Commit Graph

10083 Commits

Author SHA1 Message Date
Priyank Patel d7b4de2348 messages: Add support for passing user IDs for pm-with clause.
We also document support for user IDs in the pm-with narrow operator.

Edited by tabbott to document on /api rather than in the /help page.

Fixes part of #9474.
2019-06-17 17:38:51 -07:00
Priyank Patel e3431976ce validator: Add validator for checking string or integer list.
This validator will be used to check if string or user IDs list is
passed in.
2019-06-17 17:33:58 -07:00
Hemanth V. Alluri b8268c66d5 webhooks/bitbucket3: Add the `diagnostics:ping` event.
Note: Unlike with the other events supported by this webhook so far,
to determine this event's event type, we need to use the X-Event-Key
HTTP Header.
2019-06-17 17:33:32 -07:00
Tim Abbott 544f9c74ce export: Use outbox emoji for managing who is exported.
This is a little more unambiguous.
2019-06-17 16:10:28 -07:00
Yashashvi Dave 0b4384f4cd custom profile fields: Extract function `validate_custom_field_data`. 2019-06-17 14:56:02 -07:00
Wyatt Hoodes e7b9b93f84 test-backend: Move `bot_avatar` to `var/<uuid>/test-backend`. 2019-06-17 13:56:34 -07:00
Wyatt Hoodes fbcb05dae6 test-backend: Move `test-log-dir` to `var/<uuid>/test-backend`. 2019-06-17 13:56:34 -07:00
Wyatt Hoodes dd339a57af test-backend: Move import/export ouput into `var/<uuid>/test-backend`. 2019-06-17 13:51:40 -07:00
Eeshan Garg f645de255c webhooks/jira: Avoid UnboundLocalErrors for event handler functions.
If the event key is None, the handler content_func never gets
defined, which leads to an UnboundLocalError. This can be easily
avoided by having a dedicated function that handles the case for
when the event key is None.
2019-06-16 15:19:53 -07:00
Hemanth V. Alluri 5a1043237c realm_plan_type: Add wide organization logo related fields to state.
Namely, here we add the "plan_includes_wide_organization_logo" and
"upgrade_text_for_wide_organization_logo" to the page_params (which
is set in zerver/lib/events.py).

"plan_includes_wide_organization_logo" is True if the plan is not of
the Realm.LIMITED type. We need to add this extra boolean parameter
instead of just using "realm_plan_type" to make things a lot easier
to work with on the frontend side, especially considering that
handlebars won't allow checking for equality in its {{#if}} blocks.
2019-06-14 15:59:28 -07:00
Hemanth V. Alluri 5d8050f822 realm_plan_type: Send an event when the realm's plan type is updated.
When a realm's plan type is updated using "do_change_plan_type" we
notify active users of the realm. This way certain plan features
could be enabled instantaneously for active users.
2019-06-14 15:59:28 -07:00
Hemanth V. Alluri 611808381f webhooks/stripe: Blacklist invoice changes under invoiceitem category. 2019-06-14 15:55:07 -07:00
Hemanth V. Alluri 325b1fd6c4 webhooks/stripe: Modify the invoice.updated event handling logic.
If the invoice was paid then the message should simply be
"Invoice is now paid." with a link to the invoice.

Also, suppress the "status_transitions" and "payment_intent"events.
2019-06-14 15:55:07 -07:00
Hemanth V. Alluri eaa89d47fa webhooks/stripe: Add fixtures for invoice and invoiceitem updates. 2019-06-14 15:55:07 -07:00
Hemanth V. Alluri 1a33d73554 management: Don't prepend "HTTP_" to non-headers in parse_headers.
Django does not prepend "HTTP_" to the "Content-Type" or
"Content-Length" headers. So parse_headers should not do that either.
2019-06-14 15:53:59 -07:00
Wyatt Hoodes 6b033c7909 test-backend: Add steps to deal with potential database leaks.
A function was written in `test_fixtures.py` to drop a test database
template if the corresponding database id doesn't belong to a file.
Alongside this fact, every file that is written is removed after 60
minutes.  Meaning any potential database template can never exist
longer than one hour.

This follow-up work was added to deal with the potential race
conditions when running `test-backend`.  Ensuring that all templates
are properly dealt with.

Essentially rewritten by tabbott for cleanliness.

Fixes the remainder of #12426.
2019-06-14 15:23:25 -07:00
Wyatt Hoodes 0b05d91e62 test_runner: Write database ids to file for reference in clean up.
The ids that will be used for each particular run of the test suite are
written to a unique file. Each file will then be used as a time
reference of when the suite was ran.

This change sets up the ability for a complete clean up of potentially
leaked database templates.

Tweaked by tabbott to remove these files after successful database
cleanup.
2019-06-14 15:23:20 -07:00
Wyatt Hoodes 900d9d3e49 test_runner: Add comment explaining abnormal database destruction.
When running the test-backend suite in serial mode, `destroy_test_db`
double appends the database id number to the template if passed an
argument for `number`.  The comment here explains this behavior.
2019-06-14 14:35:39 -07:00
Vishnu Ks cc91e6cb94 auth: Allow setting GOOGLE_OAUTH2_CLIENT_ID from dev-secrets.
This makes it much more convenient to use Google/GitHub authentication
in a Zulip development environment for testing; one only has to set it
up once.
2019-06-14 10:16:26 -07:00
Vishnu Ks 16a345a766 auth: Document setting social_auth_github_key in dev-secrets.
This is a simpler configuration method for development, because
dev_settings.py isn't something the developer can edit.
2019-06-14 10:16:26 -07:00
vinitS101 a6eda858d0 ldap: Fix avatar sync not working with the S3 backend.
This fixes an issue that caused LDAP synchronization to fail for
avatars.  The problem occurred due to the lack of a 'name' attribute
on the BytesIO object that we pass to the upload backend (which is
only used in the S3 backend for computing Content-Type).

Fixes #12411.
2019-06-13 15:12:13 -07:00
Yago González 57c80669a0 api docs: Remove migrated endpoints from deprecated specs. 2019-06-13 14:31:55 -07:00
Yago González c9920ab902 api docs: Add explanatory note to zulip-2.0.yaml. 2019-06-13 14:31:55 -07:00
Mateusz Mandera 8f15884c7d retention: Delete objects tied to a Message in one query with archiving.
Rather than relying on the CASCADING property of the ForeignKey to the
Message table to clean up these objects, we delete them in the same
query as we archive them - since it's guaranteed that any of these
objects that we archive will be deleted due to their Message being
deleted later.
We don't have this guarantee for Attachment objects, which is why we
can't apply this scheme to them.
2019-06-13 11:18:11 -07:00
Mateusz Mandera 25810752fe retention: Fully process each Message chunk in a transaction.
To ensure the database retains a consistent state if archiving gets
interrupted, we process each Messages chunk together with related
objects in a single atomic transaction.
2019-06-13 11:17:54 -07:00
Mateusz Mandera 55eb46433b retention: Use yield when batching instead of returning a list of lists.
This generator architecture will be cleaner for supporting the
transactionality model we want.
2019-06-13 11:11:34 -07:00
Mateusz Mandera 37a22844b9 retention: Clean up code of move_messages_to_archive(). 2019-06-13 11:02:11 -07:00
Mateusz Mandera a68c460a14 retention: Clean up code for archiving attachment_messages.
We had two duplicate functions for archiving zerver_attachment_messages
rows, doing the same thing - archiving by message_id. One of them had a
redundant INNER JOIN, so we get rid of that too.
2019-06-13 11:02:11 -07:00
Mateusz Mandera cbee5beeac retention: Log progress through the archiving process. 2019-06-13 11:02:11 -07:00
Mateusz Mandera e3c7a5d896 retention: Loop over realms in archive_messages.
Since we loop over realms in the functions for archiving stream messages
and then personal+huddle messages, and also want to split cleaning up
attachments by realm - it makes sense to do it all in one single loop.
2019-06-13 11:02:11 -07:00
Yashashvi Dave 8e269b4651 models: Rename notification to `enable_stream_audible_notifications`.
Rename notification property `enable_stream_sounds` to
`enable_stream_audible_notifications` to match with other
notification property patterns.

Fixes part of #12304
2019-06-12 16:24:51 -07:00
Mateusz Mandera 5b8140cf75 retention: Group stream message archiving by realm.
We group the process of archiving stream message by realm, to allow
logging and keeping track of time taken per realm.
2019-06-11 09:25:25 -07:00
Mateusz Mandera f06a4b4eab retention: Batch Message archiving queries.
We batch queries that archive Messages, to limit the maximum amount of
Message objects archived in a single query. This leads to the archiving
of other related objects being batched as well, because we loop over
chunks of archived messages and archive their related objects per-chunk.
2019-06-11 09:25:25 -07:00
Yashashvi Dave f25e00bcaa zever/lib/validator: Rename function to `validate_choice_field_data`. 2019-06-06 22:36:16 -07:00
Wyatt Hoodes ccfcd6b4d8 test_runner: Fix accumulation of uniquely named test templates.
N = self.parallel templates are created, and these templates were
previously named 'zulip_test_template_<1, N>'.  However, to support
running multiple instances of `test-backend`, a unique
`random_id_range_start` was created for each template database.

There was no problem prior because the templates would simply be
used again and thus did not require any clean up. Now that there are
unique database names being created, every time `test-backend` is run
these templates can accumulate on disk.  Instead, we clean up our
templates at the end of every complete run of the test suite, or upon a
SIGINT.

Fixes: #12426
2019-06-06 14:45:32 -07:00
Wyatt Hoodes d3500867d9 test_runner: Fix small typo in comment. 2019-06-06 14:29:57 -07:00
Tim Abbott 1de4b94fbe openapi: Add validation of parameter lists against actual code.
This validation is incomplete, in large part because of the long list
of TODOs in this code.  But this test should provide a ton of support
for us in avoiding regressions as we work towards having complete API
documentation.

See https://github.com/zulip/zulip/issues/12521 for a bunch of
follow-up improvements.
2019-06-06 13:41:07 -07:00
Tim Abbott 065575debf retention: Add a quick comment explaining how deletion works. 2019-06-06 11:41:07 -07:00
Mateusz Mandera 323be57151 retention: If stream has no retention policy set, use realm policy.
We add the following behavior:
If stream has message_retention_days set to -1, archiving for it is
disabled.
If stream has message_retention_days set to null, use the realm's
policy. If the realm has no policy, we don't archive for this stream.
2019-06-06 11:17:42 -07:00
Mateusz Mandera 8bef82c7f9 retention: Clean up redundant code for special handling of UserMessages.
UserMessages no longer need special handling, they can be archived by
move_models_with_message_key_to_archive and automatically cleaned up
like the other models with a message key with CASCADING=True.
2019-06-06 11:17:42 -07:00
Mateusz Mandera 0e9fa4f028 retention: Support stream-based retention policies.
We change the archiving scheme to allow having stream based retention
policies. In the first step of the archiving process, we loop over
streams and archive their expired messages and related objects.
Then we separately archive all expired personal and huddle messages and
related objects. As the last step, we scan for redundant attachments
which can now be deleted.
To achieve this, we have to rewrite a significant portion of the
retention code and rework some of the database queries.
For the sake of simplicity, we neither archive nor delete cross-realm
messages, except cross-realm stream messages – in their case they can
be processed in the same manner as ordinary stream messages.
In the query for archiving personal and huddle messages we simply
exclude those sent by cross-realm bots.
We change the tests to adapt to these modifications.
2019-06-06 11:17:42 -07:00
Mateusz Mandera aa45325b5f retention: Rename move_expired_rows to move_rows. 2019-06-06 11:17:42 -07:00
Mateusz Mandera d481ee9a40 retention: Add message_retention_days field to Stream model. 2019-06-06 11:17:42 -07:00
Mateusz Mandera d373a16910 retention: Remove realm_id check when archiving attachments.
Since we archive attachments and attachment_messages tied to a list of
ids of Messages that we just archived (so from the current realm), it's
unnecessary to check their realm in the queries. This could potentially
cause archiving of an attachment with realm_id of another realm, but
this isn't an issue, as long as we make sure we don't end up deleting
the original Attachment object incorrectly - but realm_id check is
included in delete_expired_attachments() to ensure that.
2019-06-06 11:17:42 -07:00
Hemanth V. Alluri c6b03432bc docs: modify update-message-flag to contain information about flags.
This makes it a lot more useful for understanding how our flag update
endpoints work.

With significant edits by tabbott to explain what these are.

Fixes #12092.
2019-06-04 00:40:47 -07:00
Tim Abbott fa77467d5d api: Don't allow editing non-editable flags.
Previously, we didn't have validation to prevent editing certain flags
that don't make sense for a client to edit, like whether a user was
mentioned in a given message.

This isn't a security issue -- the user could only mess up their own
personal search results (etc.), but it does seem worth fixing to avoid
confusion for folks developing Zulip clients.

While we're at it, clearly document the situation in comments.
2019-06-04 00:33:21 -07:00
sahil839 5a130097bf settings: Add display setting for demoting inactive streams.
This adds a setting to control Zulip's default behavior of sorting to
bottom and graying out inactive streams.  The previous logic is still
the default "automatic", but this gives users more control.  See the
models.py comment for details.

Fixes #11524.
2019-06-03 23:07:56 -07:00
Tim Abbott 710fc6767f test_fixtures: Fix buggy reuse of migrations_hash path.
We were apparently reusing the path for both the development and test
databases, which meant that we would not always correctly run
`generate_fixtures` when changes were required.

This was a recent regression introduced when we added this cache a few
days ago.
2019-06-03 23:07:56 -07:00
Eeshan Garg 0d519ab146 webhooks/bitbucket2: Account for missing username in user data.
Not all payloads contain the user's username. In such cases, we
should use the user's display name or nickname instead.
2019-06-03 20:00:17 -07:00
Tim Abbott 807e5c7a1a auth: Fix fetching personal API key with email addresses hidden.
This was a corner case missed in the main migration.
2019-06-03 15:55:07 -07:00
Tim Abbott 2bd6d275a7 test_import_realm: Fix test robustness.
The previous version hardcoded assumptions about user IDs.
2019-06-02 15:00:16 -07:00
Tim Abbott a85fee333c test_import_export: Fix some spelling errors. 2019-06-02 14:57:03 -07:00
Mateusz Mandera 6c3ba25474 retention: Use RETURNING to speed up database queries.
We add RETURNING to fetch relevant message and usermessage ids in
archiving queries and use them to make other queries faster and slower.
A side-effect of this implementation is that with cross-realm messages,
the UserMessage of the recipient and the Message will not be deleted -
but cross-realm messages are rare, will still get correctly put in the
archive tables and so failing to delete should not be a problem for now.
They will be fully handled later.
2019-06-02 14:55:14 -07:00
Mateusz Mandera 426e3bbbd9 retention: Remove redundant LEFT JOIN in archiving UserMessages.
zerver_archivedmessage is already INNER JOIN-ed earlier in the query, so
we check the pub_date in it, instead of joining zerver_message, which
would just redundantly join the analogical rows.
2019-06-02 14:55:14 -07:00
Vishnu Ks a9a44392d1 tests: Add test to highlight that bs4 modifies html sometimes. 2019-06-02 14:53:13 -07:00
Vishnu Ks 8718846c2a import: Use html.parser instead of lxml in bs4.
lxml parser appends html and body tags to the soup object which
are not reqired. There are no other major parsing diffrences between
the two parsers as long the HTML input is perfectly formated.
lxml parser is much faster than html.parser but it hardly matters
in our case.
https://www.crummy.com/software/BeautifulSoup/bs4/doc/#differences-
between-parsers
2019-06-02 14:53:13 -07:00
Puneeth Chaganti 64c40287f1 url preview: Rename type_ variable to oembed_resource_type. 2019-06-02 14:31:39 -07:00
Puneeth Chaganti 30dcf805ea url preview: Use oEmbed preview for Vimeo, instead of custom code. 2019-06-02 14:31:39 -07:00
Mateusz Mandera 569d79b9d8 email_mirror: Add support for "+include-quotations" in address.
We add an option to disable the stripping of quotations from the email
body, if "+include-quotations" token is included in the email address.
2019-06-02 10:50:59 -07:00
Mateusz Mandera e4138c5463 email_mirror: Add support for "+include-footers" in address.
In addition to the "+show-sender" option, we now add "+include-footers"
which disables stripping of the footer from the email body if this token
is included in the email address.
2019-06-02 10:50:59 -07:00
Mateusz Mandera a5aa4adb54 email_mirror: Add general support for optional tokens in the address.
To enable a comfortable way of adding more optional tokens in the
address (like current '+show-sender') we change decode_email_address to
return a general dictionary containing options specified through adding
these optional tokens in the To: address. For now, we only have
"+show-sender", but more can be easily added using this change.
2019-06-02 10:50:59 -07:00
Wyatt Hoodes db69cdbcde public_export: Add support for deleting export after access.
The RealmAuditLog object ID was stored in the event sent to the
deferred_work queue as a means to update the row's extra_data field.
The extra_data field then stores the location of the export.
2019-05-31 22:54:27 -07:00
Wyatt Hoodes 4a79be331e test_fixtures.py: Use hashing to optimize migrations check.
Instead of running `what_to_do_with_migrations` unconditionally, we
first hash and compare the files located in `*/migrations/*`. Only if
a migration file has changed (or the hash file does not exist yet) do we
call `what_to_do_with_migrations`.

It was discovered that the call to Django's `showmigrations.py` file was
causing roughly a 500ms increase in `test-backend`'s start up time.

However, this fix only saves about 100ms, apparently because a lot of
that work was importing Django dependnecies we need for most tests
anyway.

Fixes: #12428.
2019-05-31 17:44:17 -07:00
Eeshan Garg cf921d5981 webhooks/buildbot: Add missing "skipped" build result code.
The payload for when a build is cancelled was causing an error
because the build result code mapping was missing one of the
codes. This commit also fixes a minor typo in the result codes.
2019-05-31 16:59:42 -07:00
Puneeth Chaganti 9aa5a2b369 url preview: Use oEmbed html for videos.
Ensure that the html is safe, before using it. The html is considered if it is
in an iframe with a http/https src, based on the recommendations here:
https://oembed.com/#section3

We directly embed the `iframe` html into the lightbox overlay.
2019-05-31 15:59:03 -07:00
Puneeth Chaganti c8cb785950 url preview: Show inline images as previews for oEmbed photo pages. 2019-05-31 15:59:03 -07:00
Puneeth Chaganti 8c0c9ca7a4 url preview: Turn Realm.inline_url_embed_preview off by default. 2019-05-31 15:28:32 -07:00
Mateusz Mandera f73600c82c rate_limiter: Create a general rate_limit_request_by_entity function. 2019-05-30 16:50:11 -07:00
Puneeth Chaganti 22d0cd9696 url preview: Don't cache embed data when fetch has network errors. 2019-05-30 16:45:22 -07:00
Vaibhav 20d4aa96a9 user_info: Add `is_cross_realm_bot` to user info dataset. 2019-05-30 15:43:18 -07:00
Vishnu Ks 55bf44152a import: Handle hidden_by_limit case for files in slack import.
Fixes #12011
2019-05-30 12:01:09 -07:00
Mateusz Mandera 4facc93670 retention: Add archiving of SubMessages. 2019-05-30 11:40:20 -07:00
Mateusz Mandera 37c42a09e5 retention: Archiving of models tied to a Message, applied to Reactions.
We add general code that will archive models that are tied to a specific
Message (such as Reactions and SubMessages). Certain details of the
model are grabbed from a list models_with_message_key, and then used to
create queries that will archive these database tables.
We put Reaction in that list in this commit, and add appropriate tests.
To have archiving of other analogical models (for example SubMessage),
one only needs to make an appropriate entry in the
models_with_message_key list.
2019-05-30 11:40:20 -07:00
Mateusz Mandera dfee559333 test_retention: Check that Reactions get correctly deleted. 2019-05-30 11:33:41 -07:00
Mateusz Mandera 29729b7748 test_retention: Check that SubMessages get correctly deleted. 2019-05-30 11:27:38 -07:00
Mateusz Mandera 6d69405f54 test_retention: Keep helper functions in a base class. 2019-05-30 11:27:38 -07:00
Mateusz Mandera 2370e6717c test_retention: Factor out _make_expired_zulip_messages helper function. 2019-05-30 11:27:38 -07:00
Mateusz Mandera 29529cf2e7 retention: Add ArchivedSubMessage model. 2019-05-29 16:26:11 -07:00
Mateusz Mandera 292b4bb0d7 retention: Add ArchivedReaction model. 2019-05-29 16:26:11 -07:00
Mateusz Mandera 2bc6d52c72 retention: Fix name of move_attachment_message_to_archive_by_message.
The first instance of the word "message" should be in plural. We rename
to move_attachment_message_to_archive_by_message.
2019-05-29 16:26:11 -07:00
Mateusz Mandera 2ca650be4d retention: Clean up move_messages_to_archive() for more clarity. 2019-05-29 16:26:11 -07:00
Tim Abbott fa37c5cee1 docs: Extract a Writing Documentation top-level section.
This should make this easier to find, and also makes "Subsystems" a
bit smaller of a catch-all.
2019-05-29 15:52:11 -07:00
Tim Abbott 7134a12231 pgroonga: Fix do_batch_update logic for all postgres versions.
Apparently, the fix in 430ed061c1
requires a sufficiently modern postgres version not available in
Xenial.

Fixes #12382.

Patch by Sutou Kouhei.
2019-05-29 14:58:44 -07:00
Eeshan Garg cecea75457 api_docs: Detect missing arguments in curl examples.
This commit adds automated tests that make sure that every curl
example command in our API docs has the '-X (POST|GET)' argument.

Fixes: #11927
2019-05-28 16:53:48 -07:00
Tim Abbott 8339c21637 test-backend: Fix db issues with running two copies in parallel.
Sometimes it's useful to run two copies of test-backend at the same
time.  The problem with doing so is that we need to make sure no two
threads are using the same test database ID.

Previously, this worked only if at most one of those copies was
running in the single-threaded mode, because we used a random database
ID for the single-threaded code path, but the same IDs counting from 0
for the parallel code path.

Fix this, mostly, by generating a random start for the range of IDs
used by the process, and then counting off database IDs starting from
there (both in the parallel and non-paralllel modes).

There's still a very low probability race, see the TODO.
Additionally, there appear to be some other races with running two
copies of test-backend at the same time not related to the database.

See https://github.com/zulip/zulip/issues/12426 for a follow-up issue
that's sorta created by this.
2019-05-28 16:38:45 -07:00
Tim Abbott 095ff28277 test-backend: Avoid dropping/rebuiling zulip_test database.
The test-backend parallel test runner system doesn't actually use the
zulip_test database; instead, it creates its own databases off the
zulip_test_template database.

We were accidentally running `tools/generate_fixtures` even when there
are no changes, because this function is shared with the
tools/lib/test_server.py codebase, which needs us to do the work of
creating a test database for it off the zulip_test_template database.

Fixing this saves about 1.5s / 4s of the runtime of a single test.
2019-05-28 16:38:45 -07:00
Tim Abbott e0afdb675f test_fixtures: Rename and document update_test_databases_if_required.
The previous name was confusing because `generate_fixtures` isn't
actually a good name in the first place.
2019-05-28 16:38:45 -07:00
Vishnu Ks 31151dadbf import: Replace data-user-group-id in rendered_content.
See the data-user-id commit for details.
2019-05-28 12:53:20 -07:00
Vishnu Ks ce1d6044db import: Replace data-stream-id in rendered_content.
See the data-user-id commit for details.
2019-05-28 12:53:20 -07:00
Vishnu Ks cb5b3f347b import: Replace data-user-id in rendered_content with new user id.
Previously, if you exported a Zulip organization and then re-imported
it, we'd end up renumbering the user IDs and all direct foreign key
references to them in the database, but not the data-user-id
references in mentions.  Fix this by parsing the message content and
doing that renumbering.

(Because we import raw markdown, not HTML, from third-party tools,
these changes won't affect data import from slack etc.)

Fixes the high-priority part of #11293.
2019-05-28 12:53:19 -07:00
Vaibhav 61371cbe9a user_info: Add `bot_owner_id` to user info dataset.
Modifies the dict with the user info to include the key `bot_owner_id`
so it can be displayed in the user info popover.

Tests concerned with changing bot owner have been modified to have
number of events=2 because while updating the bot info, two events
are fired -- updating the `realm_bot` and `realm_user` since the
key `bot_owner_id` is a part of realm user info.
2019-05-28 12:41:52 -07:00
Anders Kaseorg 802d3dbbf4 authenticate: Use keyword-only parameters.
Since positional arguments are interpreted differently by different
backends in Django's authentication backend system, it’s safer to
disallow them.

This had been the motivation for previously declaring the parameters
with default values when we were on Python 2, but that was not super
effective because Python has no rule against positional default
arguments and that convention for our authentication backends was
solely enforced by code review.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-05-27 23:49:54 -07:00
Anders Kaseorg 082f23a659 authenticate: Remove default values for required parameters.
It is now the caller’s responsibility to check that realm is not None.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-05-27 23:47:22 -07:00
Anders Kaseorg 725582850f login_or_register_remote_user: Remove unused invalid_subdomain parameter.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-05-27 23:47:22 -07:00
David Wood 2e9e653af1 webhooks/jira: Fix markup conversion of full links.
This commit modifies the regex used when parsing JIRA's full links of
the form `[text|link]` so that if you have two in a message, Zulip
markup conversion doesn't think that the first link extends to the
closing `]` of the second link.
2019-05-27 23:41:11 -07:00
Wyatt Hoodes 5c82c52b52 export.py: Clean up redundant import statements.
There existed duplicate import statements for the S3 backend as a
result of the prior refactoring work.
2019-05-27 20:13:56 -07:00
Wyatt Hoodes 46790b3280 public_export.py: Reorder the creation of the RealmAuditLog object.
This reordering was originally made with regard to the delete after
access feature for the public export.  However, this reordering is
more correct overall, i.e., the object should be created before the
event pertaining to the object is sent.
2019-05-27 20:07:28 -07:00
Wyatt Hoodes 55ff497099 test_realm_export.py: Remove unnecessary variable assignments.
The `queue_data` variable is an intermediate step that's unnecessary.
Instead, the values from the queue event are assigned dierectly.

Also, the `worker` variable is not worth an assignment as it is only
referenced a single time per test case.
2019-05-27 20:07:28 -07:00
Wyatt Hoodes d32ce0ed7e test_realm_export.py: Clean up the mocking of `do_export_realm`.
A FileNotFound error was set as the side-effect of the do_export_realm
mock and the DeferredWorker was made to consume the event explicitly.

Previously, the mock of do_export_realm was producing spammy output
as a result of a FileNotFound error coming from the queue processing of
`do_write_stats_file_for_realm_export`.
2019-05-27 20:07:28 -07:00
Wyatt Hoodes d3a39d6a5b test_realm_export.py: Remove the `create_s3_buckets` call.
We don't need to create any s3 buckets before hand in this test
case, as we don't get to a point in the testing flow where this
is necessary.
2019-05-27 20:07:28 -07:00
Wyatt Hoodes 551458e776 test_realm_export.py: Remove setUp method to match standard style.
The setUp method for two basic lines of code was removed to match
the typical style of Zulip's unit tests.
2019-05-27 20:07:28 -07:00
Wyatt Hoodes c0ef6c2fc6 export: Add LOCAL_UPLOADS_DIR support to the export feature.
A unique path was created using the `LOCAL_UPLOADS_DIR` backend, similar
to the code used in `LocalUploadBackend`.  The exported tarball was
copied to the directory, and an nginx url was created to serve the file
publicly.

Tweaked by tabbott to output an actual URL.
2019-05-27 20:06:35 -07:00
Mohit Gupta a98447b312 bots: Bots can post to announcement-only streams if their owner can.
Bot owned by a non-admin gets blocked but bots owned by an admin
can post to announcement-only stream.

Fixes: #12310.
2019-05-27 18:43:07 -07:00
Mohit Gupta d60f6c9ad9 test_messages: Fix buggy test, bot tests must use /api/v1 endpoint. 2019-05-27 18:43:07 -07:00
Mayank Madan 7fedcbd840 tests: Extract and use assert_logged_in_user_id test helper.
This cleans up the pattern for how we check which user is logged in
during Zulip's backend unit tests to be much more readable (replacing
the arcane session code that does this check).
2019-05-27 18:32:27 -07:00
Mateusz Mandera 0bf90be886 retention: Clean up and rewrite test_retention.py.
test_retention.py had various issues - we opt for keeping its essence
(what should the tests do and verify), but rewriting a lot of it in
order to have more clarity in what's happening there.
2019-05-27 12:53:32 -07:00
Mateusz Mandera c5ac66b9c8 retention: Split archive_messages code into two functions.
We split archive_messages code into two functions: moving to archive and
cleanup. This allows cleaning up the tests - they can call
these functions directly instead of copying several lines of
archive_messages here and there in multiple tests.
2019-05-27 12:53:32 -07:00
Mateusz Mandera db86043195 test_retention: Quick fix for the remaining test failure.
test_cross_realm_messages_archiving_two_realm_expired doesn't run the
code path patched in commit 3d1aa98b2ea344fba7fbb2373a37d4cf30f53e08i,
so it can still fail. We apply the analogical change in the test as
in the cited commit.
2019-05-22 14:15:18 -07:00
Tim Abbott 3d1aa98b2e retention: Use a consistent ordering for processing realms.
This is probably a good idea for the production use case, since then
there's some consistency of behavior, and if we extend logging, one
knows exactly which realms were or were not executed before a logged
failure.

This fixes the nondeterministic test failures we've been seeing in CI:
if you use `-id` in that order_by, it happens consistently.
2019-05-22 10:48:53 -07:00
Yashashvi Dave 3e50ed2075 org settings: Add organization profile preview option.
This should make it convenient and obvious how verify that their
organization profile looks nice after being markdown-rendered.

Fixes #12105.
2019-05-21 17:53:34 -07:00
Mayank Madan d5e61e20e5 dev tooling: Add "create new user" and "create new realm" buttons.
Significantly tweaked by tabbott to clean up and expand the tests.

Fixes: #6018.
2019-05-21 15:03:37 -07:00
Tim Abbott 6a42280e31 auth: Fix devlogin "All realms" view.
This was apparently accidentally broken (making it 500) by the
refactoring in 9efda71a4b.
2019-05-21 14:46:15 -07:00
Vishnu Ks 9273d0e7be tests: Check for private message special case in consented realm export. 2019-05-21 14:10:29 -07:00
Vishnu Ks 0a466914c4 tests Add PM tests for test_export_realm_with_exportable_user_ids. 2019-05-21 14:10:29 -07:00
Vishnu Ks 044d620ee0 tests Add PM tests for test_zulip_realm. 2019-05-21 14:10:29 -07:00
Vishnu Ks 5a8ddffb0b tests: Split export realm with exportable_user_ids into seperate test. 2019-05-21 14:10:29 -07:00
Vishnu Ks 3f24cc4b86 tests: Move find_by_id to ZulipTestCase. 2019-05-21 14:10:29 -07:00
Vishnu Ks 68387b9f00 tests: Move get_set to ZulipTestCase. 2019-05-21 14:10:29 -07:00
Vishnu Ks cc88f95229 tests: Test for PMs in test_export_realm_with_member_consent.
Sending PM from a hamlet(consented) to othello is a case
of sending message from a consented user to a non consented
user. This result in the generation of more than one message
files during realm export. To handle this case _export_realm
is updated.
2019-05-21 14:10:29 -07:00
Vishnu Ks 21e7763886 export: Remove unnecessary query from export_partial_message_files.
The query is not required anymore after the refactoring done
while merging #12225.
2019-05-21 14:10:29 -07:00
Vishnu Ks 319fe62d44 test: Fix third huddle variable name in test_import_export. 2019-05-21 14:10:29 -07:00
Tim Abbott bde9b28589 test_retention: Update debugging code for CI failures.
This should provide more helpful output for the next stage of
debugging.
2019-05-21 14:10:15 -07:00
Tim Abbott 3de3cb7385 tests: Remove accidentally added empty test file. 2019-05-21 10:03:01 -07:00
Tim Abbott 55b15ba117 test_retention: Improve and extent print-debugging.
We needed flush=True to have output not be lost.

Also print the original messages, so we can compare what's missing.
2019-05-21 09:28:03 -07:00
Tim Abbott 3996b62c79 test_messages: Fix logging in as a bot user.
Bot users can't login, but apparently one could force it with the
Django test system's login function.  We should just post to the API.
2019-05-20 20:21:35 -07:00
Wyatt Hoodes 4dd8c133a9 export: Rename `--upload-to-s3` to be `--upload`.
The upload option will no longer be limited to strictly S3 uploads. This
commit serves as a preliminary step for supporting LOCAL_UPLOADS_DIR as
part of the public only export feature.
2019-05-20 19:59:57 -07:00
Tim Abbott 1353e94b29 test_retention: Add print-debugging.
We've been seeing nondeterministic failures in this test suite in CI
that we can't reproduce locally; these print statements should help
track them down.
2019-05-20 19:43:28 -07:00
Mateusz Mandera 4f9bbe9d66 email_mirror: Clean up and expand test_get_missed_message_token.
This is the only function in TestEmailMirrorLibrary, so we rename this
class to more appropriate TestGetMissedMessageToken, clean it up a bit
and add some extra checks to finally get email_mirror.py to 100% test
coverage.
2019-05-20 19:35:32 -07:00
Mateusz Mandera a0efd76f4e email_mirror: Rewrite log_and_report and cover it with tests.
log_and_report and its helper functions were mostly old code no longer
well adapted to how email mirror works currently, as well as having no
test coverage. We rewrite this part of the email to report errors in a
similar manner, and add tests for it. We're able to get rid of the
clunky and now useless debug_info dictionary in process message, as
log_and_report only needs the recipient email in its third argument.
2019-05-20 19:35:32 -07:00
Mateusz Mandera 2adcdd0c25 email_mirror: Don't pass debug_info to process_stream_message.
The only place in which process_stream_message used debug_info was to
set the 'stream' key, which would only be used if ZulipEmailForwardError
was raised after this line in the code - which is impossible, because after
that line only send_zulip (which doesnt raise this exception) and
logger.info get called, then process_stream_message successfully returns
and then process_message succesfully returns as well. So this debug_info
code wasn't doing anything. We remove it.
2019-05-20 19:35:32 -07:00
Aditya Bansal 43591d6c71 archives: Display globe icon for web public streams.
In this commit we start reserving the globe icon for web public
streams and replace instances of mixed usage of globe for public
streams with a '#'.
2019-05-20 18:13:31 -07:00
Joshua Pan c28c301506 org_settings: Display delivery_email to admins always is admin UI.
Mostly rewritten by Tim Abbott to ensure it correctly implements the
desired security model.

Administrators should have access to users' real email address so that
they can contact users out-of-band.
2019-05-20 18:13:31 -07:00
Tim Abbott 07856ad648 api: Override client_gravatar for EMAIL_ADDRESS_VISIBILITY_ADMINS.
Clients won't have access to user email addresses, and thus won't be
able to compute gravatars.

The tests for this are a bit messy, in large part because our tests
for get_events call subsections of it, rather than the main function.
2019-05-20 18:13:31 -07:00
Tim Abbott bcc6949461 zilencer: Add better error handling for IntegrityError.
This provides a clean warning and 40x error, rather than a 500, for
this corner case which is very likely user error.

The test here is awkward because we have to work around
https://github.com/zulip/zulip/issues/12362.
2019-05-20 17:53:43 -07:00
Rishi Gupta fbc2239f73 keyboard shortcuts: Update description for S. 2019-05-20 15:10:11 -07:00
Puneeth Chaganti bdf8183f55 open graph: Uploaded realm icon URL is absolute for some backends.
The `LocalUploadBackend` returns a relative URL, while the `S3UploadBackend`
returns an absolute URL. This commit switches to using `urljoin` to obtain the
absolute URL, instead of simply joining strings.
2019-05-20 13:14:28 -07:00
Hemanth V. Alluri 1a12e112d9 devtools: Completely overhaul the frontend for the integrations devtool.
This commit also adds a small functionality change where the results of
each webhook fixture message sent is now displayed to the user.

With a small tweak by tabbott to fix a styling bug.

Fixes #12122.
2019-05-20 12:36:15 -07:00
Hemanth V. Alluri ef98211f68 devtools: Add support for non-json fixtures for the integrations tool.
Note: If you're going to send fixtures which are not JSON or of the
text/plain content type, make sure you set the correct content type
in the custom headers.

E.g. For the wordpress fixtures the "Content-Type" should be set to
"application/x-www-form-urlencoded".
2019-05-20 12:31:52 -07:00
Hemanth V. Alluri 8214d65336 devtools: Add support for send_all in the integrations dev panel.
Using this feature a reviewer can easily send and view all fixture
messages for any given integration - with only JSON fixtures that is.
2019-05-20 12:29:10 -07:00
K.Kanakhin e930851d16 retention-period: Add more core code for retention policy.
This is a very old commit for #106, which has been on hiatus for a few
years.  It was significantly modified by tabbott to:
* Improve coding style and variable names
* Update mypy annotations style
* Clean up the testing logic
* Update for API changes elsewhere in our system

But the actual runtime code is essentially unmodified from the
original work by Kirill.

It contains basic support for archiving Messages, UserMessages, and
Attachments with a nice test suite.  It's still not usable in
production (e.g. it will probably break Reactions, SubMessages, etc.),
but upcoming commits will address that.
2019-05-19 20:22:47 -07:00
Tim Abbott ec4221abb9 actions: Return message ID from internal_send_*_message.
This is handy for code that needs to do something with the sent
message.  We need it for a retention policy code path, but it seems
likely we'll use it a lot down the line.
2019-05-19 19:18:06 -07:00
Hemanth V. Alluri 2bd9c8cb42 devtools: Add custom HTTP headers support to the integrations dev panel.
This commit introduces a simple field where the user can now specify custom
HTTP headers. This commit does not introduce an improved system for storing
HTTP headers as fixtures - such a change would modify both the existing unit
tests as well as this devtool.
2019-05-17 11:50:53 -07:00
Hemanth V. Alluri 64b4fd5923 management: Extract parse_headers in send_webhook_fixture_message.
This way other devtools such as the integrations development panel
could also use the method.
2019-05-17 11:50:53 -07:00
Puneeth Chaganti 5f50c4d799 cache: Clear realm descriptions when description is changed.
Clear both the rendered and the text descriptions for a realm, when the
description of the realm is changed.
2019-05-17 11:31:14 -07:00
Puneeth Chaganti f084075c2a cache: Clear rendered realm text description when flushing realm.
The commit 2d9c5b3575 introduced this cache key,
and didn't setup clearing of the cache correctly.
2019-05-17 11:31:14 -07:00
Anders Kaseorg cdd09d0441 meta_tags: Remove twitter:title, twitter:description, twitter:image.
Twitter falls back to og:title, og:description, og:image, which we set
identically, so these are redundant.

https://developer.twitter.com/en/docs/tweets/optimize-with-cards/overview/markup.html

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-05-16 21:19:08 -07:00
Hemanth V. Alluri bae8295c52 devtools: Add integrations dev panel.
This commit adds a new developer tool: The "integrations dev panel"
which will serve as a replacement for the send_webhook_fixture_message
management command as a way to test integrations with much greater ease.
2019-05-15 13:07:44 -07:00
Vishnu Ks 06983298ba export: Add support for exporting realm with member consent.
This lets us handle directly in our tooling the user experience that
we document for exporting a realm with member consent (before, it
required unpleasant manual work).
2019-05-15 12:35:32 -07:00
Vishnu Ks 8ebdbea4d5 tests: Remove commented out print statements in test_import_export. 2019-05-15 11:57:31 -07:00
vinitS101 611f1f8fd2 left_sidebar: Add "+Add streams" to bottom of streamlist.
Added a new button at the bottom of the stream list which redirects
users to '/#streams/all' where they can create new streams or subscribe
to new streams.
The button is not visible to guests.

Fixes #11642.
2019-05-14 16:12:51 -07:00
Tim Abbott dc2aa031f6 linkifiers: Allow semicolons in target URLs.
This fixes another user report.  The right fix is to convert this to
use a normal URL validator after substitution.
2019-05-14 11:17:55 -07:00
Puneeth Chaganti 4ac9778d69 url preview: Catch network errors during get for page content.
We may be successfully able to get the page once, to get the content type, but
the server or network may go down and cause problems when fetching the page for
parsing its meta tags.
2019-05-13 13:55:00 -07:00
Puneeth Chaganti 9fd1c40bb1 url preview: Timeout requests after 15 seconds. 2019-05-13 13:54:59 -07:00
Puneeth Chaganti 0b76b16101 url preview: Set a custom user agent for requests.
Some sites seem to block the default user agent of the requests
library. Using a custom user agent lets us show previews for some of
these sites.
2019-05-13 13:54:43 -07:00
Puneeth Chaganti 59555ee7e5 url preview: Confirm content-type before trying to show previews.
Currently, we only show previews for URLs which are HTML pages, which could
contain other media. We don't show previews for links to non-HTML pages, like
pdf documents or audio/video files. To verify that the URL posted is an HTML
page, we verify the content-type of the page, either using server headers or by
sniffing the content.

Closes #8358
2019-05-13 13:45:17 -07:00
Tim Abbott 0f2d7a354f realm_filters: Allows more use of & and friends in URLs.
We had some excessively tight rules about what characters were
allowed, which in particular prevented using `?foo=bar&baz=quux`
structures in the realm filters URLs.

Fixes #12239.
2019-05-13 13:31:09 -07:00
Yashashvi Dave 6f6e87f45d models: Rename 'Jitsi' to 'Jitsi Meet' in Realm model.
Fixes #12293
2019-05-13 12:08:04 -07:00
Yashashvi Dave 3efead6f5f /json/realm: Return error on not-avaliable video chat provider value.
This adds some missing error handling from the video chat provider
code path.
2019-05-13 12:07:44 -07:00
Yashashvi Dave cb85ca8601 models: Alter video_chat_provider field type to integer.
Migration rewritten by tabbott because it did not work.
2019-05-13 12:02:28 -07:00
Yashashvi Dave d7f9b7eea3 models: Add dict for video chat providers in Realm model. 2019-05-13 11:34:34 -07:00
Yashashvi Dave bbff797a93 zerver/tests/test_realm: Fix not-avaliable video-chat-provider value. 2019-05-13 11:34:34 -07:00
Tim Abbott 45305c93ae auth: Fix Apache SSO port number in confirmation links.
We had a report in the thread around
https://chat.zulip.org/#narrow/stream/31-production-help/topic/Apache-based.20SSO/near/741013
that confirmation links were taking the user to the /register form on
the Apache server, which of course doesn't work because the Apache
server architecture we have is intended to only serve a single
endpoint, /accounts/login/sso, and not any static assets (etc.).

This manifested as users getting a broke page with a bunch of JS
errors about missing static assets when trying to sign up for an
account.  The right fix is to ensure that we serve these confirmation
links (and maybe in the future, redirects) to the nginx server.
2019-05-13 11:14:41 -07:00
Puneeth Chaganti 5dee17dca0 bugdown: Show previews for manually created youtube playlists.
Youtube playlists can be created by manually listing video_ids, as follows:
https://youtube.com/watch_videos?video_ids=vid1,vid2,vid3. This commit adds
previews for URLs of this type, using the first video ID.
2019-05-12 22:24:42 -07:00
Puneeth Chaganti a1f0713b2c bugdown: Show previews for youtube playlist URLs, if possible.
If a youtube playlist URL has a video-id, we show a preview for the URL.

Closes #8562
2019-05-12 22:24:42 -07:00
Puneeth Chaganti 4de261c2de bugdown: Don't show previews for youtube URLs without video ids.
`youtube.com/playlist?list=<list-id>` incorrectly matches the regex since the
change in 8afda1c1bb. The regex was modified to
match URLs of the form `youtu.be/<id>` and this playlist URL incorrectly matches
with the `<id>` set to `playlist`.

This commit avoids this match by verifying that the ID is not playlist.
2019-05-12 22:24:42 -07:00
Roman Godov a50824e031 models: Rename Subscription.in_home_view field to is_muted.
This renames Subscription.in_home_view field to is_muted, for greater
clarity as to what it does just from seeing the setting name, without
having to look it up.

Also disabled an obsolete test_migrations test.

Fixes #10042.
2019-05-12 22:08:10 -07:00
Tim Abbott 5ec8f6e812 test_migrations: Disable migrations test.
Our migrations test suite only really works when no future migrations
conflict with it.
2019-05-12 22:06:17 -07:00
Mateusz Mandera f71c1e6a0c email_mirror: Add test for process_message not finding valid recipient. 2019-05-12 18:55:47 -07:00
Mateusz Mandera fb3fa7883b email_mirror: Clean up TestStreamEmailMessagesEmptyBody.
These tests have some code and comments that only used to apply when
these empty body scenarios used to raise the regular
ZulipEmailForwardError - now they raise ZulipEmailForwardUserError.
We adapt the tests to this fact and test by mocking logging.warning and
making sure it gets called with the intended warning message. This is
also needed to cover the ZulipEmailForwardUserError case with tests to
get to 100% coverage of email_mirror.py.
2019-05-12 18:55:24 -07:00
Mateusz Mandera f892ef1d3f email_mirror: Add tests for messages with attachments. 2019-05-12 18:55:24 -07:00
Mateusz Mandera ede0991ef5 email_mirror: Add tests for filter_footer. 2019-05-12 18:55:24 -07:00
Mateusz Mandera 30c588636b email_mirror: Add test for stream missed message emails. 2019-05-12 18:55:24 -07:00
Mateusz Mandera c2e5661f69 email_mirror: Add test for redis_client.hmget returning None values.
We add a test for the case "if not all(val is not None for val in result):"
on result returned by redis_client.hmget in send_to_missed_message_address.
2019-05-12 18:55:24 -07:00
Mateusz Mandera 2e9dfc1739 email_mirror: Add test to cover empty body after footer stripping. 2019-05-12 18:55:24 -07:00
Puneeth Chaganti 62d9ad534c digest: Trigger additional query to make tests more deterministic.
A couple of tests asserted that the number of queries were within a range,
because they ran one additional query when they were run individually, as
compared to running all the tests in `TestDigestEmailMessages`. We now trigger
these additional queries within the tests, to make the tests deterministic and
assert that the number of queries is a number, instead of a range.
2019-05-09 15:10:05 -07:00
Puneeth Chaganti ab2850c225 digest: Re-enable digest emails for soft deactivated users.
Digest emails were disabled for soft deactivated users, since UserMessage
objects are created for such users lazily when they return.

We now compute the message list for gathering hot conversations by looking at
all the messages sent to the streams where the user is subscribed, while they
were subscribed.

Fixes #6297
2019-05-09 15:10:05 -07:00
Puneeth Chaganti 6abed82fb9 digest: Use one hour cutoff to generate digest emails in test.
Otherwise, the test may flake on a slow/hosed machine, where simulating a
conversation takes longer than 1 sec.
2019-05-09 15:10:05 -07:00
Pragati Agrawal a838de63d8 settings: Add option for fixed/fluid width.
This adds a new option for "fluid width" under `Display settings` section
of SETTINGS/DISPLAY SETTINGS tab.

Fixes: #11844.
2019-05-09 13:57:57 -07:00
Eeshan Garg 49ac516b50 webhooks/pivotal: Use proper punctuation. 2019-05-09 13:05:49 -07:00
Eeshan Garg 550ea64b4d webhooks/trello: Use proper punctuation. 2019-05-09 13:05:49 -07:00
Eeshan Garg b434cc7054 webhooks/taiga: Use proper punctuation and new style string formatting. 2019-05-09 13:05:49 -07:00
Eeshan Garg eaf1c4d8b2 webhooks/teamcity: Improve punctuation and message formatting. 2019-05-09 13:05:49 -07:00
Eeshan Garg 6d62522ff0 webhooks/raygun: Improve punctuation and message formatting. 2019-05-09 13:05:49 -07:00
Eeshan Garg 44038fc431 webhooks/raygun: Raise exception for unsupported event types. 2019-05-09 13:05:49 -07:00
Eeshan Garg 5a1f70f6e8 webhooks/opsgenie: Improve punctuation and message formatting. 2019-05-09 13:05:49 -07:00
Eeshan Garg 120937a1f4 webhooks/mention: Use proper message formatting. 2019-05-09 13:05:49 -07:00
Eeshan Garg 1948924cdb webhooks/jira: Improve punctuation and message formatting. 2019-05-09 13:05:49 -07:00
Eeshan Garg ef070ef019 webhooks/github: Improve punctuation. 2019-05-09 13:05:49 -07:00
Eeshan Garg 9516699c73 webhooks/front: Use proper punctuation and fix spelling errors. 2019-05-09 13:05:49 -07:00
Mateusz Mandera 40f5755546 email_mirror: Handle case of unspecified charset in Content-Type header.
If the text part of an email message didn't specify the charset in the
Content-Type header, the text content wouldn't be found. We fix this, by
assuming us-ascii charset in those cases, as specified by RFC6657:
https://tools.ietf.org/html/rfc6657
2019-05-09 09:57:40 -07:00
Harshit Bansal b553507412 subscriptions: Migrate notification setting defaults model.
This commit migrates the Subscription's notification fields from a
BooleanField to a NullBooleanField where a value of None means to
inherit the value from user's profile.

Also includes a migrations to set the corresponding settings to None
if they match the user profile's values. This migration helps us in
getting rid of the weird "Apply to all" widget that we offered on
subscription settings page.

The mobile apps can't handle None appearing as the stream-level
notification settings, so for backwards-compatibility we arrange to
only send True/False to the mobile apps by applying those defaults
server-side.  We introduce a notification_settings_null value within a
client_capabilities structure that newer versions of the mobile apps
can use to request the new model.

This mobile compatibility code is pretty effectively tested by the
existing test_events tests for the subscriptions subsystem.
2019-05-08 17:45:10 -07:00
Tim Abbott 8a2057e3cf test_events: Print out the state when it hasn't changed.
This is often a part of what one will want when debugging.
2019-05-08 17:43:21 -07:00
vinitS101 18a424be79 uploads: Remove unusable UI elements if file uploading is disabled.
If MAX_FILE_UPLOAD_SIZE is set to 0, then UI elements like the upload
icon in the compose and message edit UI and "Attachments" menu in
"/#settings" are not displayed.
A different error message is also displayed if a user tries to drag and
drop or paste a file into the compose message box.

Fixes #12152.
2019-05-08 17:10:07 -07:00
Rishi Gupta 98da11c558 support: Rename deactive to deactivated. 2019-05-08 15:09:48 -07:00
Rishi Gupta d2b32749cc audit log: Log acting_user for do_deactivate_realm.
Currently there's no way to tell the difference between "a server admin
deactivated a realm due to it being spammy" vs "a realm admin deactivated
the realm".
2019-05-08 15:09:48 -07:00
Tim Abbott ddd3a36536 soft deactivation: Remove useless conditional.
Due to my misreading the code and a sloppy search, I thought in
8218bf101c that
all_stream_subscription_logs didn't filter for streams.

While changing this, we'll switch to using `.modified_stream_id` for
potentially better performance.
2019-05-08 14:40:33 -07:00
Puneeth Chaganti d474a41c03 digest: Turn off digest_emails_enabled flag for realms by default. 2019-05-08 14:39:12 -07:00
Shubham Padia dd28413c4a ldap: Do not modify self._LDAPUser.attrs in test_query_email_attr.
Fixes #12273.
When running the test_query_email_attr test in reverse, the test failed
because self._LDAPUser.attrs was being modified and it was being shared
with other tests.
2019-05-08 09:55:44 -07:00
Vishnu Ks ce88df5069 tests: Patch maybe_update_first_visible_message_id properly.
Otherwise this will result in error when running
tests with --parallel=1 --reverse.
2019-05-08 09:55:03 -07:00
Yashashvi Dave 02a1b2e26b server events: Add realm-time-sync event to update stream privacy.
Add event to update stream settings whenever stream privacy is changed
accordingly.

Fixes #9470
2019-05-07 17:33:59 -07:00
Eeshan Garg 3af4b7d750 webhooks/freshdesk: Improve message formatting and punctuation. 2019-05-07 16:45:01 -07:00
Eeshan Garg c4e4ceadb5 webhooks/git: Use proper punctuation for branch creation templates. 2019-05-07 16:45:01 -07:00
Eeshan Garg f1228b3550 webhooks/bitbucket3: Use proper punctuation. 2019-05-07 16:45:01 -07:00
Eeshan Garg ab8aae6d0c git_webhooks: Use proper punctuation for PR/issue messages. 2019-05-07 16:45:01 -07:00
Eeshan Garg 2d4b734415 webhooks/solano: Improve message formatting and punctuation.
This commit also removes the conditional for when a build status
does not have a corresponding emoji. In such a case, it is better
to have no emoji than displaying some boilerplate text about no
appropriate emoji being available.
2019-05-07 16:45:01 -07:00
Anders Kaseorg 9efda71a4b get_realm: raise DoesNotExist instead of returning None.
This makes the implementation of `get_realm` consistent with its
declared return type of `Realm` rather than `Optional[Realm]`.

Fixes #12263.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-05-06 21:58:16 -07:00
Puneeth Chaganti 735b6cb761 digest: Remove code to gather new users and unread pms. 2019-05-06 17:43:53 -07:00
Puneeth Chaganti 77e0154800 test_users: Add assertion that guests can't see all realm users.
This commit adds coverage for the `UserProfile.can_access_all_realm_members`
method outside of tests in `test_digest`.
2019-05-06 17:43:53 -07:00
Puneeth Chaganti be762f9485 digest: Strip down the digest email removing a lot of fluff. 2019-05-06 17:43:52 -07:00
David Wood 34d810aac3 settings: Migrate to create_stream_policy structure.
This commit replaces the `create_stream_by_admins_only` setting with a
new `create_stream_policy` setting, which mirroring the structure of
the existing `invite_to_stream_policy`.

This is important preparation for migrating the waiting period feature
to be its own independent setting.

Fixes #12236.
2019-05-06 16:27:55 -07:00
Tim Abbott 3ecdabdc77 soft_deactivation: Add temporary nocoverage to fix CI. 2019-05-06 16:14:31 -07:00
Tim Abbott cf0fc7c221 test_link_embed: Fix unused variable.
This should have been in bc2ebd0f09.
2019-05-06 16:04:37 -07:00
Puneeth Chaganti bc2ebd0f09 url preview: Refactor test code to create mock responses. 2019-05-06 12:37:32 -07:00
Puneeth Chaganti da33b72848 url preview: Use in-memory caching in dev environment. 2019-05-06 12:37:32 -07:00
Puneeth Chaganti 1f6306a5a7 url preview: Cleanup import ordering. 2019-05-06 12:37:32 -07:00
Puneeth Chaganti d56b16b275 url preview: Ignore open graph tags without a content attribute. 2019-05-06 12:37:32 -07:00
Puneeth Chaganti d02eb99831 url preview: Return generic parser <p> text as str (not bs4 string). 2019-05-06 12:37:32 -07:00
Wyatt Hoodes 2b3de816a8 test_classes.py: Add activation to default language.
Running the backend tests with a high number of processes can cause
unexpected errors with language changes.  When certain tests that change
the default language, (without explicitly overriding the teardown method
to reset the default language), interleave with other tests that are
expecting the language to be in English, discrepancies arise.

This fixes a common nondeterministic test failure with high levels of
parallelization.
2019-05-06 11:16:08 -07:00
Tim Abbott 1af4f8fe20 soft_deactivation: Add some explanatory comments.
This function still doesn't make sense in the way I'd like it to, but
this at least records what algorithm we're trying to implement.
2019-05-05 18:33:15 -07:00
Tim Abbott eb97e9fae0 soft_deactivation: Fix buggy handling of race condition.
If a soft deactivated user had a subscription double-toggled without
any new messages being sent in between, add_missing_messages might
incorrectly process those two subscription changes in the wrong order.

Fortunately, the failure mode was usually to throw this exception:

django.db.utils.IntegrityError: duplicate key value violates unique
constraint "zerver_usermessage_user_profile_id_message_id_4936d0df_uniq"
DETAIL:  Key (user_profile_id, message_id)=(4, 57) already exists.

Our unit tests actually had this precise setup some fraction of the
time, because a bit of the test setup code subscribed+unsubscribed the
target user without sending any messages in between, resulting in a
test failure something like 50% of the time.

The original exception was hard to reproduce reliably originally
(resulting in an extremely annoying nondetermnistic test failure), but
is easily reproducible by changing the "id" to "-id" in this change to
always mis-order the processing of those RealmAuditLog events.
2019-05-05 18:29:20 -07:00
Tim Abbott 8218bf101c soft_deactivation: Fix buggy handling of other streams.
Previously, our soft-deactivation logic incorrectly did not filter the
set of stream subscription changes to look at to only include the
target stream.

This could result in unspecified buggy behavior.
2019-05-05 18:29:20 -07:00
Tim Abbott 8e995deab5 soft_deactivation: Make the stream_messages mutation logic clearer.
This changes our code from something that's pretty nasty bad behavior
mutating objects during a loop to only somewhat bad behavior.
2019-05-05 18:29:20 -07:00
Tim Abbott a1d2b73790 soft_deactivation: Clarify loop logic around stream_messages.
Break will do the same thing as continue here, as each iteration will
have the same result, and it's also worth explaining why this isn't
one layer up in the loop setup.
2019-05-05 18:29:20 -07:00
Puneeth Chaganti dc1571426e open graph: Use the complete URL for open graph URLs.
Closes #12199
2019-05-05 17:51:53 -07:00
Vishnu Ks 176d37a513 tests: Make tests work in reverse order in mattermost_importer. 2019-05-04 08:49:34 -07:00
vinitS101 81b5a72252 admin_settings: Change maxfilesize to max_file_upload_size.
Renamed maxfilesize to max_file_upload_size for consistency.
Related to #12152.
2019-05-03 17:36:09 -07:00
David Wood 16e9892ab2 webhooks/jira: Include issue title in messages. 2019-05-03 20:03:31 -02:30
Vishnu Ks 123bcea518 management: Don't use sys.exit(1).
Using sys.exit in a management command makes it impossible
to unit test the code in question.  The correct approach to do the same
thing in Django management commands is to raise CommandError.

Followup of b570c0dafa
2019-05-03 14:20:39 -07:00
Tim Abbott 687f1bcdaf test_classes: Fix missing flushing of per-request caches.
We should definitely be starting each test case with an empty copy of
the per-request caches, since their intended duration is even shorter
than a request.

This was masked by the fact that these caches are automatically
flushed when one makes an actual request to the Zulip API; so the
problems were only manifesting in tests like test_events, where we
call lower-level functions that access a per-request cache without
using the Zulip API.
2019-05-03 13:52:56 -07:00
Tim Abbott 5cddf07361 test_mattermost_importer: Fix buggy use of wrong paths.
This worked for me when testing locally because it was using a
constant path, and so was accessing the product under `var/` of the
previous test run.
2019-05-03 13:30:54 -07:00
Tim Abbott 0adb93db32 test_classes: Fix data import path reuse parallelism errors.
The make_import_output_dir helper function used a path determined
primarily by the filename of the fixture being used, and expected to
have complete control over that path for the duration of the test.

This resulted in nondeterministic errors if our two test classes that
ran Mattermost import code ran at the same time.
2019-05-03 13:23:09 -07:00
Shubham Padia 884c19c3bc settings: NAME_CHANGES_DISABLED disables the name change button.
Fixes #12251.

Previously when disabling name changes in server settings, instead
of realm settings, the name edit button did not get disabled.
Changing name resulted in a message stating `no changes made`.
2019-05-03 12:58:51 -07:00
Shubham Padia 8c01f396db settings: Add AVATAR_CHANGES_DISABLED server setting.
Fixes #12132.
Realm setting to disable avatar changes is already present.
The `AVATAR_CHANGES_DISABLED` setting now follows the same
2-setting model as `NAME_CHANGES_DISABLED`.
2019-05-03 12:58:51 -07:00
Shubham Padia 4994a6c2c9 realm: Add setting to disable avatar changes.
This is useful when syncing avatars from an integrated LDAP/active
directory.

The upload avatar and delete avatar buttons are hidden if avatar
changes are disabled and the user is a non-admin.
If the user has a gravatar set, then the user will not be able to
upload an image as their avatar if avatar changes are disabled.

Part of #12132.
2019-05-03 12:52:43 -07:00
Tim Abbott b570c0dafa convert_mattermost_data: Don't sys.exit(1).
This behavior ends up being quite problematic for running unit tests;
the thing one is supposed to do in management commands is raise
CommandError.
2019-05-02 11:13:41 -07:00
neiljp (Neil Pilgrim) 7bbc620b28 mypy: Require topics to be non-None in Addressee. 2019-05-01 17:52:37 -07:00
Puneeth Chaganti bfc3e3c0c7 html_to_text: Add delimiters between text from different elements.
This module is used to render the HTML of pages like our user documentation 
into text for use in open graph previews of those articles.  It provided somewhat
confusing output in the case that there were paragraph breaks in the original message,
because text with multiple paragraphs and list items does't read very well. This commit
adds `|` as a delimiter between paragraphs, and prefixes list items with a `*`.

Closes #12228
2019-05-01 17:35:20 -07:00
okay 1694831029 bugdown: Fix double processed emoji tags inside inline tags.
When an emoji is nested inside another inline tag - like em or strong -
it was getting double processed because of the way the inlinePattern
TreeProcessor runs (it runs recursively). With this fix, we set the
inner text of the emoji span as an AtomicString, preventing us from
double processing the emoji's text.

Fixes #11621

Test Plan:

* Add test case for **😄**, verify it passes.
* Go into local dev server and send "**😄**" to self and verify the DOM
does not have double <span> tags for the emoji.
* Run zerver.tests.test_push_notifications and verify the markdown test case matches
the text_content field properly
2019-05-01 17:03:15 -07:00
Eeshan Garg 28109e0f9e webhooks/jira: Ignore comment_created message.
comment_created payloads may not contain the required issue data
to format a useful notification, therefore, it is better to handle
issue comments through issue_updated events (which we already do).

Fixes: #11995.
2019-05-01 16:32:58 -07:00
Mateusz Mandera c1ceba9037 rate_limiter: Move email_mirror limiter to use rate_limit_entity.
We change the rate limiting code in the email mirror to use the new,
general rate_limit_entity function.
2019-05-01 12:54:32 -07:00
Mateusz Mandera 40763070b7 rate_limiter: Generalise some rate_limit_user code.
We create rate_limit_entity as a general rate-limiting function for
RateLimitedObjects, from code that was possible to abstract away from
rate_limit_user and that will be used for other kinds of rate limiting.
We make rate_limit_user use this new general framework from now.
2019-05-01 12:54:32 -07:00
neiljp (Neil Pilgrim) 827c016b98 mypy: Enable strict-optional for messages.py.
This also fixes an issue where we were setting num_after to None,
which should have been 0, to disable fetching messages after the
anchor.
2019-05-01 11:24:47 -07:00
neiljp (Neil Pilgrim) 47de637c3d mirrors: Refactor create_mirrored_message_users to use new Exception.
This enables the function to either return a valid UserProfile or raise
InvalidMirrorInput, which is clearer and more pythonic than the previous
approach of a tuple of a bool and Optional[UserProfile].

In making the type clearer, this improves checking with mypy.

Tests updated.
2019-05-01 11:18:22 -07:00
neiljp (Neil Pilgrim) 3d43682f1b mypy: Enable strict-optional for auth.py. 2019-05-01 10:49:25 -07:00
Tim Abbott edb956091f export: Add a blank line in S3 upload output.
This should be more readable.
2019-04-30 16:37:23 -07:00
Tim Abbott 98a1e340ad filters: Don't include realm_counts/installation_counts in errors.
This avoids unnecessarily long error emails for these.
2019-04-30 14:10:59 -07:00
David Wood 272ed90685 settings: Create an explicit invite_to_stream_policy setting.
This commit creates a new organization setting that determines whether
a user can invite other users to streams. Previously this was linked
to the waiting period threshold, but this was both not documented and
overly limiting.

With significant tweaks by tabbott to change the database model to not
involve two threshhold fields, edit the tests, etc.

This requires follow-up work to make the create stream policy setting
work how this code implies it should.

Fixes #12042.
2019-04-29 17:11:28 -07:00
Vishnu Ks 89ada6c770 tests: Remove unecessary word from convert_data test name. 2019-04-29 11:23:17 -07:00
Vishnu Ks ecdd3bea43 billing: Add cron job to run invoice_plans once a day.
Fixes #11960
2019-04-29 11:23:17 -07:00
Puneeth Chaganti f3f172991b open graph: Use uploaded realm icon for open graph images. 2019-04-28 22:50:27 -07:00