Commit Graph

10610 Commits

Author SHA1 Message Date
Vishnu Ks ad16c55c77 docs: Make remove-subscriptions use curl example system. 2019-10-15 15:40:44 -07:00
Vishnu Ks 4364e31d69 docs: Make update-subscription-properties use curl example system. 2019-10-15 15:40:44 -07:00
Vishnu Ks 89d82ab827 openapi: Improve description of subscriptions in /users/me/subscriptions. 2019-10-15 15:40:44 -07:00
Vishnu Ks c35ef2d390 bugdown: Add option to include only selected fields in generate_curl_example. 2019-10-15 15:40:44 -07:00
Vishnu Ks 3f9835ed07 docs: Make mark-as-read-bulk use curl example system. 2019-10-15 15:40:44 -07:00
Vishnu Ks f0b099dbc0 docs: Make update-message-flags use curl example system. 2019-10-15 15:40:44 -07:00
Vishnu Ks 19ee518f2e docs: Make get-message-history use curl example system. 2019-10-15 15:40:44 -07:00
Vishnu Ks ed2e89f167 docs: Make delete-message use curl example system. 2019-10-15 15:40:44 -07:00
Vishnu Ks 0dccebbe39 docs: Make update-message use curl example system. 2019-10-15 15:40:44 -07:00
Vishnu KS 9bc3594f75 docs: Make get-raw-message use curl example system. 2019-10-15 15:40:44 -07:00
Vishnu Ks cb64fc8732 bugdown: Show example value for path params in curl example. 2019-10-15 15:40:44 -07:00
Vishnu Ks a03765bbe2 bugdown: Extract out function for getting openapi example value. 2019-10-15 15:40:44 -07:00
Rafid Aslam 718b70ec8b Rename `not_subscribed key` to `not_removed` in users/me/subscriptions.
Rename `not_subscibed_key` to `not_removed` in
`users/me/subscriptions` DELETE response.

Fixes #13277.
2019-10-13 10:30:34 +07:00
Mateusz Mandera a94b04dede test_mattermost_importer: Fix leftover references to message.pub_date.
The field is now called date_sent.
2019-10-11 04:07:12 +02:00
Tim Abbott d2970a56c2 lint: Remove some unused imports.
These were introduced in ae5bc92602.
2019-10-10 18:06:30 -07:00
Vishnu KS 1585ad7bf4 mattermost: Add support for exporting DMs and huddles. 2019-10-10 16:37:03 -07:00
Vishnu KS ae5bc92602 queue: Don't create confirmation objects twice during invite.
A confirmation object is already created when
do_send_confirmation_email is called just above.

Tweaked by tabbott to remove an unnecessary somewhat hacky database
query.
2019-10-10 16:19:42 -07:00
Mateusz Mandera 4dc3ed36c3 auth: Add initial SAML authentication support.
There are a few outstanding issues that we expect to resolve beforce
including this in a release, but this is good checkpoint to merge.

This PR is a collaboration with Tim Abbott.

Fixes #716.
2019-10-10 15:44:34 -07:00
Rohitt Vashishtha b69213808a messages: Render topic links in context of stream realm.
Priviously, we rendered the topic links using the msg.sender.realm.
This resulted in issues with Zulip's internal bots not having access
to the realm_filters of the destination stream's realm. For example,
sending a message via the email gateway or notification would not
linkify any realm filters that a user would expect them to.
2019-10-10 14:33:35 -07:00
Anders Kaseorg f25968f0ff exceptions: Fix AbstractEnum typing.
This will be needed for mypy 0.730.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-10-08 17:49:28 -07:00
Rishi Gupta 360cd7f147 remote data: Send RealmAuditLog data. 2019-10-08 17:27:29 -07:00
Rishi Gupta b86142089b billing: Sync realm activation events for on-prem billing. 2019-10-08 17:18:30 -07:00
Rishi Gupta 01bf99ec65 billing: Keep track of role count changes in RealmAuditLog. 2019-10-08 17:18:30 -07:00
Anders Kaseorg cc791afc54 requirements: Upgrade django-sendfile2.
django-sendfile2 now always sends a Content-Disposition header even if
it’s inline.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-10-08 17:15:00 -07:00
Vishnu KS cd06e0ab79 tests: Remove /messages/{message_id} from buggy endpoints. 2019-10-07 11:45:11 -07:00
Vishnu KS bfc9089124 tests: Remove /users/me/subscriptions/muted_topics from buggy endpoints.
Even though required attribute of stream and stream_id params is marked
false in openapi specification, the API expects atleast one of the
params to be set. There is no way to specify relationships like this
openapi and they dont seem to have any plan to implement this in future.

https://github.com/OAI/OpenAPI-Specification/issues/256
2019-10-07 11:42:52 -07:00
Vishnu KS e7419f815a tests: Remove /settings/notifications from buggy endpoints. 2019-10-07 11:42:52 -07:00
Vishnu KS 21ab8c542a bugdown: Cast enum elements to string in APIArgumentsTablePreprocessor.
So that enums other than of type string gets rendered without any error.
2019-10-07 11:42:52 -07:00
overide dd67b1b3bb topic: Change before bound limit from 2 to 7 days.
This limit was introduced in c588c79 as a part of the
feature and not due to performance crisis. So we are
increasing this limit to 7 days. Since topics tends to
naturally fizzle after day or two so 7 days limit
would be good enough.
2019-10-06 20:12:59 -07:00
Rishi Gupta 48dc1d1128 remote data: Refactor remote_server_post_analytics to be more generic.
One small change in behavior is that this creates an array with all the
row_objects at once, rather than creating them 1000 at a time.

That should be fine, given that the client batches these in units of
10000 anyway, and so we're just creating 10K rows of a relatively
small data structure in Python code here.
2019-10-06 16:55:41 -07:00
Rishi Gupta e10361a832 models: Replace is_guest and is_realm_admin with UserProfile.role.
This new data model will be more extensible for future work on
features like a primary administrator.
2019-10-06 16:24:37 -07:00
Rishi Gupta 4256ee61cf billing: Change RealmAuditLog.event_type from str to int.
This is a more robust long-term model for storing these data.
2019-10-06 15:55:56 -07:00
Shikhar Varshney fe5d975b33 settings: Add support for overriding APNS_TOPIC and ZULIP_IOS_APP_ID.
Documentation added by tabbott.
2019-10-05 22:53:59 -07:00
Mateusz Mandera dbe508bb91 models: Migration of Message.pub_date to date_sent, part 2.
Fixes #1727.

With the server down, apply migrations 0245 and 0246. 0246 will remove
the pub_date column, so it's essential that the previous migrations
ran correctly to copy data before running this.
2019-10-05 19:01:34 -07:00
Mateusz Mandera f198475281 models: Migration of Message.pub_date to date_sent, part 1.
1. Apply migration 0243 to add date_sent column.
2. Apply migration 0244 to copy pub_date over to date_sent. Can be done
with the server running.
3. With the server down (for consistency between memory and
database state of Django objects), verify consistency with
Message.objects.exclude(date_sent=F("pub_date")).count() == 0
2019-10-05 19:01:34 -07:00
Vishnu Ks 30ad650afb bugdown: Don't show path params as data in curl example.
With test added by tabbott.
2019-10-05 18:17:36 -07:00
Vishnu Ks bfd5f4066a bugdown: Remove unecessary call to get_openapi_parameters. 2019-10-05 18:10:55 -07:00
Vishnu Ks dff62a240b bugdown: Rename openapi_example_params to openapi_params in generate_curl_example. 2019-10-05 18:10:55 -07:00
Vishnu Ks f4e3e6068f bugdown: Rename pocket to param in generate_curl_example. 2019-10-05 18:10:55 -07:00
Mateusz Mandera 94d5ca838f social auth tests: Extract prepare_login_url_and_headers method.
It will be reused in SAML tests in upcoming commits.
2019-10-01 16:39:12 -07:00
Mateusz Mandera 0e5f964363 social auth tests: Consistently refer to hamlet's name via self.name. 2019-10-01 16:38:50 -07:00
Tim Abbott c869a3bf82 upload: Fix browser caching of uploads with local uploads backend.
Apparently, our change in b8a1050fc4 to
stop caching responses on API endpoints accidentally ended up
affecting uploaded files as well.

Fix this by explicitly setting a Cache-Control header in our Sendfile
responses, as well as changing our outer API caching code to only set
the never cache headers if the view function didn't explicitly specify
them itself.

This is not directly related to #13088, as that is a similar issue
with the S3 backend.

Thanks to Gert Burger for the report.
2019-10-01 15:15:17 -07:00
Ivan Mitev 16c9d63056 emoji: Fix sort order of emoji choices.
The previous code for ensuring the sort order of emoji choices was
correct relied on an OrderedDict structure, which isn't guaranteed to
be preserved when passed to the frontend via JSON (in fact, it isn't,
since we converted the way page_params is passed to use
sort_keys=True).  Switch it to a list of dictionaries to correct this.

Fixes #13220.
2019-10-01 13:54:55 -07:00
Mateusz Mandera 4166c901ef do_update_user_custom_profile_data: Rename to ..._if_changed.
This adds clarity to the fact that the function no longer does
anything if the field values haven't changed.
2019-10-01 13:52:43 -07:00
Mateusz Mandera d66cbd2832 do_update_user_custom_profile_data: Don't notify if value not changed. 2019-10-01 13:50:14 -07:00
Tim Abbott 2b227d6b63 models: Add comments documenting our name field decisions. 2019-09-27 18:01:56 -07:00
Tim Abbott bf088519a7 api: Add avatar URL to the GET /profile endpoint.
This endpoint is legacy in a lot of ways, but it seems reasonable that
it should have these data.
2019-09-27 12:07:03 -07:00
Vishnu Ks 1c81538570 mattermost: Read fixures inside test cases instead of setUp. 2019-09-25 20:07:00 +05:30
Vishnu KS a21856c569 mattermost: Rename user_id to sender_user_id in process_raw_message_batch. 2019-09-25 20:06:47 +05:30
Vishnu KS 23d70bb685 mattermost: Rename get_recipient_id to get_recipient_id_from_receiver_name. 2019-09-25 20:06:04 +05:30
Vishnu KS a743991935 mattermost: Fix mirror dummy fixture data.
The channel where Snape sends message should be gryffindor-quidditch-team
and not slytherin-quidditch-team since he is mirror dummy in gryffindor.
2019-09-25 19:03:43 +05:30
Tim Abbott 7e0ea61b00 upload: Support S3-compatible S3 hosting providers.
Previously, we were hardcoding the domain s3.amazonaws.com.  Given
that we already have an interface for configuring the host in
/etc/zulip/boto.cfg (which in turn, automatically configures boto), we
just need to actually use the value configured in boto for what S3
hostname to use.

We don't have tests for this new use case, in part because they're
likely annoying to write with `moto` and there hasn't been a huge
amount of demand for it.  Since this doesn't regress existing S3
backend support, it seems worth merging.
2019-09-24 17:17:21 -07:00
Tim Abbott b8b0ae362c uploads: Only initialize S3 connection once in __init__.
This should be a mild performance optimization for the S3
authentication backend, since we aren't initializing unnecessary
duplicate connections.
2019-09-24 17:15:44 -07:00
Hemanth V. Alluri 98a33c0fda page_params: Fix list of config_options sent for incoming webhook bots.
This patches an issue in f37535044 where we mistakenly tried to send
the function as part of the page_params. Instead, we should just try
to send the list of configuration options (in their user displayable
form).
2019-09-24 16:04:51 -07:00
Laurel Michaels a67769091d docs: Add OpenAPI data for `/messages/{message_id}/reactions`.
This change adds the OpenAPI data needed to document the POST and
DELETE methods associated with this endpoint.

Descriptions edited slightly by tabbott.
2019-09-24 15:57:51 -07:00
Tim Abbott 96726c00ce export: Fix broken URLs in UI with S3 backend.
Apparently, the Zulip notifications (and resulting emails) were
correct, but the download links inside the Zulip UI were incorrectly
not including S3 prefix on the URL, making them not work.

While we're at this, we rewrite the somewhat convoluted previous
system for formatting the data export output.
2019-09-24 13:56:49 -07:00
Tim Abbott 8024b1179a bots: Fix bot email addresses with EMAIL_ADDRESS_VISIBILITY_ADMINS.
When using our EMAIL_ADDRESS_VISIBILITY_ADMINS feature, we were
apparently creating bot users with different email and delivery_email
properties, due to effectively an oversight in how the code was
written (the initial migration handled bots correctly, but not bots
created after the transition).

Following the refactor in the last commit, the fix for this is just
adding the missing conditional, a test, and a database migration to
fix any incorrectly created bots leaked previously.
2019-09-23 15:53:53 -07:00
Tim Abbott 6e5c99328a models: Extract a helper function for whether emails are realm-public.
This is also a useful preparatory refactor for having a user setting
controlling whether one's own email address is publicly available
within the organization.
2019-09-23 15:53:53 -07:00
Anders Kaseorg 9339402541 messages: Sort IDs in the by_streams query deterministically.
This fixes a test flake after upgrading SQLAlchemy.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-09-23 11:56:52 -07:00
Anders Kaseorg cb0652071f zulip_otp_required: Avoid _user_is_authenticated compatibility shim.
This private shim is not needed with Django ≥ 1.10, and was removed
from django-otp 0.7.0.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-09-23 11:56:52 -07:00
Rishi Gupta 3646936cfb integrations: Advertise our native Zapier integration. 2019-09-22 18:47:42 -07:00
Tim Abbott 767a1b9f0c user_activity: Skip an unnecessary database query for new entries.
While new entries are an overwhelming minority of calls to this
function, it's still worth having a clean implementation.
2019-09-21 11:49:43 -07:00
Tim Abbott 1c73ce2450 user_activity: Use LoopQueueProcessingWorker strategy.
This should dramatically improve the queue processor's performance in
cases where there's a very high volume of requests on a given endpoint
by a given user, as described in the new docstring.

Until we test this more broadly in production, we won't know if this
is a full solution to the problem, but I think it's likely.  We've
never seen the UserActivityInterval worker end up backlogged without a
total queue processor outage, and it should have a similar workload.

Fixes #13180.
2019-09-21 11:48:24 -07:00
Tim Abbott 52a2b2c6a8 tests: Move user_activity test next to slow_query test.
This will conveniently put all the LoopQueueProcessingWorkers
together.
2019-09-21 11:47:14 -07:00
Tim Abbott f0d8951035 do_update_user_activity: Refactor to support passing a count.
We'll use this in upcoming commits.
2019-09-21 11:47:14 -07:00
Tim Abbott 5c960b3e0f user_activity: Make the queue processor a bit more efficient.
We don't actually need to go to the memcached (falling back to the
database) to fetch either user or client objects on every event.  For
user objects, we actually can just pass through the user ID
transparently; for client objects, we can use an in-process cache,
since the mapping of string to ID never changes.
2019-09-21 11:47:14 -07:00
Mateusz Mandera 3271235200 email_mirror: Ignore missed message email if the user isn't active. 2019-09-20 17:58:10 -07:00
Mateusz Mandera 21459ed193 test_email_mirror: Test missed message emails in a single ZulipTestCase.
With the way these tests are, it's unnecessary to have 3 separate
classes, and it makes it confusing to decide where to add a potential
additional mm email test.
2019-09-20 17:58:10 -07:00
Tim Abbott c19b7ec89d test_events: Remove unnecessary check_none validator.
This can be done cleanly using the existing equals(None).
2019-09-20 11:51:50 -07:00
Anders Kaseorg cf5a00d94b bulk_get_subscriber_user_ids: Sort each user list by ID.
This simple backwards-compatible change saves approximately 12% in the
compressed size of the chat.zulip.org page_params.  We can do much,
much better by changing the format, but this seems like a good
intermediate step.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-09-19 20:17:43 -07:00
Rishi Gupta 74d5d3db26 integration docs: Remove angle brackets from trello bash command.
If you accidentally leave them in, you'll get a confusing error message.
2019-09-19 17:48:16 -07:00
Yago González e1fbf6dddb actions: Make do_regenerate_api_key return the new key.
This way, the new API key can be fetched without needing to read it from
the UserProfile object.
2019-09-19 13:03:56 -07:00
Anders Kaseorg fbc2de157e templates: Move page_params to a <div> at the bottom of <body>.
In a gigantic realm where we send several MB of `page_params`, it’s
slightly better to have the rest of the `<body>` available to the
browser earlier, so it can show the “Loading…” spinner and start
fetching subresources.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-09-19 12:38:10 -07:00
Mateusz Mandera 935702b6ec auth: Rename any_oauth_backend_enabled to any_social_backend_enabled.
any_oauth_backend_enabled is all about whether we will have extra
buttons on the login/register pages for logging in with some non-native
backends (like Github, Google etc.). And this isn't about specifically
oauth backends, but generally "social" backends - that may not rely
specifically rely on Oauth. This will have more concrete relevance when
SAML authentication is added - which will be a "social" backend,
requiring an additional button, but not Oauth-based.
2019-09-19 12:35:27 -07:00
Anders Kaseorg fc0554fcb8 emoji: Finish removing leftover code from banned Emoji sets.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-09-19 12:08:52 -07:00
Vishnu Ks c4af0b7bc4 mattermost: Support importing messages without team name.
Mattermost doesn't place private messages within a particular team,
which is what this is needed for.
2019-09-18 11:57:37 -07:00
Vishnu Ks bf5f531e90 import_util: Support huddles in SubscriberHandler. 2019-09-18 11:53:13 -07:00
Rishi Gupta 2646d70cbd emails: Add realm name to footer of missed message emails.
Missed message emails for mentions come from streams and possibly orgs you
don't normally get missed message emails for, so they can be hard to place.
2019-09-17 17:45:59 -07:00
Anders Kaseorg fe7d814e8d team: Move contributors_list into page_params.
This sidesteps tricky escaping issues, and will make it easier to
build a strict Content-Security-Policy.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-09-17 16:06:33 -07:00
Anders Kaseorg 7494f1600c templates: Move page_params from an inline script to the <body> dataset.
This sidesteps tricky escaping issues, and will make it easier to
build a strict Content-Security-Policy.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-09-17 16:06:33 -07:00
Wyatt Hoodes d2c3ddc765 test_events: Use stdout_suppressed in export test. 2019-09-13 11:54:14 -07:00
Wyatt Hoodes e37c91d195 test_realm_export: Use stdout_suppressed in tests.
From a testing perspective, calling `export_realm_wrapper`
prints unnecessary data to stdout.
2019-09-13 11:54:14 -07:00
Wyatt Hoodes e3595e4b3f test_slack_message_conversion: Use subTest for fixtures test. 2019-09-13 11:54:14 -07:00
Wyatt Hoodes dbaf6ac7e7 test_push_notifications: Remove fixtures print statement. 2019-09-13 11:54:14 -07:00
Wyatt Hoodes 4a64a487d3 bugdown: Remove print statement from fixtures test. 2019-09-13 11:54:14 -07:00
Wyatt Hoodes 698b0f1b91 test_management_commands: Use subTest with help tests.
We remove the print statement, and instead use
subTest to improve debuggability.
2019-09-13 11:54:14 -07:00
Anders Kaseorg 78b018989e export: Refuse to overwrite an existing directory or tarball.
Previously, incorrectly passing an existing directory to the
`manage.py export --output` option would remove its contents without
warning.  Abort instead.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-09-12 16:04:55 -07:00
Anders Kaseorg 780ecb672b CVE-2019-16216: Fix MIME type validation.
* Whitelist a small number of image/ types to be served as
  non-attachments.
* Serve the file using the type that we validated rather than relying
  on an independent guess to match.

This issue can lead to a stored XSS security vulnerability for older
browsers that don't support Content-Security-Policy.

It primarily affects servers using Zulip's local file uploads backend
for servers running Ubuntu 16.04 Xenial or newer; the legacy local
file upload backend for (now EOL) Ubuntu 14.04 Trusty was not affected
and it has limited impact for the S3 upload backend (which uses an
unprivileged S3 bucket domain to serve files).

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-09-11 15:46:36 -07:00
Thomas Ip 574c35c0b8 markdown: Render ordered lists using <ol> markup.
This brings us in line, and also allows us to style these more like
unordered lists, which is visually more appealing.

On the backend, we now use the default list blockprocessor + sane list
extension of python-markdown to get proper list markup; on the
frontend, we mostly return to upstream's code as they have followed
CommonMark on this issue.

Using <ol> here necessarily removes the behaviour of not renumbering
on lists written like 3, 4, 7; hopefully users will be OK with the
change.

Fixes #12822.
2019-09-08 16:42:20 -07:00
Wyatt Hoodes 4a176714e1 actions: Remove leftover print statement. 2019-09-08 15:36:30 -07:00
Tim Abbott 7e75f987df ldap: Fix logging of warning for deactivated users.
Also cleans up the interface between the management command and the
LDAP backends code to not guess/recompute under what circumstances
what should be logged.

Co-authored-by: mateuszmandera <mateusz.mandera@protonmail.com>
2019-09-08 09:35:23 -07:00
Tim Abbott d1a2784d52 ldap: Fix attempting to sync data for deactivated users.
The order of operations for our LDAP synchronization code wasn't
correct: We would run the code to sync avatars (etc.) even for
deactivated users.

Thanks to niels for the report.

Co-authored-by: mateuszmandera <mateusz.mandera@protonmail.com>
2019-09-08 09:35:23 -07:00
Mateusz Mandera 2ce2024bd7 ldap: Fix unintended user deactivation in case of connection failure.
Fixes #13130.

django_auth_ldap doesn't give any other way of detecting that LDAPError
happened other than catching the signal it emits - so we have to
register a receiver. In the receiver we just raise our own Exception
which will properly propagate without being silenced by
django_auth_ldap. This will stop execution before the user gets
deactivated.
2019-09-05 11:59:20 -07:00
Mateusz Mandera dfd6771237 stream settings: Use the .show-sender version of email address.
As requested in #13134.
2019-09-05 11:48:32 -07:00
Mateusz Mandera 307d8d8758 test_email_mirror: Better document the various address formats. 2019-09-05 11:48:32 -07:00
Vishnu KS 30440cf466 mattermost: Fix test flake in test_convert_channel_data.
So the reason 38f8cf612c seems
to be flaking is because the value of harry id switches between
1 and 2 in Xenial while in Bionic it would be fixed at 2. The
reason behind this is that Bionic ships with Python3.6 which
preserves dict insert order while Python3.5 that ships with Xenial
dont preserve the order. In initialize_stream_membership_dicts
we iterate user_data_map dict and the order in which the iteration
happens affects the ID of the users.
2019-09-05 11:20:33 -07:00
Puneeth Chaganti df134be235 webhooks/papertrail: Requests from Papertrail are not JSON requests.
Papertrail sends requests with the content type
`application/x-www-form-urlencoded`, with the payload parameter holding the
JSON body. This commit fixes the papertrail integration to use the payload
parameter in the request's POST data instead of trying to parse the
request's entire body as JSON.

Papertrail documentation here:
https://help.papertrailapp.com/kb/how-it-works/web-hooks#encoding
2019-09-04 13:16:17 -07:00
Wyatt Hoodes fc2574fdfd test_mattermost_importer: Add test coverage.
This is a rebase from a recent revert. A test
was failing non-deterministically relating to
`convert_channel_data` and use of an incorrect
`realm_id`.
2019-09-03 12:36:41 -07:00
Mateusz Mandera a7ec1a089c errors: Fix infinite recursion during memcached outages.
We have a very useful piece of code, _RateLimitFilter, which is
designed to avoid sending us a billion error emails in the event that
a Zulip production server is down in a way that throws the same
exception a lot. The code uses memcached to ensure we send each
traceback roughly once per Zulip server per 10 minutes (or if
memcached is unavailable, at most 1/process/10 minutes, since we use
memcached to coordinate between processes)

However, if memcached is down, there is a logging.error call internal
to the Django/memcached setup that happens inside the cache.set() call,
and those aren't caught by the `except Exception` block around it.

This ends up resulting in infinite recursion, eventually leading to
Fatal Python error: Cannot recover from stack overflow., since this
handler is configured to run for logging.error in addition to
logging.exception.

We fix this using a thread-local variable to detect whether we are
being called recursively.

This change should prevent some nasty failure modes we've had in the
past where memcached being down resulted in infinite recursion
(resulting in extra resources being consumed by our error
notifications code, and most importantly, the error notifications not
being sent).

Fixes #12595.
2019-09-03 11:44:39 -07:00
Tim Abbott 70c513a640 analytics: Fix logging for errors connecting to push bouncer.
There's no reason for this to be a category of error that emails the
server administrator, since there's a good chance that fixing it will
need to be done in the Zulip codebase, not administrator action.
2019-09-02 18:47:10 -07:00
Mateusz Mandera d70e1bcdb7 settings: Add FAKE_EMAIL_DOMAIN setting.
Fixes #9401.

This adds a FAKE_EMAIL_DOMAIN setting, which should be used if
EXTERNAL_HOST is not a valid domain, and something else is needed to
form bot and dummy user emails (if email visibility is turned off).
It defaults to EXTERNAL_HOST.

get_fake_email_domain() should be used to get this value. It validates
that it's correctly set - that it can be used to form valid emails.

If it's not set correctly, an exception is raised. This is the right
approach, because it's undesirable to have the server seemingly
peacefully operating with that setting misconfigured, as that could
mask some hidden sneaky bugs due to UserProfiles with invalid emails,
which would blow up the moment some code that does validate the emails
is called.
2019-08-30 14:59:00 -07:00
Tim Abbott 62c9ea7cf9 linkifiers: Fix problems with capture groups called "name".
Apparently, due to poor naming of the outer capture group we use to
separate the actual match from the surrounding whitespace (etc.) we
use to determine if the syntax is a possible linkifier start/end, if
you created a linkifier using "name" as the capture group, we'd try to
compile a pattern with two capture groups called "name", which would
500, preventing anyone from accessing the organization.
2019-08-30 09:36:14 -07:00
Anders Kaseorg c3e395b7d8 search: Reimplement ts_locs_array in pure PostgreSQL.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-28 17:59:12 -07:00
Anders Kaseorg d2d4800b1c test_narrow: Compile queries with the current SQLAlchemy dialect.
We’re about to start using PostgreSQL-specific syntax that can’t be
stringified without a specified dialect.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-28 17:58:01 -07:00
rht 07808e35be parse_lsb_release: Use /etc/os-release instead of /etc/lsb-release. 2019-08-28 17:53:27 -07:00
Yashashvi Dave e05429b6a9 custom fields: Add default external account custom fields. 2019-08-28 15:35:53 -07:00
Anders Kaseorg e8ee71e9a2 restore-backup, test_docs: Import settings from django.conf.
A settings module isn’t supposed to be imported directly.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-28 15:33:16 -07:00
Mateusz Mandera 6bd34afa0d models: Migration of UserMessage.id to bigint, part 2.
This requires part 1 (which can take hours to run but generally
doesn't require downtime) to be completed first.

This portion of the migration will require the server to be completely
down for a brief period; for chat.zulip.org with 250M UserMessage
rows, it took about 60s to run; that time will vary depending on
hardware details like whether the server has an SSD, but fundamentally
shouldn't be long.

Our upgrade-zulip and upgrade-zulip-from-git tools can apply this
migration correctly; nothing special needs to be done.

Fixes #13040.
2019-08-26 21:25:19 -07:00
Mateusz Mandera b008515d63 models: Migration of UserMessage.id to bigint, part 1.
As part of adding support for more than 2B UserMessage rows in a Zulip
server, we need to change UserMessage.id (a field we don't access but
is needed by Django) from an int to a bigint.  This commit is a series
of migrations which create a `bigint_id` column and populates it correctly.

This migration will take a long time to run; on chat.zulip.org (a
server with a lot of history), it took about 4 hours to complete.

How to migrate with minimal downtime:

1. Run `upgrade-zulip-from-git` through this commit.  It will install
migration 0238 and then more or less hang while applying migration
0239.  Once migration 0238 is completed, however, your server should
be able to be started back up safely while migration 0239 is running.

2. Run `/home/zulip/deployments/next/scripts/restart-server` in a
separate terminal to get Zulip running again.

3. When the `upgrade-zulip-from-git` command finishes, it will
automatically re-restart the Zulip server, leaving you in a consistent
state and ready to do part 2 of the migration.

A useful `manage.py shell` query for checking the state after this
commit is consistent is this:

    assert UserMessage.objects.exclude(bigint_id=F("id")).count() == 0

Part of #13040.
2019-08-26 21:14:15 -07:00
Tim Abbott 89aeefed76 urls: Tighten many unintentionally broad URL patterns.
Previously, several of our URL patterns accidentally did not end with
`$`, and thus ended up controlling just the stated URL, but actually a
much broader set of URLs starting with it.

I did an audit and fixed what I believe are all instances of this URL
pattern behavior.  In the process, I fixed a few tests that were
unintentionally relying on the behavior.

Fixes #13082.
2019-08-26 20:51:49 -07:00
Tim Abbott e0f8228d6e auth: Add a test for the legacy /accounts/login/google/ mobile flow.
This is needed to return 100% URL coverage, and should also help
ensure we don't accidentally break the fix from
a43b231f90.
2019-08-26 20:51:49 -07:00
Tim Abbott c037ba49a0 i18n: Tag push notification subtitles for translation. 2019-08-26 15:02:04 -07:00
Tim Abbott 0ed0bb6828 messages: Add email/push notifications for wildcard mentions.
Historically, Zulip's implementation of wildcard mentions never
triggered either email or push notifications, instead being limited to
desktop notifications and the "mentions" counter.

We fix this just by plumbing the "wildcard_mentioned" flag through our
system.

Implements much of
https://github.com/zulip/zulip/issues/6040#issuecomment-510157264.
We're also now ready to seriously work on #3750.
2019-08-26 14:39:53 -07:00
Vishnu KS d434c0ee88 slack: Remove unnecessary comments.
Remove comments that tries to explain code that is already readable.
Also remove some todo comments that has been already taken care of.
2019-08-26 14:10:19 -07:00
Vishnu KS 99d34fd11d slack: Rename default_channels to slack_default_channels. 2019-08-26 14:10:19 -07:00
Vishnu KS b919514f7f slack: Rename customprofilefield_id to custom_profile_field_id. 2019-08-26 14:10:19 -07:00
Vishnu KS c31355f9c1 slack: Rename custom_field_id_count to custom_profile_field_value_id_count. 2019-08-26 14:10:19 -07:00
Vishnu KS 138c659c97 slack: Rename slack_custom_field_name_to_zulip_custom_field_id.
Rename custom_field_map to
slack_custom_field_name_to_zulip_custom_field_id.
2019-08-26 14:10:19 -07:00
Vishnu KS 9560736d86 slack: Rename slack_user_id_to_custom_profile_fields.
Renames slack_user_custom_field_map to
slack_user_id_to_custom_profile_fields for readability.
2019-08-26 14:10:19 -07:00
Vishnu KS 01a51c8f4e slack: Rename added_recipient to slack_recipient_name_to_zulip_recipient_id. 2019-08-26 14:10:19 -07:00
Vishnu KS 9d51a1b527 slack: Rename added_users to slack_user_id_to_zulip_user_id. 2019-08-26 14:10:19 -07:00
Vishnu KS 3650f19692 slack: Lookup dir_name key in dict instead of in dict_keys.
No reason to do the lookup in O(n) when we can do
it in average O(1) time complexity.
2019-08-26 14:10:19 -07:00
Vishnu Ks 1e5c49ad82 slack: Support importing shared channels. 2019-08-26 14:10:19 -07:00
Vishnu Ks e09a29f4d3 slack: Refactor get_slack_api_data to accept multiple query params. 2019-08-26 14:10:19 -07:00
Tim Abbott 73258efb16 development: Accept ToS for create user buttons.
We added default ToS for the development environment a few months
back; as a side effect, we now need to accept ToS when going through
the development environment registration flow, including for our
one-click account creation buttons.
2019-08-26 13:55:55 -07:00
Tim Abbott b8bfdebea1 onboarding: Mark the most recent 20 messages as unread.
After a new user joins an active organization, it isn't obvious what
to do next; this change causes there to be recent unread messages in
the stream sidebar for the user to click on to get a feel for what's
happening in the organization and experiment with Zulip.

Fixes #6512.
2019-08-26 13:51:52 -07:00
Rohitt Vashishtha 8b443a25b8 markdown: Show link href if title is empty.
Fixes #6221.
2019-08-25 21:36:42 -07:00
Rohitt Vashishtha abe2dab88c markdown: Upgrade to use InlineProcessor for links.
This commit wraps up the major work that we held back when upgrading
py-markdown 2.6.11 to 3.0.1. Since we were making our custom changes
to the link syntax, at the time we stuck to using the old method of
parsing links. This lays the groundwork for further changes to our
link and image link handling, and brings us on par with upstream.

Also, we now better document the ways in which our link handling is
different from upstream.
2019-08-25 21:36:42 -07:00
Kanishk Kakar e4f0d3d79b notifications: Add 'none' to unread count options. 2019-08-25 21:29:10 -07:00
Yashashvi Dave 330d065a69 zerver/tests/test_custom_profile_data: Group tests cases. 2019-08-25 20:59:13 -07:00
Yashashvi Dave ed5cb6c28b custom profile fields: Extract field validation function. 2019-08-25 20:59:08 -07:00
Tim Abbott 5e19546c14 messages: Fix unread_msgs accounting of wildcard mentions.
Previously, the unread_msgs data structure accounting (used for both
the web and mobile apps to determine the "Unread mentions" count
displayed in the UI) did not include wildcard mentions at all.

We fix this by adding the logic required to include properly that
data, with tests.  As discussed in #6040, it makes sense to include
muted streams and topics for the purpose of this calculation.

Fixes part of #6040.
2019-08-25 20:32:12 -07:00
Tim Abbott 625eb53b08 actions: Fetch presence idle data for non-basic mentions.
Apparently, get_active_presence_idle_user_ids, which is carefully
optimized to only fetch data for users who might actually need
notification processing, was only considering PMs and direct mentions,
not wildcard mentions or alert words.

This caused some pretty weird failure modes when working on adding
support for broader mention notifications, because users who had one
of these types of notifications would be treated as never
presence-idle, which was just confusing.

This is part of adding support for notifications for wildcard mentions
and alert words; it's worth merging this as an early commit because
the consequence of not doing this are very difficult to debug.
2019-08-25 19:58:33 -07:00
Tim Abbott 4e65f1dd2b test_event_queue: Clean up testing setup.
Rather than continually resetting the contents of an existing event
queue, we allocate a new one for each subtest.

We also fix a rather confusing bundle of comments.
2019-08-25 19:37:08 -07:00
Mateusz Mandera 16479fa606 populate_db: Extract internal realm creation into a function. 2019-08-23 12:57:24 -07:00
Mateusz Mandera 8e1a7cfb52 python_examples: Specify which user the hard-coded user ids refer to.
We add an ensure_users function and use it in tests which have
hard-coded user ids, to make it clear to which users the ids refer to
and have it verified.
This makes it easier to see what's happening
in these tests and to keep track of any renumberings of user ids due to
changes in how we populate the database.
2019-08-23 12:57:24 -07:00
Rishi Gupta e058558a52 emails: Send invitation reminder email two days before expiry.
Hopefully this does a better job of spurring people to action, and also
suggests a self-service fix if they don't (i.e. contacting the person that
invited them).
2019-08-23 12:53:11 -07:00
Rishi Gupta 2d260031ed emails: Use referrer.delivery_email in invitation emails. 2019-08-23 12:53:11 -07:00
Mohit Gupta e5482adec0 search: Add streams:public to search entire history of public streams.
Add ability to search entire message history of all public streams at
once. It includes all subscibed, non subscribed public streams messages
and even historical public stream messages sent before user had joined
an organization or stream.

Fixes #8859.
2019-08-22 13:40:49 -07:00
Rishi Gupta e1848e83bc integrations: Update instructions for AppFollow. 2019-08-22 13:17:06 -07:00
Rishi Gupta 02cfb47315 exports: Update wording on settings page, /help and /features. 2019-08-22 13:17:03 -07:00
Mateusz Mandera c437fd4344 test_users: Eliminate hard-coded gravatar urls.
Instead of having a hard-coded url, it seems better to replace it with
get_gravatar_url - which returns the correct url, without breaking if
the email/id of the example user changes.
2019-08-21 21:28:09 -07:00
Mateusz Mandera 55bf13db88 test_user_groups: Eliminate hard-coded user ids. 2019-08-21 21:28:09 -07:00
Mateusz Mandera 52d4583987 test_push_notifs: Eliminate hard-coded user ids. 2019-08-21 21:28:09 -07:00
Mateusz Mandera 0edc7d74ff test_events: Eliminate hard-coded user id. 2019-08-21 21:28:09 -07:00
Mateusz Mandera afc9650ebb test_email_change: Eliminate hard-coded user id. 2019-08-21 21:28:09 -07:00
Tim Abbott 7844f87d65 actions: Tag already has an account string for translation.
And also simplify an another validation string.

Fixed #9473.
2019-08-21 21:16:47 -07:00
Rohitt Vashishtha 400d0367dc tests: Improve logging for fixture tests in push_notifications. 2019-08-21 16:34:40 -07:00
Tim Abbott d0d16ed81c Revert "test_mattermost_importer: Fix missing coverage."
This reverts commit c48e53f587.

This new test code had nondeterministic failures; reverting so we can
debug without disruption.
2019-08-21 11:20:20 -07:00
Anders Kaseorg 5aeb562700 migrations.0001_initial: Expand \n in long SQL string.
One occasionally finds that a 1580 character string of SQL queries
might not most readably be presented on a single line.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-21 11:00:38 -07:00
Hemanth V. Alluri f375350446 page_params: Send the list of config_options for incoming webhook bots.
Send the config_options for each supported incoming webhook bot along
with the initial state (not present in apply_events since this is
mostly just static data).
2019-08-20 17:00:48 -07:00
Hemanth V. Alluri d73a37726d bots: Allow incoming webhook bots to be configured via /bots.
Without disturbing the flow of the existing code for configuring
embedded bots too much, we now use the config_options feature to
allow incoming webhook type bot to be configured via. the "/bots"
endpoint of the API.
2019-08-20 17:00:48 -07:00
Hemanth V. Alluri 94c351ead4 bot_config: Have check_valid_bot_config also take the bot_type.
This is a prep commit to allow us to validate user provided bot
config data using the same function for incoming webhook type
bots alongside embedded bots (as opposed to creating a new
function just for incoming webhook bots).
2019-08-20 16:44:56 -07:00
Hemanth V. Alluri f5c57361d6 bots: Specify config options for an incoming webhook integration.
In integrations.py we have a class called Integration which we then usually
subclass and then use to define the meta-data for all of our integrations.
Now, we want to allow all of our bots, specifically incoming webhook bots,
to be configured (i.e. we should let the user provide BotConfigData).

For this we create a new instance member of the Integration class called
config_options which will be a list of tuples containing the displayable
integration name, the configuration key form of the integration name and
the validator that it's value is supposed to adhere to.
2019-08-20 16:44:56 -07:00
Wyatt Hoodes c48e53f587 test_mattermost_importer: Fix missing coverage. 2019-08-20 15:23:41 -07:00
Mateusz Mandera 26f7dd09ce display_recipient: Eliminate raw_display_recipient from message dicts.
This was used as a helper to construct the final display_recipient when
fetching messages. With the new mechanism of constructing
display_recipient by fetching appropriate users/streams from the
database and cache, this shouldn't be needed anymore.
2019-08-20 12:22:52 -07:00
Tim Abbott b25abf3de9 display_recipient: Deduplicate list of display_recipient fields. 2019-08-20 12:22:37 -07:00
Mateusz Mandera d200cb921d display_recipient: Only fetch the necessary fields from the database.
There is no need to fetch the entire Stream or UserProfile objects, as
only several fields are needed. We use Django's .values() method to only
get what's needed.

For UserProfiles, it means that we get from the queries are dictionaries
already in the display_recipient form (UserDisplayRecipient type) - so
we can remove the user_profile_to_display_recipient_dict function, as
there's no need for this UserProfile -> UserDisplayRecipient conversion
anymore.
2019-08-20 12:15:30 -07:00
Mateusz Mandera 87ccb4f6c7 types: Consistently use DisplayRecipientT for display_recipient objects.
Instead of having the rather unclear type Union[str,
List[UserDisplayRecipient]] where display_recipient of message dicts was
involved, we use DisplayRecipientT (renamed from DisplayRecipientCacheT
- since there wasn't much reason to have the word Cache in there), which
makes it clearer what is the actual nature of the objects and gets rid
of this pretty big type declaration.
2019-08-20 12:15:30 -07:00
Mateusz Mandera 3ba0a37a92 types: Define UserDisplayRecipient type using TypedDict.
Since the display_recipients dictionaries corresponding to users are
always dictionaries with keys email, full_name, short_name, id,
is_mirror_dummy - instead of using the overly general Dict[str, Any]
type, we can define a UserDisplayRecipient type,
using an appropriate TypedDict.

The type definitions are moved from display_recipient.py to types.py, so
that they can be imported in models.py.

Appropriate type adjustments are made in various places in the code
where we operate on display_recipients.
2019-08-20 12:15:30 -07:00
Mateusz Mandera c779bb1959 messages: Don't use display_recipient values from cached message_dicts.
The user information in display_recipient in cached message_dicts
becomes outdated if the information is changed in any way.

In particular, since we don't have a way to find all the message
objects that might contain PMs after an organization toggles the
setting to hide user email addresses from other users, we had a
situation where client might see inaccurate cached data from before
the transition for a period of up to hours.

We address this by using our generic_bulk_cached_fetch toolchain to
ensure we always are fetching display_recipient data from the database
(and/or a special recipient_id -> display_recipient cache, which we
can flush easily).

Fixes #12818.
2019-08-20 12:15:30 -07:00
Pragati Agrawal 48efd46bc6 realm_logo: Remove redundant `realm_logo_url` function.
This function is nothing but a redundant layer of call to
`get_realm_logo_url`.
2019-08-20 12:07:20 -07:00
Pragati Agrawal 7326acdc2b realm_logo: Refactor `get_realm_logo_url` function. 2019-08-20 12:07:20 -07:00
Pragati Agrawal 1da5001875 realm_logo: Refactor/Deduplicate the code to sent the events.
This also creates a function to get source and url for logo.
2019-08-20 12:07:20 -07:00
Tim Abbott 69505c30a4 mattermost: Handle users who aren't on any team correctly.
It's not clear to me how this is intended to work in Mattermost's
system in that they don't document this behavior, but some users have
`null` as their list of teams, and presumably are not meant to be
included in any team at all.
2019-08-19 16:06:39 -07:00
Tim Abbott e02806ccc5 request: Fix other syntax error on Xenial Python. 2019-08-19 15:58:12 -07:00
Tim Abbott 0435556fd5 request: Fix syntax error on Xenial Python. 2019-08-19 15:47:40 -07:00
Tim Abbott 3781a0dae2 urls: Reformat how we handle API-only mobile endpoints.
This restructures the API endpoints that we currently have implemented
more or less for exclusive use by the mobile and desktop apps (things
like checking what authentication methods are supported) to use a
system that can be effectively parsed by our test_openapi
documentation.

This brings us close to being able to eliminate
`buggy_documentation_endpoints` as a persistently nonempty list.
2019-08-19 15:34:40 -07:00
Tim Abbott 27bfed7896 openapi: Remove presence from untestable endpoints.
This add some regular expression manipulation hacks to make it
possible for us to validate the documentation for the presence
endpoint with a slightly more complex regular expression capture
group.
2019-08-19 15:17:28 -07:00
Tim Abbott 44db59bbe0 openapi: Fix handling of parameters passed via the URL/path.
Previously, our OpenAPI documentation validation was failing for some
endpoints because it didn't account for the `in: path` type of
parameter, resulting in a mismatch between what was declared via REQ
and what was declared in the OpenAPI docs.

We fix this by excluding the path type parameters in both places from
what's considered by documentation using the `path_only` flag.

I doubt this is the correct long-term fix; in particular, I don't
think we're actually running the validators for these path-only
parameters.  The examples that exist today are all IDs with validators
for being non-negative numbers, but longer-term I think we'll want to
do something different (possibly at the REQ layer, see the TODO).
2019-08-19 15:06:08 -07:00
Hemanth V. Alluri 2e20dcc053 webhooks/jira: Update webhook to support improved comment events.
Atlassian announced that it will no longer provide information about
comments along with their "issue" type event payloads for Jira. So
we must now update the Jira integration to appropriately respond to
"comment" type events (the reason why we didn't do this before was
that initially, the "comment" type event payloads didn't contain
sufficient information about their issues, but this payload has
since been improved).

Note: This commit does *not* remove support for the older "issue"
type event payloads where information about comments was included.
This way we can maintain compatibility with old self-hosted versions
of self hosted Jira (2016 and before).

Source:
https://developer.atlassian.com/cloud/jira/platform/change-notice-
removal-of-comments-from-issue-webhooks/

Fixes #13012
2019-08-19 14:22:42 -07:00
Wyatt Hoodes 92dd5e2c4f test_queue_worker: Remove timing from test_slow_queries_worker.
Testing a specific query time was causing spurious failures
in CI.
2019-08-18 12:32:45 -07:00
okmanl 2a1305de9f lint: Add a rule to avoid msgid as a Python variable name.
This is for consistency with our usual patterns, see #12995.  We will
need a similar commit for JavaScript to complete #12995.
2019-08-17 12:47:13 -07:00
Wyatt Hoodes dc191b0be4 test_queue_worker: Test actual code path for slow_queries.
Instead of just mocking some fake events, we use the code
path that generates slow query events and publishes them
to SlowQueryWorker.

This test improvement would have got a recent potential regression
caught in code review.
2019-08-17 12:38:02 -07:00
Brutus5000 a386e3aebb github webhook: For release events show actions and tag name. 2019-08-17 12:35:16 -07:00
Hemanth V. Alluri 243d8ffc51 openapi: Add a test to ensure generated curl examples work.
This new test runs each generated curl example against the Zulip API,
checking whether it returns successfully without errors.

Significantly modified by tabbott for simplicity.
2019-08-17 11:35:08 -07:00
Tim Abbott 26ac3ebd3e openapi: Suggest the -sS options to curl.
These options prevent curl from doing the downloading progress bar
(which is clutter).
2019-08-17 11:35:08 -07:00
Tim Abbott 2366490ffc openapi: Pass api_url to curl example generation.
Our new curl example generation logic was broken, in that it hardcoded
localhost:9991 (without an HTTP method or anything) as the API URL.

It requires a bit of plumbing to make this possible.
2019-08-17 11:35:08 -07:00
Vishnu KS f32382f7b7 emails: Show preheader block only if preheader is present. 2019-08-17 11:32:28 -07:00
Priyank Patel 3680393b47 messages: Support passing user ID for stream operator.
ok_to_include_history fuction was updated to expect stream ID.

Fixes part of #9474.
2019-08-17 11:19:12 -07:00
Priyank Patel 5eab5bbc3e streams: Refactor can_access_stream_history.
This refactor extracts the code logic of checking if user can access
stream history into it's own function: can_access_stream_history
that takes in user_profile and stream. Then we make seperate function
can_access_stream_by_name that takes in stream_name and retrives stream
and pass it to can_access_stream_history. This will make it easily to later
add a function that does the same thing with stream ID.
2019-08-17 11:10:00 -07:00
Priyank Patel 0307b6d684 refactor: Make get_stream_name_from_narrow return stream.
This function was used only once in exclude_muting_conditions where
it returned stream name so the function to fetch stream id.
Since we exepect the narrow to also include stream id we refactor it to
return a stream object and since we use get_stream_by_narrow_operand_access_unchecked
we don't need to worry about handling cases where stream id is passed since
the function handles it.
2019-08-17 11:10:00 -07:00
Priyank Patel 463ecb375f refactor: Add get_stream_by_narrow_operand_access_unchecked.
This let's us clean up the linter that excludes the use of get_stream
and by adding the access_unchecked in the name we make it clear that
it should be used with caution.

Refactoring idea by Tim Abbott.
2019-08-17 11:10:00 -07:00
Priyank Patel 1f8f8867cd message_fetch: Rename handle_user_ids_supported_operators.
This renames handle_user_ids_supported_operators to
handle_operators_supporting_id_based_api.
2019-08-17 11:10:00 -07:00
Priyank Patel 1edde4a989 Rename user_id(s)_supported_operators -> operators_supporting_id(s). 2019-08-17 11:10:00 -07:00
Priyank Patel 99450f336c messages: Extract mypy typing for narrow list to variable.
Makes it obvious and readable and as an added bonus take up less space.

Note, some types used Iterable instead of List that were change
to used List since the narrow_paramter converter return a List.
2019-08-17 11:10:00 -07:00
Gloria Elston 092662d83e api: Use new endpoint for handling reactions. 2019-08-16 14:36:15 -07:00
Anders Kaseorg a5596011a0 queue_processors, python_examples: Fix mypy errors.
zerver/openapi/python_examples.py:105: error: Argument 1 to "get_user_presence" of "Client" has incompatible type "str"; expected "Dict[str, Any]"
    zerver/openapi/python_examples.py:563: error: Argument 1 to "add_reaction" of "Client" has incompatible type "Dict[str, object]"; expected "Dict[str, str]"
    zerver/openapi/python_examples.py:576: error: Argument 1 to "remove_reaction" of "Client" has incompatible type "Dict[str, object]"; expected "Dict[str, str]"
    zerver/worker/queue_processors.py:587: error: Argument "client" to "extract_query_without_mention" has incompatible type "EmbeddedBotHandler"; expected "ExternalBotHandler"

These were only missed because mypy daemon mode requires us to set
`follow_imports = skip` for the `zulip` package.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-16 14:13:40 -07:00
Mateusz Mandera c6b3d0212d models: Move some display_recipient code to display_recipient.py. 2019-08-15 17:15:32 -07:00
Mateusz Mandera cb2c9b04b3 generic_bulk_cached_fetch: Only call query_function if necessary. 2019-08-15 17:14:02 -07:00
Mateusz Mandera 898bc52538 models: Add bulk_get_huddle_user_ids function. 2019-08-15 17:03:17 -07:00
Mateusz Mandera f70a0bba82 display_recipient: Extract user_profile_to_display_recipient_dict function. 2019-08-15 16:59:28 -07:00
Tim Abbott 27a0e307b6 cache: Fix typing for generic_bulk_cached_fetch.
The typing for generic_bulk_cached_fetch is complicated, and was
recorded incorrectly previously for the case where a cache_transformer
function is required.  We fix this by adding the new CacheItemT, and
additionally add comments explaining what's going on with these types
for future reference.

Thanks to Mateusz Mandera for raising this issue.
2019-08-14 11:00:40 -07:00
Tim Abbott 2ada0a9bad management: Fix password reset emails being sent to deactivated users.
Apparently, the filters written for the send_password_reset_email (and
some other management commands) didn't correctly consider the case of
deactivated users.

While some commands, like syncing LDAP data (which can include whether
a user should be deactivated) want to process all users, other
commands generally only want to interact with active users.  We fix
this and add some tests.
2019-08-14 10:50:45 -07:00
Wyatt Hoodes 001ff72861 data exports: Add upload and message limit test. 2019-08-12 19:52:46 -07:00
Tim Abbott 8ac8247dbe data export: Add limits on exporting large organizations.
It seems possible that attempting to export large organizations could
result in high resource consumption that justifies having a technician
manage the exports manually.
2019-08-12 18:21:09 -07:00
Tim Abbott 1d63f129bf data export: Add summary logging of runtime.
This should help us investigate cases where this runs a very long
time.
2019-08-12 18:21:08 -07:00
Wyatt Hoodes 896886b412 realm_export: Add endpoint for export deletion. 2019-08-12 17:51:46 -07:00
Wyatt Hoodes 777dcdd272 test_realm_export: Clean up export testing.
* Added comments.
* Refactored `export_object` to `audit_log_entry`.
* Cleaned up white space.
* Removed uses of `getattr`.
2019-08-12 17:51:46 -07:00
Wyatt Hoodes 7c118096e7 test_events: Remove print statement from self.client_post.
Leftover from prior debugging.
2019-08-12 17:51:46 -07:00
Wyatt Hoodes 7d178bbb0f queue_processors: Clean up the extra_data dict code.
We don't want to add a `deleted_timestamp` key until
the export is actually deleted.
2019-08-12 17:51:46 -07:00
Wyatt Hoodes 6f6efa516d exports: Refactor extra_data to export_data. 2019-08-12 17:51:46 -07:00
Wyatt Hoodes 7a2a1f29ad exports: Refactor event_time to export_time timestamp.
The time of the event was incorrectly being sent
as a datetime object.
2019-08-12 17:51:46 -07:00
Anders Kaseorg e0cf3d0e76 zerver: Accept HEAD requests wherever GET requests are accepted.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-12 16:47:41 -07:00
Hemanth V. Alluri d425ab5ae5 test_templates: Add a verbose error messages for rendering failures.
If a template fails to render in test_templates because of a missing
context variable, we should better explain the reason for the failure.
2019-08-12 16:24:08 -07:00
Anders Kaseorg 72655611ce requirements: Use maintained fork django-sendfile2 of django-sendfile
The original seems to be unmaintained
(johnsensible/django-sendfile#65).  Notably, this fixes a bug in the
filename parameter, which perviously showed the Python 3 repr of a
byte string (johnsensible/django-sendfile#49).

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-12 15:40:08 -07:00
Anders Kaseorg 73330f3136 rest_dispatch: Remove invalid Content-Length: 0 from 204 responses.
There’s an apparent contradiction between RFC 7230 §3.3.2
Content-Length:

“A server MUST NOT send a Content-Length header field in any response
with a status code of 1xx (Informational) or 204 (No Content).”

and RFC 7231 §4.3.7 OPTIONS:

“A server MUST generate a Content-Length field with a value of "0" if
no payload body is to be sent in the response.”

The only resolution within the existing language would be to disallow
all 204 responses to OPTIONS requests.  However, I don’t think that
was the intention, so I submitted this erratum report:

https://www.rfc-editor.org/errata/eid5806

and updated the code accordingly.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-12 15:34:42 -07:00
Tim Abbott 7953e23d49 event_queue: Actually fix missing copy for edit-message events.
Apparently, our edit-message events did not guarantee that the outer
wrapper dictionary, which is intended to be unique for each client,
was unique for every client (instead only ensuring it was unique for
each user).

This led to clients unexpectedly getting last_event_id validation
errors in this code path when a user had multiple connected clients,
because the linear ordering of event IDs within a given queue was
corrupted.

In fd2a63b049, we accidentally fixed
this issue with a different set of userdata events, without fixing the
edit-message event bug.  This commit fixes the remaining issue.
2019-08-12 15:17:10 -07:00
Anders Kaseorg 7bf09067d1 event_queue: Clean up type ignores.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-09 17:42:33 -07:00
Anders Kaseorg 7cd11d0559 test_users: Clean up type ignores.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-09 17:42:33 -07:00
Anders Kaseorg 75a01810e3 test_outgoing_webhook_interfaces: Clean up type ignores.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-09 17:42:33 -07:00
Anders Kaseorg 1ffbc0009d test_openapi: Clean up type ignores.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-09 17:42:33 -07:00
Anders Kaseorg 9d33200fb2 test_messages: Clean up type ignores.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-09 17:42:33 -07:00
Anders Kaseorg 4671e4ca5c test_i18n: Clean up type ignores.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-09 17:42:33 -07:00
Anders Kaseorg c453c4b602 test_events: Clean up type ignores.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-09 17:42:33 -07:00
Anders Kaseorg d05578817e soft_deactivate_users: Clean up type ignores.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-09 17:42:33 -07:00
Anders Kaseorg 6d875dd211 topic: Clean up type ignores.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-09 17:42:33 -07:00
Priyank Patel 72bf7e2181 test_narrow: Rename stream_id -> stream_recipient_id.
It looks like this isn't actually a stream id, doing
get_stream_by_id_in_realm(stream_id, stream_messages[0].get_realm())
throws error.
2019-08-09 17:31:53 -07:00
Anders Kaseorg a0f4c99dad test_runner: Clean up type ignores.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-09 16:39:16 -07:00
Anders Kaseorg aa384c03a6 push_notifications: Clean up type ignores.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-09 16:39:16 -07:00
Anders Kaseorg 95658a2a54 logging_util: Clean up type ignores.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-09 16:39:16 -07:00
Anders Kaseorg df1fbbfee3 emoji: Clean up type ignores.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-09 16:39:16 -07:00
Anders Kaseorg bb5507009f camo: Clean up type ignores.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-09 16:39:16 -07:00
Anders Kaseorg 9b392917c5 addressee: Clean up type ignores.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-09 16:39:16 -07:00
Anders Kaseorg e417d3a040 slack_message_conversion: Clean up type ignores.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-09 16:39:16 -07:00
meiomorphism 1af72a2745 subscriptions: Avoid sending unneeded subscriber information.
The `users/me/subscriptions` endpoint accidentally started returning
subscriber information for each stream.  This is convenient, but
unnecessarily costly for those clients which either don't need it
(most API apps) or already acquire this information via /register
(including Zulip's apps).

This change removes that data set from the default response.  Clients
which had come to rely on it, or would like to rely on it in future,
may still access it via an additional documented API parameter.

Fixes #12917.
2019-08-09 16:03:21 -07:00
Tim Abbott fb01f2208d api: Verify that all registered OpenAPI examples are called.
This should help make this test suite robust against potential
mistakes when updating it.
2019-08-08 11:45:05 -07:00
Hemanth V. Alluri c2f9227892 openapi/python_examples: Replace TEST_FUNCTIONS with a decorator.
Use a decorator called openapi_test_function instead of the hard-coded
TEST_FUNCTIONS list for increased readability and maintainablity (at
the cost of performance).
2019-08-08 11:45:05 -07:00
Alexandra Ciobica e5e45c9a25 auth: Change page title and add description for the list.
I changed the class of the title in order to use the same styling as the
 other similar pages (like `/accounts/go` or `/login`).

Changed the related test.
2019-08-08 11:12:51 -07:00
Alexandra Ciobica f109dcce9c auth: Add logic for avatars to the GitHub auth email selection page.
For the emails that are associated to an existing account in an
organisation, the avatars will be displayed in the email selection
page.  This includes avatar data in what is passed to the page.

Added `avatar_urls` to the context in `test_templates.py`.
2019-08-08 11:12:51 -07:00
Alexandra Ciobica d4ccd73ae3 auth: Remove `@users.noreply.github.com` from the email selection list.
Apparently GitHub changed the email address for these; we need to
update our code accordingly.

One cannot receive emails on the username@users.noreply.github.com, so
if someone tries creating an account with this email address, that
person would not be able to verify the account.
2019-08-08 11:12:51 -07:00
Anders Kaseorg 7e92ff9d0a request: Delete request.pyi and make request.py type check.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-07 15:37:21 -07:00
Anders Kaseorg 13a2ac7b8e request: Remove ExtractRecipients type safety hole on REQ.
It was allowing us to get away with wrong types on a few functions:
`check_send_typing_notification` and `send_notification_backend` can be
(and are) called with a list of `int` as `notification_to`, not just a
list of `str`.

The problem it was working around already had a better solution using
the dummy `type` argument.  Use that.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-07 15:26:59 -07:00
Tim Abbott 4e631876f4 test_events: Fix realm export test to actually work.
The previous iteration still had the failure mode of not actually
testing anything, because it didn't trigger the data export code path
(and in fact was getting an HTTP 401 authentication denied error).
2019-08-07 14:18:27 -07:00
Wyatt Hoodes 22842dab34 events: Rename notify_export_completed.
notify_realm_export is more reasonable for the context of doing
deletion events as well.
2019-08-07 14:18:27 -07:00
Wyatt Hoodes 7853e9d2de test_events: Fix broken test case.
This test was broken due to using an empty `RealmAuditLog`
table.  We fix this by mocking the creation of an export,
thus creating an entry, similar to what we do in our other
tests.
2019-08-07 12:05:44 -07:00
Wyatt Hoodes f93efcb89c test_helpers: Refactor dummy file creation for broader use. 2019-08-07 12:05:44 -07:00
Wyatt Hoodes 11db0c23fb exports: Update extra_data field to a JSON structure.
We add the `deleted_timestamp` key to the new `extra_data`
dictionary.
2019-08-07 12:04:28 -07:00
Anders Kaseorg 0962393933 cleanup: Delete trailing newlines.
Delete trailing newlines from all files, except
tools/ci/success-http-headers.txt and tools/setup/dev-motd, where they
are significant, and static/third, where we want to stay close to
upstream.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-06 23:29:11 -07:00
Anders Kaseorg becef760bf cleanup: Delete leading newlines.
Previous cleanups (mostly the removals of Python __future__ imports)
were done in a way that introduced leading newlines.  Delete leading
newlines from all files, except static/assets/zulip-emoji/NOTICE,
which is a verbatim copy of the Apache 2.0 license.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-06 23:29:11 -07:00
neiljp (Neil Pilgrim) 12f3ed12c4 mypy: Refactor extract_recipients in actions.py to remove type ignore. 2019-08-06 23:24:56 -07:00
neiljp (Neil Pilgrim) 5ab64daecc mypy: Remove type ignore by defining ProfileDataElement using TypedDict. 2019-08-06 23:24:56 -07:00
neiljp (Neil Pilgrim) accf4411f0 mypy: Remove type ignore on MissedMessageWorker.stop_timer. 2019-08-06 23:24:56 -07:00
neiljp (Neil Pilgrim) 62a1c3443d mypy: Remove type ignores for python 3 stubs now present in typeshed.
Namely:
* posix stubs
* readfp stub
2019-08-06 23:24:56 -07:00
neiljp (Neil Pilgrim) 5f673f5820 mypy: Remove type ignores after boto stub improvements. 2019-08-06 23:24:56 -07:00
neiljp (Neil Pilgrim) ba54e20def mypy: Remove type ignore for merged typeshed PR 1762. 2019-08-06 23:24:56 -07:00
Tim Abbott 6b4300904a models: Add block comment explaining per-request cache. 2019-08-06 23:22:48 -07:00
Tim Abbott fd2a63b049 event_queue: Fix missing copy for edit-message events.
Apparently, our edit-message events did not guarantee that the outer
wrapper dictionary, which is intended to be unique for each client,
was unique for every client (instead only ensuring it was unique for
each user).

This led to clients unexpectedly getting last_event_id validation
errors in this code path when a user had multiple connected clients,
because the linear ordering of event IDs within a given queue was
corrupted.
2019-08-06 13:40:30 -07:00
Hemanth V. Alluri ac69717604 openapi: Migrate /messages to use the new cURL example system.
Now that we can create cURL examples based on the OpenAPI
documentation. We can begin using simple one line tags in
the documentation instead of manually creating cURL examples.

Fixes part of #12878.
2019-08-05 21:27:24 -07:00
Hemanth V. Alluri 5af753d940 api_code_examples: Parse kwargs/mods for example code generation.
Now we can also include extra keyword arguments to specify
modifications in how the example code should be generated
in the generate_code_example template tag.

E.g. generate_code_example(curl, exclude=["param1", "param2"])
2019-08-05 21:27:23 -07:00