Commit Graph

10390 Commits

Author SHA1 Message Date
Rishi Gupta 360cd7f147 remote data: Send RealmAuditLog data. 2019-10-08 17:27:29 -07:00
Rishi Gupta b86142089b billing: Sync realm activation events for on-prem billing. 2019-10-08 17:18:30 -07:00
Rishi Gupta 01bf99ec65 billing: Keep track of role count changes in RealmAuditLog. 2019-10-08 17:18:30 -07:00
Anders Kaseorg cc791afc54 requirements: Upgrade django-sendfile2.
django-sendfile2 now always sends a Content-Disposition header even if
it’s inline.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-10-08 17:15:00 -07:00
Vishnu KS cd06e0ab79 tests: Remove /messages/{message_id} from buggy endpoints. 2019-10-07 11:45:11 -07:00
Vishnu KS bfc9089124 tests: Remove /users/me/subscriptions/muted_topics from buggy endpoints.
Even though required attribute of stream and stream_id params is marked
false in openapi specification, the API expects atleast one of the
params to be set. There is no way to specify relationships like this
openapi and they dont seem to have any plan to implement this in future.

https://github.com/OAI/OpenAPI-Specification/issues/256
2019-10-07 11:42:52 -07:00
Vishnu KS e7419f815a tests: Remove /settings/notifications from buggy endpoints. 2019-10-07 11:42:52 -07:00
Vishnu KS 21ab8c542a bugdown: Cast enum elements to string in APIArgumentsTablePreprocessor.
So that enums other than of type string gets rendered without any error.
2019-10-07 11:42:52 -07:00
overide dd67b1b3bb topic: Change before bound limit from 2 to 7 days.
This limit was introduced in c588c79 as a part of the
feature and not due to performance crisis. So we are
increasing this limit to 7 days. Since topics tends to
naturally fizzle after day or two so 7 days limit
would be good enough.
2019-10-06 20:12:59 -07:00
Rishi Gupta 48dc1d1128 remote data: Refactor remote_server_post_analytics to be more generic.
One small change in behavior is that this creates an array with all the
row_objects at once, rather than creating them 1000 at a time.

That should be fine, given that the client batches these in units of
10000 anyway, and so we're just creating 10K rows of a relatively
small data structure in Python code here.
2019-10-06 16:55:41 -07:00
Rishi Gupta e10361a832 models: Replace is_guest and is_realm_admin with UserProfile.role.
This new data model will be more extensible for future work on
features like a primary administrator.
2019-10-06 16:24:37 -07:00
Rishi Gupta 4256ee61cf billing: Change RealmAuditLog.event_type from str to int.
This is a more robust long-term model for storing these data.
2019-10-06 15:55:56 -07:00
Shikhar Varshney fe5d975b33 settings: Add support for overriding APNS_TOPIC and ZULIP_IOS_APP_ID.
Documentation added by tabbott.
2019-10-05 22:53:59 -07:00
Mateusz Mandera dbe508bb91 models: Migration of Message.pub_date to date_sent, part 2.
Fixes #1727.

With the server down, apply migrations 0245 and 0246. 0246 will remove
the pub_date column, so it's essential that the previous migrations
ran correctly to copy data before running this.
2019-10-05 19:01:34 -07:00
Mateusz Mandera f198475281 models: Migration of Message.pub_date to date_sent, part 1.
1. Apply migration 0243 to add date_sent column.
2. Apply migration 0244 to copy pub_date over to date_sent. Can be done
with the server running.
3. With the server down (for consistency between memory and
database state of Django objects), verify consistency with
Message.objects.exclude(date_sent=F("pub_date")).count() == 0
2019-10-05 19:01:34 -07:00
Vishnu Ks 30ad650afb bugdown: Don't show path params as data in curl example.
With test added by tabbott.
2019-10-05 18:17:36 -07:00
Vishnu Ks bfd5f4066a bugdown: Remove unecessary call to get_openapi_parameters. 2019-10-05 18:10:55 -07:00
Vishnu Ks dff62a240b bugdown: Rename openapi_example_params to openapi_params in generate_curl_example. 2019-10-05 18:10:55 -07:00
Vishnu Ks f4e3e6068f bugdown: Rename pocket to param in generate_curl_example. 2019-10-05 18:10:55 -07:00
Mateusz Mandera 94d5ca838f social auth tests: Extract prepare_login_url_and_headers method.
It will be reused in SAML tests in upcoming commits.
2019-10-01 16:39:12 -07:00
Mateusz Mandera 0e5f964363 social auth tests: Consistently refer to hamlet's name via self.name. 2019-10-01 16:38:50 -07:00
Tim Abbott c869a3bf82 upload: Fix browser caching of uploads with local uploads backend.
Apparently, our change in b8a1050fc4 to
stop caching responses on API endpoints accidentally ended up
affecting uploaded files as well.

Fix this by explicitly setting a Cache-Control header in our Sendfile
responses, as well as changing our outer API caching code to only set
the never cache headers if the view function didn't explicitly specify
them itself.

This is not directly related to #13088, as that is a similar issue
with the S3 backend.

Thanks to Gert Burger for the report.
2019-10-01 15:15:17 -07:00
Ivan Mitev 16c9d63056 emoji: Fix sort order of emoji choices.
The previous code for ensuring the sort order of emoji choices was
correct relied on an OrderedDict structure, which isn't guaranteed to
be preserved when passed to the frontend via JSON (in fact, it isn't,
since we converted the way page_params is passed to use
sort_keys=True).  Switch it to a list of dictionaries to correct this.

Fixes #13220.
2019-10-01 13:54:55 -07:00
Mateusz Mandera 4166c901ef do_update_user_custom_profile_data: Rename to ..._if_changed.
This adds clarity to the fact that the function no longer does
anything if the field values haven't changed.
2019-10-01 13:52:43 -07:00
Mateusz Mandera d66cbd2832 do_update_user_custom_profile_data: Don't notify if value not changed. 2019-10-01 13:50:14 -07:00
Tim Abbott 2b227d6b63 models: Add comments documenting our name field decisions. 2019-09-27 18:01:56 -07:00
Tim Abbott bf088519a7 api: Add avatar URL to the GET /profile endpoint.
This endpoint is legacy in a lot of ways, but it seems reasonable that
it should have these data.
2019-09-27 12:07:03 -07:00
Vishnu Ks 1c81538570 mattermost: Read fixures inside test cases instead of setUp. 2019-09-25 20:07:00 +05:30
Vishnu KS a21856c569 mattermost: Rename user_id to sender_user_id in process_raw_message_batch. 2019-09-25 20:06:47 +05:30
Vishnu KS 23d70bb685 mattermost: Rename get_recipient_id to get_recipient_id_from_receiver_name. 2019-09-25 20:06:04 +05:30
Vishnu KS a743991935 mattermost: Fix mirror dummy fixture data.
The channel where Snape sends message should be gryffindor-quidditch-team
and not slytherin-quidditch-team since he is mirror dummy in gryffindor.
2019-09-25 19:03:43 +05:30
Tim Abbott 7e0ea61b00 upload: Support S3-compatible S3 hosting providers.
Previously, we were hardcoding the domain s3.amazonaws.com.  Given
that we already have an interface for configuring the host in
/etc/zulip/boto.cfg (which in turn, automatically configures boto), we
just need to actually use the value configured in boto for what S3
hostname to use.

We don't have tests for this new use case, in part because they're
likely annoying to write with `moto` and there hasn't been a huge
amount of demand for it.  Since this doesn't regress existing S3
backend support, it seems worth merging.
2019-09-24 17:17:21 -07:00
Tim Abbott b8b0ae362c uploads: Only initialize S3 connection once in __init__.
This should be a mild performance optimization for the S3
authentication backend, since we aren't initializing unnecessary
duplicate connections.
2019-09-24 17:15:44 -07:00
Hemanth V. Alluri 98a33c0fda page_params: Fix list of config_options sent for incoming webhook bots.
This patches an issue in f37535044 where we mistakenly tried to send
the function as part of the page_params. Instead, we should just try
to send the list of configuration options (in their user displayable
form).
2019-09-24 16:04:51 -07:00
Laurel Michaels a67769091d docs: Add OpenAPI data for `/messages/{message_id}/reactions`.
This change adds the OpenAPI data needed to document the POST and
DELETE methods associated with this endpoint.

Descriptions edited slightly by tabbott.
2019-09-24 15:57:51 -07:00
Tim Abbott 96726c00ce export: Fix broken URLs in UI with S3 backend.
Apparently, the Zulip notifications (and resulting emails) were
correct, but the download links inside the Zulip UI were incorrectly
not including S3 prefix on the URL, making them not work.

While we're at this, we rewrite the somewhat convoluted previous
system for formatting the data export output.
2019-09-24 13:56:49 -07:00
Tim Abbott 8024b1179a bots: Fix bot email addresses with EMAIL_ADDRESS_VISIBILITY_ADMINS.
When using our EMAIL_ADDRESS_VISIBILITY_ADMINS feature, we were
apparently creating bot users with different email and delivery_email
properties, due to effectively an oversight in how the code was
written (the initial migration handled bots correctly, but not bots
created after the transition).

Following the refactor in the last commit, the fix for this is just
adding the missing conditional, a test, and a database migration to
fix any incorrectly created bots leaked previously.
2019-09-23 15:53:53 -07:00
Tim Abbott 6e5c99328a models: Extract a helper function for whether emails are realm-public.
This is also a useful preparatory refactor for having a user setting
controlling whether one's own email address is publicly available
within the organization.
2019-09-23 15:53:53 -07:00
Anders Kaseorg 9339402541 messages: Sort IDs in the by_streams query deterministically.
This fixes a test flake after upgrading SQLAlchemy.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-09-23 11:56:52 -07:00
Anders Kaseorg cb0652071f zulip_otp_required: Avoid _user_is_authenticated compatibility shim.
This private shim is not needed with Django ≥ 1.10, and was removed
from django-otp 0.7.0.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-09-23 11:56:52 -07:00
Rishi Gupta 3646936cfb integrations: Advertise our native Zapier integration. 2019-09-22 18:47:42 -07:00
Tim Abbott 767a1b9f0c user_activity: Skip an unnecessary database query for new entries.
While new entries are an overwhelming minority of calls to this
function, it's still worth having a clean implementation.
2019-09-21 11:49:43 -07:00
Tim Abbott 1c73ce2450 user_activity: Use LoopQueueProcessingWorker strategy.
This should dramatically improve the queue processor's performance in
cases where there's a very high volume of requests on a given endpoint
by a given user, as described in the new docstring.

Until we test this more broadly in production, we won't know if this
is a full solution to the problem, but I think it's likely.  We've
never seen the UserActivityInterval worker end up backlogged without a
total queue processor outage, and it should have a similar workload.

Fixes #13180.
2019-09-21 11:48:24 -07:00
Tim Abbott 52a2b2c6a8 tests: Move user_activity test next to slow_query test.
This will conveniently put all the LoopQueueProcessingWorkers
together.
2019-09-21 11:47:14 -07:00
Tim Abbott f0d8951035 do_update_user_activity: Refactor to support passing a count.
We'll use this in upcoming commits.
2019-09-21 11:47:14 -07:00
Tim Abbott 5c960b3e0f user_activity: Make the queue processor a bit more efficient.
We don't actually need to go to the memcached (falling back to the
database) to fetch either user or client objects on every event.  For
user objects, we actually can just pass through the user ID
transparently; for client objects, we can use an in-process cache,
since the mapping of string to ID never changes.
2019-09-21 11:47:14 -07:00
Mateusz Mandera 3271235200 email_mirror: Ignore missed message email if the user isn't active. 2019-09-20 17:58:10 -07:00
Mateusz Mandera 21459ed193 test_email_mirror: Test missed message emails in a single ZulipTestCase.
With the way these tests are, it's unnecessary to have 3 separate
classes, and it makes it confusing to decide where to add a potential
additional mm email test.
2019-09-20 17:58:10 -07:00
Tim Abbott c19b7ec89d test_events: Remove unnecessary check_none validator.
This can be done cleanly using the existing equals(None).
2019-09-20 11:51:50 -07:00
Anders Kaseorg cf5a00d94b bulk_get_subscriber_user_ids: Sort each user list by ID.
This simple backwards-compatible change saves approximately 12% in the
compressed size of the chat.zulip.org page_params.  We can do much,
much better by changing the format, but this seems like a good
intermediate step.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-09-19 20:17:43 -07:00