Commit Graph

400 Commits

Author SHA1 Message Date
Ryan Rehman 174b2abcfd settings: Migrate to stream_post_policy structure.
This commit includes a new `stream_post_policy` setting,
by replacing the `is_announcement_only` field from the Stream model,
which is done by mirroring the structure of the existing
`create_stream_policy`.

It includes the necessary schema and database migrations to migrate
the is_announcement_only boolean field to stream_post_policy,
a smallPositiveInteger field similar to many other settings.

This change is done to allow organization administrators to restrict
new members from creating and posting to a stream. However, this does
not affect admins who are new members.

With many tweaks by tabbott to documentation under /help, etc.

Fixes #13616.
2020-02-04 17:08:08 -08:00
Steve Howell e3ad9baf1d presence: Add process_presence_event.
This lets us conditionally remove the email
field from a presence event if the client
has registered with the slim_presence flag.
2020-02-04 12:30:36 -08:00
Steve Howell a672a00677 presence: Add user_id to presence event.
In a later commit, we will eliminate email for
clients who have set slim_presence as their
preference.
2020-02-04 12:30:36 -08:00
Steve Howell bf9144ff69 presence: Add slim_presence flag.
This flag affects page_params and the
payload you get back from POSTs to this
url:

    users/me/presence

The flag does not yet affect the
presence events that get sent to a
client.
2020-02-04 12:30:34 -08:00
Tim Abbott 5825a155cc users: Use format_user_row in events system as well.
This completes the deduplication of our logic for turning users into
dictionaries in the Zulip API.
2020-01-31 14:47:16 -08:00
Tim Abbott 6e75c27f74 test_events: Fix nested use of check_events_dict.
In addition to making our schema check stricter, it also makes it
possible for us to extend check_events_dict to do additional
validation that's only expected for the full event object.
2020-01-29 13:02:05 -08:00
Tim Abbott 79f18138f5 realm: Add private_message_policy setting.
This experimental setting disables sending private messages in Zulip
in a crude way (i.e. users get an error when they try to send one).
It makes no effort to adjust the UI to avoid advertising the idea of
sending private messages.

Fixes #6617.
2020-01-13 12:20:42 -08:00
Mateusz Mandera 9f2b0c769f stream_recipient: Eliminate unnecessary queries.
We should take adventage of the recipient field being denormalized into
the Stream model. We don't need to make queries to figure out a stream's
recipient id, so we take advantage of that to eliminate some of
those redundant queries and simplify StreamRecipientMap.
2020-01-08 14:34:43 -08:00
Tim Abbott 7ccc8373e2 bugdown: Fix logic for extracting attachment path_id.
In 3892a8afd8, we restructured the
system for managing uploaded files to a much cleaner model where we
just do parsing inside bugdown.

That new model had potentially buggy handling of cases around both
relative URLs and URLS starting with `realm.host`.

We address this by further rewriting the handling of attachments to
avoid regular expressions entirely, instead relying on urllib for
parsing, and having bugdown output `path_id` values, so that there's
no need for any conversions between formats outside bugdowm.

The check_attachment_reference_change function for processing message
updates is significantly simplified in the process.

The new check on the hostname has the side effect of requiring us to
fix some previously weird/buggy test data.

Co-Author-By: Anders Kaseorg <anders@zulipchat.com>
Co-Author-By: Rohitt Vashishtha <aero31aero@gmail.com>
2019-12-12 20:30:26 -08:00
Ryan Rehman 6110bf96ca tests: Rename prereg_users to prereg_user in test_events.
This is a typo fix.
2019-12-10 17:21:04 -08:00
Nat1405 d5f005fd61 wildcard_mentions_notify: Add per-stream override of global setting.
Adds required API and front-end changes to modify and read the
wildcard_mentions_notify field in the Subscription model.

It includes front-end code to add the setting to the user's "manage
streams" page. This setting will be greyed out when a stream is muted.
The PR also includes back-end code to add the setting the initial state of
a subscription.

New automated tests were added for the API, events system and front-end.
In manual testing, we checked that modifying the setting in the front end
persisted the change in the Subscription model. We noticed the notifications
were not behaving exactly as expected in manual testing; see
https://github.com/zulip/zulip/issues/13073#issuecomment-560263081 .

Tweaked by tabbott to fix real-time synchronization issues.

Fixes: #13429.
2019-12-09 16:09:38 -08:00
Mateusz Mandera dda3ff41e1 messages: Optimize get_recent_private_conversations.
Previously, get_recent_private_messages could take 100ms-1s to run,
contributing a substantial portion of the total runtime of `/`.

We fix this by taking advantage of the recent denormalization of
personal_recipient into the UserProfile model, allowing us to avoid
the complex join with Recipient that was previously required.

The change that requires additional commentary is the change to the
main, big SQL query:
1. We eliminate UserMessage table from the query, because the condition
m.recipient_id=%(my_recipient_id)d
implies m is a personal message to the user being processed - so joining
with usermessage to check for user_profile_id and flags&2048 (which
checks the message is private) is redundant.
2. We only need to join the Message table with UserProfile
(on sender_id) and get the sender's personal_recipient_id from their
UserProfile row.

Fixes #13437.
2019-12-09 15:23:10 -08:00
Rohitt Vashishtha 68e93d2435 update-message: Use MentionData in the update_message_backend code.
This is a performance optimization, since we can avoid doing work
related to wildcard mentions in the common case that the message can't
have any.  We also add a unit test for adding wildcard mentions in a
message edit.
2019-12-02 12:12:35 -08:00
Tim Abbott 1fe4f795af settings: Add notification settings checkboxes for wildcard mentions.
This change makes it possible for users to control the notification
settings for wildcard mentions as a separate control from PMs and
direct @-mentions.
2019-11-20 16:58:46 -08:00
Anders Kaseorg 6717daf4a6 test_events: Fix apparent typo.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-11-19 15:34:05 -08:00
Tim Abbott 993ed9c2b1 tornado: Remove stale user_profile_email field.
Since years ago, this field hasn't been used for anything other than
some logging that would be better off logging the user ID anyway.

It existed in the first place simply because we weren't passing the
user_profile_id to Tornado at all.
2019-11-15 17:07:52 -08:00
Tim Abbott ddd1a0eb00 actions: Convert do_delete_messages to take a Realm.
The function only used the user's realm anyway, so this is a cleaner
API.

This should also make it more convenient to permanently delete
messages manually, since one doesn't have to fetch a random user in
the realm in order to delete a message using the management shell.

No functional change.
2019-11-12 12:20:31 -08:00
Tim Abbott 54e357e154 avatars: Fix gravatar URLs with EMAIL_ADDRESS_VISIBILITY_ADMINS.
Previously, we were using user_profile.email rather than
user_profile.delivery_email in all calculations involving Gravatar
URLs, which meant that all organizations with the new
EMAIL_ADDRESS_VISIBILITY_ADMINS setting enabled had useless gravatars
not based on the `user15@host.domain` type fake email addresses we
generate for the API to refer to users.

The fix is to convert these calculations to use the user's
delivery_email.  Some refactoring is required to ensure the data is
passed through to the parts of the codebase that do the check;
fortunately, our automated tests of schemas are effective in verifying
that the new `sender_delivery_email` field isn't visible to the API.

Fixes #13369.
2019-11-05 12:33:20 -08:00
Tim Abbott fb59f98304 test_events: Fix handling of client_gravatar flag.
Previously, we weren't properly passing through the value of the
client_gravatar flag from the caller, resulting in buggy results if
the caller passed client_gravatar=False to do_test().

We happened to not have any uses of this before, but we're about to
add one.
2019-11-05 12:33:20 -08:00
Matheus Melo c96762b7a9 settings: Add setting for who can edit user groups.
Fixes #12380.
2019-11-03 16:45:13 -08:00
Hemanth V. Alluri 34849d7355 test_events: Fix a small typo for test_realm_bots_admin. 2019-10-23 14:35:32 -07:00
Pragati Agrawal 857470c5ca events: Add delivery_email to user initial state data.
For admins, this adds `delivery_email` in addition to `email` for the
case when email address visibility is set to admins only.
2019-10-21 15:43:49 -07:00
Mateusz Mandera bbf2474bd0 tests: setUp overrides should call super().setUp().
MigrationsTestCase is intentionally omitted from this, since migrations
tests are different in their nature and so whatever setUp()
ZulipTestCase may do in the future, MigrationsTestCase may not
necessarily want to replicate.
2019-10-19 17:27:01 -07:00
Rishi Gupta 4256ee61cf billing: Change RealmAuditLog.event_type from str to int.
This is a more robust long-term model for storing these data.
2019-10-06 15:55:56 -07:00
Mateusz Mandera 4166c901ef do_update_user_custom_profile_data: Rename to ..._if_changed.
This adds clarity to the fact that the function no longer does
anything if the field values haven't changed.
2019-10-01 13:52:43 -07:00
Tim Abbott 96726c00ce export: Fix broken URLs in UI with S3 backend.
Apparently, the Zulip notifications (and resulting emails) were
correct, but the download links inside the Zulip UI were incorrectly
not including S3 prefix on the URL, making them not work.

While we're at this, we rewrite the somewhat convoluted previous
system for formatting the data export output.
2019-09-24 13:56:49 -07:00
Tim Abbott c19b7ec89d test_events: Remove unnecessary check_none validator.
This can be done cleanly using the existing equals(None).
2019-09-20 11:51:50 -07:00
Anders Kaseorg fc0554fcb8 emoji: Finish removing leftover code from banned Emoji sets.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-09-19 12:08:52 -07:00
Wyatt Hoodes d2c3ddc765 test_events: Use stdout_suppressed in export test. 2019-09-13 11:54:14 -07:00
Tim Abbott 5e19546c14 messages: Fix unread_msgs accounting of wildcard mentions.
Previously, the unread_msgs data structure accounting (used for both
the web and mobile apps to determine the "Unread mentions" count
displayed in the UI) did not include wildcard mentions at all.

We fix this by adding the logic required to include properly that
data, with tests.  As discussed in #6040, it makes sense to include
muted streams and topics for the purpose of this calculation.

Fixes part of #6040.
2019-08-25 20:32:12 -07:00
Mateusz Mandera 0edc7d74ff test_events: Eliminate hard-coded user id. 2019-08-21 21:28:09 -07:00
Hemanth V. Alluri f375350446 page_params: Send the list of config_options for incoming webhook bots.
Send the config_options for each supported incoming webhook bot along
with the initial state (not present in apply_events since this is
mostly just static data).
2019-08-20 17:00:48 -07:00
Mateusz Mandera 26f7dd09ce display_recipient: Eliminate raw_display_recipient from message dicts.
This was used as a helper to construct the final display_recipient when
fetching messages. With the new mechanism of constructing
display_recipient by fetching appropriate users/streams from the
database and cache, this shouldn't be needed anymore.
2019-08-20 12:22:52 -07:00
Wyatt Hoodes 896886b412 realm_export: Add endpoint for export deletion. 2019-08-12 17:51:46 -07:00
Wyatt Hoodes 7c118096e7 test_events: Remove print statement from self.client_post.
Leftover from prior debugging.
2019-08-12 17:51:46 -07:00
Wyatt Hoodes 7d178bbb0f queue_processors: Clean up the extra_data dict code.
We don't want to add a `deleted_timestamp` key until
the export is actually deleted.
2019-08-12 17:51:46 -07:00
Wyatt Hoodes 6f6efa516d exports: Refactor extra_data to export_data. 2019-08-12 17:51:46 -07:00
Wyatt Hoodes 7a2a1f29ad exports: Refactor event_time to export_time timestamp.
The time of the event was incorrectly being sent
as a datetime object.
2019-08-12 17:51:46 -07:00
Anders Kaseorg c453c4b602 test_events: Clean up type ignores.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-09 17:42:33 -07:00
Tim Abbott 4e631876f4 test_events: Fix realm export test to actually work.
The previous iteration still had the failure mode of not actually
testing anything, because it didn't trigger the data export code path
(and in fact was getting an HTTP 401 authentication denied error).
2019-08-07 14:18:27 -07:00
Wyatt Hoodes 22842dab34 events: Rename notify_export_completed.
notify_realm_export is more reasonable for the context of doing
deletion events as well.
2019-08-07 14:18:27 -07:00
Wyatt Hoodes 7853e9d2de test_events: Fix broken test case.
This test was broken due to using an empty `RealmAuditLog`
table.  We fix this by mocking the creation of an export,
thus creating an entry, similar to what we do in our other
tests.
2019-08-07 12:05:44 -07:00
Wyatt Hoodes 11db0c23fb exports: Update extra_data field to a JSON structure.
We add the `deleted_timestamp` key to the new `extra_data`
dictionary.
2019-08-07 12:04:28 -07:00
Tim Abbott 3a6e5dad53 tests: Move EventQueueTest to test_event_queue.py.
This seems more appropriate, especially as we expand this library.
2019-08-02 14:43:12 -07:00
Mateusz Mandera d3d62f99e4 tests: Adjust to new changes in populate_db. 2019-07-31 16:01:37 -07:00
Wyatt Hoodes bbbea9ec87 events: Rewrite system for managing realm exports.
This feature is intended to cover all of our ways of exporting a
realm, not just the initial "public export" feature, so we should name
things appropriately for that goal.

Additionally, we don't want to include data exports in page_params;
the original implementation was actually buggy and would have.
2019-07-26 16:38:52 -07:00
Harshit Bansal bf14a0af4d auth: Migrate google auth to python-social-auth.
This replaces the two custom Google authentication backends originally
written in 2012 with using the shared python-social-auth codebase that
we already use for the GitHub authentication backend.  These are:

* GoogleMobileOauth2Backend, the ancient code path for mobile
  authentication last used by the EOL original Zulip Android app.

* The `finish_google_oauth2` code path in zerver/views/auth.py, which
  was the webapp (and modern mobile app) Google authentication code
  path.

This change doesn't fix any known bugs; its main benefit is that we
get to remove hundreds of lines of security-sensitive semi-duplicated
code, replacing it with a widely trusted, high quality third-party
library.
2019-07-21 20:51:34 -07:00
Rishi Gupta c9f194be99 notification bot: Update message for stream name changes.
Also properly internationalizes string formatting arguments for the stream
created message.
2019-07-20 14:39:15 -07:00
David Wood 9bace3f2cd notifications: Allow only notifiable in unread count.
This commit adds a new setting to the user's notification settings that
will change the behaviour of the unread count in the title bar and
desktop application.

When enabled, the title bar will show the count of unread private messages
and mentions. When disabled, the title bar will act as before, showing
the total number of unread messages.

Fixes #1736.
2019-07-13 15:49:04 -07:00
Wyatt Hoodes 62566f02d7 tests: Use TEST_WORKER_DIR for file access. 2019-07-12 17:42:18 -07:00