Commit Graph

14473 Commits

Author SHA1 Message Date
Sahil Batra e47662aa19 user_groups: Rename get_memberships_of_users.
This is a prep commit for new permissions model in
which a user group would be able to have a subgroup.

This commit renames get_memberships_of_users to
get_direct_memberships_of_users to specify that
the function is used only to fetch the direct
memberships and not memberships of subgroups of
the direct group.

Extracted this commit from #19866.

Co-authored-by: Anders Kaseorg <anders@zulip.com>
2021-10-12 13:57:47 -07:00
Sahil Batra 07e2ecf221 user_groups: Rename get_user_groups to get_direct_user_groups.
This is a prep commit for new permissions model in which a user
group would be able to have a subgroup.

This commit renames get_user_groups to get_direct_user_groups
to specify that the function is used only to fetch the direct
groups that user is part of and not subgroups of the direct
group.

Extracted this commit from #19866.

Co-authored-by: Anders Kaseorg <anders@zulip.com>
2021-10-12 13:57:32 -07:00
Sahil Batra 82af2d3db5 user_groups: Rename get_user_group_members to get_user_group_direct_members.
This is a prep commit for new permissions model in which a user group would
be able to have a subgroup.

This commit renames get_user_group_members to get_user_group_direct_members
to specify that the function is used only to fetch direct members of group
and excludes the subgroup's members.

Extracted this commit from #19866.

Co-authored-by: Anders Kaseorg <anders@zulip.com>
2021-10-12 13:57:26 -07:00
Dinesh e2df0d171f user_settings: Add send_read_receipts setting.
This will be useful to let users enable/disable
sharing read receipts once we add that feature.

Note: Added "I've" to IGNORED_PHRASES in
tools/lib/capitalization.py to avoid capitalization
errors for the label text of this setting.
2021-10-07 17:46:05 -07:00
Dinesh 9cea88c1a5 typing_notifications: Make privacy settings functional.
Changes the view code to skip sending typing events
when these settings are disabled. Returns a json success
response with a msg.
2021-10-07 17:39:22 -07:00
Dinesh 6d2b8f5ca9 user_settings: Add settings to configure sending typing notifications.
Note: These are not functional in enabling/disabling sending of
typing notifications with this commit.

Refactored the privacy settings update to keep the code less
duplicated along with making the addition of new settings easier.
2021-10-07 17:39:21 -07:00
Dinesh 0b68f264c6 apply_event: Do not include new user settings in the top level.
From 430c5cb, in `fetch_initial_state_data`,
we only include legacy settings in the top level of
`state` and the newer ones are stored in `state['user_settings']`.
That should've had a corresponding change in apply_event().

Also, fixed a test related to this logic.
2021-10-07 16:39:51 -07:00
Dinesh 47da353366 Fix change_user_settings_test to consider sending only one event.
From 430c5cb, we do not send events for settings added after
introduction of `user_settings` event.

This test wasn't considering that and started failing on
adding new user_settings.
2021-10-07 16:39:51 -07:00
Eeshan Garg 29b354346b update_realm: Allow demo orgs to be converted to regular orgs.
This commit adds support to the `PATCH /realm` endpoint for
converting a demo organization to a regular organization.

This is a part of #19523.
2021-10-07 14:43:03 -07:00
Sahil Batra e575f83351 test_subs: Remove _test_user_settings_for_adding_streams.
This commit removes _test_user_settings_for_adding_streams
and its callers for testing public and private streams
because it uses excessive mocking and we also test the same
thing in _test_user_settings_for_creating_streams without
mocking, so this test doesn't add anything.
2021-10-07 14:22:10 -07:00
Eeshan Garg f25230c7d4 analytics/support: Add ability to edit realm org type. 2021-10-07 14:08:11 -07:00
rht 58b19761b8 slack import: Fix requests.get usage of get_slack_api_data.
We also rewrite the tests using the `responses` module to avoid the
problematic mocking that made this bug possible.

Fixes #19833.
2021-10-07 11:46:23 -07:00
rht d8e1409fe5 Slack import: Use Python ZipFile to unzip.
This should handle the case when non-ASCII Unicode folder names are
created on Windows.

Fixes #19899.
2021-10-07 09:24:19 -07:00
Alya Abbott 8045355eee user docs: Document public vs. private stream creation permissions. 2021-10-06 17:33:19 -07:00
Aman Agrawal 3b29c61409 auth: Add login buttons for web public access.
For users who are not logged in and for those who don't have
'prefers_web_public_view' set in session, we redirect them
to the default login page where they can choose to login
as spectator or authenticated user.
2021-10-05 16:30:40 -07:00
Sahil Batra ab0d852fd5 register: Add can_create_web_public_streams to register response. 2021-10-05 10:10:32 -07:00
Sahil Batra 2c2c19c4d4 streams: Restrict creating web public streams based on new setting.
This commit restricts creating web public streams based on the
recently added create_web_public_stream_policy setting.
2021-10-05 09:56:00 -07:00
Sahil Batra 3916181770 models: Add can_create_web_public_streams helper.
This commit adds can_create_web_public_streams helper
in models.py which will be used to validate whether
user is allowed to create a web-public stream or not.

This commit also adds the checks for Realm.POLICY_OWNERS_ONLY
in check_has_permission_policies.
2021-10-05 09:48:50 -07:00
Sahil Batra 5f950e3efd models: Add create_web_public_stream_policy setting.
This commit adds create_web_public_stream_policy
field to Realm table which controls the roles that
can create web-public streams and by default its
value is set to POLICY_OWNERS_ONLY.
2021-10-05 09:28:28 -07:00
Sahil Batra be0387b189 test_subs: Enforce invite_only argument to be named.
This commit enforces invite_only argument to be named
in _test_user_settings_for_creating_streams. This will
help in improving readability especially when we will
add is_web_public argument in further commits.
2021-10-05 09:12:56 -07:00
Sahil Batra a63385f47f tests: Check all cases in check_has_permission_policies.
This commit adds tests for POLICY_EVERYONE and POLICY_NOBODY
in check_has_permission_policies test. The original code
used these values but these were not covered in test.
2021-10-05 09:12:56 -07:00
Sahil Batra c07b8e9b44 tests: Validate owner users also in check_has_permission_policies.
We should validate owner user also for all policy values. This is
important when we will add tests for POLICY_NOBODY in next commit.
2021-10-05 09:12:56 -07:00
Sahil Batra 855f7158ea models: Remove redundant check for POLICY_EVERYONE.
We check whether policy value is POLICY_EVERYONE in
has_permission itself so there is no need to handle
that in can_edit_topic_of_any_message.
2021-10-05 09:12:56 -07:00
Sahil Batra dbcdd08324 test_events: Test the email event when changing delivery_email.
We send three events when changing delivery email of a user - one
for updating the delivery_email field of user, one for avatar url
change, and one for changing email field if email_address_visibility
is set to 'EMAIL_ADDRESS_VISIBILITY_EVERYONE'.

There is already a test for delivery_email and avatar_url event with
the visibility setting set to 'EMAIL_ADDRESS_VISIBILITY_ADMINS_ONLY',
but no test for verifying the email update event sent when email
address is public, so this commit adds a test for checking the schema
of event for updating email field.
2021-10-05 09:12:20 -07:00
Aman Agrawal eca8f0ffbf openapi: Fix broken link to documentation. 2021-10-04 19:46:29 -07:00
Sahil Batra eedd832ca9 events: Fix updating user email on changing email_address_visibility.
When email_address_visibility is changed and either the old value
or the updated value is EMAIL_ADDRESS_VISIBILITY_EVERYONE then
email field of all users is updated and we also send the corresponding
event to clients. But apply_event code did not update the data on
receiving the event, so this commit fixes the code to correctly
handle the event in apply_event.

(We also use this event when just changing a user's email address).

This commit also adds the tests and openapi schema for the event.
2021-10-04 17:24:31 -07:00
Sahil Batra 402968ccf3 test_events: Use the list defined in models.py for realm update tests.
We use the lists defined in models.py like Realm.COMMON_POLICY_TYPES,
Realm.COMMON_MESSAGE_POLICY_TYPES, etc. in do_set_realm_property_test
instead of using defining list there (eg - [4, 3, 2, 1]). We do the
same thing in do_set_realm_property_test in test_realm.py.

We skip email_address_visibility values in this commit because it
requires some change in openapi schema as well.
2021-10-04 17:04:54 -07:00
Eeshan Garg 8123e62643 topics: Add test for translation issue with resolve notifications.
This commit adds a test that makes sure that that the fix in
PR #19828 is working as expected.
2021-10-04 16:20:16 -07:00
Eeshan Garg c4aeb159c4 topics: Fix translation issue with moved topic notifications.
Since the calls to the translation function `_()` are made outside
of the `send_message_moved_breadcrumbs` function, these strings are
translated outside of the `with override_language` block, leading to
translated strings even when we don't intend them to be translated.

We now use gettext_lazy with appropriate testing to avoid this.
2021-10-04 16:20:16 -07:00
Alex Vandiver db934be064 CVE-2021-41115: Use re2 for user-supplied linkifier patterns.
Zulip attempts to validate that the regular expressions that admins
enter for linkifiers are well-formatted, and only contain a specific
subset of regex grammar.  The process of checking these
properties (via a regex!) can cause denial-of-service via
backtracking.

Furthermore, this validation itself does not prevent the creation of
linkifiers which themselves cause denial-of-service when they are
executed.  As the validator accepts literally anything inside of a
`(?P<word>...)` block, any quadratic backtracking expression can be
hidden therein.

Switch user-provided linkifier patterns to be matched in the Markdown
processor by the `re2` library, which is guaranteed constant-time.
This somewhat limits the possible features of the regular
expression (notably, look-head and -behind, and back-references);
however, these features had never been advertised as working in the
context of linkifiers.

A migration removes any existing linkifiers which would not function
under re2, after printing them for posterity during the upgrade; they
are unlikely to be common, and are impossible to fix automatically.

The denial-of-service in the linkifier validator was discovered by
@erik-krogh and @yoff, as GHSL-2021-118.
2021-10-04 21:26:24 +00:00
Alex Vandiver f1c61fbea1 openapi: Use more carefully-constructed regexes for markdown extension.
This removes a false-positive ReDoS, since the input is always
checked-in code.  It also incidentally refactors to make the regexes
be more explicit about the values they expect, and removes unnecessary
capturing groups.

It removes an optional parenthesized status code for fixtures,
unnecessary since 981e4f8946, as well as
optional key-value language options, unnecessary since
a2be9a0e2d.

Thank you to @erik-krogh and @yoff for bringing this to our attention.
2021-10-03 18:49:38 -07:00
Pradyumna Sinha 18b36e5b8c markdown/tabbed_sections: Raise exception for missing tab name.
This fixes the issue  where 'None' would appear in the rendered
html in case of a missing tab display_name. Now,
'test-help-documentation' will fail in case of any tab display_name
being missing.

In case of a tab_section with no tabs, currently a single tab with
the name 'null_tab' gets added. Added the display name 'None' for
'null_tab', to keep in line with the existing behaviour.

Fixes #19822
2021-10-01 16:25:17 -07:00
Pradyumna Sinha 78692e9c14 markdown/tabbed_sections: Don't set tab label as 'None' in case of no tabs. 2021-10-01 16:22:11 -07:00
Pradyumna Sinha fe5de4a128 markdown/tabbed_sections: Rename TAB_DISPLAY_NAMES to TAB_SECTION_LABELS. 2021-10-01 16:22:11 -07:00
Shlok Patel 1e18d36328 realm: Change default permissions for education organizations.
This makes our onboarding guide for education organizations much
simpler, since new organizations will start with these settings
correctly configured.

Fixes #19682
2021-10-01 15:33:33 -07:00
Ganesh Pawar fa928d5cd1 streams: Split setting for stream creation policy.
Users wanted a feature where they could specify
which users can create public streams and which users can
create private streams.

This splits stream creation code into two parts,
public and private stream creation.

Fixes #17009.
2021-10-01 10:26:42 -07:00
sahil839 9dd69c17ee settings: Add moderators and members option in delete_own_message_policy.
This commit adds moderators, full members and members options to
delete_own_message_policy in backend.
2021-09-30 14:59:31 -07:00
sahil839 909a3cde76 realm: Replace allow_message_deleting with delete_own_message_policy.
This commit replaces 'allow_message_deleting' boolean setting
with an integer setting 'delete_own_message_policy'. We have a
separate dropdown now for deciding which user-roles can delete
messages sent by themselves and the time-limit setting droddown
is different.

This new setting has two options - everyone and admins only. Other
options including moderators will be added further.

We also remove the "Never" option from the original time-limit
dropdown, as admins are always allowed to delete message. This
never option resembled the case of only admins being allowed to
delete but this state is now resembled by setting the dropdown
to "admins only" and we also disable the time-limit dropdown in
this case as admins are allowed to delete irrespective of limit.

Note, this setting is only for deleting messages sent by the
deleting user themselves, and only admins are allowed to delete
messages sent by others as before.
2021-09-30 14:59:31 -07:00
sahil839 b13bfa09c5 message: Make zero invalid value for message_content_delete_limit_seconds.
We make zero invalid value for message_content_delete_limit_seconds and
for handling the case of "Allow to delete message any time", the API-level
value of message_content_delete_limit_seconds is "anytime" and "None"
as the DB-level value. We also use these values for message retention
setting, so it helps maintain consistency.
2021-09-30 14:45:39 -07:00
Mateusz Mandera 0af7c84c99 push_notifs: Log the number of devices notification was sent to. 2021-09-29 15:50:06 -07:00
Sahil Batra d1732fb9da settings: Remove realm-level default of enable_login_emails setting.
This commit does not remove the 'enable_login_emails' field from
RealmUserDefault table but it is just not used and cannot be
changed from UI or API similar to 'enable_marketing_emails' setting.
2021-09-28 12:41:09 -07:00
Aman Agrawal ef84224eed message_edit: Allow spectators to access raw message content.
We allow spectators to fetch the raw / original content of a
message which is used by the spectator to "View source" of
the message.
2021-09-28 10:07:36 -07:00
Tim Abbott e556481ba0 streams: Remove duplicates of get_web_public_streams_queryset.
This is a somewhat subtle function, that deserves a few comments
explaining subtle details of its logic, and there's no good reason to
have multiple copies of that logic that are slightly inconsistent.

Because the main changes here are just checking for invariant
failures, the behavioral change here should be limited to ensuring
deactivated streams are not considered available even if they were
tagged as web public streams before deactivation.
2021-09-28 10:07:36 -07:00
Tim Abbott 272e81988b tornado: Make process_notification hack avoid import loops.
This fixes a problem where we could not import zerver.lib.streams from
zerver.lib.message, which would otherwise be reasonable, because the
former implicitly imported many modules due to this issue.
2021-09-28 10:07:36 -07:00
Mateusz Mandera 8b906b5d2f request_notes: Set the realm appropriately for the root subdomain.
Requests to the root subdomain weren't getting request_notes.realm set
even if a realm exists on the root subdomain - which is actually a
common scenario, because simply having one organization, on the root
subdomain, is the simplest and common way for self-hosted deployments.
2021-09-28 10:02:52 -07:00
Anders Kaseorg 0f7d0a23c9 Revert "validator: Add generic check_or."
This reverts commit cd93d0967f.

This check_or is redundant with check_union; it gives a misleading
error message for the non-matching case; and it has no type safety.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-09-28 09:28:56 -07:00
seiwailai cd93d0967f validator: Add generic check_or.
Added generic check_or function and tests.
Fixes part of #17914.

Co-authored-by: Gaurav Pandey <gauravguitarrocks@gmail.com>
2021-09-27 17:30:26 -07:00
Kevin Scott 09851ff290 markdown: Document built-in preprocessor priorities for convenience.
Fixes #19810
2021-09-27 10:42:09 -07:00
Eeshan Garg bf64cb2d1c topics: Fix translation issue with resolve topic notifications.
In maybe_send_resolve_topic_notifications, since the calls to the
translation function `_()` are made outside of the `override_language`
block, the strings are not translated correctly.

This commit refactors the function to make sure that the translation
happens in the right block of code.

Fixes #19730.
2021-09-23 17:21:33 -07:00
Tim Abbott 417c32629d outgoing webhooks: Fix inconsistencies with Slack's API.
Apparently, our slack compatible outgoing webhook format didn't
exactly match Slack, especially in the types used for values.  Fix
this by using a much more consistent format, where we preserve their
pattern of prefixing IDs with letters.

This fixes a bug where Zulip's team_id could be the empty string,
which tripped up using GitLab's slash commands with Zulip.

Fixes #19588.
2021-09-23 11:19:20 -07:00
Sahil Batra c233ee9935 settings: Migrate twenty_four_hour_time setting to RealmUserDefault.
This commit removes the existing default_twenty_four_hour_time field in
Realm table which was used to set the twenty_four_hour_time setting of
new user on joining and instead we now use the twenty_four_hour_time
field of RealmUserDefault table for the same.

With some tweaks by tabbott to clarify the documentation.
2021-09-23 10:44:42 -07:00
Alya Abbott 0f9d0f5a86 help_docs: Reorganize notification docs under three umbrella pages.
With minor URL migration work by Eeshan Garg.
2021-09-22 16:46:46 -07:00
Mateusz Mandera 491f763f4e models: Improve conversion of CustomProfileField values.
These values are currently either a string already or a List[int]. We
should do the conversion in
do_update_user_custom_profile_data_if_changed properly: if the value is
already a string, it can be used directly - if it's not, orjson.dumps is
a more future-proof way of converting than str(). Using orjson.dumps
here also allows us to change the converter of the USER type
CustomProfileField to orjson.loads, which is nicer to have than
ast.literal_eval.
While orjson.dumps() and str() give the same output when
given the special case of List[int],
ast.literal_eval was previously used due to orjson.loads not being
a good inverse function to str in general. That gets straightened out
now.
2021-09-22 16:22:46 -07:00
Mateusz Mandera cacff28578 types: Fix declared type of custom profile field values.
None of the existing custom profile field types have the value as an
integer like declared in many places - nor is it a string like currently
decalred in types.py. The correct type is Union[str, List[int]]. Rather
than tracking this in so many places throughout the codebase, we add a
new ProfileDataElementValue type and insert it where appropriate.
2021-09-22 16:22:46 -07:00
Mateusz Mandera 91ea21a3fc actions: Fix type-incorrect custom profile field value assignment.
The old assignment is incorrect - field_value.value is a TextField() and
should always be a string. This didn't strictly break anything, because
django converts the value to a string when .save()ing to the db, but
field_value.value persists as a non-string for the rest of this
codepath. After fixing this, the small codeblock in
notify_user_update_custom_profile_data handling conversion of
field_value.value to a string becomes redundant.

We're assured that we're not breaking event format by the test
test_custom_profile_field_data_events in test_events.py.
2021-09-22 16:22:46 -07:00
Mateusz Mandera c4edda016f models: Remove property decorator from UserProfile.profile_data.
Having this as a property is misleading, given that it does db queries
to fetch the CustomProfileField data.
2021-09-22 16:22:46 -07:00
Anders Kaseorg bd3c099978 test_decorators: Convert "".format to Python 3.6 f-strings.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-09-22 14:59:42 -07:00
Alya Abbott c7544d620d user docs: Fix missing tab name on /help/getting-your-organization-started-with-zulip. 2021-09-22 14:29:44 -07:00
Alya Abbott b773048031 user docs: Add documentation for configuring new user defaults.
Also remove a redundant link on customize-settings-for-new-users page.
2021-09-21 16:43:35 -07:00
Eeshan Garg 4118c4a56b find_team: Send one email per email address, not per organization.
With changes to the copy for these emails by tabbott.

Fixes #19659.
2021-09-21 15:56:50 -07:00
Aman Agrawal 233d250eff do_make_stream_web_public: Send update to client.
Send update event to client after a stream is made web public.

This has been documented in the API documentation since feature level
73; previously the value was always false.
2021-09-21 12:16:10 -07:00
Aman Agrawal 5138652810 update_stream_backend: Add ability to make streams web public.
We allow clients to make existing streams web public via the API.

This feature is still disabled via settings in production
environments, because we may have additional policy rules or UI
warnings we wish to add to this sort of conversion.
2021-09-21 12:16:09 -07:00
Aman Agrawal 6a78112940 subscribe: Allow web public stream creation via the API.
User can now create web public stream via the /subscribe API.
So, when a web public stream present in the API request does not
exist, it will be created now by specifying the is_web_public
parameter. The parameter would have been ignored without this
commit.
2021-09-21 11:20:36 -07:00
Tim Abbott eb62693d26 list_to_streams: Use simpler if statement flow.
Since each if statement just raises an exception if triggered,
removing the else blocks makes the logic more readable.
2021-09-21 11:06:05 -07:00
Tim Abbott 71b8a1794a streams: Use standard error message when requiring owner.
The new error message is more clear about why, "User cannot create
stream with this settings." was bad English, and in any case removing
an unnecessary string is always an improvement for translators.
2021-09-21 11:05:30 -07:00
Tim Abbott 8fe61674d4 settings: Add web_public_streams_enabled server setting.
This new setting both serves as a guard to allow us to merge API
support for web public streams to main before we're ready for this
feature to be available on Zulip Cloud, and also long term will
protect self-hosted servers from accidentally enabling web-public
streams (which could be a scary possibility for the administrators of
a corporate Zulip server).
2021-09-21 10:50:53 -07:00
Aman Agrawal 9566d06429 home: Use already imported redirect shortcut for redirects. 2021-09-21 10:05:30 -07:00
Eeshan Garg 5839f16969 markdown: Run settings_tab/relative links extensions after include.
Recently, we discovered that our settings_tab/relative Markdown
directives didn't work when they were in a macro that was included
in another Markdown file. Note that without this commit, the
/help/create-your-organization-profile page is broken. This commit
changes the respective priorities of these two extensions such that
these directives are rendered *after* the macro is included in
another file.

Thanks to Alya Abbott for reporting this bug!
2021-09-20 16:57:43 -07:00
Eeshan Garg bfbd77ca5c markdown: Organize preprocessor priorities in one place.
All of our custom Markdown extensions have priorities that govern
the order in which the preprocessors will be run. It is more
convenient to have these all in one file so that you can easily
discern the order at first glance.

Thanks to Alya Abbott for reporting the bug that led to this
refactoring!
2021-09-20 16:57:43 -07:00
Alya Abbott 2f3ebb5b16 help: Merge "Change who can add custom emoji" into "Custom emoji" page. 2021-09-17 18:08:59 -07:00
Alya Abbott 302aba035a help: Rename add-custom-emoji -> custom-emoji. 2021-09-17 18:08:59 -07:00
Abhijeet Prasad Bodas 743712c267 user_topic: Filter on visibility_policy in muted_topic code.
This is the final step required for it to be possible for us to add
UserTopic rows in the future with a differen visibility_policy.

Fixes #19272.
2021-09-17 12:14:28 -07:00
Abhijeet Prasad Bodas bc60109867 models: Add visibility policy indexes on UserTopic.
Part of #19272.
2021-09-17 12:14:28 -07:00
Abhijeet Prasad Bodas 2153aa9495 models: Add field UserTopic.visibility_policy.
Part of #19272
2021-09-17 12:14:28 -07:00
Abhijeet Prasad Bodas 4455dac424 models: Use default db table name for UserTopic.
Also update the realm import/export code and tests accordingly.
2021-09-17 12:14:28 -07:00
Abhijeet Prasad Bodas 2aea944a7e models: Rename UserTopic.date_muted to last_updated.
This is a follow-up to #19388.

We will in the future allow patch requests to change the visibility
of an existing topic, so `last_updated` is better name for this field.

This commit does not affect the API or events in any way, but only the
database.
2021-09-17 12:14:28 -07:00
Mateusz Mandera 8ad7520180 ldap: Support alternative attrs to userAccountControl.
Fixes #17456.

The main tricky part has to do with what values the attribute should
have. LDAP defines a Boolean as
Boolean = "TRUE" / "FALSE"
so ideally we'd always see exactly those values. However,
although the issue is now marked as resolved, the discussion in
https://pagure.io/freeipa/issue/1259 shows how this may not always be
respected - meaning it makes sense for us to be more liberal in
interpreting these values.
2021-09-16 13:29:13 -07:00
Alya Abbott 310b8736af user docs: Add a bunch of "getting started" content. 2021-09-16 11:41:43 -07:00
Gilbert Bishop-White 28932ba8ee test_signup: Refactor test_signup_without_full_name to use verify_signup.
The test now uses submit_reg_form_for_user, meaning a blank
full_name is posted to /accounts/register/ rather than the
parameter being excluded.

Fixes part of #7564
2021-09-16 10:51:01 -07:00
Gilbert Bishop-White 330d1385d9 test_signup: Refactor test_signup_without_password to use verify_signup.
I had to pass stop_after_reg_form=True, as the call to get_user in
verify_signup fails. I am not sure whether this is the expected
behavior. Also this causes the test to use submit_reg_form_for_user,
meaning a blank password is posted to /accounts/register/ rather than
no password.

Fixes part of #7564
2021-09-16 10:51:01 -07:00
Gilbert Bishop-White a632fc0e10 test_signup: Refactor test_signup_invalid_name to use verify_signup.
Extend `signup` helper function to enable it's use, and add additional
checks from  `test_signup_invalid_name`

Fixes part of #7564
2021-09-16 10:51:01 -07:00
Mateusz Mandera 63162210c1 test_signup: Fix the fetching of UserProfile in verify_signup.
get_user_by_delivery_email should be used, given that the email variable
is the realm email address that the account is being created with, not
the .email field which can be a dummy address based on org settings.
2021-09-16 10:51:01 -07:00
Mateusz Mandera 99ade3f535 test_signup: Fix asserted redirect Location in verify_signup.
The expected Location should depend on the realm.
2021-09-16 10:51:01 -07:00
Mateusz Mandera 05fe62c96c docs: Adjust phrasing of "From your desktop, click on the gear...".
We actually mean the main Zulip UI here, so "from your desktop" sounds
potentially misleading; reodering the phrasing makes the meaning
unambiguous.
2021-09-16 10:47:56 -07:00
Vishnu KS 392b2900cf plans: Redirect to /upgrade when clicked on buy standard from root.
Currently we used to redirect to /new when the user click on buy
standard from the root domain. Instead we redirect to /upgrade page.
The /upgrade page redirect would ask user to enter the subdomain
of their organization and would then redirect them to /upgrade
page of their organization.
2021-09-15 17:38:50 -07:00
Tim Abbott 39c897a2b0 help: Rename about-streams-and-topics to streams-and-topics.
This better matches the title of the page and more generally our
conventions around naming /help/ articles.  We include a redirect
because this is referenced from Welcome Bot messages, and we
definitely don't want those links to break.
2021-09-15 09:48:36 -07:00
Alex Vandiver 8c72959951 push_notifications: Handle empty rendered_messages.
This parallels fe25517295, but for mobile notifications.  It also
adds a test, which verifies that such content does not crash either
mobile or email notifications.
2021-09-15 09:45:38 -07:00
Alex Vandiver 039b869ca5 email_notifications: Fix inline-ing of image-URL-only messages.
fe25517295 adjusted the email_notifications codepath to use
`lxml.html.fragment_fromstring` method when parsing
`rendered_content`, but left the tests using a helper which called
`fromstring`.

Switching the tests to match the code as run reveals a bug -- using
`drop_tree` on all `message_inline_image` classes now _does_ remove
all of a top-level image-URL-only message.  Previously, such messages
were "safe" from the block that calls `drop_tree` only by dint of
`drop_tree` being a silent no-op for the root element.  When parsed
using `fragment_fromstring`, they are no longer the root, and as such
an empty message results.

Reorder relative_to_full_url to check for only one `message_inline_image`
within the top `<div>`, and only run the `drop_tree` path in the
alternate case.  Tests must be adjusted for their output now including
one more layer of `<div>`.
2021-09-15 09:45:38 -07:00
Mateusz Mandera c460351898 auth: Add logging of successful attempts to social codepath.
The previous commit introduced logging of attempts for username+password
backends. For completeness, we should log, in the same format,
successful attempts via social auth backends.
2021-09-14 15:39:41 -07:00
Mateusz Mandera 5d54cd1041 tests: Use a HttpRequest object in assert_login_failure.
Our convention is to always have authenticate() called with a request
object. We need to be consistent with that in tests too, to avoid test
failures resulting from breaking that assumption.
We modify assert_login_failure to call client.login() in the same way as
the other similar helpers - with a properly initialized HttpRequest
instance.
2021-09-14 15:39:41 -07:00
Aman Agrawal 354491d037 models: Add is_web_public property to Realm model.
This will allow us easily know if a realm has web_public
enabled or not.
2021-09-14 10:33:53 -07:00
Aman Agrawal 5a2a787b76 user_passes_test: Extract zulip_redirect_to_login from it.
This function will be re-used to
setup redirect logic in home view.
2021-09-14 10:33:53 -07:00
Aman Agrawal dff4ab0daf spectators: Add comments and assertions on security model.
These tweaks help make reasoning about the spectators security model
easier to understand.
2021-09-14 10:33:14 -07:00
Riken Shah 8c31e6f96e emoji: Add backend changes to support still image for animated emojis.
Now, when we add a custom animated emoji to the realm
we also save a still image of it (1st frame of the gif). So
we can avoid showing an animated emoji every time.
2021-09-12 07:13:04 +00:00
Mateusz Mandera ad188130ba do_send_confirmation_email: Make invite_expires_in_days optional.
create_confirmation_link has validity time as an optional argument,
because it has reasonable defaults. Thus it's a better API for
do_send_confirmation_email to make this optional as well, allowing
relying on create_confirmation_link's defaults.
2021-09-10 16:53:03 -07:00
Mateusz Mandera bea9205c4a create_confirmation_link: Turn some args into kwargs-only. 2021-09-10 16:53:03 -07:00
Mateusz Mandera 3205f680c1 do_invite_users: Turn some args into kwargs-only. 2021-09-10 16:53:03 -07:00
shanukun 8c1ea78d7d invite: Extend invite api for handling expiration duration.
This extends the invite api endpoints to handle an extra
argument, expiration duration, which states the number of
days before the invitation link expires.

For prereg users, expiration info is attached to event
object to pass it to invite queue processor in order to
create and send confirmation link.
In case of multiuse invites, confirmation links are
created directly inside do_create_multiuse_invite_link(),

For filtering valid user invites, expiration info stored in
Confirmation object is used, which is accessed by a prereg
user using reverse generic relations.

Fixes #16359.
2021-09-10 16:53:03 -07:00
Eeshan Garg 85b1f1bed9 email: Update follow-up day 1 emails to mention demo orgs.
This is a part of #19523.
2021-09-10 16:31:54 -07:00
Eeshan Garg bf9c35ded4 onboarding: Mention demo orgs in Welcome Bot's PMs.
This is a part of #19523.
2021-09-10 16:31:54 -07:00
Abhijeet Prasad Bodas 17b8d53612 email notifs: Update scheduled timestamps after batching period change.
The API for changing the batching period was added in
5db4fe8652.

This is a follow up to that commit. We also update the timestamps for
existing scheduled email notifications entries so that the effect of
changing the setting is immediate.

Part of #15280
2021-09-10 13:25:47 -07:00
Mateusz Mandera f064e3ebac auth: Ensure a realm can't be created on SOCIAL_AUTH_SUBDOMAIN. 2021-09-10 10:47:15 -07:00
Mateusz Mandera fa58bc340d auth: Make the deactivated user error message clearer. 2021-09-10 10:47:15 -07:00
Mateusz Mandera c5806d9728 auth: Include user-input email in some error messages in the login form.
Fixes #13126.
2021-09-10 10:47:15 -07:00
Mateusz Mandera fb3864ea3c auth: Change the look of SOCIAL_AUTH_SUBDOMAIN when directly opened.
SOCIAL_AUTH_SUBDOMAIN was potentially very confusing when opened by a
user, as it had various Login/Signup buttons as if there was a realm on
it. Instead, we want to display a more informative page to the user
telling them they shouldn't even be there. If possible, we just redirect
them to the realm they most likely came from.
To make this possible, we have to exclude the subdomain from
ROOT_SUBDOMAIN_ALIASES - so that we can give it special behavior.
2021-09-10 10:47:15 -07:00
Anders Kaseorg eeee095b10 openapi: Reformat Markdown descriptions with Prettier.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-09-10 10:39:25 -07:00
Anders Kaseorg e01f051638 openapi: Rewrap to avoid line break in inline code span.
This works around https://github.com/prettier/prettier/issues/11372.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-09-10 10:39:25 -07:00
Anders Kaseorg 5360bd76ac help: Add newline after admonition header for Prettier compatibility.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-09-10 10:39:25 -07:00
Anders Kaseorg c14549f275 name_restrictions: Reserve ‘mg’ and ‘front-mail’.
These hostnames only have MX records for Mailgun and Front, and will
not work as a Zulip organization.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-09-10 10:25:56 -07:00
Sahil Batra c7cb983ebd settings: Move check_settings_values to user_settings.py.
This commit moves check_settings_values to user_settings.py
from validator.py such that we can import the functions at
the top without any issue of cyclic imports.
2021-09-09 15:03:55 -07:00
Tim Abbott f1c0309971 test_users: Fix comparison of differently sorted sets.
This fixes a nondeterministic test failure in main.
2021-09-09 11:30:43 -07:00
Sahil Batra 88346949b5 messages: Do not allow mentioning system user groups.
We do not allow mentioning system user groups for now
because this can lead to circumventing the wildcard
mention restrictions. It will be enabled once we add
a setting to control that.

This is implemented by just ignoring it as one of the
mentioned user group even if the message content
inlcudes the mention syntax for it and the message
is sent normally.

We still keep the for_mention parameter for accessing
user group while sending email and push notifications
as mentioning system user groups will be allowed in
future.
This commit also removes the test for email notifications
for system user groups as we are not allowing mentioning
them.

This commit is only for backend change as we already
exclude the system groups from mention typeaheads and
other UI.
2021-09-09 11:25:33 -07:00
Parth 9ed707c34b authentication: Fix placeholder realms for redirects.
The name of the new realm created as a tombstone after renaming
a realm's subdomain is the constant 'placeholder-realm'.
This would confuse the user when shown the deactivation notice
and asking to join the realm at a new subdomain.
This PR replaces it with the original realm name to avoid confusion.
Fixes: #19677
2021-09-09 11:02:45 -07:00
sahil839 8bab712cb4 create_user: Initialize settings according to realm-level defaults.
This commit modifies the copy_user_settings code such that instead
of source user profile, we can have two types of sources - a user
profile and RealmUserDefault table of realm and then set the
settings from RealmUserDefault only is there is no user profile
as a source.

We also rename copy_user_settings to copy_default_settings for
clarity.
2021-09-09 10:55:19 -07:00
Sahil Batra 693d58265e realm: Add 'PATCH /realm/user_settings_defaults' endpoint.
The realm-level default value of settngs for new users will
be updated using this endpoint.
2021-09-09 10:55:18 -07:00
Sahil Batra 17087cf06f actions: Add do_set_realm_user_default_setting.
This commit adds do_set_realm_user_default_setting which
will be used to change the realm-level defaults of settings
for new users.

We also add a new event type "realm_user_settings_defaults"
for these settings and a "realm_user_settings_default" object
in '/register' response containing all the realm-level default
settings.
2021-09-09 10:38:38 -07:00
sahil839 7d64a9053b models: Ensure every realm has a RealmUserDefault object.
Because we create all realms with do_create_user (including in the
test suite), we just need to change that function, add a migration for
existing realms, and ensure the data import code path correctly
creates these objects.

Note that the import code path will create a RealmUserDefault row with
default values if it is not present in the import data, which is
important for importing data from other tools like Slack.
2021-09-09 10:28:44 -07:00
Sahil Batra 97f41b5abd users: Change type of enable_marketing_emails parameter in create_user.
This commit changes the type of enable_marketing_emails parameter of
create_user to Optional[bool].
The value of this parameter will be None in certain cases when user
registers through SSO and 'TERMS_OF_SERVICE=False' when there will
be no registration form and thus no value of enable_marketing_emails.
2021-09-09 10:15:07 -07:00
Sahil Batra 03a0cfda64 settings: Set enable_marketing_emails after copying user settings.
We set the enable_marketing_emails setting after copying user
settings to override the value selected in registration form.
This change is also necessary because enable_marketing_emails
field is present in RealmUserDefault to avoid copying code
but we do not use this value actually and instead we want
the setting to be set according to the value in registration
form.

We set this setting only for non-bot users since we generally
do not set any settings for bots.
2021-09-09 10:15:07 -07:00
Sahil Batra 2eec0772fb user_settings: Extract setting values checks to a function.
We extract the checks for default_language, notification_sound,
and email_notifications_batching_period_seconds setting values
in json_change_settings to a new function check_settings_values.
2021-09-09 10:15:07 -07:00
Tim Abbott c5261af2dc migrations: Fix system bot realm not having a name.
This prevented migration 0345
(517c2ed39d / #19696) from applying on
systems that were created after the refactoring that resulted in the
system bot realm potentially having null as its name.

(We've already confirmed that normal realms, created via
`do_create_realm`, shouldn't be able to have this unusual state).
2021-09-09 09:50:43 -07:00
Tim Abbott 545911b051 markdown: Remove useless locless_schemes check.
This check was copied from upstream python-markdown's "safe mode"
before they removed that feature.  The upstream history is that they
introduced this check in
2db5d1c8e4,
which was not a complete security check, and then added the
immediately following check (with an allowlist of schemes) in
0b4ffbb60e.

Their first, incomplete check provides no security benefit and makes
the code hard to reason about, so we remove it.
2021-09-09 09:03:40 -07:00
rht c24ab8c4d3 markdown: Expand list of safelisted URL schemes to match HTML spec. 2021-09-09 09:03:40 -07:00
Sahil Batra 9c4eb764e2 create_user: Do not set enter_sends value twice.
enter_sends is already included in property_types
dict, so there is no need to set it again using
setattr in copy_user_settings.
2021-09-09 09:03:06 -07:00
Sahil Batra 8b711695cb event_schema: Use notification_settings_legacy to get type.
The 'update_global_notifications' type event is sent only for
existing settings and will not be sent for new settings, so we
should use notification_settings_legacy dict to check the type
of setting value in check_update_global_notifications instead
of notification_settings_types dict.
2021-09-09 09:03:06 -07:00
Sahil Batra 526de3f7d1 settings: Remove unnecessary loop of notification_setting_types.
We still used notification_setting_types in copy_user_settings
function of create_user.py and in a test in test_event_system.py.
It is not required to do so since we have added all settings in
property_types already and we loop over property_types at both
these places which includes all settings.
2021-09-09 09:03:06 -07:00
Tim Abbott 517c2ed39d models: Make realm.name no longer nullable.
This was likely initiall created with null=True in
5c5ffd6ea3 just because we didn't have a
plan for backfilling this field, but I verified that Zulip Cloud has
no realms without a name set, and that's the place most likely to have
any form of super-legacy nameless realms.

So we can clean up this aspect of the data model without a special
migration to do something with existing realms with name=None (which I
suspect would have resulted in a 500 anyway).
2021-09-08 17:38:56 -07:00
Anders Kaseorg 1376feda93 docs: Add missing space in “emoji set”.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-09-08 15:31:28 -07:00
Wesley Aptekar-Cassels f743dda008 emoji: Change default emojiset from Google classic to Google modern.
This is a step towards updating our emoji to support newer Unicode
emojis (#19371).

See CZO for discussion:
https://chat.zulip.org/#narrow/stream/2-general/topic/emoji.20version.3F/near/1239756
2021-09-09 02:00:36 +05:30
Sahil Batra a229231dd4 tests: Remove redundant test_notify_settings test.
We already test all the notification settinsg in
test_toggling_boolean_display_settings (which is
now renamed to test_toggling_boolean_user_settings)
as all settings are now moved to property_types and
we are merging other parts also to consider all the
settings under one category.
2021-09-08 11:04:44 -07:00
Sahil Batra d6d1a139d0 settings: Use single function to test all non-bool user settings. 2021-09-08 11:04:44 -07:00
Sahil Batra 5d7ed2c63b tests: Add separate test for invalid values of user settings.
This commit adds a separate test for invalid values of user settings
and remove the existing code for it in test_change_user_setting.

This change will enable us to merge the tests for notification
settings to this because email_notification_batching_period_settings
has different invalid values than other integer values and we do the
same for realm settings also.
2021-09-08 11:04:44 -07:00
Sahil Batra de4c0d9fe9 settings: Validate desktop_icon_display_count value in 'PATCH /settings'. 2021-09-08 11:04:44 -07:00
Sahil Batra 2f61afea9f settings: Remove do_change_notification_settings function.
This commit removes the do_change_notification_settings
function and we use do_change_user_setting directly to
change all the user settings now.
2021-09-08 11:04:44 -07:00
Sahil Batra 991efaf450 actions: Make acting_user manadatory kwarg for do_change_user_setting. 2021-09-08 11:04:44 -07:00
Sahil Batra 550d97a593 settings: Refactor callers of do_change_user_setting to pass acting_user. 2021-09-08 11:04:44 -07:00
Eeshan Garg f0ea002d94 events: Add demo_organization_scheduled_deletion_date to /register.
This commit adds `demo_organization_scheduled_deletion_date` to
the `realm` section of the `/register` response so that it is
available to clients when enabled.

This is a part of #19523.
2021-09-07 17:40:33 -07:00
Shelly 6d92534d51 models: Add setters for is_realm_owner and is_moderator.
This fixes a regression where one could end up deactivating all owners
of a realm when trying to synchronize LDAP with the `is_realm_admin`
flag configured in `AUTH_LDAP_USER_FLAGS_BY_GROUP`.

With tweaks by tabbott to add is_moderator as well.

Fixes #18677.
2021-09-07 17:15:39 -07:00
Anders Kaseorg 66ad6a4583 docs: Inline code spans are not blocks.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-09-07 16:12:39 -07:00
Tim Abbott dd6cc32ae5 do_change_notification_settings: Merge into do_change_settings.
We preserve the function as a small wrapper until we can resolve the
TODO noted in the comment.
2021-09-07 10:49:56 -07:00
Tim Abbott a5d46e5cab do_change_notification_settings: Add conditional around RealmAuditLog work. 2021-09-07 10:49:56 -07:00
Tim Abbott fecdc234bf models: Rename setting-changed RealmAuditLog entry.
We will likely in the future want to start creating these for all
settings changes, and this new name will make the code more readable
when we do so.
2021-09-07 10:49:56 -07:00
Tim Abbott bd6f12329e do_change_notification_settings: Rename value->setting_value.
This is for consistency with do_change_user_setting; it'll make the
diffs a lot more readable when we merge these functions.
2021-09-07 10:49:56 -07:00
Tim Abbott 54cbdd8c46 do_change_notification_settings: Rename name->setting_name.
This is for consistency with do_change_user_setting; it'll make the
diff a lot more readable when we merge these functions.
2021-09-07 10:49:51 -07:00
Dinesh 9443e01a5d refactor: Rename do_set_user_display_setting to do_set_user_setting. 2021-09-07 10:16:42 -07:00
Dinesh 430c5cb8e7 user_settings: Create _legacy dicts for existing settings.
Since 84742a0, all settings are sent in the `user_settings` dictionary
which were previously sent inline with other fields in /register
response.

In order to simplify the process of adding new personal settings, we
want to transition to a world where new settings only need to consider
the `property_types` object, and code that needs to reference the
legacy behavior interacts with an object with `legacy` in its name.

This way, contributors working on new settings don't need to think
about the legacy code paths at all.

See https://chat.zulip.org/#narrow/stream/378-api-design/topic/user.20settings.20response.20in.20.2Fregister
to understand this better.
2021-09-07 10:14:45 -07:00
Tim Abbott 5ab64d46a6 test_signup: Fix buggy test setup.
676edb3802 incorrectly converted this to
have the link still be valid.
2021-09-06 17:55:04 -07:00
Mateusz Mandera 676edb3802 confirmation: Migration to add expiry_date step 1.
The commit:
1. Adds the new field as nullable.
2. Adds code that'll create new Confirmation with the field set
   correctly.
3. For verifying validity of Confirmation object this still uses the old
   logic in get_object_from_key() to keep things functioning until we
   backfill the old objects in the next step.

Thus this commit is deployable. Next we'll have a commit to run a
backfill migration.
2021-09-06 17:28:29 -07:00
Mateusz Mandera 1c4ad6769d actions: Fix incorrect calls to do_invite_users.
An integer or no argument is supposed to be passed.
These weren't caught by mypy because booleans are integers in python,
see https://github.com/python/mypy/issues/1757
2021-09-06 17:28:29 -07:00
Anders Kaseorg 646c04eff2 Rename default branch to ‘main’.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-09-06 12:56:35 -07:00
Aman Agrawal 34a37f55d6 spectators: Rename is_web_public_compatible functions. 2021-09-06 09:00:16 -07:00
Aman Agrawal 3e7538b974 filter: Copy is_web_public_compatible from server to frontend.
This will be used to check if the narrow being requested by
spectator requires authentication without requesting the server.
Having this check locally, makes this process look snappy to
the user and doesn't result in 404s in the browser log.
2021-09-06 09:00:16 -07:00
Anders Kaseorg 9399b95fec push_notifications: Remove redundant APNs retry loop.
aioapns already has a retry loop.  By default it retries forever on
ConnectionError and ConnectionClosed, so our own retry loop would
never be reached.  Remove our retry loop, and configure aioapns to
retry APNS_MAX_RETRIES times on ConnectionError like the previous
version did.  It still retries forever on ConnectionClosed; that’s not
configurable but probably fine.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-09-03 11:49:33 -07:00
Anders Kaseorg 4e2cba1ce1 test_push_notifications: Add test for unexpected APNs error.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-09-03 11:49:33 -07:00
Anders Kaseorg 56a9d669f8 test_push_notifications: Create futures for the right event loop.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-09-03 11:49:33 -07:00
Eeshan Garg f5c26990c2 settings: Add constant for demo org deadlines.
We set the 30 days deadline timedelta for demo orgs as a constant
in default_settings.py so that it can be overridden in tests.
2021-09-03 09:06:29 -07:00
PIG208 82a0063cef push_notifications: Remove unused stream_name.
These are some leftovers from #10745.
2021-09-03 08:48:45 -07:00
PIG208 e73d55af91 push_notifications: Refactor trigger from Message objects.
This is a cleaner way to reduce monkey-patched attributes we added
to the Message objects.
2021-09-03 08:48:45 -07:00
PIG208 53888e5a26 request: Refactor ZulipRequestNotes to RequestNotes.
This utilizes the generic `BaseNotes` we added for multipurpose
patching. With this migration as an example, we can further support
more types of notes to replace the monkey-patching approach we have used
throughout the codebase for type safety.
2021-09-03 08:48:45 -07:00
PIG208 dba7b1e3a7 request: Extract generic BaseNotes from RequestNotes.
The motive of adding `BaseNotes` was to support monokey patching
temporary attributes to objects (such as `.trigger` on `Message`) when
working on the django-stubs migration in #18777.
2021-09-03 08:47:51 -07:00
Tim Abbott 356e6e5018 status: Extract format_user_status helper.
This will allow us to reuse this formatting logic for a single-user
version of this endpoint.
2021-09-02 16:50:25 -07:00
rht a13f3d4386 import_realm: Use generate_message_upload_path() to get upload file path.
This way, we no longer have to manually keep the upload path code in
sync with the upload path code in zerver/lib/upload.py.
This was originally suggested in
https://github.com/zulip/zulip/pull/19478#issuecomment-911479530.

This change fixes a bug when importing into a server using the local
file uploads backend, where the `import_realm.py` copy wasn't using
our standard 256-directory approach to avoid putting too many files in
a single directory.
2021-09-02 16:36:48 -07:00
rht 6ff659d199 upload: Extract generate_message_upload_path helper.
This helper will let us avoid copying this logic in the data import
code path.
2021-09-02 16:31:08 -07:00
Alex Vandiver 094f22e6b4 notifications: Add safeties to prevent incorrect notifications.
de04f0ad67 changed now notifications recipients were calculated, in
a manner that caused them to be sent when they should not have been.
ac70a2d2e1 was supposed to resolve this, but appears to have been
insufficient, as all three of these cases have been observed to still
happen.

Add safety checks immediately before notification, until the
underlying logic error can be sussed out.
2021-09-02 02:49:10 -07:00
Alex Vandiver 5b45f8a128 queue_processors: Include queue name in the timeout exception.
This information can be gleaned from the stacktrace, but making it
explicit in the stringification makes it much easier to differentiate
types of errors at a glance, particularly in Sentry.
2021-09-02 02:48:34 -07:00
Sahil Batra 307b55e93d models: Move emojiset_choices method to UserBaseSettings class.
We move the emojiset_choices method from UserProfile class to
UserBaseSettings class because emojiset_choices exists in
UserBaseSettings class and this would be used for realm-level
settings as well along with existing user-level settings.
2021-09-01 13:29:32 -07:00
Sahil Batra d093ea4442 tests: Rename system user group used in user group tests.
This commit changes the name of system group used in the
tests to be "Moderators", as we will be using system user
groups for role-based groups only.
2021-09-01 13:05:33 -07:00
Sahil Batra 83d61fedf2 api: Rename system user group used in api docs example.
We rename the user group in the example for 'GET /user_groups'
with is_system_group=True, to be 'Moderators' as is_system_group
will be set to True for role-based user groups only.
2021-09-01 13:05:33 -07:00
Alex Vandiver 66aa2a2505 fetch-contributor-data: Use builtin backoff.
This correctly handles connection timeouts, not just non-200 status
codes.
2021-09-01 05:34:13 -07:00
Alex Vandiver bf9780267d outgoing_http: Give an easy way to configure retries.
The default is kept as no retries.  Since retries with exponential
backoff are a good thing to make easy, the int form defaults to
setting a backoff_factor.

Unfortunately, urllib3 retry backoff does not implement jitter.
Switching this to use the `backoff` library[1] rather than urllib3's
native Retry is left as future extension.

[1] https://pypi.org/project/backoff/
2021-09-01 05:34:13 -07:00
Alex Vandiver 4d428490fd outgoing_http: Use OutgoingSession subclasses in more places.
This adds the X-Smokescreen-Role header to proxy connections, to track
usage from various codepaths, and enforces a timeout.  Timeouts were
kept consistent with their previous values, or set to 5s if they had
none previously.
2021-09-01 05:34:13 -07:00
Priyansh Garg 1e51c23494 markdown: Remove unnecessary checks for zulip_message.
This commits removes some unnecessary checks for `self.md.zulip_message`,
which were put there historically, as earlier we used to add the additional
properties like mentions_user_ids, alert_words, etc. to Message dict
only. These were later moved to MessageRenderingResult class in commit
75cea329b but the checks weren't removed.

This is important because while rendering the messages imported from
other chat tools (like Rocket.Chat), the Message dict is not passed to
the markdown, due to which the checks for `self.md.zerver_message` fails
and hence, things like user mentions, stream/topic mentions are not
rendered in the imported messages properly.
2021-08-31 16:53:42 -07:00
Alex Vandiver 5e5166d872 email_mirror: Downgrade a couple ZulipEmailForwardErrors.
These errors may occur in the course of normal operations, and
shouldn't generate a message in the error logs.
2021-08-31 16:37:54 -07:00
Alex Vandiver 422be80d94 email_mirror: Downgrade ZulipEmailForwardUserError to info.
This does not need to clog the logs as much, but may be useful from a
product decision standpoint.
2021-08-31 16:37:54 -07:00
Alex Vandiver be669c8d98 email_mirror: Move ZulipEmailForwardUserError into email_mirror_helpers. 2021-08-31 16:37:54 -07:00
Alex Vandiver 621d0e5123 send_email: Log IP address sources (and more realms) of send_email. 2021-08-31 16:36:42 -07:00
Alex Vandiver b011869777 migrations: Add a (profile,end) index on useractivityinterval.
The `user_activity_interval` worker calls:

```python3
    last = UserActivityInterval.objects.filter(user_profile=user_profile).order_by("-end")[0]
`````

Which results in a query like:
```sql
SELECT "zerver_useractivityinterval"."id", "zerver_useractivityinterval"."user_profile_id", "zerver_useractivityinterval"."start", "zerver_useractivityinterval"."end" FROM "zerver_useractivityinterval" WHERE "zerver_useractivityinterval"."user_profile_id" = 12345 ORDER BY "zerver_useractivityinterval"."end" DESC LIMIT 1
```

For users which have at least one matching row, this results in a
query plan like:
```
 Limit  (cost=0.56..711.38 rows=1 width=24) (actual time=0.078..0.078 rows=1 loops=1)
   ->  Index Scan Backward using zerver_useractivityinterval_7f021a14 on zerver_useractivityinterval  (cost=0.56..1031399.46 rows=1451 width=24) (actual time=0.077..0.078 rows=1 loops=1)
         Filter: (user_profile_id = 12345)
         Rows Removed by Filter: 98
 Planning Time: 0.059 ms
 Execution Time: 0.088 ms
```

But for users that have just been created, with no matching rows, this
is considerably more expensive:
```
 Limit  (cost=0.56..711.38 rows=1 width=24) (actual time=10798.146..10798.146 rows=0 loops=1)
   ->  Index Scan Backward using zerver_useractivityinterval_7f021a14 on zerver_useractivityinterval  (cost=0.56..1031399.46 rows=1451 width=24) (actual time=10798.145..10798.145 rows=0 loops=1)
         Filter: (user_profile_id = 12345)
         Rows Removed by Filter: (count of every single row in the table, redacted)
 Planning Time: 0.053 ms
 Execution Time: 10798.158 ms
```

Regular vacuuming can force the use of the index on `user_profile_id`
as long as there are few enough users, which is fast -- however, at
some point, the query planner decides that is insufficiently specific,
always chooses the effective-whole-table-scan.

Add an index on `(user_profile_id, end)`, which is expected to be
sufficiently specific that it is used even with large numbers of user
profiles.

Ref #19250.
2021-08-31 16:33:40 -07:00
Anders Kaseorg 1ce12191aa docs: Update links for other repository branch renames.
GitHub redirects these, but we should use the canonical URLs.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-08-31 15:28:01 -07:00
Anders Kaseorg 817146c28b python: Upgrade SQLAlchemy from 1.3.24 to 1.4.23.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-08-31 06:47:39 -07:00
Alex Vandiver 6c15df68e8 downloads: Serve desktop downloads from desktop-download.zulip.com.
This makes them work for sites which block github.com.
2021-08-28 23:08:45 -07:00
Alex Vandiver fe25517295 email_notifications: Handle empty rendered_messages.
The transforms called from `build_message_payload` use
`lxml.html.fromstring` to parse (and stringify, and re-parse) the HTML
generated by Markdown.  However, this function fails if it is passed
an empty document.  "empty" is broader than just the empty string; it
also includes any document made entirely out of control characters,
spaces, unpaired surrogates, U+FFFE, or U+FFFF, and so forth.  These
documents would fail to parse, and raise a ParserError.

Using `lxml.html.fragment_fromstring` handles these cases, but does by
wrapping the contents in a <div> every time it is called.  As such,
replacing each `fromstring` with `fragment_fromstring` would nest
another layer of `<div>`.

Instead of each of the helper functions re-parsing, modifying, and
stringifying the HTML, parse it once with `fragment_fromstring` and
pass around the parsed document to each helper, which modifies it
in-place.  This adds one outer `<div>`, requiring minor changes to
tests and the prepend-sender functions.

The modification to add the sender is left using BeautifulSoup, as
that sort of transform is much less readable, and more fiddly, in raw
lxml.

Partial fix for #19559.
2021-08-25 15:50:55 -07:00
Alex Vandiver 42e3c4e6ec email_notifications: Rename append_sender_to_message to prepend.
The sender goes on the beginning, which is prepending to the message.
2021-08-25 15:50:55 -07:00
Anders Kaseorg 3e78de4ce8 sync_ldap_user_data: Log all exceptions.
This is a roundabout way to appease a semgrep complaint about
‘error_msg = error_msg % (string_id,)’ while also improving the code.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-08-24 07:51:48 -07:00
Mateusz Mandera 7ef1a024db management: Rename clear_auth_rate_limit_history command. 2021-08-23 11:52:35 -07:00
Eeshan Garg 8697d98652 webhooks/clubhouse: Ignore label removals for story batch updates.
9ac55a8cf6 introduced support for
batch updates to stories. However, that commit didn't skip label
removals, as we already do in non-batch story payloads. This led
to an exception for batch story update payloads where labels were
removed but none were added.
2021-08-20 23:27:14 -07:00
Alex Vandiver 4d98b0552e missedmessage_emails: Ensure forward progress.
maybe_send_batched_emails handles batches of emails from different
users at once; as it processes each user's batch, it enqueues messages
onto the `email_senders` queue.  If `handle_missedmessage_emails`
raises an exception when processing a single user's email, no events
are marked as handled -- including those that were already handled and
enqueued onto `email_senders`.  This results in an increasing number
of users being sent repeated emails about the same missed messages.

Catch and log any exceptions when handling an individual user's
events.  This guarantees forward progress, and that notifications are
sent at-most-once, not at-least-once.
2021-08-20 07:21:39 -07:00
Eeshan Garg c9861b3c74 marketing_emails: Ask for user's consent at sign-up. 2021-08-20 06:43:26 -07:00
Zeeshan 56f7282096 api_docs: Mention special value of realm_message_retention_days feild.
This commit indicates that the realm_message_retention_days field can have
a special value, similar to its stream counterpart, and also explains how
the special value changed over different server versions.

With an extension from tabbott to double-enter the changelog entry.

Related discussion: https://chat.zulip.org/#narrow/stream/378-api-design/topic/realm_message_retention_days
2021-08-20 06:35:35 -07:00
Vishnu KS d11d454206 auth: Redirect password reset page to /accounts/go when required. 2021-08-20 06:18:49 -07:00
PIG208 8fd89f87e0 typing: Avoid redefining variable with different types. 2021-08-20 06:02:56 -07:00
PIG208 fa09404dd7 typing: Use assertions for responses when appropriate.
This is part of #18777.
2021-08-20 06:02:56 -07:00
PIG208 f9644c8cf3 typing: Fix function signatures with django-stubs. 2021-08-20 06:02:55 -07:00
PIG208 04f5f25478 typing: Replace `File` with `IO[bytes]`. 2021-08-20 06:02:28 -07:00
PIG208 7386918539 typing: Use accurate type hints for dictionaries.
This fixes the mypy errors related to dictionaries with django-stubs.
2021-08-20 06:02:28 -07:00
PIG208 3b11c36ed9 typing: Fix function signatures.
This fixes mypy errors for function signatures discovered with
django-stubs.
2021-08-20 05:54:19 -07:00
PIG208 71427239d0 typing: Replace CursorObj by CursorWrapper. 2021-08-20 05:54:19 -07:00
PIG208 254f706465 typing: Fix argument type for models in function signatures. 2021-08-20 05:54:19 -07:00
PIG208 aa9d73c9f6 typing: Improve typing with assertions.
This fixes some mypy errors discovered with django-stubs.
2021-08-20 05:54:19 -07:00
PIG208 9d8e80a4d7 push_notifications: Refactor testcases to fix mypy errors.
This fixes errors found with django-stubs and it is a part of #18777.

It mostly renames variables and adds non-check assertions.
2021-08-20 05:54:19 -07:00
PIG208 bb36bd8b6b profile_request: Replace MockRequest with HostRequestMock.
Since previously we use a custom mock request that doesn't inherit
HttpRequest, there are some typing problems when we use it.
2021-08-20 05:54:19 -07:00