Commit Graph

6259 Commits

Author SHA1 Message Date
Steve Howell 29bd1e8bd3 tests: Avoid clutter within long list of getters.
The diff here is ugly, but to summarize:

    BEFORE IMPORT:
        define get_user_id
        define get_huddle_hashes

    AFTER IMPORT AND MAKING GETTERS:
        check realm id
        define assert_realm_values
        verify emoji codes
        check huddle hashes
2021-12-07 12:27:01 -08:00
Steve Howell 93761cd237 tests: Add getter decorator for import test. 2021-12-07 12:27:01 -08:00
Steve Howell 5892748c7b tests: Avoid lambdas in import test. 2021-12-07 12:27:01 -08:00
Steve Howell 54a6c82282 tests: Avoid equal flag for huddle hashes.
There's no need to complexify the codepath
for all the normal use cases.
2021-12-07 12:27:01 -08:00
Eeshan Garg 2cdaae681d actions: Rename do_change_plan_type -> do change_realm_plan_type.
We will soon be adding an equivalent function for RemoteZulipServer,
so it makes sense to rename this function to be more descriptive.
2021-12-06 16:18:53 -08:00
Tim Abbott 8aafce5619 test_signup: Fix test failures with week old test database.
The comment explains in more detail, but basically we'd skip
exercising a bit of code in the signup code path if there were no
messages in the last week, resulting in the query count not matching.
2021-12-06 14:08:37 -08:00
Steve Howell 4088be6017 import/export: Add UserStatus table.
(We support both realm and single-user exports.)
2021-12-06 13:27:25 -08:00
Steve Howell f83907d3bb export: Add MutedUser table.
Note that the import was already implemented, but
its test was flawed.
2021-12-06 13:27:25 -08:00
Steve Howell dffe776287 import tests: Have Cordelia react with a realm emoji.
This exercises some rather unusual database-related
manipulations that happen when creating reactions
from either the app or the import process.
2021-12-06 13:27:25 -08:00
Steve Howell 6ec5a5193a tests: Verify emoji codes. 2021-12-06 13:22:47 -08:00
Palash e1796b0c8f onboarding: Manage occurrence of 'help' command.
"help" command occurs in the command list in
initial pms or when bot doesn't understand the message. It doesn't
occur when the bot is respoding to the "help" command itself.
2021-12-06 11:32:00 -08:00
Sahil Batra b68ebf5a22 message: Check wildcard mention restrictions while editing message.
This commit adds code to check whether a user is allowed to use
wildcard mention in a large stream or not while editing a message
based on the realm settings.

Previously this was only checked while sending message, thus user
was easily able to use wildcard mention by first sending a normal
message and then using a wildcard mention by editing it.
2021-12-06 10:22:29 -08:00
Tim Abbott a915e1cc26 test_push_notifications: Sort pm_users as integers, not strings.
Fixes the last commit not working as intended.
2021-12-03 17:15:25 -08:00
Tim Abbott eb3ad83560 test_push_notifications: Sort pm_users list.
The production code sorts this list, so this test would fail
nondeterministically if the database returned these elements in
another order.
2021-12-03 16:56:01 -08:00
Sahil Batra a34dd11b5b test_events: Rename functions to use dark_theme and light_theme.
This commit renames the tests for changing logo in test_events
to use dark_theme and light_theme instead of night_mode and
day_mode.
2021-12-03 14:25:52 -08:00
Mateusz Mandera 39ce424fd5 confirmation: Delete rendundant confirmation_link_expired_error page.
This template provides nothing that link_expired.html does not and is
redundant.
2021-12-02 09:50:41 -08:00
Mateusz Mandera 4a030f616f auth: Improve Confirmation link error handling in social auth codepath.
render_confirmation_key_error provides better error responses than
simply rendering the "link expired" template.
2021-12-02 09:50:41 -08:00
Mateusz Mandera 7c5daac8b6 get_object_from_key: Accept multiple allowed Confirmation types.
This allows making check_prereg_key significantly cleaner.
2021-12-02 09:50:41 -08:00
Anders Kaseorg 5344e86c57 requirements: Upgrade Python requirements.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-12-02 08:16:20 -08:00
Palash a444560684 onboarding: Replace 'edit profile' with 'profile' in bot command list.
And change the tests accordingly.
2021-12-02 08:12:51 -08:00
Alya Abbott 8070b84785 help center: Remove outdated intro from index page. 2021-12-01 17:05:59 -08:00
Palash 6404b7bf61 onboarding: Make Welcome Bot more interactive.
1. The initial welcome message now contains less detail.
2. The bot now responds to these commands: "apps", "edit profile",
"dark mode", "light mode", "streams", "topics", "message formatting",
"keyboard shortcuts" and "help" - the bot still responds if there are
slight variations in these commands.
3. Tests have been made to check if bot responds to the advertised
commands (with variations) and gives a negative message if it doesn't
understand the message.

With substantial tweaks by tabbott.

Fixes #19900.
2021-12-01 16:30:48 -08:00
Mateusz Mandera a014ef75a3 CVE-2021-43791: Validate confirmation keys in /accounts/register/ codepath.
A confirmation link takes a user to the check_prereg_key_and_redirect
endpoint, before getting redirected to POST to /accounts/register/. The
problem was that validation was happening in the check_prereg_key_and_redirect
part and not in /accounts/register/ - meaning that one could submit an
expired confirmation key and be able to register.

We fix this by moving validation into /accouts/register/.
2021-12-01 23:14:04 +00:00
Mateusz Mandera a1cd660147 confirmation: Use error status codes for confirmation link error pages. 2021-12-01 23:14:04 +00:00
Eeshan Garg f976bf723e links: Rename developer-community -> development-community. 2021-12-01 13:41:46 -08:00
Lauryn Menard 7713b371a5 api: Migrate `/update-subscription-settings` response value.
Migrates the `/update-subscription-settings` api endpoint to the
`ignored_parameters_unsupported` model, which is also currently used
by `/update-settings` and `update-realm-user-settings-defaults`.

This change is a step towards preparing for an eventual migration to
have all endpoints return an `ignored_parameters_unsupported` block.

Previously the `/update-subscription-settings` endpoint returned a
copy of the data object sent in the request.

Fixes #15307.
2021-11-26 22:25:53 -08:00
Sahil Batra 88e21d0387 misc: Replace "night mode" with "dark theme" in comments. 2021-11-26 22:03:29 -08:00
Mateusz Mandera 994ab8115e scim: Order Users by id when queried using filter syntax.
django-scim2 doesn't order the rows when fetching them in reponse to a
query using the filter syntax. We ensure that ORDER BY id is always
appended to the SQL queries.
2021-11-26 16:06:16 -08:00
strifel 11211bd9c4 integrations: Switch BigBlueButton over to SHA256.
This commit switches the BigBlueButton integration
to use SHA256 instead of SHA1 as BigBlueButton supports
it and scalelite does now, too.

Fixes #19966.
2021-11-25 07:58:27 -08:00
Steve Howell 939032b45f test flake: Sort users for scim test. 2021-11-25 07:57:20 -08:00
Alya Abbott 74b4640727 portico: Add Rust case study. 2021-11-24 15:17:33 -08:00
Sahil Batra 30c190a120 test: Check json_fetch_raw_message raises error.
This commit adds a check to verify that json_fetch_raw_message
raises error when enable_spectator_access is False.
2021-11-24 10:37:51 -08:00
Sahil Batra 56a8443f05 models: Check enable_spectator_access value in web_public_streams_enabled.
This commit changes web_public_streams_enabled to return False if
realm.enable_spectator_access is False. This is added so that
creating web-public streams is not allowed if enable_spectator_access
is False.
2021-11-23 10:48:20 -08:00
Sahil Batra ad99b4fac9 streams: Allow changing stream to be web-public based on creation setting.
We allow a user to make an existing stream web-public only if user is
allowed to create web-public streams.
2021-11-23 10:48:20 -08:00
Sahil Batra 2c1a87166c register: Pass settings.WEB_PUBLIC_STREAMS_ENABLED to clients.
This commit adds 'server_web_public_streams_enabled' field to
the register response to pass settings.WEB_PUBLIC_STREAMS_ENABLED
to clients.
2021-11-23 10:48:20 -08:00
Shlok Patel b3c58f454f api: Prevent special characters in topics.
Special characters, including `\r`, `\n`, and more esoteric codepoints
like non-characters, can negatively affect rendering and UI behaviour.

Check for, and prevent making new messages with, characters in the
Unicode categories of `Cc` (control characters), `Cs`, (surrogates),
and `Cn` (unassigned, non-characters).

Fixes #20128.
2021-11-22 22:09:06 -08:00
Sahil Batra eef9dd963b zcommand: Replace "mode" with "theme" in the returned message.
This commit replaces "dark mode" and "light mode" with "dark theme"
and "light theme" in the message returned and shown in a little
popup in the UI, when color scheme settings are changed through
slash commands.
2021-11-22 14:59:16 -08:00
Mateusz Mandera 8af7ffd9da rate_limit: Fix logging string when rate limiting email gateway.
realm.name is not the right "name" to log, we should use realm.subdomain
like everywhere else.
2021-11-22 10:28:56 -08:00
Aman Agrawal 334f046842 events: Don't send data related to custom profile field to spectator.
Since spectators can't access personal profile settings and
can't view profile for other users. Hence, we don't send realm
custom profile field data and user's profile data to spectators.

Fixes #20301.
2021-11-20 21:01:38 -08:00
Sahil Batra b72035abfe misc: Replace "Light mode" with "Light theme". 2021-11-19 11:49:59 -08:00
Sahil Batra 5024ba98cd misc: Rename "dark mode" to "dark theme".
Fixes part of #20228.
2021-11-18 16:57:11 -08:00
Aman Agrawal 00d3b88257 realm: Enable spectator access by default in development.
Enable spectator access for test `zulip` realm in developement
setup.

Add option in `do_create_realm` to configure
`enable_spectator_access` field of `Realm`.
2021-11-18 12:55:12 -08:00
Aman Agrawal 2a922409aa realm: Add option to enable/disable spectator login.
We restrict access of messages from web public streams if
anonymous login is disabled via `enable_spectator_access`.

Display of `Anonymous login` button is now controlled by
the value of `enable_spectator_access`.

Admins can toggle `enable_spectator_access` via org settings in UI.
2021-11-18 12:55:12 -08:00
Aman Agrawal 28488d9939 test_decorators: Call mock_home via patched zulip_login_required.
This avoids a weird issue that when mocking both
zulip_login_required and home_real, home_realm doesn't stay
patched.
2021-11-18 12:55:12 -08:00
Mateusz Mandera 0abbb87155 auth: Include user_id in the params returned at the end of mobile flow.
The user id is a very useful piece of information that the mobile
client should have access to - instead of only getting the email. This
makes it much simpler to impleent clients that might be robust to
changes in email address.
2021-11-16 12:12:59 -08:00
Alex Vandiver 7c3507feef queue: Allow passing down a prefetch count to pika. 2021-11-16 11:48:50 -08:00
Alex Vandiver 8d239f4984 tests: Move lambda in simulated_queue_client to help typing. 2021-11-16 11:48:50 -08:00
Alex Vandiver d1822b5630 tests: Move simulated_queue_client to the only test it is used in. 2021-11-16 11:48:50 -08:00
Alex Vandiver 49ad188449 rate_limit: Add a flag to lump all TOR exit node IPs together.
TOR users are legitimate users of the system; however, that system can
also be used for abuse -- specifically, by evading IP-based
rate-limiting.

For the purposes of IP-based rate-limiting, add a
RATE_LIMIT_TOR_TOGETHER flag, defaulting to false, which lumps all
requests from TOR exit nodes into the same bucket.  This may allow a
TOR user to deny other TOR users access to the find-my-account and
new-realm endpoints, but this is a low cost for cutting off a
significant potential abuse vector.

If enabled, the list of TOR exit nodes is fetched from their public
endpoint once per hour, via a cron job, and cached on disk.  Django
processes load this data from disk, and cache it in memcached.
Requests are spared from the burden of checking disk on failure via a
circuitbreaker, which trips of there are two failures in a row, and
only begins trying again after 10 minutes.
2021-11-16 11:42:00 -08:00
Alex Vandiver a2f89f21e3 email_change: Show deactivated page if realm is now deactivated.
We also catch if the user is now deactivated.
2021-11-14 18:55:40 -08:00
Alex Vandiver 6c14978cd1 zilencer: Truncate "remove" notifications from remote servers.
This is 4d055a6695, but for notifications which are received from
remote hosts.
2021-11-10 13:39:35 -08:00
akshatdalton f2c7e8e80f markdown: Ensure all markdown fixtures have unique names.
This commit ensures that all markdown fixtures have unique
test names by rewriting the names of some of them and adding
a test in `test_markdown.py`.

Earlier this was over-writing the value for same keys in
`load_markdown_tests` in `test_markdown.py`.
2021-11-10 12:36:47 -08:00
Mateusz Mandera 17485e2f4d saml: Refactor get_issuing_idp to rely on class polymorphism. 2021-11-10 12:08:56 -08:00
Alya Abbott e106caa68f portico: Add Lean case study. 2021-11-07 09:53:28 -08:00
Priyansh Garg 42f231c85c data_import: Ignore Rocket.Chat livechat streams/messages.
This resolves the issues reported in #20108, major chunk of which were
due to the incomplete support for importing the livechat streams/messages
in the tool. So, it's best not to import any livechat streams/messages for
now until a complete support for importing the same is developed.
2021-11-07 09:50:55 -08:00
Anders Kaseorg 8df6eccd42 dependencies: Upgrade JavaScript dependencies.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-11-05 17:34:13 -07:00
Alex Vandiver c8badbd858 reset_password: Show user-facing page on rate-limit. 2021-11-04 20:34:39 -07:00
Alex Vandiver d3ecbf96a8 rate_limit: Rate-limit password reset attempts by IP, as well. 2021-11-04 20:34:39 -07:00
Tim Abbott 1cad29fc3a settings: Add rate limiting for email address changes.
Co-authored-by: Alex Vandiver <alexmv@zulip.com>
2021-11-04 20:34:39 -07:00
Alex Vandiver f0532aecc8 tests: Stop adding an api_by_user rate limit always.
The decorator form is clearer by being more explicit; additionally,
the api_by_user rate-limit only currently used in one place, and makes
it difficult to test per-user rate-limits that are more specific.
2021-11-04 20:34:39 -07:00
Tim Abbott 01e2a495fc rate_limit: Fix missing IP rate limiting on confirmation.
Co-authored-by: Alex Vandiver <alexmv@zulip.com>
2021-11-04 20:34:39 -07:00
Alex Vandiver 48ba2e7cc6 tests: Hitting a rate-limit on find accounts should not send emails. 2021-11-04 20:34:39 -07:00
Alex Vandiver 0cfb156545 rate_limit: Merge two IP rate limits domains that send emails.
Both `create_realm_by_ip` and `find_account_by_ip` send emails to
arbitrary email addresses, and as such can be used to spam users.
Lump their IP rate limits into the same bucket; most legitimate users
will likely not be using both of these endpoints at similar times.

The rate is set at 5 in 30 minutes, the more quickly-restrictive of
the two previous rates.
2021-11-04 20:34:39 -07:00
Alex Vandiver 5f0897e6f7 tests: Add a test that IP rate-limiting is by IP.
The existing test did no verify that the rate limit only applied to
127.0.0.1, and that other IPs were unaffected.  For safety, add an
explicit test of this.
2021-11-04 20:34:39 -07:00
Alex Vandiver 9495dad850 tests: Simplify by always clearing IP limits in rate_limit_rule.
The only use case of rate_limit_rule which does not clear the
RateLimitedIPAddr history is test_hit_ratelimits_as_remote_server,
which is not made any worse by clearing out the IP history for a
non-existent `api_by_remote_server` domain.
2021-11-04 20:34:39 -07:00
Alex Vandiver 260ccc9620 tests: Factor out common non-api rate-limit assert function.
The same `assert_func` is used in multiple places, for non-API
requests.  Factor it out and make it a flag to do_test_hit_ratelimits.
2021-11-04 20:34:39 -07:00
Alex Vandiver 272e78de8b tests: Split out test_find_account_rate_limiting_multiple.
This lets rate_limit_rule be used consistently as a decorator, and
improves how parallel the code is here, with other tests.
2021-11-04 20:34:39 -07:00
Alex Vandiver 328a28d772 tests: Correct the assert in the post-rate-limit test. 2021-11-04 20:34:39 -07:00
Aman Agrawal 0879e5e0e2 home: Delete prefers_web_public_view key after user is logged in.
Since `prefers_web_public_view` key in session is only
relevant to users without an account, this key should no longer
be present in the user's session object.

Fixes #19907
2021-11-03 16:52:51 -07:00
Priyansh Garg 17409a78be data_import: Fix a few KeyError bugs in Rocket.Chat import tool.
This commit fixes a few bugs in Rocket.Chat import tool as reported on CZO.

Link: https://chat.zulip.org/#narrow/stream/9-issues/topic/Rocketchat.20Import
2021-11-03 16:50:56 -07:00
Shlok Patel 893c9bc896 export: Remove `--delete-after-upload` flag in realm export.
For export realm following changes have been made:
- `./manage.py export --upload` would delete `.tar.gz` and unpacked dir
- `./manage.py export` would only delete `unpacked dir`

Besides, we have removed `--delete-after-upload` as we have set it as
the default.

Fixes #20081
2021-11-03 11:14:02 -07:00
Aman Agrawal 3e689ebae9 users: Allow spectators to view user avatars.
If realm is web_public, spectators can now view avatar of other
users.

There is a special exception we had to introduce in rest model to
allow `/avatar` type of urls for `anonymous` access, because they
don't have the /api/v1 prefix.

Fixes #19838.
2021-11-02 11:26:19 -07:00
Priyansh Garg 0db9b7287b data_import: Import Rocket.Chat messages from direct discussions.
This commit adds functionality to import messages from the
Discussions having direct channels as their parent. As we don't
have topics in the PMs, the messages are imported in interleaved
form in the imported direct channels/PMs.

This was completely unsupported earlier and would have resulted in
an error.
2021-11-01 17:09:11 -07:00
Sahil Batra e6106cb334 invites: Update error message when max limit for the day is reached.
This commit updates the error message returned when the maximum
invite limit for the day. We update the error returned by API to
only mention that the limit is reached and add the suggestion
to use multi-use link or contact support in the message shown
in webapp.
2021-11-01 16:36:26 -07:00
Sahil Batra 9a03a78e3d test_users: Use delivery_email instead of email to generate gravatar url.
We always use delivery_email to generate gravatar_url, but in
test_admin_api_hide_emails we were passing email to get_gravatar_url
and matched with the avatar_url field of the fetched user object.

The tests were passing because the email_address_is_realm_public
was using old realm object and thus email field was incorrectly
set to delivery_email even when email_address_visibility was set
to EMAIL_ADDRESS_VISIBILITY_ADMINS.

This commit fixes the test to pass delivery_email to get_gravatar_url.
2021-10-29 16:19:55 -07:00
Suyash Vardhan Mathur d2c5067839 openapi: Document remove-attachment endpoint. 2021-10-29 15:01:50 -07:00
Tim Abbott 5d5cf371d8 lint: Fix typo in test email address. 2021-10-27 16:49:30 -07:00
Mateusz Mandera 4105ccdb17 saml: Implement IdP-initated logout for Keycloak.
Fixes #13948.
2021-10-27 13:13:55 -07:00
Alex Vandiver 6a40c17ccf markdown: CSS-escape preview links.
This adds `soupsieve` as an explicit dependency, but intentionally
does not adjust the provision version, as it was already an indirect
dependency.
2021-10-26 18:17:23 -07:00
Alex Vandiver 52f74bbd9b markdown: Run URL preview links through camo.
Not proxying these requests through camo is a security concern.
Furthermore, on the desktop client, any embed image which is hosted on
a server with an expired or otherwise invalid certificate will trigger
a blocking modal window with no clear source and a confusing error
message; see zulip/zulip-desktop#1119.

Rewrite all `message_embed_image` URLs through camo, if it is enabled.
2021-10-26 18:17:23 -07:00
Alex Vandiver 9381a3bd45 linkifiers: Support URL percent-encoded bytes.
Supporting URL percent-encoded bytes is possible using `%%20`, but this
is not necessarily very understandable to end-users, even those that
understand percent encoding.

Allow `%20` in linkifier URL format strings, and transform them into
`%%20` in the pattern just before they are applied in markdown
translation.  Care must be taken here, such that already-escaped `%`s
are not escaped an extra time.

We do this before rendering, and not before storage, as
a simplification; the JS-side linkifier at present only understands
`%(foo)s` and thus needs no changes, and to avoid an un-escaping pass
before showing in the admin UI.
2021-10-22 13:00:20 -07:00
Alex Vandiver d6768814a1 linkifiers: Explicitly only check format strings after verifying as a URL.
This makes the errors less confusing -- they might otherwise be jammed
together in the frontend.
2021-10-22 13:00:20 -07:00
Alex Vandiver 8dd9b4e812 linkifiers: Loosen regex that validates URLs.
User-supplied custom realm filter has had some sort of regex-based
validation of the format URL since their introduction in
d7e1e4a2c0 -- and this has always been
in addition to the URLValidator.  The URLValidator is the one which
does the security-relevant work of validating that the schema is
reasonable, and that the overall shape of the URL is well-formed.  The
regex has served primarily to arbitrary limit the characters that can
appear in the URL, in the mistaken name of safety.

Adjust the regex, such that its only purpose is to verify that the
usages of `%` characters in the URL are reasonable, and leave the URL
validation to the URLValidator, which can do a far better job.  This
includes broadening the support to include `%%` as an escape
character; this is likely such a niche case as to be unnecessary, but
costs little.

Fixes #16013.
2021-10-22 13:00:20 -07:00
Jonny Tran ddf4053a73 composebox_typeahead: Remove redundant options from the `/` typeahead.
Removes the `/day` and `/night` options from the typeahead menu while
still allowing the commands to be used. Typing `/day` and `/night`
will now suggest `/light` and `/dark`, respectively. Also changes the
`Dark mode` and `Light mode` popups that appear after using the
corresponding command.

Fixes #18318.
2021-10-21 10:50:27 -07:00
Alex Vandiver 111ee64e36 push_notifications: Pass down the remote server and user-id for logs.
This makes logging more consistent between FCM and APNs codepaths, and
makes clear which user-ids are for local users, and which are opaque
integers namespaced from some remote zulip server.
2021-10-19 22:04:24 -07:00
Alex Vandiver 5bcd3c01cb push_notifications: Add log line with user-id, UUID, and devices.
Being able to determine how many distinct users are getting push
notifications per remote host is useful, as is the distribution of
device counts.  This parallels the log line in
handle_push_notification for push notifications from local realms,
handled via the event queue.
2021-10-19 22:04:24 -07:00
Sahil Batra 2285338b11 tests: Use more selective query in user group tests.
We should use more selective query for UserGroupMembership
objects in tests for checking adding and removing members.
This is done by checking the membership counts for the
particular user group only.

This will help in keeping the tests more understandable
after we add members to the role-based system groups,
since that would create a lot of membership objects.
2021-10-19 17:02:10 -07:00
Sahil Batra 599b3cc58e test_user_groups: Make UserGroup queries more selective.
We make the UserGroup queries in user group creation and
deletion tests more selective by fitering the user groups
which belong to the realm and not the one included in
lear realm, etc.

This will help us to keep the tests more understandable
when the counts of UserGroup increases due to addition of
system groups. There is no need to consider system groups
of other realms in these tests.
2021-10-19 17:02:10 -07:00
Sahil Batra 8b2213a43e test_user_groups: Fix comment in test_user_group_create. 2021-10-19 17:02:10 -07:00
rht bb8504d925 lint: Fix typos found by codespell. 2021-10-19 16:51:13 -07:00
Eeshan Garg b325a4f1be realm: Rename plan type constants to be more descriptive.
It is confusing to have the plan type constants not be namespaced
by the thing they represent. We already have a namespacing
convention in place for constants, so we should use it for
Realm.plan_type as well.
2021-10-19 12:20:39 -07:00
Alex Vandiver cbbd4b128d push_notifications: Provide a hint when the server is not registered. 2021-10-19 12:17:30 -07:00
Gaurav Pandey 1c1a1e2cad api: Remove encoding of string in update_user endpoint.
* Remove unnecessary json_validator for full_name parameter.
* Update frontend to pass the right parameter.
* Update documentation and note the change.

Fixes #18409.
2021-10-18 19:03:43 -07:00
Alex Vandiver b02754adec html_diff: Handle empty differences between empty strings.
`rendered_content` in historical messages may be empty; examining the
history of them may thus require diff'ing two empty strings, which
itself produces an empty string.

Use `lxml.html.fragment_fromstring` to be able to successfully parse
these, rather than 500.

Part of #19559.
2021-10-18 18:27:40 -07:00
Mateusz Mandera f5a65846a8 scim: Override django-scim2 logic of exception handling in views.
As detailed in the comments, the default behavior is undesirable for us
because we can't really predict all possibilities of exceptions that may
be raised - and thus putting str(e) in the http response is potentially
insecure as it may leak some unexpected sensitive information that was
in the exception.

As a hypothetical example - KeyError resulting from some buggy
some_dict[secret_string] call would leak information. Though of course
we aim to never write code like that.
2021-10-17 21:33:03 -07:00
Mateusz Mandera 9b5a27813a scim: Test the /scim/v2/Users/.search endpoint.
This endpoint has some basic support in our and django-scim2
implementation and we should be testing it.

https://datatracker.ietf.org/doc/html/rfc7644#section-3.4.3
for details on what this endpoint is.
2021-10-17 21:33:03 -07:00
Vishnu KS aff52722a7 billing: Add PLUS as a plan_type to Realm. 2021-10-15 17:27:50 -07:00
Sahil Batra 6be20a8c6d users: Allow admins to get subscription status in private stream.
We pass allow_realm_admin as True to access_stream_by_id for
`GET users/{user_id}/subscriptions{stream_id}` endpoint
because we want to allow non-subscribed admins to get
subscription status in private streams.

Fixes #19077.
2021-10-15 10:58:56 -07:00
Anders Kaseorg 3bd3173b1f avatar: Remove ?x=x kludge.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-10-14 12:47:43 -07:00
Anders Kaseorg cf7e8e3947 backends: Fix URL encoding in redirect_deactivated_user_to_login.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-10-14 12:47:43 -07:00
Mateusz Mandera 73a6f2a1a7 auth: Add support for using SCIM for account management. 2021-10-14 12:29:10 -07:00
Anders Kaseorg 1e5157b66c user_groups: Add a recursive group membership model.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-10-13 10:08:06 -07:00
Sahil Batra e47662aa19 user_groups: Rename get_memberships_of_users.
This is a prep commit for new permissions model in
which a user group would be able to have a subgroup.

This commit renames get_memberships_of_users to
get_direct_memberships_of_users to specify that
the function is used only to fetch the direct
memberships and not memberships of subgroups of
the direct group.

Extracted this commit from #19866.

Co-authored-by: Anders Kaseorg <anders@zulip.com>
2021-10-12 13:57:47 -07:00
Sahil Batra 07e2ecf221 user_groups: Rename get_user_groups to get_direct_user_groups.
This is a prep commit for new permissions model in which a user
group would be able to have a subgroup.

This commit renames get_user_groups to get_direct_user_groups
to specify that the function is used only to fetch the direct
groups that user is part of and not subgroups of the direct
group.

Extracted this commit from #19866.

Co-authored-by: Anders Kaseorg <anders@zulip.com>
2021-10-12 13:57:32 -07:00
Dinesh 9cea88c1a5 typing_notifications: Make privacy settings functional.
Changes the view code to skip sending typing events
when these settings are disabled. Returns a json success
response with a msg.
2021-10-07 17:39:22 -07:00
Dinesh 0b68f264c6 apply_event: Do not include new user settings in the top level.
From 430c5cb, in `fetch_initial_state_data`,
we only include legacy settings in the top level of
`state` and the newer ones are stored in `state['user_settings']`.
That should've had a corresponding change in apply_event().

Also, fixed a test related to this logic.
2021-10-07 16:39:51 -07:00
Dinesh 47da353366 Fix change_user_settings_test to consider sending only one event.
From 430c5cb, we do not send events for settings added after
introduction of `user_settings` event.

This test wasn't considering that and started failing on
adding new user_settings.
2021-10-07 16:39:51 -07:00
Eeshan Garg 29b354346b update_realm: Allow demo orgs to be converted to regular orgs.
This commit adds support to the `PATCH /realm` endpoint for
converting a demo organization to a regular organization.

This is a part of #19523.
2021-10-07 14:43:03 -07:00
Sahil Batra e575f83351 test_subs: Remove _test_user_settings_for_adding_streams.
This commit removes _test_user_settings_for_adding_streams
and its callers for testing public and private streams
because it uses excessive mocking and we also test the same
thing in _test_user_settings_for_creating_streams without
mocking, so this test doesn't add anything.
2021-10-07 14:22:10 -07:00
Eeshan Garg f25230c7d4 analytics/support: Add ability to edit realm org type. 2021-10-07 14:08:11 -07:00
rht 58b19761b8 slack import: Fix requests.get usage of get_slack_api_data.
We also rewrite the tests using the `responses` module to avoid the
problematic mocking that made this bug possible.

Fixes #19833.
2021-10-07 11:46:23 -07:00
Aman Agrawal 3b29c61409 auth: Add login buttons for web public access.
For users who are not logged in and for those who don't have
'prefers_web_public_view' set in session, we redirect them
to the default login page where they can choose to login
as spectator or authenticated user.
2021-10-05 16:30:40 -07:00
Sahil Batra ab0d852fd5 register: Add can_create_web_public_streams to register response. 2021-10-05 10:10:32 -07:00
Sahil Batra 2c2c19c4d4 streams: Restrict creating web public streams based on new setting.
This commit restricts creating web public streams based on the
recently added create_web_public_stream_policy setting.
2021-10-05 09:56:00 -07:00
Sahil Batra 3916181770 models: Add can_create_web_public_streams helper.
This commit adds can_create_web_public_streams helper
in models.py which will be used to validate whether
user is allowed to create a web-public stream or not.

This commit also adds the checks for Realm.POLICY_OWNERS_ONLY
in check_has_permission_policies.
2021-10-05 09:48:50 -07:00
Sahil Batra 5f950e3efd models: Add create_web_public_stream_policy setting.
This commit adds create_web_public_stream_policy
field to Realm table which controls the roles that
can create web-public streams and by default its
value is set to POLICY_OWNERS_ONLY.
2021-10-05 09:28:28 -07:00
Sahil Batra be0387b189 test_subs: Enforce invite_only argument to be named.
This commit enforces invite_only argument to be named
in _test_user_settings_for_creating_streams. This will
help in improving readability especially when we will
add is_web_public argument in further commits.
2021-10-05 09:12:56 -07:00
Sahil Batra dbcdd08324 test_events: Test the email event when changing delivery_email.
We send three events when changing delivery email of a user - one
for updating the delivery_email field of user, one for avatar url
change, and one for changing email field if email_address_visibility
is set to 'EMAIL_ADDRESS_VISIBILITY_EVERYONE'.

There is already a test for delivery_email and avatar_url event with
the visibility setting set to 'EMAIL_ADDRESS_VISIBILITY_ADMINS_ONLY',
but no test for verifying the email update event sent when email
address is public, so this commit adds a test for checking the schema
of event for updating email field.
2021-10-05 09:12:20 -07:00
Sahil Batra eedd832ca9 events: Fix updating user email on changing email_address_visibility.
When email_address_visibility is changed and either the old value
or the updated value is EMAIL_ADDRESS_VISIBILITY_EVERYONE then
email field of all users is updated and we also send the corresponding
event to clients. But apply_event code did not update the data on
receiving the event, so this commit fixes the code to correctly
handle the event in apply_event.

(We also use this event when just changing a user's email address).

This commit also adds the tests and openapi schema for the event.
2021-10-04 17:24:31 -07:00
Sahil Batra 402968ccf3 test_events: Use the list defined in models.py for realm update tests.
We use the lists defined in models.py like Realm.COMMON_POLICY_TYPES,
Realm.COMMON_MESSAGE_POLICY_TYPES, etc. in do_set_realm_property_test
instead of using defining list there (eg - [4, 3, 2, 1]). We do the
same thing in do_set_realm_property_test in test_realm.py.

We skip email_address_visibility values in this commit because it
requires some change in openapi schema as well.
2021-10-04 17:04:54 -07:00
Eeshan Garg 8123e62643 topics: Add test for translation issue with resolve notifications.
This commit adds a test that makes sure that that the fix in
PR #19828 is working as expected.
2021-10-04 16:20:16 -07:00
Eeshan Garg c4aeb159c4 topics: Fix translation issue with moved topic notifications.
Since the calls to the translation function `_()` are made outside
of the `send_message_moved_breadcrumbs` function, these strings are
translated outside of the `with override_language` block, leading to
translated strings even when we don't intend them to be translated.

We now use gettext_lazy with appropriate testing to avoid this.
2021-10-04 16:20:16 -07:00
Alex Vandiver db934be064 CVE-2021-41115: Use re2 for user-supplied linkifier patterns.
Zulip attempts to validate that the regular expressions that admins
enter for linkifiers are well-formatted, and only contain a specific
subset of regex grammar.  The process of checking these
properties (via a regex!) can cause denial-of-service via
backtracking.

Furthermore, this validation itself does not prevent the creation of
linkifiers which themselves cause denial-of-service when they are
executed.  As the validator accepts literally anything inside of a
`(?P<word>...)` block, any quadratic backtracking expression can be
hidden therein.

Switch user-provided linkifier patterns to be matched in the Markdown
processor by the `re2` library, which is guaranteed constant-time.
This somewhat limits the possible features of the regular
expression (notably, look-head and -behind, and back-references);
however, these features had never been advertised as working in the
context of linkifiers.

A migration removes any existing linkifiers which would not function
under re2, after printing them for posterity during the upgrade; they
are unlikely to be common, and are impossible to fix automatically.

The denial-of-service in the linkifier validator was discovered by
@erik-krogh and @yoff, as GHSL-2021-118.
2021-10-04 21:26:24 +00:00
Alex Vandiver f1c61fbea1 openapi: Use more carefully-constructed regexes for markdown extension.
This removes a false-positive ReDoS, since the input is always
checked-in code.  It also incidentally refactors to make the regexes
be more explicit about the values they expect, and removes unnecessary
capturing groups.

It removes an optional parenthesized status code for fixtures,
unnecessary since 981e4f8946, as well as
optional key-value language options, unnecessary since
a2be9a0e2d.

Thank you to @erik-krogh and @yoff for bringing this to our attention.
2021-10-03 18:49:38 -07:00
Pradyumna Sinha 18b36e5b8c markdown/tabbed_sections: Raise exception for missing tab name.
This fixes the issue  where 'None' would appear in the rendered
html in case of a missing tab display_name. Now,
'test-help-documentation' will fail in case of any tab display_name
being missing.

In case of a tab_section with no tabs, currently a single tab with
the name 'null_tab' gets added. Added the display name 'None' for
'null_tab', to keep in line with the existing behaviour.

Fixes #19822
2021-10-01 16:25:17 -07:00
Pradyumna Sinha 78692e9c14 markdown/tabbed_sections: Don't set tab label as 'None' in case of no tabs. 2021-10-01 16:22:11 -07:00
Shlok Patel 1e18d36328 realm: Change default permissions for education organizations.
This makes our onboarding guide for education organizations much
simpler, since new organizations will start with these settings
correctly configured.

Fixes #19682
2021-10-01 15:33:33 -07:00
Ganesh Pawar fa928d5cd1 streams: Split setting for stream creation policy.
Users wanted a feature where they could specify
which users can create public streams and which users can
create private streams.

This splits stream creation code into two parts,
public and private stream creation.

Fixes #17009.
2021-10-01 10:26:42 -07:00
sahil839 9dd69c17ee settings: Add moderators and members option in delete_own_message_policy.
This commit adds moderators, full members and members options to
delete_own_message_policy in backend.
2021-09-30 14:59:31 -07:00
sahil839 909a3cde76 realm: Replace allow_message_deleting with delete_own_message_policy.
This commit replaces 'allow_message_deleting' boolean setting
with an integer setting 'delete_own_message_policy'. We have a
separate dropdown now for deciding which user-roles can delete
messages sent by themselves and the time-limit setting droddown
is different.

This new setting has two options - everyone and admins only. Other
options including moderators will be added further.

We also remove the "Never" option from the original time-limit
dropdown, as admins are always allowed to delete message. This
never option resembled the case of only admins being allowed to
delete but this state is now resembled by setting the dropdown
to "admins only" and we also disable the time-limit dropdown in
this case as admins are allowed to delete irrespective of limit.

Note, this setting is only for deleting messages sent by the
deleting user themselves, and only admins are allowed to delete
messages sent by others as before.
2021-09-30 14:59:31 -07:00
sahil839 b13bfa09c5 message: Make zero invalid value for message_content_delete_limit_seconds.
We make zero invalid value for message_content_delete_limit_seconds and
for handling the case of "Allow to delete message any time", the API-level
value of message_content_delete_limit_seconds is "anytime" and "None"
as the DB-level value. We also use these values for message retention
setting, so it helps maintain consistency.
2021-09-30 14:45:39 -07:00
Mateusz Mandera 0af7c84c99 push_notifs: Log the number of devices notification was sent to. 2021-09-29 15:50:06 -07:00
Sahil Batra d1732fb9da settings: Remove realm-level default of enable_login_emails setting.
This commit does not remove the 'enable_login_emails' field from
RealmUserDefault table but it is just not used and cannot be
changed from UI or API similar to 'enable_marketing_emails' setting.
2021-09-28 12:41:09 -07:00
Aman Agrawal ef84224eed message_edit: Allow spectators to access raw message content.
We allow spectators to fetch the raw / original content of a
message which is used by the spectator to "View source" of
the message.
2021-09-28 10:07:36 -07:00
Mateusz Mandera 8b906b5d2f request_notes: Set the realm appropriately for the root subdomain.
Requests to the root subdomain weren't getting request_notes.realm set
even if a realm exists on the root subdomain - which is actually a
common scenario, because simply having one organization, on the root
subdomain, is the simplest and common way for self-hosted deployments.
2021-09-28 10:02:52 -07:00
Anders Kaseorg 0f7d0a23c9 Revert "validator: Add generic check_or."
This reverts commit cd93d0967f.

This check_or is redundant with check_union; it gives a misleading
error message for the non-matching case; and it has no type safety.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-09-28 09:28:56 -07:00
seiwailai cd93d0967f validator: Add generic check_or.
Added generic check_or function and tests.
Fixes part of #17914.

Co-authored-by: Gaurav Pandey <gauravguitarrocks@gmail.com>
2021-09-27 17:30:26 -07:00
Tim Abbott 417c32629d outgoing webhooks: Fix inconsistencies with Slack's API.
Apparently, our slack compatible outgoing webhook format didn't
exactly match Slack, especially in the types used for values.  Fix
this by using a much more consistent format, where we preserve their
pattern of prefixing IDs with letters.

This fixes a bug where Zulip's team_id could be the empty string,
which tripped up using GitLab's slash commands with Zulip.

Fixes #19588.
2021-09-23 11:19:20 -07:00
Sahil Batra c233ee9935 settings: Migrate twenty_four_hour_time setting to RealmUserDefault.
This commit removes the existing default_twenty_four_hour_time field in
Realm table which was used to set the twenty_four_hour_time setting of
new user on joining and instead we now use the twenty_four_hour_time
field of RealmUserDefault table for the same.

With some tweaks by tabbott to clarify the documentation.
2021-09-23 10:44:42 -07:00
Mateusz Mandera cacff28578 types: Fix declared type of custom profile field values.
None of the existing custom profile field types have the value as an
integer like declared in many places - nor is it a string like currently
decalred in types.py. The correct type is Union[str, List[int]]. Rather
than tracking this in so many places throughout the codebase, we add a
new ProfileDataElementValue type and insert it where appropriate.
2021-09-22 16:22:46 -07:00
Mateusz Mandera c4edda016f models: Remove property decorator from UserProfile.profile_data.
Having this as a property is misleading, given that it does db queries
to fetch the CustomProfileField data.
2021-09-22 16:22:46 -07:00
Anders Kaseorg bd3c099978 test_decorators: Convert "".format to Python 3.6 f-strings.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-09-22 14:59:42 -07:00
Eeshan Garg 4118c4a56b find_team: Send one email per email address, not per organization.
With changes to the copy for these emails by tabbott.

Fixes #19659.
2021-09-21 15:56:50 -07:00
Aman Agrawal 233d250eff do_make_stream_web_public: Send update to client.
Send update event to client after a stream is made web public.

This has been documented in the API documentation since feature level
73; previously the value was always false.
2021-09-21 12:16:10 -07:00
Aman Agrawal 5138652810 update_stream_backend: Add ability to make streams web public.
We allow clients to make existing streams web public via the API.

This feature is still disabled via settings in production
environments, because we may have additional policy rules or UI
warnings we wish to add to this sort of conversion.
2021-09-21 12:16:09 -07:00
Aman Agrawal 6a78112940 subscribe: Allow web public stream creation via the API.
User can now create web public stream via the /subscribe API.
So, when a web public stream present in the API request does not
exist, it will be created now by specifying the is_web_public
parameter. The parameter would have been ignored without this
commit.
2021-09-21 11:20:36 -07:00
Tim Abbott 71b8a1794a streams: Use standard error message when requiring owner.
The new error message is more clear about why, "User cannot create
stream with this settings." was bad English, and in any case removing
an unnecessary string is always an improvement for translators.
2021-09-21 11:05:30 -07:00
Tim Abbott 8fe61674d4 settings: Add web_public_streams_enabled server setting.
This new setting both serves as a guard to allow us to merge API
support for web public streams to main before we're ready for this
feature to be available on Zulip Cloud, and also long term will
protect self-hosted servers from accidentally enabling web-public
streams (which could be a scary possibility for the administrators of
a corporate Zulip server).
2021-09-21 10:50:53 -07:00
Abhijeet Prasad Bodas 743712c267 user_topic: Filter on visibility_policy in muted_topic code.
This is the final step required for it to be possible for us to add
UserTopic rows in the future with a differen visibility_policy.

Fixes #19272.
2021-09-17 12:14:28 -07:00
Abhijeet Prasad Bodas 2aea944a7e models: Rename UserTopic.date_muted to last_updated.
This is a follow-up to #19388.

We will in the future allow patch requests to change the visibility
of an existing topic, so `last_updated` is better name for this field.

This commit does not affect the API or events in any way, but only the
database.
2021-09-17 12:14:28 -07:00
Mateusz Mandera 8ad7520180 ldap: Support alternative attrs to userAccountControl.
Fixes #17456.

The main tricky part has to do with what values the attribute should
have. LDAP defines a Boolean as
Boolean = "TRUE" / "FALSE"
so ideally we'd always see exactly those values. However,
although the issue is now marked as resolved, the discussion in
https://pagure.io/freeipa/issue/1259 shows how this may not always be
respected - meaning it makes sense for us to be more liberal in
interpreting these values.
2021-09-16 13:29:13 -07:00
Gilbert Bishop-White 28932ba8ee test_signup: Refactor test_signup_without_full_name to use verify_signup.
The test now uses submit_reg_form_for_user, meaning a blank
full_name is posted to /accounts/register/ rather than the
parameter being excluded.

Fixes part of #7564
2021-09-16 10:51:01 -07:00
Gilbert Bishop-White 330d1385d9 test_signup: Refactor test_signup_without_password to use verify_signup.
I had to pass stop_after_reg_form=True, as the call to get_user in
verify_signup fails. I am not sure whether this is the expected
behavior. Also this causes the test to use submit_reg_form_for_user,
meaning a blank password is posted to /accounts/register/ rather than
no password.

Fixes part of #7564
2021-09-16 10:51:01 -07:00
Gilbert Bishop-White a632fc0e10 test_signup: Refactor test_signup_invalid_name to use verify_signup.
Extend `signup` helper function to enable it's use, and add additional
checks from  `test_signup_invalid_name`

Fixes part of #7564
2021-09-16 10:51:01 -07:00
Mateusz Mandera 63162210c1 test_signup: Fix the fetching of UserProfile in verify_signup.
get_user_by_delivery_email should be used, given that the email variable
is the realm email address that the account is being created with, not
the .email field which can be a dummy address based on org settings.
2021-09-16 10:51:01 -07:00
Mateusz Mandera 99ade3f535 test_signup: Fix asserted redirect Location in verify_signup.
The expected Location should depend on the realm.
2021-09-16 10:51:01 -07:00
Vishnu KS 392b2900cf plans: Redirect to /upgrade when clicked on buy standard from root.
Currently we used to redirect to /new when the user click on buy
standard from the root domain. Instead we redirect to /upgrade page.
The /upgrade page redirect would ask user to enter the subdomain
of their organization and would then redirect them to /upgrade
page of their organization.
2021-09-15 17:38:50 -07:00
Alex Vandiver 8c72959951 push_notifications: Handle empty rendered_messages.
This parallels fe25517295, but for mobile notifications.  It also
adds a test, which verifies that such content does not crash either
mobile or email notifications.
2021-09-15 09:45:38 -07:00
Alex Vandiver 039b869ca5 email_notifications: Fix inline-ing of image-URL-only messages.
fe25517295 adjusted the email_notifications codepath to use
`lxml.html.fragment_fromstring` method when parsing
`rendered_content`, but left the tests using a helper which called
`fromstring`.

Switching the tests to match the code as run reveals a bug -- using
`drop_tree` on all `message_inline_image` classes now _does_ remove
all of a top-level image-URL-only message.  Previously, such messages
were "safe" from the block that calls `drop_tree` only by dint of
`drop_tree` being a silent no-op for the root element.  When parsed
using `fragment_fromstring`, they are no longer the root, and as such
an empty message results.

Reorder relative_to_full_url to check for only one `message_inline_image`
within the top `<div>`, and only run the `drop_tree` path in the
alternate case.  Tests must be adjusted for their output now including
one more layer of `<div>`.
2021-09-15 09:45:38 -07:00
Mateusz Mandera c460351898 auth: Add logging of successful attempts to social codepath.
The previous commit introduced logging of attempts for username+password
backends. For completeness, we should log, in the same format,
successful attempts via social auth backends.
2021-09-14 15:39:41 -07:00
Aman Agrawal 354491d037 models: Add is_web_public property to Realm model.
This will allow us easily know if a realm has web_public
enabled or not.
2021-09-14 10:33:53 -07:00
Riken Shah 8c31e6f96e emoji: Add backend changes to support still image for animated emojis.
Now, when we add a custom animated emoji to the realm
we also save a still image of it (1st frame of the gif). So
we can avoid showing an animated emoji every time.
2021-09-12 07:13:04 +00:00
Mateusz Mandera bea9205c4a create_confirmation_link: Turn some args into kwargs-only. 2021-09-10 16:53:03 -07:00
Mateusz Mandera 3205f680c1 do_invite_users: Turn some args into kwargs-only. 2021-09-10 16:53:03 -07:00
shanukun 8c1ea78d7d invite: Extend invite api for handling expiration duration.
This extends the invite api endpoints to handle an extra
argument, expiration duration, which states the number of
days before the invitation link expires.

For prereg users, expiration info is attached to event
object to pass it to invite queue processor in order to
create and send confirmation link.
In case of multiuse invites, confirmation links are
created directly inside do_create_multiuse_invite_link(),

For filtering valid user invites, expiration info stored in
Confirmation object is used, which is accessed by a prereg
user using reverse generic relations.

Fixes #16359.
2021-09-10 16:53:03 -07:00
Eeshan Garg 85b1f1bed9 email: Update follow-up day 1 emails to mention demo orgs.
This is a part of #19523.
2021-09-10 16:31:54 -07:00
Eeshan Garg bf9c35ded4 onboarding: Mention demo orgs in Welcome Bot's PMs.
This is a part of #19523.
2021-09-10 16:31:54 -07:00
Abhijeet Prasad Bodas 17b8d53612 email notifs: Update scheduled timestamps after batching period change.
The API for changing the batching period was added in
5db4fe8652.

This is a follow up to that commit. We also update the timestamps for
existing scheduled email notifications entries so that the effect of
changing the setting is immediate.

Part of #15280
2021-09-10 13:25:47 -07:00
Mateusz Mandera f064e3ebac auth: Ensure a realm can't be created on SOCIAL_AUTH_SUBDOMAIN. 2021-09-10 10:47:15 -07:00
Mateusz Mandera fa58bc340d auth: Make the deactivated user error message clearer. 2021-09-10 10:47:15 -07:00
Mateusz Mandera c5806d9728 auth: Include user-input email in some error messages in the login form.
Fixes #13126.
2021-09-10 10:47:15 -07:00
Mateusz Mandera fb3864ea3c auth: Change the look of SOCIAL_AUTH_SUBDOMAIN when directly opened.
SOCIAL_AUTH_SUBDOMAIN was potentially very confusing when opened by a
user, as it had various Login/Signup buttons as if there was a realm on
it. Instead, we want to display a more informative page to the user
telling them they shouldn't even be there. If possible, we just redirect
them to the realm they most likely came from.
To make this possible, we have to exclude the subdomain from
ROOT_SUBDOMAIN_ALIASES - so that we can give it special behavior.
2021-09-10 10:47:15 -07:00
Tim Abbott f1c0309971 test_users: Fix comparison of differently sorted sets.
This fixes a nondeterministic test failure in main.
2021-09-09 11:30:43 -07:00
Sahil Batra 88346949b5 messages: Do not allow mentioning system user groups.
We do not allow mentioning system user groups for now
because this can lead to circumventing the wildcard
mention restrictions. It will be enabled once we add
a setting to control that.

This is implemented by just ignoring it as one of the
mentioned user group even if the message content
inlcudes the mention syntax for it and the message
is sent normally.

We still keep the for_mention parameter for accessing
user group while sending email and push notifications
as mentioning system user groups will be allowed in
future.
This commit also removes the test for email notifications
for system user groups as we are not allowing mentioning
them.

This commit is only for backend change as we already
exclude the system groups from mention typeaheads and
other UI.
2021-09-09 11:25:33 -07:00
sahil839 8bab712cb4 create_user: Initialize settings according to realm-level defaults.
This commit modifies the copy_user_settings code such that instead
of source user profile, we can have two types of sources - a user
profile and RealmUserDefault table of realm and then set the
settings from RealmUserDefault only is there is no user profile
as a source.

We also rename copy_user_settings to copy_default_settings for
clarity.
2021-09-09 10:55:19 -07:00
Sahil Batra 693d58265e realm: Add 'PATCH /realm/user_settings_defaults' endpoint.
The realm-level default value of settngs for new users will
be updated using this endpoint.
2021-09-09 10:55:18 -07:00
Sahil Batra 17087cf06f actions: Add do_set_realm_user_default_setting.
This commit adds do_set_realm_user_default_setting which
will be used to change the realm-level defaults of settings
for new users.

We also add a new event type "realm_user_settings_defaults"
for these settings and a "realm_user_settings_default" object
in '/register' response containing all the realm-level default
settings.
2021-09-09 10:38:38 -07:00
sahil839 7d64a9053b models: Ensure every realm has a RealmUserDefault object.
Because we create all realms with do_create_user (including in the
test suite), we just need to change that function, add a migration for
existing realms, and ensure the data import code path correctly
creates these objects.

Note that the import code path will create a RealmUserDefault row with
default values if it is not present in the import data, which is
important for importing data from other tools like Slack.
2021-09-09 10:28:44 -07:00
Sahil Batra 03a0cfda64 settings: Set enable_marketing_emails after copying user settings.
We set the enable_marketing_emails setting after copying user
settings to override the value selected in registration form.
This change is also necessary because enable_marketing_emails
field is present in RealmUserDefault to avoid copying code
but we do not use this value actually and instead we want
the setting to be set according to the value in registration
form.

We set this setting only for non-bot users since we generally
do not set any settings for bots.
2021-09-09 10:15:07 -07:00
Sahil Batra 526de3f7d1 settings: Remove unnecessary loop of notification_setting_types.
We still used notification_setting_types in copy_user_settings
function of create_user.py and in a test in test_event_system.py.
It is not required to do so since we have added all settings in
property_types already and we loop over property_types at both
these places which includes all settings.
2021-09-09 09:03:06 -07:00
Anders Kaseorg 1376feda93 docs: Add missing space in “emoji set”.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-09-08 15:31:28 -07:00
Wesley Aptekar-Cassels f743dda008 emoji: Change default emojiset from Google classic to Google modern.
This is a step towards updating our emoji to support newer Unicode
emojis (#19371).

See CZO for discussion:
https://chat.zulip.org/#narrow/stream/2-general/topic/emoji.20version.3F/near/1239756
2021-09-09 02:00:36 +05:30
Sahil Batra a229231dd4 tests: Remove redundant test_notify_settings test.
We already test all the notification settinsg in
test_toggling_boolean_display_settings (which is
now renamed to test_toggling_boolean_user_settings)
as all settings are now moved to property_types and
we are merging other parts also to consider all the
settings under one category.
2021-09-08 11:04:44 -07:00
Sahil Batra d6d1a139d0 settings: Use single function to test all non-bool user settings. 2021-09-08 11:04:44 -07:00
Sahil Batra 5d7ed2c63b tests: Add separate test for invalid values of user settings.
This commit adds a separate test for invalid values of user settings
and remove the existing code for it in test_change_user_setting.

This change will enable us to merge the tests for notification
settings to this because email_notification_batching_period_settings
has different invalid values than other integer values and we do the
same for realm settings also.
2021-09-08 11:04:44 -07:00
Sahil Batra 2f61afea9f settings: Remove do_change_notification_settings function.
This commit removes the do_change_notification_settings
function and we use do_change_user_setting directly to
change all the user settings now.
2021-09-08 11:04:44 -07:00
Sahil Batra 550d97a593 settings: Refactor callers of do_change_user_setting to pass acting_user. 2021-09-08 11:04:44 -07:00
Eeshan Garg f0ea002d94 events: Add demo_organization_scheduled_deletion_date to /register.
This commit adds `demo_organization_scheduled_deletion_date` to
the `realm` section of the `/register` response so that it is
available to clients when enabled.

This is a part of #19523.
2021-09-07 17:40:33 -07:00
Shelly 6d92534d51 models: Add setters for is_realm_owner and is_moderator.
This fixes a regression where one could end up deactivating all owners
of a realm when trying to synchronize LDAP with the `is_realm_admin`
flag configured in `AUTH_LDAP_USER_FLAGS_BY_GROUP`.

With tweaks by tabbott to add is_moderator as well.

Fixes #18677.
2021-09-07 17:15:39 -07:00
Tim Abbott fecdc234bf models: Rename setting-changed RealmAuditLog entry.
We will likely in the future want to start creating these for all
settings changes, and this new name will make the code more readable
when we do so.
2021-09-07 10:49:56 -07:00
Dinesh 9443e01a5d refactor: Rename do_set_user_display_setting to do_set_user_setting. 2021-09-07 10:16:42 -07:00
Dinesh 430c5cb8e7 user_settings: Create _legacy dicts for existing settings.
Since 84742a0, all settings are sent in the `user_settings` dictionary
which were previously sent inline with other fields in /register
response.

In order to simplify the process of adding new personal settings, we
want to transition to a world where new settings only need to consider
the `property_types` object, and code that needs to reference the
legacy behavior interacts with an object with `legacy` in its name.

This way, contributors working on new settings don't need to think
about the legacy code paths at all.

See https://chat.zulip.org/#narrow/stream/378-api-design/topic/user.20settings.20response.20in.20.2Fregister
to understand this better.
2021-09-07 10:14:45 -07:00
Tim Abbott 5ab64d46a6 test_signup: Fix buggy test setup.
676edb3802 incorrectly converted this to
have the link still be valid.
2021-09-06 17:55:04 -07:00
Mateusz Mandera 676edb3802 confirmation: Migration to add expiry_date step 1.
The commit:
1. Adds the new field as nullable.
2. Adds code that'll create new Confirmation with the field set
   correctly.
3. For verifying validity of Confirmation object this still uses the old
   logic in get_object_from_key() to keep things functioning until we
   backfill the old objects in the next step.

Thus this commit is deployable. Next we'll have a commit to run a
backfill migration.
2021-09-06 17:28:29 -07:00
Mateusz Mandera 1c4ad6769d actions: Fix incorrect calls to do_invite_users.
An integer or no argument is supposed to be passed.
These weren't caught by mypy because booleans are integers in python,
see https://github.com/python/mypy/issues/1757
2021-09-06 17:28:29 -07:00
Anders Kaseorg 646c04eff2 Rename default branch to ‘main’.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-09-06 12:56:35 -07:00
Aman Agrawal 34a37f55d6 spectators: Rename is_web_public_compatible functions. 2021-09-06 09:00:16 -07:00
Anders Kaseorg 9399b95fec push_notifications: Remove redundant APNs retry loop.
aioapns already has a retry loop.  By default it retries forever on
ConnectionError and ConnectionClosed, so our own retry loop would
never be reached.  Remove our retry loop, and configure aioapns to
retry APNS_MAX_RETRIES times on ConnectionError like the previous
version did.  It still retries forever on ConnectionClosed; that’s not
configurable but probably fine.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-09-03 11:49:33 -07:00
Anders Kaseorg 4e2cba1ce1 test_push_notifications: Add test for unexpected APNs error.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-09-03 11:49:33 -07:00
Anders Kaseorg 56a9d669f8 test_push_notifications: Create futures for the right event loop.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-09-03 11:49:33 -07:00