Commit Graph

15824 Commits

Author SHA1 Message Date
Sahil Batra b9e0575829 realm: Allow only owners to change waiting_period_threshold setting.
We allow only owners to change the waiting period setting to become
full member. This commit contains only backend changes, frontend
changes will be done separately.
2022-09-16 15:27:52 -07:00
Sahil Batra f23bfbd462 realm_domains: Allow only owners to add, edit or delete domains.
We allow only owners to add, edit or delete the allowed domains.
This commit only contains backend changes, frontend changes will
be done in a separate commit.
2022-09-16 15:27:52 -07:00
Sahil Batra 1396fbab6e realm: Allow only owners to change restricted-login related settings.
We allow only owners to change disallow_disposable_email_addresses
and emails_restricted_to_domains settings. This commit only contains
change in backend part, frontend changes will be done separately.
2022-09-16 15:27:52 -07:00
Sahil Batra 63fa2a9338 realm: Allow only owners to change invite_required in backend.
We allow only owners to change the invite_required setting.
This commit only adds the restriction in backend, frontend
changes will be done separately.

We also add a helper function in test_realm.py to avoid
writing same code repeatedly and this helper will also
be used in tests for other settings to be added in
further commits.
2022-09-16 15:04:54 -07:00
Tim Abbott 76bcb96414 realm_export: Improve estimate of data export size.
As suggested by the new comments, the cost for a Zulip data export
scales with messages actually included in the export, so an
organizations with 1M private messages but only 50K public stream
messages should not be modeled the same as one with 1M public stream
messages for the purpose of the limits here.

Also improve the comments and variable names more generally.
2022-09-16 14:54:45 -07:00
Anders Kaseorg 31b4be3b9b event_queue: Replace sequential queue IDs with UUIDs.
This prevents ID conflicts when resharding between multiple Tornado
shards.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-09-16 10:06:07 -07:00
Tim Abbott 183e7be31c models: Fix subscribed typo. 2022-09-15 16:21:09 -07:00
Mateusz Mandera d86e4ac34d models: Improve comment documentation around Recipients and Huddles. 2022-09-15 16:12:17 -07:00
Anders Kaseorg 987ab741f9 sharding: Support Tornado sharding by regexes.
One should now be able to configure a regex by appending _regex to the
port number:

[tornado_sharding]
9802_regex = ^[l-p].*\.zulipchat\.com$

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-09-15 16:07:50 -07:00
Sahil Batra c3fe8420fd streams: Allow unsubscribing others based on can_remove_subscribers_group.
Current value of can_remove_subscribers_group field is admins system group
only so behavior is not changed. We would provide support to change this
setting from API and UI in further commits.
2022-09-14 16:14:15 -07:00
Sahil Batra b9248c75f4 stream: Add do_change_can_remove_subscribers_group and field to objects.
This commit adds do_change_can_remove_subscriber_group function for
changing can_remove_subscribers_group field of a stream. We also add
can_remove_subscribers_group_id field to stream and subscription
objects.

This function will be helpful for writing tests in next commit.
We would add API and UI support to change this setting in further
commits.
2022-09-14 16:03:11 -07:00
Sahil Batra 86c2f6881e models: Update can_remove_subscribers_group field to not be null.
This commit udpates can_remove_subscribers_group to be not null.
We already added a migration to set the value of this field for
existing streams and also added a commit to set this field to
admins system group for now while creating streams.
2022-09-14 16:03:11 -07:00
Sahil Batra c9c230c2da migrations: Add migration to set default of can_remove_subscribers_group.
This migration sets can_remove_subscribers_group value to admins system
group for all the existing streams. In further commit we would change
can_remove_subscribers_group to be not null and thus we add this migration
to ensure all existing streams have this setting value set.
2022-09-14 16:03:11 -07:00
Sahil Batra b0de5c0f36 streams: Set can_remove_subscribers_group while creating streams.
This commit sets can_remove_subscribers_group to admins system
group while creating streams as it will be the default value
of this setting. In further we would provide an option to set
value of this setting to any user group while creating streams
using API or UI.
2022-09-14 16:03:11 -07:00
Sahil Batra 941bb5a372 import: Fix import related code for user groups.
We change the import order to import UserGroup objects before
Stream such that we can set can_remove_subscribers_group correctly.

We do not import UserGroupMembership objects here along with
UserGroup since UserProfile objects are not imported and
GroupGroupMembership are also imported later as these are not
required before.
2022-09-14 16:03:11 -07:00
Sahil Batra 49d61d19f4 models: Add can_remove_subscribers_group field to Stream.
This new setting will be used to check which user group
is allowed to unsubscribe others from the stream.
2022-09-14 16:03:11 -07:00
Zixuan James Li e2bf8c49af test_decorators: Remove cachify test cases.
cachify has been removed in 9d448e73d2.
We don't need to keep its tests.

TODO: functools.lru_cache can be replaced by functools.cache when we
drop Python 3.8.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-09-14 09:40:46 -07:00
Anders Kaseorg 61e149dbb5 subdomain: Remove impossible None case from user_matches_subdomain.
The only two callers pass get_subdomain(request) which can’t be None.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-09-14 09:35:36 -07:00
Lauryn Menard f0dac7beef docs: Update base titles for documentation pages.
Updates the base titles so that they begin with the page content,
and end with "| Zulip" +  information about the type of doc: "help
center", "API documentation", "terms and policies", or "integrations".
2022-09-13 11:36:52 -07:00
Lauryn Menard 09d94c62a5 integrations: Revise integration CATEGORY dict structure.
Adds a META_CATEGORY dict for categories that are not best described
as groups of 'tools', so that in the subsequent commit the PAGE_TITLE
can be set accordingly.

Also, removes 'tools' from the 'Miscellaneous' category text and
spells out 'Human resources' instead of using 'HR'.
2022-09-13 11:36:52 -07:00
Sahil Batra 1e55e7641e export: Do not export direct_members and direct_subgroups field.
We do not need direct_members and direct_subgroups field of
UserGroup objects in the export data since we already have
UserGroupMembership and GroupGroupMembership object data.

While importing we keep these fields empty when creating
UserGroup objects and direct_members and direct_subgroups
fields will get set when UserGroupMembership and
GroupGroupMembership objects are created.

This change will also help us in further changes when we
will change the order of importing to import UserGroup
objects just after Realm objects.
2022-09-13 11:07:09 -07:00
Sahil Batra c714ad993b test_import_export: Add tests for GroupGroupMembership objects. 2022-09-13 11:07:09 -07:00
Anders Kaseorg bd9a1dc971 tests: Consistently JSON-encode ‘to’ parameter
Although our POST /messages handler accepts the ‘to’ parameter with or
without JSON encoding, there are two problems with passing it as an
unencoded string.

Firstly, you’d fail to send a message to a stream named ‘true’ or
‘false’ or ‘null’ or ‘2022’, as the JSON interpretation is prioritized
over the plain string interpretation.

Secondly, and more importantly for our tests, it violates our OpenAPI
schema, which requires the parameter to be JSON-encoded.  This is
because OpenAPI has no concept of a parameter that’s “optionally
JSON-encoded”, nor should it: such a parameter cannot be unambiguously
decoded for the reason above.

Our version of openapi-core doesn’t currently detect this schema
violation, but after the next upgrade it will.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-09-13 11:05:37 -07:00
Anders Kaseorg a1de8d95a8 openapi: Fix ‘to’ parameter schema.
The previous schema incorrectly prohibited the string, integer, and
string-array forms that we do in fact accept.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-09-13 11:05:37 -07:00
Anders Kaseorg a5a12398fe openapi: Fix ‘narrow’ parameter schema.
The previous schema incorrectly prohibited the two-element array form
that we do in fact accept, and didn’t specify anything about the
contents of the object form.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-09-13 11:05:37 -07:00
Yogesh Sirsat 50c13f9db0 custom_profile_fields: Backend tests for default external account type. 2022-09-12 14:29:11 -07:00
Yogesh Sirsat c355934179 custom_profile_fields: Change data of default external account type.
change the names of "github" and "twitter" external account fields to
"GitHub username" and "Twitter username" respectively and remove the
hints of them.
2022-09-12 14:29:11 -07:00
Florian Pritz a276603766 rocketchat: Deduplicate and ignore huddle rooms with same users.
If there are more than 1 room with the same set of users, the import
will fail due to a unique constraint on the huddle_hash. Figuring out
why and which room is causing this database error is kinda difficult.

We deduplicate those cases here and simply merge the rooms together.
Note however, that the deduplication does not work as expected so we
simply ignore them all together for now and only raise an exception
along some logging output. At least this way, it is pretty clear what is
wrong and you do not have to wait to get a database error during the
actual import.

We also ignore empty huddle rooms since those are the duplicates that
caused problems for me and if they are empty, ignoring them is easier
than trying to get the merge to work.

Not sure where those channels come from since we discovered this with
production data.

Signed-off-by: Florian Pritz <bluewind@xinu.at>
2022-09-09 16:57:24 -07:00
Florian Pritz 3677aabcbd rocketchat: Ignore mention mapping failures.
Not sure where those come from since we discovered this with production
data.

Signed-off-by: Florian Pritz <bluewind@xinu.at>
2022-09-09 16:57:24 -07:00
Florian Pritz c308799133 rocketchat: Only set message content if it exists.
Not sure where those come from since we discovered this with production
data.

Signed-off-by: Florian Pritz <bluewind@xinu.at>
2022-09-09 16:57:24 -07:00
Florian Pritz 1cc2764d45 rocketchat: Ignore reactions from non-existant users.
Not sure where those come from since we discovered this with production
data. Somehow there were reactions with usernames that were old and no
longer existed.

Signed-off-by: Florian Pritz <bluewind@xinu.at>
2022-09-09 16:57:24 -07:00
Florian Pritz 26fe028534 rocketchat: Truncate long stream names.
These will lead to an error during import otherwise.

Signed-off-by: Florian Pritz <bluewind@xinu.at>
2022-09-09 16:57:24 -07:00
Florian Pritz 3a27919b5b rocketchat: Ignore rocketchat attachments without types.
Not sure where those come from since we discovered this with production
data.

There only was a single instance of this in my entire batch of data in
an old message from the time when we started using Rocket.Chat. This
might be an old issue or it might require some special settings that
were later changed.

Signed-off-by: Florian Pritz <bluewind@xinu.at>
2022-09-09 16:57:24 -07:00
Florian Pritz 5ec8f4ef09 rocketchat: Ignore missing rocketchat attachments.
Not sure where those come from since we discovered this with production
data.

Signed-off-by: Florian Pritz <bluewind@xinu.at>
2022-09-09 16:57:24 -07:00
Florian Pritz 96fa0991f8 rocketchat: Handle long or invalid rocketchat attachment names.
Signed-off-by: Florian Pritz <bluewind@xinu.at>
2022-09-09 16:57:24 -07:00
David Rosa 620854d437 integrations: Add incoming webhook macro to relevant docs.
Several integration docs instruct the user to create a bot, but don't
specify that the type of bot should be "Incoming webhook".

Renames create-a-bot.md -> create-an-incoming-webhook.md for clarity,
and replaces all incomplete instructions with this macro.
Renames bot_types.png -> bot_types_incoming_webhook.png and updates
the image with a screenshot of the latest UI.
2022-09-09 16:31:33 -07:00
David Rosa 217915a50c integrations: Remove "Yo".
The Yo company shut down in 2016.
https://en.wikipedia.org/wiki/Yo_(app)#History

Removes `yo` instances from `zerver/lib/integrations.py`.
Removes `zerver/webhooks/yo`.
Removes `static/images/integrations/yo-app`.
2022-09-09 16:31:33 -07:00
Raghav Luthra 4dad9fa158 user_settings: Add user setting to control the user list style.
Added a user_list_style personal user setting to the bottom of
Settings > Display settings > Theme section which controls the look
of the right sidebar user list.

The radio button UI includes a preview of what the styles look like.

The setting is intended to eventually have 3 possible values: COMPACT,
WITH_STATUS and WITH_AVATAR; the final value is not yet implemented.

Co-authored-by: Tim Abbott <tabbott@zulip.com>
2022-09-09 16:30:54 -07:00
Mateusz Mandera 95bc77ae4f help center: Update "SAML authentication" to follow help center guidelines.
Co-authored-by: Alya Abbott <alya@zulip.com>
2022-09-09 10:30:05 -07:00
Anders Kaseorg 7d485aa58f docs: Correct more “webapp” to “web app”.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-09-08 20:23:28 -07:00
Anders Kaseorg 4c38b1371b test_auth_backends: Fix SOCIAL_AUTH_GITHUB_TEAM_ID mock.
SOCIAL_AUTH_GITHUB_TEAM_ID is expected to be the string of an integer.

The requests mock for the bogus /None URL is unused because the
function that would request it is itself mocked.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-09-08 20:23:28 -07:00
Anders Kaseorg 6f5346cc7d stripe: Fix WildValue usage.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-09-08 20:22:38 -07:00
yogesh sirsat e4b74c2099 streams: Fix misleading comment in "test_subs". 2022-09-08 16:56:07 -07:00
Zixuan James Li 8a80443b73 uptimerobot: Use the create-bot-construct-url macro for uptimerobot.
Since the setup for uptimerobot does not deviate from what we have in
the "create-bot-construct-url" macro, we can reuse it and event
filtering instructions for uptimerobot will be automatically included.

TODO: Add event filtering documentation for buildbot when it supports
custom endpoint URL.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-09-08 13:37:35 -07:00
Zixuan James Li 6b527dfa61 integrations: Add documentation for the event filtering system.
We create "event-filter-instruction.md" and add it to
"create-bot-construct-url.md". This allows the user to keep track of the
supported event types for most of the integrations that implement this
feature. Note that not all integrations use "create-bot-construct-url.md".

We also need to rename "function" to "view_function" to make this change
type-check.

This is relevant to #18392.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-09-08 13:37:35 -07:00
Anders Kaseorg a9b12ae067 message_fetch: Remove obsolete UnicodeDecodeError handler.
This handler from commit a696141a25
(#7418) was almost certainly made unreachable when commit
c3e395b7d8 (#13092) removed anything
that might have thrown a UnicodeDecodeError from highlight_string.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-09-08 11:07:43 -07:00
Lauryn Menard 7f738803f0 help-docs: Rename organization language help center file.
Renames article about organization language used for automated
messages and invitation emails. Creates URL redirect and updates
links in repository (web app, help center and api documentation).

Prior to this change, the article was named:
'change-the-default-language-for-your-organization'.

Fixes #21949.
2022-09-07 09:45:37 -07:00
Christopher Chong 28173cafc8 message_flags: Fix deadlocks when updating message flags.
Previously, an active production Zulip server would experience a class
of deadlocks caused by two or more concurrent bulk update operations
on the UserMessage table.

This is because UPDATE ... SET ... WHERE statements that execute in
parallel take row-level UPDATE locks as they get results; since the
query plans may result in getting rows in different orders between two
queries, this can result in deadlocks.

Some databases allow ORDER BY on their UPDATE ... WHERE statements;
PostgreSQL does not. In PostgreSQL, the answer is to do a sub-select
with an ORDER BY ... FOR UPDATE to ensure consistent ordering on row
locks.

We do this all code paths using bitand or bitor as part of bulk
editing message flags, which should ensure that these concurrent
operations obtain row level locks on the table in the same order.

Fixes #19054.
2022-09-06 16:06:58 -07:00
Matt Keller e1023f45cf sentry: Properly stringify a possibly-None result. 2022-09-06 15:06:05 -07:00
Mateusz Mandera 01d659b672 internal_prep_private_message: Remove redundant realm arg.
This is redundant and is taken based on recipient_user anyway.
2022-09-06 15:03:17 -07:00