Commit Graph

600 Commits

Author SHA1 Message Date
Anders Kaseorg f27a709a5e populate_db: Spell “stickiness” correctly.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-07-05 13:09:34 -07:00
Zixuan James Li c72fe80525 management: Remove migrate_stream_notifications.
The script has been broken since `Subscription.notifications`
was removed.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-06-21 12:57:01 -07:00
Tim Abbott 1304c435ce populate_db: Restore test database predictable set of topics.
856eff0fe6 perturbed the initial state
of the 30 messages in the test database, which unfortunately some
tests depend on.
2022-06-08 12:34:45 -07:00
Tim Abbott 856eff0fe6 populate_db: Ensure many streams have "more topics" in development.
We increase the total number of messages, since increasing the number
of topics would otherwise have the side effect of making it hard to
find longer conversations.
2022-06-08 12:08:56 -07:00
Sahil Batra 6e8523015c populate_db: Add users to initial private stream in mit and lear realm.
We do not allow keeping vacant private streams as we deactivate them
when all users are unsubscribed, so it is better to add at least one
user in the initial private stream created while creating realm.
2022-06-02 13:42:40 -07:00
Zixuan James Li d5517932cd typing: Use BaseDatabaseSchemaEditor in place of DatabaseSchemaEditor.
This is a part of #18777.

Signed-off-by: Zixuan James Li <359101898@qq.com>
2022-05-30 14:18:53 -07:00
Sahil Batra aa98b39429 bulk_create: Copy realm-level defaults in bulk_create_users.
This commit adds code to copy the realm-level default of
settings while creating users through bulk_create_users.

We do not directly call 'copy_default_settings' as it
calls ".save()" but here we want to bulk_create the objects
for efficiency.

We also add the code to set realm-default of enter_sends as
True for the Zulip dev server as done in 754b547e8 and thus
we remove enter_sends argument from create_user_profile as
it is of no use now.
2022-05-04 12:52:43 -07:00
Anders Kaseorg c9faefd50e cache: Instantiate only one BMemcached cache backend.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-05-02 17:41:49 -07:00
Anders Kaseorg a2825e5984 python: Use Python 3.8 typing.{Protocol,TypedDict}.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-04-27 12:57:49 -07:00
evykassirer e4c51488c0 populate_db: Add support for creating messages older than 5 days ago.
Fixes #21646.
2022-04-25 16:16:15 -07:00
Tim Abbott 6e831db55b populate_db: Don't mark messages as read for bot users.
Bots don't generally do API requests to mark messages as read. If they
did, it's likely because the developer of the bot wants them to appear
in read receipts or similar (E.g. as an indication of what messages
have been processed).

So we should avoid setting the read flag on bot messages the test
database.
2022-04-25 13:41:38 -07:00
Alex Vandiver ede4a88b49 populate_db: Remove unnecessary pre-population of URL cache.
76deb30312 changed this to not just be the URL, but rather a
prefixed hash of the URL, but failed to update this location which
wrote to it.  This meant that this pre-population step was writing to
the wrong keys in the durable cache, and thus ineffective.

Then, da33b72848 switched the cache to be in-memory, making this
write to the wrong keys in an in-process memory store.  There is no
way to pre-fill this sort of cache, except at server start-up.

Finally, and most fundamentally, 8c0c9ca7a4 then disabled
`inline_url_embed_preview` by default, making the code entirely moot.

Remove the triply-unnecessary code.
2022-04-15 14:48:12 -07:00
Anders Kaseorg e01faebd7e actions: Split out zerver.actions.create_realm.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-04-14 17:14:37 -07:00
Anders Kaseorg e5500a2226 actions: Split out zerver.actions.reactions.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-04-14 17:14:35 -07:00
Anders Kaseorg cbad5739ab actions: Split out zerver.actions.create_user.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-04-14 17:14:35 -07:00
Anders Kaseorg 5fcbc412cf actions: Split out zerver.actions.streams.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-04-14 17:14:35 -07:00
Anders Kaseorg 975066e3f0 actions: Split out zerver.actions.message_send.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-04-14 17:14:34 -07:00
Anders Kaseorg ec6355389a actions: Split out zerver.actions.user_settings.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-04-14 17:14:34 -07:00
Anders Kaseorg d7981dad62 actions: Split out zerver.actions.users.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-04-14 17:14:34 -07:00
Anders Kaseorg bbce879c81 actions: Split out zerver.actions.custom_profile_fields.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-04-14 17:14:33 -07:00
Anders Kaseorg a29f1b39da actions: Move part into zerver.lib.streams.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-04-14 17:14:32 -07:00
Anders Kaseorg 385616f27f actions: Split out zerver.actions.realm_emoji.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-04-14 17:14:31 -07:00
Tim Abbott 2328a81f55 devtools: Remove development-only user creation management commands.
We now have nicer version available not only to developers, and it's
definitely better to deduplicate these.
2022-03-21 12:05:59 -07:00
Tim Abbott 6b00c748fd onboarding: Deduplicate realm creation initial user logic.
We now call this function inside do_create_user(...,
realm_creation=True), which generally improves readability and
robustness of the codebase.

This fixes a bug where this onboarding content was not correctly done
when creating a realm via LDAP, and also will be important as we add
new code paths that might let you create a realm.
2022-03-21 12:05:59 -07:00
Sahil Batra 9345c344d7 user_groups: Add members to the System groups created.
This commit adds users to the appropriate system user group
based on their role. We also change the user groups when
changing role of the user.

We also add migration to add existing users to the appropriate
user groups.

This commit adds update_users_in_full_members_system_group which
is currently used to update the full members group on changing
role of a user. This function will be modified in next commit such
that it can be used to update full members group on changing
waiting_period_threshold setting of realm.
2022-03-14 18:53:46 -07:00
Steve Howell 8f99894302 streams: Extract stream_color library.
This is a pure code move.
2022-03-14 18:01:36 -07:00
Mateusz Mandera f90beae616 zilencer: Drop the index from RemotePushDeviceToken.user_id.
The index isn't used, because our unique_index entries provide better
indexes for the queries.
2022-03-14 17:47:30 -07:00
Mateusz Mandera 0677c90170 zilencer: Change push bouncer API to accept uuids as user identifier.
This is the first step to making the full switch to self-hosted servers
use user uuids, per issue #18017. The old id format is still supported
of course, for backward compatibility.

This commit is separate in order to allow deploying *just* the bouncer
API change to production first.
2022-03-14 17:47:30 -07:00
Alex Vandiver f531f3a27f push_notifications: Drop FCM retries to 2, not 10.
This reverts bc15085098 (which provided
not justification for its change) and moves further, down to 2 retries
from the default of 5.

10 retries, with exponential backoff, is equivalent to sleeping 2^11
seconds, or just about 34 minutes (though the code uses a jitter which
may make this up to 51 minutes).  This is an unreasonable amount of
time to spend in this codepath -- as only one worker is used, and it
is single-threaded, this could effectively block all missed message
notifications for half an hour or longer.

This is also necessary because messages sent through the push bouncer
are sent synchronously; the sending server uses a 30-second timeout,
set in PushBouncerSession.  Having retries which linger longer than
this can cause duplicate messages; the sending server will time out
and re-queue the message in RabbitMQ, while the push bouncer's request
will continue, and may succeed.

Limit to 2 retries (APNS currently uses 3), and results expected max
of 4 seconds of sleep, potentially up to 6.  If this fails, there
exists another retry loop above it, at the RabbitMQ layer (either
locally, or via the remote server's queue), which will result in up to
3 additional retries -- all told, the request will me made to FCM up
to 12 times.
2022-03-08 12:52:58 -08:00
Anders Kaseorg 21cd1c10b3 docs: Add missing space in “time zone”.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-02-24 14:05:12 -08:00
evykassirer 8c3ff92964 populate_db: Add timezones for some test users.
After failing to notice a place where we wanted to hide timezone
information, we decided to add timezones to some of the test
users, so that we can better consider the effects of timezones
when manually testing.

Testing:

* ran populate_db and confirmed users had timezones in the UI
* updated test_populate_db.py
2022-02-22 11:14:58 -08:00
Sahil Batra 947b2b55fe populate_db: Set owners for bots in development and test database.
Since we do not allow to remove owners from bots, it is better
to keep owners for the bots in development environment as well.

We need to change puppeteer tests here because now desdemona
already has bots in dev server and thus "Active bots" section
is opened by default in the settings instead of "Add a new bot"
section.
2022-02-22 10:35:18 -08:00
Anders Kaseorg 1629d6bfb3 python: Reformat with Black 22 (stable).
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-02-18 18:03:13 -08:00
Anders Kaseorg b0ce4f1bce docs: Fix many spelling mistakes.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-02-07 18:51:06 -08:00
Lauryn Menard 3be622ffa7 backend: Add request as parameter to json_success.
Adds request as a parameter to json_success as a refactor towards
making `ignored_parameters_unsupported` functionality available
for all API endpoints.

Also, removes any data parameters that are an empty dict or
a dict with the generic success response values.
2022-02-04 15:16:56 -08:00
Eeshan Garg 3bc0f8c6f9 zilencer: Add endpoint for deactivating remote server registration. 2022-01-21 14:57:04 -08:00
Eeshan Garg 7a1ed9a4cb zilencer: Remove management command for adding remote server.
As @timabbott mentioned on #20577, this command was mostly useful
during early development of the feature, and is no longer needed now
that we have an API for accomplishing the same thing.
2022-01-19 13:13:52 -08:00
Anders Kaseorg 1dfddffc8d profile_request: Use modern Django middleware API.
Fixes “RemovedInDjango40Warning: Passing None for the middleware
get_response argument is deprecated.” from LogRequests().

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-01-13 21:03:22 -08:00
Mateusz Mandera 868ed17661 remote_server: Handle invalid server uuid being given authing to API.
get_remote_server_by_uuid (called in validate_api_key) raises
ValidationError when given an invalid UUID due to how Django handles
UUIDField. We don't want that exception and prefer the ordinary
DoesNotExist exception to be raised.
2022-01-04 14:40:49 -08:00
Alex Vandiver 1b395b6403 zilencer: Truncate APNS notifications correctly.
APNs payloads nest the zulip-custom data further than the top level,
as Android notifications do.  This led to APNs data silently never
being truncated; this case was not caught in tests because the mocks
provided the wrong data for the APNs structure.

Adjust to look in the appropriate place within the APNs data, and
truncate that.
2022-01-03 15:24:16 -08:00
Eeshan Garg 0b5324f345 corporate: Add helper for deactivating remote server registrations. 2022-01-03 14:02:48 -08:00
Eeshan Garg 4cc35c339b migrations: Backfill audit log entries for remote server creation.
This is a follow-up to #20408.
2022-01-03 12:58:00 -08:00
Mateusz Mandera 4153b5c517 remote_server: Improve uuid validation at the server/register endpoint.
As explained in the comments in the code, just doing UUID(string) and
catching ValueError is not enough, because the uuid library sometimes
tries to modify the string to convert it into a valid UUID:

>>> a = '18cedb98-5222-5f34-50a9-fc418e1ba972'
>>> uuid.UUID(a, version=4)
UUID('18cedb98-5222-4f34-90a9-fc418e1ba972')
2021-12-31 11:18:01 -08:00
Mateusz Mandera c5c3ab66d6 remote_server: Migrate RemoteZulipServer.uuid to be UUIDField.
Given that these values are uuids, it's better to use UUIDField which is
meant for exactly that, rather than an arbitrary CharField.

This requires modifying some tests to use valid uuids.
2021-12-28 10:11:34 -08:00
Mateusz Mandera e48120fd12 remote_server: Validate zulip_org_id submitted by registering server.
zulip_org_id is supposed to be a UUID, so we want to actually validate
the format, not only check the length.
2021-12-28 10:11:34 -08:00
Tim Abbott 4cb189fc63 settings: Rename TOS_VERSION to TERMS_OF_SERVICE_VERSION.
The previous version was appropriate in a setting where it was only
used for Zulip Cloud, but it's definitely clearer to spell it out.
2021-12-09 17:51:16 -08:00
Eeshan Garg 79e9ba13e2 billing: Add do_change_remote_server_plan_type.
This is a part of the plumbing we need to support billing for
self-hosted customers.

With documentation changes from tabbott.
2021-12-07 10:25:37 -08:00
Aman Agrawal 00d3b88257 realm: Enable spectator access by default in development.
Enable spectator access for test `zulip` realm in developement
setup.

Add option in `do_create_realm` to configure
`enable_spectator_access` field of `Realm`.
2021-11-18 12:55:12 -08:00
Alex Vandiver 800e38016a queue_rate: Output to CSV, and run multiple prefetch values. 2021-11-16 11:48:50 -08:00
Alex Vandiver 6c14978cd1 zilencer: Truncate "remove" notifications from remote servers.
This is 4d055a6695, but for notifications which are received from
remote hosts.
2021-11-10 13:39:35 -08:00