Commit Graph

17787 Commits

Author SHA1 Message Date
Alex Vandiver adf9e8de3d home: Redirect https://auth.zuliphostname/ to https://zuliphostname/.
It is confusing for folks directing accessing the page to get a
"Organization does not exist" message.
2024-01-15 17:55:42 -08:00
Mateusz Mandera 100cef9186 templates: Move remote_realm_server_mismatch_error.html to zerver.
This is rendered by regular self-hosted servers, so doesn't belong in
zilencer.
2024-01-15 16:50:48 -08:00
Mateusz Mandera fc247cba3f remote_billing: Fix /self-hosted-billing/ handling for desktop app.
When you click "Plan management", the desktop app opens
/self-hosted-billing/ in your browser immediately. So that works badly
if you're already logged into another account in the browser, since that
session will be used and it may be for a different user account than in
the desktop app, causing unintended behavior.

The solution is to replace the on click behavior for "Plan management"
in the desktop app case, to instead make a request to a new endpoint
/json/self-hosted-billing, which provides the billing access url in a
json response. The desktop app takes that URL and window.open()s it (in
the browser). And so a remote billing session for the intended user will
be obtained.
2024-01-15 16:50:48 -08:00
Mateusz Mandera 80f5963bbc auth: Add a configurable wrapper around authenticate calls. 2024-01-15 12:18:48 -08:00
Mateusz Mandera d3b4cbd182 auth: Add hardening authenticate(use_dummy_backend=True) in do_login.
As explained in the comment, this is to prevent bugs where some strange
combination of codepaths could end up calling do_login without basic
validation of e.g. the subdomain. The usefulness of this will be
extended with the upcoming commit to add the ability to configure custom
code to wrap authenticate() calls in. This will help ensure that some
codepaths don't slip by the mechanism, ending up logging in a user
without the chance for the custom wrapper to run its code.
2024-01-15 12:18:48 -08:00
Mateusz Mandera 56ca307bc1 tests: Delete test_social_auth_backends.
This test is ancient and patches so much that it's almost unreadable,
while being redundant considering we have comprehensive tests via the
SocialAuthBase subclasses. The one missing case was the one with the
backend we disabled. We replace that with a proper
test_social_auth_backend_disabled test in SocialAuthBase.
2024-01-15 12:18:48 -08:00
Prakhar Pratyush 676c07be5b tests: Rename *topic local variables to *topic_name.
This is preparatory work towards adding a Topic model.
We plan to use the local variable name as 'topic' for
the Topic model objects.

Currently, we use *topic as the local variable name for
topic names.

We rename local variables of the form *topic to *topic_name
so that we don't need to think about type collisions in
individual code paths where we might want to talk about both
Topic objects and strings for the topic name.
2024-01-15 09:40:43 -08:00
Prakhar Pratyush b7e56ccbdc lib: Rename *topic local variables to *topic_name.
This is preparatory work towards adding a Topic model.
We plan to use the local variable name as 'topic' for
the Topic model objects.

Currently, we use *topic as the local variable name for
topic names.

We rename local variables of the form *topic to *topic_name
so that we don't need to think about type collisions in
individual code paths where we might want to talk about both
Topic objects and strings for the topic name.
2024-01-15 09:40:43 -08:00
Prakhar Pratyush bc66eaee7d views: Rename *topic local variables to *topic_name.
This is preparatory work towards adding a Topic model.
We plan to use the local variable name as 'topic' for
the Topic model objects.

Currently, we use *topic as the local variable name for
topic names.

We rename local variables of the form *topic to *topic_name
so that we don't need to think about type collisions in
individual code paths where we might want to talk about both
Topic objects and strings for the topic name.
2024-01-15 09:40:43 -08:00
Prakhar Pratyush 1eef052bd1 actions: Rename *topic local variables to *topic_name.
This is preparatory work towards adding a Topic model.
We plan to use the local variable name as 'topic' for
the Topic model objects.

Currently, we use *topic as the local variable name for
topic names.

We rename local variables of the form *topic to *topic_name
so that we don't need to think about type collisions in
individual code paths where we might want to talk about both
Topic objects and strings for the topic name.
2024-01-15 09:40:43 -08:00
Lauryn Menard f27b6d896a audit-logs: Add generic property changed for Customer and CustomerPlan.
Change audit logs for changing a Customer or CustomerPlan property
to use a generic event type and specify the property name in the
extra data.
2024-01-15 03:10:04 -08:00
Tim Abbott bafa476fd3 presence: Avoid having incomplete missing test coverage.
Rename and restructure these comparison variables such that we don't
have a possibly impossible case for presence.last_connected_time being
None.

Fixes #25498.
2024-01-14 18:32:01 -08:00
Alex Vandiver e6a0284275 queue_processors: Defer initial email connection creation.
We previously created the connection to the outgoing email server when
the EmailSendingWorker was first created.  Since creating the
connection can fail (e.g. because of firewalls or typos in the
hostname), this can cause the `QueueProcessingWorker` creation to
raise an exception.  In multi-threaded mode, exceptions in the worker
threads which are _not_ during the handling of a specific event
percolate out to `log_and_exit_if_exception` and trigger the
termination of the entire process -- stopping all worker threads from
making forward progress.

Contain the blast radius of misconfigured email servers by deferring
the opening of the connection until it is first needed.  This will not
cause any overall performance change, since it only affects the
latency of the very first email after startup.
2024-01-12 08:38:46 -08:00
Alex Vandiver e1acd7b974 process_queue: For threaded workers, create them when they start.
Creating the QueueProcessingWorker objects when the ThreadedWorker is
created can lead to a race which caused confusing error messages:

1. A thread tries to call `self.worker = get_worker()`
2. This call raises an exception, which is caught by
   `log_and_exit_if_exception`
3. `log_and_exit_if_exception` sends our process a SIGUSR1, _but
    otherwise swallows the error_.
4. The thread's `.run()` is called, which tries to access
   `self.worker`, which was never set, and throws another exception.
5. The process handles the SIGUSR1, restarting.

Move the creation of the worker to when it is started, so the worker
object does not need to be stored, and possibly have a decoupled
failure.
2024-01-12 08:38:46 -08:00
Alex Vandiver e0bd169c77 docs: Fix other help pages that were renamed or moved, to save a redirect. 2024-01-11 13:52:12 -08:00
Alex Vandiver c217280c23 docs: Fix help links for pm -> dm rename, to save a redirect. 2024-01-11 13:52:12 -08:00
Prakhar Pratyush f7d4d04130 api_docs: Update doc for realm_push_notification_enabled_end_timestamp.
This commit updates the doc to explicitly mention that
'realm_push_notifications_enabled_end_timestamp' is a UNIX timestamp.
2024-01-11 12:32:16 -08:00
Sahil Batra c0c9623ae4 message: Allow system bots to mention group if everyone else can.
We now allow system bots to mention a group if can_mention_group
setting is set to "role:everyone" group and not when it is set
to some other group.
2024-01-10 14:57:21 -08:00
sanchi-t 0a626f5e3c csrf_failure: Update error page.
Switches from Django's default error page to Zulip standard error
template.  Also updates template for 405 error code to not use the 404
art.

Fixes #25626.
2024-01-10 09:49:24 -08:00
Alex Vandiver 7720d362ba messages: Only lock UserMessage rows, not other joined tables.
By default, `SELECT FOR UPDATE` will also lock any rows which are
`JOIN`ed into the selected rows; in the case of UserMessage rows, this
can mean arbitrary Message rows.

Since the messages themselves are not being changed, it is not
necessary to lock them -- and doing so may lead to deadlocks, in the
case that the UserMessage row is locked for update before the Message,
and some other request has already taken a read lock on the Message
and is blocked on the UserMessage write lock.

Change `select_for_update_query` to explicitly only lock UserMessage.
2024-01-09 13:13:59 -08:00
Evgenii 3f06596cf0
dev_ldap_directory: Use f-strings for better readability. 2024-01-09 12:09:09 -08:00
Lauryn Menard 2994685399 support: Add admin support for updating end date of active plan.
This currently will only apply to tier.SELF_HOSTED_LEGACY plans.
2024-01-09 10:13:21 -08:00
Alex Vandiver 9c336e4374 documentation: Support having no sidebar for policies docs.
We document the sidebar as being optional, so make it so.
2024-01-07 19:11:16 -08:00
Alex Vandiver 5b1387c9ba send_password_reset_email: Add a flag to only email users who need it.
Emailing the password reset email to users who have already logged in
is not as useful.
2024-01-07 16:43:31 -08:00
Alex Vandiver 6635f3ea12 send_password_reset_email: Narrow type of users.
5a32ea52ae changed the return type of `get_users`, enabling this.
2024-01-07 16:43:31 -08:00
Lauryn Menard d8044a1b07 api-docs: Revise text of /api/test-notify endpoint.
Updates title and main description to follow the general style
of the API endpoint documentation.

Updates `token` description to clarify suggested mobile client
behavior.
2024-01-05 17:07:28 -08:00
Lauryn Menard 5051ad6ca5 api-tests: Exclude some documented endpoints from curl tests.
Adds a set of excluded endpoints for the test of generated curl
examples in the API documentation.

Currently, only the `api/test-notify` endpoint is excluded since
there would need to be a push notification bouncer set up to test
that generated curl example.
2024-01-05 17:07:28 -08:00
Sahil Batra 63363ba50a push_notification: Check users count for plans to be downgraded.
We return expected_end_timestamp as "None" for the plans to be
downgraded if number of users is not more than MAX_USERS_WITHOUT_PLAN
since they will be downgraded to self-managed plan and would
have push notifications enabled.
2024-01-05 13:30:57 -08:00
Mateusz Mandera 3ec3ac63f2 zilencer: Have server send realm_uuid to remaining bouncer endpoints.
Requests to these endpoint are about a specified user, and therefore
also have a notion of the RemoteRealm for these requests. Until now
these endpoints weren't getting the realm_uuid value, because it wasn't
used - but now it is needed for updating .last_request_datetime on the
RemoteRealm.
2024-01-05 13:09:09 -08:00
Mateusz Mandera cbfbdd7337 zilencer: Add last_request_datetime to RemoteRealm + RemoteZulipServer.
For the RemoteRealm case, we can only set this in endpoints where the
remote server sends us the realm_uuid. So we're missing that for the
endpoints:

- remotes/push/unregister and remotes/push/unregister/all
- remotes/push/test_notification

This should be added in a follow-up commit.
2024-01-05 13:09:09 -08:00
Mateusz Mandera 42343cd8d9 remote_billing: Improve nocoverage tagging on self_hosting_auth_redirect. 2024-01-05 12:06:43 -08:00
Mateusz Mandera 1ee45314c1 tests: Fix how test_is_outdated_server works.
os.path.getmtime needs to be mock.patched or otherwise the success of
the test depends on the filesystem state and breaks if version.py hasn't
been modified in a while.
2024-01-05 12:06:43 -08:00
Alex Vandiver 4ab9cd7cf2 markdown: Prevent OverflowError with large time integers.
`<time:1234567890123>` causes a "signed integer is greater than
maximum" exception from dateutil.parser; datetime also cannot handle
it ("year 41091 is out of range") but that is a ValueError which is
already caught.

Catch the OverflowError thrown by dateutil.
2024-01-05 12:01:06 -08:00
Alex Vandiver 75d6f35069 s3: Add a setting for S3 addressing style.
This controls if boto3 attempts to use
`https://bucketname.endpointname/` or `https://endpointname/bucket/`
as its prefix.  See
https://botocore.amazonaws.com/v1/documentation/api/latest/reference/config.html

Fixes: #28424.
2024-01-05 11:12:18 -08:00
Alex Vandiver 3aea67a8ed s3: Only use get_bucket to get to boto3 clients and resources.
boto3 has two different modalities of making API calls -- through
resources, and through clients.  Resources are a higher-level
abstraction, and thus more generally useful, but some APIs are only
accessible through clients.  It is possible to get to a client object
from a resource, but not vice versa.

Use `get_bucket(...).meta.client` when we need direct access to the
client object for more complex API calls; this lets all of the
configuration for how to access S3 to sit within `get_bucket`.  Client
objects are not bound to only one bucket, but we get to them based on
the bucket we will be interacting with, for clarity.

We removed the cached session object, as it serves no real purpose.
2024-01-05 11:12:18 -08:00
Alex Vandiver 214bd4ed88 s3: Stop caching get_boto_client, which is only ever called once.
e883ab057f started caching the boto client, which we had identified
as slow call.  e883ab057f went further, calling
`get_boto_client().generate_presigned_url()` once and caching that
result.

This makes the inner cache on the client useless.  Remove it.
2024-01-05 11:12:18 -08:00
Lauryn Menard fb29a35262 support: Add updating minimum licenses requirement after discount.
Adds a support action for updating the minimum licenses on a
customer object once a default discount has also been set.

In the case that the current billing entity has a current active
plan or a scheduled upgrade to a new plan, then the minimum
licenses will not be updated.
2024-01-04 18:42:07 -08:00
Tim Abbott 880133295e zilencer: Skip deactivating registrations when locally deleted.
This protects us from incorrectly handling situations where someone
tested and upgrade to 8.0 for a backup on a separate hostname, and
left the test system live while upgrading the main system, in a way
that results in duplicate RemoteRealm objects that are all marked as
locally deleted.

Further word is required to figure out how to avoid the original
duplication problem.
2024-01-04 18:31:16 -08:00
Alex Vandiver 511cc4d8f7 management: Provide a way to deactivate the push registration. 2024-01-04 15:34:28 -08:00
Alex Vandiver bd38e6cb69 send_email: Distinct emails means distinct, case-insensitively. 2024-01-04 10:46:53 -08:00
Alex Vandiver 8d9ead0f6d send_custom_email: Order by delivery_email if necessary.
If we `.distinct("delivery_email")` then we must also
`.order_by("delivery_email")`; adc987dc43 added the `.order_by`
call, which broke the newsletter codepath, since it did not contain
the `delivery_email` in the ordering fields.

Add a flag to distinct on emails in `send_custom_email`.
2024-01-04 10:46:53 -08:00
Alex Vandiver 0697247d59 send_custom_email: Send to recently-active plus owners and admins.
The set of `enable_marketing_emails=True` are those that have opted
into getting marketing newsletter emails -- but we previously limited
further to only those users active in the last month.

Broaden that to "opted in, and either recently active or an owner or
an admin," with the goal of providing information to folks who may
have tried out Zulip in the past.

Co-authored-by: Tim Abbott <tabbott@zulip.com>
2024-01-04 10:46:53 -08:00
Prakhar Pratyush c4330be2b1 user_topics: Validate 'topic' parameter length <= max_topic_length.
Earlier, 'topic' parameter length for
'/users/me/subscriptions/muted_topics' and '/user_topics' endpoints
were not validated before DB operations which resulted in exception:
'DataError: value too long for type character varying(60)'.

This commit adds validation for the topic name length to be
capped at 'max_topic_length' characters.

The doc is updated to suggest clients that the topic name should
have a maximum length of 'max_topic_length'.

Fixes #27796.
2024-01-04 09:43:27 -08:00
Anders Kaseorg 9703b4d70b login: Remove external_authentication_methods from page_params.
This was used by a work-in-progress social login implementation in
zulip-desktop that was reverted for security reasons and never
released.

https://github.com/zulip/zulip/pull/13782
https://github.com/zulip/zulip-desktop/pull/863

We replaced it by a secure implementation with a maintainable API.

https://github.com/zulip/zulip-desktop/pull/943

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-12-29 13:02:12 -08:00
Mateusz Mandera d3e8ddd6ce test_push_notifications: Use self.server instead of .get expression.
It makes no sense to do RemoteZulipServer.objects.get with
self.server_uuid, when there simply is self.server.
2023-12-23 14:58:27 -08:00
Mateusz Mandera 82ee93d967 zilencer: Set .remote_realm for existing RemotePushDeviceToken.
Old RemotePushDeviceTokens were created without this attribute. But when
processing a notification, if we have remote_realm, we can take the
opportunity to to set this for all the registrations for this user.
2023-12-23 14:58:27 -08:00
Aman Agrawal d4a852e97c stripe: Save a query to get CustomerPlan for sponsored customers. 2023-12-18 12:55:34 -08:00
Tim Abbott 5dc1d36f73 zilencer: Fix community plan push notifications. 2023-12-17 09:36:47 -08:00
Anders Kaseorg b38f3d2472 models: Extract zerver.models.lookups.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-12-16 22:08:44 -08:00
Anders Kaseorg c343d7c30e models: Move query_for_ids to zerver.lib.query_helpers.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-12-16 22:08:44 -08:00