Commit Graph

2507 Commits

Author SHA1 Message Date
Simon Michalke b6a25840a1 zerver: Implement ldap group synchronization.
Fixes #9957.

Co-authored-by: Mateusz Mandera <mateusz.mandera@zulip.com>
2023-10-05 16:12:52 -07:00
Mateusz Mandera a095c34503 saml: Add OneLogin_Saml2_ValidationError to caught parsing exceptions.
This is an exception that we should be generally catching like the
others, which will give our standard /login/ redirect and proper logging
- as opposed to a 500 if we don't catch.

Addresses directly a bug we occurred in the wild, where a SAMLResponse
was submitted without issuers specified in a valid way, causing this
exception. The added test tests this specific type of scenario.
2023-09-27 10:32:18 -07:00
Alex Vandiver 5ee4b642ad views: Add a /health healthcheck endpoint.
This endpoint verifies that the services that Zulip needs to function
are running, and Django can talk to them.  It is designed to be used
as a readiness probe[^1] for Zulip, either by Kubernetes, or some other
reverse-proxy load-balancer in front of Zulip.  Because of this, it
limits access to only localhost and the IP addresses of configured
reverse proxies.

Tests are limited because we cannot stop running services (which would
impact other concurrent tests) and there would be extremely limited
utility to mocking the very specific methods we're calling to raising
the exceptions that we're looking for.

[^1]: https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/
2023-09-20 09:53:59 -07:00
Anders Kaseorg ddba558f1d mypy: Remove no_implicit_reexport override.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-09-08 16:46:44 -07:00
Zixuan James Li 1e1f98edb2 transaction_tests: Remove testing URL.
Rewrite the test so that we don't have a dedicated URL for testing.
dev_update_subgroups is called directly from the tests without using the
test client.
2023-09-06 09:13:02 -07:00
Zixuan James Li a081428ad2 user_groups: Make locks required for updating user group memberships.
**Background**

User groups are expected to comply with the DAG constraint for the
many-to-many inter-group membership. The check for this constraint has
to be performed recursively so that we can find all direct and indirect
subgroups of the user group to be added.

This kind of check is vulnerable to phantom reads which is possible at
the default read committed isolation level because we cannot guarantee
that the check is still valid when we are adding the subgroups to the
user group.

**Solution**

To avoid having another transaction concurrently update one of the
to-be-subgroup after the recursive check is done, and before the subgroup
is added, we use SELECT FOR UPDATE to lock the user group rows.

The lock needs to be acquired before a group membership change is about
to occur before any check has been conducted.

Suppose that we are adding subgroup B to supergroup A, the locking protocol
is specified as follows:

1. Acquire a lock for B and all its direct and indirect subgroups.
2. Acquire a lock for A.

For the removal of user groups, we acquire a lock for the user group to
be removed with all its direct and indirect subgroups. This is the special
case A=B, which is still complaint with the protocol.

**Error handling**

We currently rely on Postgres' deadlock detection to abort transactions
and show an error for the users. In the future, we might need some
recovery mechanism or at least better error handling.

**Notes**

An important note is that we need to reuse the recursive CTE query that
finds the direct and indirect subgroups when applying the lock on the
rows. And the lock needs to be acquired the same way for the addition and
removal of direct subgroups.

User membership change (as opposed to user group membership) is not
affected. Read-only queries aren't either. The locks only protect
critical regions where the user group dependency graph might violate
the DAG constraint, where users are not participating.

**Testing**

We implement a transaction test case targeting some typical scenarios
when an internal server error is expected to happen (this means that the
user group view makes the correct decision to abort the transaction when
something goes wrong with locks).

To achieve this, we add a development view intended only for unit tests.
It has a global BARRIER that can be shared across threads, so that we
can synchronize them to consistently reproduce certain potential race
conditions prevented by the database locks.

The transaction test case lanuches pairs of threads initiating possibly
conflicting requests at the same time. The tests are set up such that exactly N
of them are expected to succeed with a certain error message (while we don't
know each one).

**Security notes**

get_recursive_subgroups_for_groups will no longer fetch user groups from
other realms. As a result, trying to add/remove a subgroup from another
realm results in a UserGroup not found error response.

We also implement subgroup-specific checks in has_user_group_access to
keep permission managing in a single place. Do note that the API
currently don't have a way to violate that check because we are only
checking the realm ID now.
2023-08-24 17:21:08 -07:00
Samuel 3ce7b77092 typing: Add typing constants to the post register api response.
Adds typing notification constants to the response given by
`POST /register`. Until now, these were hardcoded by clients
based on the documentation for implementing typing notifications
in the main endpoint description for `api/set-typing-status`.

This change also reflects updating the web-app frontend code
to use the new constants from the register response.

Co-authored-by: Samuel Kabuya <samuel.mwangikabuya@kibo.school>
Co-authored-by: Wilhelmina Asante <wilhelmina.asante@kibo.school>
2023-08-23 16:36:44 -07:00
Zixuan James Li 37660dd0e7 linkifier: Support reordering linkifiers.
This adds API support to reorder linkifiers and makes sure that the
returned lists of linkifiers from `GET /events`, `POST /register`, and
`GET /realm/linkifiers` are always sorted with the order that they
should processed when rendering linkifiers.

We set the new `order` field to the ID with the migration. This
preserves the order of the existing linkifiers.

New linkifiers added will always be ordered the last. When reordering,
the `order` field of all linkifiers in the same realm is updated, in
a manner similar to how we implement ordering for
`custom_profile_fields`.
2023-08-14 15:21:48 -07:00
Anders Kaseorg 562a79ab76 ruff: Fix PERF401 Use a list comprehension to create a transformed list.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-08-07 17:23:55 -07:00
Anders Kaseorg 733083c65d ruff: Collapse short multi-line import statements.
isort did this by default, though it’s unclear whether that was
intended; see https://github.com/astral-sh/ruff/issues/4153.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-08-02 17:41:41 -07:00
Anders Kaseorg 3b09197fdf ruff: Fix RUF015 Prefer `next(...)` over single element slice.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-07-23 15:20:53 -07:00
Anders Kaseorg b285813beb error_notify: Remove custom email error reporting handler.
Restore the default django.utils.log.AdminEmailHandler when
ERROR_REPORTING is enabled.  Those with more sophisticated needs can
turn it off and use Sentry or a Sentry-compatible system.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-07-20 11:00:09 -07:00
Alex Vandiver d957559371 uploads: Allow uploads to set storage class.
Uploads are well-positioned to use S3's "intelligent tiering" storage
class.  Add a setting to let uploaded files to declare their desired
storage class at upload time, and document how to move existing files
to the same storage class.
2023-07-19 16:19:34 -07:00
Anders Kaseorg 63be67af80 logging_util: Remove dependence on get_current_request.
Pass the HttpRequest explicitly through the two webhooks that log to
the webhook loggers.

get_current_request is now unused, so remove it (in the same commit
for test coverage reasons).

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-07-11 22:23:47 -07:00
Anders Kaseorg f66e2c3112 sentry: Remove dependence on get_current_request.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-07-11 22:23:47 -07:00
Lauryn Menard dd59e83d54 welcome-emails: Make some code comments and docstrings more evergreen. 2023-07-11 14:15:52 -07:00
Alex Vandiver 8a77cca341 middleware: Detect reverse proxy misconfigurations.
Combine nginx and Django middlware to stop putting misleading warnings
about `CSRF_TRUSTED_ORIGINS` when the issue is untrusted proxies.
This attempts to, in the error logs, diagnose and suggest next steps
to fix common proxy misconfigurations.

See also #24599 and zulip/docker-zulip#403.
2023-07-02 16:20:21 -07:00
Alex Vandiver cf0b803d50 zproject: Prevent having exactly 17/18 middlewares, for Python 3.11 bug.
Having exactly 17 or 18 middlewares, on Python 3.11.0 and above,
causes python to segfault when running tests with coverage; see
https://github.com/python/cpython/issues/106092

Work around this by adding one or two no-op middlewares if we would
hit those unlucky numbers.  We only add them in testing, since
coverage is a requirement to trigger it, and there is no reason to
burden production with additional wrapping.
2023-07-02 16:20:21 -07:00
Lalit a686c0cc02 docs: Add documentation for delete emoji endpoint.
This endpoint was previously marked as `intentionally_undocumented`
but that was mistake.

Removed `intentionally_undocumented` and added proper documentation
with valid `python_example` for this Endpoint.

Fixes: #24084
2023-06-28 10:23:47 -07:00
Alex Vandiver 76d7a5a53a dev_settings: Remove `THUMBNAIL_IMAGES` from test_extra_settings.
THUMBNAIL_IMAGES was previously set to true as there were tests on a new
thumbnail functionality. The feature was never stable enough to remain in
the codebase and the setting was left enabled. This setting also doesn't
reflect how the production deployments are and it has been decided that we
should drop setting from test_extra_settings altogether.

Co-authored-by: Joseph Ho <josephho678@gmail.com>
2023-06-12 16:26:55 -07:00
Alex Vandiver fbb831ff3b uploads: Allow access to the /download/ variant anonymously.
This was mistakenly left off of b799ec32b0.
2023-06-12 12:55:27 -07:00
Alex Vandiver 0dbe111ab3 test_helpers: Switch add/remove_ratelimit to a contextmanager.
Failing to remove all of the rules which were added causes action at a
distance with other tests.  The two methods were also only used by
test code, making their existence in zerver.lib.rate_limiter clearly
misplaced.

This fixes one instance of a mis-balanced add/remove, which caused
tests to start failing if run non-parallel and one more anonymous
request was added within a rate-limit-enabled block.
2023-06-12 12:55:27 -07:00
evykassirer aa270bcef0 search: Remove support for experimental search pills.
This in-progress feature was started in 2018 and hasn't
been worked on much since. It's already in a broken state,
which makes it hard to iterate on the existing search bar
since it's hard to know how those changes will affect search
pills.

We do still want to add search pills eventually, and when
we work on that, we can refer to this diff to readd the
changes back.
2023-06-06 18:36:02 -07:00
Anders Kaseorg b7909db987 ruff: Fix PLC0208 Use a sequence type when iterating over values.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-06-06 14:58:11 -07:00
Alex Vandiver cdc94d2719 prod_settings_template: Document REALM_HOSTS configuration.
This was in docs/production/multiple-organizations.md, but not the
settings template.
2023-06-06 09:17:42 -07:00
Anders Kaseorg b5e5728112 coverage: Clean up coverage configuration.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-05-31 13:53:04 -07:00
Lauryn Menard 957382253a scheduled-messages: Split out edit scheduled message endpoint.
Part of splitting creating and editing scheduled messages.
Should be merged with final commit in series. Breaks tests.

Splits out editing an existing scheduled message into a new
view function and updated `edit_scheduled_message` function.
2023-05-26 18:05:55 -07:00
Mateusz Mandera 45e3626bd2 saml: Clean up additional session vars if authentication fails.
This doesn't have any obvious security implications right now, but
nonetheless such information is not meant to stick around in the session
if authentication didn't succeed and not cleaning up would be a bug.
2023-05-23 13:01:15 -07:00
Mateusz Mandera c9fb55dd20 saml: Improve the comments about the SAMLDocument processing logic. 2023-05-23 13:01:15 -07:00
Mateusz Mandera 8fb0fe96c6 saml: Save SessionIndex in session and use when making a LogoutRequest.
This is a useful improvement in general for making correct
LogoutRequests to Idps and a necessary one to make SP-initiated logout
fully work properly in the desktop application. During desktop auth
flow, the user goes through the browser, where they log in through their
IdP. This gives them a logged in  browser session at the IdP. However,
SAML SP-initiated logout is fully conducted within the desktop
application. This means that proper information needs to be given to the
the IdP in the LogoutRequest to let it associate the LogoutRequest with
that logged in session that was established in the browser. SessionIndex
is exactly the tool for that in the SAML spec.
2023-05-23 13:01:15 -07:00
Mateusz Mandera 5dd4dcdebb saml: Make SP-initiated SLO work in the desktop application. 2023-05-23 13:01:15 -07:00
Mateusz Mandera 3f55c10685 saml: Rework SP-initiated logout config to support IdP-level config.
This gives more flexibility on a server with multiple organizations and
SAML IdPs. Such a server can have some organizations handled by IdPs
with SLO set up, and some without it set up. In such a scenario, having
a generic True/False server-wide setting is insufficient and instead
being able to specify the IdPs/orgs for SLO is needed.
2023-05-23 13:01:15 -07:00
Mateusz Mandera e8f3b87b17 saml: Sign LogoutRequests and LogoutResponses if certs are set up. 2023-05-23 13:01:15 -07:00
Mateusz Mandera 0bb0220ebb saml: Implement SP-initiated Logout.
Closes #20084

This is the flow that this implements:
1. A logged-in user clicks "Logout".
2. If they didn't auth via SAML, just do normal logout. Otherwise:
3. Form a LogoutRequest and redirect the user to
https://idp.example.com/slo-endpoint?SAMLRequest=<LogoutRequest here>
4. The IdP validates the LogoutRequest, terminates its own user session
and redirects the user to
https://thezuliporg.example.com/complete/saml/?SAMLRequest=<LogoutResponse>
with the appropriate LogoutResponse. In case of failure, the
LogoutResponse is expected to express that.
5. Zulip validates the LogoutResponse and if the response is a success
response, it executes the regular Zulip logout and the full flow is
finished.
2023-05-23 13:01:15 -07:00
Mateusz Mandera 01498add9b auth: Rename authentication_method session var. 2023-05-23 13:01:15 -07:00
Sahil Batra 7f01b3fb63 users: Set tos_version to -1 for users who have not logged-in yet.
We now set tos_version to "-1" for imported users and the ones
created using API or using other methods like LDAP, SCIM and
management commands. This value will help us to allow users to
change email address visibility setting during first login.
2023-05-16 13:52:56 -07:00
Mateusz Mandera 254ea4b0c8 social_auth: Save authentication method information in the session.
The immediate application of this will be for SAML SP-initiated logout,
where information about which IdP was used for authenticating the
session needs to be accessed. Aside of that, this seems like generally
valuable session information to keep that other features may benefit
from in the future.
2023-05-12 16:21:26 -07:00
Anders Kaseorg e88b2caeef Revert "db: Force use of TimeTrackingCursor to work around Django 4.2 bug."
This reverts commit f1925487e8.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-05-10 19:44:47 -07:00
Alex Vandiver 1285c39098 zerver: Remove now-unused report/ endpoints. 2023-05-09 13:16:28 -07:00
Aman Agrawal d60d6e9115 urls: Add new endpoint to create scheduled messages.
This will help us remove scheduled message and reminder logic
from `/messages` code path.

Removes `deliver_at`/`defer_until` and `tz_guess` parameters. And
adds the `scheduled_delivery_timestamp` instead. Also updates the
scheduled message dicts to return `scheduled_delivery_timestamp`.

Also, revises some text in `/delete-scheduled-message` endpoint
and in the `ScheduledMessage` schema in the API documentation.
2023-04-28 17:25:00 -07:00
AcKindle3 4544eb4576 email: Replace `uri` with `url` in templates and backend.
In #23380 we want to change all ocurrences of `uri` to `url`. This
commit changes the ocurrences of `uri` appeared in files related to
email, including templates (`.html`, `.txt`) and backend (`.py`)
codes.

In `email.md`, `base_images_uri` is changed to `images_base_url` -
the words `base` and `images` are swapped and plural form is added
for `image`.  This is becasue the former is not found anywhere in
the codebase while the later appears a lot. To reduce confusion,
this doccumentation changed accordingly.
2023-04-26 16:37:16 -07:00
Tim Abbott 027b67be80 presence: Rewrite the backend data model.
This implements the core of the rewrite described in:

For the backend data model for UserPresence to one that supports much
more efficient queries and is more correct around handling of multiple
clients.  The main loss of functionality is that we no longer track
which Client sent presence data (so we will no longer be able to say
using UserPresence "the user was last online on their desktop 15
minutes ago, but was online with their phone 3 minutes ago").  If we
consider that information important for the occasional investigation
query, we have can construct that answer data via UserActivity
already.  It's not worth making Presence much more expensive/complex
to support it.

For slim_presence clients, this sends the same data format we sent
before, albeit with less complexity involved in constructing it.  Note
that we at present will always send both last_active_time and
last_connected_time; we may revisit that in the future.

This commit doesn't include the finalizing migration, which drops the
UserPresenceOld table.
The way to deploy is to start the backfill migration with the server
down and then start the server *without* the user_presence queue worker,
to let the migration finish without having new data interfering with it.
Once the migration is done, the queue worker can be started, leading to
the presence data catching up to the current state as the queue worker
goes over the queued up events and updating the UserPresence table.

Co-authored-by: Mateusz Mandera <mateusz.mandera@zulip.com>
2023-04-26 14:26:47 -07:00
Anders Kaseorg 9db3451333 Remove statsd support.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-04-25 19:58:16 -07:00
Ruchir Harbhajanka e08256ce88 urls: Move jwt_fetch_api_key endpoint to v1_api_mobile_patterns.
Moves jwt_fetch_api_key endpoint to v1_api_mobile_patterns so
that tools/test-api detects it as an API endpoint that is pending
documentation.

Fixes #24982.
2023-04-18 15:44:31 -07:00
Mateusz Mandera eb4fc7568c auth_enabled_helper: Add realm_authentication_methods argument.
This allows removing pointless db queries when calling
*_auth_enabled(realm) repeatedly.
2023-04-18 09:22:56 -07:00
Mateusz Mandera 72d56d5d59 auth: Remove Realm.AUTHENTICATION_FLAGS class attribute.
With the removal of the authentication_methods bitfield this is now
useless and just duplicates AUTH_BACKEND_NAME_MAP keys.
2023-04-18 09:22:56 -07:00
Mateusz Mandera ffa3aa8487 auth: Rewrite data model for tracking enabled auth backends.
So far, we've used the BitField .authentication_methods on Realm
for tracking which backends are enabled for an organization. This
however made it a pain to add new backends (requiring altering the
column and a migration - particularly troublesome if someone wanted to
create their own custom auth backend for their server).

Instead this will be tracked through the existence of the appropriate
rows in the RealmAuthenticationMethods table.
2023-04-18 09:22:56 -07:00
Aman Agrawal a06f3d26d0 scheduled_messages: Add endpoints to fetch and delete them. 2023-04-14 17:38:37 -07:00
Alex Vandiver d888bb3df2 error-bot: Remove ERROR_BOT support.
This isn't sufficiently useful to keep the added complexity.  Users
should use the email error reporting, or set up Sentry error
reporting.
2023-04-13 14:59:58 -07:00
Alex Vandiver e536a14b61 report_error: Remove API endpoint for client error reporting. 2023-04-13 14:59:58 -07:00
Alex Vandiver cdaf12da44 jinja2: Make sure that 500 pages have support_email set. 2023-04-12 16:41:07 -07:00
Anders Kaseorg d2af06f4df middleware: Remove ZulipCommonMiddleware patch.
My fix for the relevant performance bug was upstreamed in Django 4.2.

https://code.djangoproject.com/ticket/33700

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-04-07 09:13:20 -07:00
Anders Kaseorg f1925487e8 db: Force use of TimeTrackingCursor to work around Django 4.2 bug.
Effectively revert commit b4cf9ad777 to
work around https://code.djangoproject.com/ticket/34466.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-04-06 17:58:44 -07:00
Prakhar Pratyush f012d079c3 user_topics: Add a new endpoint to update visibility_policy.
This commit adds a new endpoint, 'POST /user_topics' which
is used to update the personal preferences for a topic.

Currently, it is used to update the visibility policy of
a user-topic row.
2023-04-03 22:31:49 -07:00
Sahil Batra 791c3030eb backends: Update ZulipLDAPAuthBackend to use PreregistrationRealm.
This commit updates ZulipLDAPAuthBackend to use the recently added
PreregistrationRealm when creating realms with LDAP used as
authentication method.
2023-03-27 15:44:42 -07:00
Alex Vandiver 93bfd3fb5f sentry: Provide a server-side tunnel.
Some well-intentioned adblockers also block Sentry client-side error
reporting.  Provide an endpoint on the Zulip server which forwards to
the Sentry server, so that these requests are not blocked.
2023-03-24 15:16:38 -07:00
Anders Kaseorg 1db7f80f94 ruff: Fix PLW1508 Invalid type for environment variable default.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-03-24 11:57:07 -07:00
Alex Vandiver bf532de8bb queue: Allow enabling TLS for the RabbitMQ connection.
This allows using cloud-based RabbitMQ services like AmazonMQ.

Fixes: #24699.
2023-03-23 16:02:10 -07:00
Aman Agrawal c928c87645 google_analytics: Track realm registration separately from user signup.
While the function which processes the realm registration and
signup remains the same, we use different urls and functions to
call the process so that we can separately track them. This will
help us know the conversion rate of realm registration after
receiving the confirmation link.
2023-03-08 11:34:36 -08:00
Alex Vandiver 8f8a9f6f04 sentry: Add frontend event monitoring.
Zulip already has integrations for server-side Sentry integration;
however, it has historically used the Zulip-specific `blueslip`
library for monitoring browser-side errors.  However, the latter sends
errors to email, as well optionally to an internal `#errors` stream.
While this is sufficient for low volumes of users, and useful in that
it does not rely on outside services, at higher volumes it is very
difficult to do any analysis or filtering of the errors.  Client-side
errors are exceptionally noisy, with many false positives due to
browser extensions or similar, so determining real real errors from a
stream of un-grouped emails or messages in a stream is quite
difficult.

Add a client-side Javascript sentry integration.  To provide useful
backtraces, this requires extending the pre-deploy hooks to upload the
source-maps to Sentry.  Additional keys are added to the non-public
API of `page_params` to control the DSN, realm identifier, and sample
rates.
2023-03-07 10:51:45 -08:00
Alex Vandiver dcd8947c1b tests: Set LOCAL_AVATARS_DIR / LOCAL_FILES_DIR in tests.
7ad06473b6 split out `LOCAL_AVATARS_DIR` and `LOCAL_FILES_DIR` as
derived values from `LOCAL_UPLOADS_DIR`.  However, this means that all
places which set `LOCAL_UPLOADS_DIR` need to potentially propagate
that change into the derived values if they come *after*
`computed_settings.py` is applied.  It did this successfully in
`zerver/lib/test_runner.py` and the `use_s3_backend` decorator, but
did not adjust the late-set `zproject/test_extra_settings.py`.

This causes tests to share a single common set of avatars and
attachments directories.  In puppeteer tests, this leads to assertion
failures checking `assert_is_local_storage_path`; in backend tests,
this leads to races when checking the contents of the local storage
directory when run in parallel mode.

Set `LOCAL_AVATARS_DIR` and `LOCAL_FILES_DIR` based on
`LOCAL_UPLOADS_DIR` when it is pulled from the environment during
testing.
2023-03-06 11:52:23 -08:00
Anders Kaseorg 43b4f10578 run-dev: Drop .py from script name.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-03-03 18:02:37 -08:00
Alex Vandiver 015a10637b docs: Document how to use SMTP without authentication.
This is the behaviour inherited from Django[^1].  While setting the
password to empty (`email_password = `) in
`/etc/zulip/zulip-secrets.conf` also would suffice, it's unclear what
the user would have been putting into `EMAIL_HOST_USER` in that
context.

Because we previously did not warn when `email_password` was not
present in `zulip-secrets.conf`, having the error message clarify the
correct configuration for disabling SMTP auth is important.

Fixes: #23938.

[^1]: https://docs.djangoproject.com/en/4.1/ref/settings/#std-setting-EMAIL_HOST_USER
2023-02-27 11:59:48 -08:00
Mateusz Mandera 52515a1560 presence: Tweak and document presence tuning values.
We're changing the ping interval from 50s to 60s, because that's what
the mobile apps have hardcoded currently, and backwards-compatibility
is more important there than the web app's previously hardcoded 50s.

For PRESENCE_PING_INTERVAL_SECS, the previous value hardcoded in both
clients was 140s, selected as "plenty of network/other latency more
than 2 x ACTIVE_PING_INTERVAL_MS". This is a pretty aggressive value;
even a single request being missed or 500ing can result in a user
appearing offline incorrectly. (There's a lag of up to one full ping
interval between when the other client checks in and when you check
in, and so we'll be at almost 2 ping intervals when you issue your
next request that might get an updated connection time from that
user).

To increase failure tolerance, we want to change the offline
threshhold from 2 x ACTIVE_PING_INTERVAL + 20s to 3 x
ACTIVE_PING_INTERVAL + 20s, aka 140s => 200s, to be more robust to
temporary failures causing us to display other users as offline.

Since the mobile apps currently have 140s and 60s hardcoded, it should
be safe to make this particular change; the mobile apps will just
remain more aggressive than the web app in marking users offline until
it uses the new API parameters.

The end result in that Zulip will be slightly less aggressive at
marking other users as offline if they go off the Internet. We will
likely be able to tune ACTIVE_PING_INTERVAL downwards once #16381 and
its follow-ups are completed, because it'll likely make these requests
much cheaper.
2023-02-21 12:10:43 -08:00
Mateusz Mandera 8ef889f392 presence: Use server-provided params in the webapp.
As of the previous commit, the server provides these parameters in
page_params. The defaults match the values hard-coded in the webapp so
far - so get rid of the hard-coded values in favor of taking them from
page_params.
2023-02-21 12:10:43 -08:00
Mateusz Mandera a593089770 presence: Return some presence params in the /register response. 2023-02-21 12:10:43 -08:00
Mateusz Mandera 4ba4305ff9 presence: Change server-side OFFLINE_THRESHOLD_SECS to 140s.
This old 300s value was meaningfully used in 2 places:
1. In the do_change_user_settings presence_enabled codepath when turning
   a user invisible. It doesn't matter there, 140s is just since the
   point is to make clients see this user as offline. And 140s is the
   threshold used by clients (see the presence.js constant).
2. For calculating whether to set "offline" "status" in
   result["presence"]["aggregated"] in get_presence_backend. It's fine
   for this to become 140s, since clients shouldn't be looking at the
   status value anymore anyway and just do their calculation based on
   the timestamps.
2023-02-21 12:10:43 -08:00
Alex Vandiver d3403dde86 rabbitmq: Add a RABBITMQ_PORT setting. 2023-02-20 12:04:25 -08:00
Alex Vandiver 63de600451 backends: Use staticfilesfor social auth logos. 2023-02-14 17:17:06 -05:00
Alex Vandiver 3559bf416d settings: Allow customization of STATIC_URL.
Some requests will still be to `/static/`, either at the realm's
domain, or at the root domain.
2023-02-14 17:17:06 -05:00
Alex Vandiver e31767dda4 settings: Make DEFAULT_LOGO_URI/DEFAULT_AVATAR_URI use staticfiles. 2023-02-14 17:17:06 -05:00
Alex Vandiver 3109d40b21 puppet: Add a sentry release class.
This installs the Sentry CLI, and uses it to send API events to Sentry
when a release is started and completed.
2023-02-10 15:53:10 -08:00
Abhijeet Prasad Bodas 9d1c131dc6 muted users: Make file naming consistent.
This makes the names of the relevant files consistant with the database
model name as well as the frontend JS files.
2023-02-10 15:39:57 -08:00
Prakhar Pratyush 906ff9243a user_mutes: Rename 'muting.py' to 'user_mutes.py'.
Rename 'muting.py' to 'user_mutes.py' because it, now
, contains only user-mute related functions.

Includes minor refactoring needed after renaming the file.
2023-02-07 00:23:47 +05:30
Prakhar Pratyush 49577bbdcd user_topics: Move topic muting functions to user_topics.py.
This commit moves topic related stuff i.e. topic muting functions
to a separate file 'views/user_topics.py'.

'views/muting.py' contains functions related to user-mutes only.
2023-02-07 00:23:47 +05:30
Anders Kaseorg 59eca10a43 ruff: Fix G004 Logging statement uses f-string.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-02-04 16:36:20 -08:00
Alessandro Toppi ff89590558 auth: Add JWT-based user API key fetch.
This adds a new endpoint /jwt/fetch_api_key that accepts a JWT and can
be used to fetch API keys for a certain user. The target realm is
inferred from the request and the user email is part of the JWT.

A JSON containing an user API key, delivery email and (optionally)
raw user profile data is returned in response.
The profile data in the response is optional and can be retrieved by
setting the POST param "include_profile" to "true" (default=false).

Co-authored-by: Mateusz Mandera <mateusz.mandera@zulip.com>
2023-02-03 15:23:35 -08:00
Lauryn Menard dbacc00f0f api-docs: Move markdown files to top level directory.
- Updates `.prettierignore` for the new directory.
- Updates any reference to the API documentation directory for
  markdown files to be `api_docs/` instead of `zerver/api/`.
- Removes a reference link from `docs/documentation/api.md` that
  hasn't referenced anything in the text since commit 0542c60.
- Update rendering of API documentation for new directory.
2023-02-02 17:25:40 -08:00
Anders Kaseorg df001db1a9 black: Reformat with Black 23.
Black 23 enforces some slightly more specific rules about empty line
counts and redundant parenthesis removal, but the result is still
compatible with Black 22.

(This does not actually upgrade our Python environment to Black 23
yet.)

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-02-02 10:40:13 -08:00
Anders Kaseorg d9182775c3 ruff: Fix S701 Consider using `autoescape=True`.
Django already provides this default, but be explicit to silence a
Ruff warning.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-01-26 10:16:30 -08:00
Anders Kaseorg 7e3a681f80 ruff: Fix S108 Probable insecure usage of temporary file.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-01-26 10:14:56 -08:00
Lauryn Menard dedea23745
help-docs: Move help center documentation to top level directory.
These files are not Jinja2 templates, so there's no reason that they needed
to be inside `templates/zerver`. Moving them to the top level reflects their
importance and also makes it feel nicer to work on editing the help center content, 
without it being unnecessary buried deep in the codebase.
2023-01-25 14:08:29 -08:00
Aman Agrawal 37431cf0b5 urls: Provide `email` as a GET parameter.
Since we want to use `accounts/new/send_confirm` to know how many
users actually register after visiting the register page, we
added it to Google Tag Manager, but GTM tracks every user
registration separately due <email> in the URL
making it harder to track.

To solve this, we want to pass <email> as a GET parameter which
can be easily filtered inside GTM using a RegEx and all the
registrations can be tracked as one.
2023-01-24 11:29:50 -08:00
Anders Kaseorg b0e569f07c ruff: Fix SIM102 nested `if` statements.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-01-23 11:18:36 -08:00
Alex Vandiver 04cf68b45e uploads: Serve S3 uploads directly from nginx.
When file uploads are stored in S3, this means that Zulip serves as a
302 to S3.  Because browsers do not cache redirects, this means that
no image contents can be cached -- and upon every page load or reload,
every recently-posted image must be re-fetched.  This incurs extra
load on the Zulip server, as well as potentially excessive bandwidth
usage from S3, and on the client's connection.

Switch to fetching the content from S3 in nginx, and serving the
content from nginx.  These have `Cache-control: private, immutable`
headers set on the response, allowing browsers to cache them locally.

Because nginx fetching from S3 can be slow, and requests for uploads
will generally be bunched around when a message containing them are
first posted, we instruct nginx to cache the contents locally.  This
is safe because uploaded file contents are immutable; access control
is still mediated by Django.  The nginx cache key is the URL without
query parameters, as those parameters include a time-limited signed
authentication parameter which lets nginx fetch the non-public file.

This adds a number of nginx-level configuration parameters to control
the caching which nginx performs, including the amount of in-memory
index for he cache, the maximum storage of the cache on disk, and how
long data is retained in the cache.  The currently-chosen figures are
reasonable for small to medium deployments.

The most notable effect of this change is in allowing browsers to
cache uploaded image content; however, while there will be many fewer
requests, it also has an improvement on request latency.  The
following tests were done with a non-AWS client in SFO, a server and
S3 storage in us-east-1, and with 100 requests after 10 requests of
warm-up (to fill the nginx cache).  The mean and standard deviation
are shown.

|                   | Redirect to S3      | Caching proxy, hot  | Caching proxy, cold |
| ----------------- | ------------------- | ------------------- | ------------------- |
| Time in Django    | 263.0 ms ±  28.3 ms | 258.0 ms ±  12.3 ms | 258.0 ms ±  12.3 ms |
| Small file (842b) | 586.1 ms ±  21.1 ms | 266.1 ms ±  67.4 ms | 288.6 ms ±  17.7 ms |
| Large file (660k) | 959.6 ms ± 137.9 ms | 609.5 ms ±  13.0 ms | 648.1 ms ±  43.2 ms |

The hot-cache performance is faster for both large and small files,
since it saves the client the time having to make a second request to
a separate host.  This performance improvement remains at least 100ms
even if the client is on the same coast as the server.

Cold nginx caches are only slightly slower than hot caches, because
VPC access to S3 endpoints is extremely fast (assuming it is in the
same region as the host), and nginx can pool connections to S3 and
reuse them.

However, all of the 648ms taken to serve a cold-cache large file is
occupied in nginx, as opposed to the only 263ms which was spent in
nginx when using redirects to S3.  This means that to overall spend
less time responding to uploaded-file requests in nginx, clients will
need to find files in their local cache, and skip making an
uploaded-file request, at least 60% of the time.  Modeling shows a
reduction in the number of client requests by about 70% - 80%.

The `Content-Disposition` header logic can now also be entirely shared
with the local-file codepath, as can the `url_only` path used by
mobile clients.  While we could provide the direct-to-S3 temporary
signed URL to mobile clients, we choose to provide the
served-from-Zulip signed URL, to better control caching headers on it,
and greater consistency.  In doing so, we adjust the salt used for the
URL; since these URLs are only valid for 60s, the effect of this salt
change is minimal.
2023-01-09 18:23:58 -05:00
Alex Vandiver ed6d62a9e7 avatars: Serve /user_avatars/ through Django, which offloads to nginx.
Moving `/user_avatars/` to being served partially through Django
removes the need for the `no_serve_uploads` nginx reconfiguring when
switching between S3 and local backends.  This is important because a
subsequent commit will move S3 attachments to being served through
nginx, which would make `no_serve_uploads` entirely nonsensical of a
name.

Serve the files through Django, with an offload for the actual image
response to an internal nginx route.  In development, serve the files
directly in Django.

We do _not_ mark the contents as immutable for caching purposes, since
the path for avatar images is hashed only by their user-id and a salt,
and as such are reused when a user's avatar is updated.
2023-01-09 18:23:58 -05:00
Alex Vandiver 7ad06473b6 uploads: Add LOCAL_AVATARS_DIR / LOCAL_FILES_DIR computed settings.
This avoids strewing "avatars" and "files" constants throughout.
2023-01-09 18:23:58 -05:00
Alex Vandiver cc9b028312 uploads: Set X-Accel-Redirect manually, without using django-sendfile2.
The `django-sendfile2` module unfortunately only supports a single
`SENDFILE` root path -- an invariant which subsequent commits need to
break.  Especially as Zulip only runs with a single webserver, and
thus sendfile backend, the functionality is simple to inline.

It is worth noting that the following headers from the initial Django
response are _preserved_, if present, and sent unmodified to the
client; all other headers are overridden by those supplied by the
internal redirect[^1]:
 - Content-Type
 - Content-Disposition
 - Accept-Ranges
 - Set-Cookie
 - Cache-Control
 - Expires

As such, we explicitly unset the Content-type header to allow nginx to
set it from the static file, but set Content-Disposition and
Cache-Control as we want them to be.

[^1]: https://www.nginx.com/resources/wiki/start/topics/examples/xsendfile/
2023-01-09 18:23:58 -05:00
Anders Kaseorg 17300f196c ruff: Fix ISC003 Explicitly concatenated string.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-01-04 16:25:07 -08:00
Anders Kaseorg 46cdcd3f33 ruff: Fix PIE790 Unnecessary `pass` statement.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-01-04 16:25:07 -08:00
Alex Vandiver 7613928e8a apps: Fix redirect from /apps -> https://zulip.com/apps/.
When this code was moved from being in zerver in 21a2fd482e, it kept
the `if ZILENCER_ENABLED` blocks.  Since ZILENCER and CORPORATE are
generally either both on or both off, the if statement became
mostly-unnecessary.

However, because tests cannot easily remove elements from
INSTALLED_APPS and re-determine URL resolution, we switch to checking
`if CORPORATE_ENABLED` as a guard, and leave these in-place.

The other side effect of this is that with e54ded49c4, most Zulip
deployments started to 404 requests for `/apps` instead of redirecting
them to `https://zulip.com/apps/` since they no longer had any path
configured for `/apps`.  Unfortunately, this URL is in widespread use
in the app (e.g. in links from the Welcome Bot), so we should ensure
that it does successfully redirect.

Add the `/apps` path to `zerver`, but only if not CORPORATE_ENABLED,
so the URLs do not overlap.
2022-12-30 17:47:16 -08:00
Mateusz Mandera f016ab4396 rate_limit: Add comments documenting how each rate limit rule works.
This avoid the need for spelunking the code to figure out the details of
how/when the rule is applied.
2022-12-15 12:56:51 -08:00
Aman Agrawal 51737b8905 registration: Add google analytics to realm creation process.
Track `create_realm` and `new_realm_send_confirm` using
google analytics.
This will help us track number of users who want to
create a new Zulip organization.
2022-12-08 12:10:28 -08:00
Anders Kaseorg 73c4da7974 ruff: Fix N818 exception name should be named with an Error suffix.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-11-17 16:52:00 -08:00
Anders Kaseorg 842a5bb54b message_flags: Allow updating flags by narrows and anchors.
Fixes #22893.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-11-16 21:13:44 -08:00
Anders Kaseorg 70dbeb197f middleware: Set the correct options on the django_language cookie.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-11-09 14:24:22 -08:00
Mateusz Mandera 34a0139c2b rate_limit: Add interface to override rate limiting rules. 2022-11-08 08:56:42 -08:00
Anders Kaseorg 55342efd33 scim: Upgrade django-scim2; remove request.user monkey patching.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-11-05 15:13:50 -07:00
Anders Kaseorg 9d2d6c8eb7 ruff: Fix M001 Unused `noqa` directive.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-11-03 12:10:15 -07:00
Anders Kaseorg adffad384c lint: Replace pycodestyle and pyflakes with ruff.
https://github.com/charliermarsh/ruff

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-11-03 12:10:15 -07:00
Mateusz Mandera 05913f5e3a saml: Extract logic for enabling wantMessagesSigned locally.
As explained in 158287f998,
wantMessagesSigned can't be enabled globally (as it'll break setups with
IdPs that sign SAMLResponse assertions) - but is needed for
LogoutRequests, and will be for LogoutResponses in the SP-initiated SLO
flow in future commits.

We extract a function with the necessary hacky logic for re-use in the
SP-initiated SLO implementation.
2022-11-02 21:58:46 -07:00
Lauryn Menard 1eda42f509 images: Create `/authentication_backends` directory for icons.
Creates `static/images/authentication_backends` directory for icons
of backend authentication methods, which are used on the log-in page.

And updates the example documentation in the API `/server_settings`
endpoint.
2022-10-31 14:36:54 -07:00
Anders Kaseorg b45484573e python: Use format string for logging str(obj).
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-10-10 08:32:29 -07:00
Anders Kaseorg fcd81a8473 python: Replace avoidable uses of __special__ attributes.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-10-10 08:32:29 -07:00
PIG208 df18bbbd48 requirements: Add django-stubs and configure plugin.
Note that django_stubs_ext is required to be placed within common.in
because we need the monkeypatched types in runtime; django-stubs
itself is for type checking only.

In the future, we would like to pin to a release instead of a git
revision, but several patches we've contributed upstream have not
appeared in a release yet.

We also remove the type annotation for RealmAuditLog.event_last_message_id
here instead of earlier because type checking fails otherwise.

Fixes #11560.
2022-10-05 16:15:56 -07:00
Anders Kaseorg 676d40d66b sentry: Initialize sentry in AppConfig ready hook.
This breaks an import cycle that prevented django-stubs from inferring
types for django.conf.settings.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-09-26 12:42:36 -07:00
Anders Kaseorg a5cf3b3975 config: Remove Django base class from ZulipSettingsError.
This breaks an import cycle that prevented django-stubs from inferring
types for django.conf.settings.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-09-26 12:42:36 -07:00
Anders Kaseorg 7908e2c3a5 computed_settings: Move logging callbacks from zerver.lib.logging_util.
This breaks an import cycle that prevented django-stubs from inferring
types for django.conf.settings.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-09-26 12:42:36 -07:00
Anders Kaseorg 9eefc290a9 template_loaders: Extract TwoFactorLoader to new module.
This breaks an import cycle that prevented django-stubs from inferring
types for django.conf.settings.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-09-26 12:42:36 -07:00
Anders Kaseorg 02be415122 settings_type: Extract new module for types used in settings.
This breaks an import cycle that prevented django-stubs from inferring
types for django.conf.settings.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-09-26 12:42:36 -07:00
Mateusz Mandera 16829bd84b auth: Fix up AzureAD Oauth2 docs and config.
This commit brings AzureAD config in line with other backends:
- SOCIAL_AUTH_AZUREAD_OAUTH2_SECRET gets fetched in computed_settings.py
  instead of default_settings, consistent with github/gitlab/etc.
- SOCIAL_AUTH_AZUREAD_OAUTH2_KEY gets fetched in default_settings via
  get_secret(..., development_only=True) like other social backends, to
  allow easier set up in dev environment, in the dev-secrets.conf file.
- The secret gets renamed from azure_oauth2_secret to
  social_auth_azuread_oauth2_secret to have a consistent naming scheme with
  other social backends and with the SOCIAL_AUTH_AZUREAD_OAUTH2_KEY
  name. This is backwards-incompatible.

The instructions for setting it up are updated to fit how this is
currently done in AzureAD.
2022-09-06 11:58:08 -07:00
Lauryn Menard 878b46e758 urls: Simplify process to add URL redirects for documentation pages.
Creates `zerver.lib.url_redirects.py` to record old and new URLs
for documentation pages that have been renamed/moved and need URL
redirects.

This file is then used by `zproject.urls.py` to redirect links and
by `zerver.test.test_urls.py` to test that all of the old URLs
return a success response with a common page header/text depending
on the type of redirect (help center, policy, or API).

Adds a section to contributor docs on writing documentation for
how to use this redirect system when renaming a help center or api
documentation page.

Fixes #21946. Fixes #17897.
2022-08-31 15:01:29 -07:00
Anders Kaseorg 81d0f5bdd9 computed_settings: Set STATICFILES_DIRS to an existing absolute path.
Django has always expected this, but Django 4.0 added a system check
that spews warnings in production.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-08-26 15:55:59 -07:00
Zixuan James Li c21ae3646a settings: Make SECRET_KEY mandatory.
Technically Django already makes SECRET_KEY mandatory by raising an
ImproperlyConfigured exception when it is not set. We use the
get_mandatory_secret helper here so that we have a narrower type.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-08-25 12:13:03 -07:00
Zixuan James Li c8d5959689 settings: Make AVATAR_SALT mandatory.
This also allows us to remove some assertions as we now know that
AVATAR_SALT will never be None.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-08-25 12:13:03 -07:00
Zixuan James Li 059d0e7be8 settings: Make SHARED_SECRET mandatory.
This implements get_mandatory_secret that ensures SHARED_SECRET is
set when we hit zerver.decorator.authenticate_notify. To avoid getting
ZulipSettingsError when setting up the secrets, we set an environment
variable DISABLE_MANDATORY_SECRET_CHECK to skip the check and default
its value to an empty string.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-08-25 12:13:03 -07:00
Aman Agrawal e54ded49c4 urls: Move URLs that don't belong on self-hosted servers to corporate URLs.
Use absolute URLs for these links in files which will be served
to self-hosted servers.
2022-08-22 15:53:43 -07:00
Aman Agrawal f1a1868de8 urls: Remove duplicate developer-community redirect.
Exactly same call exists below this with a comment.
2022-08-22 15:53:43 -07:00
Zixuan James Li eb88fee745 rest: Remove kwargs from rest_path.
The only caller that passes the kwargs argument is the avatar rest_path.
The application of kwargs can be rewritten with a wrapper.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-08-15 19:51:08 -07:00
Zixuan James Li 929311642b legacy_urls: Add type annotation to legacy_urls.
Mypy previously infers this to be `List[URLPattern]` which is
incompatible with other urls lists that we concatenate this with.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-08-12 17:08:04 -07:00
Zixuan James Li c9f54766c3 rate_limiter: Extract rate limit related functions.
This refactors rate limit related functions from `zerver.decorator` to
zerver.lib.rate_limiter.

We conditionally import `RemoteZulipServer`, `RequestNotes`, and
`RateLimitedRemoteZulipServer` to avoid circular dependency.

Most instances of importing these functions from `zerver.decorator` got
updated, with a few exceptions in `zerver.tests.test_decorators`, where
we do want to mock the rate limiting functions imported in
`zerver.decorator`. The same goes with the mocking example in the
"testing-with-django" documentation.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-08-12 16:51:55 -07:00
Dinesh 48d2783559 read_receipts: Add support for displaying read receipts.
Adds an API endpoint for accessing read receipts for other users, as
well as a modal UI for displaying that information.

Enables the previously merged privacy settings UI for managing whether
a user makes read receipts data available to other users.

Documentation is pending, and we'll likely want to link to the
documentation with help_settings_link once it is complete.

Fixes #3618.

Co-authored-by: Tim Abbott <tabbott@zulip.com>
2022-08-12 13:16:35 -07:00
Zixuan James Li 1d7abb3f5d settings: Extract unnecessary test specific settings.
APPLE_ID_TOKEN_GENERATOR_KEY and EXAMPLE_JWK are exclusively used in
test_auth_backends.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-08-11 13:45:28 -07:00
Zixuan James Li b0ffe45835 settings: Move TEST_WORKER_DIR to computed_settings.
mypy_django_plugin can only access settings from zproject.settings.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-08-11 13:45:28 -07:00
Zixuan James Li f994ba30b8 settings: Remove settings.BACKEND_DATABASE_TEMPLATE.
BACKEND_DATABASE_TEMPLATE was introduced in a507a47778.
This setting is only available for the test cases and it is not that
necessary to have it configurable.

We define it as a global variable in zerver.lib.test_fixtures.

This avoids requiring mypy_django_plugin to know the type of
settings.BACKEND_DATABASE_TEMPLATE for type checking purposes, given the fact
that settings.test_extra_settings is not available in production/development
setup.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-08-11 13:45:28 -07:00
Zixuan James Li 448253c009 settings: Make CAMO_KEY Optional.
This ensures that CAMO_KEY is always defined, so that mypy_django_plugin
will be able to identify its type.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-08-11 13:45:28 -07:00
Zixuan James Li 77eef5b1ad types: Add OIDCIdPConfigDict.
The presence of `auto_signup` in idp_settings_dict in the test case
test_social_auth_registration_auto_signup is incompatible with the
previous type annotation of SOCIAL_AUTH_OIDC_ENABLED_IDPS, where `bool`
is not allowed.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-08-03 15:03:15 -07:00
Anders Kaseorg b945aa3443 python: Use a real parser for email addresses.
Now that we can assume Python 3.6+, we can use the
email.headerregistry module to replace hacky manual email address
parsing.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-07-29 15:47:33 -07:00
Zixuan James Li a2bf76c610 email_backends: Fix type annotation of EmailLogBackEnd.
The base class has the methods to accept `Sequence` of `EmailMessage`.
Because our implementation in fact only supports `EmailMultiAlternatives`,
isinstance checks with assertions need to be added along with the
signature change.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-07-29 14:58:11 -07:00
Zixuan James Li 678c7fdd5f email_backends: Inline the construction of html_alternative.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-07-29 14:58:11 -07:00
Zixuan James Li a3efc777fa urls: Remove a legacy assertion.
This check was added for a legacy implementation of the GitHub integration in
bb6d189fa8,
which later got removed in
a73e8109b7.

No other webhook integration can now have a Falsy `url_object` attribute.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-07-26 18:00:24 -07:00
Zixuan James Li b5f03b9637 settings: Type JITSI_SERVER_URL as being Optional.
We fixed the case when handling `JITSI_SERVER_URL` being `None`, but the
type annotation didn't get updated along with the fix
2f9d4f5a96

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-07-26 18:00:24 -07:00
Zixuan James Li 036a90f375 settings: Add isinstance check before filtering.
This is a follow-up to https://github.com/typeddjango/django-stubs/pull/1038.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-07-26 18:00:24 -07:00
Zixuan James Li fe9ed2e69d settings: Make INVITATION_LINK_VALIDITY_MINUTES optional.
Type inference does not work when the default value of `REQ` is
non-optional while `ResultT` is optional. Mypy tries to unify
`json_validator` with `Validator[int]` in `invite_users_backend` instead
of the desired `Validator[Optional[int]]` because of the presence of the
default value `settings.INVITATION_LINK_VALIDITY_MINUTES`, which is
inferred to be an `int`. Mypy does not resort to a less specific type but
instead gives up early.

This issue applies to invite_users_backend and generate_multiuse_invite_backend
in zerver.views.invite.

There might be a way that we can add an overload to get around this, but
it's probably not worth the complexity until it comes up again more frequently.

We do in fact allow `invite_expires_in_minutes` to be `None` in places
like `do_invite_users`, `invite_users_backend`, etc, and we have
`settings.INVITATION_LINK_VALIDITY_MINUTES` as the default for them. So
it makes sense to allow having an optional value for this setting. And
since there isn't a way to independently set the value of this constant,
we move it to a different place.

TODO:

This is a temporary fix that should be refactored when the bug is fixed.

The encountered mypy issue: https://github.com/python/mypy/issues/13234

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-07-26 17:07:48 -07:00
Zixuan James Li 2e248cdbec settings: Add CUSTOM_HOME_NOT_LOGGED_IN for type narrowing.
django-stubs dynamically collects the type annotation for us from the
settings, acknowledging mypy that `HOME_NOT_LOGGED_IN` is an
`Optional[str]`. Type narrowing with assertions does not play well with
the default value of the decorator, so we define the same setting
variable with a different name as `CUSTOM_HOME_NOT_LOGGED_IN` to bypass
this restriction.

Filed python/mypy#13087 to track this issue.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-07-25 14:11:32 -07:00
Zixuan James Li 52be020d0c custom_profile: Apply ProfileDataElementUpdateDict.
We explicitly annotate variables or parameters with
`ProfileDataElementUpdateDict` as necessary.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-07-15 14:55:03 -07:00
Zixuan James Li 0952b024c1 scim: Add SCIMConfigDict.
This adds a `TypedDict` to provide better type annotation for the
configuration object for SCIM.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-07-15 14:00:56 -07:00
Zixuan James Li 75925fe059 middleware: Reorder middleware to clean up LogRequests hasattr checks.
Similar to the previous commit, we should access request.user only
after it has been initialized, rather than having awkward hasattr
checks.

With updates to the settings comments about LogRequests by tabbott.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-07-14 17:28:50 -07:00
Zixuan James Li 37a7d1fe7b middleware: Reorder middleware to avoid hasattr checks.
`request.user` gets set in Django's `AuthenticationMiddleware`, which
runs after our `HostDomainMiddleware`.

This makes `hasattr` checks necessary in any code path that uses the
`request.user` attribute. In this case, there are functions in
`context_processors` that get called in the middleware.

Since neither `CsrfMiddleware` nor `HostDomainMiddleware` are required
to run before `AuthenticationMiddleware`, moving it two slots up in
`computed_settings` is sufficient to avoid the `hasattr` checks.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-07-14 17:24:24 -07:00
Ganesh Pawar 135854db97 emoji: Use path converter for emoji name in URL.
If the emoji name contains forward slashes, the `str` converter
would treat it as a URL delimiter. Instead use the path converter, so
that forward slashes are included in the emoji name variable.

Fixes #22377
2022-07-14 14:29:40 -07:00
Anders Kaseorg 6b4474d4b5 computed_settings: Remove deprecated USE_L10N setting.
This is deprecated and defaults to True in Django 4.0.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-07-13 16:07:17 -07:00
Zixuan James Li 3a27a60b2c ldap: Remove unused isinstance check.
`realm_access_control` is always a `dict`.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-07-11 11:46:25 -07:00
Anders Kaseorg 2439914a50 settings: Add two_factor.plugins.phonenumber to INSTALLED_APPS.
I missed this in commit feff1d0411
(#22383) for upgrading to django-two-factor-auth 1.14.0.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-07-06 17:23:53 -07:00
Anders Kaseorg feff1d0411 requirements: Upgrade Python requirements.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-07-05 17:54:17 -07:00
Anders Kaseorg b4cf9ad777 db: Use cursor_factory psycopg2 option.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-07-05 17:54:17 -07:00
Zixuan James Li c968b299db typing: Tighten type annotation for settings.CACHES.
In zliencer.management.commands.populate_db, we assign the value of
settings.CACHES["default"] to `default_cache`.

django-stubs infers `settings.CACHES` to be `Dict[str, object]`. We make
the type specific enough so that we can access `default_cache` as a
dict.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-06-23 22:09:05 -07:00
Anders Kaseorg 9ee636e920 requirements: Upgrade Python requirements.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-06-01 16:01:54 -07:00
Tim Abbott 0255cbda03 settings: Allow customizing HOME_NOT_LOGGED_IN.
This can be useful if one wants a third-party authentication system to
apply first.
2022-06-01 15:37:31 -07:00
Zixuan James Li 4c6f2ae7be typing: Add assertions for authentication.
Signed-off-by: Zixuan James Li <359101898@qq.com>
2022-05-31 09:43:55 -07:00
Zixuan James Li 4a5043dd6e typing: Add none-checks for miscellaneous cases.
Signed-off-by: Zixuan James Li <359101898@qq.com>
2022-05-31 09:43:55 -07:00