Commit Graph

691 Commits

Author SHA1 Message Date
Anders Kaseorg 6988622fe8 ruff: Enable B023 Function definition does not bind loop variable.
Python’s loop scoping is misdesigned, resulting in a very common
gotcha for functions that close over loop variables [1].  The general
problem is so bad that even the Go developers plan to break
compatibility in order to fix the same design mistake in their
language [2].

Enable the Ruff rule function-uses-loop-variable (B023) [3], which
conservatively prohibits functions from binding loop variables at all.

[1] https://docs.python-guide.org/writing/gotchas/#late-binding-closures
[2] https://go.dev/s/loopvar-design
[3] https://beta.ruff.rs/docs/rules/function-uses-loop-variable/

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-09-11 18:03:45 -07:00
Alex Vandiver b94402152d models: Always search Messages with a realm_id or id limit.
Unless there is a limit on `id`, always provide a `realm_id` limit as
well.  We also notate which index is expected to be used in each
query.
2023-09-11 15:00:37 -07:00
Alex Vandiver 4d72f81257 analytics: Only join through messages once.
Using `COUNT(*) FILTER (WHERE ...)` allows getting counts of different
subsets with only one giant join.  This makes the query significantly
more performant.
2023-09-11 15:00:37 -07:00
Anders Kaseorg 0ce6dcb905 mypy: Upgrade mypy from 1.4.1 to 1.5.1.
_default_manager is the same as objects on most of our models. But
when a model class is stored in a variable, the type system doesn’t
know which model the variable is referring to, so it can’t know that
objects even exists (Django doesn’t add it if the user added a custom
manager of a different name). django-stubs used to incorrectly assume
it exists unconditionally, but it no longer does.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-09-07 17:51:42 -07:00
Anders Kaseorg 81bd63cb46 ruff: Fix PIE808 Unnecessary `start` argument in `range`.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-09-01 14:57:01 -07:00
Anders Kaseorg e32366638a requirements: Upgrade Python requirements.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-08-17 17:05:34 -07:00
Anders Kaseorg 710d1f7f51 analytics: Do not reseed the global random generator.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-08-15 17:57:16 -07:00
Steve Howell 751b8b5bb5 tests: Flush per-request caches automatically for query counts. 2023-08-11 11:09:34 -07:00
Anders Kaseorg ca40e60469 ruff: Enable PERF rules.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-08-07 17:23:55 -07:00
Anders Kaseorg 562a79ab76 ruff: Fix PERF401 Use a list comprehension to create a transformed list.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-08-07 17:23:55 -07:00
Anders Kaseorg c2c96eb0cf python: Annotate type aliases with TypeAlias.
This is not strictly necessary but it’s clearer and improves mypy’s
error messages.

https://docs.python.org/3/library/typing.html#typing.TypeAlias
https://mypy.readthedocs.io/en/stable/kinds_of_types.html#type-aliases

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-08-07 10:02:49 -07:00
Anders Kaseorg 2ae285af7c ruff: Fix PLR1714 Consider merging multiple comparisons.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-07-23 15:21:33 -07:00
Anders Kaseorg 3b09197fdf ruff: Fix RUF015 Prefer `next(...)` over single element slice.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-07-23 15:20:53 -07:00
Anders Kaseorg 143baa4243 python: Convert translated positional {} fields to {named} fields.
Translators benefit from the extra information in the field names, and
need the reordering freedom that isn’t available with multiple
positional fields.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-07-18 15:19:07 -07:00
Anders Kaseorg 7e707270f0 models: Convert deprecated index_together option to indexes.
index_together is slated for removal in Django 5.1:
https://docs.djangoproject.com/en/4.2/internals/deprecation/#deprecation-removed-in-5-1

We set the optional index names to match the previously generated
index names to avoid adding new migrations.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-07-12 07:12:43 -07:00
Alex Vandiver 6c3969f893 name_restrictions: Reject anything with zulip or kandra in it.
This is primarily to prevent impersonation, such as `zulipteam`.  We
only enable these protections for CORPORATE_ENABLED, since `zulip` is
a reasonable test name for self-hosters.
2023-06-23 10:45:40 -07:00
Lauryn Menard 47d0bb6b7d analytics: Replace "private message" with "direct message". 2023-06-19 13:40:08 -07:00
Mateusz Mandera 2b7877bcb4 support: Make user search by email case-insensitive.
It's pretty troublesome to fail to find a user in our database because
they used a different capitalization when signing up.
2023-05-16 12:38:12 -07:00
Mateusz Mandera 632e856240 support: Add option to delete user from /activity/support. 2023-05-16 12:38:12 -07:00
Alex Vandiver 505eec4bac analytics: Create a RealmAuditLog entry when subscribing test users. 2023-05-15 16:09:44 -07:00
Mateusz Mandera 24d9d3d90c support: Add option to upgrade an org to Plus plan on /activity/support.
This is so that we don't need ops to run the
manage.py switch_realm_from_standard_to_plus_plan command on the server
to handle these upgrades.
2023-04-10 17:32:45 -07:00
Mateusz Mandera 1a15449e16 support: Rename "downgrade plan" to "modify plan".
Just a naming change to facilitate adding more options to that <select>
- mainly, an option to upgrade the plan to Plus.
2023-04-10 17:32:45 -07:00
Sahil Batra f684d36710 test_classes: Add submit_realm_creation_form helper.
This commit adds a new helper submit_realm_creation_form,
similar to existing submit_reg_form_for_user, to avoid
duplicate code for creating realms in tests.
2023-03-27 15:44:42 -07:00
Sahil Batra 54771cfe94 registration: Create PreregistrationRealm object when creating realm.
This commit adds code to create PreregistrationRealm object when
creating realm and set it to the content_object field of
Confirmation object.
2023-03-27 15:44:42 -07:00
Alex Vandiver 0b80397cfd web: Save a 301 redirect from /support to /support/. 2023-03-24 14:51:01 -07:00
Anders Kaseorg afa218fa2a semgrep: Detect some unsafe uses of markupsafe.Markup.
Use the built-in HTML escaping of Markup("…{var}…").format(), in order
to allow Semgrep to detect mistakes like Markup("…{var}…".format())
and Markup(f"…{var}…").

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-03-22 11:23:27 -07:00
Alex Vandiver 330141f55d invites: Switch new LIMITED-plan heuristic to enforcing. 2023-03-16 11:41:49 -07:00
Anders Kaseorg 2d9b2a2a05 models: Remove type prefixes from __str__ values.
The Django convention is for __repr__ to include the type and __str__
to omit it.  In fact its default __repr__ implementation for models
automatically adds a type prefix to __str__, which has resulted in the
type being duplicated:

    >>> UserProfile.objects.first()
    <UserProfile: <UserProfile: emailgateway@zulip.com <Realm: zulipinternal 1>>>

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-03-08 22:56:55 -08:00
Alex Vandiver 36da7783ce invites: _max_invites is currently never None.
dc1eeef30a made the column nullable, with the meaning for null of
"use the current `settings.INVITES_DEFAULT_REALM_DAILY_MAX`."
However, 8a95526ced switched to calling `do_change_plan_type` during
realm creation, which sets `realm.max_invites` based on the plan type,
thus ensuring that no new realms have their `_max_invites` set to
null.

Check `max_invites` instead of `_max_invites`.  This requires test
adjustments for the fact that `apply_invite_realm_heuristics` is now
run.
2023-03-07 15:04:39 -08:00
Anders Kaseorg d3efd4c095 python: Import F, Q, QuerySet from their canonical module.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-03-05 14:46:28 -08:00
Anders Kaseorg 0628c3cac8 migrations: Import BaseDatabaseSchemaEditor from its canonical module.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-03-05 14:46:28 -08:00
Alex Vandiver 04e7621668 upload: Rename upload_message_image_from_request.
The table is named Attachment, and not all of them are images.
2023-03-02 16:36:19 -08:00
Sahil Batra 3950a8e19d test_helpers: Rename reset_emails_in_zulip_realm.
This commit renames reset_emails_in_zulip_realm function to
reset_email_visibility_to_everyone_in_zulip_realm which makes
it more clear to understand what the function actually does.

This commit also adds a comment explaining what this function
does.
2023-03-01 12:17:11 -08:00
Lauryn Menard a0fd7b2afc private-messages: Update translated backend strings use "direct message".
Updates user-facing translated strings containing "private message" on
the backend to use "direct message" instead.
2023-02-24 11:47:26 -08:00
Sahil Batra 0ed5f76063 settings: Add backend code for using user email_address_visibility setting.
This commits update the code to use user-level email_address_visibility
setting instead of realm-level to set or update the value of UserProfile.email
field and to send the emails to clients.

Major changes are -

- UserProfile.email field is set while creating the user according to
RealmUserDefault.email_address_visbility.

- UserProfile.email field is updated according to change in the setting.

- 'email_address_visibility' is added to person objects in user add event
and in avatar change event.

- client_gravatar can be different for different users when computing
avatar_url for messages and user objects since email available to clients
is dependent on user-level setting.

- For bots, email_address_visibility is set to EVERYONE while creating
them irrespective of realm-default value.

- Test changes are basically setting user-level setting instead of realm
setting and modifying the checks accordingly.
2023-02-10 17:35:49 -08:00
Anders Kaseorg df001db1a9 black: Reformat with Black 23.
Black 23 enforces some slightly more specific rules about empty line
counts and redundant parenthesis removal, but the result is still
compatible with Black 22.

(This does not actually upgrade our Python environment to Black 23
yet.)

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-02-02 10:40:13 -08:00
Anders Kaseorg ff1971f5ad ruff: Fix SIM105 Use `contextlib.suppress` instead of try-except-pass.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-01-23 11:18:36 -08:00
Anders Kaseorg b0e569f07c ruff: Fix SIM102 nested `if` statements.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-01-23 11:18:36 -08:00
Anders Kaseorg 3025d9a63a ruff: Fix Q002 Single quote docstring found.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-01-04 16:25:07 -08:00
Alex Vandiver 0e10ccc137 analytics: Pass the right value for realm. 2022-12-08 12:42:41 -08:00
Lauryn Menard b686e6ce9e analytics-realm-activity: Add link to realm's stats page.
Adds the  pie chart icon stats page link to the realm_activity
page header.
2022-12-02 17:30:30 -08:00
Lauryn Menard c05c25e742 analytics-activity: Fix UTC timezone and adjust header type.
Fixes an error with how the `utctime` was being generated and makes
the header on the summary table the same type as the other tabs.
2022-12-02 17:30:30 -08:00
Lauryn Menard c75e5c8ba6 analytics-activity: Add links for realm's Zulip and support page. 2022-12-02 17:30:30 -08:00
Lauryn Menard 90d0531e20 analytics-activity: Remove link with realm owners emails.
Note that a link to copy these emails to the clipboard is still
availabe on the realm's support page.
2022-12-02 17:30:30 -08:00
Anders Kaseorg 73c4da7974 ruff: Fix N818 exception name should be named with an Error suffix.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-11-17 16:52:00 -08:00
Anders Kaseorg 924d530292 ruff: Fix N813 camelcase imported as lowercase.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-11-16 09:29:11 -08:00
Lauryn Menard a3f6220fe4 analytics: Add summary statistic for guest users in realm.
Adds the count of users with the role of guest to the stats view
`page_params` via a database query. This information is then added
to the summary statistics section of the analytics page after being
formatted by `stats.js`.

Creates Bassanio as a guest user in the database for the analytics
realm.

Fixes #20162.
2022-10-17 11:53:59 -07:00
Zixuan James Li 46329a2710 test_classes: Create a dedicate helper for query count check.
This adds a helper based on testing patterns of using the "queries_captured"
context manager with "assert_length" to check the number of queries
executed for preventing performance regression.

It explains the rationale of checking the query count through an
"AssertionError" and prints the queries captured as assert_length does,
but with a format optimized for displaying the queries in a more
readable manner.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-10-17 11:32:52 -07:00
Mateusz Mandera 00b3546c9f models: Add denormalized .realm column to Message.
This commit adds the OPTIONAL .realm attribute to Message
(and ArchivedMessage), with the server changes for making new Messages
have this set. Old Messages still have to be migrated to backfill this,
before it can be non-nullable.

Appropriate test changes to correctly set .realm for Messages the tests
manually create are included here as well.
2022-10-07 10:09:38 -07:00
Lauryn Menard df7a1cec93 analytics: Add summary statistic for upload/storage space in use.
Adds the realm's used storage space for attachments to the stats
view `page_params`. This information is then added to the summary
statistics section of the analytics page after being formatted by
`stats.js`.

Uses the emoji test image to create an `Attachment` in the database
for the analytics realm. Even though it doesn't create a message
to claim the attachment, it still is sent as storage space used
data for the analytics `/stats/` page.
2022-10-07 10:08:09 -07:00