Commit Graph

706 Commits

Author SHA1 Message Date
Lauryn Menard c996bb3c19 corporate: Move functions for realm installation data to new file.
Moves two functions in corporate/lib/stripe.py that are used to
get data for the main installation activity analytics page to a
separate file: corporate/lib/analytics.py.

Also, updates these functions for the possibility of realm being
None for a Customer object.
2023-10-30 10:08:45 -07:00
Lauryn Menard b19a7a8b8a analytics: Remove Durations tab and hourly columns. 2023-10-27 12:19:13 -07:00
Alya Abbott f8f741c4a2 analytics: Replace get_realm_day_counts with a faster query.
Use analytics_realmcount to improve query runtime. There are two
known difference with previous query:
- Messages that are deleted after hourly stats are aggregated
are not decremented in new query.
- Messages sent since the hourly aggregation last ran are not counted.
2023-10-26 12:54:56 -07:00
Mateusz Mandera 986f2fd962 send_analytics_to_remote_server: Rename to ..._to_push_bouncer.
The former name is kind of misleading - this function is for the remote
server to send analytics to the push bouncer. Under our usual
terminology, a "remote server" is a self-hosted Zulip server. So data is
sent FROM not TO a remote server.
2023-10-25 11:09:49 -07:00
Mateusz Mandera 6116679a95 remote_support: Remote redundant remote_installation_stats_link context.
This function was used in the html template in an early version of
PR #27033 but then ended up unused and didn't get cleaned up.
2023-10-17 18:05:27 -07:00
Lakshay Mittal d94bdb3900 stats: Support fetching StreamCount based on stream_id.
As a preliminary step for including graphs of StreamCount data in our
analytics pages, add API support for fetching the chart data.

Care is taken to limit access to streams that the current user has
access to, which isn't necessary in similar views where the current
user is a server administrator by assumption.

Fixes part of #19653.

Co-authored-by: Tim Abbott <tabbott@zulip.com>
2023-10-16 15:10:14 -07:00
Anders Kaseorg a50eb2e809 mypy: Enable new error explicit-override.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-10-12 12:28:41 -07:00
Anders Kaseorg 84a196641a test_support_views: Move incorrectly class-scoped code inside method.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-10-12 11:22:23 -07:00
Lauryn Menard da4b54d414 analytics: Remove out of date tabs from main activity page.
Removes the various 'ad_hoc_queries' in the main activity page
that no longer have relevant data or information.
2023-10-05 12:31:57 -07:00
Lauryn Menard 40e6455579 analytics: Move integrations by client to separate activity page.
Moves the tab in the main installation activity page to be a
separate page: "/activity/integrations".
2023-10-05 12:31:57 -07:00
Lauryn Menard 0f009c439d analytics: Move remote server view to separate file.
Moves the view for "/activity/remove" from the main installation
activity view to a separate file.
2023-10-05 12:31:57 -07:00
Mateusz Mandera f71e2c8247 support: Add basic support endpoint for remote servers. 2023-10-04 16:55:47 -07:00
Lauryn Menard afc1d2a409 activity: Add separate page for remote server information/table.
Moves the "Remote Zulip servers" tab in the "/activity" page for
an installation to a separate page, "/activity/remote".

Prototype for moving other tabs in "/activity" to separate pages.
2023-10-04 12:48:39 -07:00
Alex Vandiver 71e297efb4 realm: Differentiate reserved realms from in-use realms.
Fixes: #23896.
2023-09-25 12:48:14 -07:00
Anders Kaseorg 2665a3ce2b python: Elide unnecessary list wrappers.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-09-13 12:41:23 -07:00
Anders Kaseorg 6988622fe8 ruff: Enable B023 Function definition does not bind loop variable.
Python’s loop scoping is misdesigned, resulting in a very common
gotcha for functions that close over loop variables [1].  The general
problem is so bad that even the Go developers plan to break
compatibility in order to fix the same design mistake in their
language [2].

Enable the Ruff rule function-uses-loop-variable (B023) [3], which
conservatively prohibits functions from binding loop variables at all.

[1] https://docs.python-guide.org/writing/gotchas/#late-binding-closures
[2] https://go.dev/s/loopvar-design
[3] https://beta.ruff.rs/docs/rules/function-uses-loop-variable/

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-09-11 18:03:45 -07:00
Alex Vandiver b94402152d models: Always search Messages with a realm_id or id limit.
Unless there is a limit on `id`, always provide a `realm_id` limit as
well.  We also notate which index is expected to be used in each
query.
2023-09-11 15:00:37 -07:00
Alex Vandiver 4d72f81257 analytics: Only join through messages once.
Using `COUNT(*) FILTER (WHERE ...)` allows getting counts of different
subsets with only one giant join.  This makes the query significantly
more performant.
2023-09-11 15:00:37 -07:00
Anders Kaseorg 0ce6dcb905 mypy: Upgrade mypy from 1.4.1 to 1.5.1.
_default_manager is the same as objects on most of our models. But
when a model class is stored in a variable, the type system doesn’t
know which model the variable is referring to, so it can’t know that
objects even exists (Django doesn’t add it if the user added a custom
manager of a different name). django-stubs used to incorrectly assume
it exists unconditionally, but it no longer does.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-09-07 17:51:42 -07:00
Anders Kaseorg 81bd63cb46 ruff: Fix PIE808 Unnecessary `start` argument in `range`.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-09-01 14:57:01 -07:00
Anders Kaseorg e32366638a requirements: Upgrade Python requirements.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-08-17 17:05:34 -07:00
Anders Kaseorg 710d1f7f51 analytics: Do not reseed the global random generator.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-08-15 17:57:16 -07:00
Steve Howell 751b8b5bb5 tests: Flush per-request caches automatically for query counts. 2023-08-11 11:09:34 -07:00
Anders Kaseorg ca40e60469 ruff: Enable PERF rules.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-08-07 17:23:55 -07:00
Anders Kaseorg 562a79ab76 ruff: Fix PERF401 Use a list comprehension to create a transformed list.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-08-07 17:23:55 -07:00
Anders Kaseorg c2c96eb0cf python: Annotate type aliases with TypeAlias.
This is not strictly necessary but it’s clearer and improves mypy’s
error messages.

https://docs.python.org/3/library/typing.html#typing.TypeAlias
https://mypy.readthedocs.io/en/stable/kinds_of_types.html#type-aliases

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-08-07 10:02:49 -07:00
Anders Kaseorg 2ae285af7c ruff: Fix PLR1714 Consider merging multiple comparisons.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-07-23 15:21:33 -07:00
Anders Kaseorg 3b09197fdf ruff: Fix RUF015 Prefer `next(...)` over single element slice.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-07-23 15:20:53 -07:00
Anders Kaseorg 143baa4243 python: Convert translated positional {} fields to {named} fields.
Translators benefit from the extra information in the field names, and
need the reordering freedom that isn’t available with multiple
positional fields.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-07-18 15:19:07 -07:00
Anders Kaseorg 7e707270f0 models: Convert deprecated index_together option to indexes.
index_together is slated for removal in Django 5.1:
https://docs.djangoproject.com/en/4.2/internals/deprecation/#deprecation-removed-in-5-1

We set the optional index names to match the previously generated
index names to avoid adding new migrations.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-07-12 07:12:43 -07:00
Alex Vandiver 6c3969f893 name_restrictions: Reject anything with zulip or kandra in it.
This is primarily to prevent impersonation, such as `zulipteam`.  We
only enable these protections for CORPORATE_ENABLED, since `zulip` is
a reasonable test name for self-hosters.
2023-06-23 10:45:40 -07:00
Lauryn Menard 47d0bb6b7d analytics: Replace "private message" with "direct message". 2023-06-19 13:40:08 -07:00
Mateusz Mandera 2b7877bcb4 support: Make user search by email case-insensitive.
It's pretty troublesome to fail to find a user in our database because
they used a different capitalization when signing up.
2023-05-16 12:38:12 -07:00
Mateusz Mandera 632e856240 support: Add option to delete user from /activity/support. 2023-05-16 12:38:12 -07:00
Alex Vandiver 505eec4bac analytics: Create a RealmAuditLog entry when subscribing test users. 2023-05-15 16:09:44 -07:00
Mateusz Mandera 24d9d3d90c support: Add option to upgrade an org to Plus plan on /activity/support.
This is so that we don't need ops to run the
manage.py switch_realm_from_standard_to_plus_plan command on the server
to handle these upgrades.
2023-04-10 17:32:45 -07:00
Mateusz Mandera 1a15449e16 support: Rename "downgrade plan" to "modify plan".
Just a naming change to facilitate adding more options to that <select>
- mainly, an option to upgrade the plan to Plus.
2023-04-10 17:32:45 -07:00
Sahil Batra f684d36710 test_classes: Add submit_realm_creation_form helper.
This commit adds a new helper submit_realm_creation_form,
similar to existing submit_reg_form_for_user, to avoid
duplicate code for creating realms in tests.
2023-03-27 15:44:42 -07:00
Sahil Batra 54771cfe94 registration: Create PreregistrationRealm object when creating realm.
This commit adds code to create PreregistrationRealm object when
creating realm and set it to the content_object field of
Confirmation object.
2023-03-27 15:44:42 -07:00
Alex Vandiver 0b80397cfd web: Save a 301 redirect from /support to /support/. 2023-03-24 14:51:01 -07:00
Anders Kaseorg afa218fa2a semgrep: Detect some unsafe uses of markupsafe.Markup.
Use the built-in HTML escaping of Markup("…{var}…").format(), in order
to allow Semgrep to detect mistakes like Markup("…{var}…".format())
and Markup(f"…{var}…").

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-03-22 11:23:27 -07:00
Alex Vandiver 330141f55d invites: Switch new LIMITED-plan heuristic to enforcing. 2023-03-16 11:41:49 -07:00
Anders Kaseorg 2d9b2a2a05 models: Remove type prefixes from __str__ values.
The Django convention is for __repr__ to include the type and __str__
to omit it.  In fact its default __repr__ implementation for models
automatically adds a type prefix to __str__, which has resulted in the
type being duplicated:

    >>> UserProfile.objects.first()
    <UserProfile: <UserProfile: emailgateway@zulip.com <Realm: zulipinternal 1>>>

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-03-08 22:56:55 -08:00
Alex Vandiver 36da7783ce invites: _max_invites is currently never None.
dc1eeef30a made the column nullable, with the meaning for null of
"use the current `settings.INVITES_DEFAULT_REALM_DAILY_MAX`."
However, 8a95526ced switched to calling `do_change_plan_type` during
realm creation, which sets `realm.max_invites` based on the plan type,
thus ensuring that no new realms have their `_max_invites` set to
null.

Check `max_invites` instead of `_max_invites`.  This requires test
adjustments for the fact that `apply_invite_realm_heuristics` is now
run.
2023-03-07 15:04:39 -08:00
Anders Kaseorg d3efd4c095 python: Import F, Q, QuerySet from their canonical module.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-03-05 14:46:28 -08:00
Anders Kaseorg 0628c3cac8 migrations: Import BaseDatabaseSchemaEditor from its canonical module.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-03-05 14:46:28 -08:00
Alex Vandiver 04e7621668 upload: Rename upload_message_image_from_request.
The table is named Attachment, and not all of them are images.
2023-03-02 16:36:19 -08:00
Sahil Batra 3950a8e19d test_helpers: Rename reset_emails_in_zulip_realm.
This commit renames reset_emails_in_zulip_realm function to
reset_email_visibility_to_everyone_in_zulip_realm which makes
it more clear to understand what the function actually does.

This commit also adds a comment explaining what this function
does.
2023-03-01 12:17:11 -08:00
Lauryn Menard a0fd7b2afc private-messages: Update translated backend strings use "direct message".
Updates user-facing translated strings containing "private message" on
the backend to use "direct message" instead.
2023-02-24 11:47:26 -08:00
Sahil Batra 0ed5f76063 settings: Add backend code for using user email_address_visibility setting.
This commits update the code to use user-level email_address_visibility
setting instead of realm-level to set or update the value of UserProfile.email
field and to send the emails to clients.

Major changes are -

- UserProfile.email field is set while creating the user according to
RealmUserDefault.email_address_visbility.

- UserProfile.email field is updated according to change in the setting.

- 'email_address_visibility' is added to person objects in user add event
and in avatar change event.

- client_gravatar can be different for different users when computing
avatar_url for messages and user objects since email available to clients
is dependent on user-level setting.

- For bots, email_address_visibility is set to EVERYONE while creating
them irrespective of realm-default value.

- Test changes are basically setting user-level setting instead of realm
setting and modifying the checks accordingly.
2023-02-10 17:35:49 -08:00