Given that most of the use cases for realms-only code path would
really like to upload audit logs too, and the others would likely
produce a better user experience if they upoaded audit logs, we
should just have a single main code path here i.e.
'send_analytics_to_push_bouncer'.
We still only upload usage statistics according to documented
option, and only from the analytics cron job.
The error handling takes place in 'send_analytics_to_push_bouncer'
itself.
If there are remote realms attached to the remote server, adds
the forms to be able to update sponsorship requests, approve
sponsorship and update a discount.
Moves the section in support views for any current plan details
to a new template: `templates/analytics/current_plan_details.html`.
Also, updates the PlanData dataclass to have a boolean that checks
if the current plan tier is the self-hosted legacy plan.
Updates `get_plan_type_string` for RemoteZulipServer plan types and
capitalizes the strings used for Realm plan types.
Also changes the string for Realm.PLAN_TYPE_STANDARD_FREE to be
"Standard free" instead of "open source" as that is used for any
100% sponsored organization, which is not restricted to open-source
projects.
This commit adds two columns named 'Guest users' and
'Non guest users' to respresent count of such users.
We query 'RemoteRealmAuditLog' to get the data.
Creates a process_support_view_request method for BillingSession
to process the various support requests that relate to the billing
system.
Moves approve_realm_sponsorship, update_realm_sponsorship_status,
and attach_discount_to_realm to this new BillingSession method.
Adds a new abstract property to BillingSession to have a string
value, billing_entity_display_name, to use for support messages
sent when these requests are processed.
The "send_invoice" and "charge_automatically" strings used by stripe
for the `collection_method` are referred to both as the "billing
method" and "billing modality" in the billing code.
Because we send this as data to stripe as either `collection_method`
or `billing_modality`, renames any references that are any form of
"billing method".
This default setup will be more realistic, matching the ordinary
conditions for a modern server.
Especially needed as we add bouncer code that will expect to have
RemoteRealm entries for realm_uuid values for which it receives
requests.
These metadata are essentially all publicily available anyway, and
making uploading them unconditional will simplify some things.
The documentation is not quite accurate in that it claims the server
will upload some metadata that is not actually uploaded yet (but will
by soon). This seems harmless.
Adds a column to the remote servers activity page for a 7 day
count of forwarded mobile push notifications from
RemoteInstallationCount for each server, which will be `None`
if there is no data for the remote server with that ID.
Refactors get_page helper function so that the updates to the
query data for each row is done in the function that processes
the request.
Adds columns to the remote installation page for both the support
and analytics links.
Adds `analytics/views/remote_activity.py` to the files without
100% backend test coverage.
Moves and generalizes `switch_realm_from_standard_to_plus_plan`
in stripe.py to be a more general function for changing a
CustomerPlan to a new and valid tier, `do_change_plan_to_new_tier`.
Adds a helper function with the previous function name to be used
for the support view and management command for changing a realm
from the Standard plan tier to the Plus plan tier.
This commit moves a major portion of the 'update_plan`
view to a new shared 'BillingSession.do_update_plan' method.
This refactoring will help in minimizing duplicate code
while supporting both realm and remote_server customers.
This reduces the giant load spike at 5 minute past the hour, when all
remote servers currently attempt to submit their records.
We do not wish to slew over a full hour, because we want to ensure
that we do not hold the lock when the next hour's analytics runs. It
is also not necessary to have that much variation; 10 minutes is
picked as an arbitrary "long enough" time to spread requests over.
Due to a bug[^1] in Django 4.2, fixed in 4.2.6, queries using
`__isnull` added an unnecessary cast. This cast was _also_ used in
`WHERE` clauses for partial indexes. This means that partial indexes
created before Zulip was using Django 4.2 (i.e. before Zulip Server
7.0 or 2c20028aa4) will not be used when the server is using Django
4.2.0 through 4.2.5 -- and, conversely, that indexes created while
Zulip had those versions of Django (i.e. Zulip Server 7.0 through 7.4
or 7807bff526) will not be used later.
We re-create the indexes, to ensure that users that installed Zulip
after Zulip Server 7.0 / 2c20028aa4 and before Zulip Server 7.5 /
7807bff526 have indexes which can be used by current Django. This
is useless work for some installations, but most analytics tables are
not large enough for this to take significant time.
[^1]: https://code.djangoproject.com/ticket/34840
Moves the 'make_end_of_cycle_updates_if_needed' function to
the 'BillingSession' abstract class.
This refactoring will help in minimizing duplicate code while
supporting both realm and remote_server customers.
Since the function is called from our main daily billing cron job
as well, we have changed 'RealmBillingSession' to accept 'user=None'
(our convention for automated system jobs).
This is a CountStat for tracking how many mobile notifications the
server requested.
1. On a self-hosted server, that means requesting from the push bouncer.
2. On a server that's its own push bouncer, that's just the number
directly sent.
This number has room for inaccuracy due to incrementing by the number of
user devices on a self-hosted server, as it doesn't account for errors
that may occur in the GCM/APNs low-level sending codepaths on the bouncer.
Also tests that a server that's its own push bouncer correctly
increments its mobile_pushes_sent::day CountStat, by basing it on the
values returned from the send_apple/android_push_notification functions
which tell us the actual number of successfully sent notifications.
Since the return values of send_..._push_notification are now
used in those codepaths, we need to tweak our mocks in some unrelated
tests to set up some return value to avoid errors.
This calculates the largest amount of messages sent within a month for
the last 3 months. The query is targeted for the specific use-case in
this function - for finding the count for a specific server. For
calculating this in bulk for a large number of remote server an
adapted, bulk query will be needed - rather than running this one in a
loop, which would likely be very inefficient.
Moves `update_billing_method_of_current_plan` to the BillingSession
abstract class.
Adds a helper function for support views for the realm case:
`update_realm_billing_method`.
Moves `update_sponsorship_status` to BillingSession abstract class
as `update_customer_sponsorship_status`.
Updates the support views to have a helper for updating this on a
realm: `update_realm_sponsorship_status`.
Makes `approve_sponshorship` an abstract method in BillingSession
abstract base class and moves the implementation for realms to the
RealmBillingSession child class.
Adds `approve_realm_sponsorship` helper function that's used in
the support view and initiates the billing session.
This moves the logic for `attach_realm_discount`, which is used in
the support view, to be in the BillingSession class.
Updates the function name to be `attach_discount_to_customer` so
that the context is generalized vs realm specific.
Updates RealmBillingSession implementation to account for actions
that are initiated by a support admin user.
Also moves the helper function `get_discount_for_realm` that is
only used in support views to `corporate/lib/support.py`.
We are about to add support for having RemoteZulipServer here, which is
a zilencer, not zerver, object. So let's rename this argument to
something more appropriately general.
An assert is appropriate here to ensure that some future additions of
other frequencies don't make this if/else logic wrong without explicitly
failing.
This commit moves constants for system group names to a new
"SystemGroups" class so that we can use these group names
in multiple classes in models.py without worrying about the
order of defining them.
Moves two functions in corporate/lib/stripe.py that are used to
get data for the main installation activity analytics page to a
separate file: corporate/lib/analytics.py.
Also, updates these functions for the possibility of realm being
None for a Customer object.
Use analytics_realmcount to improve query runtime. There are two
known difference with previous query:
- Messages that are deleted after hourly stats are aggregated
are not decremented in new query.
- Messages sent since the hourly aggregation last ran are not counted.
The former name is kind of misleading - this function is for the remote
server to send analytics to the push bouncer. Under our usual
terminology, a "remote server" is a self-hosted Zulip server. So data is
sent FROM not TO a remote server.
As a preliminary step for including graphs of StreamCount data in our
analytics pages, add API support for fetching the chart data.
Care is taken to limit access to streams that the current user has
access to, which isn't necessary in similar views where the current
user is a server administrator by assumption.
Fixes part of #19653.
Co-authored-by: Tim Abbott <tabbott@zulip.com>
Moves the "Remote Zulip servers" tab in the "/activity" page for
an installation to a separate page, "/activity/remote".
Prototype for moving other tabs in "/activity" to separate pages.
Using `COUNT(*) FILTER (WHERE ...)` allows getting counts of different
subsets with only one giant join. This makes the query significantly
more performant.
_default_manager is the same as objects on most of our models. But
when a model class is stored in a variable, the type system doesn’t
know which model the variable is referring to, so it can’t know that
objects even exists (Django doesn’t add it if the user added a custom
manager of a different name). django-stubs used to incorrectly assume
it exists unconditionally, but it no longer does.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
Translators benefit from the extra information in the field names, and
need the reordering freedom that isn’t available with multiple
positional fields.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
This is primarily to prevent impersonation, such as `zulipteam`. We
only enable these protections for CORPORATE_ENABLED, since `zulip` is
a reasonable test name for self-hosters.
Alex Vandiver
Lauryn Menard
Prakhar Pratyush
Anders Kaseorg
Tim Abbott
Mateusz Mandera
Sahil Batra
Alya Abbott
Lakshay Mittal
Lauryn Menard
Steve Howell