When profiling the database queries for the remote support view,
getting the user counts for remote realms was identifed as an
expensive query. Adds an index on RemoteRealmAuditLog to improve
this relatively common query for remote billing information.
We no longer want to migrate Legacy plans from server to realms, since
Legacy plans are not really a thing in the original sense anymore, since
February 15th.
Now they're just a tool to give temporary extensions of access to the
push notification service for users, when needed. And as such, it makes
no sense to migrate like that.
The remaining code in this function is for migrating (any) plan from the
server object to the realm object, if the server has just a single
realm.
This commit renames the realm-level setting 'notifications_stream'
to 'new_stream_announcements_stream'.
The new name reflects better what the setting does.
The logic in the case where there's only one realm and the function
tries to migrate the server's plan to it, had two main unhandled edge
cases that would throw exceptions:
1.
```
remote_realm = RemoteRealm.objects.get(
uuid=realm_uuids[0], plan_type=RemoteRealm.PLAN_TYPE_SELF_MANAGED
)
```
This could throw an exception if the RemoteRealm exists, but has an
active e.g. Legacy plan. Then there'd be no object matching the
plan_type in the query, raising RemoteRealm.DoesNotExist.
2. If the RemoteRealm had e.g. a Legacy plan in the past, that's now
expired, then it'd have a Customer object. Meaning that the attempt
to move the server's customer to the realm:
`server_plan.customer = remote_realm_customer`
would trigger an IntegrityError since a RemoteRealm can't have two
Customer objects.
In simple cases the situation in (2) can still be easily migrated, by
moving the plan from the server's customer to the realm's customer.
Just like deactivated realms should be excluded, so should locally
deleted realms.
In particular, failure to exclude locally deleted realms breaks
handle_customer_migration_from_server_to_realms.
When profiling the database query in `remote_activity.py`,
push_forwarded_count was identified as an expensive part of
the overall work. Adds an index on RemoteInstallationCount
so this is more efficient.
The presence of the name "François" in the random first name list led
to email addresses of the form `françois1234@zulip.com` which are
invalid.
Strip invalid ASCII from the email addresses generated in populate_db,
and validate them before tossing them verbatim into the database.
Pulls out shared code in get_remote_realm_guest_and_non_guest_count
and get_remote_server_guest_and_non_guest_count for generating the
RemoteCustomerUserCount so that it can be used in logic for getting
these counts for all remote servers and remote realms on an
installation.
This is preparatory work towards adding a Topic model.
We plan to use the local variable name as 'topic' for
the Topic model objects.
Currently, we use *topic as the local variable name for
topic names.
We rename local variables of the form *topic to *topic_name
so that we don't need to think about type collisions in
individual code paths where we might want to talk about both
Topic objects and strings for the topic name.
This was a bug from 4715a058b0 where this
was just incorrectly called. get_realms_info_for_push_bouncer() is a
function meant to be called on a self-hosted server - and this
handle_... call happens on the bouncer. Therefore this returns all
zulipchat realms in product.
With the way, handle_... is being called right now, there's no reason
for it to have an argument for passing a list of realms. It should just
fetch the relevant RemoteRealm entries by itself, given the server arg.
Requests to these endpoint are about a specified user, and therefore
also have a notion of the RemoteRealm for these requests. Until now
these endpoints weren't getting the realm_uuid value, because it wasn't
used - but now it is needed for updating .last_request_datetime on the
RemoteRealm.
For the RemoteRealm case, we can only set this in endpoints where the
remote server sends us the realm_uuid. So we're missing that for the
endpoints:
- remotes/push/unregister and remotes/push/unregister/all
- remotes/push/test_notification
This should be added in a follow-up commit.
Earlier, the 'handle_customer_migration_from_server_to_realms'
function was called during the send analytics step.
It resulted in an error for customers having multiple Zulip servers,
one for testing and the others for not-testing, sharing a
push bouncer registration.
The migration step when run in a test instance caused customers to
have their legacy plan migrated to a test realm, resulting in them
losing their legacy plan.
This commit moves the migration step to run during plan management
login step. This reduces the chances of losing legacy
plan as we expect them to only verify that 8.0 upgrade works and
not bother trying to login to plan management from their test instance.
This protects us from incorrectly handling situations where someone
tested and upgrade to 8.0 for a backup on a separate hostname, and
left the test system live while upgrading the main system, in a way
that results in duplicate RemoteRealm objects that are all marked as
locally deleted.
Further word is required to figure out how to avoid the original
duplication problem.
It seems most correct to answer the question about whether push
notifications are working specifically for the exact set of realms
that the server self-reported to us in fact exist.
Sending data on any additional realms that were not referenced in the
request (if that's somehow possible without them being locally
deleted) is likely to only be confusing.
And the client should reasonably be able to expect to get a response
covering exactly the realms it told us about.
Old RemotePushDeviceTokens were created without this attribute. But when
processing a notification, if we have remote_realm, we can take the
opportunity to to set this for all the registrations for this user.
This moves the function which computes can_push and
expected_end_timestamp outside RemoteRealmBillingSession
because we might use this function for RemoteZulipServer
as well and also renames it.
Also avoid prompting for full name time more than once.
Adds TOS version field to Remote server user.
Co-authored-by: Karl Stolley <karl@zulip.com>
Co-authored-by: Aman Agrawal <amanagr@zulip.com>
We need to update 'last_audit_log_update' before calling the
'sync_license_ledger_if_needed' method to avoid 'MissingDataError'
due to 'has_stale_audit_log' being True.
Also, we made the code block that creates audit logs,
updates 'last_audit_log_update', and syncs LicenseLedger in
an atomic operation.
This helps to rely on 'last_audit_log_update' to assume
'RemoteRealmAuditLog' and 'LicenseLedger' are up-to-date.
- The server sends the list of registrations it believes to have with
the bouncer.
- The bouncer includes in the response the registrations that it doesn't
actually have and therefore the server should delete.
When a self-hosted Zulip server does a data export and then import
process into a different hosting environment (i.e. not sharing the
RemoteZulipServer with the original, we'll have various things that
fail where we look up the RemoteRealm by UUID and find it but the
RemoteZulipServer it is associated with is the wrong one.
Right now, we ask user to contact support via an error page but
might develop UI to help user do the migration directly.
The way the flow goes now is this:
1. The user initiaties login via "Billing" in the gear menu.
2. That takes them to `/self-hosted-billing/` (possibly with a
`next_page` param if we use that for some gear menu options).
3. The server queries the bouncer to give the user a link with a signed
access token.
4. The user is redirected to that link (on `selfhosting.zulipchat.com`).
Now we have two cases, either the user is logging in for the first time
and already did in the past.
If this is the first time, we have:
5. The user is asked to fill in their email in a form that's shown,
pre-filled with the value provided inside the signed access token.
They POST this to the next endpoint.
6. The next endpoint sends a confirmation email to that address and asks
the user to go check their email.
7. The user clicks the link in their email is taken to the
from_confirmation endpoint.
8. Their initial RemoteBillingUser is created, a new signed link like in
(3) is generated and they're transparently taken back to (4),
where now that they have a RemoteBillingUser, they're handled
just like a user who already logged in before:
If the user already logged in before, they go straight here:
9. "Confirm login" page - they're shown their information (email and
full_name), can update
their full name in the form if they want. They also accept ToS here
if necessary. They POST this form back to
the endpoint and finally have a logged in session.
10. They're redirected to billing (or `next_page`) now that they have
access.
For the last form (with Full Name and ToS consent field), this pretty
shamelessly re-uses and directly renders the
corporate/remote_realm_billing_finalize_login_confirmation.html
template. That's probably good in terms of re-use, but calls for a
clean-up commit that will generalize the name of this template and the
classes/ids in the HTML.
When a remote server uploads statistics, we update the
LicenseLedger using the audit logs uploaded.
We iterate over the RemoteRealmAuditlog data for the concerned
realm starting from the event_time of the last LicenseLedger
created for that customer and update the ledger based on each event.
If the RemoteRealmAuditLog has stale data, it means the server
stopped or never uploaded data. We raise MissingDataError in such
cases when a user action led to calculating licenses count from
stale data.
We add a 'get_remote_realm_guest_and_non_guest_count'
function that queries 'RemoteRealmAuditLog' to get
the guest and non_guest count for that remote_realm.
This function is used in 'RemoteRealmBillingSession'
to calculate the current count of billed licenses.
In cloud:
Sponsored organizations have plan_type=STANDARD_FREE, don't have
a CustomerPlan object and thus no tier value.
With self-hosting:
Sponsored organizations have a CustomerPlan object with tier
TIER_SELF_HOSTED_COMMUNITY and a plan_type of PLAN_TYPE_COMMUNITY.
As of c9b0602320 and
8b55d60f9e we create a default
registration for the dev env "RemoteZulipServer" (and also RemoteRealms
for the dev realms) in populate_db. However these models weren't listed
in clear_database, meaning the state wasn't properly cleaned up at the
start of the command.
Most important, this would manifest in getting:
```
django.db.utils.IntegrityError: duplicate key value violates unique
constraint "zilencer_remotezulipserver_uuid_key"
DETAIL: Key (uuid)=(......) already exists
```
if you re-run populate_db.
1. When we get data and it includes realm info, we should automatically
link the new records with the appropriate RemoteRealm.
2. For old records, when we receive realm data, we have an opportunity
to update those old record to link them to the right RemoteRealm.
This logic doesn't need to always run, just after a remote server
upgrade, since that's when this shift in remote server behavior will
occur.
This creates a valid registration, for two reasons:
1. Avoid the need to run "manage.py register_server" in dev env to
register, when wanting to to test stuff with
`PUSH_NOTIFICATION_BOUNCER_URL = "http://localhost:9991"`.
2. Avoid breaking RemoteRealm syncing, due to duplicate registrations
(first set of registrations that gets set up with the dummy
RemoteZulipServer in populate_db, and the second that gets set up via
the regular syncing mechanism with the new RemoteZulipServer created
during register_server).
These names were picked when I still thought these endpoints would serve
both the RemoteRealm and RemoteZulipServer based flows. Now that it's
known these are RemoteRealm-only endpoints, the _server in the names no
longer makes sense.
This commit adds two columns named 'Guest users' and
'Non guest users' to respresent count of such users.
We query 'RemoteRealmAuditLog' to get the data.
Also adds `SWITCH_PLAN_TIER_AT_PLAN_END` for `CustomerPlan`
which will be used to mark status of remote server legacy
plans which are scheduled for an upgrade.
This is a prep commit to return, for each remote realm, the 'uuid',
'can_push', and 'expected_end_timestamp'.
This data will be used in 'initialize_push_notifications'.
Mateusz Mandera
Lauryn Menard
Tim Abbott
Prakhar Pratyush
Alex Vandiver
Anders Kaseorg
Aman Agrawal
Karl Stolley
Sahil Batra
Vector73