The previous implementation using Django's `get_or_create` for
`do_increment_logging_stat` involved two separate database queries,
potentially leading to race conditions.
Use an `ON CONFLICT ... DO UPDATE` (aka "upsert") query, which
eliminates race conditions and improves performance. This is mildly
complicated due to the different unique indexes across the various
tables, and the need for bug-for-bug compatibility with the previous
implementation.
Fixes#28947.
Co-authored-by: Alex Vandiver <alexmv@zulip.com>
The "invites" worker exists to do two things -- make a Confirmation
object, and send the outgoing email. Making the Confirmation object
in a background process from where the PreregistrationUser is created
temporarily leaves the PreregistrationUser in invalid state, and
results in 500's, and the user not immediately seeing the sent
invitation. That the "invites" worker also wants to create the
Confirmation object means that "resending" an invite invalidates the
URL in the previous email, which can be confusing to the user.
Moving the Confirmation creation to the same transaction solves both
of these issues, and leaves the "invites" worker with nothing to do
but send the email; as such, we remove it entirely, and use the
existing "email_senders" worker to send the invites. The volume of
invites is small enough that this will not affect other uses of that
worker.
Fixes: #21306Fixes: #24275
This prevents users from hammering the invitation endpoint, causing
races, and inviting more users than they should otherwise be allowed
to.
Doing this requires that we not raise InvitationError when we have
partially succeeded; that behaviour is left to the one callsite of
do_invite_users.
Reported by Lakshit Agarwal (@chiekosec).
Factor out the repeated pattern of taking a lock, or immediately
aborting with a message if it cannot be acquired. The exit code in
that situation is changed to be exit code 1, rather than the successful
0; we are likely missing new work since that process started.
We move the lockfiles to a common directory under `/srv/zulip-locks`
rather than muddy up `/home/zulip/deployments`.
Updates translated JsonableError strings that relate to streams
to use channel instead of stream. Separated from other error string
updates as this is a dense area of changes for this rename.
Part of stream to channel rename project.
Updates the labels for the "Messages sent by recipient type" chart
to use "channel", and updates the error message that would be sent
for the "messages_sent_by_stream" chart (that has not yet been
implemented) for a missing channel/stream.
Part of the stream to channel rename project.
Replaced HUDDLE attribute with DIRECT_MESSAGE_GROUP using VS Code search,
part of a general renaming of the object class.
Fixes part of #28640.
Co-authored-by: JohnLu2004 <JohnLu10212004@gmail.com>
Adds a re-usable lockfile_nonblocking helper to context_managers.
Relying on naive `os.mkdir` is not enough especially now that the
successful operation of this command is necessary for push notifications
to work for many servers.
We can't use `lockfile` context manager from
`zerver.lib.context_managers`, because we want the custom behavior of
failing if the lock can't be acquired, instead of waiting.
That's because if an instance of this gets stuck, we don't want to start
queueing up more processes waiting forever whenever the cronjob runs
again and fail->exit is preferrable instead.
LoggingCountStats with a daily duration and that are directly stored
on the RealmCount table (not via aggregation in process_count_stat),
can be in a state, after the hourly cron job to update analytics
counts, where the logged value will be live-updated later, because
the end time for the stat is still in the future.
As these logging counts are designed to be used on the self-hosted
installation for either debugging or rate limiting, sending these
partial/incomplete counts to the bouncer has low value.
The previous logic incorrectly used the server-level number of users
even when a (presumably smaller) realm-level count was available.
Fixes a bug introduced in 2e1ed4431a.
We don't have an index on `(server_id, id)`, and in any case, we have
a stronger guarantee that `remote_id` is time-sorted, from the
construction of the analytics tables, than that the `id`s given these
entries when uploaded are time-sorted.
View functions in `analytics/views/support.py` are moved to
`corporate/views/support.py`.
Shared activity functions in `analytics/views/activity_common.py`
are moved to `corporate/lib/activity.py`, which was also renamed
from `corporate/lib/analytics.py`.
Updates the total row for the installation and remote activity
charts to be in the table footer. Makes the footer class sticky
to the bottom of the view so that it is always visible on the
chart.
Also, updates the installation activity column for revenue to
be formatted as a dollar string, since this formatting was being
applied in the updated total row.
To estimate the annual recurring revenue for remote server and
remote realm CustomerPlans, we prefetch the current license
ledger as part of the CustomerPlan query.
Also adds a select related to the remote realm audit log query
so that we don't go to the database for the remote realm ID.
With the test added in the previous commit, the query count for
the remote activity view goes from 27 to 11, as we are no longer
hitting the database multiple times for every current plan or
for every remote realm with audit log data.
Refactors get_customer_plan_renewal_amount so that a license
ledger is always passed and make_end_of_cycle_updates_if_needed
does not need to be called.
Creates multiple remote servers and remote realms with active
plan data and audit logs.
Shows multiple small queries to get revenue data from plan license
ledger, as well as remote server and remote realm IDs.
Combines the IDs for the remote realm and remote server in the
first column.
Moves the Realm name to the second column.
For the host/hostname column, displays the remote realm host if
it is a remote realm row. Otherwise, the remote server hostname
is displayed.
Instead of showing the next invoice date for the plan, show the
date for the next billing cycle start (e.g. the next plan renewal
charge date), except for plans currently on a free trial.
For plans on a free trial, the next plan renewal date will be when
the free trial ends, which is stored as the next invoice date on
the plan.
Instead of querying the database for every remote server and realm
in the remote activity chart, we now get the server and realm data
for the installation in two queries.
Adds columns for remote realm ID, name and organization type. If
a remote server has remote realms attached that are not marked
as deactivated by the remote server, then there will be a row in
the chart for each remote realm (which duplicates some remote
server data).
Updates the plan data, revenue and user counts to be for the realm
if present and otherwise for the server.
Updates the user counts to be total users and guest users, instead
of non guest and guest users.
The total row for mobile data (users and pushes forwarded) sums
each remote server's data once, so while the column duplicates
data, the total row should be an accurate total for the installation.
Adds 5 queries to the remote activity page test. One is for the
additional query for the remote realm plans. The other four are
getting the remote realm object and then the user count data for
the two remote realms in the test.
This will help with updating this function for updating/changing
the columns in the chart as only the constant needs to be updated
and not every instance the integer is used in the loops over the
rows of data.
Merges the two charts remaining to have just one chart for the
realm activity view.
Removes the columns for different clients and adds a column to
show/sort by the user type (human or bot type).
Deletes templates/analytics/activity.html because it is no
longer used for any activity pages/views.
Removes the summary tab/chart from the user activity view.
Moves get_user_activity_summary from common_activity.py to
realm_activity.py since it's no longer used in the user
activity view.
Updates the title of the page to incluce the realm name since
an email address may be used for multiple user accounts.
Adds a support action for updating the minimum licenses on a
customer object once a default discount has also been set.
In the case that the current billing entity has a current active
plan or a scheduled upgrade to a new plan, then the minimum
licenses will not be updated.
Previously, the message string was sent as a success response to
the context, which could have been confusing or ignored when shown
in the support admin view.
Since the latest end time for reported active users on a remote
server is not used to determine if a remote server has current
data for billing purposes, we instead display the last audit log
update datetime for the remote server object.
Removes the note in the remote activity key about the Zulip
version.
Also adds a sum in the total row for mobile push notifications
sent.
Changes the emoji that highlights when a remote realm or server has
an active discount.
Adds a remote server UUID field to the remote server information.
Moves the 'has remote realms' field to be after the Zulip version
information, and adds an extra break there to visually highlight
that field as well as the following 'max monthly messages' field.
Mahhheshh
Alex Vandiver
Sahil Batra
Lauryn Menard
John Lu
Mateusz Mandera
Anders Kaseorg
Tim Abbott
Prakhar Pratyush
Alya Abbott