Earlier, when adding a new user failed due to no spare licenses
available, a message was sent to the "New user announcements"
stream.
We plan to disable the stream by default as a part of improving
onboarding experience.
Now, we send a group DM to admins when adding a new user fails
due to no spare licenses available. It makes it independent of
the "New user announcements" setting. These warning messages
are important and shouldn't be missed.
Earlier, low licenses warning message was sent to the
"New user announcements" stream.
We plan to disable the stream by default as a part of improving
onboarding experience.
Now, we send a group DM to admins for low licenses warning
to make it independent of the setting. These warning messages
are important and shouldn't be missed.
This is a prep commit to add a 'recipient_users' parameter to
the 'internal_send_huddle_message' function.
'emails' is no longer a required parameter. We can use either
of the 'emails' or 'recipient_users' parameter. 'emails' is
eventually used to fetch 'recipient_users', so if the
'recipient_users' is already available we should use that to
skip database query.
The `zerver/0501_delete_dangling_usermessages` was backported to the
`8.x` branch (and the 8.3 release) in 3db1733310. However, because
`main` contained migrations which `8.x` did not, it was backported
with a different `dependencies`:
```
dependencies = [
("zerver", "0496_alter_scheduledmessage_read_by_sender"),
]
```
...as opposed to in `main`:
```
dependencies = [
("zerver", "0500_realm_zulip_update_announcements_stream"),
]
```
This causes upgrades from 8.3 to `main` to fail:
```
django.db.migrations.exceptions.InconsistentMigrationHistory:
Migration zerver.0501_delete_dangling_usermessages is applied before
its dependency zerver.0500_realm_zulip_update_announcements_stream on
database 'default'.
```
Adjust the dependencies in `main` to match those in `8.x` where many
deploys will first have encountered the migration.
For organizations with "Zulip update announcements" stream set
to a default value, we wait for one day after sending group
DM to admins to allow them to change the stream from it's
default value if they wish to.
This prep commit refactors the function
'is_group_direct_message_sent_to_admins_atleast_one_week_ago' to
'is_group_direct_message_sent_to_admins_within_days' allowing us
to use a flexible timeframe instead of hardcoded 1 week.
We will reuse this function as a part of determinig whether
group DM to admins was sent within 1 day.
Previously, users were allowed to signup or change their names to
those which already existed in the realm.
This commit adds an Organization Permission, that shall enforce
users to use unique names while signing up or changing their
names. If a same or normalized full name is found in realm,
then a validation error is thrown.
Fixes#7830.
Previously, email addresses that weren't connected to a Zulip account
were ignored but now they receive an email stating their email isn't
connected to a Zulip account.
Also, removes the "Thanks for using Zulip!" line at the end of the
find accounts email that's sent when a Zulip account is found.
Updates the i18n test that used this string with another in the
German translation from this a successful account found email.
Fixes part of #3128
Co-authored-by: Lauryn Menard <lauryn@zulip.com>
Updates the help link in the find team emails to use the external
host information.
Removes the link for the external host since the realm links are
what the user should click on to login.
Also, passes corporate_enabled to the find team email to adjust
the text for Zulip Cloud emails.
Restructures the integration documentation pages to use a style
that's more similar to the help center documentation, with an
instruction block for setting up the integration, and sections
for additional configuration information and related documentation
links.
Updates the doc pages for the airbrake, azuredevops and gitlab
integrations as examples of the updated style.
Also updates the URL specification section of the incoming webhook
overview in the API documentation so that the documented URL
parameters can be linked to directly in the integration doc pages.
Co-authored-by: Alya Abbott <alya@zulip.com>
This was only used in the undocumented narrow_stream mode, and relied
on a deprecated synchronous XHR request.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
For multiline strings in triple quotes, a '\n' is included
at the end of each line.
Earlier, to skip '\n' we used to add an escape character '\'
at the end of each line.
This commit adds a function to avoid manually adding '\'.
As a part of the zulip news feature, we send an initial
group DM to admins suggesting them to update or set
the 'zulip_update_announcements_stream'.
This commit updates those messages to make it better.
Updates the check email translation test for updated email text in
confirm_new_email.html and onboarding_zulip_topics.html for current
translated strings in German.
As noted in the docstring for `bulk_insert_ums`, this is at least one
order of magnitude faster than using `bulk_create`. This also
includes a `ON CONFLICT DO NOTHING` which allows multiple
soft-reactivations to run at once without failing. We also adjust the
update of `last_active_message_id` to be safe against races.
Rather than use a bulk insert via Django, use the faster
`bulk_insert_all_ums` that we already have. This also adds a `ON
CONFLICT` clause, to make the insert resilient to race conditions.
There are currently two callsites, with different desired `ON
CONFLICT` behaviours:
- For `notify_reaction_update`, if the `UserMessage` had already been
created, we would have done nothing to change it.
- For `do_update_message_flags`, we would have ensured a specific bit
was (un)set.
Extend `create_historical_user_messages` and `bulk_insert_all_ums` to
support `ON CONFLICT (...) UPDATE SET flags = ...`.
The bots do not exist in the user table to look up their active
status, and attempting to import them into the analytics table will
result in duplicate rows.
Replace the long string for organisations that have notification
body/content disabled (settings.PUSH_NOTIFICATION_REDACT_CONTENT
set to true) with "New message".
This allows more of the limited space on the mobile device screen to
be used for additional messages rather than this verbose content.
Fixes#29152
Using --rotate-key without write access to the secrets file is currently
quite painful, since you end up rotating your registration's secret with
no local record of it; so effectively you lose your registration and
need help from support. We should just prevent this failure mode.
Previously, #26419 addressed the majority of these calls, but did not
prevent more from creeping in. Remove the one remaining
callsite (after the cleanup from the previous commits), and ban any
future use of the pattern.
For the common case of not needing to reference the UserMessage row
later, and for being a stream without private history, the UserMessage
row is irrelevant. Convert `has_user_message` to a thunk, and defer
loading it unless necessary.
Calling `.select_related()` with no arguments joins through every
possible table, recursively. In this case, this currently produces a
query which joins through forty-three tables.
This is rather inefficient, particularly for what is a very common
call which should be very fast.
No callsite depends on having prefetched any joined table on the
object; drop all of the joins.
Replaced HUDDLE attribute with DIRECT_MESSAGE_GROUP using VS Code search,
part of a general renaming of the object class.
Fixes part of #28640.
Co-authored-by: JohnLu2004 <JohnLu10212004@gmail.com>
Adds a line to the top of the internal_billing_notice email with
the billing entity's display name.
Makes sure all internal_billng_notice email subjects also include
the billing entity's display name.
Makes small updates to the notice text for some cases.
This commit adds a management command that will run regularly
as a cron job to send zulip updates to realms based on their
current and latest zulip_update_announcements_level.
For realms with:
* level = None: Send a group DM to admins notifying them about
this new feature & suggestion to set the stream accordingly.
* level = 0:
* If stream is still not configured, wait for a week
before setting their level to latest level. They will
miss updates until their configure the stream.
* If stream is configured, send updates.
* level > 0: Send one message/update per level & increase
the level by 1 till the latest level.
Fixes#28604.
This is a prep commit to extract out the logic to
create message from 'internal_send_huddle_message'
into a separate function 'internal_prep_huddle_message'.
We will use this new function to get the huddle message
without sending it immediately.
In general, we never want to use savepoints.
This prep commit adds savepoint=False in do_send_messages
as we don't want to just rollback to this savepoint and
proceed if we encounter any error while sending zulip updates
via cron.
A user who was no longer subscribed to a private stream kept their
UserMessage row for a message sent while they were in it; this is
expected. However, they _also_ kept that row even if the message was
moved to a different private stream that they were also not subscribed
to. This violates the invariant that users without subscriptions
never have UserMessage rows.
This `if new_stream is not None` block was improperly indented,
causing it to only run if the propagation mode was not `change_one`.
Since the block controlled creation and deletion of UserMessage rows,
this led to messages being improperly still visible to members of the
old stream if they were being moved from public to private streams.
Clients also failed to receive `delete_message` events, so the
messages remained visible in their feeds until they reloaded the
application.
We don't want to show one-time modals introducing 'Inbox' and
'Recent conversations' views to existing users.
When a user views a modal, we mark it as read by storing a row
in the 'OnboardingStep' model. Once marked as read, the user will
no longer see the modal.
This commit adds a migration to create
two rows per user (mark them as read) in OnboardingStep model.
To improve onboarding experience, this commit adds a
one-time modal which introduces the recent conversations view.
Users see this one-time modal on visiting the recent
conversations view.
Fixes#29073.
To improve onboarding experience, this commit adds
a one-time modal which introduces the inbox view.
Users see this one-time modal on visiting the inbox view.
Fixes part of #29073.
Replace a separate call to subprocess, starting `node` from scratch,
with an optional standalone node Express service which performs the
rendering. In benchmarking, this reduces the overhead of a KaTeX call
from 120ms to 2.8ms. This is notable because enough calls to KaTeX in
a single message would previously time out the whole message
rendering.
The service is optional because he majority of deployments do not use
enough LaTeX to merit the additional memory usage (60Mb).
Fixes: #17425.
Links to the available message flag table in the feature level 224
changelog entry, as there are relevant **Changes** notes for this
feature level in that part of the API documentation.
Updates the order and formatting of these new and deprecated flags
in the available flags table. Also, adds a link to the topic
wildcard mentions section of the help center documentation.
Makes small clean ups to the changes notes for this feature level,
as well as the changelog entry itself.
The original commit for these feature level 224 API changes was
c597de6a1d.
Refactor `parse_client` view to use `typed_endpoint decorator`
instead of `has_request_variables`. This change improves code consistency
and enhances codebase comprehension.
PostgreSQL's estimate of the number of usermessage rows for a single
message can be wildly off, due to poor statistics generation. This
causes this query, with 100-message batch sizes, to incorrectly
estimate millions of matched rows, causing it to perform a full-table
index scan, rather than piecemeal using the `message_id` index.
Reduce the batch size to 50, which is enough to tip in favor of a
rational query plan.
Refactor `report_csp_violations` view to use `typed_endpoint` decorator
instead of `has_request_variables`. This change improves code
consistency and enhances codebase comprehension.
Depending on the kind of config error being shown, different "go back"
links may be more appropriate.
We probably hard-coded /login/ for it, because these config errors are
most commonly used for authentication backend config error, where it
makes sense to have /login/ as "go back", because the user most likely
indeed got there from the login page.
However, for remote_billing_bouncer_not_configured, it doesn't make
sense, because the user almost surely is already logged in and got there
by clicking "Plan management" inside the gear menu in the logged in app.
It's best for these to just be consistent. Therefore:
1. The .../not-configured/ error page endpoint should be restricted to
.has_billing_access users only.
2. For consistency, self_hosting_auth_view_common is tweaked to also do
the .has_billing_access check as the first thing, to avoid revealing
configuration information via its redirect/error-handling behavior.
The revealed configuration information seems super harmless, but it's
simpler to not have to worry about it and just be consistent.
Just shows a config error page if the bouncer is not enabled. Uses a new
endpoint for this so that it can work nicely for both browser and
desktop app clients.
It's necessary, because the desktop app expects to get a json response
with either an error or billing_access_url to redirect to. Showing a
nice config error page can't be done via the json error mechanism, so
instead we just serve a redirect to the new error page, which the app
will open in the browser in a new window or tab.
Only affects zulipchat, by being based on the BILLING_ENABLED setting.
The restricted backends in this commit are
- AzureAD - restricted to Standard plan
- SAML - restricted to Plus plan, although it was already practically
restricted due to requiring server-side configuration to be done by us
This restriction is placed upon **enabling** a backend - so
organizations that already have a backend enabled, will continue to be
able to use it. This allows us to make exceptions and enable a backend
for an org manually via the shell, and to grandfather organizations into
keeping the backend they have been relying on.
Adds a re-usable lockfile_nonblocking helper to context_managers.
Relying on naive `os.mkdir` is not enough especially now that the
successful operation of this command is necessary for push notifications
to work for many servers.
We can't use `lockfile` context manager from
`zerver.lib.context_managers`, because we want the custom behavior of
failing if the lock can't be acquired, instead of waiting.
That's because if an instance of this gets stuck, we don't want to start
queueing up more processes waiting forever whenever the cronjob runs
again and fail->exit is preferrable instead.
When a server doesn't submit a remote realm info which was
previously submitted, we mark it as locally deleted.
If such a realm has paid plan attached to it, we should investigate.
This commit adds logic to send an email to sales@zulip.com for
investigation.
This commit updates default for delete_own_message_policy
setting to "Everyone" as it is helpful to allow everyone
to delete their own messages in a new organization where
users might be using Zulip for the first time.
This commit updates default for move_messages_between_streams_policy
setting to "Members and above" as it is helpful to allow members
to move messages between streams in new organizations where users
might be using Zulip for first time.
The presence of `len(messages)` outside the transaction caused the
full resultset to be fetched outside of the transaction. This should
ideally be inside the transaction, and also only need be the count.
However, also note that the process of counting matching rows, and
then executing a second query which embeds the same query, is
susceptible to phantom reads, where a query with the same conditions
returns different resultsets, under PostgreSQL's default transaction
isolation of "read committed." While this is possible to resolve by
pulling the returned IDs into a Python list, it would not address the
issue that concurrent updates which change the resultset would make
the overall algorithm still incorrect.
Add a comment clarifying the conditions under which the algorithm is
correct. A more correct algorithm would walk the UserMessage rows
which are unread and in the stream, but this requires a
whole-UserMessage index which would be quite large for such an
infrequent use case.
This makes no immediate reloads the default for runtornado, matching
the production configuration, and changes the development incantation
to be the one to specify the departure from the norm, with
--immediate-reloads.
LoggingCountStats with a daily duration and that are directly stored
on the RealmCount table (not via aggregation in process_count_stat),
can be in a state, after the hourly cron job to update analytics
counts, where the logged value will be live-updated later, because
the end time for the stat is still in the future.
As these logging counts are designed to be used on the self-hosted
installation for either debugging or rate limiting, sending these
partial/incomplete counts to the bouncer has low value.
Due to the channel_map_to_topics URL parameter in the Slack webhook,
it was not migrated to use the check_send_webhook_message.
By using check_send_webhook_message, any topic parameter in the
webhook URL will be prioritized over mapping Slack channels to
topics, e.g. when channel_map_to_topics is true. This is because
the default behaviour for incoming webhooks is to send a default
topic as a parameter to check_send_webhook_message in case there
is no topic specified in the URL.
In contrast, we can override the stream passed in the URL when
channel_map_to_topics is false by passing the Slack channel name
to check_send_webhook_message. The default behaviour for incoming
webhooks is to send a direct message if there is no specified
stream in the URL, so a default stream is not generally passed
to check_send_webhook_message.
Fixes#27601.
This commit adds a realm-level setting named
'zulip_update_announcements_stream' that configures the
stream to which zulip updates should be posted.
Fixes part of #28604.
- Adds instructions for downloading a zuliprc file for a bot or for
yourself.
- Updates the button label to "Download zuliprc", since that's the
filename it downloads.
Fixes#28881.
The previous logic incorrectly used the server-level number of users
even when a (presumably smaller) realm-level count was available.
Fixes a bug introduced in 2e1ed4431a.
This commit renames the realm-level setting
'signup_notifications_stream' to 'signup_announcements_stream'.
The new name reflects better what the setting does.
This commit renames the realm-level setting 'notifications_stream'
to 'new_stream_announcements_stream'.
The new name reflects better what the setting does.
5c96f94206 mistakenly appended, rather than prepended, the edit to
the history. This caused AssertionErrors when attempting to view the
history of moved messages, which check that the `last_edit_time`
matches the timestamp of the first edit in the list.
Fix the ordering, and update the `edit_history` for messages that were
affected. We limit to only messages edited since the commit was
merged, since that helps bound the affected messages somewhat.
RemoteRealm customer takes precedence over RemoteServer
in general. But if an inactive plan is associated with
RemoteRealm and an active plan with RemoteServer, the
ACTIVE plan takes precendence.
Co-authored-by: Prakhar Pratyush <prakhar@zulip.com>
Previously, in DM disabled org messaging to bot was not working when
starting new conversation and adding bot as recipient because of not
updating on recipient change. And secondly, self messaging was not
allowed.
This commit ensures that the DM to bot and self are allowed irrespective
of dm restrictions.
tests: Verify DMs adhere to DM restriction policy.
Fixes#28412
Signed-off-by: sayyedarib <sayyedaribhussain4321@gmail.com>
The widening of the time between when a process is marked for
reload (at Tornado startup) and when it sends reload events makes it
unlikely-to-impossible that a single `/` request will span both of
them, and thus hit the WebReloadClientError corner case.
Remove it, as it is not worth the complication. The bad behaviour it
is attempting to prevent (of a reload right after opening `/`) was
always still possible -- if the `/` request completed right before
Tornado restarted -- so it is not clear that it was ever worth the
complication.
Collapsing was done incorrectly, as 65c400e06d added `zulip_version`
and `zulip_feature_level`, but did not update the virtual event logic
to copy those new values into the virtual event.
However, it is unlikely that a server will be upgraded multiple times
in quick enough succession for this to ever be relevant. Remove the
logic, which is additional complication for little or no gain.
Commit bd6471f0e3 (#28691) added this
reference to the old name, even though it had already been renamed in
commit b220d29fed (#17775), presumably
because that had failed to update the OpenAPI description.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
Having a non-identity `cache_transformer` is no different from running
it on every row of the query_function. Simplify understanding of the
codepath used in caching by merging the pieces of code.
Rather than pass around a list of message objects in-memory, we
instead keep the same constructed QuerySet which includes the later
propagated messages (if any), and use that same query to pick out
affected Attachment objects, rather than limiting to the set of ids.
This is not necessarily a win -- the list of message-ids *may* be very
long, and thus the query may be more concise, easier to send to
PostgreSQL, and faster for PostgreSQL to parse. However, the list of
ids is almost certainly better-indexed.
After processing the move, the QuerySet must be re-defined as a search
of ids (and possibly a very long list of such), since there is no
other way which is guaranteed to correctly single out the moved
messages. At this point, it is mostly equivalent to the list of
Message objects, and certainly takes no less memory.
Rather than use `bulk_update()` to batch-move chunks of messages, use
a single SQL query to move the messages. This is much more efficient
for large topic moves. Since the `edit_history` field is not yet
JSON (see #26496) this requires that PostgreSQL cast the current data
into `jsonb`, append the new data (also cast to `jsonb`), and then
re-cast that as text.
For single-message moves, this _increases_ the SQL query count by one,
since we have to re-query for the updated data from the database after
the bulk update. However, this is overall still a performance
improvement, which improves to 2x or 3x for larger topic moves. Below
is a table of duration in seconds to run `do_update_message` to move a
topic to a new stream, based on messages in the topic, for before and
after this change:
| Topic size | Before | After |
| ---------- | -------- | ------- |
| 1 | 0.1036 | 0.0868 |
| 2 | 0.1108 | 0.0925 |
| 5 | 0.1139 | 0.0959 |
| 10 | 0.1218 | 0.0972 |
| 20 | 0.1310 | 0.1098 |
| 50 | 0.1759 | 0.1366 |
| 100 | 0.2307 | 0.1662 |
| 200 | 0.3880 | 0.2229 |
| 500 | 0.7676 | 0.4052 |
| 1000 | 1.3990 | 0.6848 |
| 2000 | 2.9706 | 1.3370 |
| 5000 | 7.5218 | 3.2882 |
| 10000 | 14.0272 | 5.4434 |
Prakhar Pratyush
Vector73
Lauryn Menard
Alex Vandiver
Alya Abbott
Tim Abbott
roanster007
swayam0322
Anders Kaseorg
Aman Agrawal
Karl Stolley
Aditya Bajaj
Mateusz Mandera
John Lu
Mahhheshh
David Rosa
Sahil Batra
sayyedarib