This commit makes passing realm mandatory to fetch_initial_state_data.
This is a prep commit for refetching the realm object with
select_related for group setting fields so that extra queries
can be avoided when computing "/registe" response.
To make better use of the limited characters in mobile push
notifications for messages quoting another message, we compress
the blockquotes and "user said" paragraphs to make space for the
actual message.
Fixes#28951.
Extracts code for generating the events table strings to a separate
function and uses templates so that the HTML is clearer.
Updates events table classes for CSS to start with "api-" for
clarity.
Creates a dataclass for getting the data needed for an individual
event's documentation and uses that dataclass in a separate
function for generating the strings for each event's documentation.
Uses templates for parts of event documentation with HTML and CSS,
and updates CSS classes to have clearer names.
Updates the self-hosted and Zulip Cloud billing articles for features
that have been added to billing management such as paying by invoice,
adding billing contact information, viewing past invoices, etc.
Revises text to be cleaner and clearer in some places.
More consistently uses bold (when not linked) for plan names.
Fixes a few errors or omissions in existing instructions.
Co-authored-by: Alya Abbott <alya@zulip.com>
Sending to a topic based on the number of firing alerts makes no
sense, and leads to conversations and alerts scattered randomly across
topics based on how on fire the alerting is.
Send a separate message for each alert in the Grafana webhook payload,
with the alert's name as its topic; if no alert name can be found,
fall back to the alert's fingerprint. Also include all alert values
in the body of the message, along with links to the alert generator,
silence, and image, if available.
Co-authored-by: Alex Vandiver <alexmv@zulip.com>
This commit removes create_public_stream_policy setting
since public channel creation permissions are now handled
by group-based setting.
We still pass "realm_create_public_stream_policy" in
"/register" response though for older clients with its
value being set depending on the value of group based
setting. If we cannot set its value to an appropriate
enum corresponding to the group setting, then we set
it to "Members only" considering that server will not
allow the users without permissions to create public
channels but the client can make sure that UI is
available to the users who have permission.
This was a bug, where in the realm.presence_disabled (synonymous to
being a zephyr mirror realm) case we would return None. We have decided
on the convention of using only integers here, and -1 representing lack
of data.
We immediately navigate the user to the conversation they just
sent a message to if they are not already in the appropriate
conversation view.
This commit adds a first-time banner to explain the same.
Fixes#29575.
4430ab9cbe changed this, assuming that all servers would send
`realm_url` -- however, only servers running that commit do. Update
to accept either `realm_url` or `realm_uri` payload properties.
This commit adds a server level setting which controls whether the setting
can be set to anonymous user groups. We only allow it in the tests for
now because the UI can only handle named user groups.
This commit fixes the event sent for updating first_message_id
when a message is deleted, to include the name field as it is
required for all "stream/update" events.
This commit also adds a test in test_events for the case when
first_message_id of a stream is updated on deleting a message.
Migration plan:
1. Add NULLable .last_update_id column to UserPresence with default 0
for new objects.
2. Backfill the value to 0 for old UserPresences, can be done in the
background while server is running.
3. Make the column non-NULL.
4. Add new model PresenceSequence and create its rows for old realms.
Fixes#26369.
There are two important fixes to make to the dicts in edit_history:
1. Update the user_id so that it points to the imported sender.
2. Apply fix_message_rendered_content to the prev_rendered_content data
to fix up mentions and other such syntax.
The docstring was misleading talking about the import from non-Zulip
platforms, when this function is also very much applicable for
Zulip-to-Zulip imports.
Previously, when the operand of id operator was more than
2147483647, it was raising server error. This is because the
maximum permissible PostgreSQL integers value is 2147483647.
This is fixed by raising a BadNarrowOperatorError in case the
id operand is larger than 2147483647.
Earlier, the 'remove_single_newlines' function wasn't working
correctly when '\n' was followed by asterix, hyphen, or number.
Specifically, they were not added as a syntax for ordered or
unordered list in markdown.
For example, see the workaround fix in 330439a83b.
This commit updates the function to replace '\n' with ' ', when
"\n" is not preceded by "\n" and not followed by:
* Another newline (\n)
* A hyphen followed by a space
* An asterisk followed by a space
* A number followed by a dot and a space
We won't have to do fixes like 330439a83b in the future.
Earlier, for new realm with zulip_update_announcements_stream
set to None, an assertion error was raised in
'is_group_direct_message_sent_to_admins_within_days' because
no 'None' to new level change took place for such realm.
A new realm is on the latest level and a default stream is set.
This commit updates the logic to simply skip sending
update messages as the stream was manually set to None
in such cases.
Removes the 'onboarding_steps' field on 'UserProfile'
model which is no longer used.
It was introduced back in 2013 (b5e22bf), and is no
longer used.
Fixes part of #30043.
This commit adds a new helper function to create or update
a UserGroup object for a setting. We could have used existing
update_or_create_user_group_for_setting but that also validates
user IDs and subgroup IDs which we can skip in tests.
This commit fixes the code store correct old value in audit
log data when changing can_mention_group setting from a
anonymous group to another anonymous group. The bug was
because the old value was being computed after updating
the UserGroup object with new members and subgroups and
is fixed by computing the old value for all the cases
and passing it to do_change_user_group_permission_setting.
requests transforms the base urllib3 exceptions into
requests.RequestExceptions -- but only within code that it is
running. When parsing the streaming body in fetch_open_graph_image,
the read itself (inside lxml) may trigger urllib3 to raise its own
timeout error -- which escapes the current catch of
requests.RequestExceptions.
Catch both requests and urllib3 exceptions.
This commit replaces occurrences of realm_uri with realm_url in email templates
and other related backend files.
Co-authored-by: Junyao Chen <junyao.chen@socitydao.org>
This adds `--automated` and `--no-automated` flags to all Zulip
management commands, whose default is based on if STDIN is a TTY.
This enables cron jobs and supervisor commands to continue to report
to Sentry, and manually-run commands (when reporting to Sentry does
not provide value, since the user can see them) to not.
Note that this only applies to Zulip commands -- core Django
commands (e.g. `./manage.py`) do not grow support for `--automated`
and will always report exceptions to Sentry.
`manage.py` subcommands in the `upgrade` and `restart-server` paths
are marked as `--automated`, since those may be run semi-unattended,
and they are useful to log to Sentry.
The documentation Creates a shared UserStatus schema that's used for
the return value of this new endpoint and for the existing user_status
objects returned by the register queue endpoint.
Co-authored-by: Suyash Vardhan Mathur <suyash.mathur@research.iiit.ac.in>
Fixes#19079.
This commit updates code, majorly in tests, to use
setting values from enums instead of directly using
the constants defined in Realm.
We still have those constants defined Realm as they
are used in a couple of places where the same code
is used for different settings. These will be
handled later.
In e726012244, we removed the
@transaction.atomic decorator from the function
'bulk_set_user_topic_visibility_policy_in_database'.
We should keep it to avoid adding it to the caller
of this function, when reused in the future.
This commits fixes the code which checks group mention permission
to handle anonymous user groups correctly. Basically we were
not checking whether the UserGroup is linked to a NamedUserGroup
and directly accessing named_user_group which results in an
error.
We also update the error messages to include the group name
which has permission to mention the groups since now there
might be a comnbination of groups and users who has permission
to mention the group.
This commit also adds tests to check sending and editing messages
when can_mention_group is set to a anonymous user group.
Earlier, we were using 'send_event' & 'queue_json_publish' in
'do_send_messages' which can lead to a situation where we enqueue
events but the transaction fails at a later stage.
Events should not be sent until we know we're not rolling back.
This commit renames are_both_setting_values_equal function
to are_both_group_setting_values_equal to make it clear
that this function is used to compare value of group
settings.
This commit moves validate_group_setting_value_change,
are_both_setting_values_equal and parse_group_setting_value
functions, which are used for updating the group settings, to
"zerver.lib.user_groups" as these functions will also be used for
group based realm and stream settings and "zerver.lib.user_groups"
file seems a better place to place such functions which are used
at multiple places.
For same reasons, we also move GroupSettingChangeRequest dataclass
to "zerver.lib.user_groups" file.
If an invalid timezone (such as +32h) was provided, the
timestamp.astimezone call would throw an exception, causing the
message send to fail. Replace that with a user-facing error.
Fixes#19658.
This is a follow-up to #29425. It assigns creator to three streams
that are used in tests and another two that aren't. For tests,
we assign a guest, an admin and an owner as creator.
To help users focus on the onboarding experience, we no longer
send the introductory "Zulip updates" message as a part of
onboarding.
Now, we send the introductory message just before the first
update message.
Fixes#30053.
This commit adds support to pass object containing both old and new
values of the can_mention_group setting, as well as detailed API
documentation for this part of the API system.
Co-authored-by: Tim Abbott <tabbott@zulip.com>
Co-authored-by: Greg PRice <greg@zulip.com>
When importing a Stream, UserProfiles don't yet exist. so trying to
import stream.creator fails with something like:
```
psycopg2.errors.ForeignKeyViolation: insert or update on table
"zerver_stream" violates foreign key constraint
"zerver_stream_creator_id_65aeba7e_fk_zerver_userprofile_id"
DETAIL: Key (creator_id)=(5) is not present in table "zerver_userprofile".
```
In 'send_initial_realm_messages', the topics names were not
being translated properly as they were computed outside the
with `override_language` block.
Now, we use 'send_initial_realm_messages' inside a with
'override_language' block in the caller. This fixes the bug.
Note: We are using 'override_language' block in the caller
and not within the function as it helps to avoid indenting
everything inside the function.
The query to fetch one of the greetings message based on
content wasn't taking translation into account that would
result in a bug in realms using non-english language.
This commit updates the query to fix the bug.
- Tighten up the content.
- Use pretty quotes.
- Reorder "experiments" topic to appear after "start a conversation"
in Inbox.
- Drop reference to messages being starred (not implemented yet).
Instead of applying 'remove_single_newlines()' to individual
contents when declared, we use it once while prepping up
message in 'internal_prep_stream_message_by_name'.
Just for the code to be less finnicky.
The function is used for any type of onboarding step
i.e. not limited to Hotspot.
This commit updates the function and variables name
to reflect the above mentioned behaviour.
Mark the channel name of the initial channel created during
realm creation for translation.
It doesn't mark the topic names and description for translation
because we are planning to remove these topics and update the
description as a part of improving the onboarding experience.
We no longer create the 'core team' private channel when
a realm is created.
Earlier, "New user announcements" channel was set to the
"core team" channel. Now it is disabled by default.
populate_db still creates the 'core team' channel to
represent a private channel.
This commit updates access_user_group_for_setting
to support setting anonymous user groups for
different settings.
There are some lines without coverage as of this
commit, but the next few commits would add tests
covering those.
This commit fixes the queries to get members and subgroups for
user group objects returned by user_groups_in_realm_serialized
to not include the UserGroup objects which are not linked to a
NamedUserGroup object, since the function only returns data for
NamedUserGroup objects.
The naming `uri` is deprecated while `url` should be used in order to
satisfy URL standards. For this reason, four endpoints are affected:
* The response content of three endpoints `/server_settings`,
`/register` and `/realm` that contain a field `realm_uri` is
changed to `realm_url`.
* In one of the common fields for all mobile push notifications payloads,
`realm_url` field is now added as an alias to `realm_uri`.
For backwards compatibility, we keep the field `realm_uri` and add
an alias `realm_url`.
Co-authored-by: Junyao Chen <junyao.chen@socitydao.org>
When the export is NOT generated by another zulip server,
while importing:
* Set the 'zulip_update_announcements_level' to the latest level
as we don't want to send all the older update messages to them.
* Send a group DM to admins, suggesting them to configure the
stream in order to avoid missing future update messages.
Fixes#29041.
Earlier, we were creating RealmAuditLog with REALM_EXPORTED
event_type when export of public data took place via organization
settings panel.
We were not creating the audit log when the export was executed
via shell i.e './manage.py export'.
This commit creates the audit log in that case too. It will
help during import to distinguish readily between imports
from another Zulip server vs imports from another product.
In #23380, we are changing all occurrences of uri with url in order to
follow the latest URL standard. Previous PRs #25038 and #25045 has
replaced the occurences of uri that has no direct relation with realm.
This commit changes just the model property, which has no API
compatibility concerns.
Also create dedicated channels and topics sections in sidebar.
Makes 'start a new topic', a section in introduction to topics
page.
Co-authored-by: Prakhar Pratyush <prakhar@zulip.com>
Gitter broke their older API as part of being integrated
into Matrix.
Their announcement blog says:
"Anything left using the Gitter APIs will need to be
updated to use the Matrix API"
This commit drops the legacy Gitter import tool and
we plan to build a new one for Matrix in future.
Updates the base hash for the streams setting overlay to be
"channels" instead of "streams".
Because there are Welcome Bot and Notification Bot messages that
would have been sent with the "/#streams" hash, we will need to
support parsing those overlay hashes as an alias for "/#channels"
permanently.
Part of the stream to channels rename project.
This fixes the annoying behaviour where a task title with a url in it,
for example, "Fix issue https://github/com/zulip/zulip/issues/1234"
would be wrongly split into a description at the `:` in the url.
The returns plugin hasn’t been updated for mypy ≥ 1.6. This
annotation is more limited in that it only supports a fixed number of
positional arguments and no keyword arguments, but is good enough for
our purposes.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
Creates an IncompatibleParametersError to be used in cases where
there are two (or more) optional parameters for an endpoint that
are incompatible with each other, e.g. there's a parameter for a
user name and a user ID but only one should be sent in the request
to identify the user.
Documents the error on the /api/rest-error-handling article.
Updates the PATCH users/me/subscriptions/muted_topics endpoint to
use this error when both the stream and stream_id parameters are
passed (note this endpoint is currently deprecated).
This commit removes name, description, is_system_group and
can_mention_group fields from UserGroup model and rename
them in NamedUserGroup model.
Fixes#29554.
This commit updates code to access name from named_user_group
field which points to the "NamedUserGroup" instead of directly
accessing name from "UserGroup", since name field will only
be present on NamedUserGroup objects in further commits.
This commit adds get_recursive_strict_subgroups function
which returns all the subgroups but not includes the user
group passed to the function.
We also update the test to check subgroups of named user
groups using the get_recursive_strict_subgroups function.
This is fine as we already test the get_recursive_subgroups
function.
Changes url from /help/all-messages to /help/combined-feed.
Also renames files "all-messages.md" and "go-to-all-messages.md"
to "combined-feed.md" and "go-to-combined-feed.md" respectively.
Fixes#27802.
Earlier, we didn't soft-reactivate users for group mentions
at all because it wasn't easy to calculate group size.
Now, we will soft reactivate if the user group mentions has
less than 12 members.
We don't reactivate all users because a user group can have a
very large size, which can lead to large backlogs in the
deferred-work queue.
Fixes part of #27586.
This commit updates the code in 'handle_push_notifications'
to use a common lib function 'get_mentioned_user_group'.
The code logically does the same thing in both the places,
hence the change.
Earlier, in 'get_mentioned_user_group_name()' we were using
'get_user_group_direct_member_ids' function which just checks
the number of direct members and not the recursive membership
of the group.
We should instead use 'get_user_group_member_ids' to calculate
the correct members count of the user group.
Factor out the repeated pattern of taking a lock, or immediately
aborting with a message if it cannot be acquired. The exit code in
that situation is changed to be exit code 1, rather than the successful
0; we are likely missing new work since that process started.
We move the lockfiles to a common directory under `/srv/zulip-locks`
rather than muddy up `/home/zulip/deployments`.
Updates various areas of the backend code that generate
JsonableErrors with translated strings to use channel
instead of stream.
Part of stream to channel rename project.
Updates the translated strings in the messages sent by the welcome
bot in new Zulip organizations to use channel instead of stream.
Comments out part of a test that uses translated Italian strings
to check these welcome bot messages and adds a TODO comment to
note that the test code should be uncommented when those strings
are translated for the stream -> channel rename.
Part of stream to channel rename project.
In zerver/lib/exceptions.py, updates translated error strings
to use channel instead of stream.
Note that the STREAM_WILDCARD_MENTION_NOT_ALLOWED and the
STREAM_DOES_NOT_EXIST error codes, and any additional fields
included with those errors, are not changed in these updates.
Part of stream to channel rename project.
Updates translated JsonableError strings that relate to streams
to use channel instead of stream. Separated from other error string
updates as this is a dense area of changes for this rename.
Part of stream to channel rename project.
In the missed message email, updates "stream_name" context variable
to be "channel_name". This varible is part of some translated
strings in the email's content.
In onboarding zulip topics email, updates content references to
streams to use channels instead for translated strings. Also,
updates "move_topic_to_different_stream_link" context variable to
be "move_channels_link".
Does not yet update the image used in the onboarding topics email
as that will be updated as part of the help center changes for
this rename.
Part of stream to channel rename project.
Updates the translated "New streams" string in the email digest to
instead by "New channels". Also, marks that for translation in the
plain text version of the email.
Updates the generated stream/channel url to use stream_narrow_url
in preparation for updating stream narrow urls for the rename.
Part of stream to channel rename project.
Generally updates variables that appear in translated strings that use
"stream" to instead use "channel".
Two exceptions are ErrorCode.STREAM_DOES_NOT_EXIST JsonableErrors as
changing the variable would also change the fields returned by these
errors to clients.
Changes to context variables in emails and variables in onboarding
welcome bot messages are addressed in separate commits.
Part of stream to channel rename project.
Adds nullable creator field, containing a reference to the user who
created the stream. When creating a stream, acting user is set as
the creator of the stream. Since API calls to create streams always
have an acting user, this field should always be set when streams
are created using the API.
Because streams can be created with no acting user, this field is
nullable. We try to backfill existing streams using RealmAuditLog table,
but not all streams are guaranteed to have a recorded create log. Thus
this new field is left null when it cannot be backfilled. We also set
this field to null when the creator user is deleted.
Fixes#28403
Uses redis to remember the last time push notifications were experienced
working. This needs to work across processes, so can't be done just in
memory.
As this is transient data that's fairly harmless to lose and thus
doesn't require the persistence benefits of the database, and we're
keeping a single "row", so don't need an entire new db table, we settle
on using redis instead of postgres. This is also consistent with how we
store other kinds of such transient data.
Earlier a extra audit log entry of type
USER_GROUP_GROUP_BASED_SETTING_CHANGED was made when a new user
group is created. This commit updates the code to not create
that audit log entry.
There is no need to create these entry as we would still
have the required data from the "OLD_VALUE" field in the
audit log entry created when changing the setting and this
also makes it consistent with the entries created for
other operations like stream creation.
This timeout strategy using asynchronous exceptions has a number of
safety caveats (read the docstring!!) and should only be used in very
specific circumstances.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
- On creating a stream, the user is redirected to the
"stream events" topic of the newly created stream.
- If this is the first time the user has created a stream,
an explanatory modal is shown.
test_hotspots.py is modified to account for the new addition
of the first_stream_created_banner.
stream_create.test.ts is also modified accordingly.
Fixes#29375.
migrated views:
- `zilencer.views.register_remote_server`
- `zilencer.views.register_remote_push_device`
- `zilencer.views.unregister_remote_push_device`
- `zilencer.views.unregister_all_remote_push_devices`
- `zilencer.views.remote_server_notify_push`
to make sure the previous checks for `remote_server_notify_push` matches
to old one, The `RemoteServerNotificationPayload` is defined.
soupsieve is a heavy-weight dependency, and Tornado pulls it in by way
of markdown rendering; since we are only using it for a very simple
process, perform that manually.
Per CSS spec[^1]:
> In quoted <string> url()s, only newlines and the character used to
> quote the string need to be escaped.
[^1]: https://drafts.csswg.org/css-values/#urls
This middleware was highly-specific to a set of URLs, and pulled in a
beautifulsoup dependency for Tornado. Move it closer to where it is
used, minimizing action at a distance, as well as trimming out a
dependency.
9a682fb20a started performing message fetching in a read-only
transaction. However, our use of `get_or_create_huddle` can violate
the read-only promise, and result in a user-facing 500.
In the cases where we're attempting to narrow to a huddle that does
not exist, this is equivalent to a false condition; catch those,
without making the huddle row, and insert a false.
For simiplicty's sake, we can avoid trying to do cache invalidation in
the variety of events that can cause the seat count to change - since
having an up to 1 day delay between users being added and the upload
limit going up is quite reasonable.
Because loading boto3 is so slow, this saves a significant amount of
time (0.3s or so) in process startup on servers which are not using
the S3 file storage backend.
Uptil now, users could add tasks to a todo widget only after creating
it through the `/todo` command in the compose box.
Users can now add an initial list of tasks using the `/todo` command,
with each task on a new line in the compose box, where the 1st `:`
would separate a task from its (optional) description. Example:
`/todo\nTask1:description1\nTask2 without description`.
Fixes part of #20213.
Users can now name task lists by providing the task list title in the
`/todo` command on the same line. Example: `/todo School Work`. If no
title is provided by the user, "Task list" (which is also the
placeholder) is used as default.
The author of a task list can later edit / update the task list title
in the todo widget, just like the question in the poll widget.
Fixes part of #20213.
Uptil now, if the user did not specify a poll question, but included
options when using the `/poll` command, the 1st option wrongly
became the poll question.
Now on leaving the poll question space blank, the next line will not be
confused for it, and the poll widget will have an input for adding a
question.
Adds backend support for "channels" operator.
This will deprecate/replace the "streams" operator eventually, but
we will keep support of the operator for backwards compatibility
for a while.
Part of renaming stream to channel project.
Adds backend support for "channel" operator.
This will deprecate/replace the "stream" operator eventually, but
we will keep support of the operator for backwards compatibility
for a while.
Part of renaming stream to channel project.
Prep commit for deprecating the "stream" search operator and
replacing it with the "channel" operator, and for depreacting the
"streams" operator and replacing it with the "channels" operator.
f92d43c690 added uses of `@overload` to probide multiple type
signatures for `access_message`, based on the `get_user_message`
parameter. Unfortunately, mypy does not check the function body
against overload signatures, so it allows type errors to go
undetected.
Replace the overloads with two functions, for one of which also
returns the usermessage. The third form, of only returning if the
usermessage exists, is not in a high-enough performance endpoint that
a third form is worth maintaining; it uses the usermessage form.
The IntegrityError shows up in the database logs, which looks
unnecessarily concerning. Use `ON CONFLICT IGNORE` to mark this as
expected, especially since the return value is never used.
This commit adds a 'skip-delay' option to the
'send_zulip_update_announcements' management command.
It will be useful for self-hosted servers after 9.0 upgrade to
avoid the 24 hour delay to receive update messages after group
DM is sent to the admins.
One can run the management command with the --skip-delay flag
to immediately send the update messages.
When an organization (without open ability for anyone to join) invites a
guest user, the invitation prompts allows them to choose whether the
guest should be added to default streams or not. This is useful, because
since we don't have per-role default streams configs, they may want
default streams to be for full Members.
SCIM provisioning doesn't have this control, since a newly provisioned
user gets created via a direct do_create_user call, thus adding them to
the organization's default streams, with no workaround possible aside of
just getting rid of default streams in the organization.
To make provisioning guests in such an organization usable, we add a
simple config option to create them with no streams. It's configured by
adding
```
"create_guests_without_streams": True
```
to the config dict in settings.SCIM_CONFIG.
Sahil Batra
Prakhar Pratyush
Lauryn Menard
Vector73
Ryan Crisanti
Alex Vandiver
Mateusz Mandera
Kenneth Rodrigues
Prakhar Pratyush
roanster007
Anders Kaseorg
Tim Abbott
Alya Abbott
bedo
evykassirer
Tomas Fonseca
Aman Agrawal
tnmkr
N-Shar-ma
nimishmedatwal
Sayam Samal
Ujjawal Modi
Kislay Verma
Mahhheshh