So far, we've used the BitField .authentication_methods on Realm
for tracking which backends are enabled for an organization. This
however made it a pain to add new backends (requiring altering the
column and a migration - particularly troublesome if someone wanted to
create their own custom auth backend for their server).
Instead this will be tracked through the existence of the appropriate
rows in the RealmAuthenticationMethods table.
Adds a new welcome email, `onboarding_zulip_guide`, to be sent four
days after a new user registers with a Zulip organization if the
organization has specified a particular organization type that has
a guide in the corporate `/for/.../` pages. If there is no guide,
then no email is scheduled or sent.
The current `for/communities/` page is not very useful for users
who are not organization administrators, so these onboarding guide
emails are further restricted for those organization types to
only go to new users who are invited/registered as admins for the
organzation.
Adds two database queries for new user registrations: one to get
the organization's type and one to create the scheduled email.
Adds two email logs because the email is sent both to a new user
who registers with an existing organization and to the organization
owner when they register a new organization.
Co-authored by: Alya Abbott <alya@zulip.com>
This commit adds code to create a "Nobody" system user group
to realms which will be used in settings to represent "Nobody"
option.
We also add a migration to add this group to existing realms.
This commit adds PreregistrationRealm class which will be
similar to PreregistrationUser and will store initial
information of the realm before its creation as we are
changing the organization creation flow as per #24307.
Fixes part of #24307.
Currently, there is a checkbox setting for whether to
"Include realm name in subject of message notification emails".
This commit replaces the checkbox setting with a dropdown
having values: Automatic [default], Always, Never.
The Automatic option includes the realm name if, and only if,
there are multiple Zulip realms associated with the user's email.
Tests are added and(or) modified.
Fixes: #19905.
The Django convention is for __repr__ to include the type and __str__
to omit it. In fact its default __repr__ implementation for models
automatically adds a type prefix to __str__, which has resulted in the
type being duplicated:
>>> UserProfile.objects.first()
<UserProfile: <UserProfile: emailgateway@zulip.com <Realm: zulipinternal 1>>>
Signed-off-by: Anders Kaseorg <anders@zulip.com>
c7d0192755 added the unique constraint on
`user_profile_id,message_id,reaction_type,emoji_code`, but left the
existing constraint on `user_profile_id,message_id,emoji_name`. As
explained in the comment added in 3cd543ee98, `emoji_name` cannot be
trusted to be unique, as it is possible to have an Unicode emoji
reaction and a custom emoji with the same name on a message.
Remove the overly-constraining unique index, now that c7d0192755 has
provided the correct one.
This commit adds backend code to set email_address_visibility when
registering a new user. The realm-level default and the value of
source profile gets overridden by the value user selected during
signup.
Ever since we started bundling the app with webpack, there’s been less
and less overlap between our ‘static’ directory (files belonging to
the frontend app) and Django’s interpretation of the ‘static’
directory (files served directly to the web).
Split the app out to its own ‘web’ directory outside of ‘static’, and
remove all the custom collectstatic --ignore rules. This makes it
much clearer what’s actually being served to the web, and what’s being
bundled by webpack. It also shrinks the release tarball by 3%.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
We add stream_permission_group_settings object which is
similar to property_types framework used for realm settings.
This commit also adds GroupPermissionSetting dataclass for
defining settings inside stream_permission_group_settings.
We add "do_change_stream_group_based_setting" function which
is called in loop to update all the group-based stream settings
and it is now used to update 'can_remove_subscribers_group'
setting instead of "do_change_can_remove_subscribers_group".
We also change the variable name for event_type field of
RealmAuditLog objects to STREAM_GROUP_BASED_SETTING_CHANGED
since this will be used for all group-based stream settings.
'property' field is also added to extra_data field to identify
the setting for which RealmAuditLog object was created.
We will add a migration in further commits which will add the
property field to existing RealmAuditLog objects created for
changing can_remove_subscribers_group setting.
This will allow us to rely on the database to detect duplicate
`UserTopic`s (with the same `topic_name` with different cases)
and thus correctly throw IntegrityErrors when expected.
This is also important from a correctness point of view, since as
of now, when checking if topic is muted or requesting the backend for
muting a topic, the frontend does not check for case insensitivity.
There might exist duplicate UserTopics (in a case insensitive sense)
which need are removed before creating the new index.
The migration was tested manually using `./manage.py shell`.
The Client.name field is only 30 characters long, but there is no
limit to the length of parsed User-Agent value which we may attempt to
store in it. This can cause requests with long user-agents to 500
when the creation of the Client row fails.
Truncate the name at 30 characters for the cache key, and passing
`name` to `get_or_create`.
This commits update the code to use user-level email_address_visibility
setting instead of realm-level to set or update the value of UserProfile.email
field and to send the emails to clients.
Major changes are -
- UserProfile.email field is set while creating the user according to
RealmUserDefault.email_address_visbility.
- UserProfile.email field is updated according to change in the setting.
- 'email_address_visibility' is added to person objects in user add event
and in avatar change event.
- client_gravatar can be different for different users when computing
avatar_url for messages and user objects since email available to clients
is dependent on user-level setting.
- For bots, email_address_visibility is set to EVERYONE while creating
them irrespective of realm-default value.
- Test changes are basically setting user-level setting instead of realm
setting and modifying the checks accordingly.
This commit renames parse_message_content_edit_or_delete_limit
to parse_message_time_limit_setting and also renames
MESSAGE_CONTENT_EDIT_OR_DELETE_LIMIT_SPECIAL_VALUES_MAP to
MESSAGE_TIME_LIMIT_SETTING_SPECIAL_VALUES_MAP.
We do this change since this function and object will also be
used for message move limit and it makes sense to have a more
generic name.
In Zulip, message topics are case-insensitive but case-preserving.
The `get_context_for_message` function erroneously did a
case-sensitive search, and thus only messages whose topic matched
exactly were pulled in as context.
Make the missed-message pipeline aware that message topics are not
case-sensitive. This means that, when collapsing adjacent messages,
we merge messages with topic headers which are "different"; create a
separate explicit "grouping" to know which to collapse.
Black 23 enforces some slightly more specific rules about empty line
counts and redundant parenthesis removal, but the result is still
compatible with Black 22.
(This does not actually upgrade our Python environment to Black 23
yet.)
Signed-off-by: Anders Kaseorg <anders@zulip.com>
This commit renames "can_edit_topic_of_any_message" function
in models.py to "can_move_messages_to_another_topic" and
"user_can_edit_topic_of_any_message" function in settings_data.js
to "user_can_move_messages_to_another_topic".
This change is done since topic editing permission does not
depend on message sender now and messages are considered same
irrespective of whether the user who is editing the topic had sent
the message or not. This also makes the naming consistent with
what we use for the label of this setting in webapp and how we
describe this action in help documentation.
This explicitly enforces ordering on the linkifiers. This is useful when
there are overlapping linkifier patterns that matches the same text. In
our current linkifier implementation, this order affects how the
patterns are handled in the markdown processor, with the earlier ones
being prioritized.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
This allows us to revoke MultiUseInvites by changing their .status
instead of deleting them (which has been deleting the helpful tracking
information on PreregistrationUsers about which MultiUseInvite they came
from).
Fixes#23517.
While this feature was added to Zulip very early, it has been troubled
for most of that time; it never looked great visually, had a lot of
implementation complexity around resize.js, and has a weird model (a
setting that changes the UI only in certain window sizes).
This option is not commonly used; while a significant portion of users
have it enabled, many of them just don't use window sizes where it
actually has an effect. So it's not clear that it will be missed if
removed; we got very few bug reports when it was completely broken for
a few days after we first integrated the new left sidebar private
messages design.
Even with it no longer being broken, it does not work very well with
the addition of the new PMs section in the left sidebar. (Having two
scrollbars in the sidebar looks quite awkward.) The new private
messages section in the left sidebar also addresses some of the use
cases for always keeping the Users list always visible, even in narrow
windows.
This option is only removed from frontend for now. To make this
decision easily reversible, the backend code of this feature
is still kept.
Storing a QuerySet rather than the list version of the result in it
has a large overhead -- and, as noted by the type annotations, the
result is only ever used as a list. This difference is particularly
important because the cached `get_realm_user_dicts` can get extremely
large for realms with large numbers of users, potentially overflowing
the 1MB default object limit in memcached.
Switch all cases of `cache_with_key` which return QuerySets to
returning the list values of them.
These were useful as a transitional workaround to ignore type errors
that only show up with django-stubs, while avoiding errors about
unused type: ignore comments without django-stubs. Now that the
django-stubs transition is complete, switch to type: ignore comments
so that mypy will tell us if they become unnecessary. Many already
have.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
This commit adds the OPTIONAL .realm attribute to Message
(and ArchivedMessage), with the server changes for making new Messages
have this set. Old Messages still have to be migrated to backfill this,
before it can be non-nullable.
Appropriate test changes to correctly set .realm for Messages the tests
manually create are included here as well.
With django-stubs, these explicit copies of Django’s implicit id
fields are no longer needed for type checking. An exception is the
BigAutoField AbstractUserMessage.id, which is left alone.
This reverts commit c08ee904d8 (#15641).
Signed-off-by: Anders Kaseorg <anders@zulip.com>
Note that django_stubs_ext is required to be placed within common.in
because we need the monkeypatched types in runtime; django-stubs
itself is for type checking only.
In the future, we would like to pin to a release instead of a git
revision, but several patches we've contributed upstream have not
appeared in a release yet.
We also remove the type annotation for RealmAuditLog.event_last_message_id
here instead of earlier because type checking fails otherwise.
Fixes#11560.
Previously, we type the model fields with explicit type annotations
manually with the approximate types. This was because the lack of types
for Django.
django-stubs provides more specific types for all these fields that
incompatible with our previous approximate annotations. So now we can
remove the inline type annotations and rely on the types defined in the
stubs. This allows mypy to infer the types of the model fields for us.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
Previously, we included all three message edit related settings
("allow_message_editing", "message_content_edit_limit_seconds" and
"edit_topic_policy") in the event data and api response irrespective
of which of these settings were changed. Now, we only include changed
settings and separate events are sent for each setting if more than
one of them is changed.
Note that the previous typed in event_schema.py for
`message_content_edit_limit_seconds` incorrectly did not allow `None`
as a value, which is used to encode no limit.
Small follow-up to d86e4ac34d.
get_ makes it sound like it doesn't have side-effects, when these are
actually much like the django ORM .get_or_create function.
SCIMClient is a type-unsafe workaround for django-scim2’s conflation
of SCIM users with Django users. Given that a SCIMClient is not a
UserProfile, it might as well not be a model at all, since it’s only
used to satisfy django-scim2’s request.user.is_authenticated queries.
This doesn’t solve the type safety issue with assigning a SCIMClient
to request.user, nor the performance issue with running the SCIM
middleware on non-SCIM requests. But it reduces the risk of potential
consequences worse than crashing, since there’s no longer a
request.user.id for Django to confuse with the ID of an actual
UserProfile.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
We are no longer writing to or reading the UserStatus.status field,
so we delete that from the model.
Fifth step in making user status `away` a deprecated way to access
`presence_enabled` for clients supporting older servers.
Part of transitioning from 'unavailable' user status feature to
'invisible mode' user presence feature.
To allow `custom_profile_field` to display in user profile popover,
added new boolean field "display_in_profile_summary" in its model class.
In `custom_profile_fields.py`, functions are edited as per conditions,
like currently we can display max 2 `custom_profile_fields` except
`LONG_TEXT` and `USER` type fields.
Default external account custom profile fields made updatable for only
this new field, as previous they were not updatable.
Fixes part of: #21215
This makes use of the type parameters of ValuesQuerySet (a.k.a
_QuerySet) to provide a more accurate type annotation for query_for_ids.
Note that QuerySet[ModelT] is equivalent to _QuerySet[ModelT, ModelT].
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
is_cross_realm_bot_email is just
`email.lower() in settings.CROSS_REALM_BOT_EMAILS` which is the same,
aside of looking at .lower() - which is actually more correct.
These limits don't appear to provide useful security benefits, and
they do impact usability because they prevented email-based users from
replying more than once, or from replying to message more than 5 days
old.
Fixes#2755.
Fixes#19994.
Updates the two UserProfile foreign key fields to have a backward
relation in the MutedUser model by changing the `related_name`
property.
This is a prep commit for removing users with a muted relationship
to the current user from read receipts.
This commit adds do_change_can_remove_subscriber_group function for
changing can_remove_subscribers_group field of a stream. We also add
can_remove_subscribers_group_id field to stream and subscription
objects.
This function will be helpful for writing tests in next commit.
We would add API and UI support to change this setting in further
commits.
This commit udpates can_remove_subscribers_group to be not null.
We already added a migration to set the value of this field for
existing streams and also added a commit to set this field to
admins system group for now while creating streams.
Added a user_list_style personal user setting to the bottom of
Settings > Display settings > Theme section which controls the look
of the right sidebar user list.
The radio button UI includes a preview of what the styles look like.
The setting is intended to eventually have 3 possible values: COMPACT,
WITH_STATUS and WITH_AVATAR; the final value is not yet implemented.
Co-authored-by: Tim Abbott <tabbott@zulip.com>
Previously, an active production Zulip server would experience a class
of deadlocks caused by two or more concurrent bulk update operations
on the UserMessage table.
This is because UPDATE ... SET ... WHERE statements that execute in
parallel take row-level UPDATE locks as they get results; since the
query plans may result in getting rows in different orders between two
queries, this can result in deadlocks.
Some databases allow ORDER BY on their UPDATE ... WHERE statements;
PostgreSQL does not. In PostgreSQL, the answer is to do a sub-select
with an ORDER BY ... FOR UPDATE to ensure consistent ordering on row
locks.
We do this all code paths using bitand or bitor as part of bulk
editing message flags, which should ensure that these concurrent
operations obtain row level locks on the table in the same order.
Fixes#19054.
This uses a more specific type `_StrPromise` to replace `Promise`
providing typing information for lazy translation strings.
In places where the callee evaluates the `_StrPromise` object in all
cases we simply force the evaluation with `str()`. This includes
`JsonableError` that ends up handled by the error handler middleware,
and `internal_send_stream_message` that depends on `check_stream_topic`,
requiring the `topic` to be evaluated anyway. In other siuations, the
callee is expected to be able to handle `StrPromise` explicitly.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
This commit changes the code to consider zero as an invalid value for
message_content_edit_time_limit_seconds. Now to represent the setting that
user can edit the message anytime, the setting value will be "None" in
database and "unlimited" will be passed to API from clients.
mypy_django_plugin cannot resolve the type of the custom manager. We add
a type annotation explicitly to skip the type inferrence.
This fixes the following error:
error: Could not resolve manager type for "zerver.models.UserGroup.objects" [django-manager-missing]
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
The `message` attirbute doesn't get defined in the class where it is used.
The cleanest solution will be using the `Self` type introduced in PEP
673 and type it as something like `ForeignKey[Union[Self, Combinable],
Self]`.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
Adds an API endpoint for accessing read receipts for other users, as
well as a modal UI for displaying that information.
Enables the previously merged privacy settings UI for managing whether
a user makes read receipts data available to other users.
Documentation is pending, and we'll likely want to link to the
documentation with help_settings_link once it is complete.
Fixes#3618.
Co-authored-by: Tim Abbott <tabbott@zulip.com>
This commit adds support to change enable_read_receipts
setting through API and also adds the field to response
of "/register" endpoint so that the setting value
is available to clients.
We now use EVERYONE_GROUP_NAME instead of writing
the actual group name at multiple places, so that we
can have all the group names coded at one place only.
We now use MEMBERS_GROUP_NAME instead of writing
the actual group name at multiple places, so that we
can have all the group names coded at one place only.
We now use MODERATORS_GROUP_NAME instead of writing
the actual group name at multiple places, so that we
can have all the group names coded at one place only.
We now use ADMINISTRATORS_GROUP_NAME instead of writing
the actual group name at multiple places, so that we can
have all the group names coded at one place only.
We now use OWNERS_GROUP_NAME instead of writing
the actual group name at multiple places, so that
we can have all the group names coded at one place
only.
We now use EVERYONE_ON_INTERNET_GROUP_NAME instead of
writing the actual group name at multiple places, so
that we can have all the group names coded at one place
only.
We now use FULL_MEMBERS_GROUP_NAME instead of
writing the actual full members system group
name at multiple places, so that we can have
all the group names coded at one place only.
During a brief period while a custom emoji is being uploaded, it could
be visible to clients even though it was still in the process of being
uploaded, an operation that can fail.
In certain cases, we call `RealmEmoji.save()` before the filename
becomes available. This result in getting invalid urls generated and
flushed. Normally we call it again shortly after, making it harder to
trigger this bug.
Fixes#22552.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
We now send a new user_topic event while muting and unmuting topics.
fetch_initial_state_data now returns an additional user_topics array to
the client that will maintain the user-topic relationship data.
This will support any future addition of new features to modify the
relationship between a user-topic pair.
This commit adds the relevent backend code and schema for the new
event.
Now that we can assume Python 3.6+, we can use the
email.headerregistry module to replace hacky manual email address
parsing.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
Fixes#21266.
We want to tie the prereg_user to the MultiUseInvite directly rather
than to the MultiUserInvite's confirmation object, because the latter is
not possible. This is because the flow is that after going through the
multiuse invite link, the PreregistrationUser is created together with a
Confirmation object, creating a confirmation link (via
create_confirmation_link) to which then the user is redirected to finish
account creation. This means that the PreregistrationUser is already
tied to a Confirmation, so that attribute is occupied.
This commit removes "role" field from subscription
objects since we are not moving forward with stream
administrator concept and instead working on new
permssions model as per #19525.
This commit removes WILDCARD_MENTION_POLICY_STREAM_ADMINS
option of wildcard_mention_policy since we are not moving
forward with stream administrator concept and instead working
on new permssions model as per #19525.
We also add a migration to change wildcard_mention_policy of
existing realms to WILDCARD_MENTION_POLICY_ADMINS. This change
is fine since we were already treating both the setting values
as same as stream admin concept was not implemented completely.
This commit removes the is_stream_admin property of Subscription
model and also updates check_stream_access_for_delete_or_update
to not return true when is_stream_admin is True.
We also removes the relevant tests.
This change is done as we would not be moving forward with the
stream administrator concept as we have decided to modify the
permissions model as per #19525.
To prevent new contributors who are adding new user settings from
being confused about the legacy dicts for display settings and
notifications settings, we create `modern_settings` and
`modern_notifications_settings` dicts with comments documenting
that new user settings should be added to one of them.
Note that the `modern_notification_settings` is a placeholder for
new notifications settings as there have been no new notification
settings added since 430c5cb8 (when the legacy dicts were created)
and is currently annotated as `Dict[str, Any]` (which can be
removed if/when a new setting is added).
TODO: For now, we import `_QuerySet` as `ValuesQuerySet`. But there
is a convenient reexport of `ValuesQuerySet` in `django_stubs_ext`
that does the same thing. Once we get django-stubs integrated,
we should import `ValuesQuerySet` from `django_stubs_ext` instead.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
This ensure that the return type is compatible with the
actual type of `realm.realmdomain_set.values`.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
Added 2 new functions which check if a flag is present in a
user message or not using bit comparison.
Refactor `where_unread/starred/active_push_notification` to use
these functions.
Existing tests are enough to check if these functions work correctly.
This function is oblivious to the existence of ArchivedAttachment, which
is incorrect. A file can be removed if and only if it is not referenced
by any Messages or ArchivedMessages.
Add none-checks, rename variables (to avoid redefinition of
the same variable with different types error), add necessary
type annotations.
This is a part of #18777.
Signed-off-by: Zixuan James Li <359101898@qq.com>
Adds `want_advertise_in_communities_directory` to the realm model
to track organizations that give permission to be listed on such
a site / directory on zulip.com.
Adds a checkbox to the organization profile admin for
organizations to give permission to be advertised in the
Zulip communities directory.
Adds a help center article about the Zulip communities directory
and uses a shared intro documentation file to create sections in
the articles on creating an organization profile and moderating
open organizations.
Co-authored-by: Alya Abbott <alya@zulip.com>
The default of a Stream is to be public - having
history_public_to_subscribers default to False is inconsistent with
that. The defaults on the model should generally be consistent.
This commit changes the invite API to accept invitation
expiration time in minutes since we are going to add a
custom option in further commits which would allow a user
to set expiration time in minutes, hours and weeks as well.
Added a setting to the bottom of Settings > Display settings > Theme section
to display the reacting users on a message when numnber of reactions are
small.
This is a preparatory commit for #20980.
The relevant function is waiting to be merged in #21299 - but we have
already used it on Zulip Cloud, creating RealmAuditLog entries with the
number 107 and thus should reserve it before another PR takes
it for another purpose, creating confusion in the logs.
This demonstrates a way to resolve the long-standing issue
of typing higher-order identity functions without using
`cast` and in a type-safe manner for decorators in `cache.py`.
Signed-off-by: Zixuan James Li <359101898@qq.com>
This is necessary for the migration 0386_fix_attachment_caches to run,
and likely makes more convenient any future parallel code interacting
with both Attachment and ArchivedAttachment.
Previously, Attachment.is_realm_public and its cousin,
Attachment.is_web_public, were properties that began as False and
transitioned to True only when a message containing a link to the
attachment was sent to the appropriate class of stream, or such a link
was added as part of editing a message.
This pattern meant that neither field was updated in situations where
the access permissions for a message changed:
* Moving the message to a different stream.
* Changing the permissions for a stream containing links to the message.
This correctness issue has limited security impact, because uploaded
files are secured both by a random URL and by these access checks.
To fix this, we reformulate these fields as a cache, with code paths
that change the permissions affecting an attachment responsible for
setting these values to the `None` (uncached) state. We prefer setting
this `None` state over computing the correct permissions, because the
correct post-edit permissions are a function of all messages
containing the attachment, and we don't want to be responsible for
fetching all of those messages in the edit code paths.
he possibility for it being null was likely an oversight -- it should
have been removed after the early migrations to backfill the field
when it was added.
We've confirmed there are no existing violations of this invariant in
Zulip Cloud.
This commit creates a new TypedDict RealmPlaygroundDict for realm
playground objects. Now the list of playgrounds in the events sent
to clients and the "added_playground" field of RealmAuditLog entry
use RealmPlaygroundDict instead of Dict.
This is in a separate commit to make deployment easier. It ensures that
this is only marked non-null after the backfill migration (backfilling
.uuid for all old UserProfiles) runs - which was added in the previous
commit.
This will make it convenient to add a handful of organizations to the
beta of this feature during its first few weeks to try to catch bugs,
before we open it to everyone in Zulip Cloud.
The correct return type of get_realm_domains should
be List[Dict[str, Union[bool, str]]] instead of
List[Dict[str, str]] because allowed_subdomains is
a bool field not str.
This model is by designed intended to exist on a 1:1 relationship with
Realms, and we attempt to ensure that with application code, but we
should have a unique constraint too, since a database with duplicate
such entries would be corrupted.
We do this via the standard Django OneToOneField.
The database value for expiry_date is None for the invite
that will never expire and the clients send -1 as value
in the API similar to the message retention setting.
Also, when passing invite_expire_in_days as an argument
in various functions, invite_expire_in_days is passed as
-1 for "Never expires" option since invite_expire_in_days
is an optional argument in some functions and thus we cannot
pass "None" value.
Mateusz Mandera
Aman Agrawal
Anders Kaseorg
Sahil Batra
Lauryn Menard
Abhijeet Prasad Bodas
Prakhar Pratyush
Daniil Fadeev
Alex Vandiver
Tim Abbott
Zixuan James Li
PIG208
evykassirer
yogesh sirsat
Jason Hildebrand
Raghav Luthra
Christopher Chong
Dinesh
Kartik Srivastava
byshen-dev
Steve Howell