Commit Graph

1453 Commits

Author SHA1 Message Date
Anders Kaseorg 143baa4243 python: Convert translated positional {} fields to {named} fields.
Translators benefit from the extra information in the field names, and
need the reordering freedom that isn’t available with multiple
positional fields.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-07-18 15:19:07 -07:00
Prakhar Pratyush 4c9d26ce17 mention: Send notifications for @topic wildcard mentions.
This commit completes the notifications part of the @topic
wildcard mention feature.

Notifications are sent to the topic participants for the
@topic wildcard mention.
2023-07-17 09:39:24 -07:00
Steve Howell b742f1241f realm emoji: Use a single cache for all lookups.
The active realm emoji are just a subset of all your
realm emoji, so just use a single cache entry per
realm.

Cache misses should be very infrequent per realm.

If a realm has lots of deactivated realm emoji, then
there's a minor expense to deserialize them, but that
is gonna be dwarfed by all the other more expensive
operations in message-send.

I also renamed the two related functions.  I erred on
the side of using somewhat verbose names, as we don't
want folks to confuse the two use cases. Fortunately
there are somewhat natural affordances to use one or
the other, and mypy helps too.

Finally, I use realm_id instead of realm in places
where we don't need the full Realm object.
2023-07-17 09:35:53 -07:00
Steve Howell e988cf9b0a emoji cache: Don't join to UserProfile table.
We only need author id, and anything else in the table
would be possibly stale anyway.
2023-07-17 09:35:53 -07:00
Anders Kaseorg 7e707270f0 models: Convert deprecated index_together option to indexes.
index_together is slated for removal in Django 5.1:
https://docs.djangoproject.com/en/4.2/internals/deprecation/#deprecation-removed-in-5-1

We set the optional index names to match the previously generated
index names to avoid adding new migrations.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-07-12 07:12:43 -07:00
Sahil Batra 2e4f7f6336 user_groups: Remove "@" from name of role-based system groups.
This commit removes "@" from name of role-based system groups
since we have added a restricion on having user group names
starting with "@" in the previous commit as they look odd in
mention syntax.

We also add a migration in this commit to update the name of
role-based system groups in existing realms to remove "@"
from the name. This migration also updates the names of
non-system user groups by removing the invalid prefixes
from their names and if there is a group already with that
name, we insted name the group as "group:{group_id}".

Fixes #26148.
2023-07-11 13:46:02 -07:00
Sahil Batra 929bf1243e user_groups: Disallow certain prefixes in group name.
We do not allow user group names to start with "@", "role:",
"user:", "stream:" and "channel:".

Group names starting with "@" look odd in mentions and
"role:", "user:" and "stream:" prefixes are reserved for
system groups which will be used in the new groups-based
permission model. We do not allow "channel:" prefix for
now just to be safe in a case where we use it instead of
"stream:" prefix for stream based groups in future.

Fixes part of #26148.
2023-07-11 13:46:02 -07:00
Sahil Batra ea3a7a9e6f user_groups: Add API restrictions for long user group names.
Previously we had database level restriction on length of
user group names. Now we add the same restriction to API
level as well, so we can return a better error response.
2023-07-11 13:46:02 -07:00
Steve Howell 89381a8072 cache: Eliminate get-stream-by-name cache.
We remove the cache functionality for the
get_realm_stream function, and we also change it to
return a thin Stream object (instead of calling
select_related with no arguments).

The main goal here is to remove code complexity, as we
have been prone to at least one caching validation bug
related to how Realm and UserGroup interact. That
particular bug was more theoretical than practical in
terms of its impact, to be clear.

Even if we were to be perfectly disciplined about only
caching thin stream objects and always making sure to
delete cache entries when stream data changed, we would
still be prone to ugly situations like having
transactions get rolled back before we delete the cache
entry. The do_deactivate_stream is a perfect example of
where we have to consider the best time to unset the
cache. If you unset it too early, then you are prone to
races where somebody else churns the cache right before
you update the database. If you set it too late, then
you can have an invalid entry after a rollback or
deadlock situation. If you just eliminate the cache as
a moving part, that whole debate is moot.

As the lack of test changes here indicates, we rarely
fetch streams by name any more in critical sections of
our code.

The one place where we fetch by name is in loading the
home page, but that is **only** when you specify a
stream name. And, of course, that only causes about an
extra millisecond of time.
2023-07-11 13:45:40 -07:00
Steve Howell 046e4c715b cache: Use DB for all bulk get-stream-by-name queries.
This changes bulk_get_streams so that it just uses the
database all the time.  Also, we avoid calling
select_related(), so that we just get back thin and
tidy Stream objects with simple queries.

About not caching any more:

It's actually pretty rare that we fetch streams by name
in the main application. It's usually API requests that
send in stream names to find more info about streams.

It also turns out that for large queries (>= ~30 rows
for my measurements) it's more efficent to hit the
database than memcached. The database is super fast at
scale; it's just the startup cost of having Django
construct the query, and then having the database do
query planning or whatever, that slows us down. I don't
know the exact bottleneck, but you can clearly measure
that one-row queries are slow (on the order of a full
millisecond or so) but the marginal cost of additional
rows is minimal assuming you have a decent index (20
microseconds per row on my droplet).

All the query-count changes in the tests revolve around
unsubscribing somebody from a stream, and that's a
particularly odd use case for bulk_get_streams, since
you generally unsubscribe from a single stream at a
time. If there are some use cases where you do want to
unsubscribe from multiple streams, we should move
toward passing in stream ids, at least from the
application. And even if we don't do that, our cost for
most queries is a couple milliseconds.
2023-07-11 13:45:40 -07:00
Zixuan James Li 3349ac9f86 user_groups: Audit UserGroup group based setting changes.
This add audit log entries when any group based setting of a user group
is updated. We store both the old and new values in extra_data, along
with the name of that setting. Entries populated during user group creation
are hardcoded to track "can_mention_group".

Potentially we can adjust "set_defaults_for_group_settings" so that it
populates realm audit logs with it, but that is out of scope for this change.

We use an atomic transaction so that the audit logs are committed
together with the updates.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2023-07-11 08:56:55 -07:00
Zixuan James Li 71de14ab43 models: Add modified_user_group.
This also adds the supported event types for changes to UserGroup.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2023-07-11 08:56:55 -07:00
Prakhar Pratyush 179d5cb37d mention: Replace 'wildcards' with 'stream_wildcards'.
This prep commit replaces the 'wildcard' keyword in the codebase
with 'stream_wildcard' at some places for better readability, as
we plan to introduce 'topic_wildcards' as a part of the
'@topic mention' project.

Currently, 'wildcards = ["all", "everyone", "stream"]' which is an
alias to mention everyone in the stream, hence better renamed as
'stream_wildcards'.

Eventually, we will have:
'stream_wildcard' as an alias to mention everyone in the stream.
'topic_wildcard' as an alias to mention everyone in the topic.
'wildcard' refers to 'stream_wildcard' and 'topic_wildcard' as a whole.
2023-07-03 22:03:17 -07:00
Sahil Batra 2763f9b575 user_groups: Add can_mention_group setting.
This commit adds a new can_mention_group setting which will be
used to determine who can mention a particular group.

Fixes a part of #25927.
2023-06-30 17:28:33 -07:00
Ujjawal Modi a361c23aac alert_words: Refactor the code to flush alert_words cache.
Subsequent commits will add "on_delete=models.RESTRICT"
relationships, which will result in the AlertWord
objects being deleted after Realm has been deleted from
the database.

In order to handle this, we update realm_alert_words_cache_key,
realm_alert_words_automaton_cache_key, and flush_realm_alert_words
functions to accept realm_id as parameter instead of realm
object, so that  the code for flushing the cache works even
after the realm is deleted. This change is fine because
eventually only realm_id is used by these functions and there
is no need of the complete realm object.
2023-06-28 18:03:32 -07:00
Ujjawal Modi f7346f36fc attachments: Refactor code for flushing used_upload_space cache.
Subsequent commits will add "on_delete=models.RESTRICT"
relationships, which will result in the Attachment
objects being deleted after Realm has been deleted from
the database.

In order to handle this, we update
get_realm_used_upload_space_cache_key function to accept
realm_id as parameter instead of realm object, so that
the code for flushing the cache works even after the
realm is deleted. This change is fine because eventually
only realm_id is used by this function and there is no
need of the complete realm object.
2023-06-28 18:03:32 -07:00
Ujjawal Modi 535a088d0b bots: Refactor code for flushing bots cache.
Subsequent commits will add "on_delete=models.RESTRICT"
relationships, which will result in the UserProfile
objects being deleted after Realm has been deleted from
the database.

In order to handle this, we update bot_dicts_in_realm_cache_key
function to accept realm_id as parameter instead of realm
object, so that  the code for flushing the cache works even
after the realm is deleted. This change is fine because
eventually only realm_id is used by this function and there is
no need of the complete realm object.
2023-06-28 18:03:32 -07:00
Ujjawal Modi fd0434a052 realm_emoji: Refactor code for flushing realm_emoji cache.
Subsequent commits will add "on_delete=models.RESTRICT"
relationships, which will result in the RealmEmoji
objects being deleted after Realm has been deleted from
the database.

In order to handle this, we update get_realm_emoji_dicts,
get_realm_emoji_cache_key, get_active_realm_emoji_cache_key,
get_realm_emoji_uncached and get_active_realm_emoji_uncached
functions to accept realm_id as parameter instead of realm
object, so that the code for flushing the cache works even
after the realm is deleted. This change is fine because
eventually only realm_id is used by these functions and
there is no need of the complete realm object.
2023-06-28 18:03:32 -07:00
Sahil Batra 138a67d97e types: Add default_group_name field to GroupPermissionSetting type.
This commit adds default_group_name field to GroupPermissionSetting
type which will be used to store the name of the default group for
that setting which would in most cases be one of the role-based
system groups. This will be helpful when we would have multiple
settings and we would need to set the defaults while creating
realm and streams.
2023-06-28 18:03:32 -07:00
Lauryn Menard d3f7cfccbc zerver: Update comments with "private message" or "PM".
Updates comments/doc-strings that use "private message" or "PM" in
files in the `/zerver` directory to instead use "direct message".
2023-06-23 11:24:13 -07:00
Anders Kaseorg c09e7d6407 codespell: Correct “requestor” to “requester”.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-06-20 16:17:55 -07:00
Zixuan James Li 5901ffb0ab models: Fix typo in comment.
"This consistent ordering is important to prevent to prevent"
->
"This consistent ordering is important to prevent"
2023-06-15 17:50:34 -04:00
Prakhar Pratyush 134058b06d settings: Configure 'enable_followed_topic_audible_notifications'.
This commit makes it possible for users to control the
audible desktop notifications for messages sent to followed topics
via a global notification setting.

There is no support for configuring this setting through the UI yet.
2023-06-13 18:01:41 -07:00
Prakhar Pratyush a848c744c3 settings: Configure 'enable_followed_topic_desktop_notifications'.
This commit makes it possible for users to control the
visual desktop notifications for messages sent to followed topics
via a global notification setting.

There is no support for configuring this setting through the UI yet.
2023-06-13 18:01:41 -07:00
Prakhar Pratyush e71d3ada87 settings: Add wildcard mention notifications for the followed topics.
This commit makes it possible for users to control the wildcard
mention notifications for messages sent to followed topics
via a global notification setting.

There is no support for configuring this setting
through the UI yet.
2023-06-13 18:01:41 -07:00
Prakhar Pratyush d73c715dc2 settings: Add push notifications for the followed topics.
This commit makes it possible for users to control
the push notifications for messages sent to followed topics
via a global notification setting.

There is no support for configuring this setting
through the UI yet.
2023-06-13 18:01:41 -07:00
Prakhar Pratyush 5e5538886f settings: Add email notifications for the followed topics.
This commit makes it possible for users to control
the email notifications for messages sent to followed topics
via a global notification setting.
Although there is no support for configuring this setting
through the UI yet.

Add five new fields to the UserBaseSettings class for
the "followed topic notifications" feature, similar to
stream notifications. But this commit consists only of
the implementation of email notifications.
2023-06-13 18:01:41 -07:00
Zixuan Li e39e04c3ce
migration: Add `extra_data_json` for audit log models.
Note that we use the DjangoJSONEncoder so that we have builtin support
for parsing Decimal and datetime.

During this intermediate state, the migration that creates
extra_data_json field has been run. We prepare for running the backfilling
migration that populates extra_data_json from extra_data.

This change implements double-write, which is important to keep the
state of extra data consistent. For most extra_data usage, this is
handled by the overriden `save` method on `AbstractRealmAuditLog`, where
we either generates extra_data_json using orjson.loads or
ast.literal_eval.

While backfilling ensures that old realm audit log entries have
extra_data_json populated, double-write ensures that any new entries
generated will also have extra_data_json set. So that we can then safely
rename extra_data_json to extra_data while ensuring the non-nullable
invariant.

For completeness, we additionally set RealmAuditLog.NEW_VALUE for
the USER_FULL_NAME_CHANGED event. This cannot be handled with the
overridden `save`.

This addresses: https://github.com/zulip/zulip/pull/23116#discussion_r1040277795

Note that extra_data_json at this point is not used yet. So the test
cases do not need to switch to testing extra_data_json. This is later
done after we rename extra_data_json to extra_data.

Double-write for the remote server audit logs is special, because we only
get the dumped bytes from an external source. Luckily, none of the
payload carries extra_data that is not generated using orjson.dumps for
audit logs of event types in SYNC_BILLING_EVENTS. This can be verified
by looking at:

`git grep -A 6 -E "event_type=.*(USER_CREATED|USER_ACTIVATED|USER_DEACTIVATED|USER_REACTIVATED|USER_ROLE_CHANGED|REALM_DEACTIVATED|REALM_REACTIVATED)"`

Therefore, we just need to populate extra_data_json doing an
orjson.loads call after a None-check.

Co-authored-by: Zixuan James Li <p359101898@gmail.com>
2023-06-07 12:14:43 -07:00
Sahil Batra 48e99657ad events: Remove realm_community_topic_editing_limit_seconds.
This commit removes realm_community_topic_editing_limit_seconds
field from register response since topic edit limit is now
controlled by move_messages_within_streams_limit_seconds
setting.
We also remove DEFAULT_COMMUNITY_TOPIC_EDITING_LIMIT_SECONDS
constant since it is no longer used.
2023-05-25 17:26:21 -07:00
Alex Vandiver c978bfaa32 models: Add a unique index on UserProfile.api_key.
This prevents `get_user_profile_by_api_key` from doing a sequential
scan.

Doing this requires moving the generation of initial api_key values
into the column definition, so that even bare calls to
`UserProfile.objects.create` (e.g. from tests) call appropriately
generate a random initial value.
2023-05-19 11:11:04 -07:00
Sahil Batra 7f01b3fb63 users: Set tos_version to -1 for users who have not logged-in yet.
We now set tos_version to "-1" for imported users and the ones
created using API or using other methods like LDAP, SCIM and
management commands. This value will help us to allow users to
change email address visibility setting during first login.
2023-05-16 13:52:56 -07:00
Lauryn Menard 90cc2716f0 scheduled-messages: Update API dicts for `failed` boolean field.
Adds the `failed` boolean from the ScheduledMessage to the API dict
returned by scheduled message events and register response, and by
fetching the user's scheduled messages.

`failed` will only be true when the server has tried to send the
scheduled message and failed due to an error.
2023-05-12 15:48:59 -07:00
Tim Abbott 602e4c2aa3 scheduled_messages: Add focused scheduled message indexes. 2023-05-09 13:48:28 -07:00
Tim Abbott 835f62617e scheduled_messages: Store the final delivered message ID.
This could be useful for debugging problems with the system
operationally.
2023-05-09 13:48:28 -07:00
Tim Abbott 7051d3416b scheduled_messages: Add reasonable failure handling.
Previously, it seemed possible for the scheduled messages API to try
to send infinite copies of a message if we had the very poor luck of a
persistent failure happening after a message was sent.

The failure_message field supports being able to display what happened
in the scheduled messages modal, though that's not exposed to the API
yet.
2023-05-09 13:48:28 -07:00
Lauryn Menard 02fafb0376 models: Update the references for API dicts for scheduled messages. 2023-05-09 07:36:05 -07:00
Mateusz Mandera 414658fc8e scheduled_message: Handle attachments properly.
Fixes #25414.

We add Attachment.scheduled_messages relation to track ScheduledMessages
which reference the attachment.

The import bits can be done after merging this, by updating #25345.
2023-05-08 09:56:02 -07:00
Aman Agrawal d60d6e9115 urls: Add new endpoint to create scheduled messages.
This will help us remove scheduled message and reminder logic
from `/messages` code path.

Removes `deliver_at`/`defer_until` and `tz_guess` parameters. And
adds the `scheduled_delivery_timestamp` instead. Also updates the
scheduled message dicts to return `scheduled_delivery_timestamp`.

Also, revises some text in `/delete-scheduled-message` endpoint
and in the `ScheduledMessage` schema in the API documentation.
2023-04-28 17:25:00 -07:00
Lauryn Menard 7739703111 scheduled-messages: Update scheduled message objects in the API for type.
Updates the objects in the API for scheduled messages so that those
for stream messages return the `to` property as an integer since it
is always the unique stream ID and so that those for direct messages
do not have a `topic` property since direct messages never have a
topic.

Also makes small update so that web app scheduled messages overlay
has the correct stream ID.
2023-04-28 17:25:00 -07:00
Aman Agrawal 4cb238fb6e models: Add method to convert ScheduleMessage objects into dicts. 2023-04-28 17:25:00 -07:00
Alex Vandiver 4f2417cfc4 soft_reactivation: Add a partial index to speed up event lookups.
The full auditlog table is moderately large, and the previously-chosen
index (on `modified_user_id`) is not terribly specific.
2023-04-28 12:43:34 -07:00
Sahil Batra 850e0046eb models: Add ORG_TYPE_IDS constant field to Realm.
This commit adds ORG_TYPE_IDS constant field to Realm class
such that it can be used when we want to validate the org_type
passed in request. This was previously defined in realm.py, but
we move it inside Realm class such that we can use it at other
places as well.
2023-04-27 12:28:37 -07:00
Mateusz Mandera a9f40a64fd presence: Support null values in UserPresence. 2023-04-26 14:26:47 -07:00
Tim Abbott 027b67be80 presence: Rewrite the backend data model.
This implements the core of the rewrite described in:

For the backend data model for UserPresence to one that supports much
more efficient queries and is more correct around handling of multiple
clients.  The main loss of functionality is that we no longer track
which Client sent presence data (so we will no longer be able to say
using UserPresence "the user was last online on their desktop 15
minutes ago, but was online with their phone 3 minutes ago").  If we
consider that information important for the occasional investigation
query, we have can construct that answer data via UserActivity
already.  It's not worth making Presence much more expensive/complex
to support it.

For slim_presence clients, this sends the same data format we sent
before, albeit with less complexity involved in constructing it.  Note
that we at present will always send both last_active_time and
last_connected_time; we may revisit that in the future.

This commit doesn't include the finalizing migration, which drops the
UserPresenceOld table.
The way to deploy is to start the backfill migration with the server
down and then start the server *without* the user_presence queue worker,
to let the migration finish without having new data interfering with it.
Once the migration is done, the queue worker can be started, leading to
the presence data catching up to the current state as the queue worker
goes over the queued up events and updating the UserPresence table.

Co-authored-by: Mateusz Mandera <mateusz.mandera@zulip.com>
2023-04-26 14:26:47 -07:00
Zixuan James Li 6f42542320 migrations: Remove filter_format_validator.
This removes the validator argument for 0423_realmfilter_url_template,
which do not really alter the database schema. It otherwise fails
the migration because the filter_format_validator function is removed.

Migration 0094_realm_filter_url_validator is modified because we can no
longer refer to filter_format_validator.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2023-04-19 12:20:49 -07:00
Zixuan James Li 268f858f39 linkifier: Support URL templates for linkifiers.
This swaps out url_format_string from all of our APIs and replaces it
with url_template. Note that the documentation changes in the following
commits  will be squashed with this commit.

We change the "url_format" key to "url_template" for the
realm_linkifiers events in event_schema, along with updating
LinkifierDict. "url_template" is the name chosen to normalize
mixed usages of "url_format_string" and "url_format" throughout
the backend.

The markdown processor is updated to stop handling the format string
interpolation and delegate the task template expansion to the uri_template
library instead.

This change affects many test cases. We mostly just replace "%(name)s"
with "{name}", "url_format_string" with "url_template" to make sure that
they still pass. There are some test cases dedicated for testing "%"
escaping, which aren't relevant anymore and are subject to removal.
But for now we keep most of them as-is, and make sure that "%" is always
escaped since we do not use it for variable substitution any more.

Since url_format_string is not populated anymore, a migration is created
to remove this field entirely, and make url_template non-nullable since
we will always populate it. Note that it is possible to have
url_template being null after migration 0422 and before 0424, but
in practice, url_template will not be None after backfilling and the
backend now is always setting url_template.

With the removal of url_format_string, RealmFilter model will now be cleaned
with URL template checks, and the old checks for escapes are removed.

We also modified RealmFilter.clean to skip the validation when the
url_template is invalid. This avoids raising mulitple ValidationError's
when calling full_clean on a linkifier. But we might eventually want to
have a more centric approach to data validation instead of having
the same validation in both the clean method and the validator.

Fixes #23124.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2023-04-19 12:20:49 -07:00
Zixuan James Li a19c80df43 linkifiers: Add url_template field.
This will later be used to expand matching linkifier patterns.
Making it nullable for now, but we will make it required in
the APIs.

As a part of this transition, we temporarily make url_format_string
nullable as well, which will be later removed. This allows us to
switch to populating url_template without caring about passing
url_format_string.

Note that the validators are imported in the migration because Django
otherwise diffs it and considers the schema to be different, generating
a migration, failing the "tools/test-migrations" test.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2023-04-19 12:20:49 -07:00
Joelute afbd0a9b43 user_settings: Add web_mark_read_on_scroll_policy field.
This is a backend change that will help support the new
`Mark messages as read on scroll` user setting.
2023-04-18 18:32:02 -07:00
Lauryn Menard 2c043c6242 message-type: Add support for "direct" as value for type parameter.
For endpoints with a `type` parameter to indicate whether the message
is a stream or direct message, `POST /typing` and `POST /messages`,
adds support for passing "direct" as the preferred value for direct
messages, group and 1-on-1.

Maintains support for "private" as a deprecated value to indicate
direct messages.

Fixes #24960.
2023-04-18 12:29:33 -07:00
Mateusz Mandera 72d56d5d59 auth: Remove Realm.AUTHENTICATION_FLAGS class attribute.
With the removal of the authentication_methods bitfield this is now
useless and just duplicates AUTH_BACKEND_NAME_MAP keys.
2023-04-18 09:22:56 -07:00
Mateusz Mandera ffa3aa8487 auth: Rewrite data model for tracking enabled auth backends.
So far, we've used the BitField .authentication_methods on Realm
for tracking which backends are enabled for an organization. This
however made it a pain to add new backends (requiring altering the
column and a migration - particularly troublesome if someone wanted to
create their own custom auth backend for their server).

Instead this will be tracked through the existence of the appropriate
rows in the RealmAuthenticationMethods table.
2023-04-18 09:22:56 -07:00
Aman Agrawal 555041c081 models: models: Extract get_recipient_ids method.
This will be used by scheduled messages to as it is used by drafts.
2023-04-14 17:38:37 -07:00
Aman Agrawal a0eac4aaf4 models: Rename _type to recipient_type_str. 2023-04-14 17:38:37 -07:00
Aman Agrawal b63f440fb1 models: Allow scheduled msgs to store rendered content.
This is required by the client to display a list of currently
scheduled messages.
2023-04-14 17:38:37 -07:00
Anders Kaseorg 0a3dc8a944 ruff: Fix DJ012 Order of model's inner classes, methods, and fields.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-04-12 17:32:38 -07:00
Sahil Batra 5237e9008f types: Add allow_nobody_group field to GroupPermissionSetting. 2023-04-11 09:02:09 -07:00
Lauryn Menard f391bfeec6 emails: Add new onboarding email with guide for organization type.
Adds a new welcome email, `onboarding_zulip_guide`, to be sent four
days after a new user registers with a Zulip organization if the
organization has specified a particular organization type that has
a guide in the corporate `/for/.../` pages. If there is no guide,
then no email is scheduled or sent.

The current `for/communities/` page is not very useful for users
who are not organization administrators, so these onboarding guide
emails are further restricted for those organization types to
only go to new users who are invited/registered as admins for the
organzation.

Adds two database queries for new user registrations: one to get
the organization's type and one to create the scheduled email.

Adds two email logs because the email is sent both to a new user
who registers with an existing organization and to the organization
owner when they register a new organization.

Co-authored by: Alya Abbott <alya@zulip.com>
2023-04-10 08:38:09 -07:00
Sahil Batra bed2bf64c4 user_groups: Add "Nobody" system user group.
This commit adds code to create a "Nobody" system user group
to realms which will be used in settings to represent "Nobody"
option.

We also add a migration to add this group to existing realms.
2023-03-28 14:26:22 -07:00
Sahil Batra 7f1bf9d6ab models: Add PreregistrationRealm class.
This commit adds PreregistrationRealm class which will be
similar to PreregistrationUser and will store initial
information of the realm before its creation as we are
changing the organization creation flow as per #24307.

Fixes part of #24307.
2023-03-27 15:44:42 -07:00
Abhijeet Prasad Bodas ba0dd70904 user_topics: Introduce visibility policy enum subclass in models.py.
This commit gives more readable code than using the `VISIBILITY_POLICY`
prefix.
This commit does not alter the database schema.
2023-03-14 17:50:45 -07:00
Prakhar Pratyush ae72777c77 user_settings: Automate 'Include realm name in message email subject'.
Currently, there is a checkbox setting for whether to
"Include realm name in subject of message notification emails".

This commit replaces the checkbox setting with a dropdown
having values: Automatic [default], Always, Never.

The Automatic option includes the realm name if, and only if,
there are multiple Zulip realms associated with the user's email.

Tests are added and(or) modified.

Fixes: #19905.
2023-03-14 17:48:57 -07:00
Anders Kaseorg 2d9b2a2a05 models: Remove type prefixes from __str__ values.
The Django convention is for __repr__ to include the type and __str__
to omit it.  In fact its default __repr__ implementation for models
automatically adds a type prefix to __str__, which has resulted in the
type being duplicated:

    >>> UserProfile.objects.first()
    <UserProfile: <UserProfile: emailgateway@zulip.com <Realm: zulipinternal 1>>>

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-03-08 22:56:55 -08:00
Daniil Fadeev dcb72448d8 giphy: Fix "PG13" typo. 2023-03-06 10:52:52 -08:00
Daniil Fadeev d606f5ba42 i18n: Add new markup for lazy translation of giphy ratings.
Also, it adds phrases such as 'rated Y' to the ignored list.
Otherwise, the linter would require to write it as 'rated y'.
2023-03-06 10:52:52 -08:00
Anders Kaseorg d3efd4c095 python: Import F, Q, QuerySet from their canonical module.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-03-05 14:46:28 -08:00
Alex Vandiver 52b8a84219 models: Drop unique index on user/message/emoji_name.
c7d0192755 added the unique constraint on
`user_profile_id,message_id,reaction_type,emoji_code`, but left the
existing constraint on `user_profile_id,message_id,emoji_name`.  As
explained in the comment added in 3cd543ee98, `emoji_name` cannot be
trusted to be unique, as it is possible to have an Unicode emoji
reaction and a custom emoji with the same name on a message.

Remove the overly-constraining unique index, now that c7d0192755 has
provided the correct one.
2023-02-26 17:10:01 -08:00
Sahil Batra 36584a3571 registration: Add code to set email_address_visibility during signup.
This commit adds backend code to set email_address_visibility when
registering a new user. The realm-level default and the value of
source profile gets overridden by the value user selected during
signup.
2023-02-24 09:23:34 -08:00
Anders Kaseorg c1675913a2 web: Move web app to ‘web’ directory.
Ever since we started bundling the app with webpack, there’s been less
and less overlap between our ‘static’ directory (files belonging to
the frontend app) and Django’s interpretation of the ‘static’
directory (files served directly to the web).

Split the app out to its own ‘web’ directory outside of ‘static’, and
remove all the custom collectstatic --ignore rules.  This makes it
much clearer what’s actually being served to the web, and what’s being
bundled by webpack.  It also shrinks the release tarball by 3%.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-02-23 16:04:17 -08:00
Sahil Batra 0cf99cf5c3 streams: Refactor code to set group-based stream settings.
We add stream_permission_group_settings object which is
similar to property_types framework used for realm settings.

This commit also adds GroupPermissionSetting dataclass for
defining settings inside stream_permission_group_settings.

We add "do_change_stream_group_based_setting" function which
is called in loop to update all the group-based stream settings
and it is now used to update 'can_remove_subscribers_group'
setting instead of "do_change_can_remove_subscribers_group".

We also change the variable name for event_type field of
RealmAuditLog objects to STREAM_GROUP_BASED_SETTING_CHANGED
since this will be used for all group-based stream settings.

'property' field is also added to extra_data field to identify
the setting for which RealmAuditLog object was created.

We will add a migration in further commits which will add the
property field to existing RealmAuditLog objects created for
changing can_remove_subscribers_group setting.
2023-02-22 12:17:46 -08:00
Abhijeet Prasad Bodas 80bf6b0777 user topic: Add case insensitive UNIQUE constraint.
This will allow us to rely on the database to detect duplicate
`UserTopic`s (with the same `topic_name` with different cases)
and thus correctly throw IntegrityErrors when expected.
This is also important from a correctness point of view, since as
of now, when checking if topic is muted or requesting the backend for
muting a topic, the frontend does not check for case insensitivity.

There might exist duplicate UserTopics (in a case insensitive sense)
which need are removed before creating the new index.
The migration was tested manually using `./manage.py shell`.
2023-02-20 21:04:13 -08:00
Alex Vandiver f8a16fb9af model: Handle parsed user-agents longer than 30 characters.
The Client.name field is only 30 characters long, but there is no
limit to the length of parsed User-Agent value which we may attempt to
store in it.  This can cause requests with long user-agents to 500
when the creation of the Client row fails.

Truncate the name at 30 characters for the cache key, and passing
`name` to `get_or_create`.
2023-02-14 14:58:15 -08:00
Sahil Batra 9d1dc20e6e settings: Remove realm-level email_address_visibility setting.
This was replaced by the new user-level version in recent commits.

Fixes #20035.
Fixes #18149.
2023-02-10 17:40:33 -08:00
Sahil Batra 0ed5f76063 settings: Add backend code for using user email_address_visibility setting.
This commits update the code to use user-level email_address_visibility
setting instead of realm-level to set or update the value of UserProfile.email
field and to send the emails to clients.

Major changes are -

- UserProfile.email field is set while creating the user according to
RealmUserDefault.email_address_visbility.

- UserProfile.email field is updated according to change in the setting.

- 'email_address_visibility' is added to person objects in user add event
and in avatar change event.

- client_gravatar can be different for different users when computing
avatar_url for messages and user objects since email available to clients
is dependent on user-level setting.

- For bots, email_address_visibility is set to EVERYONE while creating
them irrespective of realm-default value.

- Test changes are basically setting user-level setting instead of realm
setting and modifying the checks accordingly.
2023-02-10 17:35:49 -08:00
Sahil Batra ea0b2286e0 settings: Add email_address_visbility user setting. 2023-02-10 17:35:49 -08:00
Sahil Batra b919dfd489 realm: Add time limit setting for moving messages between streams.
This commit adds "move_messages_between_streams_limit_seconds"
setting which would be used to set a time limit to move messages
between streams.
2023-02-08 12:46:05 -08:00
Sahil Batra 73f0eae394 realm: Add time limit setting for moving message within stream.
This commit adds "move_messages_within_streams_limit_seconds"
setting which would be used to set a time limit to move messages
within stream.
2023-02-08 12:46:05 -08:00
Sahil Batra 5b1f6a696e message: Refactor code for parsing message time limit settings.
This commit renames parse_message_content_edit_or_delete_limit
to parse_message_time_limit_setting and also renames
MESSAGE_CONTENT_EDIT_OR_DELETE_LIMIT_SPECIAL_VALUES_MAP to
MESSAGE_TIME_LIMIT_SETTING_SPECIAL_VALUES_MAP.

We do this change since this function and object will also be
used for message move limit and it makes sense to have a more
generic name.
2023-02-08 10:59:28 -08:00
Alex Vandiver e3c976fed9 missed_messages: Include messages from topic which differ by case.
In Zulip, message topics are case-insensitive but case-preserving.
The `get_context_for_message` function erroneously did a
case-sensitive search, and thus only messages whose topic matched
exactly were pulled in as context.

Make the missed-message pipeline aware that message topics are not
case-sensitive.  This means that, when collapsing adjacent messages,
we merge messages with topic headers which are "different"; create a
separate explicit "grouping" to know which to collapse.
2023-02-07 11:04:21 -08:00
Anders Kaseorg da3cf5ea7a ruff: Fix RSE102 Unnecessary parentheses on raised exception.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-02-04 16:34:55 -08:00
Tim Abbott 8d27d14354 models: Improve documentation for FOLLOWED UserTopics. 2023-02-03 11:25:05 -08:00
Anders Kaseorg df001db1a9 black: Reformat with Black 23.
Black 23 enforces some slightly more specific rules about empty line
counts and redundant parenthesis removal, but the result is still
compatible with Black 22.

(This does not actually upgrade our Python environment to Black 23
yet.)

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-02-02 10:40:13 -08:00
Anders Kaseorg 7a7513f6e0 ruff: Fix SIM201 Use `… != …` instead of `not … == …`.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-01-23 11:18:36 -08:00
Anders Kaseorg ff1971f5ad ruff: Fix SIM105 Use `contextlib.suppress` instead of try-except-pass.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-01-23 11:18:36 -08:00
Anders Kaseorg b0e569f07c ruff: Fix SIM102 nested `if` statements.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-01-23 11:18:36 -08:00
Sahil Batra 939a6edf0f settings: Rename helper function to check who can edit topics.
This commit renames "can_edit_topic_of_any_message" function
in models.py to "can_move_messages_to_another_topic" and
"user_can_edit_topic_of_any_message" function in settings_data.js
to "user_can_move_messages_to_another_topic".

This change is done since topic editing permission does not
depend on message sender now and messages are considered same
irrespective of whether the user who is editing the topic had sent
the message or not. This also makes the naming consistent with
what we use for the label of this setting in webapp and how we
describe this action in help documentation.
2022-12-13 23:11:50 -08:00
Sahil Batra ad9a7d2e06 message_edit: Add "Nobody" option for move_messages_between_streams_policy. 2022-12-13 23:11:50 -08:00
Sahil Batra 02eee3a04f message_edit: Add "Nobody" option for edit_topic_policy setting. 2022-12-13 23:11:50 -08:00
Zixuan James Li 5f4d857d3c linkifier: Order linkifiers by id on query.
This explicitly enforces ordering on the linkifiers. This is useful when
there are overlapping linkifier patterns that matches the same text. In
our current linkifier implementation, this order affects how the
patterns are handled in the markdown processor, with the earlier ones
being prioritized.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-12-13 15:16:20 -08:00
Anders Kaseorg 73c4da7974 ruff: Fix N818 exception name should be named with an Error suffix.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-11-17 16:52:00 -08:00
Anders Kaseorg 1b016bef73 ruff: Enable naming errors.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-11-16 09:29:11 -08:00
Mateusz Mandera d201229df8 signup: Implement use of MultiUseInvite.status attribute.
This allows us to revoke MultiUseInvites by changing their .status
instead of deleting them (which has been deleting the helpful tracking
information on PreregistrationUsers about which MultiUseInvite they came
from).
2022-11-14 17:13:16 -08:00
Aman Agrawal fbe9a9e539 left_side_userlist: Remove feature from frontend.
Fixes #23517.

While this feature was added to Zulip very early, it has been troubled
for most of that time; it never looked great visually, had a lot of
implementation complexity around resize.js, and has a weird model (a
setting that changes the UI only in certain window sizes).

This option is not commonly used; while a significant portion of users
have it enabled, many of them just don't use window sizes where it
actually has an effect. So it's not clear that it will be missed if
removed; we got very few bug reports when it was completely broken for
a few days after we first integrated the new left sidebar private
messages design.

Even with it no longer being broken, it does not work very well with
the addition of the new PMs section in the left sidebar. (Having two
scrollbars in the sidebar looks quite awkward.) The new private
messages section in the left sidebar also addresses some of the use
cases for always keeping the Users list always visible, even in narrow
windows.

This option is only removed from frontend for now. To make this
decision easily reversible, the backend code of this feature
is still kept.
2022-11-14 12:23:55 -08:00
Anders Kaseorg 69e94b5991 ruff: Fix C413 Unnecessary `list` call around `sorted()`.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-11-03 12:10:15 -07:00
Alex Vandiver 55c0e670d9 cache: Only cache list results of QuerySets, not the QuerySet itself.
Storing a QuerySet rather than the list version of the result in it
has a large overhead -- and, as noted by the type annotations, the
result is only ever used as a list.  This difference is particularly
important because the cached `get_realm_user_dicts` can get extremely
large for realms with large numbers of users, potentially overflowing
the 1MB default object limit in memcached.

Switch all cases of `cache_with_key` which return QuerySets to
returning the list values of them.
2022-10-12 22:25:48 -07:00
Anders Kaseorg 1385a827c2 python: Clean up getattr, setattr, delattr calls with literal names.
These were useful as a transitional workaround to ignore type errors
that only show up with django-stubs, while avoiding errors about
unused type: ignore comments without django-stubs.  Now that the
django-stubs transition is complete, switch to type: ignore comments
so that mypy will tell us if they become unnecessary.  Many already
have.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-10-10 08:40:28 -07:00
Anders Kaseorg fcd81a8473 python: Replace avoidable uses of __special__ attributes.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-10-10 08:32:29 -07:00
Mateusz Mandera 42b3575183 models: Make Message.realm and ArchivedMessage.realm non-null.
In the previous commits, these have been backfilled, so now we can
enforce non-nullable.
2022-10-07 10:10:01 -07:00
Mateusz Mandera 00b3546c9f models: Add denormalized .realm column to Message.
This commit adds the OPTIONAL .realm attribute to Message
(and ArchivedMessage), with the server changes for making new Messages
have this set. Old Messages still have to be migrated to backfill this,
before it can be non-nullable.

Appropriate test changes to correctly set .realm for Messages the tests
manually create are included here as well.
2022-10-07 10:09:38 -07:00
Sahil Batra 2bf70fe4db custom_profile_field: Add "Pronouns" custom field type.
This commit adds "Pronouns" custom profile field type. We also
add "Pronouns" type field in the development environment
2022-10-06 17:56:26 -07:00
Anders Kaseorg ad2795698b models: Remove explicit id fields.
With django-stubs, these explicit copies of Django’s implicit id
fields are no longer needed for type checking.  An exception is the
BigAutoField AbstractUserMessage.id, which is left alone.

This reverts commit c08ee904d8 (#15641).

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-10-06 13:49:11 -07:00