Commit Graph

472 Commits

Author SHA1 Message Date
sahil839 2dc99aa90f settings: Add new setting for controlling who can move msgs to stream.
This commit adds a new setting 'move_messages_between_streams_policy`
for controlling who can move messages from one stream to other.
2021-04-16 15:10:39 -07:00
Aman Agrawal 802c450b3f realm: Add setting to configure GIPHY rating.
Organization admins can use this setting to restrict the maximum
rating of GIFs that will be retrieved from GIPHY. Also, there
is option to disable GIPHY too.
2021-04-14 10:29:39 -07:00
sahil839 4c8339fa8c settings: Replace invite_by_admins_policy with invite_to_realm_policy.
This commit replaces invite_by_admins_policy, which was a bool field,
with a new enum field invite_by_realm_policy.

Though the final goal is to add moderators and full members option
using COMMON_POLICY_TYPES, but this will be done in a separate
commit to make this easy for review.
2021-04-07 09:02:33 -07:00
Sumanth V Rao 40228972b9 models/realm: Add a model for storing realm playground information.
Tweaked exports.py to add the config object there so that our export
tool can include the table when exporting. Also includes all the
changes required to import the new table from the exported data.

Helper function `get_realm_playgrounds` added to fetch all
playgrounds in a realm.

Tests amended.
2021-04-07 08:20:53 +05:30
Abhijeet Prasad Bodas 3bfcaa3968 mute user: Add backend infrastructure code.
Adds backend code for the mute users feature.
This is just infrastructure work (database
interactions, helpers, tests, events, API docs
etc) and does not involve any behavioral/semantic
aspects of muted users.

Adds POST and DELETE endpoints, to keep the
URL scheme mostly consistent in terms of `users/me`.

TODOs:
1. Add tests for exporting `zulip_muteduser` database table.
2. Add dedicated methods to python-zulip-api to be used
   in place of the current `client.call_endpoint` implementation.
2021-04-06 18:44:08 -07:00
Abhijeet Prasad Bodas 89f6139505 mute user: Add model and makemigrations.
This commit adds a new database table to support
muting users, and generates a migration file for
the same.
2021-04-06 18:44:08 -07:00
Mateusz Mandera 1e37fde59f migrations: Subscription.is_user_active denormalization - step two.
This adds a migration to do the backfill and add the index that'll be
used to optimize the bulk_get_subscriber_user_ids query in the next
commit.
2021-03-30 09:29:36 -07:00
Mateusz Mandera f329878376 migrations: Subscription.is_user_active denormalization - step one.
This adds the is_user_active with the appropriate code for setting the
value correctly in the future. In the following commit a migration to
backfill the value for existing Subscriptions will be added.

To ensure correct user_profile.is_active handling also in tests, we
replace all direct .is_active mutation with calls to appropriate
functions.
2021-03-30 09:19:03 -07:00
Aman Agrawal e587c029f6 display_settings: Allow user to set default_view.
TextField is used to allow users to set long stream + topic narrow
names in the urls.

We currently restrict users to only set "all_messages" and
"recent_topics" as narrows.

This commit achieves 3 things:
* Removes recent topics as the default view which loads when
  hash is empty.
* Loads default_view when hash is empty.
* Loads default_view on pressing escape key when it is unhandled by
  other present UI elements.

NOTE: After this commit loading zulip with an empty hash will
automatically set hash to default_view.  Ideally, we'd just display
the default view without a hash, but that involves extra complexity.

One exception is when user is trying to load an overlay directly,
i.e. zulip is loaded with an overlay hash. In this case,
we render recent topics is background irrespective of default_view.

We consider this last detail to be a bug not important enough to block
adding this setting.
2021-03-11 18:09:08 -08:00
Anders Kaseorg e128f92ee1 migrations: Swap out deprecated JSONField import in-place.
A deprecated import shouldn’t be used even in a migration, since the
migration will need to remain runnable in the future.  We never needed
a migration for this switch anyway; we just needed to edit the old
migration, since no actual state changes are involved.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-03-03 21:46:05 -08:00
Anders Kaseorg 6e4c3e41dc python: Normalize quotes with Black.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-02-12 13:11:19 -08:00
Anders Kaseorg 11741543da python: Reformat with Black, except quotes.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-02-12 13:11:19 -08:00
Anders Kaseorg 69890f36b1 migrations: Enforce evaluation order in 0306 WHERE clause.
Depending on PostgreSQL’s query plan, it was possible for the value
condition to be evaluated before the field_type condition was checked,
leading to errors like

psycopg2.errors.InvalidDatetimeFormat: invalid value "stri" for "YYYY"
DETAIL:  Value must be an integer.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-01-29 09:35:52 -08:00
Mateusz Mandera 1432067959 dependencies: Upgrade to Django 3.1.
https://docs.djangoproject.com/en/3.1/releases/3.1/

- django.contrib.postgres.fields.JSONField is deprecated and should be
  replaced with models.JSONField
-  The internals of the implementation in the postgresql backend have
   changed a bit in
   f48f671223
   and thus we need to make an ugly tweak in test_runner.
- app_directories.Loader.get_dirs() now returns a list of PosixPath so
  we need to make a small tweak in TwoFactorLoader for that (PosixPath
  is not iterable)

Fixes #16010.
2021-01-26 10:20:00 -08:00
Mateusz Mandera 160cc5120a api: Require can_create_users permission to create users via API.
Allowing any admins to create arbitrary users is not ideal because it
can lead to abuse issues.  We should require something stronger that
requires the server operator's approval and thus we add a new
can_create_users permission.
2020-12-21 13:20:21 -08:00
Mateusz Mandera c9b6d8ddad models: Remove redundant Meta.permissions on Realm model.
This is dead code leftover from the old way of handling admin
permissions.
2020-12-21 13:15:40 -08:00
Mateusz Mandera d0dc04a093 models: Rename is_api_super_user to can_forge_sender, 2020-12-21 13:15:39 -08:00
Anders Kaseorg 2ab0b3d4fc validator: Reject ISO 8601 dates missing leading zeros.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-12-15 16:36:50 -08:00
Mateusz Mandera b652cc786c django3: Remove remaining postgresql_psycopg2 use.
Removed in Django 3.0.
2020-12-15 11:52:32 -08:00
Siddharth Asthana 82f5759299 Realm: Add a deactivated_redirect URLField to Realm object.
We export a realm's data, and disable the realm, because the user
is moving from Zulip Cloud (e.g. https://example.zulipchat.com/) to
self-hosting or another platform (e.g. https://zulip.example.com/)
which we do not control. This commit adds a field in the realm object
called deactivated_redirect to store the url to which the realm has
moved.
2020-12-14 21:04:52 -08:00
Anders Kaseorg d0d8c358b3 lint: Migrate typing.Text check to semgrep.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-11-16 18:17:41 -08:00
Anders Kaseorg dc84e9696c mypy: Fix types for redis.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-11-10 15:46:04 -08:00
sahil839 7106069d4d migration: Add migration to remove default status of private streams.
This commit adds migration which removes default status of exisitng
default private streams, i.e. private stream exists but they are no
longer default.
2020-10-29 15:47:34 -07:00
ryanreh99 dfa7ce5637 uploads: Support non-AWS S3-compatible server.
Boto3 does not allow setting the endpoint url from
the config file. Thus we create a django setting
variable (`S3_ENDPOINT_URL`) which is passed to
service clients and resources of `boto3.Session`.

We also update the uploads-backend documentation
and remove the config environment variable as now
AWS supports the SIGv4 signature format by default.
And the region name is passed as a parameter instead
of creating a config file for just this value.

Fixes #16246.
2020-10-28 21:59:07 -07:00
Alex Vandiver 5eb8064a1a install: Rename postgres options to postgresql. 2020-10-28 11:55:32 -07:00
Anders Kaseorg 72d6ff3c3b docs: Fix more capitalization issues.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-10-23 11:46:55 -07:00
sahil839 78b98d8067 realm: Add wildcard_mention_policy setting.
We add a new wildcard_mention_policy setting to handle wildcard
mentions in large streams, with a wide range of policies available to
organizations.

We set the default to the safe option for preventing accidental spam:
only stream administrators being able to use wildcard mentions in
large streams.
2020-10-01 12:18:03 -07:00
Anders Kaseorg 42d038f09b migrations: Fix 0301 to replace a Python loop with SQL.
The previous code is correctly flagged by semgrep 0.23 as a violation
of our sql-format rule.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-09-13 20:59:09 -07:00
Mateusz Mandera 70bcc7a8e9 migrations: Add case-insensitive unique index on realm and stream name.
This is closely analogical to b9b146c809.
2020-09-13 15:36:43 -07:00
Anders Kaseorg b7b7475672 python: Use standard secrets module to generate random tokens.
There are three functional side effects:

• Correct an insignificant but mathematically offensive bias toward
repeated characters in generate_api_key introduced in commit
47b4283c4b4c70ecde4d3c8de871c90ee2506d87; its entropy is increased
from 190.52864 bits to 190.53428 bits.

• Use the base32 alphabet in confirmation.models.generate_key; its
entropy is reduced from 124.07820 bits to the documented 120 bits, but
now it uses 1 syscall instead of 24.

• Use the base32 alphabet in get_bigbluebutton_url; its entropy is
reduced from 51.69925 bits to 50 bits, but now it uses 1 syscall
instead of 10.

(The base32 alphabet is A-Z 2-7.  We could probably replace all of
these with plain secrets.token_urlsafe, since I expect most callers
can handle the full urlsafe_b64 alphabet A-Z a-z 0-9 - _ without
problems.)

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-09-09 15:52:57 -07:00
Anders Kaseorg 3b301f522b python: Tweak some magic trailing commas to avoid Black bugs.
https://github.com/psf/black/issues/1658
https://github.com/psf/black/issues/1671

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-09-03 17:51:09 -07:00
Anders Kaseorg a276eefcfe python: Rewrite dict() as {}.
Suggested by the flake8-comprehensions plugin.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-09-02 11:15:41 -07:00
Anders Kaseorg ab120a03bc python: Replace unnecessary intermediate lists with generators.
Mostly suggested by the flake8-comprehension plugin.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-09-02 11:15:41 -07:00
Mateusz Mandera 9b50c49ea7 streams: Mark all messages as read when deactivating a stream.
The query to finds and marks all unread UserMessages in the stream as read
can be quite expensive, so we'll move that work to the deferred_work
queue and split it into batches.

Fixes #15770.
2020-09-01 11:24:27 -07:00
Tim Abbott f2c9ee8000 migrations: Fix migration 0283 with orjson.
This function now matches the copy in zerver/lib/actions.py.

This is the same migration as
b250e42f61c525029bd2b3bbb8f4ea93ece62072; orjson enforces that we
don't use integers as keys in JSON dictionaries.
2020-08-14 11:22:03 -07:00
Clara Dantas 05bf72a75c attachments: Add is_web_public field.
This commit adds the is_web_public field in the AbstractAttachment
class. This is useful when validating user access to the attachment,
as otherwise we would have to make a query in the db to check if
that attachment was sent in a message in a web-public stream or not.
2020-08-12 17:26:03 -07:00
sahil839 78da9fd3ab subscription: Add role field to Subscription class.
This commit adds role field to the Subscription class. Currently,
there are two option of roles - STREAM_ADMINISTRATOR and MEMBER.

We also add a property 'is_stream_admin' for checking whether the
user is stream admin or not.
2020-08-12 16:47:04 -07:00
Anders Kaseorg 61d0417e75 python: Replace ujson with orjson.
Fixes #6507.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-08-11 10:55:12 -07:00
Anders Kaseorg c155403884 docs: Fix various capitalization errors.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-08-11 10:25:52 -07:00
Anders Kaseorg 60a25b2721 docs: Fix spelling errors caught by codespell.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-08-11 10:23:06 -07:00
Tim Abbott e12225ace7 realm_audit_log: Fix malformed RealmAuditLog entries.
In f8bcf39014, we fixed buggy
marshalling of Streams and similar data structures where we were
including the Stream object rather than its ID in dictionaries passed
to ujson, and ujson happily wrote that large object dump into the
RealmAuditLog.extra_data field.

This commit includes a migration to fix those corrupted RealmAuditLog
entries, and because the migration loop is the same, also fixes the
format of similar RealmAuditLog entries to be in a more natural format
that doesn't weirdly nest and duplicate the "property" field.

Fixes #16066.
2020-08-09 16:51:29 -07:00
Hemanth V. Alluri 0e893b9045 models/drafts: Add a model for storing Draft messages.
Also add a Draft object-to-dictionary conversion method.
The following commits will provide an API around this
model using which our clients can sync drafts across each
other (if they so wish too). As of making this commit, we
haven't finalized exactly how our clients will use this.

See https://chat.zulip.org/#narrow/stream/2-general/topic/drafts
For some of the discussion around this model and in general,
around this feature.

Signed-off-by: Hemanth V. Alluri <hdrive1999@gmail.com>
2020-07-28 17:18:35 -07:00
Steve Howell c44500175d database: Remove short_name from UserProfile.
A few major themes here:

    - We remove short_name from UserProfile
      and add the appropriate migration.

    - We remove short_name from various
      cache-related lists of fields.

    - We allow import tools to continue to
      write short_name to their export files,
      and then we simply ignore the field
      at import time.

    - We change functions like do_create_user,
      create_user_profile, etc.

    - We keep short_name in the /json/bots
      API.  (It actually gets turned into
      an email.)

    - We don't modify our LDAP code much
      here.
2020-07-17 11:15:15 -07:00
Mateusz Mandera b9b146c809 migrations: Add case-insensitive unique indexes on realm and email.
Tweaked by tabbott to use clearer conventions and to document it in
the upgrade notes.

Fixes #15772.
2020-07-15 09:42:40 -07:00
Steve Howell 0b65abcdf5 pointer: Remove pointer from UserProfile.
Most of the changes here are just that we no
longer need to provide a value for pointer
when we create UserProfile objects.
2020-07-03 13:08:40 +00:00
Tim Abbott 2a6ef2b9eb models: Add missing migrations from invite_as change.
I neglected to `git add` after doing `renumber-migrations` for
1f8f227444.
2020-06-24 11:39:01 -07:00
Mateusz Mandera 7a03e2a7fe retention: Replace Realm.message_retention_days None value with -1.
To be more consistent with the meaning in the Stream model, and to make
it easier to have a reasonable settings API, we get rid of the None
value for Realm.message_retention_days in favor of the value -1 to
represent the "don't delete messages" default policy.
2020-06-24 10:33:21 -07:00
Tim Abbott 16cdc3c66b migrations: Fix migration 290 not being marked atomic.
Migrations that add items to a table in the same migration htat
craetes the table require atomic=False.

This fixes this exception:

Running migrations:
  Applying zerver.0287_clear_duplicate_reactions... OK
  Applying zerver.0288_reaction_unique_on_emoji_code... OK
  Applying zerver.0289_tighten_attachment_size... OK
  Applying zerver.0290_remove_night_mode_add_color_scheme...Traceback (most recent call last):
  File "/home/zulip/deployments/2020-06-22-23-20-36/zulip-py3-venv/lib/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
  File "/home/zulip/deployments/2020-06-22-23-20-36/zerver/lib/db.py", line 33, in execute
    return wrapper_execute(self, super().execute, query, vars)
  File "/home/zulip/deployments/2020-06-22-23-20-36/zerver/lib/db.py", line 20, in wrapper_execute
    return action(sql, params)
psycopg2.errors.ObjectInUse: cannot ALTER TABLE "zerver_userprofile" because it has pending trig
2020-06-22 17:08:32 -07:00
MariaGkoulta b10f156250 settings: Add automatic theme detection feature.
With this implementation of the feature of the automatic theme
detection, we make the following changes in the backend, frontend and
documentation.

This replaces the previous night_mode boolean with an enum, with the
default value being to use the prefers-color-scheme feature of the
operating system to determine which theme to use.

Fixes: #14451.

Co-authored-by: @kPerikou <44238834+kPerikou@users.noreply.github.com>
2020-06-21 01:09:01 -07:00
Tim Abbott f98d244ed6 models: Change attachment.size to not be nullable.
Attachment objects in production are only created in one place, which
passses a size.  Additionally, I verified in multiple production
environments with old data that this never actually happens (or has
happened).

So we should make the data model correctly reflect the possibilities here.
2020-06-20 13:15:02 -07:00