Commit Graph

478 Commits

Author SHA1 Message Date
Ryan Rehman 174b2abcfd settings: Migrate to stream_post_policy structure.
This commit includes a new `stream_post_policy` setting,
by replacing the `is_announcement_only` field from the Stream model,
which is done by mirroring the structure of the existing
`create_stream_policy`.

It includes the necessary schema and database migrations to migrate
the is_announcement_only boolean field to stream_post_policy,
a smallPositiveInteger field similar to many other settings.

This change is done to allow organization administrators to restrict
new members from creating and posting to a stream. However, this does
not affect admins who are new members.

With many tweaks by tabbott to documentation under /help, etc.

Fixes #13616.
2020-02-04 17:08:08 -08:00
Ryan Rehman 3dc7d60ffe muting: Record DateTime when a Topic is muted.
This includes the necessary migration to add
the date_muted field to the MutedTopic class
and populates it with a hard coded value.
2020-02-02 20:49:53 -08:00
Tim Abbott 3a9568b4d4 migrations: Fix zulipinternal migration corner case.
It's theoretically possible to have configured a Zulip server where
the system bots live in the same realm as normal users (and may have
in fact been the default in early Zulip releases?  Unclear.).  We
should handle these without the migration intended to clean up naming
for the system bot realm crashing.

Fixes #13660.
2020-01-15 13:59:31 -08:00
Tim Abbott 8e7ce7cc79 python: Sort migrations/management command imports with isort.
This is a preparatory commit for using isort for sorting all of our
imports, merging changes to files where we can easily review the
changes as something we're happy with.

These are also files with relatively little active development, which
means we don't expect much merge conflict risk from these changes.
2020-01-14 13:07:47 -08:00
Tim Abbott 79f18138f5 realm: Add private_message_policy setting.
This experimental setting disables sending private messages in Zulip
in a crude way (i.e. users get an error when they try to send one).
It makes no effort to adjust the UI to avoid advertising the idea of
sending private messages.

Fixes #6617.
2020-01-13 12:20:42 -08:00
Mateusz Mandera c011d2c6d3 email_mirror: Migrate missed message addresses from redis to database.
Addresses point 1 of #13533.

MissedMessageEmailAddress objects get tied to the specific that was
missed by the user. A useful benefit of that is that email message sent
to that address will handle topic changes - if the message that was
missed gets its topic changed, the email response will get posted under
the new topic, while in the old model it would get posted under the
old topic, which could potentially be confusing.

Migrating redis data to this new model is a bit tricky, so the migration
code has comments explaining some of the compromises made there, and
test_migrations.py tests handling of the various possible cases that
could arise.
2020-01-07 13:03:22 -08:00
Mateusz Mandera 9077bbfefd models: Add MissedMessageEmailAddress class.
Preparatory commit for making the email mirror use the database instead
of redis for missed message addresses.

This model will represent missed message email addresses, which
currently have their data stored in redis.
The redis data will be converted and migrated into these models and
the email mirror will start using them in the main commit.
2020-01-07 12:46:55 -08:00
Vishnu KS 6901087246 install: Use crudini for storing value of POSTGRES_MISSING_DICTIONARIES.
This simplifies the RDS installation process to avoid awkwardly
requiring running the installer twice, and also is significantly more
robust in handling issues around rerunning the installer.

Finally, the answer for whether dictionaries are missing is available
to Django for future use in warnings/etc. around full-text search not
being great with this configuration, should they be required.
2019-12-13 12:05:39 -08:00
Mateusz Mandera 1926649dae migrations: Avoid triggering backend initalization in migration 0209.
Fixes #13528.
The email_auth_enabled check caused all enabled backends to get
initialized, and thus if LDAP was enabled the check_ldap_config()
check would cause an error if LDAP was misconfigured
(for example missing the new settings).
2019-12-13 10:54:05 -08:00
Tim Abbott 7bde70bb52 migrations: Batch fix_has_link_attribute migration.
This avoids risk of OOM issues on servers with relatively limited RAM
and millions of messages of history; apparently, fetching all messages
ordered by ID could be quite memory-intensive even with an iterator
usage model.

Fortunately, we have other migrations that already follow this pattern
of iterating over messages, so it's easy to borrow existing code to
make this migration run reasonably.
2019-12-12 15:29:49 -08:00
Tim Abbott cd3535ad0c migrations: Improve has_link migration.
This fixes a few minor issues with the migration:

* Skips messages with empty rendered_content, fixing an exception that
  affected 4 messages on chat.zulip.org.
* Accesses messages in order.
* Provides some basic output on the progress made.

This should make life substantially better for any organizations that
run into trouble with this migration, either due to it taking a long
time to run or due to any new exceptions.
2019-12-12 14:05:08 -08:00
Tim Abbott e7cf1112c8 notifications: Enable online push notifications by default.
For new user onboarding, it's important for it to be easy to verify
that Zulip's mobile push notifications work without jumping through
hoops or potentially making mistakes.  For that reason, it makes sense
to toggle the notification defaults for new users to the more
aggressive mode (ignoring whether the user is currently actively
online); they can set the more subtle mode if they find that the
notifications are annoying.
2019-12-12 13:04:10 -08:00
Vishnu KS 6c97a36355 install: Support remote database services like RDS.
Documentation and variable names edited by tabbott.
2019-12-12 12:59:45 -08:00
Tim Abbott 60d307ac3f message: Add migrations to correct has_* fields.
This follows up on changes to correctly set has_link, has_attachment,
and has_image when rendering messages.

Fixes #12912.
2019-12-11 11:03:49 -08:00
Mateusz Mandera 8acfa17fe6 models: Add recipient foreign key in UserProfile and Stream.
This is adds foreign keys to the corresponding Recipient object in the
UserProfile on Stream tables, a denormalization intended to improve
performance as this is a common query.

In the migration for setting the field correctly for existing users,
we do a direct SQL query (because Django 1.11 doesn't provide any good
method for doing it properly in bulk using the ORM.).

A consequence of this change to the model is that a bit of code needs
to be added to the functions responsible for creating new users (to
set the field after the Recipient object gets created).  Fortunately,
there's only a few code paths for doing that.

Also an adjustment is needed in the import system - this introduces a
circular relation between Recipient and UserProfile. The field cannot be
set until the Recipient objects have been created, but UserProfiles need
to be created before their corresponding Recipients. We deal with this
by first importing UserProfiles same way as before, but we leave the
personal_recipient field uninitialized. After creating the Recipient
objects, we call a function to set the field for all the imported users
in bulk.

A similar change is made for managing Stream objects.
2019-12-09 15:14:41 -08:00
Mateusz Mandera 0c2cc41d2e CVE-2019-18933: Fix insecure account creation via social authentication.
A bug in Zulip's new user signup process meant that users who
registered their account using social authentication (e.g. GitHub or
Google SSO) in an organization that also allows password
authentication could have their personal API key stolen by an
unprivileged attacker, allowing nearly full access to the user's
account.

Zulip versions between 1.7.0 and 2.0.6 were affected.

This commit fixes the original bug and also contains a database
migration to fix any users with corrupt `password` fields in the
database as a result of the bug.

Out of an abundance of caution (and to protect the users of any
installations that delay applying this commit), the migration also
resets the API keys of any users where Zulip's logs cannot prove the
user's API key was not previously stolen via this bug.  Resetting
those API keys will be inconvenient for users:

* Users of the Zulip mobile and terminal apps whose API keys are reset
  will be logged out and need to login again.
* Users using their personal API keys for any other reason will need
  to re-fetch their personal API key.

We discovered this bug internally and don't believe it was disclosed
prior to our publishing it through this commit.  Because the algorithm
for determining which users might have been affected is very
conservative, many users who were never at risk will have their API
keys reset by this migration.

To avoid this on self-hosted installations that have always used
e.g. LDAP authentication, we skip resetting API keys on installations
that don't have password authentication enabled.  System
administrators on installations that used to have email authentication
enabled, but no longer do, should temporarily enable EmailAuthBackend
before applying this migration.

The migration also records which users had their passwords or API keys
reset in the usual RealmAuditLog table.
2019-11-21 10:23:37 -08:00
Tim Abbott 1fe4f795af settings: Add notification settings checkboxes for wildcard mentions.
This change makes it possible for users to control the notification
settings for wildcard mentions as a separate control from PMs and
direct @-mentions.
2019-11-20 16:58:46 -08:00
Matheus Melo c96762b7a9 settings: Add setting for who can edit user groups.
Fixes #12380.
2019-11-03 16:45:13 -08:00
Mateusz Mandera 5aded51b73 register: Pre-populate Name in social backend flow.
By adding some additional plumbing (through PreregistrationUser) of the
full_name and an additional full_name_validated option, we
pre-populate the Full Name field in the registration form when coming
through a social backend (google/github/saml/etc.) and potentially skip
the registration form (if the user would have nothing to do there other
than clicking the Confirm button) and just create the account and log
the user in.
2019-11-03 16:15:48 -08:00
Mateusz Mandera 4dc3ed36c3 auth: Add initial SAML authentication support.
There are a few outstanding issues that we expect to resolve beforce
including this in a release, but this is good checkpoint to merge.

This PR is a collaboration with Tim Abbott.

Fixes #716.
2019-10-10 15:44:34 -07:00
Rishi Gupta e10361a832 models: Replace is_guest and is_realm_admin with UserProfile.role.
This new data model will be more extensible for future work on
features like a primary administrator.
2019-10-06 16:24:37 -07:00
Rishi Gupta 4256ee61cf billing: Change RealmAuditLog.event_type from str to int.
This is a more robust long-term model for storing these data.
2019-10-06 15:55:56 -07:00
Mateusz Mandera dbe508bb91 models: Migration of Message.pub_date to date_sent, part 2.
Fixes #1727.

With the server down, apply migrations 0245 and 0246. 0246 will remove
the pub_date column, so it's essential that the previous migrations
ran correctly to copy data before running this.
2019-10-05 19:01:34 -07:00
Mateusz Mandera f198475281 models: Migration of Message.pub_date to date_sent, part 1.
1. Apply migration 0243 to add date_sent column.
2. Apply migration 0244 to copy pub_date over to date_sent. Can be done
with the server running.
3. With the server down (for consistency between memory and
database state of Django objects), verify consistency with
Message.objects.exclude(date_sent=F("pub_date")).count() == 0
2019-10-05 19:01:34 -07:00
Tim Abbott 8024b1179a bots: Fix bot email addresses with EMAIL_ADDRESS_VISIBILITY_ADMINS.
When using our EMAIL_ADDRESS_VISIBILITY_ADMINS feature, we were
apparently creating bot users with different email and delivery_email
properties, due to effectively an oversight in how the code was
written (the initial migration handled bots correctly, but not bots
created after the transition).

Following the refactor in the last commit, the fix for this is just
adding the missing conditional, a test, and a database migration to
fix any incorrectly created bots leaked previously.
2019-09-23 15:53:53 -07:00
Mateusz Mandera 6bd34afa0d models: Migration of UserMessage.id to bigint, part 2.
This requires part 1 (which can take hours to run but generally
doesn't require downtime) to be completed first.

This portion of the migration will require the server to be completely
down for a brief period; for chat.zulip.org with 250M UserMessage
rows, it took about 60s to run; that time will vary depending on
hardware details like whether the server has an SSD, but fundamentally
shouldn't be long.

Our upgrade-zulip and upgrade-zulip-from-git tools can apply this
migration correctly; nothing special needs to be done.

Fixes #13040.
2019-08-26 21:25:19 -07:00
Mateusz Mandera b008515d63 models: Migration of UserMessage.id to bigint, part 1.
As part of adding support for more than 2B UserMessage rows in a Zulip
server, we need to change UserMessage.id (a field we don't access but
is needed by Django) from an int to a bigint.  This commit is a series
of migrations which create a `bigint_id` column and populates it correctly.

This migration will take a long time to run; on chat.zulip.org (a
server with a lot of history), it took about 4 hours to complete.

How to migrate with minimal downtime:

1. Run `upgrade-zulip-from-git` through this commit.  It will install
migration 0238 and then more or less hang while applying migration
0239.  Once migration 0238 is completed, however, your server should
be able to be started back up safely while migration 0239 is running.

2. Run `/home/zulip/deployments/next/scripts/restart-server` in a
separate terminal to get Zulip running again.

3. When the `upgrade-zulip-from-git` command finishes, it will
automatically re-restart the Zulip server, leaving you in a consistent
state and ready to do part 2 of the migration.

A useful `manage.py shell` query for checking the state after this
commit is consistent is this:

    assert UserMessage.objects.exclude(bigint_id=F("id")).count() == 0

Part of #13040.
2019-08-26 21:14:15 -07:00
Anders Kaseorg 5aeb562700 migrations.0001_initial: Expand \n in long SQL string.
One occasionally finds that a 1580 character string of SQL queries
might not most readably be presented on a single line.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-21 11:00:38 -07:00
Mateusz Mandera 9b2b24c617 production: Rename the 'zulip' system bot realm to 'zulipinternal'. 2019-07-31 16:01:36 -07:00
Wyatt Hoodes b01ba48369 migration_0060: Remove unreachable return statement.
This was introduced in a past refactoring.
2019-07-29 15:23:34 -07:00
Wyatt Hoodes a75abe096b migration_0064: Fix typing for file_key. 2019-07-29 15:23:10 -07:00
Wyatt Hoodes e106f95c8b migration_0064: Remove usage of Text type. 2019-07-29 15:23:10 -07:00
Wyatt Hoodes b3323366b3 migration_0077: Migrate from Text to str.
In `force_str` we assume that python 2 strings should be
considered.  This is no longer the case, so we replace all
occurences of `Text` with `str`, and remove the unreachable
condition.

(Probably further cleanup is possible, but this code shouldn't be
modified again in any case).
2019-07-25 17:54:26 -07:00
Conner Bondurant c25dcf048d models: Enforce stricter requirements on the full_name field.
This changes the requirements for UserProfile to disallow some
additional characters, with the overall goal of being able to use
formataddr in send_mail.py.

We don't need to be particularly careful in the database migration,
because user full_names are not required to be unique.
2019-07-22 18:13:34 -07:00
David Wood 9bace3f2cd notifications: Allow only notifiable in unread count.
This commit adds a new setting to the user's notification settings that
will change the behaviour of the unread count in the title bar and
desktop application.

When enabled, the title bar will show the count of unread private messages
and mentions. When disabled, the title bar will act as before, showing
the total number of unread messages.

Fixes #1736.
2019-07-13 15:49:04 -07:00
Yashashvi Dave d7ee2aced1 models: Add `external_account` in custom profile field types.
Add new custom profile field type, External account.
External account field links user's social media
profile with account. e.g. GitHub, Twitter, etc.

Fixes part of #12302
2019-07-09 17:21:54 -07:00
Mateusz Mandera b8839302c0 retention: Fix migration making archive_transaction field not null.
DELETing from archive tables and ALTERing ArchivedMessage needs to be
split into separate transactions.

zerver_archivedattachment_messages needs to be cleared out before
zerver_archivedattachment.
2019-07-04 16:42:42 -07:00
vinitS101 04f3fce761 ldap: Fix LDAP avatar synchronization to check if avatar has changed.
When "manage.py sync_ldap_user_data" is run, user avatars are now only
updated if they have changed in LDAP.

Fixes #12381.
2019-07-02 17:52:48 -07:00
Mateusz Mandera 74ba0a7082 retention: Make archive_timestamp field not nullable. 2019-07-02 17:25:31 -07:00
Mateusz Mandera a2cce62c1c retention: Use new ArchiveTransaction model.
We add a new model, ArchiveTransaction, to tie archived objects together
in a coherent way, according to the batches in which they are archived.
This enables making a better system for restoring from archive, and it
seems just more sensible to tie the archived objects in this way, rather
the somewhat vague setting of archive_timestamp to each object using
timezone_now().
2019-06-26 12:05:59 -07:00
Anders Kaseorg 6112d020f4 migrate, create_large_indexes: Use CREATE INDEX IF NOT EXISTS.
We no longer support Postgres versions missing this syntax.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-06-26 11:01:30 -07:00
Yashashvi Dave 8e269b4651 models: Rename notification to `enable_stream_audible_notifications`.
Rename notification property `enable_stream_sounds` to
`enable_stream_audible_notifications` to match with other
notification property patterns.

Fixes part of #12304
2019-06-12 16:24:51 -07:00
Mateusz Mandera d481ee9a40 retention: Add message_retention_days field to Stream model. 2019-06-06 11:17:42 -07:00
sahil839 5a130097bf settings: Add display setting for demoting inactive streams.
This adds a setting to control Zulip's default behavior of sorting to
bottom and graying out inactive streams.  The previous logic is still
the default "automatic", but this gives users more control.  See the
models.py comment for details.

Fixes #11524.
2019-06-03 23:07:56 -07:00
Puneeth Chaganti 8c0c9ca7a4 url preview: Turn Realm.inline_url_embed_preview off by default. 2019-05-31 15:28:32 -07:00
Mateusz Mandera 29529cf2e7 retention: Add ArchivedSubMessage model. 2019-05-29 16:26:11 -07:00
Mateusz Mandera 292b4bb0d7 retention: Add ArchivedReaction model. 2019-05-29 16:26:11 -07:00
Yashashvi Dave cb85ca8601 models: Alter video_chat_provider field type to integer.
Migration rewritten by tabbott because it did not work.
2019-05-13 12:02:28 -07:00
Roman Godov a50824e031 models: Rename Subscription.in_home_view field to is_muted.
This renames Subscription.in_home_view field to is_muted, for greater
clarity as to what it does just from seeing the setting name, without
having to look it up.

Also disabled an obsolete test_migrations test.

Fixes #10042.
2019-05-12 22:08:10 -07:00
Pragati Agrawal a838de63d8 settings: Add option for fixed/fluid width.
This adds a new option for "fluid width" under `Display settings` section
of SETTINGS/DISPLAY SETTINGS tab.

Fixes: #11844.
2019-05-09 13:57:57 -07:00
Harshit Bansal b553507412 subscriptions: Migrate notification setting defaults model.
This commit migrates the Subscription's notification fields from a
BooleanField to a NullBooleanField where a value of None means to
inherit the value from user's profile.

Also includes a migrations to set the corresponding settings to None
if they match the user profile's values. This migration helps us in
getting rid of the weird "Apply to all" widget that we offered on
subscription settings page.

The mobile apps can't handle None appearing as the stream-level
notification settings, so for backwards-compatibility we arrange to
only send True/False to the mobile apps by applying those defaults
server-side.  We introduce a notification_settings_null value within a
client_capabilities structure that newer versions of the mobile apps
can use to request the new model.

This mobile compatibility code is pretty effectively tested by the
existing test_events tests for the subscriptions subsystem.
2019-05-08 17:45:10 -07:00
Puneeth Chaganti d474a41c03 digest: Turn off digest_emails_enabled flag for realms by default. 2019-05-08 14:39:12 -07:00
David Wood 34d810aac3 settings: Migrate to create_stream_policy structure.
This commit replaces the `create_stream_by_admins_only` setting with a
new `create_stream_policy` setting, which mirroring the structure of
the existing `invite_to_stream_policy`.

This is important preparation for migrating the waiting period feature
to be its own independent setting.

Fixes #12236.
2019-05-06 16:27:55 -07:00
Shubham Padia 4994a6c2c9 realm: Add setting to disable avatar changes.
This is useful when syncing avatars from an integrated LDAP/active
directory.

The upload avatar and delete avatar buttons are hidden if avatar
changes are disabled and the user is a non-admin.
If the user has a gravatar set, then the user will not be able to
upload an image as their avatar if avatar changes are disabled.

Part of #12132.
2019-05-03 12:52:43 -07:00
David Wood 272ed90685 settings: Create an explicit invite_to_stream_policy setting.
This commit creates a new organization setting that determines whether
a user can invite other users to streams. Previously this was linked
to the waiting period threshold, but this was both not documented and
overly limiting.

With significant tweaks by tabbott to change the database model to not
involve two threshhold fields, edit the tests, etc.

This requires follow-up work to make the create stream policy setting
work how this code implies it should.

Fixes #12042.
2019-04-29 17:11:28 -07:00
Puneeth Chaganti f3d0ccb9f6 models: Add a digest_weekday field to the Realm model.
Allow realms to specify the day of the week when the digest should be sent out.
When enqueue-ing digests, pick only the realms that chose the current weekday as
the day to send out digests.
2019-04-27 15:20:41 -07:00
Mateusz Mandera a269c4bdd4 email_mirror: Make email_token a unique column of Stream. 2019-03-17 12:55:35 -07:00
Raymond Akornor 89351cdd19 send_email: Add ScheduledEmail support for multiple recipients.
Follow up on 92dc363. This modifies the ScheduledEmail model
and send_future_email to properly support multiple recipients.

Tweaked by tabbott to add some useful explanatory comments and fix
issues with the migration.
2019-03-15 11:02:12 -07:00
Challa Venkata Raghava Reddy b69aec2dbc streams: Add first_message_id tracking first message in stream.
This field is primarily intended to support avoiding displaying the
"more topics" feature in new organizations and streams, where we might
know that all messages in the stream are already available in the
browser.

Based on original work by Roman Godov, and significantly modified by
tabbott.

The second migration involved here could be expensive on Zulip Cloud,
but is unlikely to be an issue on other servers.
2019-03-11 13:30:49 -07:00
Hemanth V. Alluri ae126c452b stream-descriptions: Create wrapper for rendering stream descriptions.
In commit de65a04 we can see that if the need ever arises to modify
how stream descriptions are rendered, we would need to make changes
at 5 different call points which can be quite cumbersome. So this
functionality has been extracted to a new method called
'render_stream_descriptions'.
2019-03-06 17:16:14 -08:00
Tim Abbott de65a04ae0 streams: Disable inline URL preview when rendering stream descriptions.
We want to use the baseline features of bugdown, but not fancy things
like inline URL previews, since the whole structure of stream
descriptions is to have a single-line thing supporting some
formatting.

The migration part of this change fixes a bug encountered by some
organizations upgrading from older versions of Zulip.
2019-02-28 17:00:40 -08:00
Anders Kaseorg 649235cfec python: Remove unused imports.
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2019-02-22 16:54:36 -08:00
sahil839 7157edf4af settings: Add support for uploading logo for night mode.
This adds a new field named realm_night_logo which is used for
displaying the organization logo when the user is in night mode.

Fixes #11176.
2019-02-18 15:15:57 -08:00
Vishnu Ks 28769e040f invites: Add backend for multiuse admin invites. 2019-02-07 15:41:00 -08:00
Anders Kaseorg 4bd28f7ae6 migrations: Remove unused imports.
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2019-02-02 17:01:04 -08:00
Hemanth V. Alluri 73d26c8b28 streams: Render and store the stream description from the backend.
This commit does the following three things:
    1. Update stream model to accomodate rendered description.
    2. Render and save the stream rendered description on update.
    3. Render and save stream descriptions on creation.

Further, the stream's rendered description is also sent whenever the
stream's description is being sent.

This is preparatory work for eliminating the use of the
non-authoritative marked.js markdown parser for stream descriptions.
2019-02-01 22:24:18 -08:00
Rishi Gupta 8df0d95559 billing: Remove deprecated RealmAuditLog.requires_billing_update. 2019-02-01 18:49:43 -08:00
Rishi Gupta 4fab08172b billing: Remove deprecated billing fields from Realm. 2019-01-31 15:12:43 -08:00
Pragati Agrawal 1e811b42ec org settings: Add realm level setting for missed message content in email.
This adds a setting under "Notification" section of
"Organization settings" tab, which enables Organization administrator to
control whether the missed message emails include the message content or
not.

Fixes: #11123.
2019-01-25 14:34:10 -08:00
Steve Howell 7e904c3a1a user status: Add status_text field to the database.
The status_text field will be something like "out to lunch."
2019-01-24 09:46:35 -08:00
Marco Burstein 9ddadd39f4 compose: Add support for using Zoom as the video chat provider.
This adds Zoom call properties to the `Realm` model, creates endpoints
for creating calls, adds a frontend and tests.

Fixes #10979.
2019-01-07 10:00:02 -08:00
Shubham Dhama 42c262b807 invite: Replace `invite_as_admin` usage with `invite_as`.
Since we have already added the `invite_as` field to models, we can now
replace usage of `invite_as_admin` properly with its equivalent `invite_as
== PreregistrationUser.INVITE_AS['REALM_ADMIN']`.

Hence, also removed now redundant `invite_as`.
2019-01-05 14:46:38 -08:00
Shubham Dhama c9414a3572 invite: Rename NORMAL_USER to MEMBER of Preregistration.INVITE_AS.
Renamed to keep the user-facing and the codebase terminologies coherent.
2019-01-05 14:43:06 -08:00
Steve Howell a8301ca14a status: Add UserStatus model and core library for away status. 2019-01-02 09:12:03 -08:00
Shubham Dhama 235b0db82b preregistrationuser: Add a model field for all type of invited users.
This adds a model field `invited_as` to PreregistrationUser model class
which will replace `invited_as_admin` in future. Intentions behind adding
this are that we can specify "types" of users for an invited person other
than admin or regular that is, guest user or maybe many others in the
future.
2018-12-30 10:16:17 -08:00
seresheim 49dbd85a89 auth: Add support for Azure Active Directory authentication.
This takes advantage of all of our work on making the
python-social-auth integration reusable for other authentication
backends.
2018-12-18 16:39:03 -08:00
Joshua Pan ad1df0ebeb settings: Add support for customizing the top-left logo.
This adds a new realm_logo field, which is a horizontal-format logo to
be displayed in the top-left corner of the webapp, and any other
places where we might want a wide-format branding of the organization.

Tweaked significantly by tabbott to rebase, fix styling, etc.

Fixing the styling of this feature's loading indicator caused me to
notice the loading indicator for the realm_icon feature was also ugly,
so I fixed that too.

Fixes #7995.
2018-12-18 12:44:52 -08:00
Vishnu Ks 8a1794caa3 message: Store the value of first_visible_message_id in Realm table.
This eliminates a bunch of potentially buggy caching code, with no
material negative side effects.
2018-12-12 15:11:17 -08:00
Marco Burstein ba46dc83c6 notifications: Add a setting for changing the notification sound.
Also, add a new notification sound, "ding". It comes from
https://freesound.org, where the original Zulip notification sound comes
from as well. In the future, new sounds can be added by adding audio
files to the `static/audio/notification_sounds` directory.

Tweaked significantly by tabbott:
* Avoided removing static/audio/zulip.ogg, because that file is
  checked for by old versions of the desktop app.
* Added a views check for the sound being valid + tests.
* Added additional tests.
* Restructured the test_events test to be cleaner.
* Removed check_bool_or_string.
* Increased max length of notification_sound.
* Provide available_notification_sounds in events data set if global
  notifications settings are requested.

Fixes #8051.
2018-12-09 21:25:30 -08:00
Tim Abbott 9d32857c37 realm: Add new Realm.email_address_visibility field.
This is preparatory work for settings controlling who can see user
emails; it includes the API-level support for editing it, but no code
to actually enforce the policy.
2018-12-06 15:56:30 -08:00
Hemanth V. Alluri 6b1d724f5c zerver: Add bugdown rendering for text custom profile fields.
This is the first step of letting users use Zulip markdown in their
SHORT_TEXT and LONG_TEXT custom profile fields, so that they can
include emphasis, links, etc.

This doesn't include any frontend logic yet, however.
2018-11-29 09:47:32 -08:00
Tim Abbott c679920c01 python: Fix unnecessary uses of str_utils library. 2018-11-27 11:44:09 -08:00
Rishi Gupta 6b637428a7 billing: Add Realm.seat_limit.
Currently unenforced, and will mostly need to be set by hand.
2018-11-20 18:50:43 -08:00
Tim Abbott e221a01638 Revert "Change if(realm.domain == mit.edu) to use Realm.is_zephyr_..."
This reverts commit 3645bb9225.

This change was incorrect, because the `is_zephyr_mirror_realm`
property on Realm is a property and thus isn't available in the
migration codebase.

Since this migration is only run for very old servers, this should
have no impact.
2018-11-02 13:16:45 -07:00
Tim Abbott c57c4cf703 notifications: Fix push notifications with multiple realms.
Previously, Zulip did not correctly handle the case of a mobile device
being registered with a push device token being registered for
multiple accounts on the same server (which is a common case on
zulipchat.com).  This was because our database `unique` and
`unique_together` indexes incorrectly enforced the token being unique
on a given server, rather than unique for a given user_id.

We fix this gap, and at the same time remove unnecessary (and
incorrectly racey) logic deleting and recreating the tokens in the
appropriate tables.

There's still an open mobile app bug causing repeated re-registrations
in a loop, but this should fix the fact that the relevant mobile bug
causes the server to 500.

Follow-up work that may be of value includes:
* Removing `ios_app_id`, which may not have much purpose.
* Renaming `last_updated` to `data_created`, since that's what it is now.

But none of those are critical to solving the actual bug here.

Fixes #8841.
2018-10-10 16:15:52 -07:00
Tim Abbott 510afd0c93 migrations: Disable atomic for delivery_email migration.
I'm not sure theoretically why this should be required only for some
installations, but these articles all suggest the root problem is
doing these two migrations together atomically (creating the field and
setting a value for it), so the right answer is to declare the
migration as not atomic:

https://stackoverflow.com/questions/12838111/django-db-migrations-cannot-alter-table-because-it-has-pending-trigger-events
https://confluence.atlassian.com/confkb/upgrade-failed-with-the-error-message-error-cannot-alter-table-content-because-it-has-pending-trigger-events-747606853.html
2018-09-28 14:14:20 -07:00
Tim Abbott 42f49d19b3 emoji: Set default emoji to google-blob for existing users too.
This fixes an inconsistent test failure with test_users.py (that
depended on the ordering between this migration and the creation of
test database users like hamlet).
2018-08-31 15:45:33 -07:00
Harshit Bansal e7311cdf5d emoji: Bring back the feature of changing emojisets.
This is largely inspired by requests from people not liking the
Google's new emojiset. A lot of people were requesting to revert
back to old blobs emojiset so we are re-enabling this feature
after making relevant infrastructure changes for supporting google's
old blob emojiset and re-adding support for twitter emojiset.

Fixes: #10158.
2018-08-31 11:52:24 -07:00
Pragati Agrawal ba4e4e38ae user settings: Add setting to make login notifications optional.
This adds a feature in the "Notification" section of "Settings" tab,
which lets user enable or disable login emails notification.

Tweaked by tabbott to simplify the test.

Fixes: #5795, progress towards #5854.
2018-08-28 14:02:31 -07:00
Vishnu Ks dcb7b15069 billing: Use UserProfile.is_billing_admin instead of Customer.billing_user. 2018-08-27 12:35:46 -07:00
Joshua Pan 79e2266d6e models: Create starred_message_counts boolean field. 2018-08-21 13:42:22 -07:00
Vishnu Ks 9bb338be11 models: Add plan_type to Realm. 2018-08-21 12:39:06 -07:00
Yashashvi Dave d5f3c9b3bc custom profile field: Rename field types in UI. 2018-08-21 11:37:51 -07:00
Tim Abbott 6827c42e02 migrations: Fix accounting in is_private migration.
The previous logic didn't correctly handle the case of total being 0.

Fixes #10378.
2018-08-20 16:49:08 -07:00
Yashashvi Dave 868362046b zerver/models.py: Change custom field name max length to 40 char. 2018-08-17 08:36:22 -07:00
Shubham Padia 40fdc4724b migrations: Set initial value for is_private flags.
This initializes the is_private flag to have the correct value for
historical messages.
2018-08-09 16:09:44 -07:00
Harshit Bansal 9057e543ac emoji: Disable support for letting users switch emojisets.
Due to copyright issues with potentially displaying Apple emojisets on
non-apple devices, as well as iamcal dropping support for the emojione
emojiset (see https://github.com/iamcal/emoji-data/pull/142), we are
dropping (perhaps temporarily) support for allowing users to switch
emojisets in Zulip.

This commit just hides the feature from the user but leaves most of
the infrastructure in place so that in the future if we decide to
re-enable the support we will not need to redo the infrastructure work
(some JS-side code is deleted, mostly because we'll want to re-add the
feature using the do_settings_change infrastructure anyway).

The most likely emoji set to add is the legacy "blobs" Google emoji
set, since it seems popular with some users.

Tweaked by tabbott to remove some additional JS code and update the
changelog.
2018-08-08 15:48:04 -07:00
Tim Abbott 35cb7528f9 models: Add new UserMessage flag active_mobile_push_notification.
This flag is used to track which user/message pairs correspond to an
active mobile push notification, that should potentially be cleared
when the user reads the message.

This flag should never appear on a message that is also marked as
read; eventually we may want a cron job to check for that condition.

We include a partial index on UserMessage for this flag.
2018-08-01 16:51:56 -07:00
Roman Godov 5e70577f84 models: Rename Realm.show_digest_email field.
This renames Realm.show_digest_email field to
digest_emails_enabled, for greater clarity as to what it does
just from seeing the setting name, without having to look it up.

Fixes part of #10042.
2018-08-01 11:05:58 -07:00
Roman Godov c0806917ec models: Rename Realm.restricted_to_domain field.
This renames Realm.restricted_to_domain field to
emails_restricted_to_domains, for greater clarity as to what it does
just from seeing the setting name, without having to look it up.

Fixes part of #10042.
2018-07-31 09:28:33 -07:00
Shubham Padia bf6dc4472b models: Add is_private flag to UserMessage and add index for it.
The is_private flag is intended to be set if recipient type is
'private'(1) or 'huddle'(3), otherwise i.e if it is 'stream'(2), it
should be unset.

This commit adds a database index for the is_private flag (which we'll
need to use it). That index is used to reset the flag if it was
already set. The already set flags were due to a previous removal of
is_me_message flag for which the values were not cleared out.

For now, the is_private flag is always 0 since the really hard part of
this migration is clearing the unspecified previous state; future
commits will fully implement it actually doing something.

History: Migration rewritten significantly by tabbott to ensure it
runs in only 3 minutes on chat.zulip.org.  A key detail in making that
work was to ensure that we use the new index for the queries to find
rows to update (which currently requires the `order_by` and `limit`
clauses).
2018-07-30 15:43:55 -07:00
Roman Godov 34ae3dfd44 models: Delete unused Subscription.notifications field.
This deletes the unused Subscription.notifications field and removes
it from some testing and analytics code (which should not have been
using it in the first place).

Fixes #10042.
2018-07-26 15:54:57 -07:00
Tim Abbott e74c37a129 migrations: Convert migration 0041 to not import from zerver.
This should help avoid problems when upgrading from very old versions of Zulip.
2018-07-24 09:00:14 -07:00
Vishnu Ks 2b28042ddf models: Rename remaining audit log event types to past tense.
This makes the realm audit event type log entries more consistent.
2018-07-22 20:00:28 -07:00
Joshua Pan 533eccd655 models: Create delivery_email field in userprofile.
This commit creates a new field called delivery_email. For now, it is
exactly the same as email upon user profile creation and should stay
that way even when email is changed, and is used only for sending
outgoing email from Zulip.

The purpose of this field is to support an upcoming option where the
existing `email` field in Zulip becomes effectively the user's
"display email" address, as part of making it possible for users
actual email addresses (that can receive email, stored in the
delivery_email field) to not be available to other non-administrator
users in the organization.

Because the `email` field is used in numerous places in display code,
in the API, and in database queries, the shortest path to implementing
this "private email" feature is to keep "email" as-is in those parts
of the codebase, and just set the existing "email" ("display email")
model field to be something generated like
"username@zulip.example.com" for display purposes.

Eventually, we'll want to do further refactoring, either in the form
of having both `display_email` and `delivery_email` as fields, or
renaming "email" to "username".
2018-07-12 12:30:20 +05:30
Rishi Gupta b5753d0ddc billing: Add initial support for seat based plans.
The main remaining todo for correctly populating
RealmAuditLog.requires_billing_update is supporting the de-seating (and
corresponding re-seating) that happens after being offline for two weeks.
2018-07-09 14:33:08 +05:30
Tim Abbott 7ccefc3e5d migrations: Remove dependence on PRIVATE_STREAM_HISTORY_FOR_SUBSCRIBERS.
We don't want to keep around a declaration of
PRIVATE_STREAM_HISTORY_FOR_SUBSCRIBERS forever, so we should just move
this to a getattr; if the user has set it on their server, we'll use
the value; otherwise, we just use False.
2018-05-29 07:36:53 -07:00
Yashashvi Dave e82c879b85 custom fields: Add user type of custom fields.
Fixes #8878
2018-05-27 23:01:21 -07:00
Tim Abbott 6967b6519d settings: Add a development-only setting for less dense mode.
This should make it easier for us to iterate on a less-dense Zulip.

We create two classes on body, less_dense_mode and more_dense_mode, so
that it's easy as we refactor to separate the two concepts from things
like colors that are independent.
2018-05-24 12:31:37 -07:00
Tim Abbott 3853e133f4 migrations: Add missing reverse_code for migration 0167. 2018-05-21 09:42:19 -07:00
Steve Howell ff097623fa Add SubMessage table. 2018-05-16 15:13:33 -07:00
Tim Abbott f0ef335412 models: Remove unused ModelReprMixin class.
It appeared to be used as a base class in various Django migrations,
but because it didn't define any model fields, it wasn't actually.
2018-05-15 19:11:22 -07:00
Shubham Padia a5759108d3 models: Add field is_announcement_only to stream. 2018-05-13 09:06:20 -07:00
Yago González 6837fc5d56 i18n: Add missing strings for custom profile fields and fix capitalization.
The "Short/Long Text" option for custom profile fields wasn't properly
capitalized (i.e. "Text" should have been all lowercase), and also
wasn't properly tagged for translation.

For the sake of consistency, the change to proper capitalization has
also been applied to the models and any tests involving this feature.

Due to a bug in Django, it complained about the models having changed
and thus not being consistent with the migrations. That isn't actually
true (since the database stores the numeric values for each key), but
the migrations have been modified to avoid this error. This does not
affect the migrations' behaviour in any way.
2018-05-06 19:44:36 -07:00
Aditya Bansal 1f358954be web_public_streams: Add is_web_public to Stream table.
Also add function do_change_stream_web_public in lib/actions.py
to help in changing a streams web public status.
2018-05-02 15:23:33 -07:00
Angelika Serwa f4f64243dd custom_profile_fields: Support changing the sort order of the fields.
Tweaked by tabbott for variable naming and the URL.

Closes #8879.
2018-04-30 18:17:41 -07:00
Yashashvi Dave 0d7d94d0db custom fields: Add support for custom URL field type. 2018-04-30 10:53:23 -07:00
Yashashvi Dave 4033f210af custom fields: Add support for custom date field type. 2018-04-30 23:04:25 +05:30
Eeshan Garg 057ff9c91e models: Add Stream.history_public_to_subscribers.
This commit adds a new field history_public_to_subscribers to the
Stream model, which serves a similar function to the old
settings.PRIVATE_STREAM_HISTORY_FOR_SUBSCRIBERS; we still use that
setting as the default value for new streams to avoid breaking
backwards-compatibility for those users before we are ready with an
actual UI for users to choose directly.

This also comes with a migration to set the value of the new field for
existing streams with an algorithm matching that used at runtime.

With significant changes by Tim Abbott.

This is an initial part of our efforts on #9232.
2018-04-28 22:54:04 -07:00
Tim Abbott 7d6bb3dcb4 settings: Remove obsolete default_desktop_notifications setting.
This actually hasn't been hooked up to do anything in years.

While we're at it, we remove the entire "Zulip Labs" settings page.
2018-04-28 13:46:07 -07:00
Shubham Dhama 30b1ec9433 org settings: Change default allow_community_topic_editing value.
This changes the default value of allow_community_topic_editing to
`True` which was merged with `False` default value temporarily.
2018-04-28 13:00:29 -07:00
YJDave 6bef44a9fa org setting: Add time limits for message deleting.
Add realm setting to set time limit for message deleitng.
Set default value of message_content_delete_limit_seconds
to 600 seconds(10 min).

Thanks to Shubham Dhama for rebasing and reworking this.  Some final
edits also done by Tim Abbott.

Fixes #7344.
2018-04-27 19:22:28 -07:00
Umair Khan cf2f6b38dd profile: Add choice field.
Fixes part of #8878
2018-04-26 00:35:51 -07:00
Vishnu Ks c9e932a7ce settings: Add support for Hangouts as the video chat provider.
The only thing that's annoying about this feature is that you need to
be a paying G Suite customer to use it.
2018-04-23 09:39:47 -07:00
Vishnu Ks d1c143de42 models: Add attribute for video provider in realm. 2018-04-23 09:15:12 -07:00
Shubham Dhama 1191f1730a guest: Add a model field for guest users.
This adds new field `is_guest` to UserProfile model and
is meant for the new type of user i.e. "Guest Users".

(Part of #8385).
2018-04-20 16:08:29 -07:00
Umair Khan 0420b89468 profile: Add hint attribute to custom profile fields.
This is the model for #8876.
2018-04-19 11:32:35 -07:00
Shubham Dhama 0e6757af5c org settings: Change default realm description to empty string.
This fixes a traceback that users would get when editing the realm
description just after creating a new organization.
2018-04-06 15:15:47 -07:00
Tim Abbott 758d7b9146 bots: Clean up editing bots impacting non-bot users.
This fixes a bug where the endpoint for editing bot users would allow
an organization administrator to edit the full name of a bot user.

A combination of this an another recently fixed bug made it possible
for this process to set a `bot_owner` for a non-bot user; so we also
include a migration to fix that for any users that might have had our
model invariants corrupted in that way.
2018-04-02 18:58:26 -07:00
Umair Khan c30a282dd9 profile: Remove integer and float fields. 2018-04-02 09:46:21 -07:00
Shubham Dhama b7aae89029 org settings: Add realm level default time format setting. 2018-03-31 16:23:26 -07:00
Tim Abbott 4570936ac2 models: Fix default value for last_reminder.
This was causing a rather confusing test flake in
test_stream_error_pm_to_bot_owner.  What was happening was that if
this test (which used that code path) ran within 5 minutes of the
populate_db run, it would fail.
2018-03-29 11:53:12 -07:00
Tim Abbott f7c563dfcc migrations: Fix 0149_realm_emoji migration for S3 case.
The original migration implementation didn't follow the S3 API correctly.
2018-03-23 10:48:50 -07:00
Sarah ecd75ccba6 models: Add allow_community_topic_editing setting. 2018-03-22 16:02:24 -07:00
Harshit Bansal a49655e0d4 emoji: Migrate realm emoji to be addressed by `id` rather than `name`.
This commit migrates realm emoji to be addressed by their `id` rather
than their name. This fixes a long standing issue which was causing
an error on uploading an emoji with same name as a deactivated realm
emoji.

Fixes: #6977.
2018-03-20 22:24:44 -07:00
Tim Abbott b94a24ffe3 reactions: Fix buggy migration for realm emoji.
The original implementation of this migration had a highly unfortunate
bug that would result in it deleting all reactions to realm emoji on
the server; we missed this in review, so essentially all historical
realm emoji reactions on chat.zulip.org were lost :(.

We both correct the problem, and also add logging of the deleted rows
that would help should anything be deleted erroneously.
2018-03-20 21:50:47 -07:00
Greg Price dc1eeef30a antispam: Make a setting for default Realm.max_invites.
This makes this value much easier for a server admin to change than it
was when embedded directly in the code.  (Note this entire mechanism
already only applies on a server open for anyone to create a realm.)

Doing this also means getting the default out of the database.
Instead, we make the column nullable, and when it's NULL in the
database, treat that as whatever the current default is.  This better
matches anyway the likely model where there are a few realms with
specially-set values, and everything else should be treated uniformly.

The migration contains a `RenameField` step, which sounds scary
operationally -- but it really does mean just the *field*, in
the model within the Python code.  The underlying column's name
doesn't change.
2018-03-16 18:00:11 -07:00
Tim Abbott 69a7069ac4 migrations: Fix text version to Custom Emoji.
The fact that we need to do this is basically a Django bug; these
strings aren't used in the database itself.
2018-03-16 17:16:18 -07:00
Greg Price 4139e6c763 reactions: Fix migration to correctly handle corner case.
If an emoji that was deleted was the only realm emoji, or more
generally if all realm emoji were deleted, then we would just leave
the reaction unchanged, with an `emoji_code` that is now corrupt.

Instead, treat this case the same as if only this emoji was deleted
while others remain.
2018-03-15 18:53:51 -07:00
Vishnu Ks a44255eedb emails: Add backend for disallowing disposable email addresses. 2018-03-11 22:05:58 -07:00
YJDave c94b21e9ac settings: Add setting to disable message content in missed message emails.
Fixes #6938.
2018-03-09 21:16:02 -08:00
Harshit Bansal 5aa8629a10 reactions: Migrate `emoji_code` to store `id` for realm emoji.
Till now, we had been storing realm emoji's name in emoji code field
in reactions' model. This commit migrates it to store realm emoji's id.
It is a part of effort to migrate realm emojis to be referenced by their
id and not by name.
2018-03-09 13:46:27 -08:00
Shubham Dhama 777b6de689 org settings: Add setting to prevent users from adding bots.
Fixes: #7908.
2018-03-09 13:21:55 -08:00
Marco Burstein bdb86f1b5e emoji: Add support for translating emoticons.
Add `translate_emoticons` to `prop_types` and `expected_keys`.
Furthermore, create a emoji-translating Markdown inline pattern.

Also use a JavaScript version of `translate_emoticons` and then use
this function during Markdown previews and as a preprocessor. This
is only needed for previews, because usually emoticon translation
happens on the backend after sending.

Add tests for emoticon translation, a settings UI, and a /help/ page
as well.

Tweaked by tabbott to fix various test failurse as well as how this
handles whitespace, requiring emoticons to not have adjacent
characters.

Fixes #1768.
2018-03-04 15:37:24 -08:00
Vishnu Ks 497ee97ce3 models: Change description attribute of user group to textfield. 2018-03-01 11:27:26 -08:00
Aastha Gupta d124597f4b org-settings: Add setting to turn off welcome emails.
This adds an organization-level setting to provide an option to turn
off the welcome emails.

Fixes #8000.
2018-02-28 12:39:01 -08:00
Aditya Bansal f50e325075 RealmAuditLog: Fill subscription events with event_last_message_id=None.
Creating the very first organization administrator user and
subscribing them to streams before any messages were sent resulted in
RealmAuditLog entries being created with a `event_last_message_id` of
None, because that's the maximum ID in the empty set.

We correct this by fixing the incorrectly created RealmAuditLog
entries, both for new servers and also fixing old broken entries on
existing servers.

This fixes an issue where if a user setup a Zulip server with just the
organization administrator, and then forgot about it (so that the
initial user became soft-deactivated), trying to sign in 3 weeks later
would throw an exception.

This fixes the issue reported here:

https://chat.zulip.org/#narrow/stream/9-issues/subject/500.20error.20on.20login/near/511981
2018-02-22 09:45:22 -08:00
Greg Price 52f1c57c2a migrations: Fix mypy error in placeholder migration. 2018-02-05 18:03:07 -08:00
Roman Godov b875fe07eb settings: Added setting to turn on and off realm name in email subject.
Users having only account in one realm will not be distracted by realm
name in subject lines of every email.  Users who have multiple
accounts in realms can turn this setting on and receive a
corresponding realm name in email's subject.

Tweaked by tabbott to rebase and address a few small issues.

Fixes #5489.
2018-02-05 18:01:54 -08:00
Greg Price 6f128279e8 migrations: Add reverser for emoji_alt_code migration.
This is easy to do, and prevents this feature from getting a server
admin stuck in potentially a pretty uncomfortable way -- unable to
roll back a deploy.
2018-02-05 16:55:21 -08:00