Commit Graph

291 Commits

Author SHA1 Message Date
Steve Howell a04a095738 Speed up alert word detection during message sends.
We no longer use all the alert words for all the users in the
entire realm when we look for alert words in a newly sent/edited
message.  Now we limit the search to only all the alert words
for all the users who will get UserMessage records.  This will
hopefully make a big difference for big realms where most messages
are only sent to a small subset of users.
2016-09-23 15:24:55 -07:00
Steve Howell 40b18094ec alert words: Refactor alert word detection.
The bugdown parser no longer has a concept of which users need which
alert words, since it can't really do anything actionable with that info
from a rendering standpoint.

Instead, our calling code passes in a set of search words to the parser.
The parser returns the list of words it finds in the message.
Then the model method builds up the list of user ids that should be
flagged as having alert words in the message.

This refactoring is a little more involved than I'd like, but there are
still some circular dependency issues with rendering code, so I need to
pass in the rather complicated realm_alert_words data structure all the way
from the action through the model to the renderer.

This change shouldn't change the overall behavior of the system, except
that it does remove some duplicate regex checks that were occurring when
multiple users may have had the same alert word.
2016-09-23 15:21:17 -07:00
Steve Howell 6b3f945a7e dead code: Remove linebreak(). 2016-09-19 18:25:03 -07:00
Steve Howell c2277afd06 Simplify bulk_get_streams().
We can always expect a realm here.
2016-09-19 18:25:03 -07:00
Steve Howell eb09dd217d Simplify get_stream_backend.
Assume that get_stream_backend() always receives a realm, not a
realm id.  We can be pretty confident of that now due to mypy.
2016-09-19 18:25:03 -07:00
Steve Howell 1dbc94bcd9 annotations: Require a Realm for get_stream(). 2016-09-19 18:25:02 -07:00
Steve Howell e0d59d9386 dead code: Remove Stream.create(). 2016-09-19 18:25:02 -07:00
Tim Abbott 83ef7e72e6 models: Fix missing mypy annotations in subdomains code. 2016-09-10 11:57:07 -07:00
Tim Abbott 2b0a486077 Eliminate cross-realm PM conversation with ADMIN_DOMAIN.
In the future, we can use UserProfile.is_staff to control this, but
there isn't a current use case for it.
2016-08-22 21:46:16 -07:00
Tim Abbott b8050ce02f Add realm.host property. 2016-08-18 18:48:40 -07:00
Tim Abbott 4fbb8c3eee templates: Add new context variables to all templates.
This adds a few new helpful context variables that we can use to
compute URLs in all of our templates:
* external_uri_scheme: http(s)://
* server_uri: The base URL for the server's canonical name
* realm_uri: The base URL for the user's realm

This is preparatory work for making realm_uri != server_uri when we
add support for subdomains.
2016-08-13 16:27:35 -07:00
Rishi Gupta 708b416ca1 Terms of Service: Fix corner cases around new users being created.
This fixes a few bugs in 7910a6e134,
related to automatically created user accounts.
2016-08-13 00:19:54 -07:00
Tim Abbott 386c6c2a31 settings: Refactor SYSTEM_ONLY_REALMS configuration. 2016-08-12 12:12:53 -07:00
Rishi Gupta 7910a6e134 Terms of Service: Add ability to update TOS and have users re-sign.
Most directly useful for the migration to zulipchat.com.

Creates a new field in UserProfile to store the tos_version, as well as two
new settings TOS_VERSION and FIRST_TIME_TOS_TEMPLATE. We check for a version
mismatch between what the user has signed and the current
settings.TOS_VERSION whenever the user hits the home page, and redirect them
if needed.

Note that accounts_accept_terms.html and
zerver.views.accounts_accept_terms were unused before this commit
(they date from c327446537)
2016-08-11 23:37:48 -07:00
Rag Sagar 2fef36f15a Add realm-level default language setting.
Adds a new field default language in the zerver_realm model.
This realm level default language will be used as default language
for newly created users. Realm level default language can be
changed from the administration page.

Fixes #1372.
2016-08-09 17:38:29 -07:00
Rag Sagar 28f0079432 Use a constant to set UserProfile default_language field max_length attribute. 2016-08-09 16:10:24 -07:00
Umair Khan ef62506fb1 Annotate zerver/models.py. 2016-08-08 15:13:29 -07:00
Tim Abbott df525ad1c5 Remove old MitUser model and related code.
The MitUser model caused a constant series of little problems for
users with mit.edu email addresses trying to sign up for different
Zulip servers.

The new implementation just uses conditionals on the realm object when
selecting the confirmation template to use.
2016-07-26 20:30:12 -07:00
Tim Abbott 375551aaa6 Clean up most hardcoding of mit.edu domain checks.
This moves all this code to be gated on a few virtual realm settings.
2016-07-26 20:30:12 -07:00
Tim Abbott 70543e059a bugdown: Remove hardcoding of mit.edu for zephyr_mirror realm config. 2016-07-26 20:30:12 -07:00
Tim Abbott c17676b00c Cleanup MIT hardcoding for disabling presence. 2016-07-26 20:30:12 -07:00
Steve Howell 57c436b32d Add TODOs for #1379 (bugdown mutual dependency) 2016-07-25 14:59:32 -07:00
Steve Howell 8ac1398b0f Use topic_name() in to_log_dict(). 2016-07-18 14:10:11 -07:00
Rishi Gupta d529a94e4d Add realm setting to time-limit editing of message content.
This is controlled through the admin tab and a new field in the Realms table.
Notes:
* The admin tab setting takes a value in minutes, whereas the backend stores it
  in seconds.
* This setting is unused when allow_message_editing is false.
* There is some generosity in how the limit is enforced. For instance, if the
  user sees the hovering edit button, we ensure they have at least 5 seconds to
  click it, and if the user gets to the message edit form, we ensure they have
  at least 10 seconds to make the edit, by relaxing the limit.
* This commit also includes a countdown timer in the message edit form.

Resolves #903.
2016-07-15 13:55:49 -07:00
Steve Howell 9d7716f368 Add message.topic_name() helper function.
This little helper will facilitate adding a new Topic
table in the future.
2016-07-14 09:54:56 -07:00
Eklavya Sharma 958335bdb3 Change all default values in models to unicode. 2016-07-11 21:30:32 -07:00
Tomasz Kolek d7c7279523 Add is_status_message method to Message model. 2016-07-11 12:09:06 -07:00
Rishi Gupta 43c2f35776 Add realm setting to disable message editing.
This is controlled through the admin tab and a new field in the Realms
table.  This mirrors the behavior of the old hardcoded setting
feature_flags.disable_message_editing.  Partially resolves #903.
2016-07-10 11:57:24 -07:00
Rishi Gupta 07e7230ae1 models.py: Add post_save to Message.
Flushes message from cache after a message.save(). Needed for tests
where we directly manipulate Message objects in the database.
2016-07-10 11:42:18 -07:00
Tim Abbott 1e2d38e790 Move stringify_message_dict into to_dict_uncached. 2016-07-08 17:58:56 -07:00
Umair Khan 035fceb814 Add dynamically loaded language dropdown. 2016-07-04 11:56:02 -07:00
Eklavya Sharma 6d8ba90db9 zerver/models.py: Modify imports for mypy in py3 mode.
Change 'from zerver.lib import bugdown' to
'import zerver.lib.bugdown as bugdown' to make zerver/models.py
pass mypy check in python 3 mode.
2016-07-02 10:38:48 -07:00
Kartik Maji f8bb7503e6 Add ability to pin streams to top of the streams sidebar list.
Based on work by Lauren Long, with some tweaks by tabbott.
2016-06-30 22:26:09 -07:00
acrefoot acd7ff7aff Fix typo in Recipient comment. 2016-06-27 18:34:02 -07:00
Tim Abbott b2a24e0306 Revert "Add authorization check before serving files."
This reverts commit e985b57259.

This commit will break production when we next do a release, because
we haven't done a migration to create Attachment objects for
previously uploaded files.
2016-06-27 12:09:56 -07:00
rahuldeve e985b57259 Add authorization check before serving files. 2016-06-27 11:24:35 -07:00
rahuldeve 8cecb37743 Modify Attachment model to track file access permissions. 2016-06-23 17:46:16 -07:00
Vishnu Ks ad1c3894d9 Add interface for creating new realms.
This is controlled by settings.OPEN_REALM_CREATION; if that setting is
off, this feature doesn't do anything.
2016-06-17 16:15:28 -07:00
Eklavya Sharma 7ec9cb7e93 Annotate model fields: Attachment.
Also fix clashing annotations.
2016-06-13 20:01:03 +05:30
Eklavya Sharma 86978cb2a3 Annotate model fields: Stream. 2016-06-13 20:01:03 +05:30
Eklavya Sharma 1ec7e124c7 Annotate model fields: Message. 2016-06-13 20:01:02 +05:30
Eklavya Sharma f27cff57c3 Annotate model fields: UserProfile. 2016-06-13 20:01:02 +05:30
Eklavya Sharma 9f39c9276f Annotate model fields: DefaultStream, Referral, ScheduledJob. 2016-06-13 19:37:07 +05:30
Eklavya Sharma 4f890cca2a Annotate model fields: UserActivity, UserActivityInterval, UserPresence. 2016-06-13 19:37:07 +05:30
Eklavya Sharma 7290f9cb83 Annotate model fields: Subscription and Huddle. 2016-06-13 19:37:07 +05:30
Eklavya Sharma ddaaa98b25 Annotate model fields: UserMessage. 2016-06-13 19:30:58 +05:30
Eklavya Sharma acd1767398 Annotate model fields: Recipient and Client. 2016-06-13 19:30:57 +05:30
Eklavya Sharma c0004a5874 Annotate model fields: PushDeviceToken and MitUser. 2016-06-13 19:30:57 +05:30
Eklavya Sharma 04740fb620 Annotate model fields: PreregistrationUser. 2016-06-13 19:30:56 +05:30
Eklavya Sharma d666e00833 Annotate model fields: RealmEmoji and RealmFilter. 2016-06-13 19:30:56 +05:30
Eklavya Sharma 6c7dd07ec2 Annotate model fields: Realm and RealmAlias. 2016-06-13 19:30:56 +05:30
Eklavya Sharma c654c4032d zerver/models.py: Annotate get_display_recipient.
get_display_recipient's annotation clashes with other wrong annotations.
Fix those wrong annotations.
Since get_display_recipient returns a Union, use isinstance checks and
casts to make mypy checks succeed.
2016-06-12 23:34:57 +05:30
Eklavya Sharma ed61c4c581 Improve model string representation.
Define __str__, __repr__ and __unicode__ correctly on models.
This will help in python 3 compatibility.
2016-06-12 09:55:12 -07:00
Eklavya Sharma 17b9422546 zerver/models.py: Fix regex strings.
Some regex strings were not declared as raw, even though they used
a lot of backslashes.
2016-06-12 09:31:19 -07:00
Eklavya Sharma 81759d56be zerver/models.py: Fix string annotations in UserPresence. 2016-06-12 09:30:53 -07:00
Eklavya Sharma e11bec28c2 zerver/models.py: Fix annotations (str -> text_type).
Change str to text_type where appropriate in annotations related to
Attachment, PreregistrationUser and Huddle.
2016-06-12 09:30:53 -07:00
Eklavya Sharma d76bea8f25 zerver/models.py: Identify functions returning QuerySets.
Identify functions which return QuerySets and give them a return type
`Sequence` with appropriate parameter.  Typing them as QuerySet will
not be useful since generic stubs for QuerySets are not available and
not knowing the type of QuerySets is hardly useful for type checking.
2016-06-12 09:30:53 -07:00
Eklavya Sharma 39060aa221 zerver/models.py: Fix annotations related to Message. 2016-06-12 09:30:53 -07:00
Eklavya Sharma 2841aa642d Fix annotations related to make_safe_digest and hashes. 2016-06-12 09:30:53 -07:00
Eklavya Sharma 9020177418 zerver/models.py: Add annotations related to Stream and Client. 2016-06-12 09:30:53 -07:00
Eklavya Sharma 10f2ec043d Fix zerver.lib.utils.generate_random_token.
generate_random_token used to return a value of type six.binary_type
and its return type was annotated as `str`.  This commit fixes that
by making it return a value of type `six.text_type` and updating
the annotation accordingly.
Also fix clashing annnotations.
2016-06-12 09:30:53 -07:00
Eklavya Sharma 56d5785c2e zerver/models.py: Change some constants to unicode literals.
Change choices of UserProfile.avatar_sources and UserProfile.tutorial_status
from str literals to unicode literals.  This is done because these fields
are CharFields, which are of type `six.text_type`.  So the set of values
which they can take should also be of the type `six.text_type`.
Also fix clashing annotations.
2016-06-12 09:30:53 -07:00
Eklavya Sharma 0cc7a6583c zerver/models.py: Fix last_reminder_tzaware annotation.
Fix annotated return type of UserProfile.last_reminder_tzaware from
`str` to `Optional[datetime.datetime]`.
2016-06-12 09:30:33 -07:00
Eklavya Sharma 8acc51218e Fix realm emoji and realm filter annotations.
Change `str` to `text_type` in annotations in zerver/models.py
related to realm emoji and realm filters.
Also fix clashing annotations in zerver/lib/bugdown/__init__.py.
2016-06-12 09:25:42 -07:00
Eklavya Sharma e68d99eb2e zerver/models.py: Make cache_keys text_type. 2016-06-12 09:25:32 -07:00
Tim Abbott f44b227b85 Refactor getting cross-realm users into a function. 2016-06-11 11:22:22 -07:00
Eklavya Sharma 53084fe03c Use text_type as type of cache keys and update users.
This changes the type annotations for the cache keys in Zulip to be
consistently text_type, and updates the annotations for values that
are used as cache keys across the codebase.
2016-06-11 09:10:34 -07:00
Eklavya Sharma d3b80d94a2 Use appropriate string types and correctly encode/decode them. 2016-06-11 17:34:23 +05:30
Tomasz Kolek 999093b227 Add new is_incoming_webhook bot type.
This type of bot is only able to send messages via webhook endpoints.
2016-06-07 21:23:35 -07:00
Tim Abbott 9c5f15e89b models: Fix use of non-lazy ugettext at import time.
Was introduced in 03debdf82f.
2016-06-04 17:46:03 -07:00
Tim Abbott 6f69053911 Annotate most of the rest of bugdown. 2016-06-04 11:35:29 -07:00
Tim Abbott ac3989c114 models: Add most missing type annotations. 2016-06-04 00:03:54 -07:00
Tim Abbott 03debdf82f Fix malformed error message when creating invalid Realm Emoji.
Thanks to Greg McCoy for his help finding this bug.
2016-06-03 23:12:36 -07:00
Max 0f4673ae3b Add type annotation to mention.py, redis_utils.py, timestamp.py, user_agent.py
Some functions in models.py had input typed as int when they needed to be typed as datetime.datetime
2016-06-03 19:00:16 -07:00
Max 1148f6ff8a Rename timestamp kwarg in to_presence_dict to dt
It is not a timestamp, it is a datetime object. This is better ducktyping
2016-06-03 19:00:16 -07:00
Ashish Kumar 31bf6b8259 Type annotation of zerver/models.py
[Substantially revised by tabbott]

This probably still has some bugs in it, but having mostly complete
annotations for models.py will help a lot for the annotations folks
are adding to other files.
2016-06-02 23:28:34 -07:00
Vishnu Ks 100d885f23 Change default announcement stream to announce.
Fixes #788.
2016-05-31 07:38:07 -07:00
Tomasz Kolek 8c18b8947f Add bot_type field to UserProfile.
This is intended to support creating different types of bots with
potentially limited permissions.
2016-05-19 22:37:37 -07:00
Tim Abbott b01196db86 to_log_dict: Add sender_id to logged fields. 2016-05-18 23:02:43 -07:00
Aristeidis Fkiaras 3ee210d9e8 Add setting to only allow admins create new streams.
Fixes: #691.

Thanks to Preston Hansen for work on this feature!
2016-05-18 18:53:13 -07:00
Tim Abbott 92bec8cfea Merge Zulip 1.3.12 security release. 2016-05-10 11:32:26 -07:00
Tim Abbott 07fc47f953 CVE-2016-4426: Fix non-admin users having access to all bot API keys.
Long ago, there was work on an experimental integration model where
every user in a realm would have administrative control over all bots,
with the goal of simplifying the process of setting up communally
administered bots for smaller teams.  While that new model was never
fully implemented (and thus never setup as an option), an error in
that original implementation meant that the data on all bots in a
realm, including their API keys, was sent to the browsers of users via
the `realm_bots` variable in `page_params`.  The data wasn't displayed
in the UI for non-admin users, but was available via e.g. the
javascript console.

This commit updates this behavior to only send sensitive bot data like
API keys to the owner of the bot (and realm admins).

We may in the future implement a model simplifying communally
administered integrations, but if we do that, those bots should be
limited in their capabilities (e.g. only able to send webhook
messages).

This bug has been present since Zulip was released as open source.
2016-05-10 09:50:02 -07:00
Tim Abbott 2a2cbd60c3 cache: Fix fragile active_bot_dicts_in_realm caching model.
The issue here is similar to that in the previous commit.
2016-05-09 10:12:35 -07:00
Tim Abbott fbc7e977ac cache: Fix fragile active_user_dicts_in_realm caching model.
Previously we relied on having two matching list of fields for the
get_active_user_dicts_in_realm, one in the actual code and the other
in the caching system.  By unifying these lists to have a single
source, we eliminate a class of caching bugs we might otherwise
regularly introduce.
2016-05-09 10:12:35 -07:00
rahuldeve dde832b158 Add Attachment model to keep track of uploads.
This commit adds the capability to keep track and remove uploaded
files.  Unclaimed attachments are files that have been uploaded to the
server but are not referred in any messages.  A management command to
remove old unclaimed files after a week is also included.

Tests for getting the file referred in messages are also included.
2016-05-02 22:14:47 -07:00
Tim Abbott d9e4968d6f Increase maximum URL length for RealmEmoji to 1000.
The default of 200 was shorter than the Camo URLs use by Zulip.
2016-05-02 19:02:56 -07:00
Tim Abbott 5bd94c15c7 Use camo to avoid mixed content warnings when displaying emoji. 2016-05-02 17:21:31 -07:00
Vladislav Manchev f5e6176aea Add custom realm emoji UI to administration page. 2016-04-26 13:15:54 -07:00
Tim Abbott 79297898f1 Remove obsolete AppleDeviceToken model. 2016-04-20 21:51:52 -07:00
Tim Abbott 49799440a4 Replace use of django-guardian with fields on UserProfile.
As documented in https://github.com/zulip/zulip/issues/441, Guardian
has quite poor performance, and in fact almost 50% of the time spent
running the Zulip backend test suite on my laptop was inside Guardian.

As part of this migration, we also clean up the old API_SUPER_USERS
variable used to mark EMAIL_GATEWAY_BOT as an API super user; now that
permission is managed entirely via the database.

When rebasing past this commit, developers will need to do a
`manage.py migrate` in order to apply the migration changes before the
server will run again.

We can't yet remove Guardian from INSTALLED_APPS, requirements.txt,
etc. in this release, because otherwise the reverse migration won't
work.

Fixes #441.
2016-04-20 21:51:52 -07:00
Tim Abbott 28d1a3105c models: Add a __repr__ for Client. 2016-04-20 15:26:51 -07:00
Tim Abbott b61d73fc93 Delete unused old StreamColor model. 2016-04-08 13:06:04 -07:00
Varshit 4e1060076d Purge 'from typing import *' from zerver/.
This is a partial implementation of #636.
2016-04-07 14:07:07 -07:00
Tim Abbott d8f7d89fb4 Add PEP-484 type annotations to zerver/models.py.
Done pair-programming with Guido.
2016-04-03 15:40:06 -07:00
Tim Abbott b99313545e Add PEP-484 type annotations to global dictionaties. 2016-04-03 15:40:06 -07:00
Tim Abbott a407f090e1 models: Document some of the more interesting model classes. 2016-04-01 09:34:45 -07:00
Ryan Moore 15cc3fde7b s/memcache/remote_cache/ in models 2016-03-31 12:54:29 -07:00
Tim Abbott df0d2a726d python3: Add missing utf-8 encoding/decoding in various places. 2016-03-08 09:14:15 -08:00
Reid Barton 9735025167 Refactor logic around restricted_to_domain.
Add a function email_allowed_for_realm that checks whether a user with
given email is allowed to join a given realm (either because the email
has the right domain, or because the realm is open), and use it
whenever deciding whether to allow adding a user to a realm.

This commit is not intended to change any behavior, except in one case
where the Zulip realm's domain was not being converted to lowercase.
2016-02-07 10:54:52 -05:00
Tim Abbott 05a827c520 need_to_render_content: Fix comparison with None.
If the content wasn't rendered, both rendered_content and
rendered_content_version would be None.  In addition to being
confusing, in Python 3, `None < 2` is an error and this code breaks.
2016-01-26 21:11:25 -08:00