Commit Graph

566 Commits

Author SHA1 Message Date
Rishi Gupta c1bbd8d70d models.py: Add get_realm_by_string_id to assist with Realm.domain migration.
Step 0 of a two step process:

1. Replace all occurances of get_realm(domain) with
   get_realm_by_string_id(string_id)

2. Rename get_realm_by_string_id to get_realm.
2016-11-02 22:46:34 -07:00
Rishi Gupta 9ef8536cc6 models.Realm: Require Realm.string_id to be non-NULL.
Adds a database migration, adds a new string_id argument to the management
realm creation command, and adds a short name field to the web realm
creation form when REALMS_HAVE_SUBDOMAINS is False.
2016-11-02 22:46:34 -07:00
Rishi Gupta 64bcd71d6e models.Realm: Rename subdomain to string_id.
Does a database migration to rename Realm.subdomain to
Realm.string_id, and makes Realm.subdomain a property.  Eventually,
Realm.string_id will replace Realm.domain as the handle by which we
retrieve Realm objects.
2016-11-02 22:46:34 -07:00
Steve Howell 7883cecf28 Send user_id to the client in more places.
This commit touches:

    realm_bot/add
    realm_user/add
    page_params.bot_list
    page_params.people_list
2016-10-28 13:40:14 -07:00
K.Kanakhin 39e0886361 retention-policy: Add tool to determine expired messages.
This is a first step towards implementing a message retention policy
feature.

- Add Realm model message_retention_days field to setup
  messages expired period for realm.
- Add migration.
- Add tool to get expired messages for each Realm.
- Add tests to cover tool for getting expired messages.
2016-10-25 15:38:08 -07:00
Tomasz Kolek 4790316b57 Add user setting option to always send push notifications.
Add option in user's settings for getting mobile push notifications
even if a Zulip browser is online.  Default is False.

Fixes: #1596.
2016-10-25 10:52:29 -07:00
Rishi Gupta b6da450b96 admin.js: Use RealmAlias instead of Realm to show the restricted domains.
Passes the allowed domains for a realm to the frontend, via
page_params.domains. Groundwork for allowing users to add and
remove domains via the admin setting page, rather than via the
realm_alias.py management command.
2016-10-25 10:07:20 -07:00
Rishi Gupta 537ee75761 Create RealmAlias entries for Realm.domain.
This is a preliminary step towards eliminating the realm.domain field
in favor of realm.subdomain.  Includes a database migration to create
these for existing realms.
2016-10-25 10:06:39 -07:00
Tim Abbott 5bea2f5e20 Remove unused AVATAR_FROM_SYSTEM code.
This is some of the code we'd need if we wanted to have Zulip generate
avatars for things.  Since it is so little useful code, and it's not
clear we will need this feature ever, we can remove this code to make
the codebase less confusing.  It'd be easy to dig this out of history
if we ever want it.

Fixes #2101.
2016-10-22 19:48:50 -07:00
Umair Khan eb7ce73628 Move name_changes_disabled to zerver.models 2016-10-10 09:27:52 -07:00
Rishi Gupta 777fcaa6a0 Add new organization type field to Realm objects.
Adds a new field org_type to Realm.  Defaults for restricted_to_domain
and invite_required are now controlled by org_type at time of realm
creation (see zerver.lib.actions.do_create_realm), rather than at the
database level.  Note that the backend defaults are all
org_type=corporate, since that matches the current assumptions in the
codebase, whereas the frontend default is org_type=community, since if
a user isn't sure they probably want community.

Since we will likely in the future enable/disable various
administrative features based on whether an organization is corporate
or community, we discuss those issues in the realm creation form.
Before we actually implement any such features, we'll want to make
sure users understand what type of organization they are a member of.

Choice of org_type (via radio button) has been added to the realm
creation flow and the realm creation management command, and the
open-realm option removed.

The database defaults have not been changed, which allows our testing code
to work unchanged.

[includes some HTML/CSS work by Brock Whittaker to make it look nice]
2016-10-05 17:01:46 -07:00
Steve Howell 43e21f6fb4 Remove unused bugdown global from models.py.
Fixes: #1379

(It was really a series of fixes leading up to this.)
2016-10-04 11:35:03 -07:00
Steve Howell b2f84f0fa4 Move render_markdown() to lib/message.py.
This removes a bugdown circular dependency.
2016-10-04 11:34:53 -07:00
Steve Howell b81a961fdd Remove Message.set_rendered_content(). 2016-10-04 11:31:21 -07:00
Steve Howell 7fb992dba3 Simplify and "fix" render_old_messages management command.
The command to render old messages now looks for all messages
not matching the bugdown version, and it no longer directly calls
into model code.  We should still be extremely cautious about
using this code.
2016-10-04 11:31:20 -07:00
Steve Howell 583a6bbadd Extract zerver/lib/message.py.
This pulls message-related code from models.py into a new
module called message.py, and it starts to break some bugdown
dependencies.  All the methods here are basically related to
serializing Message objects as dictionaries for caches and
events.

    extract_message_dict
    stringify_message_dict
    message_to_dict
    message_to_dict_json
    MessageDict.to_dict_uncached
    MessageDict.to_dict_uncached_helper
    MessageDict.build_dict_from_raw_db_row
    MessageDict.build_message_dict

This fix also removes a circular dependency related
to get_avatar_url.

Also, there was kind of a latent bug in Message.need_to_render_content
where it was depending on other calls to Message to import bugdown
and set it globally in the namespace.  We really need to just
eliminate the function, since it's so small and used by code that
may be doing very sketchy things, but for now I just fix it.  (The
bug would possibly be exposed by moving build_message_dict out to the
library.)
2016-10-04 11:31:20 -07:00
Steve Howell ac994fdd51 Move three functions from models.py to lib/cache.py.
I move these three functions to lib/cache.py:

    to_dict_cache_key_id
    to_dict_cache_key
    flush_message

This will prepare us for a more significant refactoring that
eventually breaks down some circular dependencies with
Message and bugdown.
2016-10-04 11:31:20 -07:00
Steve Howell 5ae8cff289 code cleanup: Remove unused Attachment methods.
This removes some unused code on Attachment.  Some of these
methods might be useful in a manage.py shell, but without tests,
they are not very trustworthy, and the Attachment model isn't that
complicated to write raw Django queries against.
2016-10-04 10:35:43 -07:00
Tim Abbott 87b9017845 upload: Extract get_avatar_url and add to UploadBackend. 2016-10-02 21:20:43 -07:00
Tim Abbott 22fd7ba02a avatar: Move avatar hash computations to their own file. 2016-10-02 21:19:10 -07:00
hackerkid ea39fb2556 Add option for hosting each realm on its own subdomain.
This adds support for running a Zulip production server with each
realm on its own unique subdomain, e.g. https://realm_name.example.com.

This patch includes a ton of important features:
* Configuring the Zulip sesion middleware to issue cookier correctly
  for the subdomains case.
* Throwing an error if the user tries to visit an invalid subdomain.
* Runs a portion of the Casper tests with REALMS_HAVE_SUBDOMAINS
  enabled to test the subdomain signup process.
* Updating our integrations documentation to refer to the current subdomain.
* Enforces that users can only login to the subdomain of their realm
  (but does not restrict the API; that will be tightened in a future commit).

Note that toggling settings.REALMS_HAVE_SUBDOMAINS on a live server is
not supported without manual intervention (the main problem will be
adding "subdomain" values for all the existing realms).

[substantially modified by tabbott as part of merging]
2016-09-27 23:24:14 -07:00
Steve Howell 1672353558 comments: Add comment to Realm.deployment().
I refer to github issue 1845 here.
2016-09-24 16:56:34 -07:00
Steve Howell bc9a75ab24 Remove unreachable remove_unreachable() method.
Sorry, couldn't resist wording the commit message like that. :)

The remove_unreachable() method on Message was no longer being
used, and the commit history made it fairly clear we won't need it
in the future.
2016-09-24 16:56:34 -07:00
Steve Howell dfa416cfc8 Remove last_reminder_tzaware().
We can now rely on UserProfile.last_reminder being time zone
aware, or even if it isn't, it's a self-correcting problem the
first time a reminder is sent.  (It's a non-problem to be off
by a few timezones if somebody still has an old value there, because
they will still be outside the 1-minute nag window even with the
timezone disparity.)
2016-09-24 16:56:34 -07:00
Steve Howell 6673b28cde Extract domain_in_local_realm_filters_cache().
This is partly a concession to testing; it's really hard to test
that we are flushing the cache properly if tests need to look
at a global variable in models.py that can be re-assigned on every
request.  Extracting this function makes it easy for tests to know
whether a domain is in the local cache.
2016-09-24 16:56:34 -07:00
Steve Howell a04a095738 Speed up alert word detection during message sends.
We no longer use all the alert words for all the users in the
entire realm when we look for alert words in a newly sent/edited
message.  Now we limit the search to only all the alert words
for all the users who will get UserMessage records.  This will
hopefully make a big difference for big realms where most messages
are only sent to a small subset of users.
2016-09-23 15:24:55 -07:00
Steve Howell 40b18094ec alert words: Refactor alert word detection.
The bugdown parser no longer has a concept of which users need which
alert words, since it can't really do anything actionable with that info
from a rendering standpoint.

Instead, our calling code passes in a set of search words to the parser.
The parser returns the list of words it finds in the message.
Then the model method builds up the list of user ids that should be
flagged as having alert words in the message.

This refactoring is a little more involved than I'd like, but there are
still some circular dependency issues with rendering code, so I need to
pass in the rather complicated realm_alert_words data structure all the way
from the action through the model to the renderer.

This change shouldn't change the overall behavior of the system, except
that it does remove some duplicate regex checks that were occurring when
multiple users may have had the same alert word.
2016-09-23 15:21:17 -07:00
Steve Howell 6b3f945a7e dead code: Remove linebreak(). 2016-09-19 18:25:03 -07:00
Steve Howell c2277afd06 Simplify bulk_get_streams().
We can always expect a realm here.
2016-09-19 18:25:03 -07:00
Steve Howell eb09dd217d Simplify get_stream_backend.
Assume that get_stream_backend() always receives a realm, not a
realm id.  We can be pretty confident of that now due to mypy.
2016-09-19 18:25:03 -07:00
Steve Howell 1dbc94bcd9 annotations: Require a Realm for get_stream(). 2016-09-19 18:25:02 -07:00
Steve Howell e0d59d9386 dead code: Remove Stream.create(). 2016-09-19 18:25:02 -07:00
Tim Abbott 83ef7e72e6 models: Fix missing mypy annotations in subdomains code. 2016-09-10 11:57:07 -07:00
Tim Abbott 2b0a486077 Eliminate cross-realm PM conversation with ADMIN_DOMAIN.
In the future, we can use UserProfile.is_staff to control this, but
there isn't a current use case for it.
2016-08-22 21:46:16 -07:00
Tim Abbott b8050ce02f Add realm.host property. 2016-08-18 18:48:40 -07:00
Tim Abbott 4fbb8c3eee templates: Add new context variables to all templates.
This adds a few new helpful context variables that we can use to
compute URLs in all of our templates:
* external_uri_scheme: http(s)://
* server_uri: The base URL for the server's canonical name
* realm_uri: The base URL for the user's realm

This is preparatory work for making realm_uri != server_uri when we
add support for subdomains.
2016-08-13 16:27:35 -07:00
Rishi Gupta 708b416ca1 Terms of Service: Fix corner cases around new users being created.
This fixes a few bugs in 7910a6e134,
related to automatically created user accounts.
2016-08-13 00:19:54 -07:00
Tim Abbott 386c6c2a31 settings: Refactor SYSTEM_ONLY_REALMS configuration. 2016-08-12 12:12:53 -07:00
Rishi Gupta 7910a6e134 Terms of Service: Add ability to update TOS and have users re-sign.
Most directly useful for the migration to zulipchat.com.

Creates a new field in UserProfile to store the tos_version, as well as two
new settings TOS_VERSION and FIRST_TIME_TOS_TEMPLATE. We check for a version
mismatch between what the user has signed and the current
settings.TOS_VERSION whenever the user hits the home page, and redirect them
if needed.

Note that accounts_accept_terms.html and
zerver.views.accounts_accept_terms were unused before this commit
(they date from c327446537)
2016-08-11 23:37:48 -07:00
Rag Sagar 2fef36f15a Add realm-level default language setting.
Adds a new field default language in the zerver_realm model.
This realm level default language will be used as default language
for newly created users. Realm level default language can be
changed from the administration page.

Fixes #1372.
2016-08-09 17:38:29 -07:00
Rag Sagar 28f0079432 Use a constant to set UserProfile default_language field max_length attribute. 2016-08-09 16:10:24 -07:00
Umair Khan ef62506fb1 Annotate zerver/models.py. 2016-08-08 15:13:29 -07:00
Tim Abbott df525ad1c5 Remove old MitUser model and related code.
The MitUser model caused a constant series of little problems for
users with mit.edu email addresses trying to sign up for different
Zulip servers.

The new implementation just uses conditionals on the realm object when
selecting the confirmation template to use.
2016-07-26 20:30:12 -07:00
Tim Abbott 375551aaa6 Clean up most hardcoding of mit.edu domain checks.
This moves all this code to be gated on a few virtual realm settings.
2016-07-26 20:30:12 -07:00
Tim Abbott 70543e059a bugdown: Remove hardcoding of mit.edu for zephyr_mirror realm config. 2016-07-26 20:30:12 -07:00
Tim Abbott c17676b00c Cleanup MIT hardcoding for disabling presence. 2016-07-26 20:30:12 -07:00
Steve Howell 57c436b32d Add TODOs for #1379 (bugdown mutual dependency) 2016-07-25 14:59:32 -07:00
Steve Howell 8ac1398b0f Use topic_name() in to_log_dict(). 2016-07-18 14:10:11 -07:00
Rishi Gupta d529a94e4d Add realm setting to time-limit editing of message content.
This is controlled through the admin tab and a new field in the Realms table.
Notes:
* The admin tab setting takes a value in minutes, whereas the backend stores it
  in seconds.
* This setting is unused when allow_message_editing is false.
* There is some generosity in how the limit is enforced. For instance, if the
  user sees the hovering edit button, we ensure they have at least 5 seconds to
  click it, and if the user gets to the message edit form, we ensure they have
  at least 10 seconds to make the edit, by relaxing the limit.
* This commit also includes a countdown timer in the message edit form.

Resolves #903.
2016-07-15 13:55:49 -07:00
Steve Howell 9d7716f368 Add message.topic_name() helper function.
This little helper will facilitate adding a new Topic
table in the future.
2016-07-14 09:54:56 -07:00
Eklavya Sharma 958335bdb3 Change all default values in models to unicode. 2016-07-11 21:30:32 -07:00
Tomasz Kolek d7c7279523 Add is_status_message method to Message model. 2016-07-11 12:09:06 -07:00
Rishi Gupta 43c2f35776 Add realm setting to disable message editing.
This is controlled through the admin tab and a new field in the Realms
table.  This mirrors the behavior of the old hardcoded setting
feature_flags.disable_message_editing.  Partially resolves #903.
2016-07-10 11:57:24 -07:00
Rishi Gupta 07e7230ae1 models.py: Add post_save to Message.
Flushes message from cache after a message.save(). Needed for tests
where we directly manipulate Message objects in the database.
2016-07-10 11:42:18 -07:00
Tim Abbott 1e2d38e790 Move stringify_message_dict into to_dict_uncached. 2016-07-08 17:58:56 -07:00
Umair Khan 035fceb814 Add dynamically loaded language dropdown. 2016-07-04 11:56:02 -07:00
Eklavya Sharma 6d8ba90db9 zerver/models.py: Modify imports for mypy in py3 mode.
Change 'from zerver.lib import bugdown' to
'import zerver.lib.bugdown as bugdown' to make zerver/models.py
pass mypy check in python 3 mode.
2016-07-02 10:38:48 -07:00
Kartik Maji f8bb7503e6 Add ability to pin streams to top of the streams sidebar list.
Based on work by Lauren Long, with some tweaks by tabbott.
2016-06-30 22:26:09 -07:00
acrefoot acd7ff7aff Fix typo in Recipient comment. 2016-06-27 18:34:02 -07:00
Tim Abbott b2a24e0306 Revert "Add authorization check before serving files."
This reverts commit e985b57259.

This commit will break production when we next do a release, because
we haven't done a migration to create Attachment objects for
previously uploaded files.
2016-06-27 12:09:56 -07:00
rahuldeve e985b57259 Add authorization check before serving files. 2016-06-27 11:24:35 -07:00
rahuldeve 8cecb37743 Modify Attachment model to track file access permissions. 2016-06-23 17:46:16 -07:00
Vishnu Ks ad1c3894d9 Add interface for creating new realms.
This is controlled by settings.OPEN_REALM_CREATION; if that setting is
off, this feature doesn't do anything.
2016-06-17 16:15:28 -07:00
Eklavya Sharma 7ec9cb7e93 Annotate model fields: Attachment.
Also fix clashing annotations.
2016-06-13 20:01:03 +05:30
Eklavya Sharma 86978cb2a3 Annotate model fields: Stream. 2016-06-13 20:01:03 +05:30
Eklavya Sharma 1ec7e124c7 Annotate model fields: Message. 2016-06-13 20:01:02 +05:30
Eklavya Sharma f27cff57c3 Annotate model fields: UserProfile. 2016-06-13 20:01:02 +05:30
Eklavya Sharma 9f39c9276f Annotate model fields: DefaultStream, Referral, ScheduledJob. 2016-06-13 19:37:07 +05:30
Eklavya Sharma 4f890cca2a Annotate model fields: UserActivity, UserActivityInterval, UserPresence. 2016-06-13 19:37:07 +05:30
Eklavya Sharma 7290f9cb83 Annotate model fields: Subscription and Huddle. 2016-06-13 19:37:07 +05:30
Eklavya Sharma ddaaa98b25 Annotate model fields: UserMessage. 2016-06-13 19:30:58 +05:30
Eklavya Sharma acd1767398 Annotate model fields: Recipient and Client. 2016-06-13 19:30:57 +05:30
Eklavya Sharma c0004a5874 Annotate model fields: PushDeviceToken and MitUser. 2016-06-13 19:30:57 +05:30
Eklavya Sharma 04740fb620 Annotate model fields: PreregistrationUser. 2016-06-13 19:30:56 +05:30
Eklavya Sharma d666e00833 Annotate model fields: RealmEmoji and RealmFilter. 2016-06-13 19:30:56 +05:30
Eklavya Sharma 6c7dd07ec2 Annotate model fields: Realm and RealmAlias. 2016-06-13 19:30:56 +05:30
Eklavya Sharma c654c4032d zerver/models.py: Annotate get_display_recipient.
get_display_recipient's annotation clashes with other wrong annotations.
Fix those wrong annotations.
Since get_display_recipient returns a Union, use isinstance checks and
casts to make mypy checks succeed.
2016-06-12 23:34:57 +05:30
Eklavya Sharma ed61c4c581 Improve model string representation.
Define __str__, __repr__ and __unicode__ correctly on models.
This will help in python 3 compatibility.
2016-06-12 09:55:12 -07:00
Eklavya Sharma 17b9422546 zerver/models.py: Fix regex strings.
Some regex strings were not declared as raw, even though they used
a lot of backslashes.
2016-06-12 09:31:19 -07:00
Eklavya Sharma 81759d56be zerver/models.py: Fix string annotations in UserPresence. 2016-06-12 09:30:53 -07:00
Eklavya Sharma e11bec28c2 zerver/models.py: Fix annotations (str -> text_type).
Change str to text_type where appropriate in annotations related to
Attachment, PreregistrationUser and Huddle.
2016-06-12 09:30:53 -07:00
Eklavya Sharma d76bea8f25 zerver/models.py: Identify functions returning QuerySets.
Identify functions which return QuerySets and give them a return type
`Sequence` with appropriate parameter.  Typing them as QuerySet will
not be useful since generic stubs for QuerySets are not available and
not knowing the type of QuerySets is hardly useful for type checking.
2016-06-12 09:30:53 -07:00
Eklavya Sharma 39060aa221 zerver/models.py: Fix annotations related to Message. 2016-06-12 09:30:53 -07:00
Eklavya Sharma 2841aa642d Fix annotations related to make_safe_digest and hashes. 2016-06-12 09:30:53 -07:00
Eklavya Sharma 9020177418 zerver/models.py: Add annotations related to Stream and Client. 2016-06-12 09:30:53 -07:00
Eklavya Sharma 10f2ec043d Fix zerver.lib.utils.generate_random_token.
generate_random_token used to return a value of type six.binary_type
and its return type was annotated as `str`.  This commit fixes that
by making it return a value of type `six.text_type` and updating
the annotation accordingly.
Also fix clashing annnotations.
2016-06-12 09:30:53 -07:00
Eklavya Sharma 56d5785c2e zerver/models.py: Change some constants to unicode literals.
Change choices of UserProfile.avatar_sources and UserProfile.tutorial_status
from str literals to unicode literals.  This is done because these fields
are CharFields, which are of type `six.text_type`.  So the set of values
which they can take should also be of the type `six.text_type`.
Also fix clashing annotations.
2016-06-12 09:30:53 -07:00
Eklavya Sharma 0cc7a6583c zerver/models.py: Fix last_reminder_tzaware annotation.
Fix annotated return type of UserProfile.last_reminder_tzaware from
`str` to `Optional[datetime.datetime]`.
2016-06-12 09:30:33 -07:00
Eklavya Sharma 8acc51218e Fix realm emoji and realm filter annotations.
Change `str` to `text_type` in annotations in zerver/models.py
related to realm emoji and realm filters.
Also fix clashing annotations in zerver/lib/bugdown/__init__.py.
2016-06-12 09:25:42 -07:00
Eklavya Sharma e68d99eb2e zerver/models.py: Make cache_keys text_type. 2016-06-12 09:25:32 -07:00
Tim Abbott f44b227b85 Refactor getting cross-realm users into a function. 2016-06-11 11:22:22 -07:00
Eklavya Sharma 53084fe03c Use text_type as type of cache keys and update users.
This changes the type annotations for the cache keys in Zulip to be
consistently text_type, and updates the annotations for values that
are used as cache keys across the codebase.
2016-06-11 09:10:34 -07:00
Eklavya Sharma d3b80d94a2 Use appropriate string types and correctly encode/decode them. 2016-06-11 17:34:23 +05:30
Tomasz Kolek 999093b227 Add new is_incoming_webhook bot type.
This type of bot is only able to send messages via webhook endpoints.
2016-06-07 21:23:35 -07:00
Tim Abbott 9c5f15e89b models: Fix use of non-lazy ugettext at import time.
Was introduced in 03debdf82f.
2016-06-04 17:46:03 -07:00
Tim Abbott 6f69053911 Annotate most of the rest of bugdown. 2016-06-04 11:35:29 -07:00
Tim Abbott ac3989c114 models: Add most missing type annotations. 2016-06-04 00:03:54 -07:00
Tim Abbott 03debdf82f Fix malformed error message when creating invalid Realm Emoji.
Thanks to Greg McCoy for his help finding this bug.
2016-06-03 23:12:36 -07:00
Max 0f4673ae3b Add type annotation to mention.py, redis_utils.py, timestamp.py, user_agent.py
Some functions in models.py had input typed as int when they needed to be typed as datetime.datetime
2016-06-03 19:00:16 -07:00
Max 1148f6ff8a Rename timestamp kwarg in to_presence_dict to dt
It is not a timestamp, it is a datetime object. This is better ducktyping
2016-06-03 19:00:16 -07:00
Ashish Kumar 31bf6b8259 Type annotation of zerver/models.py
[Substantially revised by tabbott]

This probably still has some bugs in it, but having mostly complete
annotations for models.py will help a lot for the annotations folks
are adding to other files.
2016-06-02 23:28:34 -07:00
Vishnu Ks 100d885f23 Change default announcement stream to announce.
Fixes #788.
2016-05-31 07:38:07 -07:00
Tomasz Kolek 8c18b8947f Add bot_type field to UserProfile.
This is intended to support creating different types of bots with
potentially limited permissions.
2016-05-19 22:37:37 -07:00
Tim Abbott b01196db86 to_log_dict: Add sender_id to logged fields. 2016-05-18 23:02:43 -07:00
Aristeidis Fkiaras 3ee210d9e8 Add setting to only allow admins create new streams.
Fixes: #691.

Thanks to Preston Hansen for work on this feature!
2016-05-18 18:53:13 -07:00
Tim Abbott 92bec8cfea Merge Zulip 1.3.12 security release. 2016-05-10 11:32:26 -07:00
Tim Abbott 07fc47f953 CVE-2016-4426: Fix non-admin users having access to all bot API keys.
Long ago, there was work on an experimental integration model where
every user in a realm would have administrative control over all bots,
with the goal of simplifying the process of setting up communally
administered bots for smaller teams.  While that new model was never
fully implemented (and thus never setup as an option), an error in
that original implementation meant that the data on all bots in a
realm, including their API keys, was sent to the browsers of users via
the `realm_bots` variable in `page_params`.  The data wasn't displayed
in the UI for non-admin users, but was available via e.g. the
javascript console.

This commit updates this behavior to only send sensitive bot data like
API keys to the owner of the bot (and realm admins).

We may in the future implement a model simplifying communally
administered integrations, but if we do that, those bots should be
limited in their capabilities (e.g. only able to send webhook
messages).

This bug has been present since Zulip was released as open source.
2016-05-10 09:50:02 -07:00
Tim Abbott 2a2cbd60c3 cache: Fix fragile active_bot_dicts_in_realm caching model.
The issue here is similar to that in the previous commit.
2016-05-09 10:12:35 -07:00
Tim Abbott fbc7e977ac cache: Fix fragile active_user_dicts_in_realm caching model.
Previously we relied on having two matching list of fields for the
get_active_user_dicts_in_realm, one in the actual code and the other
in the caching system.  By unifying these lists to have a single
source, we eliminate a class of caching bugs we might otherwise
regularly introduce.
2016-05-09 10:12:35 -07:00
rahuldeve dde832b158 Add Attachment model to keep track of uploads.
This commit adds the capability to keep track and remove uploaded
files.  Unclaimed attachments are files that have been uploaded to the
server but are not referred in any messages.  A management command to
remove old unclaimed files after a week is also included.

Tests for getting the file referred in messages are also included.
2016-05-02 22:14:47 -07:00
Tim Abbott d9e4968d6f Increase maximum URL length for RealmEmoji to 1000.
The default of 200 was shorter than the Camo URLs use by Zulip.
2016-05-02 19:02:56 -07:00
Tim Abbott 5bd94c15c7 Use camo to avoid mixed content warnings when displaying emoji. 2016-05-02 17:21:31 -07:00
Vladislav Manchev f5e6176aea Add custom realm emoji UI to administration page. 2016-04-26 13:15:54 -07:00
Tim Abbott 79297898f1 Remove obsolete AppleDeviceToken model. 2016-04-20 21:51:52 -07:00
Tim Abbott 49799440a4 Replace use of django-guardian with fields on UserProfile.
As documented in https://github.com/zulip/zulip/issues/441, Guardian
has quite poor performance, and in fact almost 50% of the time spent
running the Zulip backend test suite on my laptop was inside Guardian.

As part of this migration, we also clean up the old API_SUPER_USERS
variable used to mark EMAIL_GATEWAY_BOT as an API super user; now that
permission is managed entirely via the database.

When rebasing past this commit, developers will need to do a
`manage.py migrate` in order to apply the migration changes before the
server will run again.

We can't yet remove Guardian from INSTALLED_APPS, requirements.txt,
etc. in this release, because otherwise the reverse migration won't
work.

Fixes #441.
2016-04-20 21:51:52 -07:00
Tim Abbott 28d1a3105c models: Add a __repr__ for Client. 2016-04-20 15:26:51 -07:00
Tim Abbott b61d73fc93 Delete unused old StreamColor model. 2016-04-08 13:06:04 -07:00
Varshit 4e1060076d Purge 'from typing import *' from zerver/.
This is a partial implementation of #636.
2016-04-07 14:07:07 -07:00
Tim Abbott d8f7d89fb4 Add PEP-484 type annotations to zerver/models.py.
Done pair-programming with Guido.
2016-04-03 15:40:06 -07:00
Tim Abbott b99313545e Add PEP-484 type annotations to global dictionaties. 2016-04-03 15:40:06 -07:00
Tim Abbott a407f090e1 models: Document some of the more interesting model classes. 2016-04-01 09:34:45 -07:00
Ryan Moore 15cc3fde7b s/memcache/remote_cache/ in models 2016-03-31 12:54:29 -07:00
Tim Abbott df0d2a726d python3: Add missing utf-8 encoding/decoding in various places. 2016-03-08 09:14:15 -08:00
Reid Barton 9735025167 Refactor logic around restricted_to_domain.
Add a function email_allowed_for_realm that checks whether a user with
given email is allowed to join a given realm (either because the email
has the right domain, or because the realm is open), and use it
whenever deciding whether to allow adding a user to a realm.

This commit is not intended to change any behavior, except in one case
where the Zulip realm's domain was not being converted to lowercase.
2016-02-07 10:54:52 -05:00
Tim Abbott 05a827c520 need_to_render_content: Fix comparison with None.
If the content wasn't rendered, both rendered_content and
rendered_content_version would be None.  In addition to being
confusing, in Python 3, `None < 2` is an error and this code breaks.
2016-01-26 21:11:25 -08:00
Tim Abbott 2ea0fce47e Add UserProfile indexes on is_active and is_bot.
Since we frequently do filters for only active users, these indexes
may help performance in some cases.
2016-01-09 20:01:37 -08:00
Tim Abbott 74853709a8 Rename NOTIFICATION_STREAM_NAME to clarify it's a default. 2015-12-05 14:14:44 -08:00
Tim Abbott f6f8f1fe36 Use new-style classes consistently for Python 3 support.
Also add the fixer for this to our list of fixers we check.
2015-11-04 08:01:52 -08:00
Nicholas Bergson-Shilcock 89a2765553 Turn off desktop notifications by default for new users.
New users will no longer get desktop and audible notifications for all streams
by default.

This also updates the `day1` follow-up email to let users know they can
customize how and when Zulip notifies them of new messages.

Lastly, this adds a `changelog.md` file, following the conventions from
keepachangelog.com, to track changes for new releases.
2015-10-15 12:25:32 -04:00
Tim Abbott 32aea4c9dd Fix get_unique_open_realm always returning None in production.
Fixes #186.
2015-10-15 10:21:04 -04:00
Tim Abbott b68d116e3a Remove old is_super_user helper functions.
(imported from commit 85655ea9369f5dc309f6f687bac578924571c9ee)
2015-09-20 11:13:20 -07:00
Tim Abbott 858d0a984b Move API super users configuration into the database.
(imported from commit 3cc702f93e7252b42930dba4bde93a915b6dbf44)
2015-09-20 11:13:20 -07:00
David Roe 3ac95ddc1a Add UserProfile flag to control whether we have a left side userlist.
Previously this was hardcoded for a single customer.

(imported from commit a6b7095050aa10cef976541505d9b09a35453f48)
2015-09-19 23:22:59 -07:00
David Roe e3f38acbce Enterprise => Voyager.
(imported from commit 41b9a67301aeaf5fd40bbbb8f34a326ca98431fd)
2015-08-21 10:33:35 -07:00
Reid Barton ecc07333af Django 1.8 compatibility: miscellaneous changes
(imported from commit 086fac44cbdce1986a07e2ade00412dd538b4f23)
2015-08-20 23:15:45 -07:00
Reid Barton 0aab583bb1 Django 1.8 compatibility: transactions
- autocommit is now the default
- commit_on_success is now atomic

(imported from commit b6166ca666ff23d34f2871ca365ce3ec66b0fa86)
2015-08-20 23:15:45 -07:00
Reid Barton 599742536b Switch from South to native Django migrations
This commit loses some indexes, unique constraints etc. that were
manually added by the old migrations. I plan to add them to a new
migration in a subsequent commit.

(imported from commit 4bcbf06080a7ad94788ac368385eac34b54623ce)
2015-08-20 23:01:26 -07:00
David Roe 3515a69e43 Remove public_streams_disabled.
(imported from commit 1d0c8db92bf8e4c44bd1c96063acfd6eea74925c)
2015-08-20 18:12:53 -07:00
Kate Buckner 4d0f7c7ea4 Add a user-visible setting for 24-hour time display.
(imported from commit d934824fd6b72e64a455aac9ff4585b262145f02)
2015-08-20 17:33:16 -07:00
David Roe def426aa34 Remove exceptions for two domains where all streams were private.
(imported from commit 1e391a50875091f6a8843d3915a6893f2d903f22)
2015-08-20 17:33:16 -07:00
David Roe 086d8eee22 Add name_changes_disabled realm field.
(imported from commit 6b04ba2f7ad64c44f2ef18302f0fbd819259d632)
2015-08-20 15:29:46 -07:00
David Roe 5b7f3466ba Add feature where only admins can invite new users.
This is controlled through the admin tab and a new field in the Realms table.

(imported from commit e78a6f48160e2a1bbc68d278beb726fe31515266)
2015-08-20 15:29:46 -07:00
David Roe 809efc4f2b Add realm option to show or not show digest e-mail.
(imported from commit e9cfe519aa9f2857a1dcc7c75f5e1889834fcb86)
2015-08-20 15:29:45 -07:00
David Roe 90e2f5053f Add mandatory topics as a realm option.
(imported from commit 929a884b8610669aa24a167367b899683e33a045)
2015-08-20 15:29:45 -07:00
David Roe 472898cfc6 Allow adding users to realms more easily in Dev VM.
Include new field on Realm to control whether e-mail invitations are required
separately from whether the e-mail domain must match.
Allow control of these fields from admin panel.
Update logic in registration page to use these fields.

(imported from commit edc7f0a4c43b57361d9349e258ad4f217b426f88)
2015-08-19 22:24:53 -07:00
Jason Michalski 802a3dd357 Enable missed message notifications for zulip.com
Also increase the number of messages sent as context from 5 to 10 and
look up to 15 minutes in to the past for context.

(imported from commit bfaed9bcff1ee2047fc3b7a63acf93cd2d47cc7d)
2015-02-20 21:13:36 -05:00
Leo Franchi 1c22b48bb1 Add an optional ios_app_id field to PushDeviceToken
(imported from commit 32d6efd1ecc58297b83e515350e6daac3bce8469)
2015-02-11 06:57:22 +00:00
Luke Faraone f2e7f4cb67 Revert "Exclude dormant users from the buddy list."
Resolves a AmbiguousTimeError.

Approved by Leo.

This reverts commit ebfaeb97ffda22b618be7a9206877f9d2ec53404.

(imported from commit 42b29c6c57eb954952a740bc89611031cef1834a)
2014-12-16 22:56:45 -08:00
Steve Howell 187469ea86 Exclude dormant users from the buddy list.
You must have accessed Zulip in the last 45 days to show up
on the list.

(imported from commit ebfaeb97ffda22b618be7a9206877f9d2ec53404)
2014-10-06 15:27:56 -07:00
Jason Michalski 6b13f4a3c9 Update message events to include is_mirror_dummy
This can be used by mirroring scripts to only forward messages to users
who do not zulip accounts.

(imported from commit 200d6bcaaf39238bfb01480a9e906d567d4d9e11)
2014-07-29 21:46:34 -07:00
Jason Michalski 710a802f49 Add context to at mention missed messages
When you are at mentioned in a stream we will now send you up to the
last five messages which were sent in the past 5 minutes on the same
topic and stream.

(imported from commit 6df6c1cf868722a7bf76e54710e38741a7ac8f31)
2014-07-15 20:21:28 -07:00
Steve Howell 72964a2194 Recognize humbug-user-uploads-test for images/attachements.
(imported from commit 8596a5c4bad6e1adbfe97cbe057d1bd6d8e7f530)
2014-03-14 20:48:45 -04:00
Steve Howell 105c742e92 Correctly identify images/attachments on humbug-user-uploads.
(imported from commit 081d00d99672e6b938c60d9e13665e21d935d8ef)
2014-03-11 13:06:18 -04:00
Jason Michalski 3f6e53db6e Add bot_data module that updated with events
(imported from commit b0bd714258132fc81db763d316a15f5a81b1f4ff)
2014-03-05 14:16:20 -05:00
Jason Michalski c17ed8dc8c Add bots to page_params and send events on bot creation
(imported from commit ce418b4f056576d57f82d26af621473c730c12d8)
2014-03-05 14:16:19 -05:00
Jason Michalski f3180b774b [schema] Add default user_profile options for register events
Allow bot owners to set which streams their will receive events for
without needing to change a configuration file.

(imported from commit 2b69e519dbc12ffbdba072031a7f7196c9e50e33)
2014-03-05 14:16:18 -05:00
Jason Michalski de545d5fa0 [schema] Add a default to stream option to user profile
This allows bot owners to configure which streams messages are delivered
to without needing to change webhook URLs or configuration files.

(imported from commit 32a0c26657c145b001cd8cb3ce0a0364d48902ce)
2014-03-05 14:16:18 -05:00
Steve Howell eec12ff268 Calculate has_* fields before saving Messages
Before saving a Message object, call update_calculated_fields()
to set the has_attachment/has_image/has_link fields.

Note that the pre_save hook we added here does not get called
if you call bulk_create, hence the explicit call to
update_calculated_fields() in do_send_messages().

(imported from commit 1d60ae5908ef186aa5ff1e39277dbb2b765e60d4)
2014-03-04 11:00:26 -05:00
Zev Benjamin 814aed7cbe Send an event when a stream is created, is deleted, becomes occupied, or becomes vacant
A stream is vacant when it has no subscribers and occupied when it has at least
one subscriber.

We have a slightly odd model where stream creation is conflated with
subscription creation.  Streams are created by attempting to subscribe to a
stream that doesn't exist.  We also hide streams with no subscribers from users
to make it seem like they've gone away.  However, we can't actually remove those
streams because we want to preserve history.

This commit moves us towards a separation of these two concepts.  By sending
events for stream creation, occupation, vacancy, and deletion, we allow clients
to directly observe the global state of streams rather than indirectly observing
subscription information.  A more complete solution would involve adding a view
for explicitly creating streams without subscribing to them.

This commit does not handle the intricacies of invite-only streams.  We
currently simply do not send these events for invite-only streams.

(imported from commit 5430e5a5eecefafcdba4f5d4f9aa665556fcc559)
2014-03-03 17:30:58 -05:00
Jessica McKellar 600786983c [schema] Split Subscription notifications field into audible + desktop.
(imported from commit e160b9eb958fa5a06f990fccdeb25fa39e13e753)
2014-03-03 16:08:31 -05:00
Jessica McKellar df39a7bde2 [schema] Add separate notification settings for streams.
(imported from commit 2122a4cc35e9a1f019eea004cd27ce703ba5a55a)
2014-03-03 16:08:30 -05:00
Steve Howell 0f96f5064e [schema] Add has_attachment/has_image/has_link to Message.
We will need to run these commands manually when deploying to staging:

    CREATE INDEX CONCURRENTLY "zerver_message_has_attachment" ON "zerver_message" ("has_attachment");
    CREATE INDEX CONCURRENTLY "zerver_message_has_image" ON "zerver_message" ("has_image");
    CREATE INDEX CONCURRENTLY "zerver_message_has_link" ON "zerver_message" ("has_link");

(imported from commit 84808dc6b1af887ddf784cb8a875ae462f4df985)
2014-03-01 12:41:00 -05:00
Steve Howell a27610f560 Make Realm.get_admin_users not do O(N) queries.
This commit makes it so that we don't individually query
auth_permission for every user on the realm during calls
to Realm.get_admin_users().  This should speed up page loads.

To apply this commit, we had to upgrade all of our servers
with this patch to django-guardian:

 https://github.com/lukaszb/django-guardian/pull/178#issuecomment-31049062

(imported from commit a1604bf573a5005c9abc128a680a7da6a20cabef)
2014-02-28 16:27:12 -05:00
Luke Faraone 6a803a88c8 Return whether the user can be mobile-pushed in the presence endpoint
This allows clients to show an indication if users are on mobile or can
be reached via mobile.

(imported from commit 819c27a141f71dee32a97fd5c1ac405b039a5038)
2014-02-27 20:25:39 -05:00
Luke Faraone f30a62e33c Use keyword arguments in UserPresence.to_presence_dict()
Also rename the function, which was previously called to_presense_dict.

This will let us be more flexible about adding additional values.

(imported from commit 323b7d2df53918928190c9ee9544b4599a5e2df3)
2014-02-27 20:25:38 -05:00
Jessica McKellar 73fc65eb99 Make enter sends the default.
(imported from commit aa256772b381466f02210881214c9882b78b044d)
2014-02-05 13:35:41 -05:00
Jessica McKellar eeb8464f4d Don't expose deactivated streams to users through clients or API.
(imported from commit c32715255b3286f52fb313d35659f9357082603a)
2014-01-29 12:41:21 -05:00
Jessica McKellar 99d58fe8b5 [schema] at a `deactivated` field to Stream.
From a user's perpsective, the stream has been deleted. From the
database's perspective, the stream has been deactivated -- the stream
messages still exist.

(imported from commit b08b30b2a822663e17d64182af1fb160c2193344)
2014-01-29 12:41:20 -05:00
Steve Howell 34f6c705e9 Rename update_realm_filter() to flush_realm_filter().
(imported from commit f52b45b627f73ece8c7348451cea48f32142e94e)
2014-01-28 17:40:01 -05:00
Steve Howell 75555ff16c Rename update_realm_emoji_cache() to flush_realm_emoji().
(imported from commit fc97377113a3c2701addcc6fb0c5d1bc6f52933d)
2014-01-28 17:40:01 -05:00
Steve Howell 912f96e7ec Rename update_stream_cache() to flush_stream().
(imported from commit c67844ef7feb1924ec495c4451e33b56b237c88a)
2014-01-28 17:40:01 -05:00
Steve Howell c5edb58cd4 Turn flush_realm() into a post-save hook for Realm.
(imported from commit 97e44b51e2b35b4b5b64b4008f6a8d3585a26e33)
2014-01-28 17:40:01 -05:00
Steve Howell 0eed5d20d2 Create cache.flush_realm() and call it from do_deactivate_realm().
This is a slight behavior change, as we now flush user_profile
caches for bots as well as humans.

(imported from commit 24c72c44d851ee4c66a67a4728cd6c548faeedcd)
2014-01-28 17:40:00 -05:00
Steve Howell 3776d11c3a Rename update_user_profile_cache() to flush_user_profile().
(imported from commit 1fddb8b8dfe7943ace70c51d762eeae2ee1203a0)
2014-01-28 17:40:00 -05:00
Leo Franchi 883157893c Use a UserMessage flag to indicate /me messages
(imported from commit ea503b0d3eb6e90230c0859be96ede60faf5c2bd)
2014-01-27 12:07:48 -05:00
Tim Abbott da90d63046 Split out zerver/lib/notifications.py from actions.py.
(imported from commit 784b82834ee4fcb4431e77f8fb1c526f8eec82ad)
2014-01-24 17:33:56 -05:00
Jason Michalski 4104f00229 [schema] Add description to streams and display it
A description was added to the streams and it is now displayed on the
subscriptions page. It can not be set in the UI yet.

(imported from commit 81d08b65eee42dba87cd99dd5bd30106c4eb6c6a)
2014-01-24 14:47:44 -05:00
Jason Michalski 4c90392261 [schema] Add an per-user option to disable notification on new streams
Added a default_desktop_notifications boolean to userprofile with a UI
in Zulip Labs. This flag is used to default the notification flag on new
subscriptions.

(imported from commit a25223cc5ecf09980cf877991e25034bb3fd4046)
2014-01-16 20:13:34 -05:00
Steve Howell 8bf9068d0b Add back end support to restrict creation of streams.
Note that this doesn't actually restrict anybody yet, but it
makes it so that UserProfile.can_create_streams must return True
for a user to create a stream.  We can modify that in the future
to have special behavior for realms that want more restrictions.

(imported from commit 432e85b1ca86aaee4a6bd1d4a6d0b2c78ecb0863)
2014-01-16 14:43:56 -05:00
Jessica McKellar a1d128b6cb Flush memcached on a stream privacy change.
Otherwise uninvited users can join a stream recently turned
invite-only.

(imported from commit ab43bf3b856a0e3b71e6e06cee1519214b7763cd)
2014-01-15 17:13:56 -05:00
Zev Benjamin 3720cb1c33 Move message-related views into their own file
(imported from commit 2c71cc1b5d3f0210c98767888c461d5c6d046b49)
2014-01-10 21:38:59 -05:00
Tim Abbott 47500d8352 [schema] Add a bit for whether inactive users are mirror dummies.
(imported from commit bb21bb2c62ac09742484d7a4ca8907e7d864b982)
2014-01-09 11:08:35 -05:00
Jessica McKellar 8e3911269e Strip whitespace from emails before looking up DB entries in a few places.
(imported from commit eafdce98450d16c0ca51c4cb17a139811a6124bc)
2014-01-07 20:24:21 -05:00
Jessica McKellar a06b5bd720 [schema] Add a deactivated field to Realm.
(imported from commit 8b873296d45dbacc0865c8d101a872be0b65ed46)
2014-01-07 20:24:20 -05:00
Jessica McKellar 8091634814 Move sent_by_human check to a method on the Message class.
(imported from commit 2ef49051932f9633ad11130952591c5c5b147f12)
2013-12-31 16:42:38 -05:00
Tim Abbott 014b5e8a91 Add a per-request realm filters cache.
This avoids doing hundreds of memcached queries in get_old_messages.

(imported from commit 8b64cac3e4e91d46762a13d6812bd3c6f517e9ef)
2013-12-19 14:04:55 -05:00
Tim Abbott c91415f318 Improve names for per-request display recipient cache.
It incorrectly advertises itself as per-process.

(imported from commit faf7ca7374d020058e80249bb16a4c6afbcb3e44)
2013-12-19 14:04:55 -05:00
acrefoot 0fd492a817 [schema] Add enable_autoscroll_past_unread setting
This is for the CUSTOMER28 folks, so that they can turn Zulip into a more "chat client" thing.

(imported from commit 373a8afae4998fce5560e7b2bd13804c8fbb39fc)
2013-12-13 11:50:03 -05:00
Kevin Mehall e4589700b6 [schema] Modify device token to support both iOS and Android
This replaces the AppleDeviceToken table with a generic
PushDeviceToken with a `kind` field to make it easier to add functionality
like per-device/per-stream settings that share code between Android and
iOS devices.

The schema must continue to work on prod with the old table name, so we
add the new table in parallel and can drop the old table once this code
hits prod and any necessary data is copied.

(imported from commit 0209a7013f2850ac6311f23c3d6f92c65ffd19e3)
2013-12-11 15:37:47 -05:00
Leo Franchi 2c4efecee2 [schema] Add data model for managing realm filters via the database.
(imported from commit c624cdda9c23d9ca42706bf64b4c8a4e69e95c50)
2013-12-11 14:39:09 -05:00
Tim Abbott 1fb9c09f02 Don't import bugdown in models.py headings.
This allows us to avoid a circular import when importing models.py
from inside bugdown for the realm-filters-from-database branch.

(imported from commit 7de85b54243132ade6818b080abdc8c5e8ad84f5)
2013-12-11 14:39:09 -05:00
Tim Abbott 59e16feae1 Check whether realm exists in completely_open check.
(imported from commit 5294c4c7cdb6c03feb6bf716a3fc761dc7af1500)
2013-12-09 15:26:15 -05:00
Tim Abbott e56e892578 Enable public streams for most legacy realms.
(imported from commit 6844df2e14c624239d99a1d3fc25e55206c8c264)
2013-12-06 16:26:10 -05:00
Jessica McKellar 1b068d5433 [schema] add boolean field enable_digest_emails to UserProfile.
We'll use this to track the user preference for receiving digest
e-mails.

(imported from commit a56644b1a359444ded45f9437c954194b9ae47a1)
2013-12-02 18:58:03 -05:00
Leo Franchi 501f063426 Replace email_to_domain with split_email_to_domain and resolve_email_to_domain
Now that we support email aliases, we have to be careful when going from
an email address to a domain that we assume we can use to get a Realm
object for. When we care about the Realm's domain, we want to follow
any RealmAliases that exist for a certain domain.

When we just care about the original email address domain itself,
for comparison or other purposes, use split_email_from_domain

This removes the ambiguity of having to decide when to use
email_to_domain + RealmAlias or just email_to_domain

(imported from commit 0e199495502d946ce2e1aae56263e7e8665be4ed)
2013-11-26 10:45:01 -05:00
Zev Benjamin d2d695dcb6 Add an in-memory Client object cache
Client objects are immutable and there are very few of them, so caching them in
memory is cheap and saves a trip to memcached.

(imported from commit 300b9b402f4e509f86a7fd86b5f898dc3f43738f)
2013-11-20 16:55:10 -05:00
Jessica McKellar d1619b3f3c Use a more accurate "is admin?" check.
UserProfile.show_admin was intended to be a check for users that have
administrative rights in other realms, which we've harmlessly but
erroneously been using to check if they are an admin in their realm.

Use the more straightforward check instead, with a more intuitive
name.

(imported from commit d81050c7dbbb19e59c5e31750be303a4630e1456)
2013-11-19 14:31:02 -05:00
Steve Howell 220678ae40 Support emojis/bolding/etc. in single-line /me messages.
(imported from commit c62ae8cf26117e30a029e025021cc4a3a67f4a83)
2013-11-15 14:34:38 -05:00
Steve Howell ee28e5e2f6 Relax 80-char limit on status messages (/me).
(imported from commit e9de2cb6c394f5b764e988e279f0fb3cfd38c6e8)
2013-11-14 13:51:07 -08:00
acrefoot f7b5a10da0 [schema] Add ScheduledJob table, and update mandrill related code
ScheduledJobs with type Email displace the usual mandrill codepaths
in the Zulip Enterprise deploys

* Email-specific helper functions will appear in deliver_email.py
* 0058_auto__add_scheduledjob.py

(imported from commit 8db08d8a279600322acfdbed792dc1a676f7a0ab)
2013-11-13 16:41:36 -05:00
Leo Franchi 62ba16a829 [schema] Allow a realm to have aliases for multiple other domains
(imported from commit 5d0340afd4c77f50db63550a3ec8d81e885c285c)
2013-11-13 14:17:43 -05:00
Tim Abbott e18a08c69e settings: Move hardcoded API super users into local_settings.py.
(imported from commit fea7550a771c837db0fb948238488f778bedf73a)
2013-11-13 12:02:49 -05:00
Tim Abbott 5293cdebe8 Rename LOCAL_SERVER to ENTERPRISE.
(imported from commit 7edf353eefe6c9e7aac74b7bbc37b923cac1b913)
2013-11-12 15:57:01 -05:00
Steve Howell 56c39728ba Support /me syntax.
If a user types "/me runs to the store", we put "runs to the store"
in bold after their name.

(imported from commit fbc11e99244e1c8fa1c03e4753e706957fcd449e)
2013-11-12 10:49:10 -05:00
Luke Faraone 3daca0ca9c Fix rebasing errors that rightfully made the linter sad
(imported from commit fbed798c7d752ea2b058aeebb5ae31a620684a3a)
2013-11-05 17:52:53 -05:00
Luke Faraone c11b65590b SSO / REMOTE_USER support
(imported from commit 4f4fad7af5d3c6099cac95d7708338c182626d72)
2013-11-05 16:14:13 -05:00
Tim Abbott 68dcc760c3 Clean up some unused imports.
(imported from commit 0c5d8e2a55ba1b8909ba807fee3afe863dcdc226)
2013-11-04 11:51:17 -05:00
Tim Abbott 0ae03eb655 Remove transaction management hacks in get_client.
I'm pretty sure this code was a hack to workaround the fact that
without database-level autocommit, .get_or_create is racy.

(imported from commit bfdd476a354aee439feeaa0030ae59cc43c165b9)
2013-11-04 07:31:39 -05:00
Tim Abbott 2de31ee287 [schema] Remove the UserMessage.archived field.
This was a precursor to UserMessage.flags.read that never got used
because we decided to use django-bitfield.

(imported from commit 868754723c07ee9b85ae951aee785e571ccfef97)
2013-11-04 07:31:11 -05:00
Steve Howell 406c4f172a Add realm.get_admin_users() method.
(imported from commit b1cb465f0c81eb9f56ad1486b92cd39a8d001474)
2013-11-02 11:37:27 -04:00
Tim Abbott 1292c5dbfe Add backend support for Jabber mirroring.
(imported from commit 12f5c708a6961aded4f4e166af84e87720be4ddc)
2013-10-29 16:49:42 -04:00
Leo Franchi e1557bef73 Cache realm user basic info dict that is used in bugdown
(imported from commit 688c47daceb73534be90bd98a031c7b4edb5546e)
2013-10-28 12:59:36 -04:00
Waseem Daher 66f48288b4 Rename LOCALSERVER -> LOCAL_SERVER.
(imported from commit b3abdd10d54d2ad7a9c463af9a291d2e2127707f)
2013-10-25 17:37:06 -04:00
Luke Faraone 81d7dd1fda [schema] Support for authenticating Deployments via the API.
Here we introduce a new Django app, zilencer. The intent is to not have
this app enabled on LOCALSERVER instances, and for it to grow to include
all the functionality we want to have in our central server that isn't
relevant for local deployments.

Currently we have to modify functions in zerver/* to match; in the
future, it would be cool to have the relevant shared code broken out
into a separate library.

This commit inclues both the migration to create the models as well as a
data migration that (for non-LOCALSERVER) creates a single default
Deployment for zulip.com.

To apply this migration to your system, run:
   ./manage.py migrate zilencer

(imported from commit 86d5497ac120e03fa7f298a9cc08b192d5939b43)
2013-10-25 14:13:30 -04:00
Leo Franchi 05564ec0e2 [schema] Add a config option for enabling/disabling push notifications
(imported from commit 5649188cd5e0e4edb6e71559ec7d7fc5e92efaf8)
2013-10-24 14:54:30 -04:00
Leo Franchi 1d82704b8d [schema] Add a last_updated field to AppleDeviceToken
(imported from commit 2ea91e462773c50591585a610b0de63625ad2b7e)
2013-10-24 14:54:30 -04:00
Tim Abbott 1b381b343d Allow starring historical messages.
This has a small bug where we don't actually filter the message out of
the home view; fixing that requires adding an index on the "flags"
field of UserMessage.

(imported from commit 492c99d0a8e87b253e577be6564bec12099bd8e9)
2013-10-23 18:51:47 -04:00
Steve Howell 58acef4098 Add get_emails_from_user_ids() action.
(imported from commit 887090de16f685373bd2f3548bd1ec94a7ad4a9e)
2013-10-22 15:51:18 -04:00
Tim Abbott d58b4e92dd Add irc-bot@zulip.com as an API superuser.
(imported from commit 36aa0faf8c3ccd8454f214986f32da4f37236921)
2013-10-21 14:37:36 -04:00
Jessica McKellar 5c5ffd6ea3 [schema] Add a name field to zerver_realm.
(imported from commit 5b5d7a40d5dff11fe9ca6624b3794cb9c8a3520a)
2013-10-17 13:47:51 -04:00
Leo Franchi f7386c3f58 [schema] Add support for keeping track of iOS APNS device tokens
In order to support iOS Push Notifications, we need to keep track
of a device's unique APNS Token. These are delivered to our iOS
code after registering for remote notifications

(imported from commit bbe34483e1380dc20a1c93e3ffa1fcfdb9087e67)
2013-10-16 12:54:28 -04:00
Tim Abbott 32861af323 [schema] Increase maximum stream name length to 60.
(imported from commit 83e9434178909d18481692a9e7fe3b8e1802bd7e)
2013-10-15 09:13:35 -04:00
Luke Faraone 1d9391e867 Initial local server configuration.
(imported from commit ac9b9896b74b78c6ca03af7f411d0788ae402cff)
2013-10-10 14:14:14 -04:00
Jessica McKellar 836f3894fb Add a comment to UserProfile that pointer is a Message.id.
And NOT a UserMessage.id. I kept having to trace the code and convince
myself of this.

(imported from commit 720a982a888dd525e63c37ddfedd28b71492d32e)
2013-10-08 10:46:52 -04:00
Leo Franchi 72c7c5b836 [schema] Give new realms a default notifications stream, and send signups/creations to it
(imported from commit 19ad761f7d9eabb865ac3526bf66dff8b7ed0224)
2013-10-07 15:46:13 -04:00
Steve Howell 53b5768b14 Double speed of message fetching in get_old_messages_backend().
When we don't already have old messages in cache, we need to
fetch data from the database and create dictionaries for the
cache.  This commit makes that process work in 50ms, instead
of 130ms, for the data set in test_bulk_message_fetching(),
which is 602 records.  Before this commit we had about 132
microseconds of unnecessary churn per message, because we
were fetching DB fields we didn't need and incurring the cost
of the Django ORM.  Now we use values() to get only the columns
we need, and we take advantage of previous commits that make
our code less OO and more function-driven, so we can pass the
values directly to build_message_dict() without having to create
objects.

A couple caveats on this commit:

1) I haven't been able to get good measurements on the overall
effect on get_old_messages_backend().  If you kill the cache to
force DB queries, you introduce noise related to sessions and
user profiles.

2) Look at the long comment in this commit related to
re-rendering messages in this codepath.  The problem precedes
this commit.

(imported from commit dcb64aa9416f0e9583355ddd6dc3adfa746b9fc7)
2013-09-30 18:22:07 -04:00
Steve Howell eb299bf5c8 Extract Message.build_message_dict()
(imported from commit 03395ea8fbffd0a52f85127c9a24f0cc3f4a6969)
2013-09-30 18:22:07 -04:00
Steve Howell 2a3747e21f Isolate message side effects in Message.to_dict_uncached().
Only call a function on the message object in the unfortunate
situation that we are rendering new content in to_dict_uncached().
Long term, it would be nice if this function didn't have side
effects, and we had a better strategy for upgrading rendered
content when bugdown versions change.

(imported from commit 2a323f52af37a6d651c171cb8234fbfa3d25d561)
2013-09-30 18:22:07 -04:00
Steve Howell f65462c891 Extract Message.need_to_render_content()
(imported from commit 466fa133ed1e8fb13b7222ba3bdcd5773d63de14)
2013-09-30 18:22:07 -04:00
Steve Howell 95254398f2 Extract get_avatar_url().
This function doesn't require the whole UserProfile object to
create the avatar url, and we call it from Message.to_dict_uncached().

(imported from commit e814caab101c4fedd1ba66df041a3408014e4085)
2013-09-30 18:22:06 -04:00
Steve Howell e4fb66fdf8 Isolate self-dot use in Message.to_dict_uncached().
For a bunch of self-dot references, move them to the top.  (This
is kind of funny out of context, but it sets us up for future
refactorings.)

(imported from commit 4ebc1c44a633d86772df1828c51180707769c3dc)
2013-09-30 18:20:55 -04:00
Steve Howell 005ad9c8e0 Kill off dead code in Message.to_dict_uncached().
If this line of code were ever called, it would crash anyway,
because it would be an unknown type, and Recipient.type_name()
would raise a KeyError.

(imported from commit db38c5f71fb2f0b044a832eb88e53fceb0d8a9cf)
2013-09-30 18:14:12 -04:00
Steve Howell dc077a708f Call get_display_recipient_by_id() in Message.to_dict_uncached().
This starts to make us less dependent on having full objects.

(imported from commit 5fb9504b077b676ecb697a909f4a0d2737f51efb)
2013-09-30 18:13:53 -04:00
Steve Howell a9fd6ddfce Extract get_display_recipient_by_id().
This is a variation of get_display_recipient that takes
values instead of an object, so that it is decoupled from
the Django object system.

(imported from commit 25bed43ecd62f1fe0176d517b7003e7f4c78bc37)
2013-09-30 18:11:29 -04:00
Steve Howell 515130396c Remove TEST_SUITE exception for get_display_recipient().
If it's ok for the tests to use memcached, it should be ok
for them to use the in-process cache too.

(imported from commit be43879c3c48f3780317fd5b4139b44d4a1f0ed3)
2013-09-30 18:08:31 -04:00
Steve Howell 826dffbbb4 Extracted Message.save_rendered_content().
This is a harmless extraction designed to allow subclasses override
the behavior of how rendered content gets saved.

(imported from commit 9df4ed9f86c857897fcb5f2b6781bfc5a0813766)
2013-09-30 18:08:31 -04:00
Steve Howell f36938c2f5 Extract Message.extractor
(imported from commit e6ee015cacdb0f6e1f725b7113f750e8b33d316b)
2013-09-30 18:08:29 -04:00
Jessica McKellar ed0fd51160 [schema] Add a date_created field to zerver_stream.
(imported from commit 1360da91b81fad59d49068fdf05205eef982e64a)
2013-09-27 16:23:56 -04:00
Jessica McKellar f22023d4cb [schema] Add a date_created field to Realm.
(imported from commit 1953f7c1f8af7f81d50d7362c704f6a81637d72e)
2013-09-24 16:19:52 -04:00
Steve Howell 4b75dd0d85 Add Stream.num_subscribers
(imported from commit 9b79ab84516c3634ef6795ef2d90db37553cc0d1)
2013-09-20 10:19:07 -04:00
Leo Franchi d88266b41d Warn bot owners when a bot sends a message to a nonexistent stream
(imported from commit 2499d7bd9d176a7ab43f751eb855813d0a479bd4)
2013-09-18 15:54:13 -04:00
Steve Howell 30b6f54a36 Don't cache UserPresence info.
(imported from commit ff590bdf5d84bb9b3cedc561c2f2fbf7f7592a66)
2013-09-17 14:51:56 -04:00
Steve Howell 1e912c7e22 Exclude bots in the user presence query.
(imported from commit 02668ba5db85e12ac0aefc0fa2480d38ffed4921)
2013-09-16 16:27:16 -04:00
Steve Howell 85f9b6695f Optimize user presence/activity query.
The get_status_dict_by_realm helper gets called whenever our
 realm user_presences cache expires, and it used to query these fields:

    "zerver_userpresence"."id", "zerver_userpresence"."user_profile_id", "zerver_userpresence"."client_id", "zerver_userpresence"."timestamp", "zerver_userpresence"."status", "zerver_userprofile"."id", "zerver_userprofile"."password", "zerver_userprofile"."last_login", "zerver_userprofile"."is_superuser", "zerver_userprofile"."email", "zerver_userprofile"."is_staff", "zerver_userprofile"."is_active", "zerver_userprofile"."is_bot", "zerver_userprofile"."date_joined", "zerver_userprofile"."bot_owner_id", "zerver_userprofile"."full_name", "zerver_userprofile"."short_name", "zerver_userprofile"."pointer", "zerver_userprofile"."last_pointer_updater", "zerver_userprofile"."realm_id", "zerver_userprofile"."api_key", "zerver_userprofile"."enable_desktop_notifications", "zerver_userprofile"."enable_sounds", "zerver_userprofile"."enter_sends", "zerver_userprofile"."enable_offline_email_notifications", "zerver_userprofile"."last_reminder", "zerver_userprofile"."rate_limits", "zerver_userprofile"."avatar_source", "zerver_userprofile"."tutorial_status", "zerver_userprofile"."onboarding_steps", "zerver_userprofile"."invites_granted", "zerver_userprofile"."invites_used", "zerver_userprofile"."alert_words", "zerver_userprofile"."muted_topics", "zerver_client"."id", "zerver_client"."name"

Now it queries just the fields it needs:

  "zerver_client"."name", "zerver_userpresence"."status", "zerver_userpresence"."timestamp", "zerver_userprofile"."email" FROM "zerver_userpresence"

Also, get_status_dict_by_realm is now namespaced under UserPresence as a static method.

(imported from commit be1266844b6bd28b6c615594796713c026a850a1)
2013-09-16 11:29:10 -04:00
Leo Franchi 6e56342cf6 Send missedmessage notifications if user is idle for >1hr
(imported from commit 573f46a77497cb2f73eae3b4a648e466977e6247)
2013-09-13 17:33:34 -04:00
Steve Howell 21456091e4 [schema] Add muted_topics to UserProfile
(imported from commit 5d804259be94ba83dc4f7541e50443dd1a42cc6b)
2013-09-11 16:47:37 -04:00
Tim Abbott 9fcca3df4e [schema] Collect data on when users are active on site.
These engagement data will be useful both for making pretty graphs of
how addicted our users are as well as for allowing us to check whether
a new deployment is actually using the product or not.

This measures "number of minutes during which each user had checked
the app within the previous 15 minutes".  It should correctly not
count server-initiated reloads.

It's possible that we should use something less aggressive than
mousemove; I'm a little torn on that because you really can check the
app for new messages without doing anything active.

This is somewhat tested but there are a few outstanding issues:

* Mobile apps don't report these data.  It should be as easy as having
  them send in update_active_status queries with new_user_input=true.

* The semantics of this should be better documented (e.g. the
  management script should print out the spec above)x.

(imported from commit ec8b2dc96b180e1951df00490707ae916887178e)
2013-09-10 13:25:59 -04:00
acrefoot 592e1ead6b Actually use the realm specified by the email mirror when doing the markdown
(imported from commit 00b909b1d0c70a3523ab53d2f91815ab14ba2a6f)
2013-09-05 18:24:55 -04:00
Kevin Mehall e58e6f1b7d Check admin permissions on the target realm, not on their own realm.
Trac #1772

Also abstracts this into a method.

(imported from commit f435b3379fb5387f8b3fdf7ca728ab760eadbf95)
2013-09-05 15:36:59 -04:00
Leo Franchi b2ddd670e4 [schema] Add backend support for per-user alert words
(imported from commit 7a9c596a010cbedbddf594c5d9c68bb9ed46d122)
2013-09-05 10:18:40 -04:00
Scott Feeney e3247de572 Factor out get_active_user_profiles_by_realm()
(imported from commit 634ebf265b02303a938f7595de3622e25bc22b49)
2013-08-29 10:28:20 -04:00
Jessica McKellar f67f5b7619 Allow superusers to send messages to even private streams.
(imported from commit b7253174fa25d61b5bda056a5e5353e3b8ea918b)
2013-08-28 16:43:40 -04:00
Jessica McKellar 2be988e6b7 Give recipient_cache and associated functions a clearer name.
(imported from commit 14b12b196e33dfc4eeca5ab0f12c130ddaef1065)
2013-08-28 10:23:40 -04:00
Jessica McKellar 1f4f799030 Flush the per-process recipient cache after every request.
This cache was created to make recipient lookups within a single
request (e.g. when fetching old messages) cheaper. To support stream
name changes, we need to invalidate this cache on every request so
that users get a consistent view of the name change.

(imported from commit 801051b9f6a108c1f50be7eca9a1242d661919b1)
2013-08-28 10:23:39 -04:00
Jessica McKellar cd4daaa41f Factor the display_recipient cache key into a function.
(imported from commit 4a80c4db6ddcf8104f45b6b068e598378e655bc0)
2013-08-28 10:23:37 -04:00
Kevin Mehall 32aa0bb72d [migration] Add RealmEmoji model for realm-specific emoji.
(imported from commit d0a08b748f458c0ab68985b4cf00c78950ab40c4)
2013-08-23 21:39:29 -04:00
Steve Howell 4a7bab9a6f Allow collapsing of recipient sections.
This is experimental, for staging only.  There might be a better
way to model this than dueling force_expand/force_collapse flags,
but it works for now.  The code in collapse_recipient_group()
could also be DRYed up relative to expand_summary_row().

(imported from commit 107151d1ecd640970fb7700d41278a003bd1abaa)
2013-08-23 15:57:31 -04:00
Steve Howell 795248a1d0 Mute any topic named "muted" on zulip.com.
This change will allow us to test the muting feature on
staging.  Any topic named "muted" will automatically be
muted.  You can also mute any other topic on the console:

    muting.mute_topic('devel', 'ios');
    current_msg_list.rerender();

More UI around this experiment will be coming soon, as well
as support for muting entire streams.

The muting module keeps track of which topics are muted, but a
user can expand muted messages, and once that happens, the
messages are marked with the "force_expand" flag that gets
persisted to the back end.

Muted messages are rendered in similar fashion to the summarized
rows, and as part of unifying some of that code, we have
made it so that expanding a summarized section doesn't remove
individual flags related to summaries; instead, the messages
get the force_expand flag set.

(imported from commit acee4190e63813d46850415c41ff8ebfae4a6953)
2013-08-23 12:10:27 -04:00
Tim Abbott c2be5ca1ef Disable public streams for a CUSTOMER5 contractor.
(imported from commit bb7997cd831b9df9c951e3ecefe8a65b06bf6732)
2013-08-22 13:57:19 -04:00
Luke Faraone 078aea862c Always convert emails to lowercase in helper functions.
(imported from commit 834be7d682167c0d428c9b8a5ab8d3ae0c0d51a7)
2013-08-15 13:17:46 -04:00
Jessica McKellar befd34d111 [schema][manual] Make Stream.email_token required.
This commit CANNOT be deployed until the previous schema change
([schema] models: add an email_token field to Streams) is on prod.

Before applying this schema change, run the populate-stream-tokens
management command to generate tokens for streams that need them.

(imported from commit 7adc81c8c317ec5d59dd59ba42a4dc1a46174007)
2013-08-13 14:28:47 -04:00
Luke Faraone ecc42bc9f8 Add administrative panel to allow for user deactivations etc.
We now show a list of users and allow you to deactivate a user using the
same process as `python manage.py deactivate_user`.

We add a new menu item accessible from the gear icon which will eventually
have much more than just this, but we have a good start here.

Here we also add a property to UserProfile which determines whether you're
eligible to access the administration panel, and then have code which shows
the menu option if so.

This introduces a new JS file, admin.js.

(imported from commit 52296fdedb46b4f32d541df43022ffccfb277297)
2013-08-13 14:20:18 -04:00
Jessica McKellar c020545e02 [schema] models: add an email_token field to Streams.
The e-mail forwarder will use this. Set it to nullable temporarily to
accomodate existing streams; later commits will a) provide a script to
give all streams a token, and b) make the field non-null.

Realm administrators will eventually have a UI to regenerate stream
tokens.

(imported from commit a084d0a7012eb9665e4da095cbc46aa9ef354eaa)
2013-08-09 14:59:26 -04:00
Jessica McKellar 99b2a90440 Move completely_open to models.py.
It will have new callers soon and that's as fine a place as any to
avoid circular imports.

(imported from commit 089a724e9ad06cb5a51ffe80f1729d789238e5f6)
2013-08-08 10:12:22 -04:00
Kevin Mehall eac6463031 Implement message summarization experiment.
When you read messages in a narrow and then un-narrow, collapse
adjacent messages read in the narrow into a summary row that can
be clicked to expand those messages.

Scoped to staging with feature flags.

The implementation of this within our current MessageList is rather ugly.

(imported from commit bcb3a39d8c0c334136fe86318f18ead03f0f50bf)
2013-08-07 10:24:03 -04:00
Tim Abbott e111a2f9a5 [manual] Rename Django app from zephyr to zerver.
This needs to be deployed to both staging and prod at the same
off-peak time (and the schema migration run).

At the time it is deployed, we need to make a few changes directly in
the database:

(1) UPDATE django_content_type set app_label='zerver' where app_label='zephyr';
(2) UPDATE south_migrationhistory set app_name='zerver' where app_name='zephyr';

(imported from commit eb3fd719571740189514ef0b884738cb30df1320)
2013-08-06 07:39:36 -04:00