Mirror
/
zulip
mirror of https://github.com/zulip/zulip.git
1
0
Fork 0
Code Issues Projects Releases Wiki Activity
12,124 Commits 28 Branches 143 Tags 624 MiB
Commit Graph

2802 Commits

Author SHA1 Message Date
Steve Howell fba7a9ca21 export: Unify top-down export configuration. 
We now just have a single configuration get_realm_config() that
handles most of the top-down realm export tables.  (It basically
does everything not related to messages or uploads/avatars.)

Unifying the configs allows us to be more strict in our
configuration about checking for anomalies.  In the future
we may need to loosen up some of those restrictions again,
but for now we are picky and paranoid.
 2016-08-12 15:27:23 -07:00
Steve Howell 5a5353b846 export: Fetch stream data only for stream recipients. 
Fetch stream data only for stream recipients, instead of
getting streams via realm_id.

(This change is kind of moot for now, since our stream recipients
include all possible stream recipients in the realm, but this
sets us up for when we start restricting users that we export
within the realm.)
 2016-08-12 15:27:22 -07:00
Steve Howell 7a429d1e30 export: Add sanity_check_stream_data(). 2016-08-12 15:27:22 -07:00
Steve Howell ec86e475b4 export: Add Config.post_process_data 2016-08-12 15:12:01 -07:00
Steve Howell 0c2c331905 export: Flip how we fetch stream subscription data. 
We now get stream subscriptions BEFORE stream recipients.
 2016-08-12 15:12:01 -07:00
Steve Howell 70a916aae3 export: Flip how we fetch user subscription data. 
We now get user subscriptions BEFORE user recipients.
 2016-08-12 15:12:01 -07:00
Steve Howell 2a2ce6ada1 export: Remove hard-to-maintain code comment. 
Subsequent changes are gonna make the top-down/bottom-up comment
no longer valid.
 2016-08-12 15:12:01 -07:00
Steve Howell 90494cfec2 export: Note user-owned bot limitations. 2016-08-12 15:10:49 -07:00
Tim Abbott ed01842f95 notifications: Fix missed message emails with multiple senders. 
Previously, missed message emails with multiple senders would
incorrectly have a "," outside the quoted sender name part of the from
address string, resulting in confusing email output.
 2016-08-12 12:28:06 -07:00
Tim Abbott 386c6c2a31 settings: Refactor SYSTEM_ONLY_REALMS configuration. 2016-08-12 12:12:53 -07:00
Tim Abbott f0e495831e forms: Fix check for whether terms of service are enabled. 2016-08-12 11:04:45 -07:00
Steve Howell 6fdd42c08b export: Create convenient soft links. 2016-08-12 10:48:33 -07:00
Steve Howell 70b68ddcc3 export: Use a config for export_single_user(). 2016-08-12 10:37:41 -07:00
Steve Howell c69a5bdec3 export: Handle more tables via export_from_config(). 
This commit introduces the ability to do custom fetches
and to essentially use temp tables for intermediate results.

(The temp table stuff deals with recipients/subscriptions
having three different flavors--user, stream, and huddle.)
 2016-08-12 10:37:35 -07:00
Steve Howell f471a1779e export: Handle simple exports with export_from_config(). 
This handles the simple tables that don't need custom fetches.
 2016-08-12 09:54:57 -07:00
Steve Howell 682155778d export: Add export_with_config(). 
Subsequent commits will start to use this.
 2016-08-12 09:54:57 -07:00
Steve Howell b0e6d20321 export: Write stats.txt for `./manage.py export <realm>`. 2016-08-12 09:06:10 -07:00
Rishi Gupta 7910a6e134 Terms of Service: Add ability to update TOS and have users re-sign. 
Most directly useful for the migration to zulipchat.com.

Creates a new field in UserProfile to store the tos_version, as well as two
new settings TOS_VERSION and FIRST_TIME_TOS_TEMPLATE. We check for a version
mismatch between what the user has signed and the current
settings.TOS_VERSION whenever the user hits the home page, and redirect them
if needed.

Note that accounts_accept_terms.html and
zerver.views.accounts_accept_terms were unused before this commit
(they date from c327446537)
 2016-08-11 23:37:48 -07:00
Tim Abbott 11cac86a58 settings: Add support for specifying a remote redis password. 2016-08-11 16:54:22 -07:00
Steve Howell df3aa39be3 export: Extract write_data_to_file(). 2016-08-11 15:51:22 -07:00
Steve Howell f29b32bbb2 export: Clarify message exporting code. 
The function to create the message partial files has been
renamed to export_partial_message_files().  It now gets its own
list of user profile ids and recipient ids from the response,
so that we can de-clutter do_export_realm().
 2016-08-11 15:51:22 -07:00
Steve Howell 5cd915694a export: Extract launch_user_message_subprocesses(). 
This is the last in a series of commits that makes it
so that do_export_realm() mostly delegates work out
to other functions.
 2016-08-11 15:21:30 -07:00
Steve Howell b383f5ca5d export: Extract fetch_user_profile_cross_realm(). 2016-08-11 15:21:30 -07:00
Steve Howell fee2106c6f export: Extract fetch_huddle_objects(). 
This also removes the dead codepath for include_private=False.
 2016-08-11 15:21:30 -07:00
Steve Howell a6235f6a60 export: Add comments to export_single_user(). 
(This is a bit of a prefactoring to hopefully create a nice
diff in a subsequent commit.)
 2016-08-11 15:21:30 -07:00
Steve Howell 6e7fe76cf4 export: s/avatar_bucket/processing_avatars 
The name avatar_bucket was confusing for a boolean, and
in some places it was used for non-S3 paths.

I considered the more concise 'is_avatar', but that
was still confusing when you are processing multiple
files, because you think it's a calculated property
on one file instead of an overall codepath switch.

I also considered splitting up some functions, but
there is a lot of common logic between handling
file uploads and avatars that's not trivial to extract
into helpers, especially on the S3 side.
 2016-08-11 15:21:30 -07:00
Steve Howell 3dab366733 export: Clean up names of upload/avatar export functions. 
I did some minor moving around of code that made us have
one fewer function without any additional conditional
logic. The names are more explicit about saying
"from_local" and "from_s3".  Also, there is less clutter
now in do_export_realm(), which is evolving into more of
a dispatcher and less of a worker.
 2016-08-11 15:21:30 -07:00
Steve Howell 9582a83d10 export: Fix usage in export_single_user command. 2016-08-11 15:21:30 -07:00
Steve Howell d62a351107 export: Add sanity_check_output(). 2016-08-11 15:21:30 -07:00
Tim Abbott 95aee366c8 Enable local browser error reporting if ZILENCER_ENABLED. 2016-08-11 13:49:18 -07:00
Umair Khan af9f308f0e Send double columned language list in page_params. 2016-08-10 16:19:28 -07:00
Umair Khan 73e5289736 Add default_language_name in page_params. 2016-08-10 16:19:28 -07:00
Umair Khan ddaf1a2b67 Return unformatted list from get_language_list. 2016-08-10 16:19:28 -07:00
Steve Howell 06b0df5efc export: Remove spurious select_related() call for Client. 2016-08-10 14:16:17 -07:00
Steve Howell cb59a11f0a export: Extract get_primary_ids(). 2016-08-10 14:16:17 -07:00
Steve Howell 90e9083b81 export: Extract filter_by_realm(). 2016-08-10 14:16:17 -07:00
Steve Howell 4b6b1b8ad4 export: Extract filter_by_users(). 2016-08-10 14:16:17 -07:00
Steve Howell db9edfce34 export: Use DATE_FIELDS in fix_datetime_fields(). 
Now we only call this once per table and use DATE_FIELDS to
look up the data fields.
 2016-08-10 14:16:17 -07:00
Steve Howell 35c59fc4d7 export: Clean up export_messages(). 
This is pretty minor cleanup, but it makes it a little more
explicit what we're writing to the shard file, and it allows
us to use a more specific mypy type when calling
floatify_datetime_fields.
 2016-08-10 14:16:17 -07:00
Steve Howell 1d1f36c0b8 export: Always use subprocesses to export UserMessage. 
We no longer have an in-process code path to export
UserMessage rows.  We want to only maintain the
subprocess code, which we'll always use in production,
and which will work fine in dev.
 2016-08-10 14:16:17 -07:00
Steve Howell 78bbefbf94 export: Create import_attachments. 2016-08-10 14:16:17 -07:00
Rag Sagar 2fef36f15a Add realm-level default language setting. 
Adds a new field default language in the zerver_realm model.
This realm level default language will be used as default language
for newly created users. Realm level default language can be
changed from the administration page.

Fixes #1372.
 2016-08-09 17:38:29 -07:00
Steve Howell 7ec6a394fe export: Filter Attachment objects by realm. 2016-08-09 16:47:14 -07:00
Steve Howell cecfaa7761 export: Extract import_message_data(). 2016-08-09 16:47:14 -07:00
Steve Howell 5386ed280e export: Extract update_id_map(). 
We also use a vanilla dictionary instead of a defaultdict, so
that we explicitly initialize what tables are being re-mapped.
 2016-08-09 16:47:14 -07:00
Steve Howell 217ef8a4d2 export: Split fix_foreign_keys() into two functions. 
We now have convert_to_id_fields for the simple case, and
re_map_foreign_keys for the more complex case. I also
renamed some parameters and variables.
 2016-08-09 16:47:14 -07:00
Umair Khan e1090118c5 Percent translated should be an integer. 2016-08-09 16:19:07 -07:00
Rag Sagar 5771e2eb25 Move i18n functions from zerver/views/__init__.py to zerver/lib/i18n.py 2016-08-09 16:10:24 -07:00
Rag Sagar 28f0079432 Use a constant to set UserProfile default_language field max_length attribute. 2016-08-09 16:10:24 -07:00
Steve Howell dd88ffccfd export: Extract make_raw() in lib/export.py. 2016-08-09 15:58:27 -07:00
Steve Howell 09fa343bdd export: Use DATE_FIELDS in floatify_datetime_fields. 
This avoids a little bit of code duplication, plus it should
make it a little easier to add new date fields in the future.
 2016-08-09 15:58:27 -07:00
Steve Howell c14ab3c91f export: Add annotations to zerver/lib/export.py. 
I also fixed some small things like removing unnecessary return
statements, and adding a TODO.

In some cases I explicitly cast stuff at run-time to set() or
str() to appease mypy, as well as make it clear to somebody
reading the code that the callee might not respect ordering
or tolerate unicode.
 2016-08-09 15:58:27 -07:00
Steve Howell f18cc4ae3a export: Added export_avatars_local_helper(). 2016-08-09 15:58:27 -07:00
Umair Khan 86920a297c Mock out loggers in push notification tests. 
Fixes: #1575
 2016-08-09 22:16:09 +05:00
Umair Khan 7a84f12a38 Verify mock calls without strict order. 
In push notification tests, call assert_has_calls with `any_order=True`
so that the order of the calls does not matter.
 2016-08-09 22:15:37 +05:00
Tim Abbott 42b63f5caa test_templates: Fix failure to report a user-facing error. 2016-08-09 09:54:44 -07:00
Tim Abbott f3365f4089 test_templates: Add integrations_dict to context. 
This fixes the fact that this test was (silently) failing.
 2016-08-09 09:54:36 -07:00
Umair Khan fe812a89bf mypy: Type of remote_username should be text_type. 2016-08-09 09:19:24 -07:00
Umair Khan 8c249f859e Push Notification: Add tests for GCM. 2016-08-09 16:13:02 +05:00
Umair Khan acd76eb604 Annotate push_notifications.py. 2016-08-08 15:17:02 -07:00
Umair Khan 9c13e9a072 Annotate test_push_notifications. 2016-08-08 15:17:02 -07:00
Umair Khan 35dd1ffb13 Pass password as string to authenticate. 2016-08-08 15:17:01 -07:00
Umair Khan 266c791e64 Email should be a unicode. 2016-08-08 15:17:01 -07:00
Umair Khan 503ad774f5 Tokens should be unicode. 2016-08-08 15:17:01 -07:00
Umair Khan 0dd7d0dda1 push_notifications: Change name to 'identifier'. 2016-08-08 15:15:29 -07:00
Umair Khan fdbd132ba4 Annotate zerver/context_processors.py. 2016-08-08 15:15:29 -07:00
Umair Khan 00eab4e526 Annotate zerver/views/__init__.py. 2016-08-08 15:15:29 -07:00
Umair Khan 25dd1aeb5c Annotate zerver/lib/actions.py. 2016-08-08 15:13:29 -07:00
Umair Khan ef62506fb1 Annotate zerver/models.py. 2016-08-08 15:13:29 -07:00
Umair Khan e34b7a0691 Fix type annotation in message_cache_items. 2016-08-08 15:13:29 -07:00
Umair Khan ec910d96fc Annotate zerver/storage.py. 2016-08-08 15:12:12 -07:00
Umair Khan 0dfc757447 Annotate test_auth_backends.py. 2016-08-08 15:12:12 -07:00
Tim Abbott 6264ff7039 Add new Zulip realm import/export tool. 
The previous export tool would only work properly for small realms,
and was missing a number of important features:
* Export of avatars and uploads from S3
* Export of presence data, activity data, etc.
* Faithful export/import of timestamps
* Parallel export of messages
* Not OOM killing for large realms

The new tool runs as a pair of documented management commands, and
solves all of those problems.

Also we add a new management command for exporting the data of an
individual user.
 2016-08-08 14:58:18 -07:00
Tim Abbott e7adaf8db1 utils: Add mkdir_p implementation. 2016-08-08 14:36:50 -07:00
Tim Abbott 54a6bdc3ad views: Fix mypy annotations in get_invitee_emails_set. 
Apparently this buggy annotation slipped through into master.
 2016-08-05 15:04:16 -07:00
Tomasz Kolek 64b5d0ce64 Add support for inviting emails in "name" <email> format. 
Often, users will copy email addresses with a name (rather than pure
email addresses) into the Zulip "invite users" UI.  Previously, that
would throw an error.

This change also adds a get_invitee_emails_set function for parsing
emails content and a test suite for this new feature.

Fixes: #1419.
 2016-08-05 14:48:04 -07:00
Umair Khan d33fc0046f Add APNS feedback server tests. 2016-08-05 10:36:57 -07:00
Umair Khan 33450d1a0b Add APNS push notification tests. 2016-08-05 10:36:57 -07:00
Umair Khan 40054ce26c Add APNS response listener test case. 2016-08-05 10:36:57 -07:00
Umair Khan df8fc30b75 Create MockRedis class. 
Redis is not available on Travis and creating a mock Redis server looks
easier than using `mock` module.
 2016-08-05 10:36:57 -07:00
Umair Khan 2ab796aeb5 Add tests for APNS message. 2016-08-05 10:36:57 -07:00
Umair Khan 882bb5558b Push to APNS in a loop. 2016-08-05 10:36:57 -07:00
Umair Khan 4e8ca0a326 Make sure apns response code is an integer. 2016-08-05 10:36:57 -07:00
Umair Khan 5c41eae7ec Don't send connection to apns response listener. 2016-08-05 10:36:57 -07:00
Umair Khan ffaf7cb2ba Create get_connection function for APNS. 2016-08-05 10:36:57 -07:00
Umair Khan 1d2bdfdfb1 Save user id instead of user profile. 2016-08-05 10:36:57 -07:00
Umair Khan 0cac1aa135 For APNS frame identifier generate random int. 2016-08-05 10:36:57 -07:00
Umair Khan 0b5dc56b8d Redis key expiry should be an integer. 2016-08-05 10:36:57 -07:00
Umair Khan cbd53fbac8 Don't import OrderedDict. 2016-08-05 10:36:57 -07:00
Umair Khan 0b88deb640 APNS_SANDBOX should be a bool. 
Fixes: #1480
 2016-08-05 10:36:57 -07:00
Tim Abbott dcb92d41b8 Annotate test_messages.py. 2016-08-04 15:57:04 -07:00
Tim Abbott 39f1024740 test_messages: Fix remaining mypy errors. 2016-08-04 15:57:03 -07:00
Tim Abbott 88550edd9e Annotate queue_processors.py. 2016-08-04 15:57:03 -07:00
Tim Abbott 4e8054b84a Refactor queue worker tests to match actual API. 2016-08-04 15:57:03 -07:00
Tim Abbott 1e54897ca7 process_queue: Add missing type annotation. 2016-08-04 15:57:03 -07:00
Tim Abbott 4ddc567c50 views/__init__: Add missing annotations. 2016-08-04 15:57:03 -07:00
Tim Abbott 641aa08721 compilemessages: Add mypy annotations. 2016-08-04 15:57:03 -07:00
Tim Abbott a787dee48b makemessages: Add missing annotations and fix string types. 2016-08-04 15:57:03 -07:00
Tim Abbott 2da6b6bfd8 views.messages: Add some more annotations. 2016-08-04 15:57:03 -07:00
Tim Abbott 2f13b0b18a actions: Fix various string annotations. 2016-08-04 15:57:03 -07:00