Commit Graph

85 Commits

Author SHA1 Message Date
Vishnu Ks f3c04f711d lib: Remove unused get_user_profile_by_email import in export.py. 2017-07-20 16:50:23 -07:00
Vishnu Ks 3cbc0cc2eb lib: Use get_system_bot in do_import_realm(export.py). 2017-07-18 17:14:05 -07:00
Rishi Gupta aa845e7f60 models: Replace ScheduledJob with ScheduledEmail.
ScheduledJob was written for much more generality than it ended up being
used for. Currently it is used by send_future_email, and nothing
else. Tailoring the model to emails in particular will make it easier to do
things like selectively clear emails when people unsubscribe from particular
email types, or seamlessly handle using the same email on multiple realms.
2017-07-17 16:05:38 -07:00
Vaida d5517bae36 Delete the old zulip.com "referrals" system.
This system hasn't been in active use for several years, and had some 
problems with it's design.  So it makes sense to just remove it to declutter
the codebase.

Fixes #5655.
2017-07-07 14:59:18 -07:00
Christian Hudon 8ab6a23a30 Fix most strict-optional issues in export.py. 2017-05-24 18:50:59 -07:00
Christian Hudon 14e871ce9c Change order of arguments so output_dir is not optional. Helps mypy too. 2017-05-24 17:32:21 -07:00
Konstantin Gukov dd76222a3f Fetch system bots using new get_system_bot function.
This eliminate a bunch of uninteresting calls to
get_user_profile_by_email.
2017-05-23 10:30:40 -07:00
Aditya Bansal b822e75a4b pep8: Add compliance with rule E261 to export.py. 2017-05-18 03:00:32 +05:30
hackerkid c4f0fa97a8 Replace timezone.make_aware with timezone_make_aware. 2017-04-16 12:28:56 -07:00
hackerkid 6ddee006bd Replace timezone.is_naive with timezone_is_naive. 2017-04-16 12:28:56 -07:00
hackerkid 55c3d12078 Replace timezone.utc with timezone_utc. 2017-04-16 12:28:56 -07:00
Harshit Bansal ac2172e233 models: Rename RealmAlias model to RealmDomain.
Includes a migration.
2017-04-04 15:48:03 -07:00
Rishi Gupta 3aae6cd421 Change if(realm.domain == zulip.com) checks to use Realm.string_id. 2017-03-13 14:17:14 -07:00
Rishi Gupta 5dc683ba8d Use Realm.string_id instead of Realm.domain when logging. 2017-03-13 09:42:14 -07:00
Raghav Jajodia a3a03bd6a5 mypy: Added Dict, List and Set imports.
Fixed mypy errors associated with the upgrade.
2017-03-04 14:33:44 -08:00
Rishi Gupta 95f5c96bec Canonicalize how we convert timestamps to UTC datetimes.
No change in behavior with this commit, just making it easier to write a
future lint rule.
2017-03-01 23:03:56 -08:00
Eklavya Sharma dd0e1f6a4c Use correct string type in boto function parameters.
boto's stubs have been updated in mypy 0.4.7, which has given us
more information about what type of strings are expected as
parameters in various functions.
2017-02-06 22:37:37 -08:00
Tim Abbott 4e171ce787 lint: Clean up E126 PEP-8 rule. 2017-01-23 22:06:13 -08:00
Tim Abbott d6e38e2a5c lint: Clean up E123 PEP-8 rule. 2017-01-23 21:34:26 -08:00
Tim Abbott e9158dd520 lint: Clean up E121 PEP-8 rule. 2017-01-23 21:02:39 -08:00
JefftheBest1 a549ed6e65 Removed accommodate typos 2017-01-12 04:53:31 -08:00
Rishi Gupta cf762eaf84 Change X.realm.id to X.realm_id across codebase.
This makes it more clearly the pattern in the Zulip codebase, and thus
decreases the risk of accidentally doing database queries.
2017-01-03 16:46:26 -08:00
nikolay abc2ff4a06 pep8: Fix many rule E128 violations.
[Tweaked by tabbott to adjust some approaches used in wrapping]
2016-12-03 13:33:31 -08:00
Sidhant Bhavnani 8c0c12c1d9 pep8: Fix E303 violations. 2016-12-02 15:34:11 -08:00
Alex Huang c8ddea16c3 pep8: Fix E122. 2016-12-01 23:16:35 -08:00
Rafid Aslam 41bd88d5ed pep8: Fix E301 pep8 violations.
Fix "E301: expected (1 or 2) blank line" pep8 violations.
2016-11-29 08:51:44 -08:00
Rafid Aslam 7a2282986a pep8: Fix E225 pep8 violations. 2016-11-28 15:21:15 -08:00
Umair Khan cfded8b5af Django 1.10: Resolve QuerySet returned by model_to_dict.
Django 1.10 resolves ManyToManyField into a QuerySet in model_to_dict.
This commit further resolves the QuerySet to the primary keys.
2016-11-09 15:29:58 -08:00
Tim Abbott 5bea2f5e20 Remove unused AVATAR_FROM_SYSTEM code.
This is some of the code we'd need if we wanted to have Zulip generate
avatars for things.  Since it is so little useful code, and it's not
clear we will need this feature ever, we can remove this code to make
the codebase less confusing.  It'd be easy to dig this out of history
if we ever want it.

Fixes #2101.
2016-10-22 19:48:50 -07:00
Tim Abbott 22fd7ba02a avatar: Move avatar hash computations to their own file. 2016-10-02 21:19:10 -07:00
Sahil Dua 058587da77 Remove extra new lines at the ends of Zulip authoried files.
Fixes #1627.

[tweaked by tabbott to avoid patching third-party modules, for now]
2016-09-26 21:05:24 -07:00
Steve Howell ffec98d85c Annotate zerver/lib/export.py.
This adds the remaining annotations to lib/export.py.
2016-09-12 08:21:46 -07:00
Steve Howell 62dd86bcce mypy: Set Path to str instead of text_type in export.py.
Using text_type for Path just breaks a lot of calls to
the core Python libraries that still want strings.
2016-09-12 08:21:46 -07:00
Tim Abbott f0a65fe52a export_messages_single_user: Fix missing order_by.
This fixes a nasty bug where exporting messages sent by a single user
might only contain some of the messages in the event that the
unspecified sort order by the database didn't happen to be sorted by
message ID.
2016-08-17 22:18:58 -07:00
Steve Howell c12bd853f7 export: Add basic export tests. (fixes #1584) 2016-08-16 13:38:37 -07:00
Steve Howell ab053845aa export: Fetch messages with two queries. 2016-08-16 13:38:37 -07:00
Steve Howell a5f651b81a export: Add some user_id filtering to do_export_realm().
This commit only addresses tables that currently derive from
user_profile_config in get_realm_config:

   zerver_userpresence
   zerver_useractivity
   zerver_useractivityinterval
   zerver_subscription
   zerver_recipient
   zerver_stream
   zerver_huddle

It also introduces an entry in realm.json for a virtual
table called "zerver_userprofile_mirrordummy" for dummy users,
which include prior dummy users and users excluded from the call
to do_export_realm().

Note that this feature is not yet exposed in the management command.
2016-08-16 13:38:37 -07:00
Tim Abbott 4a46b879ee import_uploads_s3: Fix setting of content-type. 2016-08-13 11:26:53 -07:00
Tim Abbott 8edc5110cd export: Set re_map_foreign_keys verbose default to False.
Otherwise, this is super spammy when doing a large import.
2016-08-13 11:26:16 -07:00
Tim Abbott 8b170665e4 export: Add assertion that ./manage.py exists in current directory.
Otherwise we'll fail starting the UserMessage export, later on.
2016-08-13 10:13:11 -07:00
Tim Abbott 856ab48ec6 export: Fix stream export sanity check. 2016-08-13 10:02:34 -07:00
Steve Howell e932b03999 export: Clean up path.join/makedirs for avatars/uploads. 2016-08-13 09:31:49 -07:00
Steve Howell 12c25e7d5c export: Filter attachments by message id. 2016-08-13 09:31:49 -07:00
Steve Howell 0f493d5000 export: Return msg ids from export_partial_message_files(). 2016-08-13 09:31:49 -07:00
Steve Howell 0c3e98fa91 export: Introduce attachment.json file.
Now attachment data gets written to its own json file.  We are
splitting this out so that will be easier for us to cross-check
attachments against messages without holding up writing a lot
of the other realm data.  (message cross-checking is coming soon)
2016-08-12 18:59:14 -07:00
Steve Howell ea0a7d87c8 export: Refactor how we fetch attachment data.
This commit doesn't change any behavior; it just moves fetching
attachments out of the Config scheme and into its own method.
This prepares us to start writing attachment data to its own
file and cross-checking against message ids (coming soon).
2016-08-12 18:59:14 -07:00
Steve Howell fba7a9ca21 export: Unify top-down export configuration.
We now just have a single configuration get_realm_config() that
handles most of the top-down realm export tables.  (It basically
does everything not related to messages or uploads/avatars.)

Unifying the configs allows us to be more strict in our
configuration about checking for anomalies.  In the future
we may need to loosen up some of those restrictions again,
but for now we are picky and paranoid.
2016-08-12 15:27:23 -07:00
Steve Howell 5a5353b846 export: Fetch stream data only for stream recipients.
Fetch stream data only for stream recipients, instead of
getting streams via realm_id.

(This change is kind of moot for now, since our stream recipients
include all possible stream recipients in the realm, but this
sets us up for when we start restricting users that we export
within the realm.)
2016-08-12 15:27:22 -07:00
Steve Howell 7a429d1e30 export: Add sanity_check_stream_data(). 2016-08-12 15:27:22 -07:00
Steve Howell ec86e475b4 export: Add Config.post_process_data 2016-08-12 15:12:01 -07:00