Commit Graph

1094 Commits

Author SHA1 Message Date
Tim Abbott 55d7947b76 email_mirror: Use ERROR_BOT for error reporting.
This has the nice side effect of getting rid of the now-unnecessary
ADMIN_DOMAIN from this codepath -- we really just want whichever
realm ERROR_BOT is in.
2016-08-22 21:28:01 -07:00
Steve Howell cd2e36d66f export: Add lib.utils.query_chunker(). 2016-08-22 19:27:32 -07:00
Steve Howell fd6ee7117f tests: Renamed AuthedTestCase to ZulipTestCase. 2016-08-22 19:07:44 -07:00
Tim Abbott 96a5388fcb mandrill: Fix hardcoded check for settings.VOYAGER.
Since this delayed sending feature is the only thing
settings.MANDRILL_API_KEY is used for, it seems reasonable for that to
be the gate as to whether we actually use Mandrill.
2016-08-22 16:25:28 -07:00
Taranjeet Singh 5e5ad9de0c bugdown/__init__.py: Fix line with length greater than 120.
This updates file zerver/lib/bugdown/__init__.py.
2016-08-19 11:56:43 -07:00
Rishi Gupta 64179b2fd2 confirmation: Use realm host in activation URLs.
This is preparation for merging the subdomains feature.
2016-08-18 20:12:36 -07:00
Brock Whittaker 6936d49202 Live update new avatars across users.
This sends an event when a new avatar is uploaded that refreshes the
avatar for all browser clients without the need to reload the browser.

Fixes: #1359.
2016-08-18 15:32:29 -07:00
Tim Abbott e7aac717e8 Fix markdown ordering bugs with some custom syntax.
!avatar, !modal_link, !gravatar, etc. were incorrectly being processed
before the escape character for code blocks.

While we're at it, we add tests for these special syntaxes.
2016-08-18 14:47:37 -07:00
Steve Howell 125dcad379 tools: Report OverflowErrors in url coverage report. 2016-08-18 13:28:58 -07:00
Tim Abbott 685d63b6cc notifications: Update digest email link URLs to use realm.uri.
This is preparation for fully supporting each realm having its own
subdomain in Zulip.
2016-08-17 22:58:35 -07:00
Tim Abbott 91ad923d80 notifications: Use realm.uri in unsubscribe links. 2016-08-17 22:44:38 -07:00
Tim Abbott 48ed7e2e31 notifications: Use realm_uri for click here link. 2016-08-17 22:19:53 -07:00
Tim Abbott 5f12136e7c notifications: Fix broken one click unsubscribe link. 2016-08-17 22:19:53 -07:00
Tim Abbott 6ef79ac66c notifications: Remove support@zulip.com comment in docstring. 2016-08-17 22:19:53 -07:00
Tim Abbott f0a65fe52a export_messages_single_user: Fix missing order_by.
This fixes a nasty bug where exporting messages sent by a single user
might only contain some of the messages in the event that the
unspecified sort order by the database didn't happen to be sorted by
message ID.
2016-08-17 22:18:58 -07:00
Steve Howell c12bd853f7 export: Add basic export tests. (fixes #1584) 2016-08-16 13:38:37 -07:00
Steve Howell ab053845aa export: Fetch messages with two queries. 2016-08-16 13:38:37 -07:00
Steve Howell a5f651b81a export: Add some user_id filtering to do_export_realm().
This commit only addresses tables that currently derive from
user_profile_config in get_realm_config:

   zerver_userpresence
   zerver_useractivity
   zerver_useractivityinterval
   zerver_subscription
   zerver_recipient
   zerver_stream
   zerver_huddle

It also introduces an entry in realm.json for a virtual
table called "zerver_userprofile_mirrordummy" for dummy users,
which include prior dummy users and users excluded from the call
to do_export_realm().

Note that this feature is not yet exposed in the management command.
2016-08-16 13:38:37 -07:00
Tim Abbott 4fbb8c3eee templates: Add new context variables to all templates.
This adds a few new helpful context variables that we can use to
compute URLs in all of our templates:
* external_uri_scheme: http(s)://
* server_uri: The base URL for the server's canonical name
* realm_uri: The base URL for the user's realm

This is preparatory work for making realm_uri != server_uri when we
add support for subdomains.
2016-08-13 16:27:35 -07:00
Tim Abbott 4a46b879ee import_uploads_s3: Fix setting of content-type. 2016-08-13 11:26:53 -07:00
Tim Abbott 8edc5110cd export: Set re_map_foreign_keys verbose default to False.
Otherwise, this is super spammy when doing a large import.
2016-08-13 11:26:16 -07:00
Tim Abbott 8b170665e4 export: Add assertion that ./manage.py exists in current directory.
Otherwise we'll fail starting the UserMessage export, later on.
2016-08-13 10:13:11 -07:00
Tim Abbott 856ab48ec6 export: Fix stream export sanity check. 2016-08-13 10:02:34 -07:00
Steve Howell e932b03999 export: Clean up path.join/makedirs for avatars/uploads. 2016-08-13 09:31:49 -07:00
Steve Howell 12c25e7d5c export: Filter attachments by message id. 2016-08-13 09:31:49 -07:00
Steve Howell 0f493d5000 export: Return msg ids from export_partial_message_files(). 2016-08-13 09:31:49 -07:00
Tim Abbott 924d0475c1 cache_helpers: Disable filling message cache.
The message cache filling script actually used both database and
memcached queries as part of filling the cache (all used to compute
the needed display_recipient values).  We would ideally fix this by
using bulk operations to fill the display_recipient cache, but until
we do so, this cache filler is counterproductive.

I believe this disabling fixes an issue where memcached would get
overloaded and stop handling requests during a server restart on busy
servers.
2016-08-13 09:27:56 -07:00
Rishi Gupta 708b416ca1 Terms of Service: Fix corner cases around new users being created.
This fixes a few bugs in 7910a6e134,
related to automatically created user accounts.
2016-08-13 00:19:54 -07:00
Steve Howell 0c3e98fa91 export: Introduce attachment.json file.
Now attachment data gets written to its own json file.  We are
splitting this out so that will be easier for us to cross-check
attachments against messages without holding up writing a lot
of the other realm data.  (message cross-checking is coming soon)
2016-08-12 18:59:14 -07:00
Steve Howell ea0a7d87c8 export: Refactor how we fetch attachment data.
This commit doesn't change any behavior; it just moves fetching
attachments out of the Config scheme and into its own method.
This prepares us to start writing attachment data to its own
file and cross-checking against message ids (coming soon).
2016-08-12 18:59:14 -07:00
Steve Howell fba7a9ca21 export: Unify top-down export configuration.
We now just have a single configuration get_realm_config() that
handles most of the top-down realm export tables.  (It basically
does everything not related to messages or uploads/avatars.)

Unifying the configs allows us to be more strict in our
configuration about checking for anomalies.  In the future
we may need to loosen up some of those restrictions again,
but for now we are picky and paranoid.
2016-08-12 15:27:23 -07:00
Steve Howell 5a5353b846 export: Fetch stream data only for stream recipients.
Fetch stream data only for stream recipients, instead of
getting streams via realm_id.

(This change is kind of moot for now, since our stream recipients
include all possible stream recipients in the realm, but this
sets us up for when we start restricting users that we export
within the realm.)
2016-08-12 15:27:22 -07:00
Steve Howell 7a429d1e30 export: Add sanity_check_stream_data(). 2016-08-12 15:27:22 -07:00
Steve Howell ec86e475b4 export: Add Config.post_process_data 2016-08-12 15:12:01 -07:00
Steve Howell 0c2c331905 export: Flip how we fetch stream subscription data.
We now get stream subscriptions BEFORE stream recipients.
2016-08-12 15:12:01 -07:00
Steve Howell 70a916aae3 export: Flip how we fetch user subscription data.
We now get user subscriptions BEFORE user recipients.
2016-08-12 15:12:01 -07:00
Steve Howell 2a2ce6ada1 export: Remove hard-to-maintain code comment.
Subsequent changes are gonna make the top-down/bottom-up comment
no longer valid.
2016-08-12 15:12:01 -07:00
Tim Abbott ed01842f95 notifications: Fix missed message emails with multiple senders.
Previously, missed message emails with multiple senders would
incorrectly have a "," outside the quoted sender name part of the from
address string, resulting in confusing email output.
2016-08-12 12:28:06 -07:00
Steve Howell 6fdd42c08b export: Create convenient soft links. 2016-08-12 10:48:33 -07:00
Steve Howell 70b68ddcc3 export: Use a config for export_single_user(). 2016-08-12 10:37:41 -07:00
Steve Howell c69a5bdec3 export: Handle more tables via export_from_config().
This commit introduces the ability to do custom fetches
and to essentially use temp tables for intermediate results.

(The temp table stuff deals with recipients/subscriptions
having three different flavors--user, stream, and huddle.)
2016-08-12 10:37:35 -07:00
Steve Howell f471a1779e export: Handle simple exports with export_from_config().
This handles the simple tables that don't need custom fetches.
2016-08-12 09:54:57 -07:00
Steve Howell 682155778d export: Add export_with_config().
Subsequent commits will start to use this.
2016-08-12 09:54:57 -07:00
Steve Howell b0e6d20321 export: Write stats.txt for `./manage.py export <realm>`. 2016-08-12 09:06:10 -07:00
Rishi Gupta 7910a6e134 Terms of Service: Add ability to update TOS and have users re-sign.
Most directly useful for the migration to zulipchat.com.

Creates a new field in UserProfile to store the tos_version, as well as two
new settings TOS_VERSION and FIRST_TIME_TOS_TEMPLATE. We check for a version
mismatch between what the user has signed and the current
settings.TOS_VERSION whenever the user hits the home page, and redirect them
if needed.

Note that accounts_accept_terms.html and
zerver.views.accounts_accept_terms were unused before this commit
(they date from c327446537)
2016-08-11 23:37:48 -07:00
Tim Abbott 11cac86a58 settings: Add support for specifying a remote redis password. 2016-08-11 16:54:22 -07:00
Steve Howell df3aa39be3 export: Extract write_data_to_file(). 2016-08-11 15:51:22 -07:00
Steve Howell f29b32bbb2 export: Clarify message exporting code.
The function to create the message partial files has been
renamed to export_partial_message_files().  It now gets its own
list of user profile ids and recipient ids from the response,
so that we can de-clutter do_export_realm().
2016-08-11 15:51:22 -07:00
Steve Howell 5cd915694a export: Extract launch_user_message_subprocesses().
This is the last in a series of commits that makes it
so that do_export_realm() mostly delegates work out
to other functions.
2016-08-11 15:21:30 -07:00
Steve Howell b383f5ca5d export: Extract fetch_user_profile_cross_realm(). 2016-08-11 15:21:30 -07:00