Commit Graph

12171 Commits

Author SHA1 Message Date
Tim Abbott bfa987f26e Add missing realm_uri and server_uri to HomeTest. 2016-08-13 16:52:56 -07:00
Rishi Gupta 205fb996ec templates: Use realm_uri and server_uri settings for server URL.
The policy here is essentially:
* Pages on the central server (documentation, etc.) are server_uri.
* Pages associated with a user's realm server (anything logged-in, plus
  things like their login page) are realm_uri.
2016-08-13 16:37:45 -07:00
Rishi Gupta 902a7fbfe9 Change relative paths to absolute paths when we don't want the subdomain.
Changes relative path to an absolute path (that doesn't contain the
subdomain) for various links to
/create_realm, /api, /apps, /integrations, /hello, /terms, and the logged
out / (the Zulip in the upper left corner of portico)

I typically left links internal to the relevant pages (e.g. a link from
integrations.html to a subpage of integrations/) as relative links, and
changed external links from within the app to the absolute path (e.g. the
link to integrations from the gear menu).
2016-08-13 16:27:36 -07:00
Rishi Gupta dff43fcc5e Change https:// to external_uri_scheme in templates.
Helps in case the development environment is not using https.
We get the value from zerver/context_processors.py (which is
ultimately reading it from settings)
2016-08-13 16:27:36 -07:00
Tim Abbott 5cad3f5707 handlebars: Add server_uri and realm_uri to page_params. 2016-08-13 16:27:36 -07:00
Tim Abbott 4fbb8c3eee templates: Add new context variables to all templates.
This adds a few new helpful context variables that we can use to
compute URLs in all of our templates:
* external_uri_scheme: http(s)://
* server_uri: The base URL for the server's canonical name
* realm_uri: The base URL for the user's realm

This is preparatory work for making realm_uri != server_uri when we
add support for subdomains.
2016-08-13 16:27:35 -07:00
Tim Abbott 4a46b879ee import_uploads_s3: Fix setting of content-type. 2016-08-13 11:26:53 -07:00
Tim Abbott 8edc5110cd export: Set re_map_foreign_keys verbose default to False.
Otherwise, this is super spammy when doing a large import.
2016-08-13 11:26:16 -07:00
Tim Abbott 8b170665e4 export: Add assertion that ./manage.py exists in current directory.
Otherwise we'll fail starting the UserMessage export, later on.
2016-08-13 10:13:11 -07:00
Tim Abbott 856ab48ec6 export: Fix stream export sanity check. 2016-08-13 10:02:34 -07:00
Steve Howell 603352630b export: Update conversion docs. 2016-08-13 09:36:08 -07:00
Steve Howell e932b03999 export: Clean up path.join/makedirs for avatars/uploads. 2016-08-13 09:31:49 -07:00
Steve Howell 12c25e7d5c export: Filter attachments by message id. 2016-08-13 09:31:49 -07:00
Steve Howell 0f493d5000 export: Return msg ids from export_partial_message_files(). 2016-08-13 09:31:49 -07:00
Tim Abbott 924d0475c1 cache_helpers: Disable filling message cache.
The message cache filling script actually used both database and
memcached queries as part of filling the cache (all used to compute
the needed display_recipient values).  We would ideally fix this by
using bulk operations to fill the display_recipient cache, but until
we do so, this cache filler is counterproductive.

I believe this disabling fixes an issue where memcached would get
overloaded and stop handling requests during a server restart on busy
servers.
2016-08-13 09:27:56 -07:00
Rishi Gupta befd665b91 Update static_header.txt with new license information.
This is the header file for the minified javascript.
2016-08-13 00:37:21 -07:00
Rishi Gupta 708b416ca1 Terms of Service: Fix corner cases around new users being created.
This fixes a few bugs in 7910a6e134,
related to automatically created user accounts.
2016-08-13 00:19:54 -07:00
Tim Abbott 3bae3cd54d settings: Add option to disable email deliverer. 2016-08-12 19:24:45 -07:00
Steve Howell 0c3e98fa91 export: Introduce attachment.json file.
Now attachment data gets written to its own json file.  We are
splitting this out so that will be easier for us to cross-check
attachments against messages without holding up writing a lot
of the other realm data.  (message cross-checking is coming soon)
2016-08-12 18:59:14 -07:00
Steve Howell ea0a7d87c8 export: Refactor how we fetch attachment data.
This commit doesn't change any behavior; it just moves fetching
attachments out of the Config scheme and into its own method.
This prepares us to start writing attachment data to its own
file and cross-checking against message ids (coming soon).
2016-08-12 18:59:14 -07:00
ukhan db81647b7b Update packages in requirements that won't break tests
upgrade moto, downgrade httpretty for compatibility

downgrade commonmark for compatibility
2016-08-12 17:56:23 -07:00
ukhan 180b438c44 Upgrade setuptools prior to updating mock 2016-08-12 17:56:23 -07:00
Steve Howell fba7a9ca21 export: Unify top-down export configuration.
We now just have a single configuration get_realm_config() that
handles most of the top-down realm export tables.  (It basically
does everything not related to messages or uploads/avatars.)

Unifying the configs allows us to be more strict in our
configuration about checking for anomalies.  In the future
we may need to loosen up some of those restrictions again,
but for now we are picky and paranoid.
2016-08-12 15:27:23 -07:00
Steve Howell 5a5353b846 export: Fetch stream data only for stream recipients.
Fetch stream data only for stream recipients, instead of
getting streams via realm_id.

(This change is kind of moot for now, since our stream recipients
include all possible stream recipients in the realm, but this
sets us up for when we start restricting users that we export
within the realm.)
2016-08-12 15:27:22 -07:00
Steve Howell 7a429d1e30 export: Add sanity_check_stream_data(). 2016-08-12 15:27:22 -07:00
Steve Howell ec86e475b4 export: Add Config.post_process_data 2016-08-12 15:12:01 -07:00
Steve Howell 0c2c331905 export: Flip how we fetch stream subscription data.
We now get stream subscriptions BEFORE stream recipients.
2016-08-12 15:12:01 -07:00
Steve Howell 70a916aae3 export: Flip how we fetch user subscription data.
We now get user subscriptions BEFORE user recipients.
2016-08-12 15:12:01 -07:00
Steve Howell 2a2ce6ada1 export: Remove hard-to-maintain code comment.
Subsequent changes are gonna make the top-down/bottom-up comment
no longer valid.
2016-08-12 15:12:01 -07:00
Steve Howell fd77585a8c export: Add docs/conversion.md.
This is not hooked into anything yet.  After this weekend's
conversion, it will probably be worthwhile to take a second pass
at it!
2016-08-12 15:10:49 -07:00
Steve Howell 90494cfec2 export: Note user-owned bot limitations. 2016-08-12 15:10:49 -07:00
Tim Abbott 88a123d5e0 Fix excessive CPU usage by rabbitmq-numconsumers Nagios checks.
The previous model for these Nagios checks was kinda crazy -- every
minute, we'd run a full `rabbitmctl list_consumers` for each of the
dozen+ consumers that we have, and then do the exact same parsing
logic for each to determine whether the target queue has a running
consumer to write out a state file.

Because `rabbitmctl list_consumers` takes a small amount of resources,
on systems where CPU is very limited (e.g. t2 style AWS instances),
this minor CPU wastage could be problematic.

Now we just do that `rabbitmqctl list_consumers` once per minute, and
output all the state files from a single command.

Further TODO items on this front include removing the hardcoded list
of queues.
2016-08-12 14:09:36 -07:00
Tim Abbott 852af83d3c check_postgres_replication_lag: Fix psql command line.
This allows the plugin to be run as users other than the "zulip" user,
where the "zulip" database would not be the default.
2016-08-12 13:19:08 -07:00
Tim Abbott 5194485d74 log2zulip: use a Zulip virtualenv if available.
This makes it easier to use log2zulip in a Zulip server environment.
2016-08-12 13:17:36 -07:00
Tim Abbott 5b53521b32 memcached: set a low retry_timeout. 2016-08-12 12:57:24 -07:00
Tim Abbott 4baf120c7c memcached: Enable TCP_NODELAY optimization. 2016-08-12 12:57:24 -07:00
Tim Abbott d8c066ba52 memcached: Pass verify_keys option. 2016-08-12 12:57:23 -07:00
Tim Abbott ed01842f95 notifications: Fix missed message emails with multiple senders.
Previously, missed message emails with multiple senders would
incorrectly have a "," outside the quoted sender name part of the from
address string, resulting in confusing email output.
2016-08-12 12:28:06 -07:00
Tim Abbott 386c6c2a31 settings: Refactor SYSTEM_ONLY_REALMS configuration. 2016-08-12 12:12:53 -07:00
Tim Abbott ecb81fabf3 settings: Extract EVENT_LOGS_ENABLED setting. 2016-08-12 12:12:53 -07:00
Tim Abbott f0e495831e forms: Fix check for whether terms of service are enabled. 2016-08-12 11:04:45 -07:00
Steve Howell 6fdd42c08b export: Create convenient soft links. 2016-08-12 10:48:33 -07:00
Steve Howell 70b68ddcc3 export: Use a config for export_single_user(). 2016-08-12 10:37:41 -07:00
Steve Howell c69a5bdec3 export: Handle more tables via export_from_config().
This commit introduces the ability to do custom fetches
and to essentially use temp tables for intermediate results.

(The temp table stuff deals with recipients/subscriptions
having three different flavors--user, stream, and huddle.)
2016-08-12 10:37:35 -07:00
Steve Howell f471a1779e export: Handle simple exports with export_from_config().
This handles the simple tables that don't need custom fetches.
2016-08-12 09:54:57 -07:00
Steve Howell 682155778d export: Add export_with_config().
Subsequent commits will start to use this.
2016-08-12 09:54:57 -07:00
Tim Abbott 6496fe2a53 travis: Remove rabbitmq nodename dependency on hostname.
Because rabbitmq doesn't support changing the nodename of a running
rabbitmq node, Zulip installations suffered a plague of issues where
e.g. a Zulip server would reboot, the hostname would change, and
suddenly the local rabbitmq instance being used by Zulip would stop
working.

We address this problem by using, by default, a fixed rabbitmq
nodename, but providing server administrators the option to set the
rabbitmq nodename used by Zulip however they choose.

To upgrade an existing server to use this new configuration, one will
need to add something like the following to /etc/zulip/zulip.conf:

[rabbitmq]
nodename = zulip@localhost

However, I don't believe we have the puppet code in place to make this
work correctly at initial installation without rabbitmq-server being
already installed (but off), as we can easily setup in Travis CI but I
haven't been willing to do for the installer.  So for now, this just
fixes our Travis CI problems.

Fixes: #1579.
2016-08-12 09:38:23 -07:00
Tim Abbott c7059c9751 travis: Update success-http-headers to match current certs.
Travis CI seems to have changed the way the snakeoil SSL certs are
generated in their infrastructure, so we need to update our expected
"success" HTTP headers accordingly.
2016-08-12 09:35:41 -07:00
Steve Howell b0e6d20321 export: Write stats.txt for `./manage.py export <realm>`. 2016-08-12 09:06:10 -07:00
Rishi Gupta 7910a6e134 Terms of Service: Add ability to update TOS and have users re-sign.
Most directly useful for the migration to zulipchat.com.

Creates a new field in UserProfile to store the tos_version, as well as two
new settings TOS_VERSION and FIRST_TIME_TOS_TEMPLATE. We check for a version
mismatch between what the user has signed and the current
settings.TOS_VERSION whenever the user hits the home page, and redirect them
if needed.

Note that accounts_accept_terms.html and
zerver.views.accounts_accept_terms were unused before this commit
(they date from c327446537)
2016-08-11 23:37:48 -07:00