Commit Graph

33568 Commits

Author SHA1 Message Date
Anders Kaseorg cb0652071f zulip_otp_required: Avoid _user_is_authenticated compatibility shim.
This private shim is not needed with Django ≥ 1.10, and was removed
from django-otp 0.7.0.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-09-23 11:56:52 -07:00
Anders Kaseorg 2573ecb7d1 requirements: Bump thumbor Django to match main Django.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-09-23 11:56:36 -07:00
Anders Kaseorg 92b42573fb requirements: Compile thumbor requirements with pip-tools on Python 2.
The reason that `pip-tools` running on Python 3 didn’t detect the
right requirements for `thumbor` on Python 2 is simply that some of
them are conditional on the Python version.

As for the requirements that had been manually added as a workaround:
`backports-abc` and `singledispatch` are now correctly detected, while
`backports.ssl-match-hostname` was vendored into `urllib3` some time
ago and `certifi` is no longer necessary.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-09-23 11:56:36 -07:00
Tim Abbott 3c0f54e242 docs: Update documented string_id for zulipinternal realm.
See 9b2b24c617 for the update of the
string_id.
2019-09-23 11:51:10 -07:00
Tim Abbott e4dee9532c nagios: Update configuration for user_activity worker change.
Since LoopQueueProcessingWorker jobs cannot be monitored by checking
for connected consumers (since they poll, rather than consuming as
events arrive), they can't be monitored with check_consumers.  It's
OK, because that monitoring was redundant with monitoring for
potential growth in their queue that we have as well.

Also clean up the block comments for the two other similar queue
procesors.
2019-09-23 11:49:46 -07:00
Rishi Gupta 3646936cfb integrations: Advertise our native Zapier integration. 2019-09-22 18:47:42 -07:00
Anders Kaseorg 8d91bebf95 restart-server: Warn if the shell’s PWD goes through an updated symlink.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-09-21 12:02:15 -07:00
Ray Kraesig a4ababb9a4 vagrant docs: specify restart command
Add a specific command to restart Vagrant to adopt the new
configuration.

(When naïvely using only `vagrant halt` + `vagrant up --provision`,
external devices remained unable to connect; per `netstat -nltp`, the
host IP of forwarded ports remained `127.0.0.1`.)
2019-09-21 11:54:14 -07:00
Tim Abbott 767a1b9f0c user_activity: Skip an unnecessary database query for new entries.
While new entries are an overwhelming minority of calls to this
function, it's still worth having a clean implementation.
2019-09-21 11:49:43 -07:00
Tim Abbott 1c73ce2450 user_activity: Use LoopQueueProcessingWorker strategy.
This should dramatically improve the queue processor's performance in
cases where there's a very high volume of requests on a given endpoint
by a given user, as described in the new docstring.

Until we test this more broadly in production, we won't know if this
is a full solution to the problem, but I think it's likely.  We've
never seen the UserActivityInterval worker end up backlogged without a
total queue processor outage, and it should have a similar workload.

Fixes #13180.
2019-09-21 11:48:24 -07:00
Tim Abbott 52a2b2c6a8 tests: Move user_activity test next to slow_query test.
This will conveniently put all the LoopQueueProcessingWorkers
together.
2019-09-21 11:47:14 -07:00
Tim Abbott f0d8951035 do_update_user_activity: Refactor to support passing a count.
We'll use this in upcoming commits.
2019-09-21 11:47:14 -07:00
Tim Abbott 5c960b3e0f user_activity: Make the queue processor a bit more efficient.
We don't actually need to go to the memcached (falling back to the
database) to fetch either user or client objects on every event.  For
user objects, we actually can just pass through the user ID
transparently; for client objects, we can use an in-process cache,
since the mapping of string to ID never changes.
2019-09-21 11:47:14 -07:00
Mateusz Mandera 3271235200 email_mirror: Ignore missed message email if the user isn't active. 2019-09-20 17:58:10 -07:00
Mateusz Mandera 21459ed193 test_email_mirror: Test missed message emails in a single ZulipTestCase.
With the way these tests are, it's unnecessary to have 3 separate
classes, and it makes it confusing to decide where to add a potential
additional mm email test.
2019-09-20 17:58:10 -07:00
Mateusz Mandera 371a7768ad backends: Extract useful is_user_active function.
This logic can be useful elsewhere, for checking whether user_profile is
active.
2019-09-20 17:58:10 -07:00
Tim Abbott c19b7ec89d test_events: Remove unnecessary check_none validator.
This can be done cleanly using the existing equals(None).
2019-09-20 11:51:50 -07:00
Anders Kaseorg 6b09e690f1 page_params: Throw an error if params are missing.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-09-20 10:34:44 -07:00
Anders Kaseorg dea6889956 templates: Make the Loading… message more robust.
Don’t hide it until both CSS and JS have loaded.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-09-20 10:34:44 -07:00
Anders Kaseorg 46e562f990 bootstrap: Change tooltip html default to false.
Bootstrap v2.2.0^2~40^2~6 changes this default to false, so this is a
prerequisite to upgrading Bootstrap, and it’s also safer.

This closes an HTML injection path via user full names in the emoji
reaction tooltip.  It doesn’t appear to be exploitable for cross-site
scripting because we disallow `>` in full names, and the code happens
to be written such that the next `>` is in a different parser
invocation.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-09-19 20:53:10 -07:00
Anders Kaseorg cf5a00d94b bulk_get_subscriber_user_ids: Sort each user list by ID.
This simple backwards-compatible change saves approximately 12% in the
compressed size of the chat.zulip.org page_params.  We can do much,
much better by changing the format, but this seems like a good
intermediate step.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-09-19 20:17:43 -07:00
Anders Kaseorg 2e1494bdbd setup-apt-repo: Add ca-certificates to pre_setup_deps.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-09-19 20:15:43 -07:00
Rishi Gupta 74d5d3db26 integration docs: Remove angle brackets from trello bash command.
If you accidentally leave them in, you'll get a confusing error message.
2019-09-19 17:48:16 -07:00
Tim Abbott a352f2e10d test_counts: Remove custom user creation code.
Like the last commit, this avoids us needing to update this random
test class in the analytics subsystem when we adjust the UserProfile
model.
2019-09-19 14:31:58 -07:00
Tim Abbott aae610f65c analytics: Fix creation of shylock user.
Previously, the shylock user was created using a bad duplicate of our
standard create_user code path; we fix this by just calling into that
code path.
2019-09-19 14:16:38 -07:00
Yago González e1fbf6dddb actions: Make do_regenerate_api_key return the new key.
This way, the new API key can be fetched without needing to read it from
the UserProfile object.
2019-09-19 13:03:56 -07:00
Anders Kaseorg fbc2de157e templates: Move page_params to a <div> at the bottom of <body>.
In a gigantic realm where we send several MB of `page_params`, it’s
slightly better to have the rest of the `<body>` available to the
browser earlier, so it can show the “Loading…” spinner and start
fetching subresources.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-09-19 12:38:10 -07:00
Mateusz Mandera 935702b6ec auth: Rename any_oauth_backend_enabled to any_social_backend_enabled.
any_oauth_backend_enabled is all about whether we will have extra
buttons on the login/register pages for logging in with some non-native
backends (like Github, Google etc.). And this isn't about specifically
oauth backends, but generally "social" backends - that may not rely
specifically rely on Oauth. This will have more concrete relevance when
SAML authentication is added - which will be a "social" backend,
requiring an additional button, but not Oauth-based.
2019-09-19 12:35:27 -07:00
Mateusz Mandera 050f814e47 auth: Clean up SOCIAL_AUTH_BACKENDS / OAUTH_BACKEND_NAMES lists.
SOCIAL_AUTH_BACKEND / OAUTH_BACKEND_NAMES are currently the same
backends. All Oauth backends are social, and all social are oauth.
So we get rid of OAUTH_BACKEND_NAMES and use only SOCIAL_AUTH_BACKENDS.
2019-09-19 12:35:27 -07:00
Rohitt Vashishtha 6a08efc065 typeahead: Precompile regexes for removing diacritics.
Precompiling regexes gives a performance increase of around 10-15%
based on tests. See https://jsperf.com/typeahead-regex. This stacks
up when we have a lot of users in an organisation.
2019-09-19 12:27:01 -07:00
Anders Kaseorg b72bb8171b nginx: Add CORS, HSTS, and X-Frame-Options headers to error responses.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-09-19 12:10:18 -07:00
Anders Kaseorg 990f08da90 dependencies: Upgrade nearly all JavaScript dependencies to latest.
Keep katex at 0.10.x, mini-css-extract-plugin at 0.6.x
(https://github.com/webpack-contrib/mini-css-extract-plugin/issues/417),
and typescript-eslint at 1.x for now.  Upgrade everything else.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-09-19 12:08:52 -07:00
Anders Kaseorg fc0554fcb8 emoji: Finish removing leftover code from banned Emoji sets.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-09-19 12:08:52 -07:00
clarammdantas cf5d3a3ef3 left sidebar: Fix bot availability status in "private messages".
This changes the availability icon for bot users to user_circle_green;
previously it was accidentally defaulting to user_circle_empty, making
it appear that bots were never available.

Fixes #13149.
2019-09-18 17:40:25 -07:00
Tim Abbott 75d579847e docs: Update changelog through current master. 2019-09-18 17:36:31 -07:00
Tim Abbott 356e68e09e test-backend: Temporarily disable coverage requirement for mattermost.
The mattermost import tool has some temporary coverage regressions.
2019-09-18 12:11:58 -07:00
Vishnu Ks c4af0b7bc4 mattermost: Support importing messages without team name.
Mattermost doesn't place private messages within a particular team,
which is what this is needed for.
2019-09-18 11:57:37 -07:00
Vishnu Ks bf5f531e90 import_util: Support huddles in SubscriberHandler. 2019-09-18 11:53:13 -07:00
Tim Abbott edee1251c8 message_list: Replace buggy rerender_the_whole_thing.
As it turns out, our rerender_the_whole_thing function (used whenever
we were adding messages and discovered that the resulting message list
would be out-of-order) was just broken and scrolled the browser to a
random location.

This caused two user-facing bugs:

* On very fast networks, if two users sent messages at very close to
  the same time, we could end up with out-of-order message deliveries,
  triggering this code path, which was intended to silently correct
  the situation, but failed.

* In some narrows to streams with muted topics in the history but some
  recent traffic, the user's browser-cached history might have some
  gaps that mean the server fetch we do after narrowing discovers the
  history is out-of-order, again triggering the
  rerender_the_whole_thing code path.

The fix is to just remove that function, adding a new option to the
well-tested rerender_preserving_scrolltop (which has explicit logic to
preserve the scroll position) instead.

Fixes #12067.  Likely also fixes #12498.
2019-09-18 11:43:21 -07:00
Tim Abbott 0815a9bd53 docs: Add a link to changelog from section on deploying master. 2019-09-18 11:18:12 -07:00
Rishi Gupta 2646d70cbd emails: Add realm name to footer of missed message emails.
Missed message emails for mentions come from streams and possibly orgs you
don't normally get missed message emails for, so they can be hard to place.
2019-09-17 17:45:59 -07:00
Anders Kaseorg fe7d814e8d team: Move contributors_list into page_params.
This sidesteps tricky escaping issues, and will make it easier to
build a strict Content-Security-Policy.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-09-17 16:06:33 -07:00
Anders Kaseorg 7494f1600c templates: Move page_params from an inline script to the <body> dataset.
This sidesteps tricky escaping issues, and will make it easier to
build a strict Content-Security-Policy.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-09-17 16:06:33 -07:00
Anders Kaseorg f1ecd3c18b templates: Defer all script execution.
This allows parsing and resource loading to continue in parallel with
script fetching.  It also means we don’t need to concern ourselves
with explicitly waiting for the DOM to become ready.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-09-17 16:06:33 -07:00
Anders Kaseorg 9b38bcc2a6 templates: Abstract render_bundle behind a Jinja2 macro.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-09-17 16:06:33 -07:00
Rohitt Vashishtha 1df5cdc41a typeahead: Align the tip text centered vertically. 2019-09-17 13:09:03 -07:00
Rohitt Vashishtha 3e3deb2f17 typeahead: Move tip text to bottom. 2019-09-17 13:09:03 -07:00
Tim Abbott 68f6f622a7 open source: Add a more complete "why zulip" explanation.
Also contains a lot of edits from Rishi, including to the content
around Zulip's features and what to highlight there.
2019-09-17 12:40:38 -07:00
Tim Abbott 5452c0b4ac open source: Document some recent and upcoming features. 2019-09-17 12:03:20 -07:00
Rishi Gupta 2ebbd9a917 portico: Fix line-height in hero image description. 2019-09-17 12:03:20 -07:00