Commit Graph

9459 Commits

Author SHA1 Message Date
Tim Abbott 1ce0e8256b zoom: Avoid sending Zoom API secret to other admin clients.
Fixing this involves fixing the backend to handle unchanged field
submissions of the Zoom credentials without trying to re-validate the
credentials (for performance) as well as to fetch the already-sent
secret.
2019-02-28 15:43:42 -08:00
Rishi Gupta d833c70dc7 org settings: Explain Zoom support is experimental, and fix a few strings.
Visually, #zoom_help_text acts like
.organization-settings-parent div:first-of-type when the Zoom option
is selected, but isn't treated as such.

No visual change with the #google_hangouts_domain change; just there to make
the code more readable/defensible.
2019-02-28 15:09:35 -08:00
Anders Kaseorg 88e90d34b9 try_git_describe: Set cwd, not --git-dir.
git describe --dirty requires a working tree, not just a repository.

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2019-02-28 14:59:33 -08:00
Anders Kaseorg c6e60ebf0e backup: Set cwd while running pg_dump.
This avoids a spurious permission error inside the Postgres
`resolve_symlinks` function if we don’t have access to the current
working directory (e.g. we’re running with cwd /root inside `su
zulip`).

While we’re here, add a defensive `--` argument.

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2019-02-28 14:40:28 -08:00
Tim Abbott bc3b864754 export: Add a bunch of comments to our export tool. 2019-02-28 12:20:08 -08:00
Tim Abbott de6f724bc5 middleware: Avoid doing work for statsd when not enabled.
This saves about 8% of the runtime of our total response middleware,
or equivalently close to 2% of the total Tornado response time.  Which
is pretty significant given that we're not sure anyone is using statsd
in production.

It's also useful outside Tornado, but the effect is particularly
significant because of how important Tornado performance is.
2019-02-27 17:53:15 -08:00
Tim Abbott c955b20131 middleware: Don't repreatedly regenerate open graph functions.
This avoids parsing these functions on every request, which was
adding roughly 350us to our per-request response times.

The overall impact was more than 10% of basic Tornado response
runtime.
2019-02-27 17:53:13 -08:00
Tim Abbott cefebd87a4 docs: Mention bug in ALWAYS_SEND_ALL_HOTSPOTS feature.
This should avoid some wasted time.
2019-02-27 10:21:30 -08:00
Rishi Gupta 6d51b25fc6 help: Rework docs for joining and inviting. 2019-02-27 09:33:37 -08:00
Tim Abbott 216d2ec1bf production: Add optional support for submitting usage statistics.
See documentation for details.
2019-02-26 17:35:10 -08:00
Greg Price de2f1ee0c4 push notif: Make O(1) database queries in handle_remove_... 2019-02-26 16:41:55 -08:00
Greg Price b0a84cd7ab message: Add an O(1)-query variant of bulk_access_messages.
We'll use this in the push-notifications code, in a context where
there should definitely already be UserMessage rows if everything's
gone normally... but explicitly checking at the top seems like the
right pattern from a secure-coding perspective.
2019-02-26 16:41:54 -08:00
Greg Price 9869153ae8 push notif: Send a batch of message IDs in one `remove` payload.
When a bunch of messages with active notifications are all read at
once -- e.g. by the user choosing to mark all messages, or all in a
stream, as read, or just scrolling quickly through a PM conversation
-- there can be a large batch of this information to convey.  Doing it
in a single GCM/FCM message is better for server congestion, and for
the device's battery.

The corresponding client-side logic is in zulip/zulip-mobile#3343 .

Existing clients today only understand one message ID at a time; so
accommodate them by sending individual GCM/FCM messages up to an
arbitrary threshold, with the rest only as a batch.

Also add an explicit test for this logic.  The existing tests
that happen to cause this function to run don't exercise the
last condition, so without a new test `--coverage` complains.
2019-02-26 16:41:54 -08:00
Greg Price 28ff9670de push notif: Push `gcm_options` logic inside "payload" helpers.
These are logically closely related.
2019-02-26 16:41:54 -08:00
Greg Price 8f26e12c85 push notif: Clarify get_*_payload, and factor another out.
This is a pure refactor; adding docstrings, making some names more
explicit, and pulling out one small helper.
2019-02-26 16:41:54 -08:00
Greg Price 69ded8b1b4 push notif: Drop irrelevant fields in `remove` payloads.
These fields don't make much sense in this case; and the client
doesn't look at them and never has.  Stop including them.
2019-02-26 16:41:54 -08:00
synicalsyntax 33bd52388a integrations: Rename social media logos to bypass Adblock Plus.
Adblock Plus's "Block social media icons tracking" setting blocked
integration logos for social media platforms from loading, so the logos
are renamed to bypass this.

Fixes #11590.
2019-02-23 10:56:43 -08:00
synicalsyntax d6483a99d0 integrations: Update xkcd logo. 2019-02-22 18:16:29 -08:00
Anders Kaseorg f1ec67c614 python: Mark intentionally unused imports with noop statements.
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2019-02-22 16:54:47 -08:00
Anders Kaseorg 649235cfec python: Remove unused imports.
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2019-02-22 16:54:36 -08:00
Eeshan Garg 8821b5a903 webhooks/pagerduty: Account for missing trigger summary data.
Some incoming payloads do not include details such as the trigger's
summary and description.
2019-02-22 15:59:28 -08:00
Eeshan Garg 912931e1bc webhooks/trello: Support updateCheckItemStateOnCard events. 2019-02-22 15:59:28 -08:00
Eeshan Garg 788dd48c93 webhooks/trello: Ignore createCheckItem events.
Notifications for every check item that is added to a card's
check list would be too noisy.
2019-02-22 15:59:28 -08:00
Eeshan Garg 9af2aa5566 webhooks/trello: Recommend wrapping webhook URL in quotes.
The ampersand in the query string of a URL can confuse bash,
leading to situations where anything in the URL beyond the
ampersand isn't parsed.
2019-02-22 15:59:28 -08:00
Challa Venkata Raghava Reddy e7fb19c8b0 invitations: Fix email validation errors for deactivated accounts.
This provides a much clearer error message when trying to invite a
user who has a deactivated account.

Fixes part of #8144.
2019-02-22 14:27:22 -08:00
Eeshan Garg cfe0f6b56d webhooks/clubhouse: Account for bug in GitHub branch payloads.
This change was prompted by a possible bug on Clubhouse's end. In
general, if a branch is added, it also prompts a workflow state
transition in its primary story.

However, our webhook error logs show an erroneous payload where
the same branch is added to another story, but there is no
workflow state transition. Also, both the stories are grouped in
the same payload, which confused/broke our code. Ideally, this
shouldn't happen and is most likely a bug on Clubhouse's end.
In most cases, changes included in Clubhouse payloads never
pertain to more than one parent entity (stories) simultaneously,
and we usually operate under the assumption that the changes
included therein are related to each other in terms of their
parent object (story or epic) and not a child object (the GitHub
branch).
2019-02-20 16:32:42 -08:00
Eeshan Garg 76b1a8379b webhooks/clubhouse: Support epic_archive events. 2019-02-20 16:32:42 -08:00
Eeshan Garg 51a1b76a1f webhooks/clubhouse: Ignore story comment updates.
Comment updates in general aren't very informative.
Plus, errors for this event type were cluttering up our
webhook logs.
2019-02-20 16:32:42 -08:00
Eeshan Garg 76e58d05cb webhooks/jira: Ignore a few noisy events.
UnexpectedWebhookEventType errors for these events were cluttering
up our webhook error logs.
2019-02-20 16:32:42 -08:00
Eeshan Garg fa29006311 webhooks/github: Ignore check_suite events.
A check suite is a collection of check runs. We care a lot more
about the outcomes of check runs in this case because check_run
payloads are a lot more informative than check_suite payloads.

(And in any case, the check_suite events are primarily for notifying
tools like CI to run checks).
2019-02-20 16:32:42 -08:00
Eeshan Garg a0717e4424 webhook/github: Support check_run events.
We only support notifications for events where a check run has
completed. Notifications for when a check run has been queued or
is in progress are not very informative and may be too noisy.
2019-02-20 16:32:39 -08:00
Tim Abbott cbc62b8e07 streams: Prevent creation of multi-line stream descriptions.
We do not anticipate our UI for showing stream descriptions looking
reasonable for multi-line descriptions, so we should just ban creating
them.

Given the frontend changes, multi-line descriptions are only likely to
show up from importing content from other tools, in which case
replacing newlines with spaces is cleaner than the alternative.
2019-02-20 12:28:00 -08:00
Rohitt Vashishtha 44ec83ef28 markdown: Render silent mentions as **name**.
This change should help people discover to distinguish
silent mentions in text as a part of Zulip syntax while
differentiating them from regular mentions.
2019-02-20 10:41:42 -08:00
Rohitt Vashishtha 57b9991396 markdown: Change syntax of silent mentions ( _@person -> @_person). 2019-02-20 10:41:42 -08:00
Rishi Gupta e40731574a integrations: Fix missing logo for trello.
Fixes regression from 6f8481c.
2019-02-19 15:41:18 -08:00
synicalsyntax 2a5fe09d16 integrations: Update Opsgenie logo.
Converts logo from PNG to SVG format.
2019-02-19 15:07:24 -08:00
synicalsyntax 1595162496 integrations: Update Jira logo.
Converts logo from PNG to SVG format.
2019-02-19 15:07:24 -08:00
synicalsyntax 5bf89b946d integrations: Update Bitbucket logo.
Converts logo from PNG to SVG format.
2019-02-19 15:07:24 -08:00
synicalsyntax 6f8481cd33 integrations: Update Trello logo.
Converts logo from PNG to SVG format.
2019-02-19 15:07:24 -08:00
Raymond Akornor 4dc7f5354d tests: Optimize test-backend performance.
This optimizes test-backend by skipping webhook
tests when run in default mode.

Tweaked by tabbott to extend the documentation and update the CI
commands.
2019-02-19 14:50:41 -08:00
Eeshan Garg c78c3f423c webhooks/github: Ignore project_card events.
The payloads for this event are missing some important details
about the Project's changes, such as the name of the project,
the card's column name, etc. Without such details, the resultant
notifications would not be useful at all!
2019-02-19 14:25:38 -08:00
Eeshan Garg ec81410b03 webhooks/github: Ignore repository_vulnerability_alert event.
This event isn't incredibly common/useful and errors for this
event were cluttering up our webhook logs.
2019-02-19 18:01:05 -03:30
sahil839 7157edf4af settings: Add support for uploading logo for night mode.
This adds a new field named realm_night_logo which is used for
displaying the organization logo when the user is in night mode.

Fixes #11176.
2019-02-18 15:15:57 -08:00
Eeshan Garg 0500639d01 api_tests: Test zulip.Client.add_reaction. 2019-02-18 10:24:13 -08:00
Tim Abbott 7a49611a94 settings_invites: Display links for multi-use invitations.
This makes it possible to figure out which is which if you have
several active at the same time.
2019-02-15 14:28:07 -08:00
Vishnu Ks 763eca6ca9 invites: Add UI for revoking multiuse invites. 2019-02-15 14:13:31 -08:00
Vishnu Ks 410e2574d1 settings: Remove unused setting ACCOUNT_ACTIVATION_DAYS.
ACCOUNT_ACTIVATION_DAYS doesn't seems to be used anywhere.
INVITATION_LINK_VALIDITY_DAYS seems to do it's job currently.

(It was only ever used in very early Zulip commits).
2019-02-15 13:56:10 -08:00
Tim Abbott 0c0aec3cc9 export: Fix finding manage.py to export usermessages.
We were using a hardcoded relative path, which doesn't work if you're
not running this from the root of the Zulip checkout.

As part of fixing this, we need to make `LOCAL_UPLOADS_DIR` an
absolute path.

Fixes #11581.
2019-02-15 11:32:36 -08:00
Anders Kaseorg ce01a4c5a3 backup: Add Zulip, OS, and PostgreSQL version to the tarball.
==> zulip-backup/os-version <==
    Ubuntu trusty

    ==> zulip-backup/postgres-version <==
    90324

    ==> zulip-backup/zulip-version <==
    2.0.0-rc1+git
    2.0.0-rc1-50-gbf169b8-dirty

Documentation added by tabbott.

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2019-02-14 15:11:57 -08:00
Greg Price 0213aa0b16 push notif: Don't forget to clear "active" flag on sending to bouncer.
Since da8f4bc0e back in August, this control flow has caused
`flags.active_mobile_push_notification` to be cleared if we don't send
these `remove` messages at all, and if we send them directly to GCM...
but not if we send them via the Zulip notification bouncer.

As a result, on a server configured to send `remove` notification-messages
via the bouncer, we accumulate "active" messages and never clear them.

If the user then does `mark_all_as_read`, we end up sending a `remove`
for each of those messages again, and all in one giant burst.  We've
seen puzzling bursts of hundreds of removals pass through the bouncer
since turning on removals on chat.zulip.org; it's likely many of them
are caused by this bug.

This issue was made more acute with f4478aad5, which unconditionally
enabled removals.

Test added by tabbott.
2019-02-14 14:52:53 -08:00
Vishnu Ks 949d098e99 management: Add tooling for transfering local uploads to s3.
This solves a common migration problem for folks who cut corners when
first setting up Zulip.

Fixes #11294.
2019-02-14 14:31:46 -08:00
Vishnu Ks 6d138bd3e5 tests: Fix broken avatar_disk_path for original size input.
string.replace returns the new string and does
not change the original string.
2019-02-14 14:18:03 -08:00
Rishi Gupta e183c316dd help: Rename help/change-your-avatar to help/set-your-avatar. 2019-02-13 17:50:39 -08:00
Abdelhadi Dyouri 4ac2db56f8 export: Correctly treat emoji author field as optional.
While we likely will eventually want to make every custom emoji have
an author, that's not the data model today.

Fixes #11518.
2019-02-13 16:12:06 -08:00
Greg Price 630481cb7a push notif: Switch from GCM to FCM endpoint.
This is the only server-side change required for the FCM migration!

Optionally, at some point in the future we might choose to migrate
to the new ("v1") API which FCM also offers.  Nothing revolutionary
but there are some nice things about it:
  https://firebase.google.com/docs/cloud-messaging/migrate-v1
2019-02-13 13:57:57 -08:00
Greg Price 84e0b68b16 push notif: Rename `gcm` to less confusing `gcm_client`.
This opens up space in the namespace for, say, the library
module itself.
2019-02-13 13:57:57 -08:00
Tim Abbott 7099d01641 email mirror: Fix missing variable for logging. 2019-02-13 13:16:55 -08:00
Greg Price f4478aad54 push notif: Unconditionally remove notifications on message read.
The client-side fix to make these not a problem was in release
16.2.96, of 2018-08-22.  We've been sending them from the
development community server chat.zulip.org since 2018-11-29.
We started forcing clients to upgrade with commit fb7bfbe9a,
deployed 2018-12-05 to zulipchat.com.

(The mobile app unconditionally makes a request to a route on
zulipchat.com to check for this kind of forced upgrade, so that
applies to mobile users of any Zulip server.)

So at this point it's long past safe for us to unconditionally
send these.  Hardwire the old `SEND_REMOVE_PUSH_NOTIFICATIONS`
setting to True, and simplify it out.
2019-02-13 13:13:45 -08:00
savish 1b334b906a tests: Use subTest in test_admin_user_can_change_profile_data.
Fixes #11070.
2019-02-13 10:50:39 -08:00
savish aa605468b6 tests: Use subTest in test_update_realm_properties. 2019-02-13 10:49:13 -08:00
savish 9c292fbeab tests: Use subTest in test_bugdown_fixtures. 2019-02-13 10:49:13 -08:00
Rishi Gupta 378d14af7e import: Fix tests related to plan_type.
Broken in 4d08461.
2019-02-12 18:11:31 -08:00
Tim Abbott 12d5e870c5 tests: Fix import test failure.
Broken in 4d08461ab1.
2019-02-12 17:46:55 -08:00
Tim Abbott 4d08461ab1 import: Set plan_type to SELF_HOSTED on import.
We've for a while had logic to set plan_type to LIMITED when importing
into Zulip Cloud; we need corresponding logic to set it to SELF_HOSTED
when importing into a self-hosted server.

Fixes #11541.
2019-02-12 16:01:02 -08:00
Vishnu Ks 868a763cec auth2: Don't use session for passing multiuse invite key.
For Google auth, the multiuse invite key should be stored in the
csrf_state sent to google along with other values like is_signup,
mobile_flow_otp.

For social auth, the multiuse invite key should be passed as params to
the social-auth backend. The passing of the key is handled by
social_auth pipeline and made available to us when the auth is
completed.
2019-02-12 15:51:11 -08:00
Eeshan Garg 179b747769 streams: Refactor multi-option helpers into separate functions.
For internal stream messages, most of the time, we have access to
a Stream object. For the few corner cases where we don't, it is a
much cleaner approach to have a separate function that accepts a
stream name than having one multi-option helper that accepts both
names and objects.
2019-02-12 11:10:26 -08:00
Eeshan Garg 1d718adce4 zerver/views: Rename integrations.py to documentation.py.
The name "integrations" is misleading because we now also use
this module for our REST API docs.
2019-02-11 21:25:14 -08:00
Eeshan Garg b0c761dfb1 zapier_app: Include bot details in response to auth requests.
The bot's details (such as ID and name) are used to format a
connection label in Zapier's UI. This allows users to distinguish
between different Zapier-to-Zulip connections set up with different
bots.
2019-02-11 21:24:52 -08:00
Tim Abbott d6140b684f notifications: Don't send error emails on bouncer 500s.
Since the individual server administrator can't do anything about
these, this should not trigger an email notification.
2019-02-11 21:19:28 -08:00
Anders Kaseorg e0a51948d9 script: Add ready-to-run tooling for doing backups.
Based on an initial version by Tim Abbott (#11204).

Fixes #552.
2019-02-11 17:30:37 -08:00
Rishi Gupta 028874bab3 open graph: Remove extraneous spaces from descriptions.
Our html collects extra spaces in a couple of places. The most prominent is
paragraphs that look like the following in the .md file:
* some text
  continued

The html will have two spaces before "continued".
2019-02-11 12:05:19 -08:00
Rishi Gupta 0119b383a6 help: Update Accounts basics section to conform to new tabbed style. 2019-02-11 12:05:19 -08:00
Rishi Gupta d3125f59e1 open graph: Omit .code-section navigation from open graph. 2019-02-11 12:05:19 -08:00
Rishi Gupta e1f02dc6f2 open graph: Include multiple paragraphs in description tags. 2019-02-11 12:05:19 -08:00
Rishi Gupta 83236dc283 help: Add tab styling to untabbed instructions as well.
This changes the border-radius to 6px for the tabbed display, which is not
in line with the current Zulip style for border-radius (4px). However 6px
really looks a lot better for this (possibly because it's a bigger box than
most of our other boxes?)
2019-02-11 12:05:19 -08:00
anhhuynh bd548eb7f3 emails: Remove newlines from translated strings in email templates.
Fixes #11390.
2019-02-10 12:28:17 -08:00
Greg Price a6c2c16666 push notif tests: Clean up how we set up device token fixtures.
I was hoping this would make things faster... it does, but sadly only
by about 70ms, 5% of this file's test runtime.

It sure does make this file rather less action-at-a-distance, though,
as well as fixing some duplication.
2019-02-08 15:18:12 -08:00
Greg Price ccc1f3cd85 push notif tests: Stop importing module as bizarre name `apn`. 2019-02-08 15:18:12 -08:00
Greg Price c372ceb7a2 push notif tests: Use mock.patch instead of `apn.gcm = ...`. 2019-02-08 15:18:12 -08:00
Greg Price e289c5835d push notif tests: Use mock.patch instead of `apn.gcm = None`.
This is what it's for -- and it cleans up after itself, too.
2019-02-08 15:18:12 -08:00
Greg Price ff12072f18 push notif: Skip four layers of setup for some pure unit tests.
This saves about 50ms, for just four test cases.
2019-02-08 15:18:12 -08:00
Greg Price 1e11e929ec push notif: Guess a GCM `priority` on behalf of old servers. 2019-02-08 15:18:12 -08:00
Greg Price a293aeee23 push notif: Explicitly set GCM priority `normal` for remove.
If we make a practice on the Zulip server of always explicitly setting
the desired priority, then when an old server doesn't set the priority
we can reasonably have the bouncer make a guess.
2019-02-08 15:18:12 -08:00
Greg Price ffabebd7f3 push notif: Set GCM priority `high` for real notifications.
This is the payoff of this branch!  Fixes zulip/zulip-mobile#3185.
2019-02-08 15:18:12 -08:00
Greg Price 699bf262ca push notif: Test GCM options parsing more comprehensively.
The payoff from making these into real unit tests.
2019-02-08 15:18:12 -08:00
Greg Price 02ba302009 push notif: Simplify tests for parsing GCM options.
Huzzah for unit tests!
2019-02-08 15:18:12 -08:00
Greg Price f1cb9be79c push notif: Consolidate and simplify GCM tests. 2019-02-08 15:18:12 -08:00
Greg Price 6da9aeee7d push notif: Expand and clarify docs on GCM options. 2019-02-08 09:44:00 -08:00
Greg Price 674b254b65 push notif: Accept GCM `priority` option.
That is, this allows a Zulip server to now set the `priority`; but if
it doesn't, we use upstream's default value, which has the same effect
as we've always previously had by not setting it at all.

But when this is deployed to the push notifications bouncer server, it
does allow another server to set priority when pushing notifications
through the bouncer.
2019-02-08 09:42:59 -08:00
Greg Price 575f320009 push notif: Factor out a helper parse_gcm_options.
Looks a bit silly as is, but this will give us a good home for
logic to start parsing more than zero allowed options.
2019-02-08 09:41:07 -08:00
Greg Price 9ebad5d505 push notif: Reorg `send_android_...` slightly for clarity.
In particular, get the `remote`/`DeviceTokenClass` bit out from
in between `json_request` and the various bits of code preparing
for it.
2019-02-08 09:40:45 -08:00
Greg Price 49fd2e65de push notif: Add GCM options to bouncer API; empty for now.
The first use case for this will be setting `priority`,
coming up shortly.
2019-02-08 09:40:43 -08:00
Eeshan Garg 3470e541c8 internal_send_stream_message: Support accepting a Stream object.
If the caller has access to a Stream object, it is wasteful to
query a database for a stream by ID or name. In addition, not
having to go through stream names eliminates various classes of
possible bugs involved with re-fetching the Stream object by name.
2019-02-08 08:59:03 -08:00
Eeshan Garg c240008edb internal_prep_stream_message: Support accepting a Stream object.
If the caller has access to a Stream object, it is wasteful to
query a database for a stream by ID or name. In addition, not
having to go through stream names eliminates various classes of
possible bugs involved with getting a Stream object back.
2019-02-08 08:53:16 -08:00
Tim Abbott 4826393ec4 hipchat: Fix help documentation for slim_mode. 2019-02-07 16:17:01 -08:00
Vishnu Ks 28769e040f invites: Add backend for multiuse admin invites. 2019-02-07 15:41:00 -08:00
Eeshan Garg 4da28970fa addressee: Rename for_stream to for_stream_name.
The name for_stream_name is more appropriate here.  The name
for_stream is more suitable for a function that takes in a Stream
object, which we're about to add.
2019-02-07 13:41:24 -08:00
Tim Abbott 970c7d1ddf notifications: Fix hash-named duplicate sound files in options.
Our hash-naming of production assets interacted badly with the "look
at files in a directory" algorithm used to determine what sound
options exist for the "notification sound" feature.  For lack of a
better solution, we fix this by excluding files with an extra `.` in
their name.
2019-02-07 13:36:41 -08:00
Eeshan Garg 361d2d0688 webhooks/stripe: Add fixture and test for invoiceitem.created. 2019-02-05 18:40:38 -03:30
Tim Abbott 11ea8ae9fe tests: Fix test broken by recent url preview change. 2019-02-05 13:45:00 -08:00