Commit Graph

8674 Commits

Author SHA1 Message Date
Rishi Gupta d76f728dda api docs: Rename webhook-walkthrough to incoming-webhooks-walkthrough. 2018-10-16 21:07:00 -07:00
Tim Abbott 78a15dd715 slack import: Fix obscure email address for Slackbot.
Since we know what slackbot is, we don't need to give it a crazy hash
as its email address.
2018-10-16 16:33:41 -07:00
Aditya Bansal 8324e2c976 thumbnails: Return original path if url is not supposed to be thumbnailed. 2018-10-16 16:00:47 -07:00
Aditya Bansal 5d68bd92ad thumbnails: Extract user_uploads_or_external as a function.
This is a prepartory commit for the upcoming changes. It was meaningful
to extract this one out because this function is essentially a condition
check on whether a given url is one of the user_uploads or an external
one. Based on its value we decide whether a url must be thumbnailed or
not and thus this function will also be used in an upcoming commit
patching lib/thumbnail.py to do the same check before thumbnail url
generation.
2018-10-16 16:00:47 -07:00
Aditya Bansal 9a411a5765 thumbnails: Stop thumbnailing urls other than external or user_uploads.
We are basically adding a check for url's to be external (belonging
to some 3rd party web site hosting the image) or be one of the
user uploaded files. User uploaded files are served by a separate
endpoint which is /user_uploads/. Any other local url such as
/user_avatars/ or /static/ should never be sent to thumbor for
thumbnailing.
Not sending /user_avatars/ to thumbor for thumbnailing makes sense
because they are already properly thumbnailed and stored properly.
/static/ urls host very few images we use for demo and can be safely
be excluded from thumbnailing.
2018-10-16 16:00:47 -07:00
Tim Abbott e4813e462b tornado: Rename async_request_{restart,stop} to mention timer.
Previously, these timer accounting functions could be easily mistaken
for referring to starting/stopping the request.  By adding timer to
the name, we make the code easier for the casual observer to read and
understand.
2018-10-16 15:39:10 -07:00
Eeshan Garg 0292ffe04b webhooks/taiga: Add test for when a task is moved between stories.
This commit adds a test for the payload that is generated when
a Task is moved from one user story to another on Taiga's Sprint
Taskboard UI.

This commit also gets up this webhook's test coverage up to 100%.
2018-10-16 13:06:20 -07:00
Eeshan Garg 48fe5ae5b2 webhooks/taiga: Stop catching KeyErrors for 'change' events.
I generated multiple payloads and verified that there are no
`change` event payloads that will not contain the values in
question, so it is useless to catch these KeyErrors. If there are
any anomalies still, it is better to be notified about them than
to silently ignore them.
2018-10-16 13:06:20 -07:00
Yago González 029d815be7 api docs: Document the /dev_fetch_api_key endpoint. 2018-10-16 12:47:12 -07:00
Yago González 3eeec94d03 api docs: Document the POST /users/me/subscriptions/properties endpoint. 2018-10-16 12:38:27 -07:00
Tim Abbott 8481a2fd2d api docs: Fix confusing discussion of "dev server".
The Zulip API is to be used on both development and production
servers, and really we just need to talk about zuliprc files.

There's a similar issue for the JS docs, but we need to fix the
copy/paste issues with those as well.
2018-10-16 12:23:23 -07:00
Yago González 3d2b3f8fa4 api docs: Document the POST /messages/flags endpoint. 2018-10-16 12:19:24 -07:00
Yago González 7a4103eee6 api docs: Document POST /mark_(all|stream|topic)_as_read. 2018-10-16 12:12:05 -07:00
Tim Abbott d13bd6e8d3 linkifiers: Rename articles to current feature name.
These were previously called "linkification filters", but we now want
to consistently refer to them as "linkifiers" in the UI and API docs.
2018-10-16 12:01:17 -07:00
Yago González 3bdc8f9946 api: Document the GET /realm/emoji endpoint. 2018-10-16 11:51:48 -07:00
Yago González 7a7b507e86 api docs: Document DELETE /realm/filters/<filter_id>. 2018-10-16 11:51:48 -07:00
Yago González a7c48acc8e api docs: Document GET /realm/filters. 2018-10-16 11:51:48 -07:00
Vishnu Ks 14a6a8db1a logs: Mock out warning logs in test_scrub_realm. 2018-10-16 11:16:47 -07:00
Vishnu Ks 57e0a24020 management: Scrub delivery email as well while running scrub_realm. 2018-10-16 11:16:46 -07:00
Kevin Lin b2c29274f3 presence: Allow bots to fetch realm presence data.
Before, presence information for an entire realm could only be queried via
the `POST /api/v1/users/me/presence` endpoint. However, this endpoint also
updates the presence information for the user making the request. Therefore,
bot users are not allowed to access this endpoint because they don't have
any presence data.

This commit adds a new endpoint `GET /api/v1/realm/presence` that just
returns the presence information for the realm of the caller.

Fixes #10651.
2018-10-15 12:51:28 -07:00
Steve Howell b1dd9a251b hipchat import: Break messages into smaller batches.
Even individual "room" files from hipchat can be large,
so we process only 1000 messages at a time
within each file, which produces smaller JSON files.
2018-10-15 10:54:23 -07:00
Steve Howell 6650bb2240 minor: Move fix_mentions() closer to caller. 2018-10-15 10:54:23 -07:00
Steve Howell 219ff0f749 hipchat import: Extract UserHandler class. 2018-10-15 10:54:23 -07:00
Steve Howell 2d523fd668 hipchat import: Extract make_user_messages(). 2018-10-15 10:54:23 -07:00
Steve Howell ca0495cbe6 hipchat import: Support attachments. 2018-10-15 10:54:23 -07:00
Steve Howell 2d4b09f59d utils: Add process_list_in_batches(). 2018-10-15 10:54:23 -07:00
Steve Howell d71f3eb1bf hipchat import: Add some more logging. 2018-10-14 09:29:04 -07:00
Steve Howell 76deb30312 preview: Hash cache keys for preview urls.
We don't want really long urls to lead to truncated
keys, or we could theoretically have two different
urls get mixed up previews.

Also, this suppresses warnings about exceeding the
250 char limit.

Finally, this gives the key a proper prefix.
2018-10-14 09:28:57 -07:00
Steve Howell d933779477 hipchat import: Support PrivateUserMessage data.
We now import PM data from HipChat.
2018-10-13 16:47:44 -07:00
Steve Howell f0c3ee0a2e hipchat import: Write smaller message files.
We now write new message files for each new input
file + message type we process.  This helps the
importer not run out of memory later.
2018-10-13 16:47:44 -07:00
Steve Howell 75fc5d41c9 hipchat import: Refactor write_message_data.
The goal here is to make it easier to handle other
message types by moving the key-specific stuff
to the top of the file.
2018-10-13 16:47:44 -07:00
Steve Howell cc55eb8154 hipchat import: Only process UserMessage rows for now. 2018-10-13 16:47:44 -07:00
Steve Howell 3baac7ddf3 hipchat import: Handle missing emails for guest users. 2018-10-13 16:47:44 -07:00
Steve Howell 8accc60ca7 import_util: Support multiple message ids for attachments. 2018-10-13 16:47:44 -07:00
Steve Howell 23d7b3d2cc import: De-dup create_converted_data_files helper. 2018-10-13 16:47:41 -07:00
Steve Howell 91905bd66a import: Add sequencer library.
This avoids some tedious code related to making ids
in conversion programs.
2018-10-13 16:47:39 -07:00
Steve Howell 85f1910f93 minor: Add link to hipchat spec to code. 2018-10-13 16:43:28 -07:00
Steve Howell 493aae2958 imports: Make loading UserMessage faster and more robust.
We use UserMessageLite to avoid Django overhead, and we
do updates in chunks of 10000.  (The export may be broken
into several files already, but a reasonable chunking at
import time is good defense against running out of memory.)
2018-10-13 16:43:28 -07:00
Tim Abbott 68ab71eb8b push: Fix exceptions when removing push notifications.
Now that we allow multiple users to have registered the same token, we
need to configure calls to unregister tokens to only query the
targeted user_id.

We conveniently were already passing the `user_id` into the push
notification bouncer for the remove API, so no migration for older
Zulip servers is required.
2018-10-12 11:19:23 -07:00
Steve Howell 0f7628280f narrow: Handle spurious emails in pm-with searches.
If cordelia searches on pm-with:iago@zulip.com,cordelia@zulip.com,
we now properly treat that the same way as pm-with:iago@zulip.com.

Before this fix, the query would initially go through the
huddle code path.  The symptom wasn't completely obvious, as
eventually a deeper function would return a recipient id
corresponding to a single PM with @iago@zulip.com, but we would
only get messages where iago was the recipient, and not any
messages where he was the sender to cordelia.

I put the helper function for this in zerver/lib/addressee, which
is somewhat speculative.  Eventually, we'll want pm-with queries
to allow for user ids, and I imagine there will be some shared
logic with other Addressee code in terms of how we handle these
strings.  The way we deal with lists of emails/users for various
endpoints is kind of haphazard in the current code, although
granted it's mostly just repeating the same simple patterns.  It
would be nice for some of this code to converge a bit.  This
affects new messages, typing indicators, search filters, etc.,
and some endpoints have strange legacy stuff like supporting
JSON-encoded lists, so it's not trivial to clean this up.

Tweaked by tabbott to add some additional tests.
2018-10-12 10:18:30 -07:00
Steve Howell 9f2aad55b5 hipchat import: Handle users without avatars. 2018-10-12 07:03:25 -04:00
Steve Howell 51bd36e448 tests: Add coverage to get_service_interface_class(). 2018-10-11 16:12:07 -07:00
Steve Howell 8379aeee15 outgoing bots: Fix header for generic servers.
For our bots that use GenericOutgoingWebhookService
(which are basically Zulip style bots), we now
include a "content-type" header of "application/json".

We accomplish this by having the service classes
implement their own custom method called
`send_data_to_server`. For the Slack-related
code, we just extracted code from `do_rest_call`,
and then for the Zulip-related code, we added
a `headers` parameter.
2018-10-11 16:12:07 -07:00
Steve Howell 8226e13e9c bot tests: Replace use of MockServiceHandler.
We'll just use a real class here, since the service
handlers are pretty lightweight and just munge data.
2018-10-11 16:12:07 -07:00
Steve Howell 8f74d99b6c Remove stubs in OutgoingWebhookServiceInterface.
If we omit methods in subclasses, they're likely to
be caught by linters or unit tests, and even if they
aren't, raising NotImplementedError doesn't actually
prevent user problems.

I've been fighting these in refactoring, and it's
just been a bunch of busy work, plus comments are
highly likely to bitrot.
2018-10-11 16:12:07 -07:00
Steve Howell 31597cf33e Remove timeout parameter in do_rest_call().
Nobody was setting it.
2018-10-11 16:12:07 -07:00
Steve Howell 69ee84bb14 refactor: Extract build_bot_request().
This fixes a couple things:

    * process_event() is a pretty vague name
    * returning tuples should generally be avoided
    * we were producing the same REST parameters in both
      subclasses
    * relative_url_path was always blank
    * request_kwargs was always empty

Now process_event() is called build_bot_request(),
and it only returns request data,
not a tuple of `rest_operation` and `request_data`.

By no longer returning `rest_operation`, there are
fewer moving parts.  We just have `do_rest_call` make
a POST call.
2018-10-11 16:12:07 -07:00
Steve Howell 16eff75e49 refactor: Simplify how we use base_url.
Before this change, we instantiated base_url into a superclass
of subclasses that returned base_url into a dictionary that
gets returned to our caller.

Now we just pull base_url out of service when we need to make
the REST call.
2018-10-11 16:12:07 -07:00
Steve Howell b89a94f730 Improve errors when we can't connect to a bot server.
We don't overwhelm people with error info when bots
fail to connect or time out.
2018-10-11 16:12:07 -07:00
Steve Howell 3790c469e9 outgoing bots: Report JSON errors to users.
We should arguably report these to bot owners
as well, but this is at least an improvement
over having the server crash.
2018-10-11 16:12:07 -07:00