Commit Graph

9037 Commits

Author SHA1 Message Date
Steve Howell 1854ddbcd0 taiga tests: Use self.TOPIC instead of "subject". 2018-11-12 15:47:11 -08:00
Steve Howell 31e328c544 webhook tests: Replace SUBJECT with TOPIC. 2018-11-12 15:47:11 -08:00
Steve Howell 1fff890b71 webhooks: Change *_SUBJECT_TEMPLATE to *_TOPIC_TEMPLATE. 2018-11-12 15:47:11 -08:00
Steve Howell ea98a44db3 webhooks: Replace SUBJECT_WITH_* with TOPIC_WITH_*. 2018-11-12 15:47:11 -08:00
Steve Howell ced4d81856 Sweep tests for expected_subject -> expected_topic.
This is all in the webhooks tests, including some
docs for how to write those tests.
2018-11-12 15:47:11 -08:00
Steve Howell 3133998af9 test_messages: Fix a lot of subject -> topic_name. 2018-11-12 15:47:11 -08:00
Steve Howell e55fc144b9 Add alias support to REQ helpers for subject/topic.
The alias support is generic here, and we use it for
migrating subject -> topic in REQ_topic().
2018-11-12 15:47:11 -08:00
Steve Howell de6c08a842 Extract messages_for_topic(). 2018-11-12 15:47:11 -08:00
Steve Howell b667dff4bc minor: Change wording in comment to avoid "subject".
We're trying to sweep "subject" out of the codebase,
even when it has nothing to do our legacy "subject"
field.  The rewording here will prevent some linter
noise.
2018-11-12 15:47:11 -08:00
Steve Howell c7515a853c Use REQ_topic in update_message_backend().
Note that we now strip the topic in the converter.
We also strip it in the view handler, which is now
redundant, but I left it in as defensive code.
2018-11-12 15:47:11 -08:00
Steve Howell 3de1289d3e Rename subject -> topic_name in update_message_backend(). 2018-11-12 15:47:11 -08:00
Steve Howell 754795c0b7 Extract REQ_topic() helper.
For now we use it only in the send path, not the update
path.
2018-11-12 15:47:11 -08:00
Steve Howell 0a171bf9bf Fix "prev_subject" with LEGACY_PREV_TOPIC.
I added `LEGACY_` prefix to the var, because
otherwise you'd have really confusing code
where you change the key from `PREV_TOPIC`
to "prev_topic".
2018-11-12 15:47:11 -08:00
Steve Howell a49ba8c577 Extract user_message_exists_for_topic(). 2018-11-12 15:47:11 -08:00
Steve Howell 39e5c61d4a Extact MATCH_TOPIC constant. 2018-11-12 15:47:11 -08:00
Steve Howell 500fcbb5e3 messages: Update subject -> topic_name.
This fixes several local vars named "subject".
2018-11-12 15:47:11 -08:00
Steve Howell f7f071b52f minor: Fix subject -> topic in a comment. 2018-11-12 15:47:11 -08:00
Steve Howell 8981b65bbc Extract topic_column_sa(). 2018-11-12 15:47:11 -08:00
Steve Howell b46fb5eaad Rename subject_matches -> topic_matches.
This is always either a var or a SQL label,
so it's safe to change.
2018-11-12 15:47:11 -08:00
Steve Howell bb959acbaf Use modern huddle URLs in digest emails. 2018-11-12 12:01:09 -08:00
Steve Howell e79e0018f7 Use modern PM URLs in digest emails. 2018-11-12 12:01:09 -08:00
Steve Howell 6a89446e80 tests: Add test for huddle digest urls. 2018-11-12 12:01:09 -08:00
Steve Howell c08a3833e2 tests: Start to check urls in digest emails. 2018-11-12 12:01:09 -08:00
Roman Godov aa3682f9fc api: Add GET /user_groups documentation.
This documents our GET endpoint for interacting with Zulip's user
groups feature.

Fixes #10298.
2018-11-08 22:19:30 -08:00
Tim Abbott e88998e6d4 import: Fix buggy handling of avatars in Slack conversion.
This was a pretty nasty error, where we were accidentally accessing
the parent list in this inner loop function.

This appears to have been introduced as a refactoring bug in
7822ef38c2.
2018-11-08 15:03:39 -08:00
Roger Souza 6179bba2bd emails: Add one-click unsubscribe link to new login emails.
Fixes: #10547.
2018-11-08 14:39:33 -08:00
Tim Abbott 17be6e9165 confirmation: Move one_click_unsubscribe_link out of notifications.py.
This helps prevent a problematic import sequence when we start using
it from signals.py.
2018-11-08 14:39:33 -08:00
Steve Howell cd4d447d90 Minor: change message.subject -> message.topic_name().
This also allows us to lock down
push_notifications.py from using "subject"
in the future.
2018-11-08 16:34:30 +00:00
Steve Howell ff9a6c5ced minor: Rename subject -> topic_name in bugdown. 2018-11-08 16:21:14 +00:00
Steve Howell e802be0fde Remove "subject" in notifications.py.
These last few are all pretty safe.
2018-11-08 15:12:10 +00:00
Steve Howell 39dc5f2691 Rename recipient_subject to bucket.
The various vars here that had recipient_subject
in the name now have either bucket or bucket_tup
there.

The shorter names are a bit easier to read, and the
original names were misleading for the PM case.

This was basically two search/replaces, and we have
good test coverage here, so it's pretty low risk
despite the messy diff.
2018-11-08 14:59:09 +00:00
Steve Howell 226a2826b5 Remove "subject" from zerver/lib/message.py.
We also lock this down file in the linter, so
that you can no longer introduce "subject"
to `message.py`.
2018-11-08 14:45:16 +00:00
Steve Howell 592dd290b2 Avoid "subject" in MessageDict. 2018-11-08 14:33:44 +00:00
Steve Howell 881e046c37 minor: Use TOPIC_NAME constant for "subject" event. 2018-11-08 14:01:45 +00:00
Steve Howell d05f731c1c Eliminate the use of arguments.db_data.
We now attach zulip_db_data to the markdown engines
for classes that need it.  This was the last remaining
global we had, so we remove `arguments.py` here.
2018-11-07 10:44:49 -08:00
Steve Howell b66304e167 refactor: Pass db_data down to helpers.
This mostly preps for the next commit.
2018-11-07 10:44:49 -08:00
Steve Howell fa6f642c9c refactor: Remove global argument.current_realm. 2018-11-07 10:44:48 -08:00
Steve Howell e1113c7011 refactor: Remove the global arguments.current_message.
The Markdown processor makes it fairly simple for
the helper classes to access the `md` engine.  We
now write `_md_engine.zulip_message` to avoid having
the current message in the global namespace.

Note that we do reuse engines for multiple messages,
but each engine is specific to a realm.  And we therefore
avoid even the theoretical possibility of leaking message
data between realms.
2018-11-07 10:44:48 -08:00
Steve Howell ab24cc2535 minor: Pass in arguments.current_message to helpers. 2018-11-07 10:44:48 -08:00
Steve Howell c26768ea63 bugdown: Import nl2br and tables extensions "normally".
This makes us consistent with how we import codehilite.

Using Python's normal import mechanism avoids some overhead
with Markdown having to parse dotted notation.

These modules are tiny, so they shouldn't impact startup
too much.  Also, by explicitly importing them, we avoid
the pitfall of having a sucessful startup and a broken
renderer.
2018-11-07 10:44:48 -08:00
Steve Howell c8a2081526 bugdown: Break out helper functions for extending bugdown.
These will make profiling a lot easier, and you
can also quickly disable features.  The overhead
of these function calls is dwarfed by other concerns.
2018-11-07 10:44:47 -08:00
Steve Howell ffa4daf936 bugdown: Reduce overhead of building link regexes.
We were building the same link regex every time
we build a Markdown engine, which happens twice
per realm.  It's an expensive operation due to
the complexity of the regex and us reading a file.
2018-11-07 10:33:11 -08:00
Steve Howell 18a76c54de bugdown: Extract build_engine.
This separates out the main job of building
an instance of Markdown from the fairly orthogonal
task of maintaining a list of engines.
2018-11-07 10:33:11 -08:00
Steve Howell eb2269c614 fenced_code: Refactor to avoid nested classes.
Nested classes are kind of expensive in Python,
particularly when you throw in mypy annotations.

Also, flatter is arguably better, although it is
kind of a pain here not to have closures.
2018-11-07 10:33:11 -08:00
Steve Howell dfadbcd3bc bugdown: Avoid ORM when there are no group names.
This change avoids hitting the Django ORM when
we don't find any possible group mentions in
the message content.

Django doesn't necessarily actually hit the database,
but it's still slow and shows up in profiles.
2018-11-07 10:33:11 -08:00
Steve Howell 88f50b97fd import: Render content before inserting messages.
By rendering content before bulk importing messages,
we avoid O(N) database hops.
2018-11-07 10:33:11 -08:00
Steve Howell bf3f7d93d0 Simplify params for fix_message_rendered_content. 2018-11-07 10:33:11 -08:00
Steve Howell 0878d86706 import: Avoid unnecessary Message lookups.
We now no longer go the DB to get a Message object
during render.
2018-11-07 10:33:11 -08:00
Steve Howell 1e12b13a56 import: Avoid unnecessary sender lookups.
This commit speeds up the import by avoiding
sender lookups and instead using the data
for users that we already have in memory.

This avoids a few DB hops, many hops to memcached,
plus some object construction.

We now call do_render_markdown() directly.  This
also makes it more explicit that the import has
never rendered alert words.
2018-11-07 10:33:10 -08:00
Steve Howell 06f8d0af7e refactor: Extract do_render_markdown().
For the import-data codepath, we will call
the extracted function directly in a
subsequent commit.

The do_render_markdown() function has more
required parameters, which allows for more
explicit code and also allows us to flatten
out some logic related to alert words.  (We
just pass in empty sets/dicts as needed).
2018-11-07 10:20:14 -08:00
Steve Howell 35e9e5928f render: Upstream calculation of translate_emoticons. 2018-11-07 10:20:14 -08:00
Steve Howell 82b808e620 bugdown: Avoid zephyr-related queries in rendering.
We can rely on `message_realm` being the same
as `message.sender.realm`, which allows us to
skip two queries to the database for the rare
Zephyr mirroring case.
2018-11-07 10:11:06 -08:00
Steve Howell 659c9dde00 bugdown: Avoid unnecessary realm queries.
We now keep realm in the arguments variable,
which avoids some lookups.

We also test settings before even trying to
get realms.
2018-11-07 10:08:46 -08:00
Steve Howell f9a7451167 import: Pass in realm to render codepath.
We avoid querying the same realm multiple times.
2018-11-07 10:08:46 -08:00
Steve Howell 92a7f04149 import: Inline save_message_rendered_content().
This function requires a message object, whereas
we want to work with JSON data to avoid necessary
queries when we import data.  Inlining the function
sets us up for a subsequent refactoring.

We change the way we deal with theoretical return
values of `None` to use an assertion; otherwise,
we would have to loosen up a bunch of mypy types
from `str` to `Optional[str]`.  It's not clear `None`
is even possible--we've moved toward throwing exceptions
there instead of silently failing.
2018-11-07 10:08:45 -08:00
Steve Howell 6b6001c46c Remove "subject" from test_classes.py.
We now use "topic" lingo.
2018-11-07 10:03:53 -08:00
Steve Howell ff60055fa4 Use topic_match_sa() for topic searches.
Note this introduce literal(), which makes the way
we handle topic mutes more consistent with general
topic searches.
2018-11-07 10:03:53 -08:00
Steve Howell 79d5e36ca3 Extract topic_match_sa() helper.
We'll also use this in zerver/views/messages.py, but
that's a bigger change.
2018-11-07 10:03:53 -08:00
Steve Howell af1acf9239 Rename constant to MAX_TOPIC_NAME_LENGTH. 2018-11-07 10:03:53 -08:00
Steve Howell 2cf46f0122 Extract save_message_for_edit_use_case().
This is mostly extracted to help clean out
all "subject" references from actions.py.
2018-11-07 10:03:53 -08:00
Steve Howell 0cf4cddc5b Extract update_messages_for_topic_edit().
This is somewhat hairy logic, so it's nice
to extract it and not worry about variable leaks.

Also, this moves some legacy "subject" references out
of actions.py.
2018-11-07 10:03:53 -08:00
Steve Howell 32232377f7 Rename bugdown.subject_links -> topic_links. 2018-11-07 10:03:53 -08:00
Steve Howell 0e854288ee Add some string constants to topic.py.
The goal here is to make it easy to
forbid "subject" in actions.py.
2018-11-07 10:03:53 -08:00
Steve Howell 7252861785 Extract filter_by_topic_name_via_message. 2018-11-07 10:03:53 -08:00
Steve Howell 2fd0cfe708 Use topic_name() helper in more places. 2018-11-07 10:03:53 -08:00
Steve Howell 55afadc286 Introduce Message.set_topic_name(). 2018-11-07 10:03:53 -08:00
Steve Howell dc8f893494 Extract filter_by_exact_message_topic(). 2018-11-07 10:03:53 -08:00
Steve Howell 56ecd227ff Add topic-related helpers for ScheduledMessage. 2018-11-07 10:03:53 -08:00
Steve Howell 50e3f85557 Extract topic.py library.
We start by including functions that do custom
queries for topic history.

The goal of this library is partly to quarantine
the legacy "subject" column on Message.
2018-11-07 10:03:53 -08:00
Rohitt Vashishtha 652477daea bugdown: Refactor name to mention_text in mention.py.
This commit renames some variables to make it clear that a mention's
text is not only the name of the user in all cases.
2018-11-07 10:02:53 -08:00
Eeshan Garg f9d867e138 webhooks/jira: Test double-escaped stream names with spaces.
A recent change to check_send_webhook_message allows webhooks to
unescape stream names before sending a message. This commit adds
a test for the edge case where the webhook URL is escaped twice by
a third-party.
2018-11-07 09:55:48 -08:00
Tim Abbott e14a35b490 import: Don't assume a last_modified key is present.
This fixes an exception when importing uploaded file data from
Slack/HipChat.
2018-11-07 09:52:35 -08:00
Eeshan Garg 006e47198c webhooks/jira: Decode stream name on our end.
Recently, one of our users reported that a JIRA webhook was not
able to send messages to a stream with a space character in its
name. Turns out that JIRA does something weird with webhook URLs,
such that escaped space characters (%20) are escaped again, so
that when the request gets to Zulip, the double escaped %20 is
evaluated as the literal characters `%20`, and not as a space.

We fix this by unescaping the stream name on our end before
sending the message forward!
2018-11-06 15:57:04 -08:00
Tim Abbott 6ef64fc752 narrow: Throw a clear error for invalid id: narrows.
This fixes an issue where searching for `id:` (with no operand) would
cause the server to 500, rather than returning a user-facing error.
2018-11-06 15:53:02 -08:00
Tim Abbott 8b661f2f03 slack import: Correctly detect the commenting user.
Fixes #10772.
2018-11-06 13:14:23 -08:00
Tim Abbott 1bf385e35f import: Avoid sending a content-type of None to S3.
The previous logic was incorrect, in that if `content_type` was set to
None (which happens with Slack/HipChat export, among other things),
then we wouldn't run the `guess_type` logic to auto-detect the
Content-Type to send to S3.
2018-11-06 13:03:14 -08:00
Tim Abbott 81a4c846f4 hipchat: Set s3_path for exported emoji.
This fixes an issue where the import process would fail when importing
to a server using the S3 backend.
2018-11-06 13:02:04 -08:00
Tim Abbott 539e84e9a1 hipchat import: Stop setting last_modified=None.
The last_modified field is intended to support setting the
orig-last-modified field in the S3 backend when importing, basically
to keep track of this bit of pre-export data for debugging.  In the
event that it isn't available, the correct thing to do is not write
out an invalid `last_modified` field; we should just not write it out
at all.
2018-11-06 12:50:36 -08:00
sameerchoubey 996caddb11 webhooks/taiga: Stop using emojis in notifications.
This fixes the fact that these emoji were sometimes not displaying
properly (because of changes in the emoji names used in the codebase),
while also making this integration more standard (since it was the
only one with such an aggressive use of emoji).
2018-11-06 12:06:49 -08:00
Steve Howell a092bee6b3 import: Reduce memory usage for UserMessage ids.
The UserMessage table can be huge, so creating a
bunch of entries in `ID_MAP` can overflow memory.

We don't have any tables that depend on `UserMessage`,
and we don't send the 'id' fields from `zerver_usermessage`
to the database, so re-mapping them was just busy-work.
2018-11-05 10:18:01 -08:00
Tim Abbott 40ff41e135 tornado: Fix populate_db failing to call send_event properly.
This isn't the right long-term fix; theoretically, send_event
shouldn't be doing anything with populate_db, but that's for later.
2018-11-02 17:07:21 -07:00
Jack Zhang 5902a573be push_notifications: Play 'default' sound for iOS notifications.
Fixes zulip/zulip-mobile#2651.

This was tested on an iPhone 7 running iOS 12.
2018-11-02 17:01:50 -07:00
Tim Abbott 0cac7e1cd3 tornado: Extract functions for Tornado queue names.
This moves all control for what queue to use for which realm in our
Tornado system to just the sharding.py file; no actual sharding is
done yet.
2018-11-02 17:00:10 -07:00
Tim Abbott 152c44b6d2 tornado: Extract function for specifying Tornado URI.
Since TORNADO_PROCESSES is 1 in all default configurations, this
doesn't have any user-facing effect.
2018-11-02 17:00:09 -07:00
Tim Abbott ec065e92ee tornado: Store port on SockJS connection object.
This will make it available for use inside our websockets code.
2018-11-02 16:55:33 -07:00
Tim Abbott ea1ec68899 events: Pass a realm object into send_event.
This is a preparator refactor for supporting hosting different Tornado
processes on different servers; to look up which Tornado server we
should be sending the event to, we'll need the realm object.
2018-11-02 16:47:39 -07:00
Tim Abbott 75e48459b5 tornado: Support using a port-aware file for dumping event queues.
This should make it possible for there to safely be multiple Tornado
processes running on different ports on the same system.

It may also fix a rare race bug in development, where previously, it
was possible for the Tornados processes for Casper and the main
development server to interfere; I haven't investigated whether this
was a real bug or not, but now those two services will use independent
Tornado files.

We still need to add something to direct traffic between the different
Tornado processes.
2018-11-02 16:47:39 -07:00
Tim Abbott 0976e3bd39 purge_queue: Use more robust Tornado patterns.
We're about to make the list of possible Tornado queue names broader.
2018-11-02 16:06:46 -07:00
Tim Abbott 9339c191da tornado: Fix missing mypy annotation. 2018-11-02 14:43:55 -07:00
Tim Abbott cf24a20185 tornado: Refactor logic for persistent queue filenames.
Now, these are computed using a function, which will make it easier to
edit these paths to depend on which Tornado process it is in coming
commits.
2018-11-02 14:19:10 -07:00
Tim Abbott 98f28fa6ce tornado: Remove unused send_notification() function.
This hasn't been used in a long time, probably since
3fddc11cc2.
2018-11-02 14:14:39 -07:00
Tim Abbott e221a01638 Revert "Change if(realm.domain == mit.edu) to use Realm.is_zephyr_..."
This reverts commit 3645bb9225.

This change was incorrect, because the `is_zephyr_mirror_realm`
property on Realm is a property and thus isn't available in the
migration codebase.

Since this migration is only run for very old servers, this should
have no impact.
2018-11-02 13:16:45 -07:00
Rishi Gupta 7dc021a6b2 billing: Improve error message for require_billing_access. 2018-11-01 14:17:36 -07:00
Vishnu Ks 677f34c2b9 billing: Use require_billing_access decorator in JSON endpoints. 2018-11-01 12:45:29 -07:00
Tim Abbott 4c55de4b58 hipchat import: Don't require the output directory to already exist.
This fixes a common exception one might see when trying to use the
HipChat importer for the first time.
2018-11-01 10:09:19 -07:00
Tim Abbott d54af3cb5b hipchat import: Handle deactivated users without an email address.
We saw this in a recent HipChat import data set.
2018-11-01 10:09:19 -07:00
Tim Abbott 48810f43be queue_processors: Remove unnecessary spammy logging output.
This logging statement was incorrectly not removed before merging
5cec566cb9.
2018-10-31 16:31:35 -07:00
Tim Abbott ff2b496067 error_notify: Stringify QUERY_STRING values directly.
Apparently, the QUERY_STRING property of the report object wasn't
actually a string; since we only care about its string representation,
we should just stringify it.
2018-10-31 16:30:38 -07:00
Tim Abbott 10ac671cd4 middleware: Fix logging of query counts in websockets requests.
Apparently, we weren't resetting the query counters inside the
websockets codebase, resulting in broken log results like this:

SOCKET  403   2ms (db: 1ms/2q) /socket/auth [transport=websocket] (unknown via ?)
SOCKET  403   5ms (db: 2ms/3q) /socket/auth [transport=websocket] (unknown via ?)
SOCKET  403   2ms (db: 3ms/4q) /socket/auth [transport=websocket] (unknown via ?)
SOCKET  403   2ms (db: 3ms/5q) /socket/auth [transport=websocket] (unknown via ?)
SOCKET  403   2ms (db: 4ms/6q) /socket/auth [transport=websocket] (unknown via ?)
SOCKET  403   2ms (db: 5ms/7q) /socket/auth [transport=websocket] (unknown via ?)
SOCKET  403   2ms (db: 5ms/8q) /socket/auth [transport=websocket] (unknown via ?)
SOCKET  403   3ms (db: 6ms/9q) /socket/auth [transport=websocket] (unknown via ?)

The correct fix for this is to call reset_queries at the start of each
endpoint within the websockets system.  As it turns out, we're already
calling record_request_start_data there, and in fact should be calling
`reset_queries` in all code paths that use that function (the other
code paths, in zerver/middleware.py, do it manually with
connection.connection.queries = []).

So we can clean up the code in a way that reduces risk for similar
future issues and fix this logging bug with this simple refactor.
2018-10-31 16:22:17 -07:00
Yashashvi Dave 4f37127ba9 /json/users/: Add user profile data variable to update values.
This commit adds user profile data variable in `/json/users/`
api view. This allows user to update user profile data
along with user's name and type.
2018-10-31 15:36:44 -07:00
Yashashvi Dave 4ceb4b607f zerver/lib/users.py: Extract func `validate_user_custom_profile_data`.
Extract function `validate_user_custom_profile_data` to validate
user's custom profile field values according to field type.
2018-10-31 15:36:44 -07:00
Jack Zhang 3d6a745047 notifications: Show stream names for private stream messages.
Fixes #10745.

Use get_display_recipient to get stream names, and remove the
references to message.stream_name in push_notifications.py which were
added in 97571a203, as the actual stream names were being retrived
only for Message objects associated with public streams.
2018-10-30 20:03:48 -07:00
Rishi Gupta 33d6baa529 portico: Add security doc to /security. 2018-10-30 19:52:24 -07:00
Steve Howell 30c493ed24 slack import: Generate message_id/reaction_id with NEXT_ID.
This avoids the need to pass tuples of ints around, which
is pretty brittle.
2018-10-29 13:24:50 -07:00
Steve Howell 2f58eb1057 slack import: Extract process_message_files().
This is mostly an extraction, but it does change the
way we calculate `content`.  We append the markdown
links from ALL files to any content that came in the
message itself.

Separating this out also allows us to add more
test coverage for the extracted code.
2018-10-29 13:24:50 -07:00
Steve Howell 00f822a26a conversion: Generate attachment_ids with helpers. 2018-10-29 13:24:50 -07:00
Steve Howell 5cb60f7bea conversions: Use subscriber_map for Slack/Gitter.
We now use subscriber_map for building UserMessage
rows in Slack/Gitter conversions.

This is mostly designed to simplify the code, rather
than having to scan the entire subscribers for each
message.

I am guessing this will improve performance for most
conversions.  We sort small lists on every message,
in order to be deterministic, but the sorting cost
is probably more than offset by avoiding the O(N)
scans across all subscriptions.  Also, it's probably
negligible in the grand scheme of things, compared
to JSON parsing, file I/O, etc.

This commits also fixes some typos with mentioned_users_id ->
mentioned_user_ids and cleans up a test a bit as well.
2018-10-29 13:24:50 -07:00
Steve Howell adb458a5df refactor: Use build_user_message for Slack/Gitter.
We now have all three third party
conversions (Gitter/Slack/Hipchat)
go through build_user_message().

Hipchat was already using this helper.

We also avoid callers having to pass in
an id to build_user_message().
2018-10-29 13:24:50 -07:00
Steve Howell 5194701787 conversions: Use NEXT_ID for usermessage_id.
This is mostly complicated due to the way that the
Slack import passes around tuples of ids to maintain
four different parallel sequences.
2018-10-29 13:24:50 -07:00
Steve Howell e90f47a530 minor: Remove unused param for get_message_url(). 2018-10-29 12:57:15 -07:00
Steve Howell fd62e71737 Clean up URLs sent by outgoing webhooks.
When you send a message to a bot that wants
to talk via an outgoing webhook, and there's
an error (e.g. server is down), we send a
message to the bot's owner that links to the
message that triggered the error.

The code to produce those links was out of
date.

Now we move the important code to the
`url_encoding.py` library and fix the PM
links to use the more modern style (user_ids
instead of emails).  We also replace "subject"
with "topic" in the stream urls.
2018-10-29 12:57:15 -07:00
neiljp (Neil Pilgrim) 482383b6f7 mypy: Add Optional to tornado/descriptors.py; remove from mypy.ini. 2018-10-29 12:53:16 -07:00
Pragati Agrawal d5df0377cc settings_users: Support guest user in admin-user-table.
This supports guest user in the user-info-form-modal as well as in the
role section of the admin-user-table.

With some fixes by Tim Abbott and Shubham Dhama.
2018-10-29 12:33:35 -07:00
Vishnu Ks 553c50ebfb messages: Add history_limited flag to messages response.
The purpose of this commit is to pass information
to the frontend whether the message response recieved
has been limited due to plan restrictions or not.

To implement this, the backend for limiting the message
history had to be rewritten as we used to fetch
only the message rows whose id was greater than
first_visible_message_id. The filtered rows gives us
no information on whether the message history was
limited or not. So the backend was rewritten to not
do any restriction of limiting the message rows while
making the query. The limiting of rows is now done in
post_process_limited_query which will also return back
the value of history_limited flag.

Tweaked by tabbott to note a few cases where the results are
incorrect.  I'm merging this despite those, because those cases don't
impact the correctness of the feature, and it may have tricky
performance implications to fix correctly.
2018-10-26 17:41:07 -07:00
Tim Abbott dbc412b8eb test_narrow: Fix missing found_oldest assert in tests.
Apparently, we weren't actually checking that found_oldest had the
correct value; fortunately, this didn't actually result in a problem,
because the values were always correct.  But this will be important as
we start extending this test.
2018-10-26 17:33:13 -07:00
Vishnu Ks cc8dd0e971 billing: Set message_visibility_limit to 10000 for Limited. 2018-10-26 17:10:38 -07:00
Aditya Bansal f90f701f03 camo: Change CAMO_URI setting value for test suite.
This is a preparatory commit which will help us with removing camo.
In the upcoming commits we introduce a new endpoint which is based
out on the setting CAMO_URI. Since camo could have been hosted on
a different server as well from the main Zulip server, this change
will help us realise in tests how that scenerio might be dealt with.
2018-10-26 16:51:54 -07:00
Aditya Bansal 86eeae6faa bugdown: Rename use_thumbnails to already_thumbnailed for clarity. 2018-10-26 16:51:54 -07:00
Aditya Bansal 7dc1b591cb camo: Extract generate_camo_url out from get_camo_url.
This will help us eliminate camo from our production installs.
Basically it helps us de duplicate some code from upcoming code
which will help us check validity of a camo url.
2018-10-26 16:51:54 -07:00
Jack Zhang f116aba490 push notifications: Reword APNs payload alert titles.
Also, rename get_alert_from_message to get_gcm_alert.

With the implementation of the and get_apns_alert_title and
get_apns_alert_subtitle, the logic within get_alert_from_message
is only relevant to the GCM payload, so we adjust the name
accordingly.

Progresses #9949.
Resolves https://github.com/zulip/zulip-mobile/issues/1316.
2018-10-26 16:02:04 -07:00
Jack Zhang 92a100798c push notifications: Remove tests for get_alert_from_message.
The string that is returned from get_alert_from_message is
dependent upon the same message that is passed into get_apns_payload
and get_gcm_payload. The contents of those payloads that are tested via
TestGetAPNsPayload and TestGetGCMPayload, which makes the tests for
get_alert_from_message redundant.

Also, simplify the logic by removing the last elif conditional.
2018-10-26 15:55:26 -07:00
Steve Howell e2ee455314 outgoing webhooks: Support widget content.
If we use an outgoing webhook and the web server
responds with `widget_content` in the payload, we
include that in what we send through the send-message
codepath.

This makes outgoing webhook bots more consistent with
generic bots.
2018-10-26 12:08:05 -07:00
Rishi Gupta 92d0511c0b portico: Make a few wording edits to the top of /hello. 2018-10-25 18:25:20 -07:00
Steve Howell 9145cd16cf minor: Change topic for imported hipchat messages. 2018-10-25 14:16:11 -05:00
Steve Howell 4b4f27fffb tests: Fix flaky test by using sets, not lists.
The test named `test_archiving_messages_with_attachment`
started flaking recently.  We use sets for comparison
instead of lists to avoid arbitrary sorting differences.
2018-10-25 13:47:37 -05:00
Steve Howell 78f6e3ac7d hipchat import: Fix data issues with PMs.
We now set the is_private flag on UserMessage
rows for PMs and set their subject to ''.
2018-10-25 09:11:36 -05:00
Steve Howell 272b954790 hipchat import: Add option to mask content.
Masking content can be useful for testing
out conversions where you're dealing
with data from customers and want to avoid
inadvertently reading their content (while
still having semi-realistic messages).
2018-10-25 08:31:01 -05:00
Steve Howell 6e8ae2e3fd hipchat import: Support private stream subscribers.
We now create private stream subscriptions that are
based off of `members` and `owner` from room data
in `rooms.json`.
2018-10-25 08:31:01 -05:00
Steve Howell 25f532ca2f refactor: Break up build_subscriptions.
Having two smaller functions should make it
easier to customize the behavior for each specific
use case.  The only reason they were ever coupled
was to keep ids in sequence, but the recent NEXT_ID
changes make that a non-issue now.
2018-10-25 08:31:01 -05:00
Steve Howell 2ed9fbd25b conversions: Use NEXT_ID for recipient and subscription ids.
The NEXT_ID scheme seems pretty robust, so I'm fixing a
few easy places.
2018-10-25 08:31:01 -05:00
Steve Howell 50f76e58ce conversions: Make NEXT_ID a true singleton.
We now instantiate NEXT_ID in sequencer.py, which avoids
having multiple modules make multiple copies of a sequencer
and possibly causing id collisions.
2018-10-25 08:31:01 -05:00
neiljp (Neil Pilgrim) a2767df51d mypy: Enforce strict-optional checking of decorator.py. 2018-10-25 07:24:47 -05:00
Tim Abbott 462b9c80c0 bots: Add basic documentation for duplicate bot names feature. 2018-10-24 17:01:34 -07:00
Steve Howell 551fc7f165 bots: Prevent bots from having duplicate full names.
Bots are not allowed to use the same name as
other users in the realm (either bot or human).

This is kind of a big commit, but I wanted to
combine the post/patch (aka add/edit) checks
into one commit, since it's a change in policy
that affects both codepaths.

A lot of the noise is in tests.  We had good
coverage on the previous code, including some places
like event testing where we were expediently
not bothering to use different names for
different bots in some longer tests.  And then
of course I test some new scenarios that are relevant
with the new policy.

There are two new functions:

    check_bot_name_available:
        very simple Django query

    check_change_bot_full_name:
        this diverges from the 3-line
        check_change_full_name, where the latter
        is still used for the "humans" use case

And then we just call those in appropriate places.

Note that there is still a loophole here
where you can get two bots with the same
name if you reactivate a bot named Fred
that was inactive when the second bot named
Fred was created.  Also, we don't attempt
to fix historical data.  So this commit
shouldn't be considered any kind of lockdown,
it's just meant to help people from
inadvertently creating two bots of the same
name where they don't intend to.  For more
context, we are continuing to allow two
human users in the same realm to have the
same full name, and our code should generally
be tolerant of that possibility.  (A good
example is our new mention syntax, which disambiguates
same-named people using ids.)

It's also worth noting that our web app client
doesn't try to scrub full_name from its payload in
situations where the user has actually only modified other
fields in the "Edit bot" UI.  Starting here
we just handle this on the server, since it's
easy to fix there, and even if we fixed it in the web
app, there's no guarantee that other clients won't be
just as brute force.  It wasn't exactly broken before,
but we'd needlessly write rows to audit tables.

Fixes #10509
2018-10-24 16:59:57 -07:00
Steve Howell fe6df1c222 hipchat import: Fix bug w/rogue UserMessage records.
This bug was introduced very recently and is an
aliasing bug.  It caused extra UserMessage rows to
be created as we inadvertently updated the underlying
subscriber_map sets for multiple messages.

This probably mostly affected PMs.

It's doubtful the bug ever got out into the field.
2018-10-24 18:44:18 -05:00
Tim Abbott 5cec566cb9 queue_processors: Rewrite MissedMessageWorker to always wait.
Previously, MissedMessageWorker used a batching strategy of just
grabbing all the events from the last 2 minutes, and then sending them
off as emails.  This suffered from the problem that you had a random
time, between 0s and 120s, to edit your message before it would be
sent out via an email.

Additionally, this made the queue had to monitor, because it was
expected to pile up large numbers of events, even if everything was
fine.

We fix this by batching together the events using a timer; the queue
processor itself just tracks the items, and then a timer-handler
process takes care of ensuring that the emails get sent at least 120s
(and at most 130s) after the first triggering message was sent in Zulip.

This introduces a new unpleasant bug, namely that when we restart a
Zulip server, we can now lose some missed_message email events;
further work is required on this point.

Fixes #6839.
2018-10-24 14:43:36 -07:00
Tim Abbott 9ed3fe3596 events: Improve logging for batched missed-message email handler. 2018-10-24 11:21:51 -07:00
Rishi Gupta 458169928c billing: Rename Zulip Premium to Zulip Standard. 2018-10-24 10:42:16 -07:00
Steve Howell 409e2b4134 hipchat import: Support sender_id == 0 use case. 2018-10-23 17:27:37 -05:00
Steve Howell 48b5b2f5d5 hipchat import: Start adding tests.
These test are for the handling of HipChat
sender info.  The data formats are somewhat
inconsistent and sometimes require us to
generate "mirror" users, so this is potentially
fragile code if we don't cover it well.
2018-10-23 17:27:37 -05:00
Steve Howell 876a72c467 hipchat import: Extract get_hipchat_sender_id(). 2018-10-23 17:27:37 -05:00
Steve Howell 481488a35e Extract make_subscriber_map().
We extract this function and put it in the shared
library `import_util.py`.

Also, we make it one time higher up in the call
stack, rather than re-building it for every batch
of messages.  I doubt this was super expensive, but
there's no reason to repeatedly execute this.
2018-10-23 17:27:37 -05:00
Steve Howell 737e02a2e6 hipchat import: Fix PM messages.
Before this fix, we were creating two copies of every
PM Message in zerver_message with only corresponding
UserMessage row.

Now we only create one PM Message per message, which
we accomplish by making sure we only use imported
messages from the sender's history.json file.  And
then we write UserMessage rows for both participants
by making sure to include sender_id in the set of
user_ids that feeds into making UserMessage.  For
the case where you PM yourself, there's just one
UserMessage row.

It does not appear that we need to support huddles
yet.
2018-10-23 17:27:37 -05:00
Steve Howell 53436b4b41 import: Rename id_maps -> ID_MAP. 2018-10-23 17:27:37 -05:00
Steve Howell bd9e4ef0c8 import: Use pub_date to sort message ids.
When we create new ids for message rows, we
now sort the new ids by their corresponding
pub_date values in the rows.

This takes a sizable chunk of memory.

This feature only gets turned on if you
set sort_by_date to True in realm.json.
2018-10-23 17:27:37 -05:00
Steve Howell d1ff903534 refactor: Rename build_user -> build_user_profile.
This makes greps less confusing.
2018-10-23 17:27:37 -05:00
Rishi Gupta b26e8eca62 portico: Redirect /plans to zulipchat.com/plans for self-hosted realms. 2018-10-23 14:47:00 -07:00
Rishi Gupta fbd884f934 billing: Rename MAX_INVITES_PREMIUM.
Largely to have parallel structure with INVITES_DEFAULT_REALM_DAILY_MAX.
2018-10-23 14:47:00 -07:00
Rishi Gupta a7fe6b1c34 billing: Fix max_invites for PREMIUM_FREE.
We could migrate all the current PREMIUM_FREE organizations to have more
invites, but this setting mainly affects orgs right as they are starting, so
it's probably fine.
2018-10-23 14:47:00 -07:00
Eeshan Garg 4c0890e8b0 webhooks/github: Handle empty 'requested_reviewers' key.
We recently received a bug report that implied that for certain
payloads, the `requested_reviewers` key was empty whereas a
singular `requested_reviewer` key containing one reviewer's
information was present in its stead. Naturally, this raised
some not so pretty IndexError exceptions.

After some investigation and generating a few similar payloads,
I discovered that in every case both the `requested_reviewers`
and the `requested_reviewer` keys were correctly populated, so I
had to manually edit the payload to reproduce the error on my end.

My guess is that this anomaly goes back to when GitHub's reviewer
request feature was new and didn't support requesting multiple
reviewers, and that the singular `requested_reviewer` key could
possibly just be there for backwards compatibility or might just
be mere oversight. Either way, the solution here is to look for the
plural `requested_reviewers` key, and if that is empty, fall back
to the singular `requested_reviewer` key.
2018-10-23 14:41:24 -07:00
Rishi Gupta 1c264dedc5 api docs: Change Shakespeare quote.
The previous quote doesn't come off well if you don't know the reference
(which the majority of our users will not).
2018-10-22 15:44:36 -07:00
Eeshan Garg 3a47a1a627 webhooks/greenhouse: Ignore ping events.
"ping" type events are usually just Greenhouse's way of indicating
that a webhook has been set up.
2018-10-22 12:27:48 -07:00
Aditya Bansal a16bf34c7f thumbnailing: Fix oversharpening of thumbnails.
We seemed to have been doing too much of sharpening on the thumbnails.
The purpose of sharpening here was to just counter the softening
effects of a resize on an image but overdoing it is bad.

Value sharpen(0.5,0.2,true) seems to look good for achieving the
best results here on different displays as revealed in the manual
hit and trial based testing.

Thanks to @borisyankov for pointing out the issue and suggesting
the values.
2018-10-22 22:28:04 +05:30
Pragati Agrawal b665b0a9a5 users: Avoid O(N) query checking for is_admin.
The main benefit is that we don't do O(N) queries for checking admin
thus leading to less processing of UserProfile.
2018-10-19 15:51:12 -07:00
Tim Abbott 39ea471cf1 error reports: Ensure we filter API keys from query strings.
For some webhook endpoints where the third-party API requires us to do
this, the user's API key might appear in error emails through
appearing in the `QUERY_STRING` parameter.  Fix that by filtering any
actual content from those; what we usually need for debugging is just
what set of parameters were provided.
2018-10-19 15:03:14 -07:00
Yashashvi Dave 41fbb16cdf org settings: Fix error when admin update realm users full names.
Currently, if there is only one admin in realm and admin tries
to updates any non-adminuser's full name it throws error,
"Cannot remove only realm admin". Because in `/json/users/<user_id>`
api check_if_last_admin_is_changed is checked even if property
is_admin is not changed.

This commit fix this issue and add tests for it.
2018-10-19 12:34:20 -07:00
Rishi Gupta aad5c3df48 user docs: Fix typo causing broken gear icon. 2018-10-18 16:30:42 -07:00
Tim Abbott 695d8d0bd1 get_apns_payload: Require a UserProfile object for the recipient.
This is preparatory work for being able to display an unread count
badge on iOS, in which case we need to know who the current user is.
2018-10-18 15:09:18 -07:00
Tim Abbott a34b79a3f4 python: Avoid importing the mock module in production.
These lazy imports save a significant amount of time on Zulip's core
import process, because mock imports pbr, which in turn import
pkgresources, which is in turn incredibly slow to import.

Fixes part of #9953.
2018-10-17 15:28:48 -07:00
Steve Howell ff61c56f47 hipchat import: Add NotificationMessage support. 2018-10-17 12:11:08 -07:00
Tim Abbott f9b6eeb488 import: Migrate from json to ujson for better perf.
We expect to get better memory performace from
ujson than json.

We also do a better job of closing file handles.

This likely fixes #10377.
2018-10-17 12:11:08 -07:00
Tim Abbott 2eebacf2dc push_notifications: Lazily import APNS libraries.
The APNS client libraries (especially the hyper.http20 one) were
determined via profiling to take significant time during the import
process, so we move them to be lazily imported in order to optimize
the overall Zulip import process.  This save up to about 100ms in
import time.

These libraries are only used in certain Django processes inside
zulipchat.com, and so are unnecessary both in development as well as
for self-hosted Zulip servers.
2018-10-17 11:59:33 -07:00
Tim Abbott 704967faa4 email_mirror: Don't import talon unless we're using it.
Talon is an expensive import; on my system, deferring this import
saves 28ms on the import time for Zulip.
2018-10-17 11:25:38 -07:00
Rishi Gupta d76f728dda api docs: Rename webhook-walkthrough to incoming-webhooks-walkthrough. 2018-10-16 21:07:00 -07:00
Tim Abbott 78a15dd715 slack import: Fix obscure email address for Slackbot.
Since we know what slackbot is, we don't need to give it a crazy hash
as its email address.
2018-10-16 16:33:41 -07:00
Aditya Bansal 8324e2c976 thumbnails: Return original path if url is not supposed to be thumbnailed. 2018-10-16 16:00:47 -07:00
Aditya Bansal 5d68bd92ad thumbnails: Extract user_uploads_or_external as a function.
This is a prepartory commit for the upcoming changes. It was meaningful
to extract this one out because this function is essentially a condition
check on whether a given url is one of the user_uploads or an external
one. Based on its value we decide whether a url must be thumbnailed or
not and thus this function will also be used in an upcoming commit
patching lib/thumbnail.py to do the same check before thumbnail url
generation.
2018-10-16 16:00:47 -07:00
Aditya Bansal 9a411a5765 thumbnails: Stop thumbnailing urls other than external or user_uploads.
We are basically adding a check for url's to be external (belonging
to some 3rd party web site hosting the image) or be one of the
user uploaded files. User uploaded files are served by a separate
endpoint which is /user_uploads/. Any other local url such as
/user_avatars/ or /static/ should never be sent to thumbor for
thumbnailing.
Not sending /user_avatars/ to thumbor for thumbnailing makes sense
because they are already properly thumbnailed and stored properly.
/static/ urls host very few images we use for demo and can be safely
be excluded from thumbnailing.
2018-10-16 16:00:47 -07:00
Tim Abbott e4813e462b tornado: Rename async_request_{restart,stop} to mention timer.
Previously, these timer accounting functions could be easily mistaken
for referring to starting/stopping the request.  By adding timer to
the name, we make the code easier for the casual observer to read and
understand.
2018-10-16 15:39:10 -07:00
Eeshan Garg 0292ffe04b webhooks/taiga: Add test for when a task is moved between stories.
This commit adds a test for the payload that is generated when
a Task is moved from one user story to another on Taiga's Sprint
Taskboard UI.

This commit also gets up this webhook's test coverage up to 100%.
2018-10-16 13:06:20 -07:00
Eeshan Garg 48fe5ae5b2 webhooks/taiga: Stop catching KeyErrors for 'change' events.
I generated multiple payloads and verified that there are no
`change` event payloads that will not contain the values in
question, so it is useless to catch these KeyErrors. If there are
any anomalies still, it is better to be notified about them than
to silently ignore them.
2018-10-16 13:06:20 -07:00
Yago González 029d815be7 api docs: Document the /dev_fetch_api_key endpoint. 2018-10-16 12:47:12 -07:00
Yago González 3eeec94d03 api docs: Document the POST /users/me/subscriptions/properties endpoint. 2018-10-16 12:38:27 -07:00
Tim Abbott 8481a2fd2d api docs: Fix confusing discussion of "dev server".
The Zulip API is to be used on both development and production
servers, and really we just need to talk about zuliprc files.

There's a similar issue for the JS docs, but we need to fix the
copy/paste issues with those as well.
2018-10-16 12:23:23 -07:00
Yago González 3d2b3f8fa4 api docs: Document the POST /messages/flags endpoint. 2018-10-16 12:19:24 -07:00
Yago González 7a4103eee6 api docs: Document POST /mark_(all|stream|topic)_as_read. 2018-10-16 12:12:05 -07:00
Tim Abbott d13bd6e8d3 linkifiers: Rename articles to current feature name.
These were previously called "linkification filters", but we now want
to consistently refer to them as "linkifiers" in the UI and API docs.
2018-10-16 12:01:17 -07:00
Yago González 3bdc8f9946 api: Document the GET /realm/emoji endpoint. 2018-10-16 11:51:48 -07:00
Yago González 7a7b507e86 api docs: Document DELETE /realm/filters/<filter_id>. 2018-10-16 11:51:48 -07:00
Yago González a7c48acc8e api docs: Document GET /realm/filters. 2018-10-16 11:51:48 -07:00
Vishnu Ks 14a6a8db1a logs: Mock out warning logs in test_scrub_realm. 2018-10-16 11:16:47 -07:00
Vishnu Ks 57e0a24020 management: Scrub delivery email as well while running scrub_realm. 2018-10-16 11:16:46 -07:00
Kevin Lin b2c29274f3 presence: Allow bots to fetch realm presence data.
Before, presence information for an entire realm could only be queried via
the `POST /api/v1/users/me/presence` endpoint. However, this endpoint also
updates the presence information for the user making the request. Therefore,
bot users are not allowed to access this endpoint because they don't have
any presence data.

This commit adds a new endpoint `GET /api/v1/realm/presence` that just
returns the presence information for the realm of the caller.

Fixes #10651.
2018-10-15 12:51:28 -07:00
Steve Howell b1dd9a251b hipchat import: Break messages into smaller batches.
Even individual "room" files from hipchat can be large,
so we process only 1000 messages at a time
within each file, which produces smaller JSON files.
2018-10-15 10:54:23 -07:00
Steve Howell 6650bb2240 minor: Move fix_mentions() closer to caller. 2018-10-15 10:54:23 -07:00
Steve Howell 219ff0f749 hipchat import: Extract UserHandler class. 2018-10-15 10:54:23 -07:00
Steve Howell 2d523fd668 hipchat import: Extract make_user_messages(). 2018-10-15 10:54:23 -07:00
Steve Howell ca0495cbe6 hipchat import: Support attachments. 2018-10-15 10:54:23 -07:00
Steve Howell 2d4b09f59d utils: Add process_list_in_batches(). 2018-10-15 10:54:23 -07:00
Steve Howell d71f3eb1bf hipchat import: Add some more logging. 2018-10-14 09:29:04 -07:00
Steve Howell 76deb30312 preview: Hash cache keys for preview urls.
We don't want really long urls to lead to truncated
keys, or we could theoretically have two different
urls get mixed up previews.

Also, this suppresses warnings about exceeding the
250 char limit.

Finally, this gives the key a proper prefix.
2018-10-14 09:28:57 -07:00
Steve Howell d933779477 hipchat import: Support PrivateUserMessage data.
We now import PM data from HipChat.
2018-10-13 16:47:44 -07:00
Steve Howell f0c3ee0a2e hipchat import: Write smaller message files.
We now write new message files for each new input
file + message type we process.  This helps the
importer not run out of memory later.
2018-10-13 16:47:44 -07:00
Steve Howell 75fc5d41c9 hipchat import: Refactor write_message_data.
The goal here is to make it easier to handle other
message types by moving the key-specific stuff
to the top of the file.
2018-10-13 16:47:44 -07:00
Steve Howell cc55eb8154 hipchat import: Only process UserMessage rows for now. 2018-10-13 16:47:44 -07:00
Steve Howell 3baac7ddf3 hipchat import: Handle missing emails for guest users. 2018-10-13 16:47:44 -07:00
Steve Howell 8accc60ca7 import_util: Support multiple message ids for attachments. 2018-10-13 16:47:44 -07:00
Steve Howell 23d7b3d2cc import: De-dup create_converted_data_files helper. 2018-10-13 16:47:41 -07:00
Steve Howell 91905bd66a import: Add sequencer library.
This avoids some tedious code related to making ids
in conversion programs.
2018-10-13 16:47:39 -07:00
Steve Howell 85f1910f93 minor: Add link to hipchat spec to code. 2018-10-13 16:43:28 -07:00
Steve Howell 493aae2958 imports: Make loading UserMessage faster and more robust.
We use UserMessageLite to avoid Django overhead, and we
do updates in chunks of 10000.  (The export may be broken
into several files already, but a reasonable chunking at
import time is good defense against running out of memory.)
2018-10-13 16:43:28 -07:00
Tim Abbott 68ab71eb8b push: Fix exceptions when removing push notifications.
Now that we allow multiple users to have registered the same token, we
need to configure calls to unregister tokens to only query the
targeted user_id.

We conveniently were already passing the `user_id` into the push
notification bouncer for the remove API, so no migration for older
Zulip servers is required.
2018-10-12 11:19:23 -07:00
Steve Howell 0f7628280f narrow: Handle spurious emails in pm-with searches.
If cordelia searches on pm-with:iago@zulip.com,cordelia@zulip.com,
we now properly treat that the same way as pm-with:iago@zulip.com.

Before this fix, the query would initially go through the
huddle code path.  The symptom wasn't completely obvious, as
eventually a deeper function would return a recipient id
corresponding to a single PM with @iago@zulip.com, but we would
only get messages where iago was the recipient, and not any
messages where he was the sender to cordelia.

I put the helper function for this in zerver/lib/addressee, which
is somewhat speculative.  Eventually, we'll want pm-with queries
to allow for user ids, and I imagine there will be some shared
logic with other Addressee code in terms of how we handle these
strings.  The way we deal with lists of emails/users for various
endpoints is kind of haphazard in the current code, although
granted it's mostly just repeating the same simple patterns.  It
would be nice for some of this code to converge a bit.  This
affects new messages, typing indicators, search filters, etc.,
and some endpoints have strange legacy stuff like supporting
JSON-encoded lists, so it's not trivial to clean this up.

Tweaked by tabbott to add some additional tests.
2018-10-12 10:18:30 -07:00
Steve Howell 9f2aad55b5 hipchat import: Handle users without avatars. 2018-10-12 07:03:25 -04:00
Steve Howell 51bd36e448 tests: Add coverage to get_service_interface_class(). 2018-10-11 16:12:07 -07:00
Steve Howell 8379aeee15 outgoing bots: Fix header for generic servers.
For our bots that use GenericOutgoingWebhookService
(which are basically Zulip style bots), we now
include a "content-type" header of "application/json".

We accomplish this by having the service classes
implement their own custom method called
`send_data_to_server`. For the Slack-related
code, we just extracted code from `do_rest_call`,
and then for the Zulip-related code, we added
a `headers` parameter.
2018-10-11 16:12:07 -07:00
Steve Howell 8226e13e9c bot tests: Replace use of MockServiceHandler.
We'll just use a real class here, since the service
handlers are pretty lightweight and just munge data.
2018-10-11 16:12:07 -07:00
Steve Howell 8f74d99b6c Remove stubs in OutgoingWebhookServiceInterface.
If we omit methods in subclasses, they're likely to
be caught by linters or unit tests, and even if they
aren't, raising NotImplementedError doesn't actually
prevent user problems.

I've been fighting these in refactoring, and it's
just been a bunch of busy work, plus comments are
highly likely to bitrot.
2018-10-11 16:12:07 -07:00
Steve Howell 31597cf33e Remove timeout parameter in do_rest_call().
Nobody was setting it.
2018-10-11 16:12:07 -07:00
Steve Howell 69ee84bb14 refactor: Extract build_bot_request().
This fixes a couple things:

    * process_event() is a pretty vague name
    * returning tuples should generally be avoided
    * we were producing the same REST parameters in both
      subclasses
    * relative_url_path was always blank
    * request_kwargs was always empty

Now process_event() is called build_bot_request(),
and it only returns request data,
not a tuple of `rest_operation` and `request_data`.

By no longer returning `rest_operation`, there are
fewer moving parts.  We just have `do_rest_call` make
a POST call.
2018-10-11 16:12:07 -07:00
Steve Howell 16eff75e49 refactor: Simplify how we use base_url.
Before this change, we instantiated base_url into a superclass
of subclasses that returned base_url into a dictionary that
gets returned to our caller.

Now we just pull base_url out of service when we need to make
the REST call.
2018-10-11 16:12:07 -07:00
Steve Howell b89a94f730 Improve errors when we can't connect to a bot server.
We don't overwhelm people with error info when bots
fail to connect or time out.
2018-10-11 16:12:07 -07:00
Steve Howell 3790c469e9 outgoing bots: Report JSON errors to users.
We should arguably report these to bot owners
as well, but this is at least an improvement
over having the server crash.
2018-10-11 16:12:07 -07:00
Steve Howell df4b665658 refactor: Parse JSON from bots in one place.
We move the JSON parsing step into the
higher level function: process_success_response().

In the unlikely event that we'll start integrating
with a solution that doesn't use JSON, we can deal
with that, and for now doing the parsing in one
place will help us make error reporting more
consistent.

In a subsequent commit we'll introduce better
error handling for malformed JSON.
2018-10-11 16:12:07 -07:00
Steve Howell 229dd5d861 outgoing webhooks: Get rid of "Success!" prefix.
The earlier code here, if it got a payload with
"response_string" as a key, would prefix the
corresponding value with "Success!".  We just
want the bot to set its own content.

The code is reorganized here so that process_success()
always produces a value keyed by "content" from
incoming data, and then process_success_response()
doesn't do any fancy munging of the data.
2018-10-11 16:12:07 -07:00
Tim Abbott 0a751567a3 upload: Fix missing mypy return type annotation. 2018-10-11 16:11:20 -07:00
Joshua Pan 971cb18cb3 user_settings: Compare new stripped email with old email.
We weren't comparing the newly stripped email with the current
old email, thus adding spaces around an email would result in
an error.
2018-10-11 15:55:32 -07:00
Tim Abbott 8cf104b643 avatar: Allow API authentication for /avatar/ routes.
This makes it feasibly for the mobile apps to correctly render user
avatars generated by the `!avatar()` syntax.
2018-10-11 15:52:29 -07:00
Aditya Bansal 3164f1a9a4 avatar: Rename user_profile to avatar_user_profile.
This is a preparatory commit for upcoming changes to move
/avatar/ to be a logged in or API accessible endpoint.
Basically we rename this variable because the new name is more
appropriate in the situation. Also user_profile will be used to
hold the user_profile of person accessing the endpoint in coming up
commit.
2018-10-11 15:50:37 -07:00
Aditya Bansal 6893f52ad9 thumbnails: Instruct thumbor to sharpen thumbnailed images.
Fixes: #10218.
2018-10-11 15:44:47 -07:00
Aditya Bansal 6e433186a1 thumbnails: Change thumbnail size to be 300px.
Fixes: #10219.
2018-10-11 15:44:47 -07:00
Vishnu Ks 962d72b58b retention: move_messages_to_archive should accept multiple message ids.
This will speed up the scrub realm management command. Calling the
function with a single message_id in a loop was extremely inefficient.
2018-10-11 15:31:12 -07:00
Vishnu Ks 6972de21be management: Add command to scrub a realm of personal data. 2018-10-11 15:30:26 -07:00
Vishnu Ks 2f5a5c2c49 test_classes: Create lear_user helper function. 2018-10-11 15:30:26 -07:00
Vishnu Ks 5bdadc8061 upload: Create function to delete avatar image. 2018-10-11 15:30:26 -07:00
Vishnu Ks 1d94fc7dbb upload: Extract function to delete file. 2018-10-11 15:30:26 -07:00
Steve Howell 4b82326376 hipchat import: Support guest users.
We simplify the code for is_realm_admin
and set is_guest as well.

I verified that build_user() is not used
by Slack/Gitter, so the extra argument there
should be fine.

Fixes #10639
2018-10-11 15:28:58 -07:00
Vishnu Ks 6aa4b64dc0 emails: Don't log emails while running test suite.
Modified the tests to ensure 100% coverage.
2018-10-11 15:12:08 -07:00
Vishnu Ks d8c19cb003 models: Move billing models from zilencer to corporate. 2018-10-11 14:54:29 -07:00
Tim Abbott c57c4cf703 notifications: Fix push notifications with multiple realms.
Previously, Zulip did not correctly handle the case of a mobile device
being registered with a push device token being registered for
multiple accounts on the same server (which is a common case on
zulipchat.com).  This was because our database `unique` and
`unique_together` indexes incorrectly enforced the token being unique
on a given server, rather than unique for a given user_id.

We fix this gap, and at the same time remove unnecessary (and
incorrectly racey) logic deleting and recreating the tokens in the
appropriate tables.

There's still an open mobile app bug causing repeated re-registrations
in a loop, but this should fix the fact that the relevant mobile bug
causes the server to 500.

Follow-up work that may be of value includes:
* Removing `ios_app_id`, which may not have much purpose.
* Renaming `last_updated` to `data_created`, since that's what it is now.

But none of those are critical to solving the actual bug here.

Fixes #8841.
2018-10-10 16:15:52 -07:00
Rishi Gupta bf22eefede api docs: Move integration-docs-guide to docs/. 2018-10-09 20:28:44 -07:00
Steve Howell c0df049a18 Allow "content" from outgoing webhooks.
We now allow outgoing webhooks to provide us a
"content" field, which is probably a more guessable
name than "response_string", particularly for folks
that use our other bot-related APIs.  And we don't
modify content as we do response_string, i.e. no
"Success!" prefix.

If we're not too concerned about backward compatibility,
we can do a subsequent commit that makes "content"
and "response_string" true synonyms and get rid of
the "Success!" prefix, which was probably accidental
to begin with.
2018-10-09 15:56:24 -07:00
Steve Howell 6c4343c86d refactor: Clean up send_response_message().
This commit starts by changing the third
argument of send_response_message to be a Dict
instead of a string, so that the data can be more
structured going forward.

That change makes the 2nd/3rd parameters both be
dicts, so to be defensive, I now have all the callers
pass in explicit keyword names.  And then I rename
message to message_info, so that the callers have
more clear code.

And that changes the implementation inside of
send_response_message() a bit.

Sorry this commit is a bit coarse, but the intermediate
commits would have been kind of ugly, too.

At the end of the day, it's pretty simple:

    bot_id: never changed
    message_info: just renamed from message
    response_data: is a Dict with the key of "content"

And the innards of send_response_message() are basically
simply dictionary lookups and function calls.
2018-10-09 15:56:24 -07:00
Steve Howell 4956107c53 refactor: Simplify return type for process_success().
There's no reason to return a failure message in
process_success(), since it's implied to be part of
the success codepath.  I didn't look at the full history
of how the strange API evolved, but the second element
of the tuple was clearly noise by the time I got here.
Neither of the subclasses ever set it, and none of the
consumers used it.
2018-10-09 15:56:24 -07:00
Steve Howell f2dd218331 refactor: Inline succeed_with_message().
This two-line function wasn't really carrying its
weight, and it just made it harder to refactor the
overall codepath.

Eliminating the function forces us to mock at a slightly
deeper level, which is probably a good thing for what
the test intends to do.  The deeper mock still verifies that
we're sending the message (good) without digging into
all the details of how we send it (good).

Note that we will still keep around the similarly named
`fail_with_message` helper, which is a lot more useful.
(The succeed/fail scenarios aren't really symmetric here.
For success, there are fewer codepaths that do more complex
things, whereas we have lots and lots of failure codepaths
that all do the same simple thing of replying with a canned
message.)
2018-10-09 15:56:24 -07:00
Steve Howell fa505a1af1 refactor: Have process_success return structured data.
Before this change subclasses of OutgoingWebhookServiceInterface
would return a raw string as the first element of its return
tuple in process_success().  This is not a very flexible
design, as it prevents the bot from passing extra data like
`widget_content`.

It's also possible in the future that we'll want to let outgoing
bots reply directly to senders who mention them on streams, and
again the original design was overly constrained for that.

This commit does not actually change any functionality yet.
2018-10-09 15:56:24 -07:00
Steve Howell 3bb8cbe0c7 minor: Dedup check_send_message() call. 2018-10-09 15:56:24 -07:00
Steve Howell e641036911 minor: Rename var to message_type. 2018-10-09 15:56:24 -07:00
Steve Howell b61612d50b minor: De-duplicate code for client. 2018-10-09 15:56:24 -07:00
Callum Fraser 383f1633e6 api: Add limit of total messages available per request of 5000.
Tweaked by tabbott to use a declared constant rather than just use
5000 in multiple places; this also means we can change the count
without updating translations.

Fixes #10446.
2018-10-09 15:43:21 -07:00
Rishi Gupta 7956c57448 user docs: Add import-from-hipchat. 2018-10-09 15:23:07 -07:00
Lyla Fischer 3c51328763 user docs: Remove icon-vector for font-awesome. 2018-10-09 14:16:16 -07:00
Pragati Agrawal 749e034863 org settings: Rename "Filter settings" to "Linkifiers".
Fixes: #10551
2018-10-09 08:50:45 -07:00
Lenny Jagielski 4fa4ca53c2 webhooks: Add Netlify integration.
Fixes: #10169.
2018-10-05 17:04:40 -07:00
Eeshan Garg ba929508e2 webhooks/ifttt: Get test coverage up to 100%.
IFTTT allows custom templating for their payloads, so the onus is
on the user to ensure that their custom templates conform to the
expectations outlined in our IFTTT webhook docs. For that reason,
these payloads weren't generated, but were manually edited.
2018-10-04 12:16:06 -07:00
Eeshan Garg 316f9aa78b webhooks/hellosign: Rewrite the integration from scratch.
After discovering a couple of bugs, I decided to thoroughly test
and rewrite this integration from scratch. The older code wasn't
generating coherent messages.

This also commit gets this integration up to 100% test coverage.
2018-10-04 12:16:06 -07:00
Eeshan Garg 5039f6dfb5 webhooks/gitlab: Get test coverage up to 100%.
Test coverage was improved by removing an unused function and
removing some code (written by me) that was actually handling
Test Hook event types incorrectly.
2018-10-04 12:16:06 -07:00
Eeshan Garg 6e2e2b9125 webhooks/github: Test commit status payloads with target_url.
It was a painful amount of work to generate the actual payload.
Since the only difference was a small build URL, I manually
edited the payload and used that for testing.

This commit gets our GitHub webhook up to 100% test coverage.
2018-10-04 12:16:06 -07:00
Eeshan Garg 42e3410df1 webhooks/github: Improve logic for page build messages.
Some of the page build message code had insufficient test coverage.
I looked at generating the payloads that would allow me to test
the lines of code in question, but it was too much work to
generate the payloads and this seemed like a vague event anyway.

So I just rewrote the logic so that the lines missing
coverage are implicitly covered.
2018-10-04 12:16:06 -07:00
Eeshan Garg 4f34ee2e6b webhooks/github: Test and improve messages for issue comment deletion.
This is a part of our efforts to get this webhook's coverage
up to 100%.

Note that apart from just testing an uncovered line of code, this
commit also fixes a minor bug in the code for messages about issue
comment deletion and editing.
2018-10-04 12:16:06 -07:00