Commit Graph

3183 Commits

Author SHA1 Message Date
Rhea Parekh 58f1f0e13e slack importer: append Recipient id from the db.
Recipient id should be updated from the database,
otherwise there will be foreign key clashes.
2018-01-17 11:46:27 -05:00
Rhea Parekh c86411d9a2 slack importer: Move message conversions to a new module. 2018-01-17 09:38:53 -05:00
Rhea Parekh c1d336f5d1 slack importer: Refactor checking for availability of realm subdomain. 2018-01-17 09:38:53 -05:00
Ricky a90e99dd3e webhooks: Add Beeminder webhook. 2018-01-17 09:24:12 -05:00
fredfishgames 4e2b99a98b integrations: Add webhook code, API endpoint, and tests for Opbeat. 2018-01-17 09:17:19 -05:00
Shreyansh Dwivedi b0fb7aa6b2 bugdown: Add preview for vimeo videos.
This also amends a commit from Brock Whittaker <brock@zulipchat.com>
that merges two separate functions for YouTube videos and Vimeo videos
into a generic video recall function.

Fixes #7550.
2018-01-17 09:11:43 -05:00
Viraat Chandra 64034241a0 webhooks: Create Intercom integration and docs. 2018-01-16 08:51:17 -05:00
Xavier Cooney b0f501ff6c test_helpers.py: Remove unecessary imports. 2018-01-16 08:16:43 -05:00
Xavier Cooney 0c48f5bf09 test_classes.py: Remove unecssary imports. 2018-01-16 08:16:43 -05:00
Aditya Bansal c64766e55e scheduledmessages: Add funcs to verify and schedule messages.
We add two functions:

1.) check_schedule_message(): This function is responsible for
doing the essential initial checkes to verify the validity of
the message. These checkes include things like if user is
allowed to send messages to some stream or not or if the user is
a super_user. All this is basically done by further calling
check_message() with appropriate parameters. This is on the same
lines as is check_send_message().

2.) do_schedule_messages(): This function is responsible for
creating ScheduleMessage table rows for a list of messages that
are to be scheduled. This basically accumulates the ScheduleMessage
objects in a list and then bulk creates the rows.
2018-01-10 09:18:02 -05:00
Andy Perez c209069fcd bugdown: Improve image inlining logic.
Fix #7537
2018-01-09 16:43:22 -05:00
Robert Hönig d1d8365a6b embedded bots: Add config data UI.
This adds UI fields in the bot settings for specifying
configuration values like API keys for a bot. The names
and placeholder values for each bot's config fields are
fetched from the bot's <bot>.conf template file in the
zulip_bots package. This also adds giphy and followup
as embedded bots.
2018-01-09 11:50:54 -05:00
infinitelooped ffa7637215 integrations: Add webhook code, API endpoint, and tests for Groove. 2018-01-09 09:32:52 -05:00
Rhea Parekh b583f1f0ff slack importer: Add regex for markdown mapping.
* For strikethrough formatting: Slack's '~strike~' to Zulip's '~~strike~~'.
    * For bold formatting: Slack's '*bold*' to Zulip's '**bold**'.
    * For italic formatting: Slack's '_italic_' to Zulip's '*italic*'.
    * For mentioning formatting: Slack's <@slack_id|short_name> to Zulip's @**full_name**.
    * Checking links.
2018-01-08 08:05:50 -05:00
Rhea Parekh a6e0bcd5f6 Slack importer: Map messages and usermessages. 2018-01-08 08:05:50 -05:00
Rhea Parekh 24a86a524b slack importer: Quiet the unzip command. 2018-01-08 08:05:50 -05:00
Rhea Parekh 4019a456d1 slack importer: Rename messages file.
Rename messages file from '/message-000001.json' to
'/messages-000001.json'
2018-01-08 08:05:50 -05:00
Rhea Parekh 400c968b0e slack importer: Remove unnecessary comment. 2018-01-08 08:05:50 -05:00
Rhea Parekh c875428a83 slack importer: Extract get_user_full_name() helper. 2018-01-08 08:05:50 -05:00
Robert Hönig c745d1901c bot_config.py: Add function for loading config file templates.
load_bot_config_template(bot) parses the <bot>.conf
template file, which can be found in the zulip_bots
package for each bot. It then returns the INI content
of that file as a dict.
2018-01-07 20:05:52 +01:00
Robert Hönig ed7208fd6e Make EmbeddedBotHandler.get_config_info comply with ExternalBotHandler. 2018-01-07 20:05:52 +01:00
Robert Hönig 9cc059f4aa bot_config.py: Raise exception when getting config data from bot without config entries. 2018-01-07 20:05:52 +01:00
Robert Hönig dcd62f92fb validator.py: Extend functionality of check_dict().
With this commit, check_dict() can be used to verify
that all keys and/or values of a dict are of a certain
type.
2018-01-07 20:04:52 +01:00
Vishnu Ks 5cadc6365a slack importer: Extract data into directory of zip file.
Currently the zip file is extracted to the root of
zulip directory no matter where the the zip file.
The extracted data is not useful after running the command
which pollutes the zulip directory. It make more sense to
extract it to the same directory of zip file especially
when the zip file gets downloaded to /temp like in the tests.
2018-01-07 09:50:27 -05:00
Robert Hönig 29a343cb44 Rename BotUserConfigData to BotConfigData.
This is in compliance with the naming of
BotStorageData.
2018-01-06 14:13:18 -05:00
Alena Volkova 5bbc46762f tests: Improve error handling of import failures.
Fixes #6776.
2018-01-05 12:03:37 -05:00
Vishnu Ks e6d3f8895f messages: Create function to calculate first visible message id. 2018-01-04 08:52:40 -05:00
Alena Volkova 45f0c76c44 settings: Limit the creation of generic bots.
This commit adds a setting to limit creation of generic bots
to admins for realms that want that restriction.  (Generic
bots, apart from being considered spammy on some realms,
have less locked down permissions than webhook bots).

Fixes #7066.
2018-01-02 18:12:22 -05:00
Aastha Gupta daf86eb664 settings: Add "text" option to emoji_set model.
We no longer have a special UI setting and model
field ("emoji_alt_code") for saying users want text-only
emojis.  We now instead make "text" be a fifth choice
for "emojiset".

Fixes #7406
2018-01-02 14:55:01 -05:00
Vishnu Ks 277d463d23 messages: Add option to limit users message history. 2018-01-02 13:45:01 -05:00
YJDave fa44d2ea69 settings: Remove autoscroll_forever setting.
Fixes #6845
2018-01-02 10:35:49 -05:00
rht 185fd99816 mypy: Use Python 3 type syntax in several files. 2017-12-30 07:34:51 -05:00
rht 2e8adbde12 mypy: Use Python 3 type syntax in zerver/lib/actions.py. 2017-12-30 07:33:07 -05:00
Eeshan Garg 89f622937d api docs: Generate Arguments table from JSON.
This commit does the following:
* Move the Arguments table data from stream-message.md and
  private-message.md to a JSON file.
* Add a Markdown extension that allows one to include and render
  a table from a JSON file like so:
    {generate_arguments_table|arguments.json|private-stream.md}
* Use Bootstrap's .table class to format the table instead of
  relying on custom CSS.
2017-12-29 17:03:36 -05:00
akashnimare ef51a39552 markdown: Update regex for strikethrough.
This uses the correct regex for strikethrough. Also, added
a test to make sure that strikethrough works when it contains
link with whitespace.

Fixes #7596.
2017-12-28 15:59:04 -05:00
Kiy4h 78e71faefe mypy: Use Python 3 type syntax in zerver/lib/test_fixtures.py. 2017-12-28 07:26:44 -05:00
Rhea Parekh ed4054d110 Import script: Check and add system bots after every import.
This checks for the existing system bots and adds them if they
aren't included in the import.
2017-12-27 07:52:45 -05:00
Rhea Parekh cd332a7f59 Slack importer: Append ids to the existing ids in the database.
For import with the condition `--import-into-nonempty`, the ids
of the mapped data will be appended to the existing ids in the
database.
2017-12-27 07:50:53 -05:00
Rhea Parekh 5b37c5562b Slack importer: Add the slack to zulip data conversion files. 2017-12-27 07:50:53 -05:00
Rhea Parekh 25e5969add zerver/lib/mobile_auth_otp: Sweep force_str. 2017-12-26 09:09:31 -05:00
Rhea Parekh cc872ad976 zerver/lib/upload.py: Sweep force_bytes from ensure_medium_avatar_image. 2017-12-26 09:09:31 -05:00
Rhea Parekh b792a90842 Remove unused imports in the codebase. 2017-12-26 09:09:31 -05:00
nyan-salmon 613856c33b mypy: Use Python 3 type syntax in zerver/lib/send_email.py. 2017-12-26 08:34:06 -05:00
Viraat Chandra de3eb89d5c mypy: Use Python 3 syntax for typing in `zerver/lib/request.py`. 2017-12-26 08:31:43 -05:00
Viraat Chandra bbf24ec68e mypy: Use Python 3 syntax for typing in `zerver/lib/webhooks/git.py`. 2017-12-26 08:31:43 -05:00
greysome f2f9efd1a7 mypy: Use Python 3 type syntax in zerver/lib/integrations.py 2017-12-26 08:30:33 -05:00
greysome fb7ee942c4 mypy: Use Python 3 type syntax in zerver/lib/export.py 2017-12-26 08:30:33 -05:00
Greg Price a34c01780f errors: Compute deployment metadata on the right deployment.
When I added this "Deployed code" feature to the error reporting,
I apparently hadn't worked out enough of how this code works to
realize that `notify_server_error` may be in a different process,
at a different time and potentially even on a different machine
from the actual error being reported.

Given that architecture, all the data about the error must be computed
in `AdminNotifyHandler`, before sending the report through the queue,
or else it risks being wrong.  The job of `notify_server_error` and
friends is only to format the data and send it off.  So, move the
implementation of this feature in order to do that.

(@showell added some "nocoverage" directives here for code that
is hard to test (exceptions being thrown, deployment files not
existing) and that was originally part of a file that didn't
require 100% coverage)
2017-12-22 11:52:13 -05:00
Greg Price eea1ceb0db errors: Show what logger and source code the message comes from. 2017-12-22 11:52:13 -05:00
Greg Price 77e47900dd errors: Stop showing a long list of Nones when there's no request.
This had contributed to making the emails look broken and of uncertain
accuracy.
2017-12-22 11:52:13 -05:00
Greg Price 68fa29c3ec errors: Make Zulip and email paths more parallel.
This helps prevent them from diverging and getting different sets of
features and fixes.  As a bonus, the email path gets a nice tweak that
the Zulip path has had for years, since f7f2ec0ac, which makes the
emails clearer and less broken-looking when logging a message with no
stack trace.
2017-12-22 11:52:13 -05:00
Greg Price b15231dfc2 logging: Rename AdminZulipHandler to AdminNotifyHandler.
This name hasn't been right since f7f2ec0ac back in 2013; this handler
sends the log record to a queue, whose consumer will not only maybe
send a Zulip message but definitely send an email.  I found this
pretty confusing when I first worked on this logging code and was
looking for how exception emails got sent; so now that I see exactly
what's actually happening here, fix it.
2017-12-22 11:52:13 -05:00
neiljp (Neil Pilgrim) 7c819f4d90 mypy: Account for Optional variables in addressee.py. 2017-12-23 01:22:22 +09:00
neiljp (Neil Pilgrim) 57e7ec94a0 mypy: Adjust 2 emoji.py functions to allow for strict-optional. 2017-12-23 01:22:22 +09:00
neiljp (Neil Pilgrim) d01dcbd018 mypy: Amend tos_version parameter of create_user_profile to be Optional.
tos_version is marked as Optional[Text] in UserProfile & is Optional
in other places within zerver/lib/create_user.py
2017-12-23 01:22:22 +09:00
Aastha Gupta 78e8d18ef3 integrations: Rename HUBOT_INTEGRATIONS_LEGACY.
It is now simply called HUBOT_INTEGRATIONS.

Fixes #7402
2017-12-22 10:38:47 -05:00
Angelika Serwa 2f575cca72 Add Dropbox webhook integration.
This is just a basic Dropbox webhook integration. It just
notifies a user when something has changed, it does not
specify what changed. Doing so would require storing data,
as Dropbox API was created mainly for file managers, not
integrations like this.
Closes #5672
2017-12-22 10:36:21 -05:00
Marco Burstein 9327048f6f mypy: Use Python 3 type syntax in `zerver/lib/events.py`. 2017-12-22 08:17:00 -05:00
Marco Burstein 017476cdfb mypy: Use Python 3 type syntax in `zerver/lib/create_user.py`. 2017-12-22 08:17:00 -05:00
Tommy Ip 9ea7b05cfe bugdown: Rewrite markdown link to relative link conditionally. 2017-12-22 08:10:42 -05:00
Tommy Ip efe2d497f4 bugdown: Avoid matching uri with regex. 2017-12-22 08:10:42 -05:00
Tommy Ip 9ce797d68c bugdown: Extract rewrite_if_relative_link function. 2017-12-22 08:10:42 -05:00
Umair Khan 898acc4278 missed-message-worker: Send email through email_senders.
We have shifted to a generic queue to send all the emails. This queue
can retry in case of network issues; this makes sure that the emails are
always sent.
2017-12-20 19:36:27 -08:00
neiljp (Neil Pilgrim) 25d5a2ee1c requirements: Upgrade mypy to 0.560.
Fixes #7835.
2017-12-20 18:09:36 -08:00
snlkapil 1589c597d2 settings: Show local time instead of UTC in invitations tab. 2017-12-20 07:08:46 -05:00
Rishi Gupta 27a8ef86e4 actions: Add realm deactivation and reactivation to RealmAuditLog. 2017-12-19 17:46:36 -08:00
Rishi Gupta 4261307e27 emails: Clear ScheduledEmail emails on realm deactivation.
This only affects the invitation reminder emails; clear_scheduled_emails was
already clearing the other two types.
2017-12-19 17:46:36 -08:00
Rishi Gupta 869b4d41ef models: Add ScheduledEmail.realm.
The two extra queries in the test are due to the assert in
send_future_email.
2017-12-19 17:46:36 -08:00
Andy Perez 44e6d9b54b testing: Add nice diffs to markdown python tests. 2017-12-18 19:03:38 -05:00
Greg Price 776af2e248 push notifs: Simplify content-truncation code slightly.
This should be a pure refactor.
2017-12-18 15:32:56 -08:00
Greg Price 014900c2e5 push notifs: Include same stream and sender info on GCM as on APNs.
This takes the information added to APNs payloads by #7080, and adds
to our GCM payloads the parts that it didn't already have.
2017-12-18 15:32:56 -08:00
Greg Price e077d6fc74 push notifs: Factor out common payload fields for APNs and GCM.
This should be a pure refactor, producing exactly the same payloads.
2017-12-18 15:32:56 -08:00
Callum Fraser 0438da5c16 mypy: Use Python 3 type syntax in zerver/lib/avatar.py. 2017-12-18 11:32:52 -05:00
Alena Volkova 3cfb2d57b2 tests: Refactor the method for encoding credentials. 2017-12-18 09:24:09 -05:00
Alena Volkova 4a303d1778 tests: Add auth methods that will replace passing credentials. 2017-12-18 09:24:09 -05:00
Andy Perez f0190333b8 bugdown: Remove whitespace strip from inline code.
Fixes #4507
2017-12-14 07:48:10 -05:00
Vishnu Ks bc37317914 invites: Differentiate users invited as admin by showing bolt icon. 2017-12-13 15:12:21 -06:00
Greg Price b830b446f1 logging: Reduce `create_logger` to new `log_to_file`.
The name `create_logger` suggests something much bigger than what this
function actually does -- the logger doesn't any more or less exist
after the function is called than before.  Its one real function is to
send logs to a specific file.

So, pull out that logic to an appropriately-named function just for
it.  We already use `logging.getLogger` in a number of places to
simply get a logger by name, and the old `create_logger` callsites can
do the same.
2017-12-12 17:17:08 -08:00
Greg Price 1c810baaef logging: Stop calling basicConfig in `create_logger`.
From the docs:

> This function does nothing if the root logger already has handlers
> configured for it.

Which we do if we've started up Django and configured settings, and in
particular allowed Django to process `settings.LOGGING`.

So, cut it out -- all it can do is confuse people about how logging
works.

If we ever actually used the `log_format` parameter, this would be
doubly confused, because only the first call would have any effect.
2017-12-12 17:17:08 -08:00
Greg Price b6491fd52a logging: Eliminate log_level from create_logger.
Because calls to `create_logger` generally run after settings are
configured, these would override what we have in `settings.LOGGING` --
which in particular defeated any attempt to set log levels in
`test_settings.py`.  Move all of these settings to the same place in
`settings.py`, so they can be overridden in a uniform way.
2017-12-12 17:17:08 -08:00
Greg Price ebcf0b4876 logging: Stop having `create_logger` force loglevels to INFO.
This is already the loglevel we set on the root logger, so this has no
effect -- except in tests, where `test_settings.py` attempts to set
some of these same loggers to higher loglevels.  Because the
`create_logger` call generally runs after we've configured settings,
it clobbers that effect.

The code in `test_settings.py` that tries to suppress logs only works
because it also sets `propagate=False`, which has nothing to do with
loglevels but does cause logs at this logger (and descendants) to be
dropped completely unless we've configured handlers for this logger
(or one of its relevant descendants.)
2017-12-12 17:17:07 -08:00
Weronika Grzybowska b54f78968f markdown: Add automatic numbering of ordered lists.
Adds a markdown preprocessor that finds ordered lists where all items
use the same number and change them to be in normal increasing order,
starting with that number.

Fixes #5159.
2017-12-12 18:35:38 -06:00
Rhea Parekh dd1b14378b Streams: Disallow certain markdown characters in stream names. 2017-12-12 10:04:31 -06:00
Greg Price 8a432c5134 APNs: Update comment on versions of our APNs payload format. 2017-12-11 20:55:20 -08:00
kunall17 98943a8333 Include data of stream/private sender details in apns push notifications 2017-12-11 20:51:09 -08:00
picapi_ 85ae723c9f mypy: Use Python 3 type syntax in zerver/lib/soft_deactivation.py. 2017-12-11 20:30:19 -08:00
Greg Price e9a7173119 actions: Cut some redundant types in docstrings.
We have a better way of writing types now!

Also move a comment that should have been a docstring,
to being a docstring.
2017-12-11 19:29:11 -08:00
Shreyansh Dwivedi 47fcb27e39 invitations: Remove custom_body.
Fixes #7672
2017-12-11 19:23:54 -08:00
Steve Howell f83a73c763 mypy: Use more specific type for _replacement_destroy_test_db. 2017-12-08 19:00:18 -08:00
Steve Howell 78041a5846 mypy: Use more specific type for instrument_url wrapper. 2017-12-08 19:00:18 -08:00
Xavier Cooney a831acd97f mypy: Use Python 3 type syntax in bugdown/__init__.py. 2017-12-08 18:34:26 -08:00
Vishnu Ks a30152cb94 models: Use is_cross_realm_bot_email whenever possible.
Fixes #7557
2017-12-07 14:55:06 -08:00
Rishi Gupta 6787fbd547 invitations: Change rate limiting to use invites_sent::day.
[greg: small tweaks]
2017-12-06 20:35:50 -08:00
Rishi Gupta fbd8dde1f8 invitations: Add LoggingCountStat to keep track of sent invitations. 2017-12-06 20:35:50 -08:00
Rishi Gupta cb5eb47470 invitations: Move resend/revoke error handling out of actions.py. 2017-12-06 20:35:50 -08:00
Rishi Gupta 968aae167b invitations: Remove get_prereg_user_by_email.
The original logic is buggy now that emails can belong to (and be
invited to) multiple realms.

The new logic in the `invites` queue worker also avoids the bug where
when the PreregistrationUser was gone by the time the queue worker got
to the invite (e.g., because it'd been revoked), we threw an exception.

[greg: fix upgrade-compatibility logic; add test; explain
revoked-invite race above]
2017-12-06 20:35:50 -08:00
Rishi Gupta 8baadc6b15 invitations: Add check for invite limit to resend invite pathway. 2017-12-06 20:30:42 -08:00
Greg Price b3482429f1 invitations: Adjust a few blank lines for clarity and compactness. 2017-12-06 20:30:42 -08:00
Rishi Gupta cb527fd3c6 invitations: Move spammer check from do_invite_users into its own function. 2017-12-06 20:30:42 -08:00
Rishi Gupta e48bc9136a invites: Change resend invitation pathway to use existing code.
This code changes frequently enough that errors are bound to creep in.  The
main change is that this sends the original invitation email instead of the
reminder email, but I think that's fine.
2017-12-06 20:30:42 -08:00
picapi_ 21e629f351 mypy: Use Python 3 type syntax in zerver/lib/test_classes.py. 2017-12-05 16:32:18 -08:00
Rishi Gupta 7d1c88f0fb invitations: Limit realms to 100 invites per day.
To guard against using zulip invites as a vector for spam. Stopgap measure
until we figure out something better.
2017-11-29 22:18:05 -08:00
Greg Price 4926228071 rabbitmq: Do a better job of retrying failed connections.
Empirically, the retry in `_on_connection_closed` didn't actually work
-- if a reconnect failed, that was it, and the exception handler
didn't get run.  A traceback would get logged, but all its frames were
in Tornado or Pika, not our own code; presumably something magic and
async was happening to the exception.

Moreover, though we would make one attempt to reconnect if we had a
connection that got closed, we didn't have any form of retry if the
original attempt at connecting failed in the first place.

Happily, upstream offers a perfectly reasonable bit of API that avoids
both of these problems: the on-open-error callback.  So use that.
2017-11-29 16:56:29 -08:00
Greg Price 7ac2b58584 rabbitmq: Reorder a bit to group our reconnect logic together. 2017-11-29 16:56:29 -08:00
Greg Price c32b16715d tornado: Use spiffy new `call_later` rather than `add_timeout`.
This method was new in Tornado 4.0.  It saves us from having to get
the time ourselves and do the arithmetic -- which not only makes the
code a bit shorter, but also easier to get right.  Tornado docs (see
http://www.tornadoweb.org/en/stable/ioloop.html) say we should have
been getting the time from `ioloop.time()` rather than hardcoding
`time.time()`, because the loop could e.g. be running on the
`time.monotonic()` clock.
2017-11-29 16:56:29 -08:00
Greg Price 73886f57d2 zerver/lib/queue: Clean up import order. 2017-11-29 16:56:29 -08:00
Greg Price 3c4e4c14c9 rabbitmq: Add on-close callback atomically in creating the connection.
Adding it afterward is inherently racy, and upstream's API is quite
reasonable for avoiding that -- just like we can pass an on-open
callback up front, we can do the same with the on-close callback.

This is a more thorough version of 4adf2d5c2 from back in 2013-04.
2017-11-29 16:56:29 -08:00
Greg Price e88c2a7ee4 rabbitmq: Cut redundant `stop_ioloop_on_close` parameter.
The default value of this parameter is already False upstream.
(It was already False in pika version 0.9.6, which we were
supposedly using when we introduced this in 4baeaaa52; not sure
what the story was there.)
2017-11-29 16:56:29 -08:00
Tim Abbott be832387c5 user_groups: Fix reporting of empty user groups.
Previously, we weren't doing a proper left join in
user_groups_in_realm_serialized, resulting in empty user groups being
excluded from the query.  We want to leave decisions about excluding
empty user groups to the UI layer, so we include these here.
2017-11-29 16:13:55 -08:00
Tim Abbott b2cb443d24 subs: Fix clearing unread counts when leaving private streams.
Because we use access_stream_by_id here, and that checks for an active
subscription to interact with a private stream, this didn't work.

The correct fix to add an option to active_stream_by_id to accept an
argument indicating whether we need an active subscription; for this
use case, we definitely do not.
2017-11-29 14:40:08 -08:00
Tim Abbott 6bc1a682de notifications: Separate push and email notifications checks.
This is an early step in a larger refactor to properly decouple the
email and push notification code paths.
2017-11-28 17:51:18 -08:00
rht ee546a33a3 zerver/lib: Use python 3 syntax for typing.
Edited by tabbott to improve various line-wrapping decisions.
2017-11-28 17:15:14 -08:00
rht 229a8b38c0 zerver/lib: Use Python 3 syntax for typing for several files.
This adds a number of annotations that had been missed in previous
passes.
2017-11-28 17:02:24 -08:00
Tim Abbott 586e18b237 auth: Allow accounts with the same email in different realms.
[Modified by greg to (1) keep `USERNAME_FIELD = 'email'`,
(2) silence the corresponding system check, and (3) ban
reusing a system bot's email address, just like we do in
realm creation.]
2017-11-28 16:23:10 -08:00
Tim Abbott e6f460f511 auth: Replace user_email_is_unique validator.
As we migrate to allow reuse of the same email with multiple realms,
we need to replace the old "no email reuse" validators.  Because
stealing the email for a system bot would be problematic, we still ban
doing so.

This commit only affects the realm creation logic, not registering an
account in an existing realm.
2017-11-28 16:23:10 -08:00
Greg Price b6cc21b438 debug: Add facility to dump tracemalloc snapshots.
Originally this used signals, namely SIGRTMIN.  But in prod, the
signal handler never fired; I debugged fruitlessly for a while, and
suspect uwsgi was foiling it in a mysterious way (which is kind of
the only way uwsgi does anything.)

So, we listen on a socket.  Bit more code, and a bit trickier to
invoke, but it works.

This was developed for the investigation of memory-bloating on
chat.zulip.org that led to a331b4f64 "Optimize query_all_subs_by_stream()".

For usage instructions, see docstring.
2017-11-28 15:52:07 -08:00
derAnfaenger c8a5ae753c embedded bots: Consistently use 'storage' instead of 'state.' 2017-11-27 21:05:34 -08:00
Tim Abbott 9645c8b31f integrations: Remove disabled googlesearch integration.
This was moved to another part of the python-zulip-api codebase
because it was unfinished.
2017-11-27 20:50:37 -08:00
Vishnu Ks 985768b2fd registration: Check realm against PreregistrationUser realm.
We would allow a user with a valid invitation for one realm to use it
on a different realm instead.  On a server with multiple realms, an
authorized user of one realm could use this (by sending invites to
other email addresses they control) to create accounts on other
realms. (CVE-2017-0910)

With this commit, when sending an invitation, we record the inviting
user's realm on the PreregistrationUser row; and when registering a
user, we check that the PregistrationUser realm matches the realm the
user is trying to register on.  This resolves CVE-2017-0910 for
newly-sent invitations; the next commit completes the fix.

[greg: rewrote commit message]
2017-11-27 14:58:26 -08:00
Tim Abbott 8b935f4e99 settings: Add setting for SYSTEM_BOT_REALM.
This fixes some subtle JavaScript exceptions we've been getting in
zulipchat.com, caused by the system bot realm there not being "zulip"
interacting with get_cross_realm_users.
2017-11-27 14:46:07 -08:00
Tim Abbott 25fd4c5508 bulk_get_users: Edit the cache keys to make them more unique.
While at this point I was to rewrite this function, this at least
plugs the issues for now.
2017-11-27 14:41:31 -08:00
Tim Abbott 339b67f7af get_cross_realm_dicts: Harden against caching bug in last commit.
This should help protect us from future issues with the way that
`bulk_get_users` does caching.

It's likely that we'll want to further restructure `bulk_get_users` to
not have this base_query code path altogether (since it's kinda
buggy), but I'm going to defer that for a time when we have another
user.
2017-11-27 14:35:50 -08:00
Tim Abbott 646ba5b9e5 bulk_get_users: Fix issues with users in multiple realms.
The previous implementation had a subtle caching bug: because it was
sharing its cache with the `get_user_profile_by_email` cache, if a
user happened to have an email in that cache, we'd return it, even
though that user didn't match `base_query`.

This causes `get_cross_realm_users` to no longer have a problematic
caching bug.
2017-11-27 14:34:45 -08:00
Rohitt Vashishtha 6734b00c0b markdown: Hide URL if message is only an image.
Hides URL if the message content == image url so that sending gifs or
images feels less cluttered. Uses the url_to_a() function to generate
the expected url string for matching.

Fixes #7324.
2017-11-27 13:30:18 -08:00
Tim Abbott 649b7b8b9f internal_send_message: Enforce being passed cross-realm bots.
We include ERROR_BOT in this set, even though it's not technically
cross-realm (it just lives in the admin realm).

This code path does not correctly handle emails that correspond to
multiple accounts (because `get_system_bot` does not).  Since it's
intended to only be used by system bots, we add an appropriate
assertion to ensure it is only used for system bots.
2017-11-26 17:14:23 -08:00
Tim Abbott ae8e203717 actions: Merge internal_prep_message into internal_send_message.
We also document the fact that only system bots are supported here.
2017-11-26 17:14:23 -08:00
Tim Abbott 6952dcbdac embedded bots: Stop using internal_send_message for non-system-bots.
This was causing problems, because internal_send_message assumes that
there is a unique user (across all realms) with the given email
address (which is sorta required to support cross-realm bot messages
the way it does).

With this change, it now, in practice, only sends cross-realm bot
messages.
2017-11-26 17:14:23 -08:00
Tim Abbott 27582782fa embedded bots: Flip around ordering of rate-limiting check. 2017-11-26 17:14:23 -08:00
Tim Abbott 5306a9634d email_mirror: Rewrite to not use internal_send_message.
This was causing problems with the fact that `get_system_bot` now only
works for actual system bot users.
2017-11-26 17:14:23 -08:00
Tim Abbott 2210f627a5 signup: Switch active mirror-dummy users to an AssertionError.
Previously, this was a ValidationError, but that doesn't really make
sense, since this condition reflects an actual bug in the code.

Because this happened to be our only test coverage the ValidationError
catch on line 84 of registration.py, we add nocoverage there for now.
2017-11-26 16:13:45 -08:00
Vishnu Ks b9bc1c2b33 Eliminate get_user_profile_by_email from test_classes. 2017-11-26 15:47:56 -08:00
Tim Abbott 9fcdb6c83a notifications: Use call_consume_in_tests for email sending.
This adds a bit of test coverage with really no downside.
2017-11-26 11:46:07 -08:00
Robert Hönig 0e0a8a2b14 queue processor tests: Call consume by default.
This significantly improves the API for queue_json_publish to not be
overly focused on what the behavior of this function should be in our
unit tests.
2017-11-26 11:45:34 -08:00
Tim Abbott 6968b540c8 mypy: Remove type: ignores that are unnecessary with new mypy.
Mostly these can be removed because the broken LXML stubs were removed
from typeshed in https://github.com/python/typeshed/issues/525.
2017-11-25 10:06:28 -08:00
Vaibhav Sagar 711f7bca3b upload: Fix typo in animated GIF error message.
s/must be have/must have the/.
2017-11-24 09:08:38 -08:00
Andy Perez 0449dc8b8a bugdown: Fix handling of nested fenced math blocks.
Change any fence delimiters (backticks and tildes) to be
interpreted literally inside of math fenced_code blocks.

Fixes #5190.
2017-11-22 12:19:43 -08:00
Harshit Bansal 2e7d0cb4cb digests: Fix new stream links.
Fixes: #7479.
2017-11-22 10:07:36 -08:00
Tim Abbott 8f6b39a1c0 mypy: Fix annotations for json_encoder_for_html. 2017-11-22 01:32:46 -08:00
rht e169bb0954 zerver: Remove simplejson dependency.
Modified by tabbott to put the third-party code in a new file.

Fixes #6970.
2017-11-21 21:51:48 -08:00
Tim Abbott b4e67fac36 bot_lib: Fix storage bug introduced by rebase error. 2017-11-21 21:37:54 -08:00
derAnfaenger e526d0c144 embedded bots: Add views to access state. 2017-11-21 21:10:39 -08:00
derAnfaenger d2af8d4cbd embedded bots: Support multi-entry state modification. 2017-11-21 21:10:39 -08:00
derAnfaenger cbadd3d430 embedded bots: Simplify StateError exception messages. 2017-11-21 21:10:12 -08:00
Tim Abbott 141cf21b86 bulk_create: Fix buggy logic for generating recipients_by_email.
This buggy logic from e1686f427c had
broken do-destroy-rebuild-test-database.

Now that we're not just trying to add the Recipient objects for every
user on the system here to profiles_by_id, we also shouldn't be
processing every Recipeint object on the system.  The fix is simple:
because of the patch we got merged into Django upstream,
recipients_to_create actually has the object IDs added to the
Recipient objects passed into Recipient.objects.bulk_create.

This was missed in manual testing, since it only broke `populate_db
--test-suite`.
2017-11-21 21:07:32 -08:00
rht 3f4bf2d22f zerver/lib: Use python 3 syntax for typing.
Extracted from a larger commit by tabbott because these changes will
not create significant merge conflicts.
2017-11-21 20:56:40 -08:00
rht 561ba33f69 zerver/lib: Use python 3 syntax for typing.
Split by tabbott from a larger commit; this covers a batch of files
with no open PRs touching them.
2017-11-21 20:45:52 -08:00
Eeshan Garg 73d0f1db81 lib/integrations: Remove Integration.doc_context.
An Integration object doesn't need access to the context dict used
to render its doc.md, since the context dict is just passed directly to
render_markdown_path.
2017-11-21 20:39:30 -08:00
Tim Abbott e1686f427c bulk_create: Remove assumption that UserProfiles are globally unique.
This isn't used in production, but that could change in the future,
and the fix to make this limited to a single realm is pretty simple.
2017-11-21 20:23:06 -08:00
Vishnu Ks 766511e519 actions: Mark all messages as read when user unsubscribes from stream.
This fixes a bug where, when a user is unsubscribed from a stream,
they might have unread messages on that stream leak.  While it might
seem to be a minor problem, it can cause significant problems for
computing the `unread_msgs` data structures, since it means we need to
add an extra filter for whether the user is still subscribed, either
in the backend or in the UI.

Fixes #7095.
2017-11-21 20:09:17 -08:00
Tim Abbott 3bfb19b5f3 Convert EmailAuthBackend and LDAPAuthBackend to accept a realm. 2017-11-21 18:23:50 -08:00
Vishnu Ks 34689370cd settings: Create UI and backend for setting signups stream. 2017-11-21 17:39:51 -08:00
Vishnu Ks 3d2c9c6098 models: Replace core team with Realm.INITIAL_PRIVATE_STREAM_NAME. 2017-11-21 17:39:51 -08:00
Vishnu Ks a0275a6257 actions: Send new user messages to signup_notifications_stream. 2017-11-21 17:39:50 -08:00
Rishi Gupta 42652713c8 onboarding: Refactor away setup_initial_private_stream.
It is a confusing name, now that the stream creation part has been taken
away.
2017-11-21 17:39:50 -08:00
Vishnu Ks 3d17897563 onboarding: Move signup notifications stream creation to do_create_realm. 2017-11-21 17:39:50 -08:00
Tim Abbott 29408d24ff emoji: Rework reactions validation to have a single function.
This feels more readable to me, and also identified a couple cases
where we were missing test coverage.
2017-11-20 15:27:21 -08:00
Harshit Bansal 87b523f3ef emoji.py: Refactor emoji data consistency checking functions.
Inorder to provide more explicit error messages I have merged the
`emoji_code_is_valid()` and `emoji_name_is_valid()` functions into
`check_emoji_code_consistency()` and `check_emoji_name_consistency()`
respectively.
2017-11-20 15:27:21 -08:00
Tim Abbott 25ddba99f6 lint: Ban general use of user_profile.save().
This often can cause minor caching problems.

Obviously, it'd be better if we had access to the AST and thus could
do this rule for UserProfile objects in general.
2017-11-20 10:57:08 -08:00
Eeshan Garg 9138d3b8d7 integrations: Move integration-specific context to integration_doc.
Instead of populating the context dict with integration-specific
information in render_markdown_path, we now do that in
zerver.views.integrations.integration_doc instead.

Fixes #7401.

Tweaked by tabbott to use cast to handle the typing issues here.
2017-11-19 20:48:46 -08:00
rht 33b1a541d7 zerver/lib: Use python 3 syntax for typing.
With tweaks by tabbott to fix line spacing.
2017-11-18 16:09:04 -08:00
Tim Abbott 81adc92db6 actions: Use get_system_bot for fetching WELCOME_BOT. 2017-11-17 17:25:56 -08:00
Tim Abbott 1ed50ee858 tests: Pass a realm option to login for non-zulip realms.
This better matches the model of how having multiple realms should
work: you need to specify which realm you're logging into.
2017-11-17 15:32:42 -08:00
Greg Price bd5905da5d error_notify: Use `git describe --always`.
This way, even if the repo lacks any tags, we still
at least get the commit ID.
2017-11-17 14:19:20 -08:00
Andy Perez 83f563aeff bugdown: Properly parse empty blockquotes.
The character ">" now only starts a blockquote if the resulting
blockquote would be non-empty.  Thus, by itself, ">" is now
interpreted literally by bugdown, fixing #687.  The message
with contents consisting of ">>>" is now parsed as a doubly
(not triply) nested blockquote with contents ">".  Properly
formed blockquotes have identical behavior as before, but now
bugdown can no longer produce empty blockquotes as output.

Fixes #2886, #687.
2017-11-17 11:41:44 -08:00
derAnfaenger 395f1e9270 embedded bots: Add database config storage.
Storage limititations are only set on the value of
a config entry, since this is the only user-accessible
part of the schema. Keys are statically set by each
embedded bot.
2017-11-16 23:06:38 -08:00
Vishnu Ks 068912ca7d tests: Add test for default stream group signup flow. 2017-11-16 21:17:31 -08:00
Vishnu Ks cc553125a1 tests: Move check_user_subscribed_only_to_streams to test_classes ZulipTestCase. 2017-11-16 21:17:31 -08:00
Harshit Bansal d9c2f613e3 api: Add new endpoint for reactions.
This endpoint will allow us to add/delete emoji reactions whose emoji
got renamed during various emoji infra changes. This was also a
required change for realm emoji migration.

This commit was tweaked significantly by tabbott for greater clarity
(with no changes to the actual logic).
2017-11-16 20:52:15 -08:00
Greg Price 3828305305 error_notify: Try `git describe` too.
Tested for graceful degradation both with `git` not existing and the
`.git` directory not existing.
2017-11-16 19:15:12 -08:00
Greg Price 41ecdd8d83 error_notify: Add some idea of the deployed code version.
Tested in dev with the help of our handy /emails/ display.
2017-11-16 19:15:12 -08:00
Greg Price 4b5c52fc99 queue: Prevent an AttributeError rather than swallowing it.
When the RabbitMQ server disappears, we log errors like these:

```
Traceback (most recent call last):
  File "./zerver/lib/queue.py", line 114, in json_publish
    self.publish(queue_name, ujson.dumps(body))
  File "./zerver/lib/queue.py", line 108, in publish
    self.ensure_queue(queue_name, do_publish)
  File "./zerver/lib/queue.py", line 88, in ensure_queue
    if not self.connection.is_open:
AttributeError: 'NoneType' object has no attribute 'is_open'

During handling of the above exception, another exception occurred:
[... traceback of connection failure inside the retried self.publish()]
```

That's a type error -- a programming error, not an exceptional
condition from outside the program.  Fix the programming error.

Also move the retry out of the `except:` block, so that if it also
fails we don't get the exceptions stacked on each other.  This is a
new feature of Python 3 which is sometimes indispensable for
debugging, and which surfaced this nit in the logs (on Python 2 we'd
never see the AttributeError part), but in some cases it can cause a
lot of spew if care isn't taken.
2017-11-16 18:33:14 -08:00
Rhea Parekh 8c4651d716 zerver/lib/actions.py: Fix translation bugs.
Fixes #7396.
2017-11-16 11:05:49 -08:00
Tim Abbott 054952a44a docs: Update links from codebase to point to ReadTheDocs. 2017-11-16 10:53:49 -08:00
David Rosa Tamsen 7072fa5b37 docs: Reorganize developer docs to improve navigation.
This commit helps reduce clutter on the navigation sidebar.
Creates new directories and moves relevant files into them.
Modifies index.rst, symlinks, and image paths accordingly.

This commit also enables expandable/collapsible navigation items,
renames files in docs/development and docs/production,
modifies /tools/test-documentation so that it overrides a theme setting,
Also updates links to other docs, file paths in the codebase that point
to developer documents, and files that should be excluded from lint tests.

Note that this commit does not update direct links to
zulip.readthedocs.io in the codebase; those will be resolved in an
upcoming follow-up commit (it'll be easier to verify all the links
once this is merged and ReadTheDocs is updated).

Fixes #5265.
2017-11-16 09:45:08 -08:00
Harshit Bansal 3b95a6d2d9 bugdown: Fix a bug in the updation logic of markdown engines.
While fixing an issue related to email gateway messages not getting
rendered properly, I unknowingly introduced a bug in the markdown
engine updation code. This commit fixes it. The issue was that for
a realm having email gateway setup, updation of realm filters would
lead to the updation of only one of the markdown engines not both.
2017-11-15 22:47:11 -08:00
Harshit Bansal b849c5e551 bugdown: Remove an unnecessary check in `do_convert()`. 2017-11-15 22:47:11 -08:00
Harshit Bansal 5e5bb02e39 bugdown: Remove `make_realm_filters()`.
This commit removes `make_realm_filters()` and merges its functionality
in `make_md_engine()` and `maybe_update_markdown_engines()`.
2017-11-15 22:47:11 -08:00
Harshit Bansal f9f9e92443 bugdown: Correct the descriptions of `Bugdown` config. 2017-11-15 22:47:11 -08:00
Harshit Bansal e615a2ab4f bugdown: Simplify the "opts" parameter of `make_md_engine()`. 2017-11-15 22:47:11 -08:00
Harshit Bansal e3f2f67d0e bugdown: Rename `maybe_update_realm_filters()`.
This commit renames `maybe_update_realm_filters()` to
`maybe_update_markdown_engines()` which sounds closer
to the semantics.
2017-11-15 22:47:11 -08:00
Umair Khan 98be0cc502 user-groups: Send delete user group event.
Fixes #7380
2017-11-15 22:35:43 -08:00
Umair Khan cc76f7deac user-groups: Create check_delete_user_group. 2017-11-15 22:35:43 -08:00
Umair Khan 79f4a7627f user-groups: Send remove members event. 2017-11-15 22:35:23 -08:00
Umair Khan cec3f19366 user-groups: Send add members event. 2017-11-15 22:35:23 -08:00
Umair Khan eb48fab495 user-groups: Send description update event. 2017-11-15 22:35:23 -08:00
Umair Khan 5571122120 user-groups: Send name update event. 2017-11-15 22:35:23 -08:00
Umair Khan 912505317a user-groups: Send create group event. 2017-11-15 22:35:23 -08:00
Umair Khan 31efe951b7 user-groups: Fix function argument bug; pass group.
In remove_members_from_group_backend, we are passing user group to
remove_members_from_user_group. In remove_members_from_user_group,
expect user_group_id.
2017-11-15 22:35:23 -08:00
Umair Khan 66179ea295 bulk_add_members_to_user_group: Pass user group.
Previously we passed user group id. If we pass user group, send event is
easier.
2017-11-15 22:35:23 -08:00
Umair Khan 5ef3974e80 user_groups_in_realm_serialized: Sort groups. 2017-11-15 22:35:23 -08:00
Tim Abbott f02e5b90f6 cross_realm: Use bulk_get_users to fix handling of missing users.
This fixes a regression in ae5ba7f4fd,
where Zulip would 500 if the newly added system bots didn't exist on
the server.

This also fixes a moderate size performance problem where we'd fetch 5
users from memcached or the database in a loop.
2017-11-15 21:24:51 -08:00
Tim Abbott c7a975e4df users: Move check_change_full_name to actions.py.
This avoids an import loop in the next commit, and better matches our
usual code structure.
2017-11-15 17:39:09 -08:00
Vishnu Ks 6f955fbf34 actions: Replace stream.add with stream.append in process_new_human_user.
stream is a list not a set. It used to be set
and was changed to stream before getting merged.
2017-11-15 17:06:43 -08:00
Tim Abbott 6a5b965a90 integrations: Rename HUBOT_LOZENGES to HUBOT_INTEGRATIONS. 2017-11-15 14:04:19 -08:00
Aastha Gupta 3b53c692a4 integrations: Add integrations/doc/assembla.
Significantly modified by tabbott and rishig to be more maintainable.

Fixes #4542.
2017-11-15 14:04:19 -08:00
Tim Abbott 85844107fd integrations: Remove code for legacy Hubot lozenges.
We no longer display the only model anyway.
2017-11-15 14:04:19 -08:00
rht 09af29b051 zerver/lib: Text-wrap long lines exceeding 110. 2017-11-15 10:58:03 -08:00
Tim Abbott 5aa6751e7a lint: Fix long lines in newly added features. 2017-11-15 10:58:03 -08:00
Vishnu Ks d2555e8769 api: Add support to change name of default stream groups. 2017-11-14 14:41:42 -08:00
Vishnu Ks 2af249dd5d api: Add description attribute to default stream group. 2017-11-14 14:41:42 -08:00
Vishnu Ks f45ba7de93 api: Pass group id instead of name to default stream group api. 2017-11-14 14:41:42 -08:00
rht cceba2023c Use makedirs directly instead of wrapping it in mkdirs. 2017-11-14 09:47:09 -08:00
rht 72ce1c6501 Use a more concise `mkdir -p` when the situation permits. 2017-11-14 09:47:09 -08:00
derAnfaenger 25bf15a3c9 bots: Use non-generic exception for state key removal. 2017-11-10 18:56:26 -08:00
derAnfaenger 8fccf36246 bots: Wrap StateError exceptions around BotUserData exceptions. 2017-11-10 18:56:26 -08:00
derAnfaenger d707e3bc1d bots: Add USER_STATE_SIZE_LIMIT setting. 2017-11-10 18:56:26 -08:00
derAnfaenger 545c73c79e bots: Rename StateHandlerError to StateError. 2017-11-10 18:54:54 -08:00
derAnfaenger 71372610e3 bots: Validate state size limit directly in set_bot_state(). 2017-11-10 18:54:39 -08:00
Andy Perez 4d51a6cfac bugdown: Fix $$ whitespace handling.
The intended use of $$ is for inline expressions, not for multiline
ones; ```math is an acceptable alternative for the latter.  Hence,
the $$-syntax for inline TeX no longer permits newlines within it.

This was also necessary for the next change to be sensible; namely
allowing for spaces around both $$ when crafting inline TeX instead of
forcing everything to be crammed together, e.g. $$x=7$$.  In order to
avoid uninentionally creating inline expressions, the opening and
closing $$'s of an inline expression must now both exactly consist of
two dollar signs, no more and no less.

Fixes: #6488.
2017-11-10 16:54:27 -08:00
Tim Abbott a23378a304 check_short_string: Verify string before checking length.
While the previous implementation was correct, this should give better
error messages.
2017-11-10 16:37:43 -08:00
neiljp (Neil Pilgrim) 72eba6a0ce mypy: Adjust validator.py to use object rather than Any. 2017-11-10 16:37:29 -08:00
neiljp (Neil Pilgrim) 331a3c3447 mypy: Migrate validator.py typing to python3 syntax. 2017-11-10 16:32:25 -08:00
rht 8e87eb4084 zerver/lib/bugdown: Text-wrap long lines exceeding 110. 2017-11-10 16:29:32 -08:00
rht 23a481c5d2 zerver/lib/actions.py: Text-wrap long lines exceeding 110. 2017-11-10 16:27:22 -08:00
Vishnu Ks f44b60a150 Implement API for default stream groups. 2017-11-10 16:05:36 -08:00
Steve Howell f54146b6d8 mypy: Add RawUnreadMessagesResult type. 2017-11-10 10:50:57 -08:00
Steve Howell c799ec0ca6 minor: Inline get_unread_message_ids_per_recipient().
This method was only used in one test.
2017-11-10 10:50:57 -08:00
Umair Khan 4c6a376fd5 user-groups: Add groups to page_params. 2017-11-09 17:35:46 -08:00
Umair Khan b1603d289c user-groups: Add update members API endpoint.
Significantly modified by tabbott to fix some bugs.
2017-11-09 17:35:37 -08:00
Umair Khan eeaf0f1742 user-groups: Add edit API endpoint.
Significantly modified by tabbott to fix some bugs.
2017-11-09 17:35:22 -08:00
Umair Khan 1bbe84af49 user-groups: Add create API endpoint.
Significantly modified by tabbott for better security structure.
2017-11-09 17:26:14 -08:00
Umair Khan 39ca38837e user-groups: Add description to groups. 2017-11-09 16:35:52 -08:00
Umair Khan 18aecb82d7 user-groups: Get members from membership. 2017-11-09 16:18:40 -08:00
Tommy Ip bcd575d8de bugdown: Rewrite links to local server to be relative links.
Fixes #7247
2017-11-09 15:59:31 -08:00
Tim Abbott 982bd9c38d Fix push notifications for soft-deactivated users.
Previously, these push notification events were being generated, but
then ignored in handle_push_notification because there was no
user_message object.
2017-11-09 15:52:29 -08:00
Tim Abbott c99d20db63 push_notifications: Shrink scope of try/except block.
There's no reason this needed to indent the entire function.
2017-11-09 15:30:21 -08:00
Tim Abbott 64b4d83038 push_notifications: Improve debugging of lookup problems.
We're having this error being thrown often, and don't know why.
2017-11-09 15:20:24 -08:00
Steve Howell 9b723c7a32 Use encode() in zerver/lib/logging_util.py. 2017-11-09 10:43:20 -08:00
Steve Howell 3158ed01bf Remove force_str() in upload.py. 2017-11-09 10:43:20 -08:00
Steve Howell 531a3ee9de refactor: Fix parse.quote(force_str(s)) calls in bugdown. 2017-11-09 10:43:20 -08:00
Steve Howell 8eaeba8615 refactor: Remove dict_with_str_keys().
This function is no longer needed in python3, as it was decoding
keys that already came for a JSON-decoded string.
2017-11-09 10:43:20 -08:00
Steve Howell b0bb09cdb8 Remove force_text() in get_signed_upload_url(). 2017-11-09 10:43:20 -08:00
Steve Howell 7da6beac99 Remove force_text() in upload.get_file_info(). 2017-11-09 10:43:20 -08:00
Steve Howell a9f42e6266 Remove force_text() from sanitize_name(). 2017-11-09 10:43:20 -08:00
Steve Howell 17cfead19a Remove force_text() in twitter_text(). 2017-11-09 10:43:20 -08:00
Steve Howell 585829f665 Remove force_text() in list_of_tlds(). 2017-11-09 10:43:19 -08:00
Steve Howell 63024b3a6f Use encode() in zerver/lib/message.py. 2017-11-09 10:38:58 -08:00
Steve Howell d42478bfea Use encode() in zerver/lib/response.py. 2017-11-09 10:38:58 -08:00
Steve Howell 64dc9e65cf Use encode() in zerver/lib/tex.py. 2017-11-09 10:38:58 -08:00
Steve Howell 7821cb3d51 Avoid unneeded force_text() in make_safe_digest(). 2017-11-09 10:38:41 -08:00
Steve Howell 17200bfbc9 tests: Simplify test_classes.fixture_data().
We eliminate the unnecessary force_text() helper and clean up
the formatting a bit.
2017-11-09 10:32:14 -08:00
Steve Howell 5d3ecf8f31 Remove force_str() in log_event(). 2017-11-09 10:32:14 -08:00
Steve Howell 786e90d774 Removed unused imports of force_bytes. 2017-11-09 10:32:14 -08:00
Steve Howell a77cdaa527 Remove force_text() in get_url_data(). 2017-11-09 10:32:14 -08:00
rht 5ee40bf718 Remove usage of six.moves.binary_type. 2017-11-09 10:00:00 -08:00
Harshit Bansal 65838bb825 email_gateway: Disable code block processor for email gateway.
Generally emails are not written with markdown in mind and hence
sometimes render in strange ways. This commit fixes a particular
issue that was causing whitespace before paragraphs to be treated
as code block due to which email content was being rendered in a
box that scrolls in right direction a lot.

Fixes: #7045.
2017-11-09 09:56:35 -08:00
derAnfaenger 19bc55aa45 Fix various typos.
The typos and their corrections were found with the
aid of https://github.com/lucasdemarchi/codespell.
2017-11-09 16:26:38 +01:00
derAnfaenger 816c4a4e79 mypy: Fix typo in type statement. 2017-11-08 11:16:18 -08:00
Steve Howell ae0b27a7ed Extract messages_for_ids. 2017-11-07 17:48:27 -08:00
Steve Howell 8616a4ebc5 Use client_gratavar=True for users on in page_params.
This change affects realm_users and realm_non_active_users.

Note that we still send full avatar urls in realm_user/add
events, so apply_events has to do something mildly hacky to
turn the avatar_url to None in that case.

Fixing the event is probably not worth the trouble, as single
urls are not bandwidth hogs; we only need this optimization
for bulk data.
2017-11-07 16:38:14 -08:00
Steve Howell a076581426 Use client_gravatar=True for avatar URLs in page_params.
This change affects these values:

    * page_params.avatar_url
    * page_params.avatar_url_medium

It requires passing the client_gravatar flag through this
codepath:

    * home_real
    * do_events_register
    * fetch_initial_state_data
    * avatar_url
2017-11-07 16:38:14 -08:00
rht 8242c15b48 zerver/lib: Remove unused imports (F401). 2017-11-07 16:37:13 -08:00
neiljp (Neil Pilgrim) 4194dcfff1 mypy: set SubSuiteList to inherit from typed List.
Modified by tabbott to add the `type: ignore`, which is basically
required by the monkey-patching we're doing here.
2017-11-07 11:39:21 -08:00
neiljp (Neil Pilgrim) 1dcc981af8 mypy: Add explicit Any type parameters for embedded data Dicts. 2017-11-07 11:26:46 -08:00
neiljp (Neil Pilgrim) 88424bed0b mypy: Clarify types in add_missing_messages in soft_deactivation.py. 2017-11-07 11:26:46 -08:00
rht 01eb22abd9 refactor: Remove all the remaining six.moves import. 2017-11-07 10:51:44 -08:00
rht ccf2792c1c refactor: Remove six.moves.configparser import. 2017-11-07 10:51:44 -08:00
rht de319b4558 refactor: Remove six.moves.StringIO import. 2017-11-07 10:51:44 -08:00
rht 14671fc7b9 refactor: Remove six.moves.zip_longest import. 2017-11-07 10:51:44 -08:00
rht 6cce0e346e refactor: Remove six.moves.filter import. 2017-11-07 10:51:44 -08:00
rht 80a8d4f9f3 refactor: Remove six.moves.map import. 2017-11-07 10:46:42 -08:00
rht 549a26860f refactor: Remove six.moves.range import. 2017-11-07 10:46:42 -08:00
rht ec5120e807 refactor: Remove six.moves.zip import. 2017-11-07 10:46:42 -08:00
rht 19bd335cbb Change urllib import to be Python 3-specific. 2017-11-07 10:46:42 -08:00
Steve Howell 2bbfda041a Support client_gravatar field for event registration.
This commit allows clients to register client_gravatar=True, and
then we recognize that flag for message events.  If the flag is
True, we will not calculate gravatar URLs and let the clients do
it themselves.  (Clients can calculate gravatar URLs based on
emails with just a little bit of code.)
2017-11-07 10:42:17 -08:00
Steve Howell 88e1e284bb Restructure send-message code for gravatars.
This refactoring doesn't change behavior, but it sets us up
to more easily handle a register setting for `client_gravatar`,
which will allow clients to tell us they're going to compute
their own gravatar URLs.

The `client_gravatar` flag already exists in our code, but it
is only used for Django views (users/messages) but not for
Zulip events.

The main change is to move the call to `set_sender_avatar` into
`finalize_payload`, which adds the boolean `client_gravatar`
parameter to that function.  And then we update various callers
to supply that flag.

One small performance benefit of this change is that we now
lazily compute the client message payloads in
`event_queue.process_message_event` now, so this will improve
performance if all interested clients have the same value of
`apply_markdown`.  But the change here is really preparing us
for the additional boolean parameter, which will cause us to
have four variations of the payload.
2017-11-07 10:36:02 -08:00
derAnfaenger 0709033858 bugdown: Fetch complete data for tweet.
This makes an extended instead of a normal query,
resulting in the full text of a tweet being fetched
instead of just a truncated version.
2017-11-07 09:13:39 -08:00
rht e311842a1b zerver/lib: Remove inheritance from object. 2017-11-06 08:53:48 -08:00
neiljp (Neil Pilgrim) 42f5eea61f mypy: Improve typing of request.pyi (REQ).
This expands request.pyi to type-check the arguments passed into REQ.

Tweaked by tabbott to fix the report.py annotations.
2017-11-04 20:26:03 -07:00
neiljp (Neil Pilgrim) 2a2870eec8 mypy: Fully specify type of Callable parameter to process_instrumented_calls. 2017-11-04 19:47:45 -07:00
neiljp (Neil Pilgrim) 692e2150c6 mypy: Explicitly specify List[Any] in test_classes.py. 2017-11-04 19:47:45 -07:00
neiljp (Neil Pilgrim) 792a580efb mypy: Specify Callable type parameters in test_helpers.py. 2017-11-04 19:47:45 -07:00
neiljp (Neil Pilgrim) fed757452c mypy: Clarify type of lookup_dict in aggregate_message_dict. 2017-11-04 19:47:44 -07:00
neiljp (Neil Pilgrim) b693f32c0b mypy: Add Dict parameters for dropbox_image return type.
Also a note re using TypedDict in future, as the Dict looks constrained.
2017-11-04 19:47:44 -07:00
neiljp (Neil Pilgrim) 458ab3020d mypy: Add explicit Any for generic types in type_debug.py. 2017-11-04 19:47:44 -07:00
neiljp (Neil Pilgrim) 7d6d8fc848 mypy: Improve type annotation of run_parallel in zerver/lib/parallel.py. 2017-11-04 16:18:27 -07:00
neiljp (Neil Pilgrim) be856bad46 mypy: Reduce use of Any in zerver/lib/url_preview/ return types. 2017-11-04 16:18:27 -07:00
rht fef7d6ba09 zerver/lib: Remove u prefix from strings.
License: Apache-2.0
Signed-off-by: rht <rhtbot@protonmail.com>
2017-11-03 15:34:37 -07:00
Greg Price 119bc14182 alias domains: Add a v1 of this feature.
The main limitation of this version is that it's controlled entirely
from settings, with nothing in the database and no web UI or even
management command to control it.  That makes it a bit more of a
burden for the server admins than it'd ideally be, but that's fine
for now.

Relatedly, the web flow for realm creation still requires choosing a
subdomain even if the realm is destined to live at an alias domain.

Specific to the dev environment, there is an annoying quirk: the
special dev login flow doesn't work on a REALM_HOSTS realm.  Also,
in this version the `add_new_realm` and `add_new_user` management
commands, which are intended for use in development environments only,
don't support this feature.

In manual testing, I've confirmed that a REALM_HOSTS realm works for
signup and login, with email/password, Google SSO, or GitHub SSO.
Most of that was in dev; I used zulipstaging.com to also test
 * logging in with email and password;
 * logging in with Google SSO... far enough to correctly determine
   that my email address is associated with some other realm.
2017-11-03 13:38:59 -07:00
Steve Howell 9767029211 Allow generic bots to be mentioned.
The original PR to allow generic bots to be mentioned had
some merge issues that we detected about a week after the
fact.  This commit restores the logic from the original PR.

The reason we didn't detect this bug earlier is that the
merge issues didn't break any existing behavior.  Instead,
they made it so that only UserMessage rows got written for
bots, but no events were being set.  The part of the commit
that got lost is restored here, so now events get sent as
well.

Thanks to @derAnfaenger for reporting this and being patient
as we tracked it down.

Fixes #7140
2017-11-03 07:45:18 -07:00
Harshit Bansal 44bff43c86 actions.py: Rename `do_remove_reaction()` to `do_remove_reaction_legacy()`. 2017-10-31 15:42:35 -07:00
Harshit Bansal 233595b2f7 actions.py: Rename `do_add_reaction()` to `do_add_reaction_legacy()`. 2017-10-31 15:42:35 -07:00
Umair Khan 636046aec9 user-groups: Add basic backend for UserGroup model.
This adds the data model and bugdown support for the new UserGroup
mention feature.

Before it'll be fully operational, we'll still need:
* A backend API for making these.
* A UI for interacting with that API.
* Typeahead on the frontend.
* CSS to make them look pretty and see who's in them.
2017-10-31 15:16:14 -07:00
Umair Khan 5914afc018 cleanup: Use a single loop in do_send_messages. 2017-10-31 12:56:14 -07:00
neiljp (Neil Pilgrim) 5ec855967d mypy: Constrain return types to match in test_runner.py. 2017-10-31 00:03:35 -07:00
neiljp (Neil Pilgrim) 7f5b69bca8 mypy: Clarify Callable parameters & return type for get_timing. 2017-10-31 00:02:17 -07:00
neiljp (Neil Pilgrim) 304e411944 mypy: Remove unused FuncT TypeVar in cache.py. 2017-10-31 00:02:17 -07:00
Aditya Bansal 31ea495842 test_uploads: Extract 'use_s3_backend' to test helpers. 2017-10-30 21:56:31 -07:00
Greg Price 26edde21f7 alias domains: Tweak a test helper that assumes hosts are subdomains.
Because this is for tests, a heuristic like this that's right in most
situations is actually fine; we can override it in the few cases where
a test might set up a situation where it fails.

So just make it clear for the next reader that that's what's going on,
and also adjust the helper's interface slightly so that its callers
do have that flexibility.
2017-10-30 18:29:29 -07:00
Steve Howell 1ac2360d2e mypy: Fix QuerySet -> QuerySet[Subscription]. 2017-10-30 16:33:51 -07:00
Steve Howell faba34dae4 Simplify bulk_remove_subscriptions().
We extract get_bulk_stream_subscriber_info() from this
function to remove some of the complexity.  Also, in that
new function we avoid a hop to the database by querying
on stream ids instead of recipient ids.  The query that
gets changed here does require a join to the recipient
table (to get the stream id), so it's a little bit of a
tradeoff.
2017-10-30 16:33:50 -07:00
Steve Howell 48d13257b6 Make `our_realm` explicit in `bulk_remove_subscriptions`.
There's an implicit assumption in bulk_remove_subscriptions
that all users belong to the same realm.  We use the realm
for things like comparing occupied streams before and
after our main operation of deactivating streams.

Before this change, we just used the user_profile variable
that leaked from some prior loop to look up the realm, which
was super brittle.

Now we're a bit more explicit.
2017-10-30 16:29:43 -07:00
rht c4fcff7178 refactor: Replace super(.*self) with Python 3-specific super().
We change all the instances except for the `test_helpers.py`
TimeTrackingCursor monkey-patching, which actually needs to specify
the base class.
2017-10-30 14:30:25 -07:00
Harshit Bansal d140451fb4 html_diff: Migrate from `Text` to `str`. 2017-10-30 14:10:35 -07:00
Harshit Bansal c863bb83a0 html_diff: Migrate to use lxml.
We were using Google's diff-match-patch library to diff HTML. The
problem with that approach is that it is a text differ, not an HTML
differ and so it ends up messing up the HTML tags. `lxml` is a safer
option.

Fixes: #7219.
2017-10-30 14:10:35 -07:00
derAnfaenger 692db3f62b embedded bots: Move StateHandler query logic to separate file. 2017-10-30 13:37:56 -07:00
Steve Howell 08ad26f913 refactor: Extract get_stream_subscriptions_for_users(). 2017-10-29 18:36:35 -07:00
Steve Howell b3192d17ab refactor: Extract get_stream_subscriptions_for_user(). 2017-10-29 18:36:35 -07:00
Steve Howell a2747517a3 Extract StreamTopicTarget.get_active_subscriptions().
Note that this code leads to a slightly different query, because
we join to one row in the small Recipient table to match
stream_id to recipient.type_id.
2017-10-29 18:36:35 -07:00
Steve Howell 8e0b417bd9 Extract get_active_subscriptions_for_stream_ids(). 2017-10-29 18:36:35 -07:00
Steve Howell 126e14d1de Add zerver/lib/stream_subscription.py.
The first method we extract to this library is
get_active_subscriptions_for_stream_id().

We also move num_subscribers_for_stream_id() to here, which
is slightly annoying (having the method on Stream was nice)
but avoids some circular dependency issues.
2017-10-29 18:36:35 -07:00
Steve Howell 138568f4f4 refactor: Extract get_stream_recipients(). 2017-10-29 18:36:35 -07:00
Steve Howell f64c780d5c Extract get_huddle_user_ids(). 2017-10-29 18:36:35 -07:00
neiljp (Neil Pilgrim) 0781237b1f mypy: Improve typing to profile.py & remove FuncT from decorator.py.
FuncT was unused in decorator.py, and only imported into profile.py.
The @profiled decorator is now more strongly typed on return-type.
Annotations were converted to python3 format.
2017-10-29 17:10:13 -07:00
Steve Howell 8302689789 Extract Message.is_stream_message().
This sets us up to denormalize on to Message some notion of
whether a message is for a stream (without having to hit the
Recipient table).
2017-10-28 17:57:39 -07:00
Steve Howell 8b012c6210 Extract get_personal_recipient(). 2017-10-28 17:57:39 -07:00
Steve Howell 64211512f8 Avoid mutation in get_recipient_from_user_ids.
We shouldn't mutate parameters in a `get_foo` function.
2017-10-28 17:57:39 -07:00
Steve Howell 370ffe389c Extract get_huddle_recipient().
This extraction moves all the huddle logic into models.py, which
hopefully can reduce friction for things like re-organizing our
caches (there are two cache entries for every huddle) and/or
just putting huddle_id on Message directly.
2017-10-28 17:57:39 -07:00
Steve Howell a28841e8aa Extract get_stream_recipient().
Do you call get_recipient(Recipient.STREAM, stream_id) or
get_recipient(stream_id, Recipient.STREAM)?  I could never
remember, and it was not very type safe, since both parameters
are integers.
2017-10-28 17:57:39 -07:00
Harshit Bansal 4c077a77ed notifications: Instead of regex use lxml for fixing emojis. 2017-10-28 10:38:33 -07:00
Steve Howell d132c30c24 Remove `active` parameter from `do_create_user()`.
Almost all callers to do_create_user were trying to
create active users, except for one test.  The
active=False codepath was kind of broken (things
like sending welcome messages had sort of undefined
behavior there), so instead of trying to maintain it,
we just update the one test (`test_people`) to flip the
`is_active` flag manually.

Fixes #7197
2017-10-28 10:32:01 -07:00
Steve Howell d6c47573b2 Eliminate ZulipTestCase.send_message().
Now all tests use these three messages instead:

    send_stream_message
    send_personal_message
    send_huddle_message
2017-10-28 10:20:59 -07:00
Tim Abbott f2e3e779eb mypy: Properly annotate generic_bulk_cached_fetch.
Along with fixing some minor bugs, this requires extracting out the
default functions so that we can do type: ignores on them properly.

While we're at it, we switch to the Python 3 syntax.
2017-10-28 10:07:15 -07:00
Tim Abbott 73c27e1277 cache: Fix type aliasing of cached_objects.
Previously, it was converted from a CompressedItemT to an ItemT
without changing the variable name.
2017-10-28 10:01:44 -07:00
Tim Abbott 94c1da7025 cache: Move generic_bulk_cached_fetch typevars up a bit. 2017-10-28 10:00:43 -07:00
Tim Abbott 6c242379f4 bugdown: Remove unused caching imports. 2017-10-28 09:02:36 -07:00
neiljp (Neil Pilgrim) c063ba72a2 mypy: Improve typing of cache_with_key and cache decorators.
Fixes #1348.
2017-10-28 08:57:49 -07:00
Tim Abbott fa55d7ed33 retry_event: Remove requirement of failed_tries being already there.
This fixes a bug where retries in the signups queue threw an exception.
2017-10-27 18:15:53 -07:00
Tim Abbott bfa148ab39 requirements: Update mypy to latest version.
This involves fixing the mypy annotations in one file
2017-10-27 17:05:52 -07:00
Tim Abbott e5df05fd35 tests: Suppress logging spam in email mirror tests. 2017-10-27 16:06:03 -07:00
Tim Abbott 073c81bbe8 slow_tests: Increase limit for TOO slow notices. 2017-10-27 15:38:17 -07:00
Greg Price 6d403ff255 tests: Simplify set_http_host to dedupe its logic.
This will make it easier to change this logic.
2017-10-27 14:42:24 -07:00
Greg Price e4b4f67b44 subdomains: Tighten search for EXTERNAL_HOST in get_subdomain.
If a Zulip install at example.org got a request at an HTTP `Host`
like foo.example.org.evil.com (or even foo.example.orgevil.com),
we would accept it as subdomain foo.  This isn't likely to happen
in practice because it shouldn't pass ALLOWED_HOSTS, and it's not
obvious to me that anything untoward could be done with it even
if ALLOWED_HOSTS were set wide open, but if nothing else it
multiplies the cases in analyzing this logic.

The reason we had a loose match like this, I assume, is to allow
the user to come from arbitrary ports -- especially in development.
So tighten the pattern to allow just that, and add some tests for
that behavior and a comment explaining why this complication is
needed.
2017-10-27 14:42:24 -07:00
Greg Price 318682fd52 auth: Use URL rather than cookie to pass signed data cross-domain.
The cookie mechanism only works when passing the login token to a
subdomain.  URLs work across domains, which is why they're the
standard transport for SSO on the web.  Switch to URLs.

Tweaked by tabbott to add a test for an expired token.
2017-10-27 14:42:04 -07:00
Henrik Pettersson 09cd47c6ad Add UI for viewing and cancelling open Zulip invitations.
Lets administrators view a list of open(unconfirmed) invitations and
resend or revoke a chosen invitation.

There are a few changes that we can expect for the future:

  * It is currently possible to invite an email that you have already
    invited, it might make sense to change this behavior.

  * Resend currently sends an invite reminder instead of resending the
    original invite, this is because 'custom_body' was not stored when
    the first invite was sent.

Tweaked in various minor ways, primarily in the backend, by tabbott,
mostly for style consistency with the rest of the codebase.

Fixes: #1180.
2017-10-27 13:07:43 -07:00
Vishnu Ks 8c68a167fe backend: Allow Administrators to invite new users as admins.
Tweaked by tabbott to have the field before the invitation is
completed be called invite_as_admins, not invited_as_admins, for
readability.

Fixes #6834.
2017-10-27 11:19:38 -07:00
Steve Howell 4b78f69a87 tests: Introduce send_huddle_message().
The tighter interface here makes for shorter lines (or
fewer multi-line calls) and it leads to stricter type
checking.
2017-10-27 11:00:23 -07:00
Steve Howell 9707767b21 tests: Introduce send_personal_message().
The tighter interface prevents the need to specify
Recipient.PERSONAL (which can often be inaccurate in the
huddle case, anyway), and it prevents tests from confusingly
specifying a "subject" field for PMs.
2017-10-27 11:00:23 -07:00
Steve Howell 25a6fbd05d tests: Introduce send_stream_message().
Having send_stream_message() avoids the need to supply
Recipient.STREAM as a parameter, and it also uses the more
modern name of `topic_name` for topics.  Under the hood, it
avoids some annoying steps for re-formatting the recipients,
since we just have a single stream name.
2017-10-27 11:00:23 -07:00
Steve Howell 65b6676844 minor: Remove unused import (internal_send_message). 2017-10-27 10:48:11 -07:00
Steve Howell e5c5d10375 Rename subject vars in do_update_message(). 2017-10-27 10:48:11 -07:00
Steve Howell fbe5f93141 Rename subject -> topic_name in internal_send_message. 2017-10-27 10:48:11 -07:00
Steve Howell 973ee97612 Rename subject -> topic_name in internal_prep_message(). 2017-10-27 10:48:11 -07:00
Steve Howell f04550d143 Rename subject -> topic_name as var in check_message().
There were two variables, subject and subject_name, that
really had no clear distinction, so now they're just
topic_name.
2017-10-27 10:48:11 -07:00
Steve Howell 70b05998b1 Rename subject_name -> topic_name in check_send_message(). 2017-10-27 10:48:11 -07:00
derAnfaenger 64025d2ce7 integrations: Add Errbot documentation.
Fixes zulip/python-zulip-api#49.
2017-10-27 09:22:51 -07:00
Tim Abbott cab39dc6eb get_service_bot_events: Add some future-proofing.
The comment is the most important part here, but the new code
structure should help, too.
2017-10-26 22:16:47 -07:00
Tim Abbott b897c763f8 get_recipient_info: Simplify active_user_ids construction. 2017-10-26 22:16:47 -07:00
Steve Howell a70ede6c75 Allow "default" bots to see mentions on all streams.
This change allows normal bots to get UserMessage rows when
they are mentioned on a stream, even if they are not actually
subscribed to the stream.

Fixes #7140.
2017-10-26 22:16:47 -07:00
Steve Howell 53dbcbe2a4 refactor: Handle service bot mentions earlier in the code.
We now find all (possibly) relevant service bots for a message
in the call to get_recipient_info.  This allows us to eliminate
some code that would patch them after we rendered.

The get_service_bot_events() function will ignore any service
bots that weren't actually mentioned in the message (due to
backticks) or part of the active user ids.
2017-10-26 22:16:47 -07:00
Steve Howell 87b48f722d Add MentionData.get_user_ids().
This prepares us to take advantage of the user_ids
in subsequent commits related to `do_send_messages()`.
2017-10-26 22:16:47 -07:00
Steve Howell 8ac26dfb9b refactor: Introduce bugdown.MentionData class.
We now have a MentionData class that encapsulates
the users who are possibly mentioned in a message.

Not that the rendering code may not keep all the mentions,
since things like backticks will suppress the mention.

We populate this now in do_send_messages, so that we can use
the info earlier in the message-sending process.  This info
now gets passed down the call stack as an optional parameter.

Note that bugdown.convert() still populates the data when its
callers decline to pass in a MentionData object.

This is mostly a preparatory commit, as we don't take advantage
of the data yet in do_send_messages.
2017-10-26 22:16:47 -07:00
Tim Abbott 8e2cdedf9a lint: Fix lines in Python codebase longer than 120 characters. 2017-10-26 17:47:30 -07:00
Tim Abbott be619fe881 lint: Wrap many very long lines in the Python codebase.
This decreases the maximum line length in our Python codebase to 130.
2017-10-26 17:31:58 -07:00
Steve Howell 635675fe48 Reduce queries needed for sending messages.
In do_send_messages, we only produce one dictionary for
the event queues, instead of different flavors for text
vs. html.  This prevents two unnecessary queries to the
database.

It also means we only put one dictionary on the "message"
event queue instead of two, albeit a wider one that has
some values that won't be sent to the actual clients.

This wider dictionary from MessageDict.wide_dict is also
used for the `feedback_messages` queue and service bot
queues.  Since the extra fields are possibly useful down
the road, and they'll just be ignored for now, we don't
bother to remove them.  Also, those queue processors won't
have access to `content_type`, which they shouldn't need.

Fixes #6947
2017-10-26 16:35:28 -07:00
Steve Howell 9b6a4d0b16 refactor: Extract MessageDict.finalize_payload(). 2017-10-26 16:35:28 -07:00
Steve Howell df93a99b50 Cache only one row per message.
Before this change, we populated two cache entries for each
message that we sent.  The entries were largely redundant,
with the only difference being whether we sent the content
as raw markdown or as the rendered HTML.

This commit makes it so we only have one cache entry per
message, and it includes both content and rendered_content.

One legacy source on confusion here is that `content`
changes meaning when you're on the front end.  Here is the
situation going forward:

    database:
        content = raw
        rendered_contented = rendered

    cache entry:
        content = raw
        rendered_contented = rendered

    payload for the frontend:
        content = raw (for apply_markdown=False)
        content = rendered (for apply_markdown=True)
2017-10-26 16:35:28 -07:00
derAnfaenger 1792dcbd09 tests: Call real consume method of queue processors.
This switches to more real tests for a first batch of
queue_json_publish() calls that don't cause trouble when
used with call_consume_tests=True.
2017-10-26 14:58:03 -07:00
derAnfaenger 18e5bcbbb1 tests: Enable call_consume_in_tests for email mirror queue. 2017-10-26 14:53:27 -07:00
derAnfaenger 17949adf11 embedded bots: Add StateHandler remove() function. 2017-10-26 14:49:50 -07:00