Commit Graph

3230 Commits

Author SHA1 Message Date
Greg Price 693a9a5e70 push notifs: For group PMs, identify the users in the group. 2018-02-16 16:06:03 -08:00
Shubham Dhama a32e1eb913 markdown: Require double-asterisk around all mentions.
This enforces `**` around all the mentions including "at-all" and
"at-everyone" mentions. Hence this makes `@all` and `@everyone`
invalid mentions, resulting into proper syntax for these mentions as
`@**all**` and `@**everyone**` respectively.

Note from tabbott: This removes an old feature/syntax, which made
sense back when @Tim was also a way to mention a user with Tim as
their first name.  Given how nice typeahead is now, the user part of
the feature was removed a while ago; this should have gone at the same
time.

Fixes: #8143.
2018-02-16 11:45:08 -08:00
Eeshan Garg 6206647641 bugdown/api_code_examples: Add support for multiple languages.
This commit modifies the Markdown extension in bugdown/api_code_examples.py
to support rendering code examples in multiple languages by specifying
the language like so:

{generate_code_example(python)|doc.md|example}

This makes us one step closer towards adding support for testable
JavaScript code examples.
2018-02-16 10:07:13 -08:00
Eeshan Garg 73a1f8aa3e api docs: Add comment about why the event queue example isn't tested.
This commit simply adds a comment in zerver/lib/api_test_helpers.py
explaining why it isn't worth the effort to explicitly test the
code example in api/get-events-from-queue.md.
2018-02-16 10:07:13 -08:00
Shubham Padia a4b686297a api: Return anchor in get_messages when use_first_unread_anchor=True.
This may be helpful for some API clients, since it avoids them needed
to do somewhat messy post-processing on the results (the data was
always available via scanning for the first unread message in the result).

Fixes #6244.
2018-02-16 10:06:20 -08:00
Robert Hönig a19a69bfe3 embedded bots: Log warning when bot quit()s.
External bots may call bot_handler.quit() when
they wish to terminate, e.g. due to a misconfiguration.
Currently, embedded bots ignore calls to quit(), even
though they signal a problem. This commit does the first
step in handling quit() calls by logging a warning.
2018-02-13 14:56:37 -08:00
neiljp (Neil Pilgrim) 756af7518c REQ: Add type parameter to act as hint for typing. 2018-02-13 11:40:52 -08:00
neiljp (Neil Pilgrim) ec829ea832 mypy: Add 2 Optional[StreamTopicTarget] annotations in actions.py. 2018-02-13 11:40:52 -08:00
neiljp (Neil Pilgrim) 79582a59f1 mypy: Move assertion into similar branch in get_recipient_info. 2018-02-13 11:40:52 -08:00
neiljp (Neil Pilgrim) 3cb12230b2 mypy: Annotate email_gateway_bot in export_files_from_s3(). 2018-02-13 11:40:52 -08:00
neiljp (Neil Pilgrim) 16b939a964 REQ: Ensure only initial parameter is positional & adjust views/messages.py. 2018-02-13 11:40:52 -08:00
neiljp (Neil Pilgrim) b0126ca8dc mypy: Amend diff_strings to return str or raise DiffException. 2018-02-13 11:40:51 -08:00
neiljp (Neil Pilgrim) 23cc7e1671 mypy: Allow acting_user of do_change_full_name() to be None.
This is based on usage in bulk_change_user_names.py, and that
the RealmAuditLog acting_user field is Optional[UserProfile].

This could be more meaningfully changed in future, perhaps to
indicate that the command was run by a specific zulip user.
2018-02-13 11:40:51 -08:00
Vishnu Ks 81f18bd69a actions: Create do_create_multiuse_invite_link function. 2018-02-13 09:31:45 -08:00
Greg Price ecbc72b857 push notifs: Add a diagnostic in API of whether push notifs enabled.
When the answer is False, this will allow the mobile app to show a
warning that push notifications will not work and the server admin
should set them up.

Based partly on Kunal's PR #7810.  Provides the necessary backend API
for zulip/zulip-mobile#1507.
2018-02-12 14:34:59 -08:00
Greg Price 8be2dfa81c APNs: Fix a case I broke while working out tests.
I got distracted, came back later to a successful test run in my
terminal, and thought I remembered finishing the change and just
kicking off a final test run to check.

In fact, there was an `assert False` right in the normal case for
production, and I just hadn't finished a test for that path. (m.-)
Definitely the most grateful I've been for our coverage checks,
which highlighted this for me.

Remove the `assert False`, and also finish writing the test it was
there to help me write.  Those lines are covered now.
2018-02-09 18:28:16 -08:00
Umair Khan 0eca2e102d cache: Add ignore_unhashable_lru_cache function.
This is a wrapper over lru_cache function. It adds following features on
top of lru_cache:

    * It will not cache result of functions with unhashable arguments.
    * It will clear cache whenever zerver.lib.cache.KEY_PREFIX changes.
2018-02-09 18:14:08 -08:00
Greg Price 7f6a1714aa APNs: Don't try to send anything when not configured. 2018-02-09 17:16:21 -08:00
Robert Hönig 323284e0b6 backend: Call real endpoint in create_test_bot().
This allows tests to be more realistical and to
directly test payload added to add_bot_backend().
2018-02-09 12:30:24 -08:00
Rhea Parekh fbc11b141b slack importer: Add 'realm_name_in_notifications' field in zerver_userprofile.
This was recently added in b875fe07eb and
hence should be updated here.
2018-02-09 12:17:10 -08:00
Rhea Parekh be05bccb5b slack importer: optimize allocation of id range before import. 2018-02-09 12:17:10 -08:00
Rhea Parekh a0b7976f77 slack importer: Fix output_dir path and remove sys.exit(0).
Also, as the exception is being handled in 'slack_data_to_zulip_data'
remove it from the manage.py command 'convert_slack_data'.
2018-02-09 12:17:10 -08:00
Rhea Parekh c0e30079f6 slack importer: Get user data from a get request to slack users api.
The fresh imported data shows that the users emails are not included
in the data. However, the data received from the older method of slack
(which is using legacy tokens) contains the email data of the users.
2018-02-09 12:17:10 -08:00
Tim Abbott 77addc5456 bugdown: Fix handling of ultra-long renderings.
If some bug in Bugdown results in a rendered message content that is
bigger than twice the message size, we now just throw an exception
from Bugdown.  This is considerably better than the old behavior,
which might result in an enormous message being placed in the database
(potentially, bigger than the 1MB limit to store in memcached), which
would in turn result in tragic consequences.

This fixes #8322, in that it prevents the super bad outcome seen there
(where basically Zulip became unusable for everyone on the stream
where the message is posted).  Now, the failure mode is just the
message failing to send.  Still not ideal (and requires further work
on the URL embed feature), but not a minor problem, not a major one.
2018-02-09 10:57:55 -08:00
Eeshan Garg 3853577486 api tests: Add example/tests for unsubscribing other users.
This commit adds tests (and thus, an extra code example) for
unsubscribing another user from a particular stream by passing in
the `principals` argument to client.remove_subscriptions. The
ability to pass in `principals` was added in the latest release
of the zulip API PyPI package.
2018-02-08 17:58:41 -08:00
Eeshan Garg ac5c355a60 api/delete-queue: Make fixtures/examples testable.
api/delete-queue now uses the Markdown extension in
bugdown/api_code_examples to generate code examples and fixtures
from tests.
2018-02-08 17:58:41 -08:00
Eeshan Garg 344cf2c8ee api_tests: Add tests for client.register_queue. 2018-02-08 17:58:41 -08:00
Eeshan Garg 66ca0c725f api/get-all-users: Make fixtures/examples testable.
api/get-all-users now uses the Markdown extension in
bugdown/api_code_examples to generate code examples and fixtures
from tests.
2018-02-08 17:58:41 -08:00
Eeshan Garg 929724e5e7 api docs: Add page for common error payloads.
We now have a separate page for common error payloads, for example,
the payload for when the client's API key is invalid. All error
payloads that are presented on this page will be tested similarly
to our other non-error sample fixtures.
2018-02-08 17:58:41 -08:00
Rhea Parekh 48640fd28f slack importer: Suppress logger output from the unit tests. 2018-02-08 16:21:35 -08:00
Eeshan Garg 61e4fc3b74 api/remove-subscriptions: Make fixtures/examples testable.
api/remove-subscriptions now uses the Markdown extension in
bugdown/api_code_examples to generate code example and fixtures
from tests.
2018-02-07 15:58:57 -08:00
Eeshan Garg f4cb5e70ed api/add-subscriptions: Make fixtures/examples testable.
api/add-subscriptions now uses the Markdown extension in
bugdown/api_code_examples to generate code example and fixtures
from tests.
2018-02-07 15:58:57 -08:00
Eeshan Garg 09721e9ec3 bugdown/api_code_examples: Support multiple fixtures per doc.
This commit adds support for passing in an argument to the macro
"call" to explicitly specify a fixture to render, like so:

{generate_code_example|doc_name|fixture(stream_message_with_args)}
2018-02-07 15:58:57 -08:00
Eeshan Garg 7ae704d1bb api/get-all-streams: Render multiple examples from single test method.
api/get-all-streams - make use of the api_code_example Markdown extension's
feature of recursively extracting multiple code examples from a single
test method.
2018-02-07 15:58:57 -08:00
Eeshan Garg 0cca39d1fc bugdown/api_code_examples: Support multiple examples in one test.
This commit adds support for recursively extracting multiple code
examples from a single test method.
2018-02-07 15:58:57 -08:00
Eeshan Garg ffe930d50d bugdown/api_generate_examples: Replace 'method' with 'example'.
To generate a code exammple,

{generate_code_example|<api_doc_md>|example} sounds better and
more intuitive than,

{generate_code_example|<api_doc_md>|method}
2018-02-07 15:58:57 -08:00
Ricky 1303248b0b integration: Add statuspage.io webhook integration. 2018-02-07 09:59:12 -08:00
Greg Price c80802ff1e bots: Clean up create_realm_internal_bots.
This code duplicated the code in setup_realm_internal_bots, with some
added logic to avoid trying to create the same bot twice.  That logic
was buggy so that it would never work at all -- it subtracted a set of
UserProfile objects from a set of email strings -- so it looked like
the command might blow up when run after the users already existed.

In fact, the buggy logic wasn't necessary, because the work the
command does after it is idempotent -- in particular `create_users`,
within its subroutine `bulk_create_users`, already filters out users
that already exist.  So just cut the buggy stuff out, deduplicate the
rest with `setup_realm_internal_bots`, and document that invariant on
the latter.

While we're here, in the common case bail early without doing any
per-realm work in Python, since we're running this on every upgrade.
2018-02-06 18:59:57 -08:00
Rhea Parekh 83a7fd84ab slack importer: Import primary owner user first.
According to https://get.slack.help/hc/en-us/articles/201912948-Owners-and-Administrators,
only one Primary owner of a slack organsation exists. This allocates the first id
to the Primary owner and hence makes sure that the primary owner is imported first.
Added tests for the same.
2018-02-06 14:48:30 -08:00
Rhea Parekh 052e3e1540 slack importer: Change organization admin mappings.
Map 'Primary owner', 'owner' and 'admin' to 'organization admin'.
Added tests for the same.
2018-02-06 14:48:30 -08:00
Rhea Parekh b3b6023230 slack importer: Always map 'is_staff' to false in user data.
"staff" is only for server administrators, which doesn't exist in Slack.
Hence, this should always be false.
2018-02-06 14:48:29 -08:00
Roman Godov b875fe07eb settings: Added setting to turn on and off realm name in email subject.
Users having only account in one realm will not be distracted by realm
name in subject lines of every email.  Users who have multiple
accounts in realms can turn this setting on and receive a
corresponding realm name in email's subject.

Tweaked by tabbott to rebase and address a few small issues.

Fixes #5489.
2018-02-05 18:01:54 -08:00
Aditya Bansal ecbb763e79 notifications: Handle an edge case in relative_to_full_url.
For messages where the entire rendered body is a message_inline_image
object, we actually don't display any text and just display the
image. These messages may have links to images which might or might
not be internal to Zulip but in both cases there is a chance of this
links being broken when accessed by an email server like Gmail that
doesn't possess the recipient user's cookies.

We don't want to have ugly looking broken images displayed in email
notifications.  So we patch this by inserting a replacement for the
`message_inline_image` block in which we essentially replace the
content with the textual link.

Edited for clarity by tabbott.
2018-02-05 17:30:55 -08:00
Greg Price ca1129dea3 Revert "Streams: Disallow certain markdown characters in stream names."
This reverts commit dd1b14378b.

Deleted a conflicted test entirely.
2018-02-05 16:55:21 -08:00
Rhea Parekh 811294c6cc slack importer: Add test for checking link in message conversion. 2018-02-05 14:46:39 -08:00
Rhea Parekh eb7a9675a4 slack importer: Add unit tests. 2018-02-05 14:46:39 -08:00
Rhea Parekh 2b11bff9ad slack importer: Refactor script for easier unit tests.
This is to break the bigger functions into smaller
ones and hence, makes unit testing easier.

Also includes renaming the stream's topic name to 'from slack'.
2018-02-05 14:46:39 -08:00
Tim Abbott b84f2223a5 avatars: Always include a working URL for the user's own avatar.
This fixes an issue where the user's own avatar was being sent down
the wire as None.  We could have fixed it, as in #8265, by adding code
in the webapp and mobile apps to compute medium-size gravatar URLs as
well, but that would be messy, and there's little benefit to that
complexity (saving at most 2 URLs from the payload).

Fixes #8253.
2018-02-05 12:45:41 -08:00
rht 9161f8c39b zerver/lib: Remove u prefix from strings. 2018-02-05 12:12:58 -08:00
Privisus 9fdd6f2bd4 integrations: Add Insping integration. 2018-01-31 14:36:54 -08:00
rht 0c4066b0f3 django-2.0: Change parameter name to app_label.
showmigrations command expects a parameter with the name app_label.
2018-01-31 12:07:36 -08:00
Eeshan Garg 162f34f0d0 api/get-profile: Make code examples/fixtures testable.
This commit uses the Markdown extension defined in
zerver/lib/bugdown/api_generate_examples to generate the
example fixture and code example, so that both are tested
in zerver/lib/api_test_helpers.
2018-01-31 07:30:54 -05:00
Eeshan Garg 2bea4b70de api/create-user: Make code examples/fixtures testable.
This commit uses the Markdown extension defined in
zerver/lib/bugdown/api_generate_examples to generate the
example fixture and code example, so that both are tested
in zerver/lib/api_test_helpers.
2018-01-31 07:30:54 -05:00
Eeshan Garg fe98a59f5c bugdown/api_code_examples: Add macro for admin zuliprc lines.
Some of our code examples can only be run with administrator
credentials (such as create-user). Thus, the Markdown extension
for generating code examples should have an option to include
the lines that recommend using an admin zuliprc instead of a
non-admin one.
2018-01-31 07:30:54 -05:00
Eeshan Garg 73a3755120 api/get-all-streams: Make code examples/fixtures testable.
This commit uses the Markdown extension defined in
zerver/lib/bugdown/api_generate_examples to generate the
example fixture and code example, so that both are tested
in zerver/lib/api_test_helpers.
2018-01-31 07:30:54 -05:00
Eeshan Garg dcf3a9cd18 api/get-subscribed-streams: Make code examples/fixtures testable.
This commit uses the Markdown extension defined in
zerver/lib/bugdown/api_generate_examples to generate the
example fixture and code example, so that both are tested
in tools/lib/api_tests.
2018-01-31 07:30:54 -05:00
Eeshan Garg 6c634d6847 api/get-stream-id: Make code examples and fixtures testable.
This commit uses the Markdown extension defined in
zerver/lib/bugdown/api_generate_examples to generate the
example fixture and code example, so that both are tested
in tools/lib/api_tests.
2018-01-31 07:30:54 -05:00
Eeshan Garg b1878f14a2 api/update-message: Make code examples and fixtures testable.
This commit uses the Markdown extension defined in
zerver/lib/bugdown/api_generate_examples to generate the
example fixture and code example, so that both are tested in
tools/lib/api_tests.
2018-01-31 07:30:54 -05:00
Eeshan Garg cf80587f47 api/private-message: Make code examples and fixtures testable.
This commit uses the Markdown extension defined in
zerver/lib/bugdown/api_generate_examples to generate the example
fixture and code example, so that both are tested in
tools/lib/api_tests.
2018-01-31 07:30:54 -05:00
Eeshan Garg f58ecee2d8 api/stream-message: Make code examples and fixtures testable.
This commit uses the Markdown extension defined in
zerver/lib/bugdown/api_generate_examples to generate the example
fixture and code example, so that both are tested in
tools/lib/api_tests.
2018-01-31 07:30:54 -05:00
Eeshan Garg c158869096 Rename tools/lib/api_tests.py to zerver/lib/api_test_helpers.py.
Now that the Markdown extension defined in
zerver/lib/bugdown/api_generate_examples depended on code in the
tools/lib/* directory, it caused the production tests to fail since
the tools/ directory wouldn't exist in a production environment.
2018-01-31 07:30:54 -05:00
Eeshan Garg c89550cfa0 api docs: Add Markdown extension for generating examples/fixtures.
This commit adds a Markdown extension that allows the following
syntax,

{generate_code_example|<md_file_name>|<fixture or method>}

to generate code examples and fixtures found in tools/lib/api_tests.py
and templates/zerver/api/fixtures.json, respectively.
2018-01-31 07:30:54 -05:00
Greg Price cad4083987 export: Fix an unnecessary Any.
This was introduced a few weeks ago in
ed4054d11 "Import script: Check and add system bots after every import."
2018-01-30 15:34:47 -08:00
rht 9a8d2244ca django-2.0: Shift to resolvers from urlresolvers.
The old name is deprecated.
2018-01-30 10:53:54 -08:00
Vishnu Ks 43a6439b3b upload: Enforce per-realm quota. 2018-01-29 16:06:11 -08:00
Vishnu Ks 035d3c5e97 upload: Remove UserProfile field for old per-user quota. 2018-01-29 16:06:11 -08:00
Greg Price 55cf54c087 upload: Remove old per-user quota feature.
We'll replace this primarily with per-realm quotas (plus the simple
per-file limit of settings.MAX_FILE_UPLOAD_SIZE, 25 MiB by default).

We do want per-user quotas too, but they'll need some more management
apparatus around them so an admin has a practical way to set them
differently for different users.  And the error handling in this
existing code is rather confused.  Just clear this feature out
entirely for now; then we'll build the per-realm version more cleanly,
and then we can later add back per-realm quotas modelled after that.

The migration to actually remove the field is in a subsequent commit.

Based in part on work by Vishnu Ks (hackerkid).
2018-01-29 16:06:11 -08:00
Rohitt Vashishtha e2a42650c9 emails: Fix link to org-admin help docs. 2018-01-27 15:33:36 -08:00
Shubham Dhama dbce49ac53 refactor: Move topic name checks to Addressee class.
In order to get test coverage on topic name checks, we
do them in Addressee, so that we don't hit an assertion
first. The assertion in question is in Addressee.topic(),
and it was added partly to appease mypy.
2018-01-25 06:21:01 -05:00
Vishnu Ks a11b742984 messages: Calculate value of first visible message ID using cron job.
[greg: Fixed buggy time conversion in estimate_recent_messages.]
2018-01-24 17:15:08 -08:00
Patrick Grave f2b322c32d mypy: Use Python 3 syntax for typing in upload.py 2018-01-23 08:49:42 -08:00
Patrick Grave 008e250668 mypy: Use Python 3 syntax for typing in test_runner.py 2018-01-23 08:49:42 -08:00
Rhea Parekh 11b549e566 slack importer: Add regex for mail links.
`<mailto:foo@foo.com>` is changed to `mailto:foo@foo.com`.
2018-01-23 10:01:15 -05:00
Rhea Parekh df1bb5fd27 slack importer: Fix link regex and add test case. 2018-01-23 10:01:15 -05:00
Rhea Parekh 5b1e9f8181 slack importer: correct the implementation of unzipping slack data file.
Slack zip file unzips in the same folder.
2018-01-23 10:01:15 -05:00
Rhea Parekh 845a8d63bf slack importer: Prevent id clashes for import in an active db.
Previously we had a problem of id clashes while importing converted
slack data into an existing zulip instance with realms which are actively
populating the database.

This counts the total objects to be imported and does a db transaction
to increase the SEQUENCE number for that table by that number,
and hence allocates a range of ids for the to be converted slack data
objects.
2018-01-23 10:01:15 -05:00
Weronika Grzybowska 7ac7100a1d messages: Make checking for status message consistent with backend.
Adds a check for newline that was present on backend, but missing in the
frontend markdown implementation. Updating messages uses is_me_message flag
received from server instead of its own partial test. Similarly, rendering
previews uses markdown code.

Fixes #6493.
2018-01-23 09:26:41 -05:00
Robert Hönig 9a15c4e3ff Add bot services to page_params.
This is the first step for allowing users
to edit a bot's service entries, name the
outgoing webhook configuration entries. The
chosen data structures allow for a future
with multiple services per bot; right now,
only one service per bot is supported.
2018-01-23 07:29:00 -05:00
Vishnu Ks 036dc53d20 messages: Rename last_visible_message_id to first_visible_message_id. 2018-01-22 19:53:44 -08:00
YJDave 8905744bb1 stream settings: Add period at the end of error messages. 2018-01-19 13:05:43 -05:00
Aditya Bansal f272ea9087 scheduledmessages: Start using/expecting delivery_type as a param. 2018-01-19 11:33:11 -05:00
Aditya Bansal c770bdaa3a reminder_bot: Add infra for adding reminder bot to every realm. 2018-01-19 11:33:11 -05:00
Aditya Bansal f46d098558 schedulemessages: Add handle_deferred_message() to handle requests.
This is responsible for:
1.) Handling all the incoming requests at the
messages endpoint which have defer param set. This is similar to
send_message_backend apart from the fact that instead of really
sending a message it schedules one to be sent later on.
2.) Does some preliminary checks such as validating timestamp for
scheduling a message, prevent scheduling a message in past, ensure
correct format of message to be scheduled.
3.) Extracts time of scheduled delivery from message.
4.) Add tests for the newly introduced function.
5.) timezone: Add get_timezone() to obtain tz object from string.
This helps in obtaining a timezone (tz) object from a timezone
specified as a string. This string needs to be a pytz lib defined
timezone string which we use to specify local timezones of the
users.
2018-01-19 11:33:11 -05:00
rht ee06da9350 bugdown: Linkify Bitcoin URI. 2018-01-19 06:07:43 -05:00
Umair Khan ea8f1efb63 tests: Re-raise exception if not import error. 2018-01-19 06:02:05 -05:00
Umair Khan 8f316183e0 tests: Make import error tracebacks terse.
This will also disable exception chaining; we do not need it here.
2018-01-19 06:02:05 -05:00
Vishnu Ks 7dbb4cdae2 uploads: Remove ExceededQuotaError class.
We are no longer using it.
2018-01-18 15:38:08 -08:00
Umair Khan a994dba583 tests: Give more descriptive import error.
This code takes care of the environment running Python 3.4 when
test label is passed directly to the test-backend command:
./tools/test-backend test_alert_words
2018-01-18 07:13:26 -05:00
Umair Khan 0eda941822 tests: Verify imports in test modules. 2018-01-18 07:13:26 -05:00
Umair Khan 7248a3366b tests: Change print_error_message to check_import_error. 2018-01-18 07:13:26 -05:00
Umair Khan f3c86c9e58 tests: Remove old import testing code.
We will add the modified version of this code in the next commit.
2018-01-18 07:13:26 -05:00
Umair Khan c26cdee95d get_test_names: Update to accept ParallelTestSuite arg. 2018-01-18 07:13:26 -05:00
Alena Volkova aa58e1fcda tests: Make import failure error messages more consistent. 2018-01-18 07:13:26 -05:00
Alena Volkova 13b1334fe4 tests: Refactor error handling of import failures. 2018-01-18 07:13:26 -05:00
Greg Price 3d423a3c7b logging: Fix corrupted default format for our logfiles.
This was changed by accident when the syntax was rearranged,
in 3f4bf2d22 "zerver/lib: Use python 3 syntax for typing"
back in November.
2018-01-17 18:10:17 -08:00
Privisus c758636c1e integrations: Add webhook code, API endpoint, and tests for Raygun. 2018-01-17 12:53:22 -05:00
Rhea Parekh 7153d20205 slack importer: append Message id from the db. 2018-01-17 11:46:27 -05:00
Rhea Parekh 58f1f0e13e slack importer: append Recipient id from the db.
Recipient id should be updated from the database,
otherwise there will be foreign key clashes.
2018-01-17 11:46:27 -05:00
Rhea Parekh c86411d9a2 slack importer: Move message conversions to a new module. 2018-01-17 09:38:53 -05:00
Rhea Parekh c1d336f5d1 slack importer: Refactor checking for availability of realm subdomain. 2018-01-17 09:38:53 -05:00
Ricky a90e99dd3e webhooks: Add Beeminder webhook. 2018-01-17 09:24:12 -05:00
fredfishgames 4e2b99a98b integrations: Add webhook code, API endpoint, and tests for Opbeat. 2018-01-17 09:17:19 -05:00
Shreyansh Dwivedi b0fb7aa6b2 bugdown: Add preview for vimeo videos.
This also amends a commit from Brock Whittaker <brock@zulipchat.com>
that merges two separate functions for YouTube videos and Vimeo videos
into a generic video recall function.

Fixes #7550.
2018-01-17 09:11:43 -05:00
Viraat Chandra 64034241a0 webhooks: Create Intercom integration and docs. 2018-01-16 08:51:17 -05:00
Xavier Cooney b0f501ff6c test_helpers.py: Remove unecessary imports. 2018-01-16 08:16:43 -05:00
Xavier Cooney 0c48f5bf09 test_classes.py: Remove unecssary imports. 2018-01-16 08:16:43 -05:00
Aditya Bansal c64766e55e scheduledmessages: Add funcs to verify and schedule messages.
We add two functions:

1.) check_schedule_message(): This function is responsible for
doing the essential initial checkes to verify the validity of
the message. These checkes include things like if user is
allowed to send messages to some stream or not or if the user is
a super_user. All this is basically done by further calling
check_message() with appropriate parameters. This is on the same
lines as is check_send_message().

2.) do_schedule_messages(): This function is responsible for
creating ScheduleMessage table rows for a list of messages that
are to be scheduled. This basically accumulates the ScheduleMessage
objects in a list and then bulk creates the rows.
2018-01-10 09:18:02 -05:00
Andy Perez c209069fcd bugdown: Improve image inlining logic.
Fix #7537
2018-01-09 16:43:22 -05:00
Robert Hönig d1d8365a6b embedded bots: Add config data UI.
This adds UI fields in the bot settings for specifying
configuration values like API keys for a bot. The names
and placeholder values for each bot's config fields are
fetched from the bot's <bot>.conf template file in the
zulip_bots package. This also adds giphy and followup
as embedded bots.
2018-01-09 11:50:54 -05:00
infinitelooped ffa7637215 integrations: Add webhook code, API endpoint, and tests for Groove. 2018-01-09 09:32:52 -05:00
Rhea Parekh b583f1f0ff slack importer: Add regex for markdown mapping.
* For strikethrough formatting: Slack's '~strike~' to Zulip's '~~strike~~'.
    * For bold formatting: Slack's '*bold*' to Zulip's '**bold**'.
    * For italic formatting: Slack's '_italic_' to Zulip's '*italic*'.
    * For mentioning formatting: Slack's <@slack_id|short_name> to Zulip's @**full_name**.
    * Checking links.
2018-01-08 08:05:50 -05:00
Rhea Parekh a6e0bcd5f6 Slack importer: Map messages and usermessages. 2018-01-08 08:05:50 -05:00
Rhea Parekh 24a86a524b slack importer: Quiet the unzip command. 2018-01-08 08:05:50 -05:00
Rhea Parekh 4019a456d1 slack importer: Rename messages file.
Rename messages file from '/message-000001.json' to
'/messages-000001.json'
2018-01-08 08:05:50 -05:00
Rhea Parekh 400c968b0e slack importer: Remove unnecessary comment. 2018-01-08 08:05:50 -05:00
Rhea Parekh c875428a83 slack importer: Extract get_user_full_name() helper. 2018-01-08 08:05:50 -05:00
Robert Hönig c745d1901c bot_config.py: Add function for loading config file templates.
load_bot_config_template(bot) parses the <bot>.conf
template file, which can be found in the zulip_bots
package for each bot. It then returns the INI content
of that file as a dict.
2018-01-07 20:05:52 +01:00
Robert Hönig ed7208fd6e Make EmbeddedBotHandler.get_config_info comply with ExternalBotHandler. 2018-01-07 20:05:52 +01:00
Robert Hönig 9cc059f4aa bot_config.py: Raise exception when getting config data from bot without config entries. 2018-01-07 20:05:52 +01:00
Robert Hönig dcd62f92fb validator.py: Extend functionality of check_dict().
With this commit, check_dict() can be used to verify
that all keys and/or values of a dict are of a certain
type.
2018-01-07 20:04:52 +01:00
Vishnu Ks 5cadc6365a slack importer: Extract data into directory of zip file.
Currently the zip file is extracted to the root of
zulip directory no matter where the the zip file.
The extracted data is not useful after running the command
which pollutes the zulip directory. It make more sense to
extract it to the same directory of zip file especially
when the zip file gets downloaded to /temp like in the tests.
2018-01-07 09:50:27 -05:00
Robert Hönig 29a343cb44 Rename BotUserConfigData to BotConfigData.
This is in compliance with the naming of
BotStorageData.
2018-01-06 14:13:18 -05:00
Alena Volkova 5bbc46762f tests: Improve error handling of import failures.
Fixes #6776.
2018-01-05 12:03:37 -05:00
Vishnu Ks e6d3f8895f messages: Create function to calculate first visible message id. 2018-01-04 08:52:40 -05:00
Alena Volkova 45f0c76c44 settings: Limit the creation of generic bots.
This commit adds a setting to limit creation of generic bots
to admins for realms that want that restriction.  (Generic
bots, apart from being considered spammy on some realms,
have less locked down permissions than webhook bots).

Fixes #7066.
2018-01-02 18:12:22 -05:00
Aastha Gupta daf86eb664 settings: Add "text" option to emoji_set model.
We no longer have a special UI setting and model
field ("emoji_alt_code") for saying users want text-only
emojis.  We now instead make "text" be a fifth choice
for "emojiset".

Fixes #7406
2018-01-02 14:55:01 -05:00
Vishnu Ks 277d463d23 messages: Add option to limit users message history. 2018-01-02 13:45:01 -05:00
YJDave fa44d2ea69 settings: Remove autoscroll_forever setting.
Fixes #6845
2018-01-02 10:35:49 -05:00
rht 185fd99816 mypy: Use Python 3 type syntax in several files. 2017-12-30 07:34:51 -05:00
rht 2e8adbde12 mypy: Use Python 3 type syntax in zerver/lib/actions.py. 2017-12-30 07:33:07 -05:00
Eeshan Garg 89f622937d api docs: Generate Arguments table from JSON.
This commit does the following:
* Move the Arguments table data from stream-message.md and
  private-message.md to a JSON file.
* Add a Markdown extension that allows one to include and render
  a table from a JSON file like so:
    {generate_arguments_table|arguments.json|private-stream.md}
* Use Bootstrap's .table class to format the table instead of
  relying on custom CSS.
2017-12-29 17:03:36 -05:00
akashnimare ef51a39552 markdown: Update regex for strikethrough.
This uses the correct regex for strikethrough. Also, added
a test to make sure that strikethrough works when it contains
link with whitespace.

Fixes #7596.
2017-12-28 15:59:04 -05:00
Kiy4h 78e71faefe mypy: Use Python 3 type syntax in zerver/lib/test_fixtures.py. 2017-12-28 07:26:44 -05:00
Rhea Parekh ed4054d110 Import script: Check and add system bots after every import.
This checks for the existing system bots and adds them if they
aren't included in the import.
2017-12-27 07:52:45 -05:00
Rhea Parekh cd332a7f59 Slack importer: Append ids to the existing ids in the database.
For import with the condition `--import-into-nonempty`, the ids
of the mapped data will be appended to the existing ids in the
database.
2017-12-27 07:50:53 -05:00
Rhea Parekh 5b37c5562b Slack importer: Add the slack to zulip data conversion files. 2017-12-27 07:50:53 -05:00
Rhea Parekh 25e5969add zerver/lib/mobile_auth_otp: Sweep force_str. 2017-12-26 09:09:31 -05:00
Rhea Parekh cc872ad976 zerver/lib/upload.py: Sweep force_bytes from ensure_medium_avatar_image. 2017-12-26 09:09:31 -05:00
Rhea Parekh b792a90842 Remove unused imports in the codebase. 2017-12-26 09:09:31 -05:00
nyan-salmon 613856c33b mypy: Use Python 3 type syntax in zerver/lib/send_email.py. 2017-12-26 08:34:06 -05:00
Viraat Chandra de3eb89d5c mypy: Use Python 3 syntax for typing in `zerver/lib/request.py`. 2017-12-26 08:31:43 -05:00
Viraat Chandra bbf24ec68e mypy: Use Python 3 syntax for typing in `zerver/lib/webhooks/git.py`. 2017-12-26 08:31:43 -05:00
greysome f2f9efd1a7 mypy: Use Python 3 type syntax in zerver/lib/integrations.py 2017-12-26 08:30:33 -05:00
greysome fb7ee942c4 mypy: Use Python 3 type syntax in zerver/lib/export.py 2017-12-26 08:30:33 -05:00
Greg Price a34c01780f errors: Compute deployment metadata on the right deployment.
When I added this "Deployed code" feature to the error reporting,
I apparently hadn't worked out enough of how this code works to
realize that `notify_server_error` may be in a different process,
at a different time and potentially even on a different machine
from the actual error being reported.

Given that architecture, all the data about the error must be computed
in `AdminNotifyHandler`, before sending the report through the queue,
or else it risks being wrong.  The job of `notify_server_error` and
friends is only to format the data and send it off.  So, move the
implementation of this feature in order to do that.

(@showell added some "nocoverage" directives here for code that
is hard to test (exceptions being thrown, deployment files not
existing) and that was originally part of a file that didn't
require 100% coverage)
2017-12-22 11:52:13 -05:00
Greg Price eea1ceb0db errors: Show what logger and source code the message comes from. 2017-12-22 11:52:13 -05:00
Greg Price 77e47900dd errors: Stop showing a long list of Nones when there's no request.
This had contributed to making the emails look broken and of uncertain
accuracy.
2017-12-22 11:52:13 -05:00
Greg Price 68fa29c3ec errors: Make Zulip and email paths more parallel.
This helps prevent them from diverging and getting different sets of
features and fixes.  As a bonus, the email path gets a nice tweak that
the Zulip path has had for years, since f7f2ec0ac, which makes the
emails clearer and less broken-looking when logging a message with no
stack trace.
2017-12-22 11:52:13 -05:00
Greg Price b15231dfc2 logging: Rename AdminZulipHandler to AdminNotifyHandler.
This name hasn't been right since f7f2ec0ac back in 2013; this handler
sends the log record to a queue, whose consumer will not only maybe
send a Zulip message but definitely send an email.  I found this
pretty confusing when I first worked on this logging code and was
looking for how exception emails got sent; so now that I see exactly
what's actually happening here, fix it.
2017-12-22 11:52:13 -05:00
neiljp (Neil Pilgrim) 7c819f4d90 mypy: Account for Optional variables in addressee.py. 2017-12-23 01:22:22 +09:00
neiljp (Neil Pilgrim) 57e7ec94a0 mypy: Adjust 2 emoji.py functions to allow for strict-optional. 2017-12-23 01:22:22 +09:00
neiljp (Neil Pilgrim) d01dcbd018 mypy: Amend tos_version parameter of create_user_profile to be Optional.
tos_version is marked as Optional[Text] in UserProfile & is Optional
in other places within zerver/lib/create_user.py
2017-12-23 01:22:22 +09:00
Aastha Gupta 78e8d18ef3 integrations: Rename HUBOT_INTEGRATIONS_LEGACY.
It is now simply called HUBOT_INTEGRATIONS.

Fixes #7402
2017-12-22 10:38:47 -05:00
Angelika Serwa 2f575cca72 Add Dropbox webhook integration.
This is just a basic Dropbox webhook integration. It just
notifies a user when something has changed, it does not
specify what changed. Doing so would require storing data,
as Dropbox API was created mainly for file managers, not
integrations like this.
Closes #5672
2017-12-22 10:36:21 -05:00
Marco Burstein 9327048f6f mypy: Use Python 3 type syntax in `zerver/lib/events.py`. 2017-12-22 08:17:00 -05:00
Marco Burstein 017476cdfb mypy: Use Python 3 type syntax in `zerver/lib/create_user.py`. 2017-12-22 08:17:00 -05:00
Tommy Ip 9ea7b05cfe bugdown: Rewrite markdown link to relative link conditionally. 2017-12-22 08:10:42 -05:00
Tommy Ip efe2d497f4 bugdown: Avoid matching uri with regex. 2017-12-22 08:10:42 -05:00
Tommy Ip 9ce797d68c bugdown: Extract rewrite_if_relative_link function. 2017-12-22 08:10:42 -05:00
Umair Khan 898acc4278 missed-message-worker: Send email through email_senders.
We have shifted to a generic queue to send all the emails. This queue
can retry in case of network issues; this makes sure that the emails are
always sent.
2017-12-20 19:36:27 -08:00
neiljp (Neil Pilgrim) 25d5a2ee1c requirements: Upgrade mypy to 0.560.
Fixes #7835.
2017-12-20 18:09:36 -08:00
snlkapil 1589c597d2 settings: Show local time instead of UTC in invitations tab. 2017-12-20 07:08:46 -05:00
Rishi Gupta 27a8ef86e4 actions: Add realm deactivation and reactivation to RealmAuditLog. 2017-12-19 17:46:36 -08:00
Rishi Gupta 4261307e27 emails: Clear ScheduledEmail emails on realm deactivation.
This only affects the invitation reminder emails; clear_scheduled_emails was
already clearing the other two types.
2017-12-19 17:46:36 -08:00
Rishi Gupta 869b4d41ef models: Add ScheduledEmail.realm.
The two extra queries in the test are due to the assert in
send_future_email.
2017-12-19 17:46:36 -08:00
Andy Perez 44e6d9b54b testing: Add nice diffs to markdown python tests. 2017-12-18 19:03:38 -05:00
Greg Price 776af2e248 push notifs: Simplify content-truncation code slightly.
This should be a pure refactor.
2017-12-18 15:32:56 -08:00
Greg Price 014900c2e5 push notifs: Include same stream and sender info on GCM as on APNs.
This takes the information added to APNs payloads by #7080, and adds
to our GCM payloads the parts that it didn't already have.
2017-12-18 15:32:56 -08:00
Greg Price e077d6fc74 push notifs: Factor out common payload fields for APNs and GCM.
This should be a pure refactor, producing exactly the same payloads.
2017-12-18 15:32:56 -08:00
Callum Fraser 0438da5c16 mypy: Use Python 3 type syntax in zerver/lib/avatar.py. 2017-12-18 11:32:52 -05:00
Alena Volkova 3cfb2d57b2 tests: Refactor the method for encoding credentials. 2017-12-18 09:24:09 -05:00
Alena Volkova 4a303d1778 tests: Add auth methods that will replace passing credentials. 2017-12-18 09:24:09 -05:00
Andy Perez f0190333b8 bugdown: Remove whitespace strip from inline code.
Fixes #4507
2017-12-14 07:48:10 -05:00
Vishnu Ks bc37317914 invites: Differentiate users invited as admin by showing bolt icon. 2017-12-13 15:12:21 -06:00
Greg Price b830b446f1 logging: Reduce `create_logger` to new `log_to_file`.
The name `create_logger` suggests something much bigger than what this
function actually does -- the logger doesn't any more or less exist
after the function is called than before.  Its one real function is to
send logs to a specific file.

So, pull out that logic to an appropriately-named function just for
it.  We already use `logging.getLogger` in a number of places to
simply get a logger by name, and the old `create_logger` callsites can
do the same.
2017-12-12 17:17:08 -08:00
Greg Price 1c810baaef logging: Stop calling basicConfig in `create_logger`.
From the docs:

> This function does nothing if the root logger already has handlers
> configured for it.

Which we do if we've started up Django and configured settings, and in
particular allowed Django to process `settings.LOGGING`.

So, cut it out -- all it can do is confuse people about how logging
works.

If we ever actually used the `log_format` parameter, this would be
doubly confused, because only the first call would have any effect.
2017-12-12 17:17:08 -08:00
Greg Price b6491fd52a logging: Eliminate log_level from create_logger.
Because calls to `create_logger` generally run after settings are
configured, these would override what we have in `settings.LOGGING` --
which in particular defeated any attempt to set log levels in
`test_settings.py`.  Move all of these settings to the same place in
`settings.py`, so they can be overridden in a uniform way.
2017-12-12 17:17:08 -08:00
Greg Price ebcf0b4876 logging: Stop having `create_logger` force loglevels to INFO.
This is already the loglevel we set on the root logger, so this has no
effect -- except in tests, where `test_settings.py` attempts to set
some of these same loggers to higher loglevels.  Because the
`create_logger` call generally runs after we've configured settings,
it clobbers that effect.

The code in `test_settings.py` that tries to suppress logs only works
because it also sets `propagate=False`, which has nothing to do with
loglevels but does cause logs at this logger (and descendants) to be
dropped completely unless we've configured handlers for this logger
(or one of its relevant descendants.)
2017-12-12 17:17:07 -08:00
Weronika Grzybowska b54f78968f markdown: Add automatic numbering of ordered lists.
Adds a markdown preprocessor that finds ordered lists where all items
use the same number and change them to be in normal increasing order,
starting with that number.

Fixes #5159.
2017-12-12 18:35:38 -06:00
Rhea Parekh dd1b14378b Streams: Disallow certain markdown characters in stream names. 2017-12-12 10:04:31 -06:00
Greg Price 8a432c5134 APNs: Update comment on versions of our APNs payload format. 2017-12-11 20:55:20 -08:00
kunall17 98943a8333 Include data of stream/private sender details in apns push notifications 2017-12-11 20:51:09 -08:00
picapi_ 85ae723c9f mypy: Use Python 3 type syntax in zerver/lib/soft_deactivation.py. 2017-12-11 20:30:19 -08:00
Greg Price e9a7173119 actions: Cut some redundant types in docstrings.
We have a better way of writing types now!

Also move a comment that should have been a docstring,
to being a docstring.
2017-12-11 19:29:11 -08:00
Shreyansh Dwivedi 47fcb27e39 invitations: Remove custom_body.
Fixes #7672
2017-12-11 19:23:54 -08:00
Steve Howell f83a73c763 mypy: Use more specific type for _replacement_destroy_test_db. 2017-12-08 19:00:18 -08:00
Steve Howell 78041a5846 mypy: Use more specific type for instrument_url wrapper. 2017-12-08 19:00:18 -08:00
Xavier Cooney a831acd97f mypy: Use Python 3 type syntax in bugdown/__init__.py. 2017-12-08 18:34:26 -08:00
Vishnu Ks a30152cb94 models: Use is_cross_realm_bot_email whenever possible.
Fixes #7557
2017-12-07 14:55:06 -08:00
Rishi Gupta 6787fbd547 invitations: Change rate limiting to use invites_sent::day.
[greg: small tweaks]
2017-12-06 20:35:50 -08:00
Rishi Gupta fbd8dde1f8 invitations: Add LoggingCountStat to keep track of sent invitations. 2017-12-06 20:35:50 -08:00
Rishi Gupta cb5eb47470 invitations: Move resend/revoke error handling out of actions.py. 2017-12-06 20:35:50 -08:00
Rishi Gupta 968aae167b invitations: Remove get_prereg_user_by_email.
The original logic is buggy now that emails can belong to (and be
invited to) multiple realms.

The new logic in the `invites` queue worker also avoids the bug where
when the PreregistrationUser was gone by the time the queue worker got
to the invite (e.g., because it'd been revoked), we threw an exception.

[greg: fix upgrade-compatibility logic; add test; explain
revoked-invite race above]
2017-12-06 20:35:50 -08:00
Rishi Gupta 8baadc6b15 invitations: Add check for invite limit to resend invite pathway. 2017-12-06 20:30:42 -08:00
Greg Price b3482429f1 invitations: Adjust a few blank lines for clarity and compactness. 2017-12-06 20:30:42 -08:00
Rishi Gupta cb527fd3c6 invitations: Move spammer check from do_invite_users into its own function. 2017-12-06 20:30:42 -08:00
Rishi Gupta e48bc9136a invites: Change resend invitation pathway to use existing code.
This code changes frequently enough that errors are bound to creep in.  The
main change is that this sends the original invitation email instead of the
reminder email, but I think that's fine.
2017-12-06 20:30:42 -08:00
picapi_ 21e629f351 mypy: Use Python 3 type syntax in zerver/lib/test_classes.py. 2017-12-05 16:32:18 -08:00
Rishi Gupta 7d1c88f0fb invitations: Limit realms to 100 invites per day.
To guard against using zulip invites as a vector for spam. Stopgap measure
until we figure out something better.
2017-11-29 22:18:05 -08:00
Greg Price 4926228071 rabbitmq: Do a better job of retrying failed connections.
Empirically, the retry in `_on_connection_closed` didn't actually work
-- if a reconnect failed, that was it, and the exception handler
didn't get run.  A traceback would get logged, but all its frames were
in Tornado or Pika, not our own code; presumably something magic and
async was happening to the exception.

Moreover, though we would make one attempt to reconnect if we had a
connection that got closed, we didn't have any form of retry if the
original attempt at connecting failed in the first place.

Happily, upstream offers a perfectly reasonable bit of API that avoids
both of these problems: the on-open-error callback.  So use that.
2017-11-29 16:56:29 -08:00
Greg Price 7ac2b58584 rabbitmq: Reorder a bit to group our reconnect logic together. 2017-11-29 16:56:29 -08:00
Greg Price c32b16715d tornado: Use spiffy new `call_later` rather than `add_timeout`.
This method was new in Tornado 4.0.  It saves us from having to get
the time ourselves and do the arithmetic -- which not only makes the
code a bit shorter, but also easier to get right.  Tornado docs (see
http://www.tornadoweb.org/en/stable/ioloop.html) say we should have
been getting the time from `ioloop.time()` rather than hardcoding
`time.time()`, because the loop could e.g. be running on the
`time.monotonic()` clock.
2017-11-29 16:56:29 -08:00
Greg Price 73886f57d2 zerver/lib/queue: Clean up import order. 2017-11-29 16:56:29 -08:00
Greg Price 3c4e4c14c9 rabbitmq: Add on-close callback atomically in creating the connection.
Adding it afterward is inherently racy, and upstream's API is quite
reasonable for avoiding that -- just like we can pass an on-open
callback up front, we can do the same with the on-close callback.

This is a more thorough version of 4adf2d5c2 from back in 2013-04.
2017-11-29 16:56:29 -08:00
Greg Price e88c2a7ee4 rabbitmq: Cut redundant `stop_ioloop_on_close` parameter.
The default value of this parameter is already False upstream.
(It was already False in pika version 0.9.6, which we were
supposedly using when we introduced this in 4baeaaa52; not sure
what the story was there.)
2017-11-29 16:56:29 -08:00
Tim Abbott be832387c5 user_groups: Fix reporting of empty user groups.
Previously, we weren't doing a proper left join in
user_groups_in_realm_serialized, resulting in empty user groups being
excluded from the query.  We want to leave decisions about excluding
empty user groups to the UI layer, so we include these here.
2017-11-29 16:13:55 -08:00
Tim Abbott b2cb443d24 subs: Fix clearing unread counts when leaving private streams.
Because we use access_stream_by_id here, and that checks for an active
subscription to interact with a private stream, this didn't work.

The correct fix to add an option to active_stream_by_id to accept an
argument indicating whether we need an active subscription; for this
use case, we definitely do not.
2017-11-29 14:40:08 -08:00
Tim Abbott 6bc1a682de notifications: Separate push and email notifications checks.
This is an early step in a larger refactor to properly decouple the
email and push notification code paths.
2017-11-28 17:51:18 -08:00
rht ee546a33a3 zerver/lib: Use python 3 syntax for typing.
Edited by tabbott to improve various line-wrapping decisions.
2017-11-28 17:15:14 -08:00
rht 229a8b38c0 zerver/lib: Use Python 3 syntax for typing for several files.
This adds a number of annotations that had been missed in previous
passes.
2017-11-28 17:02:24 -08:00
Tim Abbott 586e18b237 auth: Allow accounts with the same email in different realms.
[Modified by greg to (1) keep `USERNAME_FIELD = 'email'`,
(2) silence the corresponding system check, and (3) ban
reusing a system bot's email address, just like we do in
realm creation.]
2017-11-28 16:23:10 -08:00
Tim Abbott e6f460f511 auth: Replace user_email_is_unique validator.
As we migrate to allow reuse of the same email with multiple realms,
we need to replace the old "no email reuse" validators.  Because
stealing the email for a system bot would be problematic, we still ban
doing so.

This commit only affects the realm creation logic, not registering an
account in an existing realm.
2017-11-28 16:23:10 -08:00
Greg Price b6cc21b438 debug: Add facility to dump tracemalloc snapshots.
Originally this used signals, namely SIGRTMIN.  But in prod, the
signal handler never fired; I debugged fruitlessly for a while, and
suspect uwsgi was foiling it in a mysterious way (which is kind of
the only way uwsgi does anything.)

So, we listen on a socket.  Bit more code, and a bit trickier to
invoke, but it works.

This was developed for the investigation of memory-bloating on
chat.zulip.org that led to a331b4f64 "Optimize query_all_subs_by_stream()".

For usage instructions, see docstring.
2017-11-28 15:52:07 -08:00
derAnfaenger c8a5ae753c embedded bots: Consistently use 'storage' instead of 'state.' 2017-11-27 21:05:34 -08:00
Tim Abbott 9645c8b31f integrations: Remove disabled googlesearch integration.
This was moved to another part of the python-zulip-api codebase
because it was unfinished.
2017-11-27 20:50:37 -08:00
Vishnu Ks 985768b2fd registration: Check realm against PreregistrationUser realm.
We would allow a user with a valid invitation for one realm to use it
on a different realm instead.  On a server with multiple realms, an
authorized user of one realm could use this (by sending invites to
other email addresses they control) to create accounts on other
realms. (CVE-2017-0910)

With this commit, when sending an invitation, we record the inviting
user's realm on the PreregistrationUser row; and when registering a
user, we check that the PregistrationUser realm matches the realm the
user is trying to register on.  This resolves CVE-2017-0910 for
newly-sent invitations; the next commit completes the fix.

[greg: rewrote commit message]
2017-11-27 14:58:26 -08:00
Tim Abbott 8b935f4e99 settings: Add setting for SYSTEM_BOT_REALM.
This fixes some subtle JavaScript exceptions we've been getting in
zulipchat.com, caused by the system bot realm there not being "zulip"
interacting with get_cross_realm_users.
2017-11-27 14:46:07 -08:00
Tim Abbott 25fd4c5508 bulk_get_users: Edit the cache keys to make them more unique.
While at this point I was to rewrite this function, this at least
plugs the issues for now.
2017-11-27 14:41:31 -08:00
Tim Abbott 339b67f7af get_cross_realm_dicts: Harden against caching bug in last commit.
This should help protect us from future issues with the way that
`bulk_get_users` does caching.

It's likely that we'll want to further restructure `bulk_get_users` to
not have this base_query code path altogether (since it's kinda
buggy), but I'm going to defer that for a time when we have another
user.
2017-11-27 14:35:50 -08:00
Tim Abbott 646ba5b9e5 bulk_get_users: Fix issues with users in multiple realms.
The previous implementation had a subtle caching bug: because it was
sharing its cache with the `get_user_profile_by_email` cache, if a
user happened to have an email in that cache, we'd return it, even
though that user didn't match `base_query`.

This causes `get_cross_realm_users` to no longer have a problematic
caching bug.
2017-11-27 14:34:45 -08:00
Rohitt Vashishtha 6734b00c0b markdown: Hide URL if message is only an image.
Hides URL if the message content == image url so that sending gifs or
images feels less cluttered. Uses the url_to_a() function to generate
the expected url string for matching.

Fixes #7324.
2017-11-27 13:30:18 -08:00
Tim Abbott 649b7b8b9f internal_send_message: Enforce being passed cross-realm bots.
We include ERROR_BOT in this set, even though it's not technically
cross-realm (it just lives in the admin realm).

This code path does not correctly handle emails that correspond to
multiple accounts (because `get_system_bot` does not).  Since it's
intended to only be used by system bots, we add an appropriate
assertion to ensure it is only used for system bots.
2017-11-26 17:14:23 -08:00
Tim Abbott ae8e203717 actions: Merge internal_prep_message into internal_send_message.
We also document the fact that only system bots are supported here.
2017-11-26 17:14:23 -08:00
Tim Abbott 6952dcbdac embedded bots: Stop using internal_send_message for non-system-bots.
This was causing problems, because internal_send_message assumes that
there is a unique user (across all realms) with the given email
address (which is sorta required to support cross-realm bot messages
the way it does).

With this change, it now, in practice, only sends cross-realm bot
messages.
2017-11-26 17:14:23 -08:00
Tim Abbott 27582782fa embedded bots: Flip around ordering of rate-limiting check. 2017-11-26 17:14:23 -08:00
Tim Abbott 5306a9634d email_mirror: Rewrite to not use internal_send_message.
This was causing problems with the fact that `get_system_bot` now only
works for actual system bot users.
2017-11-26 17:14:23 -08:00
Tim Abbott 2210f627a5 signup: Switch active mirror-dummy users to an AssertionError.
Previously, this was a ValidationError, but that doesn't really make
sense, since this condition reflects an actual bug in the code.

Because this happened to be our only test coverage the ValidationError
catch on line 84 of registration.py, we add nocoverage there for now.
2017-11-26 16:13:45 -08:00
Vishnu Ks b9bc1c2b33 Eliminate get_user_profile_by_email from test_classes. 2017-11-26 15:47:56 -08:00
Tim Abbott 9fcdb6c83a notifications: Use call_consume_in_tests for email sending.
This adds a bit of test coverage with really no downside.
2017-11-26 11:46:07 -08:00
Robert Hönig 0e0a8a2b14 queue processor tests: Call consume by default.
This significantly improves the API for queue_json_publish to not be
overly focused on what the behavior of this function should be in our
unit tests.
2017-11-26 11:45:34 -08:00
Tim Abbott 6968b540c8 mypy: Remove type: ignores that are unnecessary with new mypy.
Mostly these can be removed because the broken LXML stubs were removed
from typeshed in https://github.com/python/typeshed/issues/525.
2017-11-25 10:06:28 -08:00
Vaibhav Sagar 711f7bca3b upload: Fix typo in animated GIF error message.
s/must be have/must have the/.
2017-11-24 09:08:38 -08:00
Andy Perez 0449dc8b8a bugdown: Fix handling of nested fenced math blocks.
Change any fence delimiters (backticks and tildes) to be
interpreted literally inside of math fenced_code blocks.

Fixes #5190.
2017-11-22 12:19:43 -08:00
Harshit Bansal 2e7d0cb4cb digests: Fix new stream links.
Fixes: #7479.
2017-11-22 10:07:36 -08:00
Tim Abbott 8f6b39a1c0 mypy: Fix annotations for json_encoder_for_html. 2017-11-22 01:32:46 -08:00
rht e169bb0954 zerver: Remove simplejson dependency.
Modified by tabbott to put the third-party code in a new file.

Fixes #6970.
2017-11-21 21:51:48 -08:00
Tim Abbott b4e67fac36 bot_lib: Fix storage bug introduced by rebase error. 2017-11-21 21:37:54 -08:00
derAnfaenger e526d0c144 embedded bots: Add views to access state. 2017-11-21 21:10:39 -08:00
derAnfaenger d2af8d4cbd embedded bots: Support multi-entry state modification. 2017-11-21 21:10:39 -08:00
derAnfaenger cbadd3d430 embedded bots: Simplify StateError exception messages. 2017-11-21 21:10:12 -08:00
Tim Abbott 141cf21b86 bulk_create: Fix buggy logic for generating recipients_by_email.
This buggy logic from e1686f427c had
broken do-destroy-rebuild-test-database.

Now that we're not just trying to add the Recipient objects for every
user on the system here to profiles_by_id, we also shouldn't be
processing every Recipeint object on the system.  The fix is simple:
because of the patch we got merged into Django upstream,
recipients_to_create actually has the object IDs added to the
Recipient objects passed into Recipient.objects.bulk_create.

This was missed in manual testing, since it only broke `populate_db
--test-suite`.
2017-11-21 21:07:32 -08:00
rht 3f4bf2d22f zerver/lib: Use python 3 syntax for typing.
Extracted from a larger commit by tabbott because these changes will
not create significant merge conflicts.
2017-11-21 20:56:40 -08:00
rht 561ba33f69 zerver/lib: Use python 3 syntax for typing.
Split by tabbott from a larger commit; this covers a batch of files
with no open PRs touching them.
2017-11-21 20:45:52 -08:00
Eeshan Garg 73d0f1db81 lib/integrations: Remove Integration.doc_context.
An Integration object doesn't need access to the context dict used
to render its doc.md, since the context dict is just passed directly to
render_markdown_path.
2017-11-21 20:39:30 -08:00
Tim Abbott e1686f427c bulk_create: Remove assumption that UserProfiles are globally unique.
This isn't used in production, but that could change in the future,
and the fix to make this limited to a single realm is pretty simple.
2017-11-21 20:23:06 -08:00
Vishnu Ks 766511e519 actions: Mark all messages as read when user unsubscribes from stream.
This fixes a bug where, when a user is unsubscribed from a stream,
they might have unread messages on that stream leak.  While it might
seem to be a minor problem, it can cause significant problems for
computing the `unread_msgs` data structures, since it means we need to
add an extra filter for whether the user is still subscribed, either
in the backend or in the UI.

Fixes #7095.
2017-11-21 20:09:17 -08:00
Tim Abbott 3bfb19b5f3 Convert EmailAuthBackend and LDAPAuthBackend to accept a realm. 2017-11-21 18:23:50 -08:00
Vishnu Ks 34689370cd settings: Create UI and backend for setting signups stream. 2017-11-21 17:39:51 -08:00
Vishnu Ks 3d2c9c6098 models: Replace core team with Realm.INITIAL_PRIVATE_STREAM_NAME. 2017-11-21 17:39:51 -08:00
Vishnu Ks a0275a6257 actions: Send new user messages to signup_notifications_stream. 2017-11-21 17:39:50 -08:00
Rishi Gupta 42652713c8 onboarding: Refactor away setup_initial_private_stream.
It is a confusing name, now that the stream creation part has been taken
away.
2017-11-21 17:39:50 -08:00
Vishnu Ks 3d17897563 onboarding: Move signup notifications stream creation to do_create_realm. 2017-11-21 17:39:50 -08:00
Tim Abbott 29408d24ff emoji: Rework reactions validation to have a single function.
This feels more readable to me, and also identified a couple cases
where we were missing test coverage.
2017-11-20 15:27:21 -08:00
Harshit Bansal 87b523f3ef emoji.py: Refactor emoji data consistency checking functions.
Inorder to provide more explicit error messages I have merged the
`emoji_code_is_valid()` and `emoji_name_is_valid()` functions into
`check_emoji_code_consistency()` and `check_emoji_name_consistency()`
respectively.
2017-11-20 15:27:21 -08:00
Tim Abbott 25ddba99f6 lint: Ban general use of user_profile.save().
This often can cause minor caching problems.

Obviously, it'd be better if we had access to the AST and thus could
do this rule for UserProfile objects in general.
2017-11-20 10:57:08 -08:00
Eeshan Garg 9138d3b8d7 integrations: Move integration-specific context to integration_doc.
Instead of populating the context dict with integration-specific
information in render_markdown_path, we now do that in
zerver.views.integrations.integration_doc instead.

Fixes #7401.

Tweaked by tabbott to use cast to handle the typing issues here.
2017-11-19 20:48:46 -08:00
rht 33b1a541d7 zerver/lib: Use python 3 syntax for typing.
With tweaks by tabbott to fix line spacing.
2017-11-18 16:09:04 -08:00
Tim Abbott 81adc92db6 actions: Use get_system_bot for fetching WELCOME_BOT. 2017-11-17 17:25:56 -08:00
Tim Abbott 1ed50ee858 tests: Pass a realm option to login for non-zulip realms.
This better matches the model of how having multiple realms should
work: you need to specify which realm you're logging into.
2017-11-17 15:32:42 -08:00
Greg Price bd5905da5d error_notify: Use `git describe --always`.
This way, even if the repo lacks any tags, we still
at least get the commit ID.
2017-11-17 14:19:20 -08:00
Andy Perez 83f563aeff bugdown: Properly parse empty blockquotes.
The character ">" now only starts a blockquote if the resulting
blockquote would be non-empty.  Thus, by itself, ">" is now
interpreted literally by bugdown, fixing #687.  The message
with contents consisting of ">>>" is now parsed as a doubly
(not triply) nested blockquote with contents ">".  Properly
formed blockquotes have identical behavior as before, but now
bugdown can no longer produce empty blockquotes as output.

Fixes #2886, #687.
2017-11-17 11:41:44 -08:00
derAnfaenger 395f1e9270 embedded bots: Add database config storage.
Storage limititations are only set on the value of
a config entry, since this is the only user-accessible
part of the schema. Keys are statically set by each
embedded bot.
2017-11-16 23:06:38 -08:00
Vishnu Ks 068912ca7d tests: Add test for default stream group signup flow. 2017-11-16 21:17:31 -08:00
Vishnu Ks cc553125a1 tests: Move check_user_subscribed_only_to_streams to test_classes ZulipTestCase. 2017-11-16 21:17:31 -08:00
Harshit Bansal d9c2f613e3 api: Add new endpoint for reactions.
This endpoint will allow us to add/delete emoji reactions whose emoji
got renamed during various emoji infra changes. This was also a
required change for realm emoji migration.

This commit was tweaked significantly by tabbott for greater clarity
(with no changes to the actual logic).
2017-11-16 20:52:15 -08:00
Greg Price 3828305305 error_notify: Try `git describe` too.
Tested for graceful degradation both with `git` not existing and the
`.git` directory not existing.
2017-11-16 19:15:12 -08:00
Greg Price 41ecdd8d83 error_notify: Add some idea of the deployed code version.
Tested in dev with the help of our handy /emails/ display.
2017-11-16 19:15:12 -08:00
Greg Price 4b5c52fc99 queue: Prevent an AttributeError rather than swallowing it.
When the RabbitMQ server disappears, we log errors like these:

```
Traceback (most recent call last):
  File "./zerver/lib/queue.py", line 114, in json_publish
    self.publish(queue_name, ujson.dumps(body))
  File "./zerver/lib/queue.py", line 108, in publish
    self.ensure_queue(queue_name, do_publish)
  File "./zerver/lib/queue.py", line 88, in ensure_queue
    if not self.connection.is_open:
AttributeError: 'NoneType' object has no attribute 'is_open'

During handling of the above exception, another exception occurred:
[... traceback of connection failure inside the retried self.publish()]
```

That's a type error -- a programming error, not an exceptional
condition from outside the program.  Fix the programming error.

Also move the retry out of the `except:` block, so that if it also
fails we don't get the exceptions stacked on each other.  This is a
new feature of Python 3 which is sometimes indispensable for
debugging, and which surfaced this nit in the logs (on Python 2 we'd
never see the AttributeError part), but in some cases it can cause a
lot of spew if care isn't taken.
2017-11-16 18:33:14 -08:00
Rhea Parekh 8c4651d716 zerver/lib/actions.py: Fix translation bugs.
Fixes #7396.
2017-11-16 11:05:49 -08:00
Tim Abbott 054952a44a docs: Update links from codebase to point to ReadTheDocs. 2017-11-16 10:53:49 -08:00
David Rosa Tamsen 7072fa5b37 docs: Reorganize developer docs to improve navigation.
This commit helps reduce clutter on the navigation sidebar.
Creates new directories and moves relevant files into them.
Modifies index.rst, symlinks, and image paths accordingly.

This commit also enables expandable/collapsible navigation items,
renames files in docs/development and docs/production,
modifies /tools/test-documentation so that it overrides a theme setting,
Also updates links to other docs, file paths in the codebase that point
to developer documents, and files that should be excluded from lint tests.

Note that this commit does not update direct links to
zulip.readthedocs.io in the codebase; those will be resolved in an
upcoming follow-up commit (it'll be easier to verify all the links
once this is merged and ReadTheDocs is updated).

Fixes #5265.
2017-11-16 09:45:08 -08:00
Harshit Bansal 3b95a6d2d9 bugdown: Fix a bug in the updation logic of markdown engines.
While fixing an issue related to email gateway messages not getting
rendered properly, I unknowingly introduced a bug in the markdown
engine updation code. This commit fixes it. The issue was that for
a realm having email gateway setup, updation of realm filters would
lead to the updation of only one of the markdown engines not both.
2017-11-15 22:47:11 -08:00
Harshit Bansal b849c5e551 bugdown: Remove an unnecessary check in `do_convert()`. 2017-11-15 22:47:11 -08:00
Harshit Bansal 5e5bb02e39 bugdown: Remove `make_realm_filters()`.
This commit removes `make_realm_filters()` and merges its functionality
in `make_md_engine()` and `maybe_update_markdown_engines()`.
2017-11-15 22:47:11 -08:00
Harshit Bansal f9f9e92443 bugdown: Correct the descriptions of `Bugdown` config. 2017-11-15 22:47:11 -08:00
Harshit Bansal e615a2ab4f bugdown: Simplify the "opts" parameter of `make_md_engine()`. 2017-11-15 22:47:11 -08:00
Harshit Bansal e3f2f67d0e bugdown: Rename `maybe_update_realm_filters()`.
This commit renames `maybe_update_realm_filters()` to
`maybe_update_markdown_engines()` which sounds closer
to the semantics.
2017-11-15 22:47:11 -08:00
Umair Khan 98be0cc502 user-groups: Send delete user group event.
Fixes #7380
2017-11-15 22:35:43 -08:00
Umair Khan cc76f7deac user-groups: Create check_delete_user_group. 2017-11-15 22:35:43 -08:00
Umair Khan 79f4a7627f user-groups: Send remove members event. 2017-11-15 22:35:23 -08:00
Umair Khan cec3f19366 user-groups: Send add members event. 2017-11-15 22:35:23 -08:00
Umair Khan eb48fab495 user-groups: Send description update event. 2017-11-15 22:35:23 -08:00
Umair Khan 5571122120 user-groups: Send name update event. 2017-11-15 22:35:23 -08:00
Umair Khan 912505317a user-groups: Send create group event. 2017-11-15 22:35:23 -08:00
Umair Khan 31efe951b7 user-groups: Fix function argument bug; pass group.
In remove_members_from_group_backend, we are passing user group to
remove_members_from_user_group. In remove_members_from_user_group,
expect user_group_id.
2017-11-15 22:35:23 -08:00
Umair Khan 66179ea295 bulk_add_members_to_user_group: Pass user group.
Previously we passed user group id. If we pass user group, send event is
easier.
2017-11-15 22:35:23 -08:00
Umair Khan 5ef3974e80 user_groups_in_realm_serialized: Sort groups. 2017-11-15 22:35:23 -08:00
Tim Abbott f02e5b90f6 cross_realm: Use bulk_get_users to fix handling of missing users.
This fixes a regression in ae5ba7f4fd,
where Zulip would 500 if the newly added system bots didn't exist on
the server.

This also fixes a moderate size performance problem where we'd fetch 5
users from memcached or the database in a loop.
2017-11-15 21:24:51 -08:00
Tim Abbott c7a975e4df users: Move check_change_full_name to actions.py.
This avoids an import loop in the next commit, and better matches our
usual code structure.
2017-11-15 17:39:09 -08:00
Vishnu Ks 6f955fbf34 actions: Replace stream.add with stream.append in process_new_human_user.
stream is a list not a set. It used to be set
and was changed to stream before getting merged.
2017-11-15 17:06:43 -08:00
Tim Abbott 6a5b965a90 integrations: Rename HUBOT_LOZENGES to HUBOT_INTEGRATIONS. 2017-11-15 14:04:19 -08:00
Aastha Gupta 3b53c692a4 integrations: Add integrations/doc/assembla.
Significantly modified by tabbott and rishig to be more maintainable.

Fixes #4542.
2017-11-15 14:04:19 -08:00
Tim Abbott 85844107fd integrations: Remove code for legacy Hubot lozenges.
We no longer display the only model anyway.
2017-11-15 14:04:19 -08:00
rht 09af29b051 zerver/lib: Text-wrap long lines exceeding 110. 2017-11-15 10:58:03 -08:00
Tim Abbott 5aa6751e7a lint: Fix long lines in newly added features. 2017-11-15 10:58:03 -08:00
Vishnu Ks d2555e8769 api: Add support to change name of default stream groups. 2017-11-14 14:41:42 -08:00
Vishnu Ks 2af249dd5d api: Add description attribute to default stream group. 2017-11-14 14:41:42 -08:00
Vishnu Ks f45ba7de93 api: Pass group id instead of name to default stream group api. 2017-11-14 14:41:42 -08:00
rht cceba2023c Use makedirs directly instead of wrapping it in mkdirs. 2017-11-14 09:47:09 -08:00
rht 72ce1c6501 Use a more concise `mkdir -p` when the situation permits. 2017-11-14 09:47:09 -08:00