Commit Graph

3304 Commits

Author SHA1 Message Date
Tim Abbott 6e55342e21 bulk_remove_subscriptions: Pass client object in.
We need the client object to pass on to do_mark_stream_as_read.
2018-08-01 16:48:31 -07:00
Tim Abbott c60f197fde do_update_message_flags: Accept a Client object.
This is important for upcoming logging changes.
2018-08-01 16:40:58 -07:00
Tim Abbott 5f0519dfb4 do_update_pointer: Pass client object from callers.
We also fix an unused import.

This is needed for upcoming logging changes.
2018-08-01 16:40:15 -07:00
Tim Abbott 54d558b128 management: Add library for getting a client object.
This is to be used in some analytics features we're adding in the near
future.
2018-08-01 16:40:15 -07:00
Tim Abbott e9f4d9db2b push_notifications: Fix interface for handle_remove_push_notification.
This really should just accept a message ID.
2018-08-01 16:36:42 -07:00
Kunal Gupta bc43eefbfb notifications: Add function for cancelling GCM notifications.
This adds a new function called handle_remove_push_notification in
zerver/lib/push_notifications.py which requires user_profile id and
the message id which has to be removed in the function.

For now, the function only supports GCM (and is mostly there for
prototyping).

The payload which is being delivered needs to contain the narrow
information and the content of the message.
2018-08-01 15:59:04 -07:00
Tim Abbott 5f8d193bb7 notifications: Include realm_uri in push notifications.
This should make it much simpler for the mobile apps to line up the
data from server_settings against the data in the notifications.

Addresses part of #10094.
2018-08-01 15:46:15 -07:00
Tim Abbott 58ee3fa8c4 page_params: Include avatars and similar data in cross-realm bots.
This ensures that the format of this data structures matches that for
in-realm bots in the main users data structure (including avatars,
etc.).

Fixes #10138.
2018-08-01 15:09:11 -07:00
Rhea Parekh ee37866687 import: Add gitter import file in zerver/data_import directory. 2018-08-01 11:52:14 -07:00
Rhea Parekh b8e1e8b31d import: Add slack import files in zerver/data_import directory. 2018-08-01 11:52:14 -07:00
Roman Godov 5e70577f84 models: Rename Realm.show_digest_email field.
This renames Realm.show_digest_email field to
digest_emails_enabled, for greater clarity as to what it does
just from seeing the setting name, without having to look it up.

Fixes part of #10042.
2018-08-01 11:05:58 -07:00
Tim Abbott 1b2a26ca83 events: Fix missing empty custom profile data dict for new users.
We were getting event-handling exceptions in JS in production if a new
user was created and then went and set a custom profile field, because
there was no `.profile_data` on their user object.  We were able to
trace the issue down to the fact that our events didn't include that
field when creating a new user.
2018-07-31 11:08:11 -07:00
Roman Godov c0806917ec models: Rename Realm.restricted_to_domain field.
This renames Realm.restricted_to_domain field to
emails_restricted_to_domains, for greater clarity as to what it does
just from seeing the setting name, without having to look it up.

Fixes part of #10042.
2018-07-31 09:28:33 -07:00
Tim Abbott 7ea5987e5d errors: Use a setting to control the stream for slow-query logs.
We already had a setting for whether these logs were enabled; now it
also controls which stream the messages go to.

As part of this migration, we disable the feature in dev/production by
default; it's not useful for most environments.

Fixes the proximal data-export issue reported in #10078 (namely, a
stream with nobody ever subscribed to having been created).
2018-07-30 17:40:20 -07:00
Aditya Bansal 77651ece39 thumbnails: Rename size value 'original' to 'full'. 2018-07-30 13:00:23 -07:00
Aditya Bansal 5b5d8bb310 thumbnails: Rename data-original to data-src-fullsize. 2018-07-30 13:00:23 -07:00
Tim Abbott 07af59d4cc tornado: Split get_events_backend into two functions.
The lower-layer function, now called get_events_backend, is intended
to be called by multiple code paths (including the upcoming
get_events_internal).
2018-07-30 12:28:31 -07:00
Tim Abbott 00f6fc2559 subs: Limit editing subscribers for waiting period users.
Does not let you subscribe other users if you are a guest or a
waiting period user.

Co-authored-by: Shubham Padia <shubhamapadia@gmail.com>
2018-07-30 12:12:51 -07:00
Shubham Padia 1015b2fc44 subs: Refactor and add tests for UserProfile.can_create_streams.
This is a preparatory refactor for adding
UserProfile.can_subscribe_other_users.
Although there existed a test for limiting users from creating
streams at `test_subs.test_user_settings_for_adding_streams`,
it did not test the logic inside can_add_streams, tests have
been added to solve that issue.
2018-07-30 12:12:51 -07:00
Rohitt Vashishtha d83ef30904 notification-bot: @-mention the user subscribing you to a stream. 2018-07-30 09:50:48 -07:00
Rhea Parekh 6e7905651f mypy: Fix type errors relate to 'mock'.
1. The type of a mock object should be 'Any'.
2. 'mock' is being imported in the wrong manner.
2018-07-28 15:34:54 -07:00
Shubham Dhama 499a70b01e notifications: Fix leaking of private stream msgs sent before subscribing.
Fixes: #9834.
2018-07-28 15:13:08 -07:00
Shubham Dhama c5d8fdf68c message: Add function to check message access in bulk.
This effectively just calls access_message in a loop.
2018-07-28 15:12:55 -07:00
Shubham Dhama bd5e39cd9e test_message: Rename StarTests to MessageAccessTests. 2018-07-28 14:58:36 -07:00
Tim Abbott a57d337113 bots: Enforce bot creation policy on reactivating bots.
It's sorta an unusual state to get into, to have a user own a
deactivated bot, when they can't create a bot of that type, but
definitely a valid possibility that we should be checking for.

Fixes #10087.
2018-07-27 10:31:38 -07:00
Harshit Bansal cf5b2b4815 emoji: Change emoticon mapping for `:)`, `(:` and `:(`.
See discussion on CZO:
https://chat.zulip.org/#narrow/stream/101-design/subject/emoji.20picker/near/617811
2018-07-26 11:17:03 -07:00
Yago González cb24756edc api docs: Merge docs for sending stream and private messages. 2018-07-26 10:24:01 -07:00
Shubham Dhama ba366ea595 bugdown: Fix test_ultra_long_rendering logic.
Previously the BugdownRenderingException is caused in except path
because of KeyError not in try block.
2018-07-26 09:27:55 -07:00
Aman Agrawal 5e3428dd8e tests: Add check for duplicate items in check_events_dict.
This check raises AssertionError if list containing duplicate items
is passed through `check_events_dict` function.
2018-07-25 09:25:27 +05:30
amanagr 284259d1c3 tests: Remove redundant test for message 'flags' in test_events.py. 2018-07-23 23:10:07 -07:00
Tim Abbott 08d890e671 test_events: Use check_dict_only for stream_weekly_traffic. 2018-07-23 17:37:24 -07:00
Rishi Gupta 0573c17819 stream settings: Set weekly traffic to None instead of -1 for new streams.
The only place this gets used seems to be subscription.handlebars, where a
value of None (or null) is probably less broken-seeming than -1 anyway.
2018-07-23 16:43:15 -07:00
Joshua Pan a9d9919648 Add delivery_email field to page_params.
This allows the frontend to use the `delivery_email`
field for display use.
2018-07-23 15:38:41 -07:00
Tim Abbott 8dca8f1087 search: Add setting to control whether search pills are enabled.
This setting isn't intended to exist long term, but instead to make it
possible to merge our search pills code before we're ready to cut over
production environments to use it.
2018-07-23 11:29:10 -07:00
Rhea Parekh a656257e6b gitter import: Process user mentions.
The gitter mentions are in the format '@usermention'
and the mentions are included in the export data as:

"mentions": [
   {
    "screenName": "usermention",
    "userId": "54d7876c15522ed4b3dbbefb",
    "userIds": []
   }]

We extract this data and map this mention to @**usermention**
for Zulip.
2018-07-23 08:36:30 -07:00
Rhea Parekh a0034bfb6e gitter import: Add basic tests. 2018-07-23 08:36:30 -07:00
Rhea Parekh fe299277f3 import: Import BotStorageData and BotConfigData. 2018-07-23 08:21:00 -07:00
Rhea Parekh 2978e025df import: Import UserGroup. 2018-07-23 08:21:00 -07:00
Rhea Parekh f444e5b628 import: Import MutedTopic. 2018-07-23 08:20:58 -07:00
Rhea Parekh 091d101e7d import: Import UserHotspot. 2018-07-23 08:20:58 -07:00
Rhea Parekh 6f7b7e143f import: Map user IDs for custom profile field of type 'USER'.
The CustomProfileField object which has the `field_type`
`USER` needs to be updated with the new user IDs.
2018-07-23 08:19:04 -07:00
Tim Abbott bdc95b5d72 slack webhooks: Eliminate unnecessary use of Service model.
The user ID of the bot user is a much more useful value to be sending
to the third-party API here.
2018-07-23 08:12:08 -07:00
Vishnu Ks 2b28042ddf models: Rename remaining audit log event types to past tense.
This makes the realm audit event type log entries more consistent.
2018-07-22 20:00:28 -07:00
Steve Howell 5d0a113634 minor: Add assertion to assert_realm_values() helper. 2018-07-19 12:07:11 -04:00
Aditya Bansal 98a4e87e1d thumbor: Complete implementation of thumbnailing.
Various pieces of our thumbor-based thumbnailing system were already
merged; this adds the remaining pieces required for it to work:

* a THUMBOR_URL Django setting that controls whether thumbor is
  enabled on the Zulip server (and if so, where thumbor is hosted).

* Replaces the overly complicated prototype cryptography logic

* Adds a /thumbnail endpoint (supported both on web and mobile) for
  accessing thumbnails in messages, designed to support hosting both
  external URLs as well as uploaded files (and applying Zulip's
  security model for access to thumbnails of uploaded files).

* Modifies bugdown to, when THUMBOR_URL is set, render images with the
  `src` attribute pointing /thumbnail (to provide a small thumbnail
  for the image), along with adding a "data-original" attribute that
  can be used to access the "original/full" size version of the image.

There are a few things that don't work quite yet:
* The S3 backend support is incomplete and doesn't work yet.
* The error pages for unauthorized access are ugly.
* We might want to rename data-original and /thumbnail?size=original
  to use some other name, like "full", that better reflects the fact
  that we're potentially not serving the original image URL.
2018-07-15 00:39:41 +05:30
Sarah de947445ca event_queue: Add stream_email_notify.
This adds support to the event queue system for triggering
missed-message notifications (whether push or email) to support the
stream push notifications feature.
2018-07-14 12:19:33 +05:30
Sarah 73f5be65e8 zerver/lib/notifications: Add stream email notifications.
This modifies the logic for formatting outgoing missed-message emails
to support the upcoming stream email notifications feature (providing
a new format for the subject, etc.).
2018-07-14 12:19:33 +05:30
Tim Abbott bc861b4441 notifications: Pass trigger to do_send_missedmessage_events_reply_in_zulip.
This change converts our logic for determining whether the current
user was mentioned in a group of messages from the implicit "if it was
sent to a stream, it's a mention" to the explicit "we actually know
there was a mention in the message".  This is an important
prerequisite for our upcoming feature to support getting email
notifications for streams always (even without a mention).
2018-07-14 12:02:08 +05:30
Tim Abbott b0c393cf97 notifications: Format missed_messages argument as a dict.
Because in upcoming commits, we'll want to pass additional per-message
data into do_send_missedmessage_events_reply_in_zulip, we need to
expand the format for how we represent messages to account for that.
2018-07-14 11:34:04 +05:30
Aditya Bansal 2e837b1407 archives: Change the topic/<topic_name> code path to use topics prefix.
We do this for maintaining consistency in naming of the endpoints.
2018-07-14 10:39:35 +05:30
Tim Abbott bf1ad714da actions.py: Refactor generate_topic_history_from_db_rows for clarity.
This refactors the generate_topic_history_from_db_rows function to not
depend upon the assumption of rows passed as parameter to be sorted in
reverse order of max_message_id field.

Additionally, we add sorting and some tests that verify correct
handling of these cases.
2018-07-14 10:11:08 +05:30
Aditya Bansal 63ec8b08b8 archives: Add endpoint to fetch topic history of web public streams.
In this commit we add a new endpoint so as to have a way of fetching
topic history for a given stream id without having to be logged in.
This can only happen if the said stream is web public otherwise we
just return an empty topics list. This endpoint is quite analogous
to get_topics_backend which is used by our main web app.

In this commit we also do a bit of duplication regarding the query
responsible for fetching all the topics from DB. Basically this
query is exactly the same as what we have in the
get_topic_history_for_stream function in actions.py. Basically
duplicating now is the right thing to do because this query is
really gonna change when we add another criteria for filtering
messages which is:
Only topics for messages which were sent during the period the
corresponding stream was web public should be returned.
Now when we will do this, the query will change and thus it won't
really be a code duplication!
2018-07-14 09:51:37 +05:30
Harshit Bansal f636882e04 build_emoji: Migrate to use `emoji_names.py` file.
This migrates Zulip to use a dramatically better set of names and
aliases for our emoji set, defined in emoji_names.py (which is in turn
manually generated from our hand-curated CSV file).

This should significantly improve the experience of using Zulip's
emoji picker and emoji typeahead for finding what one is looking for.
2018-07-13 21:18:02 +05:30
Aastha Gupta dfde4fac85 invitations: Send 'invites_changed' event for invitations events.
Fixes #7665

In case of invitation events, 'invites_changed' event without
any real payload is sent to all the realm admins and the user.
The event is handled by reloading the list to view recent changes.

Commit tweaked by shubhamdhama:
* Send an `invite_changed` event when an user accept an invite.
    Also, added the test for the same.
* No need to delete the invite list in frontend, current logic
    handles the case when the invite data is changed properly.
* Extracted the common logic for sending an event into
    `notify_invites_changed`.
2018-07-13 18:00:06 +05:30
Rhea Parekh fff0f64dea tests: Refactor tests for importing huddles. 2018-07-13 01:14:42 +05:30
Rhea Parekh ee830ddee9 tests: Fix mypy bug in test_import_realm. 2018-07-13 01:14:42 +05:30
Yago González 0e135b69f9 alert words: Return the updated list when adding/removing words.
POST and DELETE operations in /users/me/alert_words may leave the
user's list of alert words in an unknown state: POSTing adds words to a
list that the client may not know from the begining, and the same with
DELETE.

Replying with the current status of the alert words list is the best way
of letting the client alter the list and knowing its contents after
being updated with a single query.

This is especially useful taking into account that POSTing words that
were already present and DELETing non-existing words both produce a
successful response.

An extra test has been added to avoid leaving GET /users/me/alert_words
too untested.
2018-07-13 01:01:00 +05:30
Rhea Parekh e9884916c9 import: Support import of huddles.
For importing huddles we have to have unique huddle hashes.
Huddle hashes are extracted from the list of users participating
in a huddle. So to extract these user ids, we first use huddle
id to getting the matching recipient, and then we use subscription
to get the user ids from the recipient id.

Added tests for the same (tests slightly tweaked by tabbott).
2018-07-12 19:06:52 +05:30
Tim Abbott 5ae90d60d7 tests: Fix a failing test in test_users.py.
This small change should have been included in
d1cda29fb4.
2018-07-12 15:05:16 +05:30
Sarah d1cda29fb4 API: Add stream email notification setting plumbing.
This is all the plumbing that makes it possible to enable the
stream_email_notifications setting via the Zulip API.  The flag still
doesn't do anything yet, but this is a nice checkpoint along the way
to implementing this feature.
2018-07-12 13:39:14 +05:30
Eeshan Garg 3eaf00444a markdown: Render nested multi-line code blocks correctly.
This commit adds a Markdown tree-processor extension that renders
multi-line code blocks that are nested inside lists with the
formatting. Note that the code block could be nested inside multiple
list levels and would still get rendered correctly.

Tim: This fixes the need for unpleasant workarounds like
f5bfa4e793 and makes nested code blocks
in our documentation look exactly how users would expect them to.
2018-07-12 12:22:04 +05:30
Rhea Parekh ecc5b75500 tests: Refactor test_import_realm for clarity. 2018-07-11 23:37:06 +05:30
Yago González e9cd0ad3d6 reactions: Allow using emoji_name for removing reactions.
Given that we allow adding emoji reactions by only using the
emoji_name, we should offer the same possibility for removing
reactions to make the experience for API clients not require looking
up emoji codes.

Since this is an additional optional parameter, this also preserves
backward compatibility.
2018-07-11 19:41:43 +05:30
Yago González 9244a6740e reactions: Make the emoji_code optional for adding a rection.
Complete, correct implementations of Zulip's emoji reactions API need
to send both emoji_code and emoji_name in order to add a reaction;
this is important for corner cases around clicking on a reaction in a
message that was first reacted to a year ago, when the emoji
name->code mappings have changed for the given code point in the
intervening time.

However, for folks building tools using the Zulip API, that corner
case is not particularly common; as a result, it makes sense to offer
an interface that allows adding a reaction by only specifying the
emoji name.

This is why the only field that needs to be required is emoji_name,
which can now be mapped to a single emoji. Both fields will be
necessary when "voting" an old reaction, but since we stil allow
specifying the two of them, these changes offer retrocompatibility.
2018-07-11 19:34:48 +05:30
Aditya Bansal 9b485f3ef4 auth: Fix bug with subdomains and GitHub auth causing apparent logouts.
This adds a new settings, SOCIAL_AUTH_SUBDOMAIN, which specifies which
domain should be used for GitHub auth and other python-social-auth
backends.

If one is running a single-realm Zulip server like chat.zulip.org, one
doesn't need to use this setting, but for multi-realm servers using
social auth, this fixes an annoying bug where the session cookie that
python-social-auth sets early in the auth process on the root domain
ends up masking the session cookie that would have been used to
determine a user is logged in.  The end result was that logging in
with GitHub on one domain on a multi-realm server like zulipchat.com
would appear to log you out from all the others!

We fix this by moving python-social-auth to a separate subdomain.

Fixes: #9847.
2018-07-10 16:49:00 +05:30
Rhea Parekh 4bbccd8287 import: import RealmAuditLog when 'zerver_realmauditlog` is missing.
* If `zerver_realmauditlog` is present in the exported data,
  `RealmAuditLog` would be imported normally.

* If it is not present, `create_subscription_events`
  function in would create the `subscription_created`
  events for RealmAuditLog. The reason this function
  is in `import_realm` module and not in the individual
  export tool scripts (like Slack) is because this
  function would be common for all export tools.

This fixes #9846 for users who have not already done an import of
their organization from Slack.

Fixes #9846.
2018-07-10 16:00:19 +05:30
Rhea Parekh 70b4794816 import: import RealmAuditLog. 2018-07-10 15:53:15 +05:30
Vishnu Ks 547b5675c6 models: Add USER_SOFT_ACTIVATED event type constant to RealmAuditLog. 2018-07-10 15:42:26 +05:30
Vishnu Ks abb218ebab models: Add SUBSCRIPTION_DEACTIVATED event type constant to RealmAuditLog. 2018-07-10 15:42:26 +05:30
Vishnu Ks ff4c1ca2c1 models: Add SUBSCRIPTION_CREATED event type constant to RealmAuditLog. 2018-07-10 15:42:26 +05:30
Vishnu Ks 20fae065f6 models: Add BOT_OWNER_CHANGED event type constant to RealmAuditLog. 2018-07-10 15:42:26 +05:30
Vishnu Ks d5b6f032d2 models: Add USER_API_KEY_CHANGED event type constant to RealmAuditLog. 2018-07-10 15:42:26 +05:30
Vishnu Ks 0d69fc482b models: Add USER_TOS_VERSION_CHANGED event type constant to RealmAuditLog. 2018-07-10 15:42:26 +05:30
Vishnu Ks 83ff688bb6 models: Add USER_EMAIL_CHANGED event type constant to RealmAuditLog. 2018-07-10 15:42:26 +05:30
Vishnu Ks 75ed3f92e2 models: Add USER_FULL_NAME_CHANGED event type constant to RealmAuditLog. 2018-07-10 15:42:26 +05:30
Vishnu Ks c8aac0fbb9 models: Add USER_CHANGE_AVATAR_SOURCE event type constant to RealmAuditLog. 2018-07-10 15:42:26 +05:30
Vishnu Ks 4c73221108 models: Add USER_CHANGE_PASSWORD event type constant to RealmAuditLog. 2018-07-10 15:42:26 +05:30
Vishnu Ks 201b99a6f8 models: Add USER_REACTIVATED event type constant to RealmAuditLog. 2018-07-10 15:42:26 +05:30
Vishnu Ks d0b89cbb44 models: Add USER_DEACTIVATED event type constant to RealmAuditLog. 2018-07-10 15:42:26 +05:30
Vishnu Ks ce3fffdbb2 models: Add USER_ACTIVATED event type constant to RealmAuditLog. 2018-07-10 15:42:26 +05:30
Vishnu Ks 2c8effe9fe models: Add USER_CREATED event type constant to RealmAuditLog. 2018-07-10 15:42:26 +05:30
Yashashvi Dave 649fccde6b zerver/lib/events.py: Fix custom field values not updated in apply_events.
Custom profile field value are stored in different structure compare to
other profile fields in events, so generic way to update fields wasn't
updating custom profile fields in `apply_event` function.

Fix this by adding check for custom fields in `apply_event`.

This also adds the appropriate test_events test to verify this code path.

Fixes part of #9875.
2018-07-10 10:30:23 +05:30
Shubham Dhama 49deb5acd3 bugdown: Move global variables to arguments.py.
This has two advantages;

* We can split bugdown/__init__.py into several modules, and each
  module can access these arguments by importing these

* We get rid of the super-ugly `global db_data` construct, replacing
  it with a only slightly ugly monkey-ish patching of the
  `zerver.lib.bugdown.arguments` module, which is at least
  considerably more clear on reading as to what it's purpose is.
2018-07-09 15:45:45 +05:30
Shubham Dhama b9a8c4df8c bugdown: Move BugdownRenderingException to exceptions.py. 2018-07-09 15:45:45 +05:30
Rishi Gupta 16334a1ba7 billing: Update billing system. 2018-07-09 14:33:07 +05:30
Armaan Ahluwalia 54d3d8e8b3 webpack: Transition app.js to be compiled by webpack.
This commit moves all files previously under the 'app' bundle in
the Django pipeline to being compiled by webpack under the 'app'
entry point. In the process, it moves assets under the app entry
to a file called app.js that consumes all relevant css and js files.

This commit also edits the webpack config to be able to expose certain
variables for third party libraries that are currently required by
some modules. This is bad coding form and should be refactored to
requiring whatever dependencies a module may have; we're just
deferring that to the future to simplify the series of transitions we
need to do here. The variable exposure is done using expose-loader in
webpack.

The app/index.html template is edited to override the newly introduced
'commonjs' block in the base template. This is done as a temporary
measure so as not to disrupt other pages on the app during the transition.

It also fixes the value of the 'this' context that was being inferred
as window by third party libraries. This is done using imports-loader
in the webpack config.  This is also messy and probably isn't how we
want things to work long term.
2018-07-05 11:03:08 +02:00
Yago González 2d1adc8d99 api docs: Allow validation against schemas for any response. 2018-07-03 20:11:05 +02:00
Tim Abbott 5a99118b3e auth: Restore a minimal SocialAuthMixin.
We need to do a small monkey-patching of python-social-auth to ensure
that it doesn't 500 the request when a user does something funny in
their browser (e.g. using the back button in the auth flow) that is
fundamentally a user error, not a server error.

This was present in the pre-rewrite version of our Social auth
codebase, without clear documentation; I've fixed the explanation
part here.

It's perhaps worth investigating with the core social auth team
whether there's a better way to do this.
2018-07-03 18:53:59 +02:00
Tim Abbott c9b0c0add4 github: Refactor email extraction to use the full emails data set.
It's possible to make GitHub social authentication support letting the
user pick which of their verified email addresses to pick, using the
python-social-auth pipeline feature.  We need to add an additional
screen to let the user pick, so we're not adding support for that now,
but this at least migrates this to use the data set of all emails that
have been verified as associated with the user's GitHub account (and
we just assume the user wants their primary email).

This also fixes the inability for very old GitHub accounts (where the
`email` field in the details might be a string the user wanted on
their GitHub profile page) to using GitHub auth to login.

Fixes #9127.
2018-07-03 18:35:29 +02:00
Anders Kaseorg d8ba378050 test_slack_importer: Remove backslashes wrongly copied from JSON data
https://github.com/houstondatavis/slack-export/blob/master/users.json

JSON or JavaScript decodes "\/" to / (and some encoders always write
"\/" to avoid accidentally creating a </script> tag), while Python
assumes "\/" is a typo for "\\/" and decodes it to \/.

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2018-07-03 16:54:46 +02:00
Anders Kaseorg 037f696d26 Enable pycodestyle W605 (invalid escape sequence).
The only changes visible at the AST level, checked using
https://github.com/asottile/astpretty, are

zerver/lib/test_fixtures.py:
'\x1b\\[(1|0)m' ↦ '\\x1b\\[(1|0)m'
'\\[[X| ]\\] (\\d+_.+)\n' ↦ '\\[[X| ]\\] (\\d+_.+)\\n'

which is fine because re treats '\\x1b' and '\\n' the same way as
'\x1b' and '\n'.

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2018-07-03 16:54:46 +02:00
Rhea Parekh fdef1d8d91 zcommand: Raise error when command doesn't begin with a slash. 2018-07-01 20:32:36 -04:00
Yago González d151f4a234 messages: Allow fetching unedited messages' history.
When GETting an unedited message's edit history, the server wasn't able
to reply properly and produced a 500 error.

Now when that happens, we return a message history that only contains
the original message.
2018-07-01 08:16:59 -07:00
Rhea Parekh 6b7b6b38ad slack import: Write messages batch-wise.
Messages can be bulky, and storing them in a single
data structure can cause a memory error.

In this commit, the messages are written to a file
batch-wise, thus avoiding the memory error.
2018-07-01 07:08:13 -07:00
Rhea Parekh 7f6c174099 slack import: Add 'id_list' field in channel_message_to_zerver_message.
The id_list would help to store the associated max ID
state between subsequent calls, which will help
in batch-wise processing of the messages.
2018-07-01 07:08:13 -07:00
Rhea Parekh af20ef4789 slack import: Save messages within convert_slack_workspace_messages.
Previously, the messages where being stored in a output file from
outside the function 'convert_slack_workspace_messages', but
now we store it from the inside the mentioned function.
This will help in processing and saving the messages batch-wise
so as to avoid a memory error.

Reactions are returned separately from 'convert_slack_workspace_messages'
rather than 'message_json'.

Also updated test for 'convert_slack_workspace_messages' and an additional
test for reactions is added.
2018-07-01 07:08:13 -07:00
Rhea Parekh 2f88ca7446 slack import: Import skype and phone data of users. 2018-07-01 07:05:40 -07:00
Eeshan Garg e1e4d523aa Replace remaining user-facing occurences of the term "invite-only".
Fixes #9611.
2018-06-27 12:34:29 -07:00
Steve Howell a2a695dfa7 test flake: Prevent flaky test with less common id.
This fixes a test flake introduced here:

    317a2fff2a

We need a higher bogus bot owner id to prevent
flakes where our userid sequence gets to 100.  (Tests
aren't completely deterministic in what data you
use, since sequences don't get rolled back when
you roll back transactions.)
2018-06-26 16:56:24 -04:00
Rishi Gupta 2de9029a69 stream settings: Add tests for weekly stream traffic calculation. 2018-06-23 12:48:15 -07:00
Yashashvi Dave 317a2fff2a bots settings: Improve API for editing bot owners to refer to users by ID.
Fixes #9502
2018-06-23 12:47:50 -07:00
Vishnu Ks 1b179ca530 signup: Prevent users from signing up with email containing +. 2018-06-23 12:03:30 -07:00
Vishnu Ks a88e44e254 registration: Use tokenized noreply address in email change. 2018-06-23 12:03:30 -07:00
Vishnu Ks e34fcf982f registration: Use tokenized noreply address in user invite. 2018-06-23 12:03:30 -07:00
Vishnu Ks 994e1a2154 registration: Use tokenized noreply address in password reset. 2018-06-23 12:03:30 -07:00
Vishnu Ks 74351fb250 send_email: Add function to generate tokenized noreply in FromAddress. 2018-06-23 12:03:30 -07:00
Raymond Akornor af5de189d8 tests_users: Use assert_json_success instead of assertTrue. 2018-06-22 09:06:43 -04:00
Marco Burstein 6dad6f3dae emoji: Add some edge-casey ordering tests for emoji.
Add 3 new Markdown emoji tests for newlines, emphasis, and links. The
goal of these tests is to ensure that Markdown operations concerning
emoji are preformed in proper order, with emoji being added correctly
based on other Markdown operations.

See suggestion here: https://git.io/flF5W.
2018-06-22 08:59:08 -04:00
Rhea Parekh d066c11b47 zcommand: Strip the slash in the backend.
The slash in command is stripped in the backend,
rather than in the client to make the client code
cleaner.
This would make client code cleaner in the slash
commands which include parameters.
2018-06-20 16:49:29 -04:00
Rhea Parekh f4bbd4f1a3 tests: Use Set instead of List in import tests. 2018-06-20 10:22:34 -07:00
Marco Burstein 02489e5fb8 emoji: Fix a bug where emoticon translation is not done after a newline.
This bug is caused by the conversion of newlines to `<br>` statements,
since `>` is not allowed as a character around an emoticon during
translation.

Also, add a new test case for preventing this bug from occurring in the
future.

Fix #9763.
2018-06-20 08:09:18 -07:00
Rishi Gupta 4f8c44a019 notifications: Simplify notification messages for new subscriptions.
We're adding more stream types, e.g. splitting private streams into
with/without shared history, adding publicly-archived streams, adding
announce-only streams, etc. So maintaining this text is going to get more
complicated over time.

Also, the right place to explain this stuff is in the stream header, or near
the z-in-a-circle.

This commit also adds translation tags to the messages.
2018-06-20 07:53:10 -04:00
Vishnu Ks 403f254557 signup: Create get_accounts_for_email function. 2018-06-19 11:25:23 -07:00
Shubham Dhama 8e032376f9 guest: Restrict guest access to user group creation and updation. 2018-06-19 11:04:14 -07:00
Rhea Parekh 200c381c91 tests: Add tests for importing files from S3 backend. 2018-06-19 10:50:11 -07:00
neiljp (Neil Pilgrim) 88a70c5f5d mypy: test_embedded_bot_system.py: add asserts & remove from mypy.ini. 2018-06-19 10:48:38 -07:00
neiljp (Neil Pilgrim) b5aa705137 mypy: test_link_embed.py: add assert & remove from mypy.ini. 2018-06-19 10:48:38 -07:00
Rhea Parekh a2a74d9271 export: The records.json IDs should be integer.
In records the IDs like the realm_id and user_profile_id
of 'records.json' should be integers. This was missing in the
S3 backend and this commit fixes that.

Added tests for this as well.
2018-06-18 23:06:09 +05:30
Rhea Parekh b2e971b9b1 tests: Add tests for the export file's records.
For the emojis, In 'records.json', the record should contain
the attribute 'file_name', which was missing in the S3 backend.
This commit adds this attribute, as well as tests for the
records of uploads, avatars and emojis in both local and S3 backend.
2018-06-18 09:19:24 -07:00
Rhea Parekh 5cbad76445 tests: Add tests for importing files from local. 2018-06-18 09:19:24 -07:00
Rhea Parekh edb506a605 tests: Add a function to setup export of files.
Add the function '_setup_export_files' to setup the files to
be exported.
2018-06-18 09:19:24 -07:00
Rhea Parekh f735fae777 tests: Add tests for a basic import logic. 2018-06-18 09:19:24 -07:00
Rhea Parekh 0e8f0e2393 tests: Rename 'message.json' to 'messages-000001.json' in export tests.
The export script produces messages in the format 'messages-%06d.json'.
Hence we rename it to 'messages-000001.json'.
2018-06-18 09:19:24 -07:00
Yashashvi Dave 8909cb1d15 custom fields: Allow list of users in user type of custom fields.
Allow user to add more than one user-value in user type of custom
fields.

Tweaked by tabbott to improve the models.py code and type annotations.
2018-06-16 09:37:49 -07:00
Vishnu Ks 4da61e2a17 signup: Copy enter_sends value when importing settings. 2018-06-16 08:50:49 -07:00
Vishnu Ks b639778d0f signup: Copy hotspots when importing settings. 2018-06-16 08:50:49 -07:00
Rhea Parekh d0bc8d0736 zcommands: Add zcommand module and separate test module.
Move the zcommands from '/views/messages.py' to
'/lib/zcommand'.

Also, move the zcommand tests from '/tests/test_messages.py'
to '/tests/test_zcommand'.
2018-06-16 10:32:42 -04:00
Shubham Dhama b0bfb6b153 streams: Hide create stream UI from guest users. 2018-06-16 06:33:14 -07:00
Shubham Dhama d40c2bb166 emoji: Make uploading new realm emoji inaccessible for guest users. 2018-06-16 06:33:14 -07:00
Shubham Dhama 596ce5b60f invite: Make inviting new users inaccessible for guest users. 2018-06-16 06:33:13 -07:00
Roman Godov 0b9d4fbb06 auth: Added time conversion in new login emails.
This commit adds code for date and time conversion in
"new login" emails according to user's timezone.

Fixes #9336.
2018-06-07 08:40:22 -07:00
Yashashvi Dave f7f039e772 custom fields: Add endpoint to delete value of custom profile data.
Add delete endpoint at `users/me/profile_data` to delete value of
custom profile data.

Fixes #9669
2018-06-07 08:35:04 -07:00
Tim Abbott 4481c74a71 GitHubAuthBackendTest: Remove token_data_dict argument.
This was always the same, and there's not much reason to customize it.
2018-06-06 15:22:52 -07:00
Vishnu Ks 53237d39aa registration: Allow users to import profile picture. 2018-06-06 13:29:42 -07:00
Tim Abbott 7d250cb2f9 attachment: Fix handling of stream history.
This fixes two issues:

* Our guest users feature gave guest users access to public stream
  attachments even if they couldn't access the public stream.

* After a user joins a private stream with our new shared history
  feature, they couldn't see images uploaded before they joined.

The tests need to check for a few types of issues:
* The actual access control permissions.
* How many database queries are used in the various
  cases for that second model, especially with multiple messages
  referencing an attachment.  This function gets called a lot, and we
  want to keep it fast.

Fixes #9372.
2018-06-06 09:40:22 -07:00
Tim Abbott 35c4a9f1d2 auth: Rewrite our social auth integration to use pipeline.
This new implementation model is a lot cleaner and should extend
better to the non-oauth backend supported by python-social-auth (since
we're not relying on monkey-patching `do_auth` in the OAuth backend
base class).
2018-06-05 23:24:48 -07:00
Yashashvi Dave 5145b24635 users: Replace duplication with generic func to validate user id.
This adds a common function `access_user_by_id` to access user id
within same realm, complete with a full suite of unit tests.

Tweaked by tabbott to make the test much more readable.
2018-06-05 11:13:13 -07:00
Tim Abbott 238ba2d486 auth: Move some simple GitHub tests to the long-term test class. 2018-06-04 23:57:09 -07:00
Tim Abbott 869d75b02f AuthBackendTest: Fix typos in error message checks.
Previously, these checks did nothing.
2018-06-04 23:45:13 -07:00
Shubham Dhama 01555e8772 streams: Handle guest user ids for stream settings changes' events. 2018-06-04 11:35:37 -07:00
Tim Abbott ec90322e5d bots: Fix behavior for bots mentioned in PMs.
We've for a long time had the behavior that a bot mentioned in a
stream message receives the notification, regardless of whether the
bot was actually subscribed to the stream.

Apparently, this behavior also triggered if you mentioned a bot in a
private message (i.e. the bot would be delievered the private message
and would probably respond unhelpfully in a new group private message
thread with the PMs original recipients plus the bot).

The fix for this bug is simple: To exclude this feature for private
messages.
2018-06-03 22:29:15 -07:00
Rishi Gupta 182215d125 portico: Update text of why-zulip.
With edits from tabbott.
2018-06-03 09:37:00 -07:00
Shubham Dhama 4483e33102 digest: Make newly registered users data inaccessible to guest users.
The new can_access_all_realm_members function is meant to act as a
base function for guest users and Zephyr realm users regarding the
accessibility of the information of other users in the realm.
2018-06-03 09:30:59 -07:00
Shubham Padia 8883107a7f streams: Allow cross-realm bots to post to announcement_only streams.
This fixes an issue where if you make #announce (the default
announcement stream) announce-only, then creating a new stream will
throw an exception (because notification-bot can't send there).

Fixes #9636.
2018-06-02 10:36:29 -07:00
Steve Howell 47b4dd6bdb slash commands: Refine /day and /night.
These two slash commands now use zcommand to talk to
the server, so we have no Message overhead, and if you're
on a stream, you no longer spam people by accident.

The commands now also give reasonable messages
if you are already in the mode you ask for.

It should be noted that by moving these commands out of
widget.py, they are no longer behind the ALLOW_SUB_MESSAGES
setting guard.
2018-06-02 09:40:12 -07:00
Steve Howell 4b2e8b83c4 slash commands: Add /ping command (via zcommand).
This adds a /ping command that will be useful for users
to see what the round trip to the Zulip server is (including
only a tiny bit of actual server time to basically give a
200).

It also introduce the "/zcommand" endpoint and zcommand.js
module.
2018-06-02 09:40:12 -07:00
Rhea Parekh f8e53511ea slack import: Add support for bold-italics formatting.
Fixes #8927
2018-06-02 09:01:55 -07:00
Shubham Dhama 8e2337509d streams: Hide public streams & subscribers from guest users.
The main change here is making can_access_public_streams
2018-06-02 08:39:39 -07:00
Shubham Dhama ca02143074 test_subs: Fix typos in test_previously_subscribed_private_streams. 2018-06-02 08:06:45 -07:00
Yago González 9f98002b76 tests: Allow testing our REST API against the OpenAPI docs. 2018-06-01 12:03:31 -07:00
Tim Abbott 0a0ae5e703 users: Move add_service to zerver.lib.users so it is reusable. 2018-06-01 08:48:17 -07:00
Sampriti Panda 46711a43f0 pgroonga: Upgrade to PGroonga 2 API. 2018-05-31 13:00:34 -07:00
Shubham Padia cd1233d3f7 upload: Add test jpg image with no exif data. 2018-05-31 08:08:59 -07:00
Tim Abbott 5967de9177 test_auth_backends: Move GitHub signup tests to new suite.
This eliminates a lot of duplicated, mocking-heavy code.
2018-05-30 18:24:27 -07:00
Tim Abbott 7d8d6f1978 test_auth_backends: Move GitHub deactivated test to new suite. 2018-05-30 18:24:27 -07:00
Tim Abbott 29492325a8 test_auth_backends: Move github_auth_enabled test to new suite.
This is step 1 of a migration to eventually delete the legacy test suite.
2018-05-30 18:24:27 -07:00
Steve Howell 554883e039 submessages: Add submessage_id to submessage event.
It's useful to have the submessage_id so that widgets
can in theory sort all the events (and not have to rely
on them arriving in sequence).
2018-05-30 14:47:36 -07:00
Shubham Padia bb8ad15fa9 streams: Set is_announcement_only when creating streams.
Adds is_announcement_only to create_stream_if_needed and
add_subscriptions_backend.
2018-05-30 14:17:33 -07:00
Steve Howell bfc13ecea4 submessages: Just send content for submessage events.
For some reason in my original version I was sending both
content and data to the client for submessage events,
where data === JSON.parse(content).  There's no reason
to not just let the client parse it, since the client
already does it for data that comes on the original
message, and since we might eventually have non-JSON
payloads.

The server still continues to validate that the payload
is JSON, and the client will blueslip if the server
regressses and sends bad JSON for some reason.
2018-05-30 13:53:11 -07:00
Rhea Parekh e2f2ea7248 test_export: Add tests for 'do_export_user'. 2018-05-30 10:28:10 -07:00
Rhea Parekh 74cc07bb2f test_export: Add tests for export of files.
Thanks @Zachary-Jackson for some preliminary work
on this.
2018-05-30 10:28:10 -07:00
Tim Abbott 99c6a82b7f i18n: Clean up logic for picking language in app.
We now have a simple algorithm: First, look at the URL path
(e.g. /de/, which is intended to be an override).  Second, look at the
language the user has specified in their settings.
2018-05-30 08:57:26 -07:00
Shubham Dhama 1d9afc7328 i18n: Pass translation data in page_params.
This is a preliminary step to make translation_data available as a
resource to `i18next.init()`.
2018-05-30 08:57:26 -07:00
Robert Hönig 647c63050f botserver: Add outgoing webhook tokens to botserverrc.
The tokens will be used to authorize the server when sending
messages to the Botserver.
2018-05-30 10:00:19 -04:00
Yashashvi Dave 47aaf4e20a users: Replace duplication with generic func to validate bot id.
This adds a common function `access_bot_by_id` to access bot id within
same realm.  It probably fixes some corner case bugs where we weren't
checking for deactivated bots when regenerating API keys.
2018-05-29 15:47:27 -07:00
Shubham Padia 8b8a9be377 upload: Rotate image according to exif data when resizing avatar and emojis.
Fixes the avatar/emoji part of #8177.

Does not address the issue with uploaded images, since we don't do
anything with them.

Also adds 3 images with different orientation exif tags to
test-images.
2018-05-29 10:39:39 -07:00
Tim Abbott 91ec0aba09 auth: Improve interactions between LDAPAuthBackend and EmailAuthBackend.
Previously, if you had LDAPAuthBackend enabled, we basically blocked
any other auth backends from working at all, by requiring the user's
login flow include verifying the user's LDAP password.

We still want to enforce that in the case that the account email
matches LDAP_APPEND_DOMAIN, but there's a reasonable corner case:
Having effectively guest users from outside the LDAP domain.

We don't want to allow creating a Zulip-level password for a user
inside the LDAP domain, so we still verify the LDAP password in that
flow, but if the email is allowed to register (due to invite or
whatever) but is outside the LDAP domain for the organization, we
allow it to create an account and set a password.

For the moment, this solution only covers EmailAuthBackend.  It's
likely that just extending the list of other backends we check for in
the new conditional on `email_auth_backend` would be correct, but we
haven't done any testing for those cases, and with auth code paths,
it's better to disallow than allow untested code paths.

Fixes #9422.
2018-05-28 22:47:47 -07:00
Tim Abbott 3842404cc0 ldap: Don't allow password reset for users in LDAP domain.
This is the analog of the last commit, for the password reset flow.
For these users, they should be managing/changing their password in
the LDAP server.

The error message for users doing the wrong thing here is nonexistent
isn't great, but it should be a rare situation.
2018-05-28 22:47:47 -07:00
Tim Abbott 8119670da1 user_settings: Prevent LDAP users from setting a Zulip password.
Previously, if both EmailAuthBackend and LDAPAuthBackend were enabled,
LDAP users could set a password using EmailAuthBackend and continue to
use that password, even if their LDAP account was later deactivated.

That configuration wasn't supported at all before, so this doesn't fix
a pre-existing security issue, but now that we're making that a valid
configuration, we need to cover this case.
2018-05-28 22:47:47 -07:00
Sampriti Panda 250015a5d5 pgroonga: Fix issues with HTML escaping in queries. 2018-05-28 16:53:30 -07:00
Shubham Dhama b410aeb897 message: Limit message access for guest users.
Tweaked by tabbott to interact correctly with the
history_public_to_subscribers feature.
2018-05-28 12:20:32 -07:00
Robert Hönig 497cd4a1f9 outgoing webhooks: Change default URL in all tests.
This reflects the changes to the default URL publicly
displayed to the user. It also changes the default
URL of the default test server outgoing webhook, which
prevented the test server flaskbotrc from working out
of the box.
2018-05-28 10:40:44 -07:00
Yashashvi Dave e82c879b85 custom fields: Add user type of custom fields.
Fixes #8878
2018-05-27 23:01:21 -07:00
Yashashvi Dave 1e948ab405 zerver/lib/users.py: Add function to check given user id is valid.
This function check if given user id exists in realm.
It also check, if user is active and not a bot.
2018-05-27 22:57:55 -07:00
Vishnu Ks c90aac56ba registration: Copy user full_name when importing settings. 2018-05-27 22:39:21 -07:00
Rhea Parekh 468afe4840 export: Support export of Custom emojis.
Export of RealmEmoji should also include the image
file of those emojis.

Here, we export emojis both for local and S3 backend
in a method with is similar to attachments and avatars.

Added tests for the same.
2018-05-27 21:54:20 -07:00
Robert Hönig 7a8c1ec9dc outgoing webhooks: Send additional useful data.
This adds the fields `trigger` and `service_email`
to each message event dispatched by outgoing webhook bots.
`trigger` will be used by the Botserver to determine if
a bot is mentioned in the message.
`service_email` will be used by the Botserver to determine
by which outgoing webhook bot the message should be handled.
2018-05-25 10:33:40 -07:00
Eeshan Garg 0a1b429d04 settings: Remove PRIVATE_STREAM_HISTORY_FOR_SUBSCRIBERS.
Now that we have a UI for this feature, we don't need this setting
anymore.
2018-05-24 17:18:14 -07:00
Tim Abbott 6967b6519d settings: Add a development-only setting for less dense mode.
This should make it easier for us to iterate on a less-dense Zulip.

We create two classes on body, less_dense_mode and more_dense_mode, so
that it's easy as we refactor to separate the two concepts from things
like colors that are independent.
2018-05-24 12:31:37 -07:00
Vishnu Ks 54a002c2e2 requirements: Upgrade pyflakes to 2.0.0.
We fix a few errors that only the new version finds.
2018-05-24 11:31:36 -07:00
Steve Howell 69517f5ac5 Support zform-based widget content in the server.
API users, particularly bots, can now send a field
called "widget_content" that will be turned into
a submessage for the web app to look at.  (Other
clients can still rely on "content" to be there,
although it's up to the bot author to make the
experience good for those clients as well.)

Right now widget_content will be a JSON string that
encodes a "zform" widget with "choices."  Our first
example will be a trivia bot, where users will see
something like this:

    Which fruit is orange in color?

        [A] orange
        [B] blackberry
        [C] strawberry

The letters will be turned into buttons on the webapp
and have canned replies.

This commit has a few parts:
    - receive widget_content in the request (simply
        validating that it's a string)
    - parse the JSON in check_message and deeply
        validate its structure
    - turn it into a submessage in widget.py
2018-05-24 09:30:22 -07:00
Shubham Dhama fc3dcbfb5a messages: Allow "no topic" topics editable indefinitely.
Fixes: #9484.
2018-05-24 07:35:52 -07:00
Umair Khan adaeaccd20 2FA: Add link in settings to setup two-factor. 2018-05-23 15:49:41 -07:00
Umair Khan e063bb0d50 2FA: Add tests for two-factor auth. 2018-05-23 15:49:35 -07:00
Umair Khan bf740f9232 2FA: Add zulip_otp_required decorator.
We need to add this because otp_required doesn't play well with tests.
2018-05-23 15:46:57 -07:00
Tim Abbott 9c29592178 mypy: Fix rebase errors in create_user.py. 2018-05-23 11:36:28 -07:00
Vishnu Ks 8369e2b15e registration: Allow users to import settings from other realm.
This should significantly improve the user experience for creating
additional accounts on zulipchat.com.

Currently, disabled in production pending some work on visual styling.
2018-05-23 10:30:00 -07:00
Vishnu Ks 2a66818e2e registration: Add function to copy user settings.
This is intended to support our upcoming feature to support copying a
user's customization settings from an existing account that user owns
in another organization.
2018-05-23 10:12:14 -07:00
Aditya Bansal 8cfb437677 provision: Don't run create_realm_internal_bots management command.
We essentially stop running create_realm_internal_bots during
every provisioing and move its operations to run from populate db.
In fact to speed things up a bit we actually make populate db call the
funcs which create_realm_internal_bots calls behind the scenes.

Fixes: #9467.
2018-05-23 13:34:52 +05:30
Aditya Bansal d343f25cc6 create_realm_internal_bots: Refactor to extract main op as a function.
We extract the entire operations of the management command to a
function create_if_missing_realm_internal_bots in the
zerver/lib/onboarding.py. The logic for determining if there are any realm
internal bots which have not been created is extracted to a function
missing_any_realm_internal_bots in actions.py.
2018-05-23 11:53:22 +05:30
Tim Abbott 998e1048e8 test_bots: Add coverage for no-op changes to bot owner.
This isn't a complete long-term fix, in that ideally we'd be doing
this check at the view layer, but various structural things make that
annoying, and we'll want this test either way.
2018-05-22 15:04:27 -07:00
Umair Khan f38d6ac6fe ldap: Make Zulip compatible with django-auth-ldap==1.5.
In version 1.5, get_or_create_user method is not used. It exists just
for the compatibility. The main function to use now is
get_or_build_user.

See the changelog:
https://django-auth-ldap.readthedocs.io/en/latest/changes.html#id1

Fixes #9307
2018-05-22 08:13:41 -07:00
Michael J. Sullivan d5ee801d60 mypy: Set local_partial_types = True.
The daemon implicitly sets this flag, so set it explicitly and fix all
of the type errors.
2018-05-21 22:41:00 -07:00
Tim Abbott cec7686f3d push notifications: Clean up unregistered/bad APNS tokens.
We've had this sort of logic for GCM for a long time; it's worth
adding for APNS as well.

Writing this is a bit of a reminder that I'm not a fan of how our unit
tests for push notifications work.
2018-05-21 11:30:56 -07:00
Tim Abbott c6909fb99c migration_tests: Document the migration test framework. 2018-05-21 09:59:55 -07:00
Aditya Bansal b489eded17 migration_test: Add test for 0145 migration. 2018-05-21 09:46:52 -07:00
Aditya Bansal e41d32a15a reminders: Add various design improvements for the UI/reminder msg. 2018-05-21 09:07:37 -07:00
Tim Abbott 43ec7ed456 narrow: Add is_web_public_compatible check for narrows.
This will be used to limit which narrows will be allowed for
web-public browsing of Zulip.
2018-05-21 08:45:48 -07:00
Umair Khan 18defcc673 2FA: Fix check for context_data attr.
This attribute only makes sense for responses that are generated using a
template.
2018-05-21 07:50:05 -07:00
Tim Abbott 7dd275328b github: Add a complete end-to-end GitHub OAuth2 test.
This revised GitHub auth backend test is inspired by the end-to-end
flow model of the Google auth backend test.  My hope is that we will
be able to migrate the rest of the important cases in the GitHub auth
backend tests to this model and then delete what is now
GitHubAuthBackendLegacyTest.

The next step after that will be to merge the GitHub and Google auth
tests (since actually, the actual test functions are basically
identical between the two).
2018-05-20 23:44:20 -07:00
Tim Abbott 86ec78eaa9 test_auth_backends: Move ResponseMock earlier in the file.
We're going to be using this in the GitHub auth backend as well.
2018-05-20 22:02:43 -07:00
Tim Abbott 9b8331cf88 test_presence: Fix nondeterministic test failures.
Apparently, the bug here was that we were aliasing the user_profile
variable, so that the results depended on what the last iteration in
the loop landed on.
2018-05-20 22:02:43 -07:00
Tim Abbott 336ad0fbb1 password reset: Handle deactivated users and realms.
Since this is a logged-out view, need to actually write code for the
case of deactivated realms.

The change to get_active_user is more for clarity; the Django password
reset form already checks for whether the user is active earlier.
2018-05-20 20:02:27 -07:00
Tim Abbott e7fa77f120 validate_email_for_realm: Clarify errors for deactivated users.
If a user's account has been deactivated, we want to provide a special
error message that makes clear what's going on.

Future work is to provide some administrative controls on whether a
user should be able to re-activate their account.
2018-05-20 19:32:53 -07:00
Tim Abbott 2609274719 streams: Don't allow adding deactivated users to streams.
This query was incorreclty not checking whether a user was deactivated
before managing their subscriptions.

This isn't an important bug, but should prevent some weird corner
cases (like trying to send a notification PM to a deactivated user,
which fails).
2018-05-20 19:17:15 -07:00
Tim Abbott 451b12d0b2 test_home: Fix broken narrow parsing exception test. 2018-05-20 19:06:14 -07:00
Vishnu Ks 61e124a2ab models: Create get_source_profile function.
This will be used in our upcoming feature to import settings from
other Zulip realms.
2018-05-20 15:30:30 -07:00
Vishnu Ks 74e823f5fa models: Add function for getting membership realms. 2018-05-20 15:17:19 -07:00
Sampriti Panda d94202c662 notifications: Add consistent order for sending email notifications.
Notification emails are sent in increasing order of the max message-id
of the unread message thread.
2018-05-20 10:19:13 -07:00
Sampriti Panda 407aa99704 test_notifications: Fix confusing variable naming in test_unread_anchor. 2018-05-20 10:16:53 -07:00
Sampriti Panda 3f4200db3c tests: Disable slow query messages in test environment.
Slow queries during backend tests sends messages to Error Bot
which affects the database state causing the tests to fail.
This fixes the occasional flakes due to that.
2018-05-20 10:16:53 -07:00
Yashashvi Dave 38db31779b /json/bots: Return "No user" if access other realms bot.
Instead of returning warning "Insufficient Permission", return
"No such bot" warning if user tries to access other realms bot."
2018-05-18 15:20:43 -07:00
Yashashvi Dave 4162e61f33 /json/users: Replace email with user_id in API to reactivate user. 2018-05-18 15:20:43 -07:00
Yashashvi Dave 06e7e933cc /json/users: Replace email with user_id in API to update/remove users. 2018-05-18 15:20:43 -07:00
Tim Abbott b8e2339a65 test_presence: Update debugging print statements. 2018-05-18 15:20:15 -07:00
jkiely ca3ce90496 mypy: Enable strict optional in lib/avatar.py.
Add assert to function and modify tests in order to pass under
strict conditions.
2018-05-17 11:11:55 -07:00
Tim Abbott d782927b4c test_webhooks_common: Add logging for a test flake case.
This seems to happen occasionally in CI, and we don't know why.
2018-05-17 09:18:31 -07:00
Vishnu Ks 372e9740ac events: Add date_joined to user_dict. 2018-05-17 07:49:35 -07:00
Tim Abbott fc32fe16c6 test_presence: Add debugging output to help track down a flake.
This test has been failing nondeterministically, and the debug output
should help us figure out why.
2018-05-17 07:44:46 -07:00
Steve Howell de47eeb6f1 Add /submessage endpoint. 2018-05-16 15:13:33 -07:00
Steve Howell 4332fd64f7 Add submessages to message payloads. 2018-05-16 15:13:33 -07:00
Steve Howell ff097623fa Add SubMessage table. 2018-05-16 15:13:33 -07:00
Lyla Fischer 98c50f7041 user docs: Update time-format docs. 2018-05-16 16:17:39 -04:00
Tim Abbott 495104bd56 mypy: Clean test_subs.py of mypy strict-optional errors.
Most of this is just asserting that the sub_dict return value from
access_stream_by_id is not None in the cases where it shouldn't be,
but additionally, we also need to pass a function into
validate_user_access_to_subscribers_helper (in this case, just `lambda:
True` works fine)
2018-05-16 12:09:52 -07:00
Tim Abbott ee0f1992bf lint: Fix lint errors in recently merged tests. 2018-05-15 20:07:33 -07:00
Baron Chandler 1e4f54560a actions: Ensure validate_user_access_to_subscribers_helper raises on invalid params. 2018-05-15 19:14:43 -07:00
Baron Chandler 024e6fc704 actions: Make sure get_recipient_info asserts on invalid recipient type. 2018-05-15 19:14:42 -07:00
Baron Chandler 7251e978d5 actions: Make sure get_typing_user_profiles raises on invalid recipient types. 2018-05-15 19:14:42 -07:00
Tim Abbott aec57baef2 test_presence: Add tests for various analytics pages.
While maybe these don't all belong in this test file, the overall
effect is that we now have quite good test coverage on
analytics/views.py.

It'd be nice to add some more assert statements for specific values
being present in the pages, but since we're not really working on that
part of the product, it's not a priority yet.
2018-05-15 18:52:18 -07:00
Tim Abbott 00861b910d test_presence: Add tests for seconds_usage_between.
This isn't widely used, but is a nice way to verify the logic further.
2018-05-15 18:37:49 -07:00
Tim Abbott 930cbabbb7 test_queue: Add a test for register_json_consumer.
This expands the coverage of the SimpleQueueProcessor logic to be
mostly complete.  We still don't have real tests for
TornadoQueueProcessor.
2018-05-15 18:24:45 -07:00
Tim Abbott 962c64a1d4 test_queue: Add basic tests for SimpleQueueClient. 2018-05-15 17:56:30 -07:00
Yashashvi Dave b949d10592 /json/bots: Replace email with user_id in API to generate bot_api_key.
Fixes #3643.
2018-05-15 16:37:06 -07:00
Yashashvi Dave d6e2f9fc88 /json/bots: Replace email with user_id in API to update bots. 2018-05-15 16:34:17 -07:00
Yashashvi Dave 8909836815 zerver/tests/test_custom_profile_data.py: Refactor tests.
Refactor custom fields creation and deletion tests to assert
if created/deleted field exist or not, instead of asserting
total count of all realm fields.
2018-05-15 12:38:18 -07:00
Shubham Padia 3703013b4f models: Add `is_announcement_only` to Stream.to_dict(). 2018-05-15 12:05:52 -07:00
Baron Chandler f59adfa67c actions: Change do_change_is_admin Exception to an Assertion Error.
- do_change_is_admin now raises AssertionError when a non-admin
  permission is given.
- adds test to test_users to ensure admin asserts on invalid
  permission values.
2018-05-15 11:27:01 -07:00
RobbieClarken 17d6d4fed1 zerver/tests: Add test for get_emoji_url for local upload backend. 2018-05-15 10:50:39 -07:00
RobbieClarken fb42b58316 zerver/tests: Test emoji upload for local storage backend. 2018-05-15 10:50:39 -07:00
RobbieClarken 2dc13d6b50 zerver/tests: Test ensure_medium_avatar_url when the file exists. 2018-05-15 10:50:39 -07:00
RobbieClarken 398141268f zerver/tests: Test get_emoji_url for S3. 2018-05-15 10:50:39 -07:00
RobbieClarken dd19ed3972 zerver/tests: Test upload emoji to S3. 2018-05-15 10:50:39 -07:00
RobbieClarken ff95c67197 zerver/tests: Test upload realm icon for S3. 2018-05-15 10:50:39 -07:00
RobbieClarken 25e91b289d zerver/tests: Test delete message image from S3 when file is missing. 2018-05-15 10:50:39 -07:00
RobbieClarken b3a6c0a253 zerver/tests: Test currently_used_upload_space function. 2018-05-15 10:50:38 -07:00
RobbieClarken efc63d4ca0 zerver/tests: Test S3 file upload with undefined content type. 2018-05-15 10:42:10 -07:00
RobbieClarken b9fff04f39 zerver/tests: Add test for resize_emoji for images that need resizing. 2018-05-15 10:42:10 -07:00
RobbieClarken 69a5ce102f zerver/tests: Remove unecessary re-open on files in tests.
The get_test_image_file helper function returns a file handle so we can
call the read method on this directly.
2018-05-15 10:42:10 -07:00
RobbieClarken f34d9dc910 zerver/tests: Test get_realm_for_filename when filename doesn't exist. 2018-05-15 10:42:10 -07:00
Baron Chandler 52d53f4a81 test_subs: Ensure gather_subscriptions_helper excludes deleted streams. 2018-05-15 10:11:32 -07:00
Tim Abbott a2ed06314d send_test_email: Throw an error if email not configured.
This should help minimize confusion when folks try to use this before
properly configuring outgoing email.

Thanks to Bruce Eckel for the report.
2018-05-14 12:02:36 -07:00
RobbieClarken 2cca5dc79f zerver/tests: Test upload when mimetype is supplied as a query param. 2018-05-14 10:58:24 -07:00
Joshua Pan 231b487bca tests: Cover check_config() in zerver/lib/management.py.
Add a comment to code to clarify what the config check does.
2018-05-14 10:46:21 -07:00
Darshan Markandaiah 607cab2a53 tests: Add 100% test coverage to zerver/apps.py.
Tweaked by tabbott to rename to test_cache.py and remove the
sender_name argument (Since it was kinda confusing).
2018-05-14 10:36:10 -07:00
Eeshan Garg 5c0d4660c1 models: Add history_public_to_subscribers to Stream.to_dict().
This commit also updates all the relevant parts where this attribute
could be useful, e.g. payloads for user subscriptions.
2018-05-13 09:15:07 -07:00
Shubham Padia 295fcb8536 models: Add `is_announcement_only` to stream_dict in `actions.py`.
Adds `is_announcement_only` to `stream_dict`s in order to access
the property in the frontend.
2018-05-13 09:11:51 -07:00
Shubham Padia 897ed17f0c api: Allow realm_admins to make a stream announcement_only. 2018-05-13 09:11:51 -07:00
Shubham Padia bb8577ba94 stream: Only realm admins can post to an announcement_only streams.
If a non-admin tries to post to an announcement_only stream, error
message will be shown.
2018-05-13 09:11:51 -07:00
Vishnu Ks 08d20dce23 docs: Cover the missing case in DocPageTest. 2018-05-13 08:56:04 -07:00
Vishnu Ks 99186952f6 integrations: Make integration_doc endpoint work only on ajax. 2018-05-13 08:56:04 -07:00
Aditya Bansal 5416d137d3 zerver/tests: Change use of typing.Text to str. 2018-05-12 15:22:39 -07:00
Aditya Bansal 2f3b2fbf59 zerver/tests: Change use of typing.Text to str. 2018-05-10 14:19:49 -07:00
Tim Abbott 81e17c7d47 test_i18n: Stop using six.moves for SimpleCookie.
This requires a 'type: ignore' because of a typeshed bug.
2018-05-09 20:45:45 -07:00
Tim Abbott 4c1777f146 tests: Add a good set of unit tests for UserActivityInterval.
This system was written years ago and has been working well the whole
time, but having unit tests for it will help future developers in
understanding what the intent is.
2018-05-09 20:45:02 -07:00
Tim Abbott fba45bb9d3 archive: Add functions for accessing web-public streams.
These will be helpful for variable upcoming projects to support the
web-public streams feature.
2018-05-08 12:12:15 -07:00
Tim Abbott f24630fd4a events: Include data for whether push notifications are enabled.
This is primarily useful for the mobile app, but could also be used to
control whether we display push-notifications related settings to
users in the web UI.
2018-05-08 11:45:13 -07:00
Eeshan Garg 34d1b0ebf1 webhooks: Add helper to extract and validate HTTP event headers.
This is a part of our efforts to close #6213.
2018-05-05 15:48:37 -07:00
Tim Abbott 956bd74905 attachments: Send events for attachment updates.
We send add events on upload, update events when sending a message
referencing it, and delete updates on removal.

This should make it possible to do real-time sync for the attachments
UI.

Based in part on work by Aastha Gupta.
2018-05-04 16:22:27 -07:00
Tim Abbott 69c4645bd2 attachments: Stop fetching attachments in / endpoint.
We only use this data in a rarely-used settings screen, and it can be
large after years of posting screenshots.

So optimize the performance of / by just loading these data when we
actually visit the page.

This saves about 300ms of runtime for loading the home view for my
user account on chat.zulip.org.
2018-05-04 16:22:26 -07:00
Tim Abbott 59d3fefc07 bots: Improve performance of fetching core bot data.
This eliminates the need to call user_ids_to_users inside the
get_service_dicts_for_bots code path, saving a database query.

This completes my refactor to fix backend performance issues in this
code path.  Previously, our messy layering of queries that resulted in
Zulip doing work even if none of the bots actually had Services or
config_data.
2018-05-04 13:44:43 -07:00
Vishnu Ks 733da0ac07 settings: Fix 500 when trying to change email to disposable email.
Fixes #9240
2018-05-04 10:52:38 -07:00
Tim Abbott 508dc5b6ed decorators: Add new decorators for guest users.
These decorators will be part of the process for disabling access to
various features for guest users.

Adding this decorator to the subscribe endpoint breaks the guest users
test we'd just added for the subscribe code path; we address this by
adding a more base-level test on filter_stream_authorization.
2018-05-04 10:25:52 -07:00
Tim Abbott 8b26f912af streams: Limit access to public streams for guest users.
With most of the tests tests written by Shubham Dhama.
2018-05-04 09:47:58 -07:00
Tim Abbott 7cbff8b521 push registration: Use standard error message for auth problems.
This avoids adding an unnecessary new translated string.
2018-05-04 09:04:39 -07:00
Tim Abbott 43098a6f7c zilencer: Add automated signup system for push notifications.
Based on an initial version by Rishi Gupta.

Fixes #7325.
2018-05-03 21:27:49 -07:00
Tim Abbott 51517fa188 request: Add new str_validator validator type.
This is helpful for cases where an argument is supposed to be a normal
string, and we want to use a Zulip validator function to do basic
things like check its length.
2018-05-03 21:11:02 -07:00
Tim Abbott f2e84f25a0 management: Refactor checkconfig code to live in library.
This makes it possible to call this from other management commands.
2018-05-03 21:11:02 -07:00
Tim Abbott ad0cfb3512 validators: Add check_string_fixed_length.
This will be useful in some upcoming work on zilencer.
2018-05-03 14:30:03 -07:00
Tim Abbott 594451707d check_capped_string: Clean up corner case.
We were rejecting strings of length equal to the max.

While we're at it, fix the unnecessary period in the error message,
which doesn't align with similar validators.
2018-05-03 14:30:03 -07:00
Armaan Ahluwalia 8fe54a533a test_home: Change requiring pygments.css to requiring app-styles.js.
This commit changes the tests based on the fact that pygments.css
will no longer be found in the template during testing. pygemnts.css
is being compiled by webpack under app-styles and so we look for the
stubentry for app-styles instead.

Tweaked by tabbott to do a cleaner test.
2018-05-02 17:18:11 -07:00
Aditya Bansal c38b70566c views/archive.py: Make 'StreamDoesNotExist' return a error msg page. 2018-05-02 15:23:33 -07:00
Aditya Bansal 5d7907b59f public_archives: Add basic infra for displaying topics.
We add very basic infra so that we can view any discussion which
happened under a topic of a global public stream without
authorization.
2018-05-02 15:23:33 -07:00
Umair Khan 29e3a1d576 two_factor: Add templates for 2-factor-auth setup.
Note from Tim: We'll likely need to do some work on the strings in
these before translating, so I removed some translation tags.
2018-05-02 15:20:49 -07:00
novokrest 036bc120c3 outgoing_webhook: Extend process_success() return value to tuple.
Change return value type of OutgoingWebhookServiceInterface.process_success
to 2-elements tuple as (success_message, failure_message)
2018-05-02 11:57:08 -07:00
Tim Abbott a0e8a37e7f topics: Fix get_topics_backend logic for zephyr realms.
This removes a check on invite_only, that should have been a check on
history_public_to_subscribers.  In addition to fixing a bug for zephyr
realms, it also makes "more topics" work correctly for realms using
the new settings for stream history being public to subscribers.
2018-05-02 09:02:57 -07:00
Rohitt Vashishtha 2f6da2661f push_notifications: Format blockquotes properly in text_output.
New output is of the format:

Hamlet said:
> Polonius said:
> > This is the.
> > Second layer of nesting.
> First layer of nesting.

Fixes #9251.
2018-05-02 08:57:17 -07:00
Vishnu Ks 8705ac1091 portico: Add noindex tags for non root realms.
There may be further work required for the /integrations pages.
2018-05-01 15:28:30 -07:00
Rohitt Vashishtha 6c96ba79e0 test_bugdown: Fix ignore-testcase feature for markdown_test_cases.
We accidentally were 'return'ing on encountering an ignored case, and thus
exiting the loop, not running further testcases.
2018-05-01 09:24:15 -07:00
Angelika Serwa f4f64243dd custom_profile_fields: Support changing the sort order of the fields.
Tweaked by tabbott for variable naming and the URL.

Closes #8879.
2018-04-30 18:17:41 -07:00
Tim Abbott de30474ddd test_custom_profile_data: Don't do a database query on import.
This is a general code cleanlyness improvement.

While we're at it, we combine together two test classes that didn't
have a particularly good reason for existing.
2018-04-30 10:57:23 -07:00
Yashashvi Dave 0a9fbe2ce6 zerver/tests/test_custom_profile_data.py: Refactor tests.
Refactor tests to include generic function for invalid
assertion and remove repetitive assertion code.
2018-04-30 10:53:23 -07:00
Yashashvi Dave a97a00a4c6 custom fields: Replace field id with field name in error message. 2018-04-30 10:53:23 -07:00
Yashashvi Dave 0d7d94d0db custom fields: Add support for custom URL field type. 2018-04-30 10:53:23 -07:00
Yashashvi Dave 4033f210af custom fields: Add support for custom date field type. 2018-04-30 23:04:25 +05:30
Yashashvi Dave 94d787aa2e zerver/test/test_custom_profile_data.py: Remove hard-coded field no's. 2018-04-30 23:04:25 +05:30
Yashashvi Dave 66759358e2 create stream: Add maxlength restriction on name and description. 2018-04-30 10:11:25 -07:00
Yashashvi Dave 7e9ccead2e stream settings: Fix server error on long stream description.
Add backend validations to check stream description length.
2018-04-30 10:11:23 -07:00
Tim Abbott 976e61d687 validators: Improve error messages for check_capped_string. 2018-04-30 10:07:06 -07:00
Eeshan Garg 057ff9c91e models: Add Stream.history_public_to_subscribers.
This commit adds a new field history_public_to_subscribers to the
Stream model, which serves a similar function to the old
settings.PRIVATE_STREAM_HISTORY_FOR_SUBSCRIBERS; we still use that
setting as the default value for new streams to avoid breaking
backwards-compatibility for those users before we are ready with an
actual UI for users to choose directly.

This also comes with a migration to set the value of the new field for
existing streams with an algorithm matching that used at runtime.

With significant changes by Tim Abbott.

This is an initial part of our efforts on #9232.
2018-04-28 22:54:04 -07:00
Tim Abbott e087be6630 home: Ignore handlebars errors in the test suite.
The handlebars error message is just for the manual development
environment; this prevents the state of compiling handlebars templates
from run-dev.py from potentially causing the unit tests to fail.
2018-04-28 13:49:24 -07:00