Commit Graph

3383 Commits

Author SHA1 Message Date
Steve Howell e2ee455314 outgoing webhooks: Support widget content.
If we use an outgoing webhook and the web server
responds with `widget_content` in the payload, we
include that in what we send through the send-message
codepath.

This makes outgoing webhook bots more consistent with
generic bots.
2018-10-26 12:08:05 -07:00
Rishi Gupta 92d0511c0b portico: Make a few wording edits to the top of /hello. 2018-10-25 18:25:20 -07:00
Steve Howell 4b4f27fffb tests: Fix flaky test by using sets, not lists.
The test named `test_archiving_messages_with_attachment`
started flaking recently.  We use sets for comparison
instead of lists to avoid arbitrary sorting differences.
2018-10-25 13:47:37 -05:00
Steve Howell 551fc7f165 bots: Prevent bots from having duplicate full names.
Bots are not allowed to use the same name as
other users in the realm (either bot or human).

This is kind of a big commit, but I wanted to
combine the post/patch (aka add/edit) checks
into one commit, since it's a change in policy
that affects both codepaths.

A lot of the noise is in tests.  We had good
coverage on the previous code, including some places
like event testing where we were expediently
not bothering to use different names for
different bots in some longer tests.  And then
of course I test some new scenarios that are relevant
with the new policy.

There are two new functions:

    check_bot_name_available:
        very simple Django query

    check_change_bot_full_name:
        this diverges from the 3-line
        check_change_full_name, where the latter
        is still used for the "humans" use case

And then we just call those in appropriate places.

Note that there is still a loophole here
where you can get two bots with the same
name if you reactivate a bot named Fred
that was inactive when the second bot named
Fred was created.  Also, we don't attempt
to fix historical data.  So this commit
shouldn't be considered any kind of lockdown,
it's just meant to help people from
inadvertently creating two bots of the same
name where they don't intend to.  For more
context, we are continuing to allow two
human users in the same realm to have the
same full name, and our code should generally
be tolerant of that possibility.  (A good
example is our new mention syntax, which disambiguates
same-named people using ids.)

It's also worth noting that our web app client
doesn't try to scrub full_name from its payload in
situations where the user has actually only modified other
fields in the "Edit bot" UI.  Starting here
we just handle this on the server, since it's
easy to fix there, and even if we fixed it in the web
app, there's no guarantee that other clients won't be
just as brute force.  It wasn't exactly broken before,
but we'd needlessly write rows to audit tables.

Fixes #10509
2018-10-24 16:59:57 -07:00
Tim Abbott 5cec566cb9 queue_processors: Rewrite MissedMessageWorker to always wait.
Previously, MissedMessageWorker used a batching strategy of just
grabbing all the events from the last 2 minutes, and then sending them
off as emails.  This suffered from the problem that you had a random
time, between 0s and 120s, to edit your message before it would be
sent out via an email.

Additionally, this made the queue had to monitor, because it was
expected to pile up large numbers of events, even if everything was
fine.

We fix this by batching together the events using a timer; the queue
processor itself just tracks the items, and then a timer-handler
process takes care of ensuring that the emails get sent at least 120s
(and at most 130s) after the first triggering message was sent in Zulip.

This introduces a new unpleasant bug, namely that when we restart a
Zulip server, we can now lose some missed_message email events;
further work is required on this point.

Fixes #6839.
2018-10-24 14:43:36 -07:00
Rishi Gupta 458169928c billing: Rename Zulip Premium to Zulip Standard. 2018-10-24 10:42:16 -07:00
Steve Howell 409e2b4134 hipchat import: Support sender_id == 0 use case. 2018-10-23 17:27:37 -05:00
Steve Howell 48b5b2f5d5 hipchat import: Start adding tests.
These test are for the handling of HipChat
sender info.  The data formats are somewhat
inconsistent and sometimes require us to
generate "mirror" users, so this is potentially
fragile code if we don't cover it well.
2018-10-23 17:27:37 -05:00
Steve Howell bd9e4ef0c8 import: Use pub_date to sort message ids.
When we create new ids for message rows, we
now sort the new ids by their corresponding
pub_date values in the rows.

This takes a sizable chunk of memory.

This feature only gets turned on if you
set sort_by_date to True in realm.json.
2018-10-23 17:27:37 -05:00
Rishi Gupta b26e8eca62 portico: Redirect /plans to zulipchat.com/plans for self-hosted realms. 2018-10-23 14:47:00 -07:00
Rishi Gupta fbd884f934 billing: Rename MAX_INVITES_PREMIUM.
Largely to have parallel structure with INVITES_DEFAULT_REALM_DAILY_MAX.
2018-10-23 14:47:00 -07:00
Rishi Gupta a7fe6b1c34 billing: Fix max_invites for PREMIUM_FREE.
We could migrate all the current PREMIUM_FREE organizations to have more
invites, but this setting mainly affects orgs right as they are starting, so
it's probably fine.
2018-10-23 14:47:00 -07:00
Aditya Bansal a16bf34c7f thumbnailing: Fix oversharpening of thumbnails.
We seemed to have been doing too much of sharpening on the thumbnails.
The purpose of sharpening here was to just counter the softening
effects of a resize on an image but overdoing it is bad.

Value sharpen(0.5,0.2,true) seems to look good for achieving the
best results here on different displays as revealed in the manual
hit and trial based testing.

Thanks to @borisyankov for pointing out the issue and suggesting
the values.
2018-10-22 22:28:04 +05:30
Pragati Agrawal b665b0a9a5 users: Avoid O(N) query checking for is_admin.
The main benefit is that we don't do O(N) queries for checking admin
thus leading to less processing of UserProfile.
2018-10-19 15:51:12 -07:00
Tim Abbott 39ea471cf1 error reports: Ensure we filter API keys from query strings.
For some webhook endpoints where the third-party API requires us to do
this, the user's API key might appear in error emails through
appearing in the `QUERY_STRING` parameter.  Fix that by filtering any
actual content from those; what we usually need for debugging is just
what set of parameters were provided.
2018-10-19 15:03:14 -07:00
Yashashvi Dave 41fbb16cdf org settings: Fix error when admin update realm users full names.
Currently, if there is only one admin in realm and admin tries
to updates any non-adminuser's full name it throws error,
"Cannot remove only realm admin". Because in `/json/users/<user_id>`
api check_if_last_admin_is_changed is checked even if property
is_admin is not changed.

This commit fix this issue and add tests for it.
2018-10-19 12:34:20 -07:00
Tim Abbott 695d8d0bd1 get_apns_payload: Require a UserProfile object for the recipient.
This is preparatory work for being able to display an unread count
badge on iOS, in which case we need to know who the current user is.
2018-10-18 15:09:18 -07:00
Tim Abbott 2eebacf2dc push_notifications: Lazily import APNS libraries.
The APNS client libraries (especially the hyper.http20 one) were
determined via profiling to take significant time during the import
process, so we move them to be lazily imported in order to optimize
the overall Zulip import process.  This save up to about 100ms in
import time.

These libraries are only used in certain Django processes inside
zulipchat.com, and so are unnecessary both in development as well as
for self-hosted Zulip servers.
2018-10-17 11:59:33 -07:00
Aditya Bansal 8324e2c976 thumbnails: Return original path if url is not supposed to be thumbnailed. 2018-10-16 16:00:47 -07:00
Aditya Bansal 9a411a5765 thumbnails: Stop thumbnailing urls other than external or user_uploads.
We are basically adding a check for url's to be external (belonging
to some 3rd party web site hosting the image) or be one of the
user uploaded files. User uploaded files are served by a separate
endpoint which is /user_uploads/. Any other local url such as
/user_avatars/ or /static/ should never be sent to thumbor for
thumbnailing.
Not sending /user_avatars/ to thumbor for thumbnailing makes sense
because they are already properly thumbnailed and stored properly.
/static/ urls host very few images we use for demo and can be safely
be excluded from thumbnailing.
2018-10-16 16:00:47 -07:00
Vishnu Ks 14a6a8db1a logs: Mock out warning logs in test_scrub_realm. 2018-10-16 11:16:47 -07:00
Vishnu Ks 57e0a24020 management: Scrub delivery email as well while running scrub_realm. 2018-10-16 11:16:46 -07:00
Kevin Lin b2c29274f3 presence: Allow bots to fetch realm presence data.
Before, presence information for an entire realm could only be queried via
the `POST /api/v1/users/me/presence` endpoint. However, this endpoint also
updates the presence information for the user making the request. Therefore,
bot users are not allowed to access this endpoint because they don't have
any presence data.

This commit adds a new endpoint `GET /api/v1/realm/presence` that just
returns the presence information for the realm of the caller.

Fixes #10651.
2018-10-15 12:51:28 -07:00
Steve Howell 76deb30312 preview: Hash cache keys for preview urls.
We don't want really long urls to lead to truncated
keys, or we could theoretically have two different
urls get mixed up previews.

Also, this suppresses warnings about exceeding the
250 char limit.

Finally, this gives the key a proper prefix.
2018-10-14 09:28:57 -07:00
Tim Abbott 68ab71eb8b push: Fix exceptions when removing push notifications.
Now that we allow multiple users to have registered the same token, we
need to configure calls to unregister tokens to only query the
targeted user_id.

We conveniently were already passing the `user_id` into the push
notification bouncer for the remove API, so no migration for older
Zulip servers is required.
2018-10-12 11:19:23 -07:00
Steve Howell 0f7628280f narrow: Handle spurious emails in pm-with searches.
If cordelia searches on pm-with:iago@zulip.com,cordelia@zulip.com,
we now properly treat that the same way as pm-with:iago@zulip.com.

Before this fix, the query would initially go through the
huddle code path.  The symptom wasn't completely obvious, as
eventually a deeper function would return a recipient id
corresponding to a single PM with @iago@zulip.com, but we would
only get messages where iago was the recipient, and not any
messages where he was the sender to cordelia.

I put the helper function for this in zerver/lib/addressee, which
is somewhat speculative.  Eventually, we'll want pm-with queries
to allow for user ids, and I imagine there will be some shared
logic with other Addressee code in terms of how we handle these
strings.  The way we deal with lists of emails/users for various
endpoints is kind of haphazard in the current code, although
granted it's mostly just repeating the same simple patterns.  It
would be nice for some of this code to converge a bit.  This
affects new messages, typing indicators, search filters, etc.,
and some endpoints have strange legacy stuff like supporting
JSON-encoded lists, so it's not trivial to clean this up.

Tweaked by tabbott to add some additional tests.
2018-10-12 10:18:30 -07:00
Steve Howell 51bd36e448 tests: Add coverage to get_service_interface_class(). 2018-10-11 16:12:07 -07:00
Steve Howell 8379aeee15 outgoing bots: Fix header for generic servers.
For our bots that use GenericOutgoingWebhookService
(which are basically Zulip style bots), we now
include a "content-type" header of "application/json".

We accomplish this by having the service classes
implement their own custom method called
`send_data_to_server`. For the Slack-related
code, we just extracted code from `do_rest_call`,
and then for the Zulip-related code, we added
a `headers` parameter.
2018-10-11 16:12:07 -07:00
Steve Howell 8226e13e9c bot tests: Replace use of MockServiceHandler.
We'll just use a real class here, since the service
handlers are pretty lightweight and just munge data.
2018-10-11 16:12:07 -07:00
Steve Howell 31597cf33e Remove timeout parameter in do_rest_call().
Nobody was setting it.
2018-10-11 16:12:07 -07:00
Steve Howell 69ee84bb14 refactor: Extract build_bot_request().
This fixes a couple things:

    * process_event() is a pretty vague name
    * returning tuples should generally be avoided
    * we were producing the same REST parameters in both
      subclasses
    * relative_url_path was always blank
    * request_kwargs was always empty

Now process_event() is called build_bot_request(),
and it only returns request data,
not a tuple of `rest_operation` and `request_data`.

By no longer returning `rest_operation`, there are
fewer moving parts.  We just have `do_rest_call` make
a POST call.
2018-10-11 16:12:07 -07:00
Steve Howell 16eff75e49 refactor: Simplify how we use base_url.
Before this change, we instantiated base_url into a superclass
of subclasses that returned base_url into a dictionary that
gets returned to our caller.

Now we just pull base_url out of service when we need to make
the REST call.
2018-10-11 16:12:07 -07:00
Steve Howell b89a94f730 Improve errors when we can't connect to a bot server.
We don't overwhelm people with error info when bots
fail to connect or time out.
2018-10-11 16:12:07 -07:00
Steve Howell 3790c469e9 outgoing bots: Report JSON errors to users.
We should arguably report these to bot owners
as well, but this is at least an improvement
over having the server crash.
2018-10-11 16:12:07 -07:00
Steve Howell df4b665658 refactor: Parse JSON from bots in one place.
We move the JSON parsing step into the
higher level function: process_success_response().

In the unlikely event that we'll start integrating
with a solution that doesn't use JSON, we can deal
with that, and for now doing the parsing in one
place will help us make error reporting more
consistent.

In a subsequent commit we'll introduce better
error handling for malformed JSON.
2018-10-11 16:12:07 -07:00
Steve Howell 229dd5d861 outgoing webhooks: Get rid of "Success!" prefix.
The earlier code here, if it got a payload with
"response_string" as a key, would prefix the
corresponding value with "Success!".  We just
want the bot to set its own content.

The code is reorganized here so that process_success()
always produces a value keyed by "content" from
incoming data, and then process_success_response()
doesn't do any fancy munging of the data.
2018-10-11 16:12:07 -07:00
Tim Abbott 8cf104b643 avatar: Allow API authentication for /avatar/ routes.
This makes it feasibly for the mobile apps to correctly render user
avatars generated by the `!avatar()` syntax.
2018-10-11 15:52:29 -07:00
Aditya Bansal 6893f52ad9 thumbnails: Instruct thumbor to sharpen thumbnailed images.
Fixes: #10218.
2018-10-11 15:44:47 -07:00
Aditya Bansal 6e433186a1 thumbnails: Change thumbnail size to be 300px.
Fixes: #10219.
2018-10-11 15:44:47 -07:00
Vishnu Ks 962d72b58b retention: move_messages_to_archive should accept multiple message ids.
This will speed up the scrub realm management command. Calling the
function with a single message_id in a loop was extremely inefficient.
2018-10-11 15:31:12 -07:00
Vishnu Ks 6972de21be management: Add command to scrub a realm of personal data. 2018-10-11 15:30:26 -07:00
Vishnu Ks 5bdadc8061 upload: Create function to delete avatar image. 2018-10-11 15:30:26 -07:00
Vishnu Ks 6aa4b64dc0 emails: Don't log emails while running test suite.
Modified the tests to ensure 100% coverage.
2018-10-11 15:12:08 -07:00
Vishnu Ks d8c19cb003 models: Move billing models from zilencer to corporate. 2018-10-11 14:54:29 -07:00
Steve Howell 4956107c53 refactor: Simplify return type for process_success().
There's no reason to return a failure message in
process_success(), since it's implied to be part of
the success codepath.  I didn't look at the full history
of how the strange API evolved, but the second element
of the tuple was clearly noise by the time I got here.
Neither of the subclasses ever set it, and none of the
consumers used it.
2018-10-09 15:56:24 -07:00
Steve Howell f2dd218331 refactor: Inline succeed_with_message().
This two-line function wasn't really carrying its
weight, and it just made it harder to refactor the
overall codepath.

Eliminating the function forces us to mock at a slightly
deeper level, which is probably a good thing for what
the test intends to do.  The deeper mock still verifies that
we're sending the message (good) without digging into
all the details of how we send it (good).

Note that we will still keep around the similarly named
`fail_with_message` helper, which is a lot more useful.
(The succeed/fail scenarios aren't really symmetric here.
For success, there are fewer codepaths that do more complex
things, whereas we have lots and lots of failure codepaths
that all do the same simple thing of replying with a canned
message.)
2018-10-09 15:56:24 -07:00
Steve Howell fa505a1af1 refactor: Have process_success return structured data.
Before this change subclasses of OutgoingWebhookServiceInterface
would return a raw string as the first element of its return
tuple in process_success().  This is not a very flexible
design, as it prevents the bot from passing extra data like
`widget_content`.

It's also possible in the future that we'll want to let outgoing
bots reply directly to senders who mention them on streams, and
again the original design was overly constrained for that.

This commit does not actually change any functionality yet.
2018-10-09 15:56:24 -07:00
Callum Fraser 383f1633e6 api: Add limit of total messages available per request of 5000.
Tweaked by tabbott to use a declared constant rather than just use
5000 in multiple places; this also means we can change the count
without updating translations.

Fixes #10446.
2018-10-09 15:43:21 -07:00
Vishnu Ks d8ddb7f69b settings: Remove reminder-bot from REALM_INTERNAL_BOTS. 2018-10-02 11:12:07 -07:00
Rishi Gupta 7f16d27015 api docs: Update api-keys. 2018-09-25 21:02:58 -07:00
Harshit Bansal 20aee7b116 emoji: Change fallback emojiset for text emojiset.
Now that we have added back support for `google-blob` emojiset,
changing the fallback emojiset for `text` emojiset to `google-blob`
make sense.
2018-09-21 18:34:12 -07:00
Shubham Padia 6bfa29b8e6 notifications: Fix soft-deactivated users don't get push notifications.
Fixes the urgent part of #10397.

It was discovered that soft-deactivated users don't get mobile push
notifications for messages on private streams that they have configured
to send push notifications.

Reason: `handle_push_notification` calls `access_message`, and that
logic assumes that a user who is a recipient of a message has an
associated UserMessage row. Those UserMessage rows are created
lazily for soft-deactivated users, so they might not exist (yet)
until the user comes back.

Solution: Ensure that userMessage row is created for
stream_push_user_ids and stream_email_user_ids in create_user_messages.
2018-09-21 12:06:18 -07:00
Rishi Gupta b470cef864 import: Set Realm.plan_type to SELF_HOSTED on import.
Tweaked by tabbott to avoid an unnecessary .save().
2018-09-21 10:57:22 -07:00
Robert Spier f4c82a1153 emails: Add detection for ChromeOS User-Agents. 2018-09-21 10:41:05 -07:00
Tim Abbott feee76eb23 export: Fix exporting files with S3 upload backend.
At some point as part of the process of supporting renumbering data,
we changed the structure of our file uploads to expect `path` to match
`s3_path`, with both having the relative path within the overall
hierarchy (including the realm ID).  This change updates the more
rarely-used S3 export code path to use that model, fixing a crash when
messages reference an Attachment object with a rewritten path_id.
2018-09-20 20:14:19 -07:00
Eeshan Garg 4f366daec0 markdown: Add extension for creating tabbed sections on /help and /api. 2018-09-18 13:49:34 -07:00
Rishi Gupta 44ea5f8492 user docs: Update message-a-stream-by-email.
Note we're no longer using subscriptions_html in the help docs, so no need
to test for it. There is already a test for subscriptions_html in
IntegrationTest.
2018-09-15 23:57:22 -07:00
Rishi Gupta a87ed67695 user docs: Add relative links for gear menu items. 2018-09-15 23:47:45 -07:00
Rishi Gupta 7acb4e9165 portico: Update text for reset_emailed.html.
The current text may have made sense with different styling, but it feels
harsh now.
2018-09-15 18:45:10 -07:00
Eeshan Garg 7a755ff520 models: Add get_user_by_id_in_realm_including_cross_realm().
Ongoing efforts surrounding #9474 and #10360 are contingent upon
such a function being available.
2018-09-12 11:43:24 -07:00
Vishnu Ks 169de2f9cb billing: Include link to /plans and /billing in gear menu. 2018-09-07 17:30:26 -07:00
Rishi Gupta 2a0523cbe5 billing: Hide /plans if Realm.plan_type=SELF_HOSTED. 2018-09-05 13:44:35 -07:00
Vishnu Ks c913eafdf4 portico: Change buttons on /plans to reflect current plan. 2018-09-05 13:44:35 -07:00
Vishnu Ks 9489ce0efc plans: Redirect to login if /plans is accessed from realm subdomain. 2018-09-05 13:44:35 -07:00
Rohitt Vashishtha 2864ce552b bugdown: Add @user|id syntax to support users with same name. 2018-08-31 14:16:47 -07:00
Rohitt Vashishtha bc37800ad5 bugdown: Store users with same full_names in mention_data.
We start by stripping the ids in front of the name before the database
lookup. This has the advantage of not mentioning anyone if an incorrect
user id and full name combination is specified, as well as not having
the query the database twice, once by fullname and next by id.

Previously, we were storing only the most recent person with the same
full name as others; this commit adds new keys to the dict such that
simply looking by name would get you the newest user with this name,
and the get_user_by_id function can index the remaining users.
2018-08-31 14:16:47 -07:00
Harshit Bansal e7311cdf5d emoji: Bring back the feature of changing emojisets.
This is largely inspired by requests from people not liking the
Google's new emojiset. A lot of people were requesting to revert
back to old blobs emojiset so we are re-enabling this feature
after making relevant infrastructure changes for supporting google's
old blob emojiset and re-adding support for twitter emojiset.

Fixes: #10158.
2018-08-31 11:52:24 -07:00
Shubham Padia 69bfa8c432 auth: Use different defaults for name and email for fakeldap.
Fixes part of #10297.
Use FAKE_LDAP_NUM_USERS which specifies the number of LDAP users
instead of FAKE_LDAP_EXTRA_USERS which specified the number of
extra users.
2018-08-31 17:07:02 +05:30
Pragati Agrawal ba4e4e38ae user settings: Add setting to make login notifications optional.
This adds a feature in the "Notification" section of "Settings" tab,
which lets user enable or disable login emails notification.

Tweaked by tabbott to simplify the test.

Fixes: #5795, progress towards #5854.
2018-08-28 14:02:31 -07:00
Tim Abbott 54e90deda8 notifications: Handle APNS "Unregistered" errors properly.
Apparently, the APNS library we're using is inconsistent about the
format of its result entries; some are strings while others are
tuples.
2018-08-28 12:12:46 -07:00
Roman Godov a2f407c11b api: Add an endpoint for listing the user groups in realm.
Adds endpoint for listing the user groups in realm.
Provides "description", "members", "name" and "id" for each user group.

Fixes #10298
2018-08-27 16:56:23 -07:00
Lyla Fischer 1efcdfdb5c templates: Merge Help and API doc main into documentation_main.
With minor fixes by eeshangarg.
2018-08-27 20:19:24 -02:30
Yago González 8b141e6dbc muting: Correct agrammatical error message.
The error message displayed when unmuting a topic that wasn't previously
muted wasn't properly formulated.
2018-08-27 14:12:03 -07:00
Vishnu Ks 62b93d551a signup: Fix broken resend email link in realm creation flow.
Also use name for selecting form in casper tests
as form with action=new is present in both /new
and /accounts/new/send_confirm/ which breaks
test in CircleCI as
waitWhileVisible('form[action^="/new/"]) never stops
waiting.
2018-08-26 22:54:05 -07:00
Vishnu Ks d2e4417a72 urls: Separate endpoint for signup and new realm email confirm.
This is preparation for the next commit.
2018-08-26 22:53:57 -07:00
Harshit Bansal ec2ebd1cc9 build_emoji: Change sprite CSS files to use hyphens in names.
This is more consistent with our other names in emoji infra.
2018-08-26 22:16:23 -07:00
Steve Howell ca272ef49d tests: Test how we handle extra data for zform messages. 2018-08-24 10:00:25 -07:00
Steve Howell 9e8930f6de tests: Test get_widget_data() helper.
We also remove some unreachable code.  Calling
split() always returns at least one token, even
if it's just the empty string.  This is tested
directly on this commit, plus messages with
empty content get rejected pretty early in
the execution path.
2018-08-24 10:00:25 -07:00
Steve Howell f3f30d29f9 tests: Add tests for /tictactoe message.
The /tictactoe widget exercises the codepath where
we don't actually extract `extra_data` from the message.
2018-08-24 10:00:25 -07:00
Steve Howell 922b1683fc tests: Add test for /poll message.
This goes through the /messages endpoint
to get more full-stack testing, and it
focuses on how SubMessage rows get
created.
2018-08-24 10:00:25 -07:00
Vishnu Ks 0f628dc4aa management: Fix send_password_reset_email command.
This had been broken in a recent refactoring of the password-reset
code path.

Fixes #10379.
2018-08-23 12:05:49 -07:00
Yashashvi Dave d5153bd136 events: Convert custom user field value to json object on update event.
In user type custom field, field value is list of user ids. We weren't
converting list to json object in update event payload. This throws
error in frontend, cause we store stringify representation of custom
field value. Therefore, after update event is recieved field-value-
type gets updated to array from string which throws json parsing error.
2018-08-22 22:45:08 -07:00
Shubham Dhama e784c95d97 guests: Prevent guests from sending to unsubscribed public streams.
This matches the overall security model of these users only having
access to streams they are subscribed to.
2018-08-22 17:53:42 -07:00
Shubham Dhama e70cf3bd67 emails/password_reset: Change text for listing other active accounts.
The main benefit of this change is that it reduces the amount of total
string that we might need to tag for translation.

Fixes: #10323.
2018-08-22 17:49:40 -07:00
Shubham Dhama aa8b3d2beb emails/password_reset: Change template string for deactivated users. 2018-08-22 17:43:23 -07:00
Steve Howell 44756a326d tests: Test duplicate default stream/groups.
This also gets us to 100% line coverage
for actions.py.
2018-08-22 16:52:30 -07:00
Steve Howell cac9a02d6d tests: Add coverage for check_invite_limit().
The function being tested here was kind of an
emergency response to some spam attacks.  It
works for a pretty specific set of circumstances,
so it requires a lot of setup.

We may eliminate this function as we improve
our realm "plan types", and if that happens, we
can either eliminate this test or repurpose it.
2018-08-22 16:52:30 -07:00
Joshua Pan 79e2266d6e models: Create starred_message_counts boolean field. 2018-08-21 13:42:22 -07:00
Joshua Pan 83d8d662dc events: Query starred message ids in fetch_initial_state_data. 2018-08-21 13:01:41 -07:00
Shubham Padia 34cb0eae29 tests: Stop using fixtures to test generate_dev_ldap_dir.
The output of generate_dev_ldap_dir was being tested against the fixture
located at zerver/tests/fixtures/ldap_dir.json. This didn't make much sense
as generate_dev_ldap_dir was itself used by developers to generate/update
the fixtures. Instead, test_generate_dev_ldap_dir checks the structure of
the dict returned by generate_dev_ldap_dir. The structure is checked by
regex checks, checking whether the dict contains some keys or not, etc.
2018-08-21 12:55:36 -07:00
Vishnu Ks 9bb338be11 models: Add plan_type to Realm. 2018-08-21 12:39:06 -07:00
Steve Howell a6bc3886e6 refactor: Extract send_peer_remove_event().
This prevents leaking some variables into an already
cluttered function.

We also add test coverage for what's now an
early-exit condition in the new function--we exempt
public MIT streams from these events.
2018-08-21 11:23:40 -07:00
Steve Howell 092bb6a728 tests: Refactor test_bulk_subscribe_MIT().
We extract a couple local vars to avoid
repeating ourself.

We also explictly set the stream as a zephyr stream.
2018-08-21 11:23:40 -07:00
Steve Howell e7062b77c8 tests: Test superuser can also mirror to private streams.
This extends a test that proved only what Cordelia
could do with/without super_user privileges when she
was trying to send to an unsubscribed stream as herself.

Now the test shows the same powers extend to Cordelia
when she's sending messages on behalf of a mirrored
user.
2018-08-21 11:23:40 -07:00
Steve Howell aee46bb1e9 tests: Test forging timestamps for mirrored messages. 2018-08-21 11:23:40 -07:00
Steve Howell e99c0929f0 tests: Test race handling for creating mirror users.
We simulate a race condition by mocking create_user
to actually create a user, but then raise an
IntegrityError (as if another process had actually
created the user, not our test).

I also changed the real code to use explicitly
named parameters.
2018-08-21 11:23:40 -07:00
Rishi Gupta 99b55b712b user docs: Shorten instructions when using relative settings links. 2018-08-20 21:26:01 -07:00
Rohitt Vashishtha 920ef2b7f7 bugdown: Add mention_data.get_user_by_id().
This will allow us to do the lookups required to support the upcoming
`@**name|id**` syntax.
2018-08-20 12:46:46 -07:00
Tim Abbott 3cfb2000cc ldap: Improve error message for username/LDAP domain mismatches. 2018-08-20 10:39:34 -07:00
zhoufeng1989 d5f752df84 tests: Tests query count when creating stream with subscriptions API.
These test cases are used to test the cost of stream creation.
Three scenarios of stream creation are covered:
1) create a public stream;
2) create a private stream;
3) create a public stream with announce=true when there is a notification stream.

Fix: #4804.
2018-08-17 08:58:38 -07:00
Eeshan Garg f476ec7fac webhooks/freshdesk/doc.md: Render example JSON correctly.
We've been getting reports from users that our Freshdesk webhook
isn't working correctly. It turns out that the issue had nothing
to do with the webhook implementation itself!

In freshdesk/doc.md, we have a JSON template we ask users to
copy/paste into a textbox in the Freshdesk UI. That JSON template
contains "{{" and "}}" characters which we escaped as Unicode
decimals to prevent clashes with Jinja2 syntax in other parts
of the same template. This worked for a while!

But thanks to the changes introduced as part of the
nested_code_blocks extension, such escaped characters were never
decoded, leading users to copy/paste the same template but with
raw escaped unicode representations of "{{" and "}}" inside. And
that eventually broke our webhook implementation.

This commit makes sure that such characters are properly "unescaped",
just for Freshdesk docs.
2018-08-17 08:55:41 -07:00
Yashashvi Dave 8ef0c6d99a custom profile field: Add length validation on field name and hint.
Add max length validation for custom field name and hint
on create-field and update-field backend views and
frontend forms.

Fixes #10284.
2018-08-17 08:37:27 -07:00
Steve Howell 948190c9ff tests: Test spam prevention feature for invites.
We have code to prevent newbies on open realms
from inviting users.  This is mostly intended
to hinder spammers.  This commit just adds some
test coverage.
2018-08-17 08:14:42 -07:00
Steve Howell 0e56fecbd9 peformance: Avoid broad StreamCount queries.
Our get_streams_traffic function used to query
all streams in the StreamCount table if you
passed in `None` for `streams`.

Now we require that you pass in a list of
stream_ids.

I don't know how much work this will save
the database, since probably the bulk of
the work is aggregating.  If we need to fine
tune DB performance, we could possibly add
`realm` as an argument and add it to the filter.

What we'll immediately get, for large multi-realm
installations, is less data over the wire and
less work for the ORM.
2018-08-17 08:14:42 -07:00
Steve Howell b05f26c6d1 tests: Add test for get_streams_traffic(). 2018-08-17 08:14:42 -07:00
Steve Howell fb283ac672 tests: Test JsonableError in lookup_default_stream_groups(). 2018-08-17 08:14:42 -07:00
Steve Howell 75e297daf2 tests: Get line coverage for empty streams results.
We have defensive code in functions for combinations
of arguments and/or inputs that doesn't happen
in practice, buts this gets us some line coverage.
2018-08-17 08:14:42 -07:00
Steve Howell 96c6af1249 tests: Test include_default option for /json/streams. 2018-08-17 08:14:42 -07:00
Steve Howell e61565fdf6 tests: Add more testing for patching bots.
This commit adds some more tests related to patching
a bot's `default_sending_stream`.

Unfortunately, this didn't reach the code that I was
intending to add line coverage to, since checks happen
higher up in the stack, but the test code I added
is probably worthwhile.
2018-08-17 08:14:42 -07:00
Steve Howell 93e8798ac7 Extract get_last_message_id().
We want our methodology for extracting the last message
id to be consistent, particularly in terms of how we
handle edge cases.  (I'll concede that the
`bulk_remove_subscriptions` codepath never hits that
corner case in practice, but it's harmless to handle
the theoretical case.)

It may also be nice to have this function show up
clearly in profiling.

This also adds some direct testing to the function.

It's not clear to me why we don't use `latest('id')`
in the implementation, but that's outside the scope
of this commit.
2018-08-17 08:14:42 -07:00
Steve Howell 0322dcc6cf tests: Add direct coverage to validate_email().
These direct tests add some line coverage.
2018-08-17 08:14:42 -07:00
Steve Howell 20c4d357cc tests: Add test for sending group mentions in messages. 2018-08-17 08:14:42 -07:00
Harshit Bansal 7be7561c68 emoji: If current emojiset is text then fallback to google emojiset.
If `TEXT_EMOJISET` is currently selected emojiset then fallback to
`GOOGLE_EMOJISET` for displaying emojis in emoji picker and
composebox typeahead. We should pre-load the spritesheets in`emoji.js`
even in case of text emojiset otherwise on slow networks emoji picker
will appear empty initially.
2018-08-14 12:01:00 -07:00
Steve Howell 6d18fa116e minor: Sort some imports. 2018-08-14 10:34:58 -07:00
Steve Howell 3fce1ec649 tests: Test logging for internally sent messages. 2018-08-14 10:34:58 -07:00
Steve Howell 2dbd11ba9d tests: Test notification bot can send to private streams. 2018-08-14 10:34:58 -07:00
Steve Howell 82469b89e5 tests: Tests bots can send to owner's streams. 2018-08-14 10:34:58 -07:00
Steve Howell edd8437c76 tests: Test API super users can send to private streams. 2018-08-14 10:34:58 -07:00
Yashashvi Dave 2a73b2b805 custom profile field: Add at-least-one-choice validation for choice field.
Add validation at-least-one-choice required for every choice type field
in backend.
2018-08-14 10:17:57 -07:00
Yago González 53b9118e97 emails: Apply the user's hour format in login notification timestamps.
The timestamp used for new login notifications always used the 12-hour
format. Instead of that, we use now the one preferred by the user, as
reflected in their settings.
2018-08-13 19:06:54 -07:00
Yago González 75b39866c7 emails: Display the timezone's name on login notifications.
Show the timezone's name instead of the UTC offset, since it's easier to
infer where does the login come from that way.
2018-08-13 19:06:54 -07:00
Steve Howell 33f576f514 tests: Add coverage for email/stream encode/decode logic. 2018-08-13 13:23:29 -07:00
Rishi Gupta 4813bac98b signup: Add opayq.com to disposable domain whitelist. 2018-08-13 10:56:47 -07:00
Rishi Gupta 228e237d51 tests: Fix incorrect comment in test_invite_using_disposable_email. 2018-08-13 10:56:47 -07:00
Steve Howell 46e25896c7 tests: Test handling of bogus widget_content. 2018-08-13 10:37:35 -07:00
Steve Howell 297f086b6a tests: Add coverage for service bot events.
We test the "skipping" logic a couple different ways.
2018-08-13 10:37:35 -07:00
Steve Howell 90844700fb tests: Test for unexpected bot types. 2018-08-13 10:37:35 -07:00
Shubham Padia e21e8c1bae compose: Hide subscribe button and change text for waiting period users.
Fixes #10124.
Users in the waiting period category cannot subscribe other users to
a stream. When a user tries to mention another unsubscribed user, a
warning message appears with a subscribe button on it to subscribe
the other user.
This commit removes the subscribe button and changes the warning text
for users in the waiting period category.
2018-08-13 10:18:35 -07:00
Shubham Padia feb2cdf378 onboarding: Change logic for preventing new login emails for a new user.
Issue: When you created a new organization with /new, the "new login"
emails were emailed. We previously had a hack of adding the
.just_registered property to the user Python object to attempt to
prevent the emails, and checking that in zerver/signals.py. This
commit gets rid of the .just_registered check.
Instead of the .just_registered check, this checks if the user has
joined more than a minute before.
A test test_dont_send_login_emails_for_new_user_registration_logins
already exists.

Tweaked by tabbott to introduce the constant JUST_CREATED_THRESHOLD.

Fixes #10179.
2018-08-13 10:16:41 -07:00
Steve Howell 3a2422a149 tests: Test corner cases for bot-related PMs. 2018-08-11 14:51:26 -07:00
Steve Howell 7e32e2bf81 tests: Cover ValueError logic in extract_recipients. 2018-08-11 14:51:26 -07:00
Steve Howell 134678849d tests: Test sending messages outside of your realm. 2018-08-11 14:51:26 -07:00
Steve Howell 05763dcb1d tests: Cover stream name collisions in deactivation test. 2018-08-11 14:51:26 -07:00
Steve Howell 413a0174f4 Extract a zephyr.py library.
Right now it only has one function, but the function
we removed never really belonged in actions.py, and
now we have better test coverage on actions.py, which
is an important module to get to 100%.
2018-08-11 14:51:26 -07:00
Steve Howell 71d46083da tests: Ensure streams get created for internal messages. 2018-08-11 14:51:26 -07:00
Steve Howell e9a31232df tests: Test RuntimeError logic for realm check. 2018-08-11 14:51:26 -07:00
Steve Howell 101446ec52 tests: Add coverage for message truncation logic. 2018-08-11 14:51:26 -07:00
Aditya Bansal 0ed2b1d574 thumbnails: Fix bug with '/user_avatar/' links in Zulip messages.
In this commit we fix a bug due to which url preview images for urls
to custom emojis, realm icons or user avatars appeared broken when
such urls would be part of a Zulip message.
2018-08-10 17:17:50 -07:00
Aditya Bansal 985b8bb843 thumbnails: Refactor to include '/user_uploads/' in encrypted url.
This is a preparatory commit to fix a bug in which a user posts
a link of custom emoji, user avatar or realm icon in a Zulip
message.
In this commit we are just adjusting the url generation in the
backend to have the '/user_uploads/' in the encrypted url generated
which the user is supposed to be redirected to and therefore
essentially reaching thumbor with the encrypted url.
This is necessary because 'user_uploads' and 'user_avatars' (or any
other item under 'user_avatars' endpoint) have a different folder
location under the local file storage backend. 'user_uploads'
endpoint's stuff is stored in a 'files' directory whereas stuff
'user_avatars' endpoint's stuff is stored in a 'avatars' directory.
Thumbor needs to know from which directory a particular local file
needs to be retrieved and therefore the zthumbor/loaders.py adds
a prefix location for the directory.

Since in an upcoming commit we are going to add user_avatars
directory location 'avatars' folder as a prefix this preparatory
commit helps simply doing the changes.
2018-08-10 17:17:50 -07:00
Rhea Parekh 3ff339c294 slack import: Add support for uploads in messages through 'files' keyword.
It appears that Slack just changed their export format, and how uses
this `files` list for user-uploaded files.
2018-08-10 16:20:36 -07:00
Rhea Parekh 20bca1409f import: Set emoji records 'last_modified' value in 'import_uploads_s3'.
The 'last_modified' value in emoji records is
needed for uploading the file to the S3 backend.
We set the same in the function 'import_uploads_s3'.

We also have to remove the keyword 'last_modified'
while building the RealmEmoji dict, as it is not
a field which exists in RealmEmoji objects.
2018-08-10 16:20:36 -07:00
Tim Abbott da8f4bc0e9 push notifications: Add support for removing GCM push notifications.
This uses the recently introduced active_mobile_push_notification
flag; messages that have had a mobile push notification sent will have
a removal push notification sent as soon as they are marked as read.

Note that this feature is behind a setting,
SEND_REMOVE_PUSH_NOTIFICATIONS, since the notification format is not
supported by the mobile apps yet, and we want to give a grace period
before we start sending notifications that appear as (null) to
clients.  But the tracking logic to maintain the set of message IDs
with an active push notification runs unconditionally.

This is designed with at-least-once semantics; so mobile clients need
to handle the possibility that they receive duplicat requests to
remove a push notification.

We reuse the existing missedmessage_mobile_notifications queue
processor for the work, to avoid materially impacting the latency of
marking messages as read.

Fixes #7459, though we'll need to open a follow-up issue for
using these data on iOS.
2018-08-10 13:58:39 -07:00
Tim Abbott 58307f80aa event_queue: Stop mocking push notifications in most tests.
Historically, queue_json_publish had a special third argument that was
basically its default mock behavior in the test suite.  We've been
migrating away from that model, because it was confusing and resulted
in poor test coverage of our queue worker code paths; this was one of
the last holdouts.

As it turns out, we don't exercise this code path in a way that
impacts tests much; the main downside of this change is a likely small
penalty to performance of the full test suite when sending private
messages.
2018-08-10 13:58:39 -07:00
Tim Abbott c3c7b33351 tests: Move clear_client_event_queues_for_testing to ZulipTestCase.
Following recent testing flakes that were traced down to this not
having been called causing `receiver_is_off_zulip` to depend on test
ordering, it makes sense to centralize this.

I think it should always have been in ZulipTestCase; it appears the
reason it wasn't from the beginning was that originally only
test_events.py interacted with it, and do_test there still needs to
call this directly (because it can be called multiple times within a
single test).  And then we did the wrong thing as expanded use of
Tornado event_queue code in tests to more of the codebase.
2018-08-10 13:58:39 -07:00
Tim Abbott 93c12fad28 test_events: Use clear_event_queue_for_testing more systematically.
This prevents these unit tests from accidentally leaking data outside
their boundaries.

Verified using a test that fails after test_events without this change.
2018-08-10 13:58:39 -07:00
Tim Abbott 8ceb3b88ad test_tornado: Fix leaking of event queue state.
Apparently, we weren't calling the proper clear functions inside the
Tornado tests, which resulted in unexpected behavior in other tests
that were relying on the Tornado event queue system being empty.

(In this case, a new test for mobile push notifications that assumed
receiver_is_off_zulip() was always true failed after this was run).
2018-08-10 13:58:39 -07:00
Vishnu Ks d64ba5d2be billing: Update the charged amount when user changes plan in upgrade page. 2018-08-10 13:52:35 -07:00
Rhea Parekh cf60b8821d outgoing webhooks: Warn user that PMs are not supported in Slack-format webhook.
Private messages are not supported in Slack-format webhook.
Instead of raising a NotImplementedError, we warn the user
that PM service is not supported by sending a message to the
user.

Added tests for the same.

Fixes #9239
2018-08-09 17:44:26 -07:00
Rhea Parekh 2357b1e145 tests: Rename 'event' to 'stream_message_event' for clarity. 2018-08-09 17:42:10 -07:00
Tim Abbott 83300c329b register: Fix period at end of heading. 2018-08-09 17:17:45 -07:00
Shubham Padia bdaff17e28 narrow: Use is_private flag index for `is:private`.
This implements a significant performance optimization for users
clicking the `Private messages` narrow in the Zulip UI, especially for
those users who do not have 50 recent private messages in an
organization with a lot of stream message traffic (because then
previously, postgres needed to scan through a huge amount of history
to find enough private messages).

The database index powering it can also support many other queries we
might want to do in the future to support "recent conversations" type
features.

Fixes #6896.
2018-08-09 16:10:01 -07:00
Shubham Padia 9dcac3479e actions.py: Set is_private flag in do_send_messages. 2018-08-09 16:08:04 -07:00
Shubham Padia a524d425ad actions.py: Block client interaction with flags in the NON_API_FLAGS.
Raise error if flag is present in NON_API_FLAGS or is not present in
UserMessage.flags.
2018-08-09 16:08:03 -07:00
Shubham Padia 2f895afdaa models: Do not leak `is_private` UserMessage flag to the API.
See the comment for why this is correct; basically, this flag is used
only for internal accounting, and would only confuse API clients.
2018-08-09 16:03:30 -07:00
Tim Abbott 25b646feaf decorator: Fix error message for accessing deactivated accounts.
The previous message was potentially a lot more ambiguous about
whether this was something about presence.  "Deactivated" makes it
explicit that some action was taken to deactivate the account.
2018-08-09 15:58:31 -07:00
Rhea Parekh 26300110ca import: Fix rendered_content in imported messages.
After the messages have been imported, set the rendered_content of the
messages instead of leaving its value to be 'None'.

This is important to ensure that:
(1) Performance for users is good after completing the import.
(2) The database's full-text indexes have all of the imported messages
(which only happens properly when Message rows have their
rendered_content field edited).

Fixes #9168.
2018-08-09 15:12:53 -07:00
Eeshan Garg 9e2a369e42 render_markdown_path: Add test for templates with absolute paths.
In certain cases we have to load a template directly because it
isn't in Jinja2's recognized template directories. This commit
adds a test to make sure that absolute paths are recognized
if they are pure Markdown files.
2018-08-09 14:34:52 -07:00
Shubham Padia d409555b2f auth: Add function for generating test ldap_dir to backends.py.
Generates ldap_dir based on the mode and the no. of extra users.
It supports three modes, 'a', 'b' and 'c', description for which
can be found in prod_settings_templates.py.
2018-08-09 13:46:44 -07:00
Steve Howell dc70bc1d3f tests: Make retention tests less time-sensitive.
We now update all test messages to have a pub_date
of "now" in the setUp() function in TestRetentionLib.

We've seen tests flake on query counts before this
patch.  It's not certain that the test flaked due
to time-related glitches, but it seems the most
plausible explanation.
2018-08-09 06:13:40 -04:00
Yago González 298aa0fcbf mobile: Make otp_encrypt_api_key accept API keys.
Since otp_encrypt_api_key only encrypts API keys, it doesn't require
access to the full UserProfile object to work properly. Now the
parameter it accepts is just the API key.

This is preparatory refactoring for removing the api_key field on
UserProfile.
2018-08-08 16:45:40 -07:00
Yago González f6219745de users: Get all API keys via wrapper method.
Now reading API keys from a user is done with the get_api_key wrapper
method, rather than directly fetching it from the user object.

Also, every place where an action should be done for each API key is now
using get_all_api_keys. This method returns for the moment a single-item
list, containing the specified user's API key.

This commit is the first step towards allowing users have multiple API
keys.
2018-08-08 16:35:17 -07:00
Yago González 13b9dd33fa test_decorators: Explain typo-looking expression.
The validate_api_key sentence may look a bit confusing since we are
using webhook_bot's email address but default_bot's API key.

At first sight, and without any context on these tests, it may look like
that's just a typo, but we do want it to be like it is right now because
that way the API key used doesn't correspond to the provided email
address (triggering some untested parts of our backend logic).
2018-08-08 16:35:17 -07:00
Harshit Bansal 9057e543ac emoji: Disable support for letting users switch emojisets.
Due to copyright issues with potentially displaying Apple emojisets on
non-apple devices, as well as iamcal dropping support for the emojione
emojiset (see https://github.com/iamcal/emoji-data/pull/142), we are
dropping (perhaps temporarily) support for allowing users to switch
emojisets in Zulip.

This commit just hides the feature from the user but leaves most of
the infrastructure in place so that in the future if we decide to
re-enable the support we will not need to redo the infrastructure work
(some JS-side code is deleted, mostly because we'll want to re-add the
feature using the do_settings_change infrastructure anyway).

The most likely emoji set to add is the legacy "blobs" Google emoji
set, since it seems popular with some users.

Tweaked by tabbott to remove some additional JS code and update the
changelog.
2018-08-08 15:48:04 -07:00
Shubham Dhama 4d07e485ff test_event_queue: Make subscription management code readable.
This test refactor makes the subscription/stream settings changes use standard
APIs and thus be easier to follow (and more robust to subtle re-fetching bugs).

This is a follow-up to #9181.
2018-08-08 12:39:19 -07:00
Shubham Padia 17d81d46dc Realm filters: Allow `#`(hash) based URLs in filter patterns.
Fixes #9817.
Use case: When people try to add a filter to a site that uses
a hash based router, which makes the URL like:
`http://some-site.domain.com/#/c/$(id)s`.
2018-08-08 11:06:33 -07:00
Yago González 5566646c45 user_groups: Handle renaming to existing names.
Renaming a user group to a name shared by other group wasn't a scenario
handled by the backend, and the server errored whenever this was
attempted.

Now a json_error is returned, letting the user know that a user group
with that name already exists.
2018-08-08 11:03:47 -07:00
Yago González 14bc5c7d5c messages: Don't require "anchor" when "use_first_unread_anchor" is set.
The use_first_unread_anchor parameter allows automatically setting the
anchor to the first message that hasn't been read in this narrow.
Therefore it isn't necessary to specify an anchor when this parameter is
enabled.

Note from Tim: Arguably, we should think about making
`use_first_unread_anchor` the default behavior when anchor is
unspecified, but that's for later consideration.
2018-08-08 10:30:08 -07:00
Yago González e1662024d1 api docs: Load the OpenAPI file only when needed.
We found out in #9953 that, appparently, loading the OpenAPI file was
taking abut a 5% of the Zulip server startup time.

Since in many cases (especially in development) having the file loaded
won't be necessary at all, we read it on the first time data from the
OpenAPI spec is needed.

Tweaked by tabbott to add a test.
2018-08-08 09:00:28 -07:00
Yago González 25d2efb9ca api docs: Live reload the OpenAPI spec on update.
Automatically detect if the OpenAPI spec file has been modified since
the last time it was loaded into memory, and if it has, automatically
reload it to have the latest version.

This feature is designed with development environments in mind. The main
benefit is being able to see the changes made to the OpenAPI document
without needing to restart the development server, which is tedious and
slows the documentation workflow down.
2018-08-08 08:54:25 -07:00
Rhea Parekh 18a4904437 import: Move 'build_attachment' to import_util. 2018-08-07 16:45:42 -07:00
Rhea Parekh b6ccc0bc52 import: Move 'build_defaultstream' to import_util. 2018-08-07 16:45:42 -07:00
Rhea Parekh bee3964f14 import: Move 'build_usermessages' to import_util. 2018-08-07 16:45:42 -07:00
Rhea Parekh 87cc1a6280 import: Move 'build_subscription' and 'build_recipient' to import_util. 2018-08-07 16:35:56 -07:00
Rhea Parekh 1117455a90 import: Move 'ZerverFieldsT' and 'build_zerver_realm' to import_util. 2018-08-07 16:35:56 -07:00
Yashashvi Dave 290388e5e0 stream settings: Fix bug in UI when last user unsubscribe private stream.
When last user(only in case of admin) unsubscribe from private stream,
stream page doesn't get updated. Cause we delete the private stream
as soon as last user unsubscribe from stream.
So `sub` get undefined in frontend, cause that stream is deleted
before unsubscribe-user-from-stream event is received.

Fix this by changing order of events sent to frontend. Event
`subscription: remove` should be sent before `stream: delete` event
from backend.
2018-08-07 13:30:53 -07:00
Yashashvi Dave 6e136be975 default stream: Allows admins to remove any default stream.
This fixes a bug where administrators couldn't remove private
unsubscribed streams from the "default streams" list, because
access_stream_by_name didn't give them access to the stream object.
2018-08-07 13:28:30 -07:00
Harshit Bansal 25fa9a25ff emoji: Add support for animated GIF images.
This commit adds 'resize_gif()' function which extracts each frame,
resize it and coalesces them again to form the resized GIF while
preserving the duration of the GIF. I read some stackoverflow
answers all of which were referring to BiggleZX's script
(https://gist.github.com/BigglesZX/4016539) for working with animated
GIF. I modified the script to fit to our usecase and did some manual
testing but the function was failing for some specific GIFs and was not
preserving the duration of animation. So I went ahead and read about
GIF format itself as well as PIL's `GifImagePlugin` code and came up
with this simple function which gets the worked done in a much cleaner
way. I tested this function on a number of GIF images from giphy.com
and it resized all of them correctly.

Fixes: #9945.
2018-08-04 11:46:58 -07:00
Yago González cf813b44b0 emails: Show HHMM timezone on new login emails.
Email notifications for new logins displayed the login timestamp's
timezone in the location format (e.g. "Asia/Taipei"). Since that can
lead users to understand the login came from that place, the timezone in
those emails is now represented in +/-HHMM format.

Fixes #10178.
2018-08-03 17:15:03 -07:00
Yago González 7d8e058e39 api docs: Implement an exception list for schema validation. 2018-08-02 15:54:41 -07:00
Tim Abbott 6e55342e21 bulk_remove_subscriptions: Pass client object in.
We need the client object to pass on to do_mark_stream_as_read.
2018-08-01 16:48:31 -07:00
Tim Abbott c60f197fde do_update_message_flags: Accept a Client object.
This is important for upcoming logging changes.
2018-08-01 16:40:58 -07:00
Tim Abbott 5f0519dfb4 do_update_pointer: Pass client object from callers.
We also fix an unused import.

This is needed for upcoming logging changes.
2018-08-01 16:40:15 -07:00
Tim Abbott 54d558b128 management: Add library for getting a client object.
This is to be used in some analytics features we're adding in the near
future.
2018-08-01 16:40:15 -07:00
Tim Abbott e9f4d9db2b push_notifications: Fix interface for handle_remove_push_notification.
This really should just accept a message ID.
2018-08-01 16:36:42 -07:00
Kunal Gupta bc43eefbfb notifications: Add function for cancelling GCM notifications.
This adds a new function called handle_remove_push_notification in
zerver/lib/push_notifications.py which requires user_profile id and
the message id which has to be removed in the function.

For now, the function only supports GCM (and is mostly there for
prototyping).

The payload which is being delivered needs to contain the narrow
information and the content of the message.
2018-08-01 15:59:04 -07:00
Tim Abbott 5f8d193bb7 notifications: Include realm_uri in push notifications.
This should make it much simpler for the mobile apps to line up the
data from server_settings against the data in the notifications.

Addresses part of #10094.
2018-08-01 15:46:15 -07:00
Tim Abbott 58ee3fa8c4 page_params: Include avatars and similar data in cross-realm bots.
This ensures that the format of this data structures matches that for
in-realm bots in the main users data structure (including avatars,
etc.).

Fixes #10138.
2018-08-01 15:09:11 -07:00
Rhea Parekh ee37866687 import: Add gitter import file in zerver/data_import directory. 2018-08-01 11:52:14 -07:00
Rhea Parekh b8e1e8b31d import: Add slack import files in zerver/data_import directory. 2018-08-01 11:52:14 -07:00
Roman Godov 5e70577f84 models: Rename Realm.show_digest_email field.
This renames Realm.show_digest_email field to
digest_emails_enabled, for greater clarity as to what it does
just from seeing the setting name, without having to look it up.

Fixes part of #10042.
2018-08-01 11:05:58 -07:00
Tim Abbott 1b2a26ca83 events: Fix missing empty custom profile data dict for new users.
We were getting event-handling exceptions in JS in production if a new
user was created and then went and set a custom profile field, because
there was no `.profile_data` on their user object.  We were able to
trace the issue down to the fact that our events didn't include that
field when creating a new user.
2018-07-31 11:08:11 -07:00
Roman Godov c0806917ec models: Rename Realm.restricted_to_domain field.
This renames Realm.restricted_to_domain field to
emails_restricted_to_domains, for greater clarity as to what it does
just from seeing the setting name, without having to look it up.

Fixes part of #10042.
2018-07-31 09:28:33 -07:00
Tim Abbott 7ea5987e5d errors: Use a setting to control the stream for slow-query logs.
We already had a setting for whether these logs were enabled; now it
also controls which stream the messages go to.

As part of this migration, we disable the feature in dev/production by
default; it's not useful for most environments.

Fixes the proximal data-export issue reported in #10078 (namely, a
stream with nobody ever subscribed to having been created).
2018-07-30 17:40:20 -07:00
Aditya Bansal 77651ece39 thumbnails: Rename size value 'original' to 'full'. 2018-07-30 13:00:23 -07:00
Aditya Bansal 5b5d8bb310 thumbnails: Rename data-original to data-src-fullsize. 2018-07-30 13:00:23 -07:00
Tim Abbott 07af59d4cc tornado: Split get_events_backend into two functions.
The lower-layer function, now called get_events_backend, is intended
to be called by multiple code paths (including the upcoming
get_events_internal).
2018-07-30 12:28:31 -07:00
Tim Abbott 00f6fc2559 subs: Limit editing subscribers for waiting period users.
Does not let you subscribe other users if you are a guest or a
waiting period user.

Co-authored-by: Shubham Padia <shubhamapadia@gmail.com>
2018-07-30 12:12:51 -07:00
Shubham Padia 1015b2fc44 subs: Refactor and add tests for UserProfile.can_create_streams.
This is a preparatory refactor for adding
UserProfile.can_subscribe_other_users.
Although there existed a test for limiting users from creating
streams at `test_subs.test_user_settings_for_adding_streams`,
it did not test the logic inside can_add_streams, tests have
been added to solve that issue.
2018-07-30 12:12:51 -07:00
Rohitt Vashishtha d83ef30904 notification-bot: @-mention the user subscribing you to a stream. 2018-07-30 09:50:48 -07:00
Rhea Parekh 6e7905651f mypy: Fix type errors relate to 'mock'.
1. The type of a mock object should be 'Any'.
2. 'mock' is being imported in the wrong manner.
2018-07-28 15:34:54 -07:00