Commit Graph

8835 Commits

Author SHA1 Message Date
Steve Howell 0f7628280f narrow: Handle spurious emails in pm-with searches.
If cordelia searches on pm-with:iago@zulip.com,cordelia@zulip.com,
we now properly treat that the same way as pm-with:iago@zulip.com.

Before this fix, the query would initially go through the
huddle code path.  The symptom wasn't completely obvious, as
eventually a deeper function would return a recipient id
corresponding to a single PM with @iago@zulip.com, but we would
only get messages where iago was the recipient, and not any
messages where he was the sender to cordelia.

I put the helper function for this in zerver/lib/addressee, which
is somewhat speculative.  Eventually, we'll want pm-with queries
to allow for user ids, and I imagine there will be some shared
logic with other Addressee code in terms of how we handle these
strings.  The way we deal with lists of emails/users for various
endpoints is kind of haphazard in the current code, although
granted it's mostly just repeating the same simple patterns.  It
would be nice for some of this code to converge a bit.  This
affects new messages, typing indicators, search filters, etc.,
and some endpoints have strange legacy stuff like supporting
JSON-encoded lists, so it's not trivial to clean this up.

Tweaked by tabbott to add some additional tests.
2018-10-12 10:18:30 -07:00
Steve Howell 9f2aad55b5 hipchat import: Handle users without avatars. 2018-10-12 07:03:25 -04:00
Steve Howell 51bd36e448 tests: Add coverage to get_service_interface_class(). 2018-10-11 16:12:07 -07:00
Steve Howell 8379aeee15 outgoing bots: Fix header for generic servers.
For our bots that use GenericOutgoingWebhookService
(which are basically Zulip style bots), we now
include a "content-type" header of "application/json".

We accomplish this by having the service classes
implement their own custom method called
`send_data_to_server`. For the Slack-related
code, we just extracted code from `do_rest_call`,
and then for the Zulip-related code, we added
a `headers` parameter.
2018-10-11 16:12:07 -07:00
Steve Howell 8226e13e9c bot tests: Replace use of MockServiceHandler.
We'll just use a real class here, since the service
handlers are pretty lightweight and just munge data.
2018-10-11 16:12:07 -07:00
Steve Howell 8f74d99b6c Remove stubs in OutgoingWebhookServiceInterface.
If we omit methods in subclasses, they're likely to
be caught by linters or unit tests, and even if they
aren't, raising NotImplementedError doesn't actually
prevent user problems.

I've been fighting these in refactoring, and it's
just been a bunch of busy work, plus comments are
highly likely to bitrot.
2018-10-11 16:12:07 -07:00
Steve Howell 31597cf33e Remove timeout parameter in do_rest_call().
Nobody was setting it.
2018-10-11 16:12:07 -07:00
Steve Howell 69ee84bb14 refactor: Extract build_bot_request().
This fixes a couple things:

    * process_event() is a pretty vague name
    * returning tuples should generally be avoided
    * we were producing the same REST parameters in both
      subclasses
    * relative_url_path was always blank
    * request_kwargs was always empty

Now process_event() is called build_bot_request(),
and it only returns request data,
not a tuple of `rest_operation` and `request_data`.

By no longer returning `rest_operation`, there are
fewer moving parts.  We just have `do_rest_call` make
a POST call.
2018-10-11 16:12:07 -07:00
Steve Howell 16eff75e49 refactor: Simplify how we use base_url.
Before this change, we instantiated base_url into a superclass
of subclasses that returned base_url into a dictionary that
gets returned to our caller.

Now we just pull base_url out of service when we need to make
the REST call.
2018-10-11 16:12:07 -07:00
Steve Howell b89a94f730 Improve errors when we can't connect to a bot server.
We don't overwhelm people with error info when bots
fail to connect or time out.
2018-10-11 16:12:07 -07:00
Steve Howell 3790c469e9 outgoing bots: Report JSON errors to users.
We should arguably report these to bot owners
as well, but this is at least an improvement
over having the server crash.
2018-10-11 16:12:07 -07:00
Steve Howell df4b665658 refactor: Parse JSON from bots in one place.
We move the JSON parsing step into the
higher level function: process_success_response().

In the unlikely event that we'll start integrating
with a solution that doesn't use JSON, we can deal
with that, and for now doing the parsing in one
place will help us make error reporting more
consistent.

In a subsequent commit we'll introduce better
error handling for malformed JSON.
2018-10-11 16:12:07 -07:00
Steve Howell 229dd5d861 outgoing webhooks: Get rid of "Success!" prefix.
The earlier code here, if it got a payload with
"response_string" as a key, would prefix the
corresponding value with "Success!".  We just
want the bot to set its own content.

The code is reorganized here so that process_success()
always produces a value keyed by "content" from
incoming data, and then process_success_response()
doesn't do any fancy munging of the data.
2018-10-11 16:12:07 -07:00
Tim Abbott 0a751567a3 upload: Fix missing mypy return type annotation. 2018-10-11 16:11:20 -07:00
Joshua Pan 971cb18cb3 user_settings: Compare new stripped email with old email.
We weren't comparing the newly stripped email with the current
old email, thus adding spaces around an email would result in
an error.
2018-10-11 15:55:32 -07:00
Tim Abbott 8cf104b643 avatar: Allow API authentication for /avatar/ routes.
This makes it feasibly for the mobile apps to correctly render user
avatars generated by the `!avatar()` syntax.
2018-10-11 15:52:29 -07:00
Aditya Bansal 3164f1a9a4 avatar: Rename user_profile to avatar_user_profile.
This is a preparatory commit for upcoming changes to move
/avatar/ to be a logged in or API accessible endpoint.
Basically we rename this variable because the new name is more
appropriate in the situation. Also user_profile will be used to
hold the user_profile of person accessing the endpoint in coming up
commit.
2018-10-11 15:50:37 -07:00
Aditya Bansal 6893f52ad9 thumbnails: Instruct thumbor to sharpen thumbnailed images.
Fixes: #10218.
2018-10-11 15:44:47 -07:00
Aditya Bansal 6e433186a1 thumbnails: Change thumbnail size to be 300px.
Fixes: #10219.
2018-10-11 15:44:47 -07:00
Vishnu Ks 962d72b58b retention: move_messages_to_archive should accept multiple message ids.
This will speed up the scrub realm management command. Calling the
function with a single message_id in a loop was extremely inefficient.
2018-10-11 15:31:12 -07:00
Vishnu Ks 6972de21be management: Add command to scrub a realm of personal data. 2018-10-11 15:30:26 -07:00
Vishnu Ks 2f5a5c2c49 test_classes: Create lear_user helper function. 2018-10-11 15:30:26 -07:00
Vishnu Ks 5bdadc8061 upload: Create function to delete avatar image. 2018-10-11 15:30:26 -07:00
Vishnu Ks 1d94fc7dbb upload: Extract function to delete file. 2018-10-11 15:30:26 -07:00
Steve Howell 4b82326376 hipchat import: Support guest users.
We simplify the code for is_realm_admin
and set is_guest as well.

I verified that build_user() is not used
by Slack/Gitter, so the extra argument there
should be fine.

Fixes #10639
2018-10-11 15:28:58 -07:00
Vishnu Ks 6aa4b64dc0 emails: Don't log emails while running test suite.
Modified the tests to ensure 100% coverage.
2018-10-11 15:12:08 -07:00
Vishnu Ks d8c19cb003 models: Move billing models from zilencer to corporate. 2018-10-11 14:54:29 -07:00
Tim Abbott c57c4cf703 notifications: Fix push notifications with multiple realms.
Previously, Zulip did not correctly handle the case of a mobile device
being registered with a push device token being registered for
multiple accounts on the same server (which is a common case on
zulipchat.com).  This was because our database `unique` and
`unique_together` indexes incorrectly enforced the token being unique
on a given server, rather than unique for a given user_id.

We fix this gap, and at the same time remove unnecessary (and
incorrectly racey) logic deleting and recreating the tokens in the
appropriate tables.

There's still an open mobile app bug causing repeated re-registrations
in a loop, but this should fix the fact that the relevant mobile bug
causes the server to 500.

Follow-up work that may be of value includes:
* Removing `ios_app_id`, which may not have much purpose.
* Renaming `last_updated` to `data_created`, since that's what it is now.

But none of those are critical to solving the actual bug here.

Fixes #8841.
2018-10-10 16:15:52 -07:00
Rishi Gupta bf22eefede api docs: Move integration-docs-guide to docs/. 2018-10-09 20:28:44 -07:00
Steve Howell c0df049a18 Allow "content" from outgoing webhooks.
We now allow outgoing webhooks to provide us a
"content" field, which is probably a more guessable
name than "response_string", particularly for folks
that use our other bot-related APIs.  And we don't
modify content as we do response_string, i.e. no
"Success!" prefix.

If we're not too concerned about backward compatibility,
we can do a subsequent commit that makes "content"
and "response_string" true synonyms and get rid of
the "Success!" prefix, which was probably accidental
to begin with.
2018-10-09 15:56:24 -07:00
Steve Howell 6c4343c86d refactor: Clean up send_response_message().
This commit starts by changing the third
argument of send_response_message to be a Dict
instead of a string, so that the data can be more
structured going forward.

That change makes the 2nd/3rd parameters both be
dicts, so to be defensive, I now have all the callers
pass in explicit keyword names.  And then I rename
message to message_info, so that the callers have
more clear code.

And that changes the implementation inside of
send_response_message() a bit.

Sorry this commit is a bit coarse, but the intermediate
commits would have been kind of ugly, too.

At the end of the day, it's pretty simple:

    bot_id: never changed
    message_info: just renamed from message
    response_data: is a Dict with the key of "content"

And the innards of send_response_message() are basically
simply dictionary lookups and function calls.
2018-10-09 15:56:24 -07:00
Steve Howell 4956107c53 refactor: Simplify return type for process_success().
There's no reason to return a failure message in
process_success(), since it's implied to be part of
the success codepath.  I didn't look at the full history
of how the strange API evolved, but the second element
of the tuple was clearly noise by the time I got here.
Neither of the subclasses ever set it, and none of the
consumers used it.
2018-10-09 15:56:24 -07:00
Steve Howell f2dd218331 refactor: Inline succeed_with_message().
This two-line function wasn't really carrying its
weight, and it just made it harder to refactor the
overall codepath.

Eliminating the function forces us to mock at a slightly
deeper level, which is probably a good thing for what
the test intends to do.  The deeper mock still verifies that
we're sending the message (good) without digging into
all the details of how we send it (good).

Note that we will still keep around the similarly named
`fail_with_message` helper, which is a lot more useful.
(The succeed/fail scenarios aren't really symmetric here.
For success, there are fewer codepaths that do more complex
things, whereas we have lots and lots of failure codepaths
that all do the same simple thing of replying with a canned
message.)
2018-10-09 15:56:24 -07:00
Steve Howell fa505a1af1 refactor: Have process_success return structured data.
Before this change subclasses of OutgoingWebhookServiceInterface
would return a raw string as the first element of its return
tuple in process_success().  This is not a very flexible
design, as it prevents the bot from passing extra data like
`widget_content`.

It's also possible in the future that we'll want to let outgoing
bots reply directly to senders who mention them on streams, and
again the original design was overly constrained for that.

This commit does not actually change any functionality yet.
2018-10-09 15:56:24 -07:00
Steve Howell 3bb8cbe0c7 minor: Dedup check_send_message() call. 2018-10-09 15:56:24 -07:00
Steve Howell e641036911 minor: Rename var to message_type. 2018-10-09 15:56:24 -07:00
Steve Howell b61612d50b minor: De-duplicate code for client. 2018-10-09 15:56:24 -07:00
Callum Fraser 383f1633e6 api: Add limit of total messages available per request of 5000.
Tweaked by tabbott to use a declared constant rather than just use
5000 in multiple places; this also means we can change the count
without updating translations.

Fixes #10446.
2018-10-09 15:43:21 -07:00
Rishi Gupta 7956c57448 user docs: Add import-from-hipchat. 2018-10-09 15:23:07 -07:00
Lyla Fischer 3c51328763 user docs: Remove icon-vector for font-awesome. 2018-10-09 14:16:16 -07:00
Pragati Agrawal 749e034863 org settings: Rename "Filter settings" to "Linkifiers".
Fixes: #10551
2018-10-09 08:50:45 -07:00
Lenny Jagielski 4fa4ca53c2 webhooks: Add Netlify integration.
Fixes: #10169.
2018-10-05 17:04:40 -07:00
Eeshan Garg ba929508e2 webhooks/ifttt: Get test coverage up to 100%.
IFTTT allows custom templating for their payloads, so the onus is
on the user to ensure that their custom templates conform to the
expectations outlined in our IFTTT webhook docs. For that reason,
these payloads weren't generated, but were manually edited.
2018-10-04 12:16:06 -07:00
Eeshan Garg 316f9aa78b webhooks/hellosign: Rewrite the integration from scratch.
After discovering a couple of bugs, I decided to thoroughly test
and rewrite this integration from scratch. The older code wasn't
generating coherent messages.

This also commit gets this integration up to 100% test coverage.
2018-10-04 12:16:06 -07:00
Eeshan Garg 5039f6dfb5 webhooks/gitlab: Get test coverage up to 100%.
Test coverage was improved by removing an unused function and
removing some code (written by me) that was actually handling
Test Hook event types incorrectly.
2018-10-04 12:16:06 -07:00
Eeshan Garg 6e2e2b9125 webhooks/github: Test commit status payloads with target_url.
It was a painful amount of work to generate the actual payload.
Since the only difference was a small build URL, I manually
edited the payload and used that for testing.

This commit gets our GitHub webhook up to 100% test coverage.
2018-10-04 12:16:06 -07:00
Eeshan Garg 42e3410df1 webhooks/github: Improve logic for page build messages.
Some of the page build message code had insufficient test coverage.
I looked at generating the payloads that would allow me to test
the lines of code in question, but it was too much work to
generate the payloads and this seemed like a vague event anyway.

So I just rewrote the logic so that the lines missing
coverage are implicitly covered.
2018-10-04 12:16:06 -07:00
Eeshan Garg 4f34ee2e6b webhooks/github: Test and improve messages for issue comment deletion.
This is a part of our efforts to get this webhook's coverage
up to 100%.

Note that apart from just testing an uncovered line of code, this
commit also fixes a minor bug in the code for messages about issue
comment deletion and editing.
2018-10-04 12:16:06 -07:00
Eeshan Garg 9c75bd3409 webhooks/github: Test pull requests with preassigned assignees.
This is a part of our efforts to get this webhook up to 100%
test coverage.
2018-10-04 12:16:06 -07:00
Eeshan Garg b40aec3a09 webhooks/freshdesk: Improve test coverage.
Note that Freshdesk allows custom templating for outgoing payloads
in their webhook UI. Therefore, the payloads added in this commit
did not have to be official payloads from Freshdesk.
2018-10-04 12:16:06 -07:00
Eeshan Garg 9e625b881b webhooks/bitbucket2: Improve commit_status_changed type messages.
Instead of just referring to the commit with the raw URL, we
should use the commit ID as the text of the hyperlink.

Note that in commit_status_changed type messages, the name of the
commit isn't available.
2018-10-04 12:16:06 -07:00
Eeshan Garg b22254835e webhooks/bitbucket2: Fix broken commit URL.
The function that generates the body of the commit_status_changed
event messages generated an invalid commit URL.

Most likely, we missed this because this event type is fairly
vague and it is possible it was never tested by users much,
if at all.
2018-10-04 12:16:06 -07:00
Eeshan Garg d8101ca139 webhooks/bitbucket2: Improve test coverage.
The lack of coverage was due to:

* An unused function that was never used anywhere.
* get_commit_status_changed_body was using a regex where it didn't
  really need to use one. And there was an if statement that
  assumed that the payload might NOT contain the URL to the commit.
  However, I checked the payload and there shouldn't be any instances
  where a commit event is generated but there is no URL to the commit.
* get_push_tag_body had an `else` condition that really can't happen
  in any payload. I verified this by checking the BitBucket webhook
  docs.
2018-10-04 12:16:06 -07:00
Eeshan Garg 185a023745 webhooks/beanstalk: Improve test coverage.
We shouldn't just ignore exceptions when encoding the incoming
auth credentials. Even if the incoming credentials are properly
encoded, it is better to know when that is the case or if
something else fails.
2018-10-04 12:16:06 -07:00
Steve Howell 4da664817b hipchat conversion: Add messages. 2018-10-02 16:55:16 -07:00
Steve Howell f296d60dad hipchat conversion: Add emoji support. 2018-10-02 16:55:16 -07:00
Steve Howell 9518b1344a hipchat conversion: Process avatars.
This processes the avatar payloads that we
get in users.json.
2018-10-02 16:55:16 -07:00
Steve Howell c0f15c3860 hipchat conversion: Include deactivated users/streams.
We now include deleted/deactivated data from the old system.
2018-10-02 16:55:16 -07:00
Steve Howell faea26783b Create convert_hipchat_data.
This is a very early version of a tool to convert Hipchat
tar files into data files that can be used by the Zulip
import process.

We include the most fundamental entities--users and
streams.  Customers who don't care about past messages
or customizations could start an instance off of this
and start communicating.

Of course, there are a lot of things missing in the
initial version:

    * messages!
    * file assets -- avatars, emojis, attachments
    * probably lots of other minor things

We currently ignore any incoming dates from Hipchat data
and just use the current time.  This is consistent with
other imports.

We also don't have any docs yet, although the process
will be extremely similar to the "Slack" process:

    https://zulipchat.com/help/import-from-slack

Also, there's a comment at the top of convert_hipchat_data.py
that describes how to test this in dev mode.

I tested this by following the steps in the comment above.
The users just "show up" in /devlogin, so that's nice, and
you can send messages to other users.  To verify the stream
data you have to go into the gear menu and click on "All
Streams", then you can subscribe and send a message.

Production users will need to get new passwords and
re-subscribe to streams.  We will probably auto-subscribe
all users to public streams.
2018-10-02 16:55:16 -07:00
Steve Howell 329154da32 import: Speed up create_subscription_events().
The code was needlessly querying the DB to get full
objects for entities where we only needed user_id,
realm_id, and stream_id.

With my test data of ~1000 records this sped up the
function from ~8s to ~0.5s.  The speedup would probably
be even more for larger data sets.
2018-10-02 16:55:16 -07:00
Greg Price 82a4dd832f search: Always encode match_subject as HTML.
The `match_subject` field is supposed to contain HTML; that's how
the highlighting is done.  But the `subject` field is plain text --
it must be encoded if we want corresponding HTML.

Of the three places the `match_subject` field is populated -- two
here in messages_in_narrow_backend, one in get_messages_backend --
two of them already do this correctly, via get_search_fields.
Fix the remaining one, where in a `/messages/matches_narrow` query
we populate `matches_subject` even if the query didn't involve a
full-text search.

This doesn't affect the webapp, which ignores `match_subject` unless
it knows it did a full-text search; nor the mobile app, which
doesn't use `/messages/matches_narrow` at all.
2018-10-02 12:07:54 -07:00
Vishnu Ks d8ddb7f69b settings: Remove reminder-bot from REALM_INTERNAL_BOTS. 2018-10-02 11:12:07 -07:00
Tim Abbott 510afd0c93 migrations: Disable atomic for delivery_email migration.
I'm not sure theoretically why this should be required only for some
installations, but these articles all suggest the root problem is
doing these two migrations together atomically (creating the field and
setting a value for it), so the right answer is to declare the
migration as not atomic:

https://stackoverflow.com/questions/12838111/django-db-migrations-cannot-alter-table-because-it-has-pending-trigger-events
https://confluence.atlassian.com/confkb/upgrade-failed-with-the-error-message-error-cannot-alter-table-content-because-it-has-pending-trigger-events-747606853.html
2018-09-28 14:14:20 -07:00
Tim Abbott a943e55863 docs: Move LDAP documentation out of plug-and-play section.
Also update the links to this section.
2018-09-27 13:38:14 -07:00
Eeshan Garg 8e6fba7f22 webhooks/trello: Fix indentation and numbering. 2018-09-27 10:04:40 -07:00
Rishi Gupta 7f16d27015 api docs: Update api-keys. 2018-09-25 21:02:58 -07:00
Rishi Gupta cb116be60c integrations: Update trello doc.
The main setup text for an integration should always be targeted at
non-technical users.
2018-09-21 19:20:23 -07:00
Harshit Bansal 20aee7b116 emoji: Change fallback emojiset for text emojiset.
Now that we have added back support for `google-blob` emojiset,
changing the fallback emojiset for `text` emojiset to `google-blob`
make sense.
2018-09-21 18:34:12 -07:00
Adam Birds 18a4239d7e integration: Add AppVeyor webhook. 2018-09-21 17:51:34 -07:00
Shubham Padia 6bfa29b8e6 notifications: Fix soft-deactivated users don't get push notifications.
Fixes the urgent part of #10397.

It was discovered that soft-deactivated users don't get mobile push
notifications for messages on private streams that they have configured
to send push notifications.

Reason: `handle_push_notification` calls `access_message`, and that
logic assumes that a user who is a recipient of a message has an
associated UserMessage row. Those UserMessage rows are created
lazily for soft-deactivated users, so they might not exist (yet)
until the user comes back.

Solution: Ensure that userMessage row is created for
stream_push_user_ids and stream_email_user_ids in create_user_messages.
2018-09-21 12:06:18 -07:00
Tim Abbott d152b84ccc trello: Clarify more what's going on with Trello's API.
I think this makes it a lot more clear what is happening here.
2018-09-21 11:24:13 -07:00
Eeshan Garg 0012e76123 webhooks/trello/doc: Require users to download Trello script.
We now have a standalone zulip_trello.py script that can be run
on Python2 and Python3 without having to install the `zulip`
package.
2018-09-21 11:12:13 -07:00
Adam Birds bd6a5ed7af integrations: Add Ansible Tower Webhook. 2018-09-21 11:05:36 -07:00
Tim Abbott a0451b692f import: Move zerver_client import before realm import.
This table is independent of the realm/stream table dance, and moving
it here helps makes the flow read more clearly.
2018-09-21 10:58:24 -07:00
Rishi Gupta b470cef864 import: Set Realm.plan_type to SELF_HOSTED on import.
Tweaked by tabbott to avoid an unnecessary .save().
2018-09-21 10:57:22 -07:00
Robert Spier f4c82a1153 emails: Add detection for ChromeOS User-Agents. 2018-09-21 10:41:05 -07:00
Tim Abbott 75376a3fc5 email_mirror: Limit message length using defined constants.
Previously, we had the somewhat arbitrary limit of 2K characters
(which some users complained about), as well as the constant 60 for
the topic.
2018-09-21 10:39:57 -07:00
Tim Abbott feee76eb23 export: Fix exporting files with S3 upload backend.
At some point as part of the process of supporting renumbering data,
we changed the structure of our file uploads to expect `path` to match
`s3_path`, with both having the relative path within the overall
hierarchy (including the realm ID).  This change updates the more
rarely-used S3 export code path to use that model, fixing a crash when
messages reference an Attachment object with a rewritten path_id.
2018-09-20 20:14:19 -07:00
Tim Abbott e2bd03365e import: Fix handling of recipient IDs for welcome bot.
If any user had sent the reply to the welcome bot recommended by our
tutorial, then the Zulip export/import process didn't work properly,
because we weren't including (and then remapping) the recipient ID for
sending PMs to the cross-realm bots.  This commit fixes that gap, by
recording the necessary data on the export side, and doing the
appropriate remapping on the import side.
2018-09-20 17:55:17 -07:00
Tim Abbott c9189439de import: Handle signup_notifications_stream_id.
Previously, our realm import logic only did the special remapping
logic for the original notifications_stream_id; when we added the new
signup_notifications_stream_id field, we neglected to handle it in the
same way.
2018-09-20 17:41:55 -07:00
Tim Abbott 165078b484 queue_processors: Fix bug in handling removed push notifications.
Apparently, we were falling through to the "add" case after correctly
processing the "remove" case, throwing a 500.
2018-09-20 17:36:54 -07:00
Eeshan Garg 0817905480 api docs: Use Markdown extension for tabbed sections. 2018-09-18 13:49:34 -07:00
Eeshan Garg 2443919a7e user docs: Use tabbed instructions on desktop-app-install-guide.
Note that the correct tab is automatically activated depending on
the user's OS.
2018-09-18 13:49:34 -07:00
Eeshan Garg ecd4f821be user docs: Automatically activate correct tab for OS-specific instructions. 2018-09-18 13:49:34 -07:00
Eeshan Garg 4f366daec0 markdown: Add extension for creating tabbed sections on /help and /api. 2018-09-18 13:49:34 -07:00
Rishi Gupta bad4a2e8ca user docs: Update invite-a-friend-to-zulip. 2018-09-16 08:24:28 -07:00
Rishi Gupta 44ea5f8492 user docs: Update message-a-stream-by-email.
Note we're no longer using subscriptions_html in the help docs, so no need
to test for it. There is already a test for subscriptions_html in
IntegrationTest.
2018-09-15 23:57:22 -07:00
Rishi Gupta c050593752 user docs: Add stream relative links. 2018-09-15 23:57:22 -07:00
Rishi Gupta a87ed67695 user docs: Add relative links for gear menu items. 2018-09-15 23:47:45 -07:00
Rishi Gupta 7acb4e9165 portico: Update text for reset_emailed.html.
The current text may have made sense with different styling, but it feels
harsh now.
2018-09-15 18:45:10 -07:00
Eeshan Garg 7a755ff520 models: Add get_user_by_id_in_realm_including_cross_realm().
Ongoing efforts surrounding #9474 and #10360 are contingent upon
such a function being available.
2018-09-12 11:43:24 -07:00
Steve Howell 67e2fd5900 Minimize race conditions for reading zulip.yaml.
In the event that two processes are racing to be the
first to load data from zulip.yaml, we now make the
race scenario be duplicated effort instead of having
the second racer get an attribute error on `data`.

We do this by declaring victory only after setting
`data`.  "Declaring victory" in this case is a matter
of setting `last_update`.

We are still possibly vulnerable to corrupted data
here, so we should investigate a mutex, or just
read the data on every call (but it's strangely
expensive, almost 3.5s on my instance), or converting
the YAML to code before launching the server.
2018-09-12 11:07:20 -07:00
Eeshan Garg 95ed9dc701 webhooks/trello/doc.md: Add context for why we need to use a script. 2018-09-12 07:24:22 -04:00
Vishnu Ks 3e86fa84ea billing: Rename RealmAuditLog.STRIPE_CARD_ADDED to STRIPE_CARD_CHANGED. 2018-09-11 00:08:38 -07:00
Rhea Parekh 7822ef38c2 import: Change absolute path of downloaded avatars in records.json to relative path. 2018-09-09 09:18:18 -04:00
Vishnu Ks 169de2f9cb billing: Include link to /plans and /billing in gear menu. 2018-09-07 17:30:26 -07:00
Rishi Gupta 2a0523cbe5 billing: Hide /plans if Realm.plan_type=SELF_HOSTED. 2018-09-05 13:44:35 -07:00
Vishnu Ks c913eafdf4 portico: Change buttons on /plans to reflect current plan. 2018-09-05 13:44:35 -07:00
Vishnu Ks 9489ce0efc plans: Redirect to login if /plans is accessed from realm subdomain. 2018-09-05 13:44:35 -07:00
Rishi Gupta 31ed4492ce billing: Add backend for downgrading. 2018-08-31 17:49:34 -07:00
Tim Abbott 42f49d19b3 emoji: Set default emoji to google-blob for existing users too.
This fixes an inconsistent test failure with test_users.py (that
depended on the ordering between this migration and the creation of
test database users like hamlet).
2018-08-31 15:45:33 -07:00
Yago González ea10f5eb2c api docs: Document GET /users/<email>/presence.
Tweaked by tabbott to describe more clearly what this is for.
2018-08-31 15:15:54 -07:00
Rohitt Vashishtha 2864ce552b bugdown: Add @user|id syntax to support users with same name. 2018-08-31 14:16:47 -07:00
Rohitt Vashishtha bc37800ad5 bugdown: Store users with same full_names in mention_data.
We start by stripping the ids in front of the name before the database
lookup. This has the advantage of not mentioning anyone if an incorrect
user id and full name combination is specified, as well as not having
the query the database twice, once by fullname and next by id.

Previously, we were storing only the most recent person with the same
full name as others; this commit adds new keys to the dict such that
simply looking by name would get you the newest user with this name,
and the get_user_by_id function can index the remaining users.
2018-08-31 14:16:47 -07:00
Harshit Bansal e7311cdf5d emoji: Bring back the feature of changing emojisets.
This is largely inspired by requests from people not liking the
Google's new emojiset. A lot of people were requesting to revert
back to old blobs emojiset so we are re-enabling this feature
after making relevant infrastructure changes for supporting google's
old blob emojiset and re-adding support for twitter emojiset.

Fixes: #10158.
2018-08-31 11:52:24 -07:00
Shubham Padia 69bfa8c432 auth: Use different defaults for name and email for fakeldap.
Fixes part of #10297.
Use FAKE_LDAP_NUM_USERS which specifies the number of LDAP users
instead of FAKE_LDAP_EXTRA_USERS which specified the number of
extra users.
2018-08-31 17:07:02 +05:30
Yago González 78f85ef960 api docs: Document the GET /messages/<message_id>/history endpoint. 2018-08-28 17:33:02 -07:00
Yago González 5c6f381f32 api docs: Document the DELETE /messages/<message_id> endpoint. 2018-08-28 17:26:49 -07:00
Yago González 9575f1b51f api docs: Document the GET /messages/<message_id> endpoint. 2018-08-28 17:22:28 -07:00
Yago González ab164ba740 api docs: Document the GET /messages endpoint. 2018-08-28 17:17:46 -07:00
Vishnu Ks 7b307fa58f billing: Set max_invites to 3000 for Premium organizations.
This is meant to be effectively infinite for legitimate use.
2018-08-28 16:03:34 -07:00
Pragati Agrawal ba4e4e38ae user settings: Add setting to make login notifications optional.
This adds a feature in the "Notification" section of "Settings" tab,
which lets user enable or disable login emails notification.

Tweaked by tabbott to simplify the test.

Fixes: #5795, progress towards #5854.
2018-08-28 14:02:31 -07:00
Tim Abbott 54e90deda8 notifications: Handle APNS "Unregistered" errors properly.
Apparently, the APNS library we're using is inconsistent about the
format of its result entries; some are strings while others are
tuples.
2018-08-28 12:12:46 -07:00
Yago González c36cf95dc8 api docs: Document GET /server_settings. 2018-08-27 17:45:50 -07:00
Roman Godov a2f407c11b api: Add an endpoint for listing the user groups in realm.
Adds endpoint for listing the user groups in realm.
Provides "description", "members", "name" and "id" for each user group.

Fixes #10298
2018-08-27 16:56:23 -07:00
Lyla Fischer 1efcdfdb5c templates: Merge Help and API doc main into documentation_main.
With minor fixes by eeshangarg.
2018-08-27 20:19:24 -02:30
Yago González 8b141e6dbc muting: Correct agrammatical error message.
The error message displayed when unmuting a topic that wasn't previously
muted wasn't properly formulated.
2018-08-27 14:12:03 -07:00
Vishnu Ks dcb7b15069 billing: Use UserProfile.is_billing_admin instead of Customer.billing_user. 2018-08-27 12:35:46 -07:00
Yago González 54464feda7 api docs: Document the PATCH /users/me/subscriptions/muted_topics endpoint. 2018-08-26 23:10:21 -07:00
Vishnu Ks 62b93d551a signup: Fix broken resend email link in realm creation flow.
Also use name for selecting form in casper tests
as form with action=new is present in both /new
and /accounts/new/send_confirm/ which breaks
test in CircleCI as
waitWhileVisible('form[action^="/new/"]) never stops
waiting.
2018-08-26 22:54:05 -07:00
Vishnu Ks d2e4417a72 urls: Separate endpoint for signup and new realm email confirm.
This is preparation for the next commit.
2018-08-26 22:53:57 -07:00
Harshit Bansal ec2ebd1cc9 build_emoji: Change sprite CSS files to use hyphens in names.
This is more consistent with our other names in emoji infra.
2018-08-26 22:16:23 -07:00
Steve Howell ca272ef49d tests: Test how we handle extra data for zform messages. 2018-08-24 10:00:25 -07:00
Steve Howell 9e8930f6de tests: Test get_widget_data() helper.
We also remove some unreachable code.  Calling
split() always returns at least one token, even
if it's just the empty string.  This is tested
directly on this commit, plus messages with
empty content get rejected pretty early in
the execution path.
2018-08-24 10:00:25 -07:00
Steve Howell f3f30d29f9 tests: Add tests for /tictactoe message.
The /tictactoe widget exercises the codepath where
we don't actually extract `extra_data` from the message.
2018-08-24 10:00:25 -07:00
Steve Howell 922b1683fc tests: Add test for /poll message.
This goes through the /messages endpoint
to get more full-stack testing, and it
focuses on how SubMessage rows get
created.
2018-08-24 10:00:25 -07:00
Tim Abbott 69b2315a6e password reset: Deduplicate code for reset URL generation. 2018-08-23 12:06:02 -07:00
Vishnu Ks 0f628dc4aa management: Fix send_password_reset_email command.
This had been broken in a recent refactoring of the password-reset
code path.

Fixes #10379.
2018-08-23 12:05:49 -07:00
Tim Abbott c313a939f7 custom profile fields: Cleanup event generation logic.
In my opinion, this makes the code somewhat more readable.
2018-08-22 22:45:08 -07:00
Yashashvi Dave d5153bd136 events: Convert custom user field value to json object on update event.
In user type custom field, field value is list of user ids. We weren't
converting list to json object in update event payload. This throws
error in frontend, cause we store stringify representation of custom
field value. Therefore, after update event is recieved field-value-
type gets updated to array from string which throws json parsing error.
2018-08-22 22:45:08 -07:00
Shubham Dhama e784c95d97 guests: Prevent guests from sending to unsubscribed public streams.
This matches the overall security model of these users only having
access to streams they are subscribed to.
2018-08-22 17:53:42 -07:00
Shubham Dhama e70cf3bd67 emails/password_reset: Change text for listing other active accounts.
The main benefit of this change is that it reduces the amount of total
string that we might need to tag for translation.

Fixes: #10323.
2018-08-22 17:49:40 -07:00
Shubham Dhama aa8b3d2beb emails/password_reset: Change template string for deactivated users. 2018-08-22 17:43:23 -07:00
Shubham Dhama 491bd6d2c9 emails/password_reset: Refactor and replace no_account_in_realm.
In place of no_account_in_realm we will use its negation
active_account_in_realm as a part of refactoring this template.
2018-08-22 17:43:23 -07:00
Yago González e7c7b19507 api docs: Document POST /realm/filters. 2018-08-22 17:42:14 -07:00
Yago González df7234f3a6 api docs: Escape HTML in the examples.
Having HTML (or HTML-like) content in the examples was making parts of
the content invisible, since the browser identified them as HTML tags
rather than verbose text.
2018-08-22 17:19:09 -07:00
Yago González c70d051031 test-api: Add function for server & realm tests.
There are some endpoints that don't fall into the currently available
categories, so this new function will be used for calling the tests for
server and realm-related endpoints.
2018-08-22 17:19:09 -07:00
Steve Howell 44756a326d tests: Test duplicate default stream/groups.
This also gets us to 100% line coverage
for actions.py.
2018-08-22 16:52:30 -07:00
Steve Howell fe6680c316 refactor: Flatten code in check_invite_limit().
Using early-exit here allows us to more easily
comment why there are certain exemptions to
this logic.

We also only require callers to pass in realm,
not the whole user object.
2018-08-22 16:52:30 -07:00
Steve Howell cac9a02d6d tests: Add coverage for check_invite_limit().
The function being tested here was kind of an
emergency response to some spam attacks.  It
works for a pretty specific set of circumstances,
so it requires a lot of setup.

We may eliminate this function as we improve
our realm "plan types", and if that happens, we
can either eliminate this test or repurpose it.
2018-08-22 16:52:30 -07:00
Steve Howell 4318f75718 Remove `code` from JsonableError.__init__.
Since this class was built, folks have always chosen
to subclass JsonableError for situations where
the default of ErrorCode.BAD_REQUEST is insufficient.

So now we simplify the use cases, which also gets
us 100% coverage on this core module.
2018-08-22 16:51:40 -07:00
Tim Abbott 2e6aaf3215 actions: Use better query for active_mobile_push_notification. 2018-08-21 15:28:05 -07:00
Tim Abbott 887d20795f message flags: Add where_starred helper and use it.
The previous query ended up doing a scan of all a user's UserMessage
rows, not just the ones tracked in the `starred` index.
2018-08-21 15:28:04 -07:00
Joshua Pan 79e2266d6e models: Create starred_message_counts boolean field. 2018-08-21 13:42:22 -07:00
Joshua Pan 83d8d662dc events: Query starred message ids in fetch_initial_state_data. 2018-08-21 13:01:41 -07:00
Shubham Padia 34cb0eae29 tests: Stop using fixtures to test generate_dev_ldap_dir.
The output of generate_dev_ldap_dir was being tested against the fixture
located at zerver/tests/fixtures/ldap_dir.json. This didn't make much sense
as generate_dev_ldap_dir was itself used by developers to generate/update
the fixtures. Instead, test_generate_dev_ldap_dir checks the structure of
the dict returned by generate_dev_ldap_dir. The structure is checked by
regex checks, checking whether the dict contains some keys or not, etc.
2018-08-21 12:55:36 -07:00
Vishnu Ks 9bb338be11 models: Add plan_type to Realm. 2018-08-21 12:39:06 -07:00
Yashashvi Dave d5f3c9b3bc custom profile field: Rename field types in UI. 2018-08-21 11:37:51 -07:00
Yashashvi Dave 6e65235a6d zerver/lib/events.py: Add FIELD_TYPE_CHOICES_DICT to page_params.
This commit add FIELD_TYPE_CHOICES_DICT to page_params and replace
FIELD_TYPE_CHOICES.

FIELD_TYPE_CHOICES_DICT includes all field types with keyword, id
and display name. Using this field-type-dict, we can access field
type information by it's keyword, and remove all static use of
field-type'a name or id in frontend.
This commit also modifies functions in js where this page_params
field-types is used.
2018-08-21 11:37:51 -07:00
Yashashvi Dave 621a5cdc35 zerver/models.py: Modify FIELD_TYPE_DATA, add keyword for field type.
This commit modifies FIELD_TYPE_DATA dict in `CustomProfileField`
model to store keyword of field types. And create new dict
FIELD_TYPE_CHOICES_DICT to store all field type information
by field type keyword, i.e. id, name.

This is preparatory commit to remove all static use of field
types in frontend and access field type with keyword instead
of display name.
2018-08-21 11:37:51 -07:00
Steve Howell a6bc3886e6 refactor: Extract send_peer_remove_event().
This prevents leaking some variables into an already
cluttered function.

We also add test coverage for what's now an
early-exit condition in the new function--we exempt
public MIT streams from these events.
2018-08-21 11:23:40 -07:00
Steve Howell 092bb6a728 tests: Refactor test_bulk_subscribe_MIT().
We extract a couple local vars to avoid
repeating ourself.

We also explictly set the stream as a zephyr stream.
2018-08-21 11:23:40 -07:00
Steve Howell e7062b77c8 tests: Test superuser can also mirror to private streams.
This extends a test that proved only what Cordelia
could do with/without super_user privileges when she
was trying to send to an unsubscribed stream as herself.

Now the test shows the same powers extend to Cordelia
when she's sending messages on behalf of a mirrored
user.
2018-08-21 11:23:40 -07:00
Steve Howell aee46bb1e9 tests: Test forging timestamps for mirrored messages. 2018-08-21 11:23:40 -07:00
Steve Howell 79fb36c599 refactor: Extract maybe_add_event() function.
This change was partially driven by a quirk in Python
where peephole optimizations make `continue` lines
appear not to be covered.

I also think it's generally a good idiom to extract
functions for loop bodies when they don't actually
accumulate values or maintain other state.  With this
commit we now prevent potential bugs for vars like
`is_stream` leaking between loop iterations.
2018-08-21 11:23:40 -07:00
Steve Howell e99c0929f0 tests: Test race handling for creating mirror users.
We simulate a race condition by mocking create_user
to actually create a user, but then raise an
IntegrityError (as if another process had actually
created the user, not our test).

I also changed the real code to use explicitly
named parameters.
2018-08-21 11:23:40 -07:00
Tim Abbott 0068a5ccd6 events: Fix can_subscribe_other_users not being set properly.
I don't understand why this didn't cause test failures in CI; this
change was clearly required and test_change_realm_property was failing
consistently for me locally.
2018-08-21 11:20:59 -07:00
Rishi Gupta 99b55b712b user docs: Shorten instructions when using relative settings links. 2018-08-20 21:26:01 -07:00
Tim Abbott 6827c42e02 migrations: Fix accounting in is_private migration.
The previous logic didn't correctly handle the case of total being 0.

Fixes #10378.
2018-08-20 16:49:08 -07:00
Abhilash Verma 0e2322a322 logging: Show timestamp in UTC in non-django production scripts.
Done in pair programming with @aero31aero.

Fixes #9678.
2018-08-20 12:52:40 -07:00
Rohitt Vashishtha 920ef2b7f7 bugdown: Add mention_data.get_user_by_id().
This will allow us to do the lookups required to support the upcoming
`@**name|id**` syntax.
2018-08-20 12:46:46 -07:00
Tim Abbott 972e1d2d8b ldap: Improve exception handling in user data sync tool.
At some point, the most likely exception here stopped being
IntegrityError.

Fixes #9218.
2018-08-20 10:40:53 -07:00
Tim Abbott 3cfb2000cc ldap: Improve error message for username/LDAP domain mismatches. 2018-08-20 10:39:34 -07:00
Rhea Parekh f70b9a3eba import: Move 'build_message' to import_util. 2018-08-19 22:27:13 -07:00
Rhea Parekh 53e9da8e1f import: Build CustomProfileField, CustomProfileFieldValue and RealmEmoji with model class. 2018-08-19 22:27:13 -07:00
Rhea Parekh d98a5925cb import: Build Reaction with the model class. 2018-08-19 22:27:13 -07:00
Rhea Parekh a5bc701181 import: Move 'build_stream' to import_util. 2018-08-19 22:27:13 -07:00
Rhea Parekh c4f8abbd30 import: Build Message with the model class. 2018-08-19 22:27:13 -07:00
Rhea Parekh 4ea7302e14 import: Add missing fields in UserProfile object.
The missing fields are checked by `full_clean()` method.
The datetime field errors are ignored as they are fixed
in the `import_realm` script. The field that are
allowed to be null are not included while building
this object.
2018-08-19 22:27:13 -07:00
Rhea Parekh 66d34b23ef import: Build Attachment with the model class. 2018-08-19 22:27:13 -07:00
Rhea Parekh 9617b1fbc5 import: Build Recipient and Subscription with model class. 2018-08-19 22:27:13 -07:00
Rhea Parekh c77763bd8e import: Move 'build_realm' to import_util. 2018-08-19 22:27:13 -07:00
Yago González aa5185fdf8 api docs: Document POST /typing. 2018-08-17 12:57:40 -07:00
Akash Nimare dab75e4990 help: Fix styling of emoticons on help pages. 2018-08-17 11:37:29 -07:00
Rishi Gupta c3a912f8b6 management commands: Add sleep_forever to lib/management. 2018-08-17 09:20:51 -07:00
zhoufeng1989 d5f752df84 tests: Tests query count when creating stream with subscriptions API.
These test cases are used to test the cost of stream creation.
Three scenarios of stream creation are covered:
1) create a public stream;
2) create a private stream;
3) create a public stream with announce=true when there is a notification stream.

Fix: #4804.
2018-08-17 08:58:38 -07:00
Eeshan Garg f476ec7fac webhooks/freshdesk/doc.md: Render example JSON correctly.
We've been getting reports from users that our Freshdesk webhook
isn't working correctly. It turns out that the issue had nothing
to do with the webhook implementation itself!

In freshdesk/doc.md, we have a JSON template we ask users to
copy/paste into a textbox in the Freshdesk UI. That JSON template
contains "{{" and "}}" characters which we escaped as Unicode
decimals to prevent clashes with Jinja2 syntax in other parts
of the same template. This worked for a while!

But thanks to the changes introduced as part of the
nested_code_blocks extension, such escaped characters were never
decoded, leading users to copy/paste the same template but with
raw escaped unicode representations of "{{" and "}}" inside. And
that eventually broke our webhook implementation.

This commit makes sure that such characters are properly "unescaped",
just for Freshdesk docs.
2018-08-17 08:55:41 -07:00
Yashashvi Dave 8ef0c6d99a custom profile field: Add length validation on field name and hint.
Add max length validation for custom field name and hint
on create-field and update-field backend views and
frontend forms.

Fixes #10284.
2018-08-17 08:37:27 -07:00
Yashashvi Dave 868362046b zerver/models.py: Change custom field name max length to 40 char. 2018-08-17 08:36:22 -07:00
Steve Howell 948190c9ff tests: Test spam prevention feature for invites.
We have code to prevent newbies on open realms
from inviting users.  This is mostly intended
to hinder spammers.  This commit just adds some
test coverage.
2018-08-17 08:14:42 -07:00
Steve Howell 0e56fecbd9 peformance: Avoid broad StreamCount queries.
Our get_streams_traffic function used to query
all streams in the StreamCount table if you
passed in `None` for `streams`.

Now we require that you pass in a list of
stream_ids.

I don't know how much work this will save
the database, since probably the bulk of
the work is aggregating.  If we need to fine
tune DB performance, we could possibly add
`realm` as an argument and add it to the filter.

What we'll immediately get, for large multi-realm
installations, is less data over the wire and
less work for the ORM.
2018-08-17 08:14:42 -07:00
Steve Howell b05f26c6d1 tests: Add test for get_streams_traffic(). 2018-08-17 08:14:42 -07:00
Steve Howell fb283ac672 tests: Test JsonableError in lookup_default_stream_groups(). 2018-08-17 08:14:42 -07:00
Steve Howell 75e297daf2 tests: Get line coverage for empty streams results.
We have defensive code in functions for combinations
of arguments and/or inputs that doesn't happen
in practice, buts this gets us some line coverage.
2018-08-17 08:14:42 -07:00
Steve Howell 96c6af1249 tests: Test include_default option for /json/streams. 2018-08-17 08:14:42 -07:00
Steve Howell 040dafbfc5 refactor: Streamline subscribed_to_stream().
The prior code uses an awkward idiom that
pre-dates the `exists()` function, and it
had an unreachable line of code.

The new version should be faster, since we
don't create a throwaway heavy Django object
or send needless data over the wire.
2018-08-17 08:14:42 -07:00
Steve Howell c41377aaab Remove _default_stream_permision_check.
This functions appears to be redundant to
`access_stream_by_name`.  The only
meaningful line of code in the function that we're
removing, the code that raises an error,
appears to be unreachable, despite reasonably
extensive tests.

The only thing the function was restricting
was that the case where the bot's owner was
unsubscribed to a private stream, which
is already locked down in
`access_stream_by_name` calls inside of
`patch_bot_backend`.

This commit increases test coverage
by removing unreachable code.

It's possible this function had
some theoretical value before we
introduced the `require_non_guest_human_user`
decorator to the `patch_bot_backend`
view, since in theory the bot itself
could have subscribed to a stream that
the owner didn't subscribe to.  Even
then it's not clear that allowing the
bot to set that as a default stream
would have been harmful, since they
can already access it.
2018-08-17 08:14:42 -07:00
Steve Howell e61565fdf6 tests: Add more testing for patching bots.
This commit adds some more tests related to patching
a bot's `default_sending_stream`.

Unfortunately, this didn't reach the code that I was
intending to add line coverage to, since checks happen
higher up in the stack, but the test code I added
is probably worthwhile.
2018-08-17 08:14:42 -07:00
Steve Howell 93e8798ac7 Extract get_last_message_id().
We want our methodology for extracting the last message
id to be consistent, particularly in terms of how we
handle edge cases.  (I'll concede that the
`bulk_remove_subscriptions` codepath never hits that
corner case in practice, but it's harmless to handle
the theoretical case.)

It may also be nice to have this function show up
clearly in profiling.

This also adds some direct testing to the function.

It's not clear to me why we don't use `latest('id')`
in the implementation, but that's outside the scope
of this commit.
2018-08-17 08:14:42 -07:00
Steve Howell 0322dcc6cf tests: Add direct coverage to validate_email().
These direct tests add some line coverage.
2018-08-17 08:14:42 -07:00
Steve Howell 20c4d357cc tests: Add test for sending group mentions in messages. 2018-08-17 08:14:42 -07:00
Puneeth Chaganti 126442a0a8 help: Generate emoticon translations table dynamically.
Closes #8586.
2018-08-16 13:50:42 -07:00
Harshit Bansal 7be7561c68 emoji: If current emojiset is text then fallback to google emojiset.
If `TEXT_EMOJISET` is currently selected emojiset then fallback to
`GOOGLE_EMOJISET` for displaying emojis in emoji picker and
composebox typeahead. We should pre-load the spritesheets in`emoji.js`
even in case of text emojiset otherwise on slow networks emoji picker
will appear empty initially.
2018-08-14 12:01:00 -07:00
Steve Howell 6d18fa116e minor: Sort some imports. 2018-08-14 10:34:58 -07:00
Steve Howell 3fce1ec649 tests: Test logging for internally sent messages. 2018-08-14 10:34:58 -07:00
Steve Howell 2dbd11ba9d tests: Test notification bot can send to private streams. 2018-08-14 10:34:58 -07:00
Steve Howell 9a7a93c80b refactor: Extract validate_sender_can_write_to_stream().
This de-clutters check_message a bit and also makes
it easy to audit our rules for who can write to a
stream.

Also, this works around a bug with Python where its
optimizations for the `pass` instruction make them
not appear to run and show up as uncovered in
coverage reports.
2018-08-14 10:34:58 -07:00
Steve Howell 82469b89e5 tests: Tests bots can send to owner's streams. 2018-08-14 10:34:58 -07:00
Steve Howell edd8437c76 tests: Test API super users can send to private streams. 2018-08-14 10:34:58 -07:00
Yashashvi Dave 2a73b2b805 custom profile field: Add at-least-one-choice validation for choice field.
Add validation at-least-one-choice required for every choice type field
in backend.
2018-08-14 10:17:57 -07:00
Yago González 53b9118e97 emails: Apply the user's hour format in login notification timestamps.
The timestamp used for new login notifications always used the 12-hour
format. Instead of that, we use now the one preferred by the user, as
reflected in their settings.
2018-08-13 19:06:54 -07:00
Yago González 75b39866c7 emails: Display the timezone's name on login notifications.
Show the timezone's name instead of the UTC offset, since it's easier to
infer where does the login come from that way.
2018-08-13 19:06:54 -07:00
Rishi Gupta aab977dbf8 RealmAuditLog: Add id to __str__ method. 2018-08-13 17:12:50 -07:00
Eeshan Garg 404439e98c webhooks/teamcity: Ignore third-party specific payload formats.
The TeamCity webhook plugin supports multiple payload formats that
are customized to be used by different services such as Slack,
Flowdock, etc. We don't support such payloads, so we should ignore
them and stick to parsing only the generic ones. We should also
notify that bot owner about the error.
2018-08-13 14:54:43 -07:00
Eeshan Garg 97995b4779 webhooks/teamcity: Update docs to conform to style guide. 2018-08-13 14:54:43 -07:00
Tim Abbott d66967f851 import: Use os.path.expanduser when accessing paths.
This fixes an issue where passing a path like `~/exports/foo` would
result in a `~` directory being created and the export/import not
working correctly.
2018-08-13 13:32:53 -07:00
Steve Howell 33f576f514 tests: Add coverage for email/stream encode/decode logic. 2018-08-13 13:23:29 -07:00
Rishi Gupta a8f96759c1 email worker: Fix typo in time.sleep call. 2018-08-13 11:16:13 -07:00
Rishi Gupta 4813bac98b signup: Add opayq.com to disposable domain whitelist. 2018-08-13 10:56:47 -07:00
Rishi Gupta 228e237d51 tests: Fix incorrect comment in test_invite_using_disposable_email. 2018-08-13 10:56:47 -07:00
Steve Howell 3112c6596c tests: Change message-type error to AssertionError.
We validate the user input upstream of this code.
2018-08-13 10:37:35 -07:00
Steve Howell 46e25896c7 tests: Test handling of bogus widget_content. 2018-08-13 10:37:35 -07:00
Steve Howell 297f086b6a tests: Add coverage for service bot events.
We test the "skipping" logic a couple different ways.
2018-08-13 10:37:35 -07:00
Steve Howell 90844700fb tests: Test for unexpected bot types. 2018-08-13 10:37:35 -07:00
Shubham Padia e21e8c1bae compose: Hide subscribe button and change text for waiting period users.
Fixes #10124.
Users in the waiting period category cannot subscribe other users to
a stream. When a user tries to mention another unsubscribed user, a
warning message appears with a subscribe button on it to subscribe
the other user.
This commit removes the subscribe button and changes the warning text
for users in the waiting period category.
2018-08-13 10:18:35 -07:00
Shubham Padia feb2cdf378 onboarding: Change logic for preventing new login emails for a new user.
Issue: When you created a new organization with /new, the "new login"
emails were emailed. We previously had a hack of adding the
.just_registered property to the user Python object to attempt to
prevent the emails, and checking that in zerver/signals.py. This
commit gets rid of the .just_registered check.
Instead of the .just_registered check, this checks if the user has
joined more than a minute before.
A test test_dont_send_login_emails_for_new_user_registration_logins
already exists.

Tweaked by tabbott to introduce the constant JUST_CREATED_THRESHOLD.

Fixes #10179.
2018-08-13 10:16:41 -07:00
Steve Howell 3a2422a149 tests: Test corner cases for bot-related PMs. 2018-08-11 14:51:26 -07:00
Steve Howell 7e32e2bf81 tests: Cover ValueError logic in extract_recipients. 2018-08-11 14:51:26 -07:00
Steve Howell 134678849d tests: Test sending messages outside of your realm. 2018-08-11 14:51:26 -07:00
Steve Howell 05763dcb1d tests: Cover stream name collisions in deactivation test. 2018-08-11 14:51:26 -07:00
Steve Howell 413a0174f4 Extract a zephyr.py library.
Right now it only has one function, but the function
we removed never really belonged in actions.py, and
now we have better test coverage on actions.py, which
is an important module to get to 100%.
2018-08-11 14:51:26 -07:00
Steve Howell 5bc33213c9 Refactor generate_topic_history_from_db_rows.
Sorting the rows first simplifies the loop logic here.

This has good test coverage--you'll get a failing
test if you comment out the sort.
2018-08-11 14:51:26 -07:00
Steve Howell 71d46083da tests: Ensure streams get created for internal messages. 2018-08-11 14:51:26 -07:00
Steve Howell e9a31232df tests: Test RuntimeError logic for realm check. 2018-08-11 14:51:26 -07:00
Steve Howell 101446ec52 tests: Add coverage for message truncation logic. 2018-08-11 14:51:26 -07:00
Aditya Bansal 0ed2b1d574 thumbnails: Fix bug with '/user_avatar/' links in Zulip messages.
In this commit we fix a bug due to which url preview images for urls
to custom emojis, realm icons or user avatars appeared broken when
such urls would be part of a Zulip message.
2018-08-10 17:17:50 -07:00
Aditya Bansal 985b8bb843 thumbnails: Refactor to include '/user_uploads/' in encrypted url.
This is a preparatory commit to fix a bug in which a user posts
a link of custom emoji, user avatar or realm icon in a Zulip
message.
In this commit we are just adjusting the url generation in the
backend to have the '/user_uploads/' in the encrypted url generated
which the user is supposed to be redirected to and therefore
essentially reaching thumbor with the encrypted url.
This is necessary because 'user_uploads' and 'user_avatars' (or any
other item under 'user_avatars' endpoint) have a different folder
location under the local file storage backend. 'user_uploads'
endpoint's stuff is stored in a 'files' directory whereas stuff
'user_avatars' endpoint's stuff is stored in a 'avatars' directory.
Thumbor needs to know from which directory a particular local file
needs to be retrieved and therefore the zthumbor/loaders.py adds
a prefix location for the directory.

Since in an upcoming commit we are going to add user_avatars
directory location 'avatars' folder as a prefix this preparatory
commit helps simply doing the changes.
2018-08-10 17:17:50 -07:00
Tim Abbott 8a22838acf slack import: Fix computation of owner email for uploaded files.
The previous code was just always returning the first user in the
organization, due to an incorrect comparison.
2018-08-10 16:20:36 -07:00
Rhea Parekh 3ff339c294 slack import: Add support for uploads in messages through 'files' keyword.
It appears that Slack just changed their export format, and how uses
this `files` list for user-uploaded files.
2018-08-10 16:20:36 -07:00
Rhea Parekh 20bca1409f import: Set emoji records 'last_modified' value in 'import_uploads_s3'.
The 'last_modified' value in emoji records is
needed for uploading the file to the S3 backend.
We set the same in the function 'import_uploads_s3'.

We also have to remove the keyword 'last_modified'
while building the RealmEmoji dict, as it is not
a field which exists in RealmEmoji objects.
2018-08-10 16:20:36 -07:00
Rishi Gupta 2855864f54 billing: Rename RealmAuditLog.REALM_PLAN_QUANTITY_RESET. 2018-08-10 15:54:31 -07:00
Rishi Gupta abc044e869 billing: Rename RealmAuditLog.REALM_PLAN_STARTED. 2018-08-10 15:54:31 -07:00
Rishi Gupta d33d7e1c28 billing: Rename RealmAuditLog.REALM_CARD_ADDED. 2018-08-10 15:54:30 -07:00
Rishi Gupta bdb6d8213f billing: Rename RealmAuditLog.REALM_STRIPE_INITIALIZED. 2018-08-10 15:46:55 -07:00
Rishi Gupta 2f042ac536 integrations: Fix typo in IFTTT doc.
Regression introduced in 85bded5.
2018-08-10 14:23:29 -07:00
Tim Abbott da8f4bc0e9 push notifications: Add support for removing GCM push notifications.
This uses the recently introduced active_mobile_push_notification
flag; messages that have had a mobile push notification sent will have
a removal push notification sent as soon as they are marked as read.

Note that this feature is behind a setting,
SEND_REMOVE_PUSH_NOTIFICATIONS, since the notification format is not
supported by the mobile apps yet, and we want to give a grace period
before we start sending notifications that appear as (null) to
clients.  But the tracking logic to maintain the set of message IDs
with an active push notification runs unconditionally.

This is designed with at-least-once semantics; so mobile clients need
to handle the possibility that they receive duplicat requests to
remove a push notification.

We reuse the existing missedmessage_mobile_notifications queue
processor for the work, to avoid materially impacting the latency of
marking messages as read.

Fixes #7459, though we'll need to open a follow-up issue for
using these data on iOS.
2018-08-10 13:58:39 -07:00
Tim Abbott cc5c8fc022 do_update_pointer: Improve docs for old mobile app code path. 2018-08-10 13:58:39 -07:00
Tim Abbott 8ba726d47d do_update_pointer: Fix missing where= declaration.
Fixes a regression introduced in 23246ff816.

However, we'll be shortly removing this feature, since it's legacy
support for an app that no longer is supported.
2018-08-10 13:58:39 -07:00
Tim Abbott 58307f80aa event_queue: Stop mocking push notifications in most tests.
Historically, queue_json_publish had a special third argument that was
basically its default mock behavior in the test suite.  We've been
migrating away from that model, because it was confusing and resulted
in poor test coverage of our queue worker code paths; this was one of
the last holdouts.

As it turns out, we don't exercise this code path in a way that
impacts tests much; the main downside of this change is a likely small
penalty to performance of the full test suite when sending private
messages.
2018-08-10 13:58:39 -07:00
Tim Abbott c3c7b33351 tests: Move clear_client_event_queues_for_testing to ZulipTestCase.
Following recent testing flakes that were traced down to this not
having been called causing `receiver_is_off_zulip` to depend on test
ordering, it makes sense to centralize this.

I think it should always have been in ZulipTestCase; it appears the
reason it wasn't from the beginning was that originally only
test_events.py interacted with it, and do_test there still needs to
call this directly (because it can be called multiple times within a
single test).  And then we did the wrong thing as expanded use of
Tornado event_queue code in tests to more of the codebase.
2018-08-10 13:58:39 -07:00
Tim Abbott 93c12fad28 test_events: Use clear_event_queue_for_testing more systematically.
This prevents these unit tests from accidentally leaking data outside
their boundaries.

Verified using a test that fails after test_events without this change.
2018-08-10 13:58:39 -07:00
Tim Abbott 8ceb3b88ad test_tornado: Fix leaking of event queue state.
Apparently, we weren't calling the proper clear functions inside the
Tornado tests, which resulted in unexpected behavior in other tests
that were relying on the Tornado event queue system being empty.

(In this case, a new test for mobile push notifications that assumed
receiver_is_off_zulip() was always true failed after this was run).
2018-08-10 13:58:39 -07:00
Vishnu Ks d64ba5d2be billing: Update the charged amount when user changes plan in upgrade page. 2018-08-10 13:52:35 -07:00
Rishi Gupta 85bded5cec integrations: Update IFTTT doc. 2018-08-10 10:37:49 -07:00
Rishi Gupta ff3ce6991b integrations: Update Zapier doc. 2018-08-10 10:37:49 -07:00
Tim Abbott 2f6f38fa7f import: Guess upload content-types when unavailable from export.
This is mostly for exports from other software like Slack, that might
not provide a content-type.
2018-08-10 09:32:28 -07:00
Tim Abbott 1ecbf49c93 import: Don't assume user_profile_id attribute is set on emojis.
The s3 import code path made a hard assumption about `user_profile_id`
being set (we'd already fixed this in the local uploads code path).

Ideally, it should be, and I've opened #10268 for fixing that, but for
now this is how it needs to work.
2018-08-10 09:32:18 -07:00
Tim Abbott cf8a0ae819 slack import: Set a last_modified timestamp for custom emoji. 2018-08-10 09:27:43 -07:00
Rhea Parekh cf60b8821d outgoing webhooks: Warn user that PMs are not supported in Slack-format webhook.
Private messages are not supported in Slack-format webhook.
Instead of raising a NotImplementedError, we warn the user
that PM service is not supported by sending a message to the
user.

Added tests for the same.

Fixes #9239
2018-08-09 17:44:26 -07:00
Rhea Parekh 2357b1e145 tests: Rename 'event' to 'stream_message_event' for clarity. 2018-08-09 17:42:10 -07:00