Commit Graph

6338 Commits

Author SHA1 Message Date
neiljp (Neil Pilgrim) 0781237b1f mypy: Improve typing to profile.py & remove FuncT from decorator.py.
FuncT was unused in decorator.py, and only imported into profile.py.
The @profiled decorator is now more strongly typed on return-type.
Annotations were converted to python3 format.
2017-10-29 17:10:13 -07:00
neiljp (Neil Pilgrim) 1cc0a3cc67 mypy: Improve constraint on return types in statsd_increment decorator. 2017-10-29 17:10:13 -07:00
Steve Howell f5fcbe453b Use get_stream_recipient() in test_narrow.mute_stream(). 2017-10-28 17:57:39 -07:00
Steve Howell 8302689789 Extract Message.is_stream_message().
This sets us up to denormalize on to Message some notion of
whether a message is for a stream (without having to hit the
Recipient table).
2017-10-28 17:57:39 -07:00
Steve Howell 8b012c6210 Extract get_personal_recipient(). 2017-10-28 17:57:39 -07:00
Steve Howell 64211512f8 Avoid mutation in get_recipient_from_user_ids.
We shouldn't mutate parameters in a `get_foo` function.
2017-10-28 17:57:39 -07:00
Steve Howell 370ffe389c Extract get_huddle_recipient().
This extraction moves all the huddle logic into models.py, which
hopefully can reduce friction for things like re-organizing our
caches (there are two cache entries for every huddle) and/or
just putting huddle_id on Message directly.
2017-10-28 17:57:39 -07:00
Steve Howell a28841e8aa Extract get_stream_recipient().
Do you call get_recipient(Recipient.STREAM, stream_id) or
get_recipient(stream_id, Recipient.STREAM)?  I could never
remember, and it was not very type safe, since both parameters
are integers.
2017-10-28 17:57:39 -07:00
Harshit Bansal 4c077a77ed notifications: Instead of regex use lxml for fixing emojis. 2017-10-28 10:38:33 -07:00
Steve Howell d132c30c24 Remove `active` parameter from `do_create_user()`.
Almost all callers to do_create_user were trying to
create active users, except for one test.  The
active=False codepath was kind of broken (things
like sending welcome messages had sort of undefined
behavior there), so instead of trying to maintain it,
we just update the one test (`test_people`) to flip the
`is_active` flag manually.

Fixes #7197
2017-10-28 10:32:01 -07:00
Steve Howell d6c47573b2 Eliminate ZulipTestCase.send_message().
Now all tests use these three messages instead:

    send_stream_message
    send_personal_message
    send_huddle_message
2017-10-28 10:20:59 -07:00
Steve Howell c978464d16 tests: Fix send_message calls in test_user.py. 2017-10-28 10:20:59 -07:00
Steve Howell 610d9b4abc tests: Fix send_message calls in test_unread.py. 2017-10-28 10:20:59 -07:00
Steve Howell a20bba04ca tests: Fix send_message calls in test_tutorial.py. 2017-10-28 10:20:59 -07:00
Steve Howell bb42004464 tests: Fix send_message calls in test_subs.py. 2017-10-28 10:20:59 -07:00
Steve Howell fba754224c tests: Fix send_message calls in test_soft_deactivation.py. 2017-10-28 10:20:59 -07:00
Steve Howell 6475b25a00 tests: Fix send_message calls in test_signup.py. 2017-10-28 10:20:59 -07:00
Steve Howell f264bbc7ca tests: Fix send_message calls in test_service_bot_system.py. 2017-10-28 10:20:59 -07:00
Steve Howell 8276442ee6 tests: Fix send_message calls in test_retention.py. 2017-10-28 10:20:59 -07:00
Steve Howell 1f077f7177 tests: Fix send_message calls in test_reactions.py. 2017-10-28 10:20:59 -07:00
Steve Howell 74943b4120 tests: Fix send_message calls in test_notifications.py. 2017-10-28 10:20:59 -07:00
Steve Howell 26e51c016d tests: Fix send_message calls in test_narrow.py. 2017-10-28 10:20:59 -07:00
Steve Howell 5c00bbc1a9 tests: Fix send_message in test_message_edit_notifications.py. 2017-10-28 10:20:59 -07:00
Steve Howell cf1a4540ef tests: Fix send_message calls in test_link_embed.py. 2017-10-28 10:20:59 -07:00
Steve Howell fd9d919fc6 tests: Fix send_message calls in test_home.py. 2017-10-28 10:20:59 -07:00
Steve Howell 1c38130626 tests: Fix send_message calls in test_export.py. 2017-10-28 10:20:59 -07:00
Steve Howell 3e319837ef tests: Fix send_message calls in test_events.py.
We mostly introduce these functions (as part of a big
code sweep):

    send_stream_message
    send_personal_message
    send_huddle_message

In two cases, where we want to specifically manipulate
queue ids, we now call check_send_message directly.  (The
above three functions deliberately don't support kwargs
to ensure simple code and better type safety.)
2017-10-28 10:20:59 -07:00
Steve Howell 25379a3b19 Fix send_message calls in test_embedded_bot_system.py. 2017-10-28 10:20:59 -07:00
Steve Howell 4372bfb4a5 tests: Fix send_message calls in test_bugdown.py. 2017-10-28 10:20:59 -07:00
Steve Howell 7cf5be2266 tests: Fix send_message calls in test_alert_words.py. 2017-10-28 10:20:59 -07:00
Steve Howell 53ad6270b4 tests: Fix send_message calls in test_event_queue.py. 2017-10-28 10:20:59 -07:00
Steve Howell 9c90c40c08 tests: Fix send_message calls in test_upload.py. 2017-10-28 10:20:59 -07:00
Tim Abbott f2e3e779eb mypy: Properly annotate generic_bulk_cached_fetch.
Along with fixing some minor bugs, this requires extracting out the
default functions so that we can do type: ignores on them properly.

While we're at it, we switch to the Python 3 syntax.
2017-10-28 10:07:15 -07:00
Tim Abbott 73c27e1277 cache: Fix type aliasing of cached_objects.
Previously, it was converted from a CompressedItemT to an ItemT
without changing the variable name.
2017-10-28 10:01:44 -07:00
Tim Abbott 94c1da7025 cache: Move generic_bulk_cached_fetch typevars up a bit. 2017-10-28 10:00:43 -07:00
Tim Abbott 5b5dffebe7 get_messages_backend: Tweak how search fields are added.
This is a no-op, but makes mypy happier, since previously it wanted to
know the type of the {} we were passing to `.update()`.
2017-10-28 09:58:51 -07:00
Tim Abbott 6c242379f4 bugdown: Remove unused caching imports. 2017-10-28 09:02:36 -07:00
neiljp (Neil Pilgrim) 232bcd98b3 mypy: Improve return type of internal_notify_view decorator. 2017-10-28 08:57:49 -07:00
neiljp (Neil Pilgrim) 5c3af83229 mypy: Improve return type of api_key_only_webhook_view decorator. 2017-10-28 08:57:49 -07:00
neiljp (Neil Pilgrim) c063ba72a2 mypy: Improve typing of cache_with_key and cache decorators.
Fixes #1348.
2017-10-28 08:57:49 -07:00
Tim Abbott 2e69d4d420 mypy: Fix type for per_request_display_recipient_cache. 2017-10-28 08:57:49 -07:00
Tim Abbott fa55d7ed33 retry_event: Remove requirement of failed_tries being already there.
This fixes a bug where retries in the signups queue threw an exception.
2017-10-27 18:15:53 -07:00
Tim Abbott bfa148ab39 requirements: Update mypy to latest version.
This involves fixing the mypy annotations in one file
2017-10-27 17:05:52 -07:00
Tim Abbott 090563fb25 test_tornado: Fix IOLoop exceptions output spam.
We weren't quite following the proper protocol for Tornado tests.
2017-10-27 16:28:33 -07:00
Tim Abbott f790e667b3 test_tornado: Fix repeated autoreload configuration in tests.
Apparently, our previous efforts to disable AUTORELOAD in the tests
were incorrect.
2017-10-27 16:19:03 -07:00
Tim Abbott aacafd7a4d test_tornado: Fix incorrect tearDown method for IOLoops.
We appear to have 2 bugs of this form, so this just cuts half of the
exception/warning log output, but it's a start.
2017-10-27 16:13:42 -07:00
Tim Abbott 49d7d49038 test_narrow: Fix sqlalchemy warning spam. 2017-10-27 16:06:03 -07:00
Tim Abbott d80d15c08d GoogleSubdomainLoginTest: Suppress unnecessary logging output.
This helps make our test output nice and clean.
2017-10-27 16:06:03 -07:00
Tim Abbott e5df05fd35 tests: Suppress logging spam in email mirror tests. 2017-10-27 16:06:03 -07:00
Tim Abbott a1fa7a3c83 JWT: Filter out logging.warning output in tests. 2017-10-27 16:06:03 -07:00
Tim Abbott fca6df09b0 test_home: Mark a couple tests as explicitly slow. 2017-10-27 16:05:55 -07:00
Tim Abbott 28b25a66c2 test_events: Mark several slower tests as explicitly slow. 2017-10-27 15:46:59 -07:00
Tim Abbott ef9679667f test_docs: Fix unnecessary info-level logging in tests. 2017-10-27 15:44:45 -07:00
Tim Abbott 073c81bbe8 slow_tests: Increase limit for TOO slow notices. 2017-10-27 15:38:17 -07:00
Tim Abbott 80a9253f89 attachments: Remove unnecessary use of REQ.
It didn't do anything, because that third argument is actually a
positional argument coming from the URL parsing logic.
2017-10-27 15:18:44 -07:00
Tim Abbott 5467296f60 decorators: Use human_users_only more aggressively. 2017-10-27 15:16:13 -07:00
Tim Abbott 1cd017288d views: Fix imports of REQ/has_request_variables from the wrong place.
These were never in zerver/decorator.py, and so it makes sense to
import them zerver/lib/request.py, mostly for ease of finding things.
2017-10-27 15:07:31 -07:00
Greg Price 6d403ff255 tests: Simplify set_http_host to dedupe its logic.
This will make it easier to change this logic.
2017-10-27 14:42:24 -07:00
Greg Price e4b4f67b44 subdomains: Tighten search for EXTERNAL_HOST in get_subdomain.
If a Zulip install at example.org got a request at an HTTP `Host`
like foo.example.org.evil.com (or even foo.example.orgevil.com),
we would accept it as subdomain foo.  This isn't likely to happen
in practice because it shouldn't pass ALLOWED_HOSTS, and it's not
obvious to me that anything untoward could be done with it even
if ALLOWED_HOSTS were set wide open, but if nothing else it
multiplies the cases in analyzing this logic.

The reason we had a loose match like this, I assume, is to allow
the user to come from arbitrary ports -- especially in development.
So tighten the pattern to allow just that, and add some tests for
that behavior and a comment explaining why this complication is
needed.
2017-10-27 14:42:24 -07:00
Greg Price 1b19af2612 subdomains: Write some tests for get_subdomain.
This logic is a bit subtle, and we're about to make changes to it.
So let's have some tests.
2017-10-27 14:42:24 -07:00
Greg Price d9cb606804 oauth: Find a better name for redirect_to_main_site.
This name was way too broad for the rather specific logic
in the actual function.
2017-10-27 14:42:24 -07:00
Greg Price ad551427ed auth: Factor out some uses of EXTERNAL_HOST with ROOT_DOMAIN_URI.
Apart from being less verbose, this makes it more manifest (on e.g. grep)
that we aren't using EXTERNAL_HOST here to construct subdomains.
2017-10-27 14:42:24 -07:00
Greg Price 318682fd52 auth: Use URL rather than cookie to pass signed data cross-domain.
The cookie mechanism only works when passing the login token to a
subdomain.  URLs work across domains, which is why they're the
standard transport for SSO on the web.  Switch to URLs.

Tweaked by tabbott to add a test for an expired token.
2017-10-27 14:42:04 -07:00
Greg Price 6efcce9374 oauth login: Refactor tests to dedupe a bit of recurring logic.
This makes the tests a little cleaner in itself, and also prepares
them to adjust with less churn when we change how
redirect_and_log_into_subdomain passes the signed token.
2017-10-27 14:28:38 -07:00
Greg Price fad3d56810 views: Move some login code from `registration` to `auth`.
Most of these have more to do with authentication in general than with
registering a new account.  `create_preregistration_user` could go
either way; we move it to `auth` so we can make the imports go only in
one direction.
2017-10-27 14:28:38 -07:00
Henrik Pettersson 09cd47c6ad Add UI for viewing and cancelling open Zulip invitations.
Lets administrators view a list of open(unconfirmed) invitations and
resend or revoke a chosen invitation.

There are a few changes that we can expect for the future:

  * It is currently possible to invite an email that you have already
    invited, it might make sense to change this behavior.

  * Resend currently sends an invite reminder instead of resending the
    original invite, this is because 'custom_body' was not stored when
    the first invite was sent.

Tweaked in various minor ways, primarily in the backend, by tabbott,
mostly for style consistency with the rest of the codebase.

Fixes: #1180.
2017-10-27 13:07:43 -07:00
Tim Abbott 237e3cdca1 registration: Clarify comment on do_activate_user code path. 2017-10-27 11:22:19 -07:00
Vishnu Ks 8c68a167fe backend: Allow Administrators to invite new users as admins.
Tweaked by tabbott to have the field before the invitation is
completed be called invite_as_admins, not invited_as_admins, for
readability.

Fixes #6834.
2017-10-27 11:19:38 -07:00
Steve Howell 4b78f69a87 tests: Introduce send_huddle_message().
The tighter interface here makes for shorter lines (or
fewer multi-line calls) and it leads to stricter type
checking.
2017-10-27 11:00:23 -07:00
Steve Howell 9707767b21 tests: Introduce send_personal_message().
The tighter interface prevents the need to specify
Recipient.PERSONAL (which can often be inaccurate in the
huddle case, anyway), and it prevents tests from confusingly
specifying a "subject" field for PMs.
2017-10-27 11:00:23 -07:00
Steve Howell 25a6fbd05d tests: Introduce send_stream_message().
Having send_stream_message() avoids the need to supply
Recipient.STREAM as a parameter, and it also uses the more
modern name of `topic_name` for topics.  Under the hood, it
avoids some annoying steps for re-formatting the recipients,
since we just have a single stream name.
2017-10-27 11:00:23 -07:00
Steve Howell 9576d25f92 Use check_send_stream_message in test_not_too_many_queries.
When possible, we want to use direct APIs for sending
stream messages.

This changes the codepath slightly, by not using
forwarded_user_profile, but it doesn't impact the number
of queries, and it's a simple check.

We also remove a couple "subject" references here.
2017-10-27 11:00:23 -07:00
Steve Howell abf16a96d1 Rename subject -> topic_name in assert_stream_message(). 2017-10-27 11:00:23 -07:00
Steve Howell c4404ed078 tests: Change some test topic names to not use "subject". 2017-10-27 11:00:23 -07:00
Steve Howell 65b6676844 minor: Remove unused import (internal_send_message). 2017-10-27 10:48:11 -07:00
Steve Howell 655f37a34b Rename subject_name in send_message_backend(). 2017-10-27 10:48:11 -07:00
Steve Howell e5c5d10375 Rename subject vars in do_update_message(). 2017-10-27 10:48:11 -07:00
Steve Howell fbe5f93141 Rename subject -> topic_name in internal_send_message. 2017-10-27 10:48:11 -07:00
Steve Howell 973ee97612 Rename subject -> topic_name in internal_prep_message(). 2017-10-27 10:48:11 -07:00
Steve Howell f04550d143 Rename subject -> topic_name as var in check_message().
There were two variables, subject and subject_name, that
really had no clear distinction, so now they're just
topic_name.
2017-10-27 10:48:11 -07:00
Steve Howell 70b05998b1 Rename subject_name -> topic_name in check_send_message(). 2017-10-27 10:48:11 -07:00
derAnfaenger 64025d2ce7 integrations: Add Errbot documentation.
Fixes zulip/python-zulip-api#49.
2017-10-27 09:22:51 -07:00
Tim Abbott cab39dc6eb get_service_bot_events: Add some future-proofing.
The comment is the most important part here, but the new code
structure should help, too.
2017-10-26 22:16:47 -07:00
Tim Abbott b897c763f8 get_recipient_info: Simplify active_user_ids construction. 2017-10-26 22:16:47 -07:00
Steve Howell a70ede6c75 Allow "default" bots to see mentions on all streams.
This change allows normal bots to get UserMessage rows when
they are mentioned on a stream, even if they are not actually
subscribed to the stream.

Fixes #7140.
2017-10-26 22:16:47 -07:00
Steve Howell 53dbcbe2a4 refactor: Handle service bot mentions earlier in the code.
We now find all (possibly) relevant service bots for a message
in the call to get_recipient_info.  This allows us to eliminate
some code that would patch them after we rendered.

The get_service_bot_events() function will ignore any service
bots that weren't actually mentioned in the message (due to
backticks) or part of the active user ids.
2017-10-26 22:16:47 -07:00
Steve Howell 87b48f722d Add MentionData.get_user_ids().
This prepares us to take advantage of the user_ids
in subsequent commits related to `do_send_messages()`.
2017-10-26 22:16:47 -07:00
Steve Howell 8ac26dfb9b refactor: Introduce bugdown.MentionData class.
We now have a MentionData class that encapsulates
the users who are possibly mentioned in a message.

Not that the rendering code may not keep all the mentions,
since things like backticks will suppress the mention.

We populate this now in do_send_messages, so that we can use
the info earlier in the message-sending process.  This info
now gets passed down the call stack as an optional parameter.

Note that bugdown.convert() still populates the data when its
callers decline to pass in a MentionData object.

This is mostly a preparatory commit, as we don't take advantage
of the data yet in do_send_messages.
2017-10-26 22:16:47 -07:00
rht 45e8ce559d zerver/views: Use python 3 syntax for typing. 2017-10-26 21:58:22 -07:00
rht 1047733486 zerver/tornado: Use python 3 syntax for typing. 2017-10-26 21:58:22 -07:00
Tim Abbott 8e2cdedf9a lint: Fix lines in Python codebase longer than 120 characters. 2017-10-26 17:47:30 -07:00
Tim Abbott b936e8c24b lint: Fix lines in Python codebase longer than 125 characters. 2017-10-26 17:36:54 -07:00
Tim Abbott be619fe881 lint: Wrap many very long lines in the Python codebase.
This decreases the maximum line length in our Python codebase to 130.
2017-10-26 17:31:58 -07:00
Steve Howell 635675fe48 Reduce queries needed for sending messages.
In do_send_messages, we only produce one dictionary for
the event queues, instead of different flavors for text
vs. html.  This prevents two unnecessary queries to the
database.

It also means we only put one dictionary on the "message"
event queue instead of two, albeit a wider one that has
some values that won't be sent to the actual clients.

This wider dictionary from MessageDict.wide_dict is also
used for the `feedback_messages` queue and service bot
queues.  Since the extra fields are possibly useful down
the road, and they'll just be ignored for now, we don't
bother to remove them.  Also, those queue processors won't
have access to `content_type`, which they shouldn't need.

Fixes #6947
2017-10-26 16:35:28 -07:00
Steve Howell 9b6a4d0b16 refactor: Extract MessageDict.finalize_payload(). 2017-10-26 16:35:28 -07:00
Steve Howell df93a99b50 Cache only one row per message.
Before this change, we populated two cache entries for each
message that we sent.  The entries were largely redundant,
with the only difference being whether we sent the content
as raw markdown or as the rendered HTML.

This commit makes it so we only have one cache entry per
message, and it includes both content and rendered_content.

One legacy source on confusion here is that `content`
changes meaning when you're on the front end.  Here is the
situation going forward:

    database:
        content = raw
        rendered_contented = rendered

    cache entry:
        content = raw
        rendered_contented = rendered

    payload for the frontend:
        content = raw (for apply_markdown=False)
        content = rendered (for apply_markdown=True)
2017-10-26 16:35:28 -07:00
Steve Howell 0cef7c9fd5 Refactor: Extract get_client_info_for_message_event().
This removes some clutter from process_message_event, and it
makes process_message_event() a bit easier to test.
2017-10-26 16:35:28 -07:00
Steve Howell a50671165a tests: Add test_content_types(). 2017-10-26 15:47:44 -07:00
rht c2a5b5f160 zerver/migrations: Use python 3 syntax for typing. 2017-10-26 15:24:56 -07:00
rht a311678190 zerver/management: Use python 3 syntax for typing. 2017-10-26 15:24:56 -07:00