Commit Graph

5739 Commits

Author SHA1 Message Date
Tim Abbott e8e876d54c events: Deduplicate anonymous user code.
This approach lets us deduplicate much of the fetch_initial_state_data
logic around logged-out users.
2020-10-01 14:43:51 -07:00
Aman Agrawal f46f251688 post_process_state: Allow web public guests.
Because the logic already works correctly, we just need to change mypy
types.
2020-10-01 14:41:49 -07:00
Aman Agrawal 190f481f49 stream_subscription: Mark notifications disabled for web public users.
Users without an account can't get notifications, so we might as well
ensure any UI displays them appropriately.
2020-10-01 14:40:48 -07:00
Anders Kaseorg 46babbe9e1 import_realm: Close the memcached connection before forking.
This prevents the memcached connection from being shared across
multiple processes, and hopefully addresses unexpected behavior from
cached functions like get_user_profile_by_id invoked inside the worker
processes.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-10-01 11:20:39 -07:00
sahil839 6c473ed75f message: Call build_message_send_dict from check_message.
We call build_message_send_dict from check_message instead of
do_send_messages.

This is a prep commit for adding a new setting for handling
wildcard mentions in large streams.
2020-09-29 17:18:04 -07:00
sahil839 f1a5fbaeb0 message: Extract build_message_send_dict function.
We extract the loop for building message dict in
do_send_messages in a separate function named
build_message_send_dict.

This is a prep commit for moving the code for building
of message dict in check_message.
2020-09-29 16:50:47 -07:00
sahil839 0514ba7ecb message: Add 'links_for_embed' to message_dict.
There is a bug where we send event for even
those messages which do not have embedded links
as we are using single set 'links_for_embed' to
check whether we have to send event for
embedded links or not.

This commit fixes the bug by adding 'links_for_embed'
in message dict itself and send the event only
if that message has embedded links.
2020-09-29 16:50:47 -07:00
Steve Howell c199571112 mypy: Add StreamDict.
This requires us to rework the view code a little
bit to explicitly assign fields.
2020-09-29 16:49:10 -07:00
Steve Howell bee18c70f0 mypy: Use str in statsd_key. 2020-09-29 16:49:10 -07:00
Steve Howell 2c496d9afd mypy: Fix do_send_user_group_update_event. 2020-09-29 16:49:10 -07:00
Steve Howell a37ef208dc mypy: Add RawReactionRow. 2020-09-29 16:49:10 -07:00
Vishnu KS 367c792968 actions: Downgrade realm before scrubbing. 2020-09-28 15:37:49 -07:00
Vishnu KS 0d30f59c97 billing: downgrade_now -> downgrade_now_without_creating_additional_invoice. 2020-09-28 15:37:49 -07:00
Steve Howell def3dac6ae event_schema: Add comments to top of the file.
The comments basically explain the common coding
patterns for making the checkers.
2020-09-28 12:19:28 -07:00
Tim Abbott 3242fc7388 soft_deactivation: Fix typo in logging output. 2020-09-28 12:12:04 -07:00
palash 0c18113910 soft_deactivation: Change root logger to zulip.soft_deactivation.
Update logger in the following files using this logger:
test_soft_deactivation, test_home, test_push_notifications
2020-09-28 12:12:00 -07:00
Tim Abbott 899cb41857 MessageDict: Remove _finalize_payload for simplicity.
finalize_payload already has a few options; there's little benefit to
this one being implemented as a separate helper function.
2020-09-28 12:00:18 -07:00
Tim Abbott 99396b25a6 MessageDict: Add a bit of docstring documentation. 2020-09-28 11:50:02 -07:00
Tim Abbott 90ff62aabc actions: Rename message local variable to message_dict.
This is a preparatory refactor to make it easy to see the changes
using `git show` in the next commit.
2020-09-28 11:14:59 -07:00
sahil839 ae74f8aafb actions: Remove unnecessary comment in do_send_messages function.
This commit removes the unnecessary comment which was added in
9454683108, when we were using message.get() for keys which
were also passed as args in do_send_messages, but there are no
such keys in the current code.
2020-09-28 10:58:35 -07:00
sahil839 76c75fea92 actions: Remove unnecessary line from do_send_messages.
This commit removes the unnecessary line of code to get
rendered_content from message dict sent by check_message
when it actually does not inlcude 'rendered_content' key.

This line was added in 9454683108, but now we do not send
rendered_content in the message dict as we render the message
in do_send_messages itself.
2020-09-28 10:58:35 -07:00
Dinesh acca870480 tests: Add a dummy request to self.client.login().
A later commit alters `authenticate` of EmailAuthBackend to
add a store `needs_to_change_password` variable to session
which is useful to insist users on changing their weak password.

The tests start failing with that change because client.login()
runs `authenticate` without a `request` object. So, this commit
sends a request object with `request.session=self.client.session`
to self.client.login() in tests wherever needed.
2020-09-25 16:24:18 -07:00
Abhijeet Prasad Bodas d9d51e32c1 i18n: Simplify logic for translation data in page_params.
This refactors the get_translation_data function to return an empty
dict when 'en' language is passed, to avoid repetition of code in
stats and home.
2020-09-25 16:21:37 -07:00
Clara Dantas 8674287192 digest: Support digest of web public streams for guest users.
This change requires some basic plumbing for test code creating
web-public streams.
2020-09-25 16:11:04 -07:00
Tim Abbott 94a9fa1891 event_schema: Add documentation and rename a few functions.
This should help make this revised subsystem readable for more new
contributors.  We still need to make updates to the high-level
documentation.
2020-09-25 12:53:00 -07:00
Steve Howell 5b7c9c4714 test_events: Add check_realm_user_remove. 2020-09-25 11:43:20 -07:00
Steve Howell 7bb7f2943f event_schema: Finish extraction with realm_emoji/update.
We now no longer define any schemas in test_events--all
of them are in event_schema, which helps our tooling
cross-check schemas for openapi and node tests.
2020-09-25 11:43:20 -07:00
Steve Howell ae4d083a5a event_schema: Extract check_realm_domains_*. 2020-09-25 11:43:20 -07:00
Steve Howell 298bed9fa1 event_schema: Split check_update_message_flags. 2020-09-25 11:43:20 -07:00
Steve Howell f6e0171d02 event_schema: Split check_reaction into add/remove.
It happens that whether you add a reaction or remove
a reaction, we send the exact same fields, just using
a different op code.

This sort of symmetry is actually kind of rare, as
usually "add" events have more fields, and "remove" events
might just send an id of something to remove.

Our openapi schema treats these as two seperate events,
so we are more consistent with it, and it helps our
schema-checking tooling for node fixtures, too.

Note that we now have to exempt the two events from
our openapi checks, due to the is_mirror_dummy field
in the deprecated user block.  We can decide how to
handle this later--one possibility is to just add it
as an optional field on the event_schema side.
2020-09-25 11:43:20 -07:00
Steve Howell b7b2546f44 event_schema: Extract check_subscription_update.
Note that we use value_type for value instead of
bool, since properties can be non-bool things
like color, which we just don't test now.  We
should test them.

We more than compensate for this by checking
the actual value of the value in
check_subscription_update.
2020-09-25 11:43:20 -07:00
Steve Howell b920ebce81 event_schema: Extract check_has_zoom_token. 2020-09-25 11:43:20 -07:00
Steve Howell 0c4286222f event_schema: Extract check_realm_update_dict. 2020-09-25 11:43:20 -07:00
Steve Howell 6ec6525624 event_schema: Extract check_delete_message.
There is a legacy format where we send
singular "message_id" instead of plural
"message_ids".

Then there are different fields for "private"
and "stream" message types.
2020-09-25 11:43:20 -07:00
Steve Howell 88165aee6b event_schema: Extract check_user_group_update. 2020-09-25 11:43:20 -07:00
Steve Howell aaaac11661 event_schema: Extract check_user_group_remove. 2020-09-25 11:43:20 -07:00
Steve Howell 1b7af13f37 event_schema: Extract check_user_group_remove_members. 2020-09-25 11:43:20 -07:00
Steve Howell 19b7739065 event_schema: Extract check_user_group_add_members. 2020-09-25 11:43:20 -07:00
Steve Howell 4084f0b949 event_schema: Extract check_realm_user_add.
Note that we make the schema for profile_data
slightly more realistic, but it doesn't actually get
exercised by our current tests (apart from
making sure it's a dict), since we don't have
profile data for our test realm.

We also don't have the optional fields for bots,
since our tests don't exercise that, nor
delivery_email.

So we exempt realm_user_add_event from openapi
checks for now.

When we try to match the openapi specs better, we
will probably want to add a few tests to test_events.

Obviously getting good coverage for adding users
would be nice for all these scenarios:

    * delivery_email matters
    * bots
    * realm has profile fields
2020-09-25 11:43:19 -07:00
Steve Howell dc2176a965 event_schema: Extract check_presence. 2020-09-25 11:43:19 -07:00
Steve Howell 6c74a44697 data_types: Generalize StringDictType.
This is a prep commit for supporting "presence"
events, where the key of the dictionary is some
arbitrary string like "website" but the value
of the dictionary is another dictionary itself
with keys that are more like variable names.
2020-09-25 11:43:19 -07:00
Steve Howell 4f3d5f2d87 event_schema: Extract check_realm_filters.
We have some known issues with representing
tuples in openapi, so we exempt realm_filters
from the relevant check.
2020-09-25 11:43:19 -07:00
Steve Howell e40a5400e5 event_schema: Extract check_muted_topics.
This also forces us to create TupleType.

We exempt this from the openapi check,
since we haven't figured out how to model
tuples in openapi with the same precision
as event_schema (and it may be impossible).

Long term we just want to stop dealing in
tuples, of course.
2020-09-25 11:43:19 -07:00
orientor 91ca1afe98 data_type: Add StringDict data type.
StringDict is a data type for representing dictionaries where
all keys and values are strings. Add this data type to data_types.py
and edit other files so that this data type is put to use and tested.

(slightly tweaked by @showell to remove a comment and shorten
a var name now that we have a proper data type)
2020-09-25 11:43:19 -07:00
Steve Howell 78a2059b8d event schema: Extract attachment checkers. 2020-09-25 11:43:19 -07:00
Steve Howell 4a947c971d event_schema: Extract check_realm_export.
These are all trivial transformations.

Note that we don't insist timestamps are
floats; the NumberType class allows ints
too.
2020-09-25 11:43:19 -07:00
Steve Howell d28c01284c event_schema: Extract check_hotspots.
This forces us to introduce a NumberType.
2020-09-25 11:43:19 -07:00
Steve Howell cf26151cea event_schema: Use realm_user_person_types.
For realm_user events, we now structure the
person type as a union of dicts, which is
more consistent with how we model this in
our openapi spec.
2020-09-25 11:43:19 -07:00
Steve Howell 10952394b0 test_events: Use int value of message_retention_days.
We also make our schema in event_schema reflect this,
which in turn makes us match the already accurate
openapi spec, so we no longer need to exempt four
types of events from our sanity checks.
2020-09-25 11:43:19 -07:00
Steve Howell 73e7f7edec check-node-fixtures: Compare python/openapi schemas.
We might want to rename the tool to something more
general now, since we are really reconciling three
things:

    - node fixtures
    - event_schema checkers for test_events
    - openapi specs

The way we compare python and openapi schemas is
as follows:

    - first convert openapi schemas to be build
      from DictType, ListType, etc. with from_opeapi

    - do a diff on the schemas

Most of the new code is just having the FooType
family of classes serialize themselves with schema().
2020-09-25 11:43:19 -07:00
Steve Howell 2b76eb767f event_schema: Change propagate_mode to an enum. 2020-09-25 11:43:19 -07:00
Steve Howell aca641a4d1 refactor: Extract data_types module.
Defining types with an object hierarchy
of type classes will allow us to build
functionality that was impossible (or
really janky) with the validators.py
approach of composing functions.

Most of the changes to event_schema.py
were automated search/replaces.

This patch doesn't really yet take
advantage of the new FooType classes,
but we will use it soon to audit our
openapi specs.
2020-09-25 11:43:19 -07:00
Wes Galbraith 9645959ac4 populate_db: Add emoji reactions to development environment database.
This change adds automated generated emoji reactions to the data in
the development environment's database.

Fixes part of #14991.
2020-09-23 16:10:37 -07:00
Aman Agrawal 5ae2325979 events: Don't send presence data for web public guests.
We disable presence and hide list of users in right sidebar
via setting realm_presence_disabled to false here for
web public guests.
2020-09-23 12:11:22 -07:00
Aman Agrawal 5153a036f2 events: Disable topic edit & msg edit/delete for web public guest.
To ensure web public guests cannot change any data, it's natural
that we disable topic & msg edit / delete ops.
2020-09-23 12:11:22 -07:00
Aman Agrawal 3ec23e1a9d fetch_initial_state_data: Handle case of web public guests.
user_profile will be None for web_public_guests here.  Hence, for
settings (of which most be inaccessible by web public guest),
which require a user_profile, we either set an empty value for
them or set them to a default value. This will help render
the frontend or extend support to our clients without breaking
a lot of code.

Tweaked by tabbott to add many comments.
2020-09-23 12:11:15 -07:00
Aman Agrawal 6012d3cff2 get_raw_user_data: Support acting_user=None.
This already had support for web_public_guest, fixed mypy
annotation.
2020-09-23 12:07:06 -07:00
Aman Agrawal c88b4cba60 format_user_row: Support acting_user=None. 2020-09-23 12:07:06 -07:00
Aman Agrawal 48492a0633 fetch_initial_state_data: Pass realm as independent parameter.
This removes dependency of the function on user_profile to get the
realm, which will be useful when user_profile is None in case of web
public guests.
2020-09-23 12:06:54 -07:00
Alex Vandiver d24869e484 webhooks: Rename is_webhook to allow_webhook_access.
This argument does not define if an endpoint "is a webhook"; it is set
for "/api/v1/messages", which is not really a webhook, but allows
access from webhooks.
2020-09-22 15:11:48 -07:00
Alex Vandiver b515c2bcbf webhooks: Add a missing format for the realm. 2020-09-22 15:11:48 -07:00
Aman Agrawal 1b5b82e712 RealmFilterPattern: Mark converted content as AtomicString.
If multiple filters match the same string, we run into an infinite
loop of converting string into urls. To fix it, we mark the matched
string as atomic after first conversion.
2020-09-22 15:10:38 -07:00
Anders Kaseorg 0228acf0f5 rest: Add rest_path shortcut for path with rest_dispatch.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-09-22 10:51:00 -07:00
Anders Kaseorg e70f2ae58d rest: Specify rest_dispatch handlers by function, not by string.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-09-22 10:46:28 -07:00
Anders Kaseorg faf600e9f5 urls: Remove unused URL names and shorten others.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-09-22 10:46:28 -07:00
Alex Vandiver 03c6a0f182 markdown: Skip other common file extensions in linking, sort. 2020-09-21 21:03:29 -07:00
Alex Vandiver 4361ce1246 markdown: Use tlds package to keep updated list of TLDs.
Also remove a useage of "blacklist."
2020-09-21 21:03:29 -07:00
Amitsinghyadav a72e9476ee api_docs: Add fragment references for all parameters.
This makes it convenient to link to a specific parameter accepted by
Zulip API endpoint.
2020-09-21 12:18:10 -07:00
Sumanth V Rao 564b199fe6 markdown: Escape lang when echoing back custom non-pygments languages.
In ae58ed5a7 we decided to echo back the text, when no Pygments lexer
matching that language was found. When we do so, we must take care to
HTML escape the lang before wrapping it in a data-code-language attribute.

Tweaked by tabbott to make clear the escaping is defensive.
2020-09-18 17:12:11 -07:00
Alex Vandiver 1928696829 tests: Remove three references to Casper in the comments. 2020-09-18 15:13:40 -07:00
Tim Abbott 9c0d6becc5 docs: Add comment links to i18n documentation. 2020-09-18 11:44:04 -07:00
Tim Abbott ae58ed5a74 markdown: Tweak data-code-language testing and comments.
This should make it clearer the precise decisions we've made about the
intended semantics of this feature.
2020-09-15 12:30:57 -07:00
Sumanth V Rao b0c9e0a295 markdown: Rename fenced code data-attribute to data-code-language. 2020-09-15 20:09:58 +05:30
Alex Vandiver 812af977d3 compilemessages: Switch to canonical override_language(). 2020-09-14 22:16:09 -07:00
Alex Vandiver 536bd3188e middleware: Move locale-setting before domain checking.
Calling `render()` in a middleware before LocaleMiddleware has run
will pick up the most-recently-set locale.  This may be from the
_previous_ request, since the current language is thread-local.  This
results in the "Organization does not exist" page occasionally being
in not-English, depending on the preferences of the request which that
thread just finished serving.

Move HostDomainMiddleware below LocaleMiddleware; none of the earlier
middlewares call `render()`, so are safe.  This will also allow the
"Organization does not exist" page to be localized based on the user's
browser preferences.

Unfortunately, it also means that the default LocaleMiddleware catches
the 404 from the HostDomainMiddlware and helpfully tries to check if
the failure is because the URL lacks a language component (e.g.
`/en/`) by turning it into a 304 to that new URL.  We must subclass
the default LocaleMiddleware to remove this unwanted functionality.

Doing so exposes a two places in tests that relied (directly or
indirectly) upon the redirection: '/confirmation_key'
was redirected to '/en/confirmation_key', since the non-i18n version
did not exist; and requests to `/stats/realm/not_existing_realm/`
incorrectly were expecting a 302, not a 404.

This regression likely came in during f00ff1ef62, since prior to
that, the HostDomainMiddleware ran _after_ the rest of the request had
completed.
2020-09-14 22:16:09 -07:00
Sumanth V Rao 033351609d markdown: Add data-codehilite-language attr for fenced code.
When converting fenced code markdown, we add the language (if specified)
in a data-attribute by tweaking the HTML generated. Doing so, allows the
frontend to make use of this attr to display view-in-playground option
for codeblocks.

We use pygments to get the lexer subclass name and use that instead of
directly using the language in the data-attribute. Doing so, helps us
map different language aliases (like `js` and `javascript`) into a common
variable (like `JavaScript`) - and avoids the client from dealing with
multiple tags corresponding to the same language.

The html structure for a message like this:

``` js
..content..
```

would now be:

<div class="codehilite" data-codehilite-language="JavaScript">
    <pre>..content..</pre>
</div>

Tests and fixtures amended.
2020-09-14 21:25:19 -07:00
Anders Kaseorg a659542d84 parallel: Remove run_parallel.
This was a broken abstraction that returned to its caller within
multiple forked processes on exceptions, and encouraged ignoring the
error code (as all of its callers did).

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-09-14 16:22:23 -07:00
Anders Kaseorg 0f16df2f13 transfer: Migrate from run_parallel to multiprocessing.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-09-14 16:22:23 -07:00
Anders Kaseorg 7f410ff0de import_realm: Migrate from run_parallel to multiprocessing.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-09-14 16:22:23 -07:00
Anders Kaseorg 73eb0aa891 test_runner: Monkey-patch to allow using multiprocessing in tests.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-09-14 16:22:23 -07:00
Aman Agrawal 2bc3924672 move_topic_to_stream: Allow moving to/between/from private streams.
Fixes #16284.

Most of the work for this was done when we implemented correct
behavior for guest users, since they treat public streams like private
streams anyway.

The general method involves moving the messages to the new stream with
special care of UserMessage.

We delete UserMessages for subs who are losing access to the message.
For private streams with protected history, we also create UserMessage
elements for users who are not present in the old stream, since that's
important for those users to access the moved messages.
2020-09-14 15:00:55 -07:00
Anders Kaseorg ddf8ec33df upload: Strip leading slash from deleted S3 export paths.
Previously, S3UploadBackend.delete_export_tarball failed to strip the
leading ‘/’ from the export path.  This mistake is now caught by Moto
1.3.15.  I expect it caused deletion failures in the real S3, although
I haven’t verified this.

We store export_path in the audit log with a leading ‘/’, but the
actual S3 keys do not have a leading ‘/’.  Changing either system
would require a migration.  So the new convention is that the
variables named ‘export_path’ have a leading ‘/’, while variables
named ‘path_id’ or ‘key’ do not.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-09-13 20:59:09 -07:00
Anders Kaseorg dfab09b17d markdown: Replace hyperlink requirement with urllib.parse.
The previous code only worked by accident and hyperlink 20.0.0 breaks
it.

>>> hyperlink.parse("example.com").replace(scheme="https")
DecodedURL(url=URL.from_text('https:example.com'))

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-09-13 15:37:28 -07:00
Anders Kaseorg 463929f349 urls: Migrate re_path routes to path.
Django treats path("<name>") like re_path(r"(?P<name>[^/]+)") and
path("<path:name>") like re_path(r"(?P<name>.+)").

This is more readable and consistent than the mix of slightly
different regexes we had before, and fixes various bugs:

• The r'apps/(.*)$' regex was missing a start anchor ^, so it
  incorrectly matched all URLs that included apps/ as a substring
  anywhere.
• The r'accounts/login/(google)/$' regex was missing a start anchor ^,
  so it incorrectly matched all URLs that ended with
  accounts/login/google/.
• The type annotation of zerver.views.realm_export.delete_realm_export
  takes export_id as an int, but it was previously passed as a string.
• The type annotation of zerver.views.users.avatar takes medium as a
  bool, but it was previously passed as a string.
• The [0-9A-Za-z]+ pattern for uidb64 was missing the - and _
  characters that can validly be part of a base64url encoded
  string (although I think the id is actually a decimal integer here,
  in which case only 012345ADEIMNOQTUYcgjkwxyz are present in its
  base64url encoding).

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-09-12 14:29:26 -07:00
Anders Kaseorg b41165fd01 dev_urls: Give the user_avatars URLPattern a name.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-09-12 11:21:40 -07:00
palash 34003fc7f6 email_mirror: Change root logger to zerver.lib.email_mirror. 2020-09-12 10:53:56 -07:00
Alex Vandiver 3f6e4ff303 webhooks: Move the extra logging information into a formatter.
This clears it out of the data sent to Sentry, where it is duplicative
with the indexed metadata -- and potentially exposes PHI if Sentry's
"make this issue public" feature is used.
2020-09-11 16:43:29 -07:00
Alex Vandiver 6323218a0e request: Maintain a thread-local of the current request.
This allows logging (to Sentry, or disk) to be annotated with richer
data about the request.
2020-09-11 16:43:29 -07:00
Alex Vandiver d04db7c5fe webhooks: Remove repetitive argument to UnsupportedWebhookEventType.
The name of the webhook can be added by the webhook decorator.
2020-09-10 17:47:21 -07:00
Alex Vandiver 9ea9752e0e webhooks: Rename UnexpectedWebhookEventType to UnsupportedWebhookEventType.
Any exception is an "unexpected event", which means talking about
having an "unexpected event logger" or "unexpected event exception" is
confusing.  As the error message in `exceptions.py` already explains,
this is about an _unsupported_ event type.

This also switches the path that these exceptions are written to,
accordingly.
2020-09-10 17:47:21 -07:00
Alex Vandiver 8016769613 webhooks: Move UnexpectedWebhookEventType into zerver.lib.exceptions.
8e10ab282a moved UnexpectedWebhookEventType into
`zerver.lib.exceptions`, but left the import into
`zserver.lib.webhooks.common` so that webhooks could continue to
import the exception from there.

This clutters things and adds complexity; there is no compelling
reason that the exception's source of truth should not move alongside
all other exceptions.
2020-09-10 17:47:21 -07:00
Mateusz Mandera f95dd628bd email: Fix race conditions with concurrent ScheduledEmail handling.
The main race conditions, which actually happened in production was with
concurrent execution of deliver_email and clear_scheduled_emails.
clear_scheduled_emails could delete all email.users in the middle of
deliver_email execution, causing it to pass empty to_user_ids list to
send_email. We mitigate this by getting the list of user ids in a single
query and moving forward with that snapshot, not having to worry about
database data being mutated anymore.

clear_scheduled_emails had potential race conditions with concurrent
execution of itself due to not locking the appropriate rows upon
selecting them for the purpose of potentially deleting them. FOR UPDATE
locks need to be acquired to prevent simultaneous mutation.

Tested manually with some print+sleep debugging to make some races
happen.

fixes #zulip-2k (sentry)
2020-09-09 15:59:01 -07:00
Anders Kaseorg b7b7475672 python: Use standard secrets module to generate random tokens.
There are three functional side effects:

• Correct an insignificant but mathematically offensive bias toward
repeated characters in generate_api_key introduced in commit
47b4283c4b4c70ecde4d3c8de871c90ee2506d87; its entropy is increased
from 190.52864 bits to 190.53428 bits.

• Use the base32 alphabet in confirmation.models.generate_key; its
entropy is reduced from 124.07820 bits to the documented 120 bits, but
now it uses 1 syscall instead of 24.

• Use the base32 alphabet in get_bigbluebutton_url; its entropy is
reduced from 51.69925 bits to 50 bits, but now it uses 1 syscall
instead of 10.

(The base32 alphabet is A-Z 2-7.  We could probably replace all of
these with plain secrets.token_urlsafe, since I expect most callers
can handle the full urlsafe_b64 alphabet A-Z a-z 0-9 - _ without
problems.)

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-09-09 15:52:57 -07:00
Dinesh c64888048f puppeteer: Rename CASPER_TESTS env variable to PUPPETEER_TESTS.
Also modified few comments to match with the changes.
2020-09-09 13:38:39 -04:00
Aman c114447e11 topic_history: Allow anonymous access in web-public streams.
For web-public streams, clients can access full topic history
without being authenticated. They only need to additionally
send "streams:web-public" narrow with their request like all
the other web-public queries.
2020-09-04 10:17:56 -07:00
Anders Kaseorg 3b301f522b python: Tweak some magic trailing commas to avoid Black bugs.
https://github.com/psf/black/issues/1658
https://github.com/psf/black/issues/1671

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-09-03 17:51:09 -07:00
Anders Kaseorg f91d287447 python: Pre-fix a few spots for better Black formatting.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-09-03 17:51:09 -07:00
Anders Kaseorg bb4fc3c4c7 python: Prefer --flag=option over --flag option.
For less inflation by Black.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-09-03 17:51:09 -07:00
Anders Kaseorg bef46dab3c python: Prefer kwargs form of dict.update.
For less inflation by Black.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-09-03 17:51:09 -07:00
Anders Kaseorg 75c59a820d python: Convert subprocess.Popen.communicate to run or check_output.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-09-03 17:42:35 -07:00
Vishnu KS 6bbcb622e1 onboarding: Move send_welcome_bot_response to onboarding. 2020-09-03 17:41:08 -07:00
Alex Vandiver 660c2e5383 tests: Add a mock to verify that queue_json_publish is valid JSON.
See also #16121.  However, that does not address the tests that
stub out the call to queue_json_publish itself.
2020-09-03 17:34:31 -07:00
Anders Kaseorg a50fae89e2 python: Elide type=str from argparse arguments.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-09-03 16:17:14 -07:00
Anders Kaseorg b4597a8ca8 python: Elide default for store_{true,false} argparse arguments.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-09-03 16:17:14 -07:00
Aman Agrawal d9431a5e66 exceptions: Raise InvalidSubdomainError when realm is invalid.
When user requests for a realm that doesn't exists, we raise
a InvalidSubdomainError.

This reduces our effort at repeatedly ensuring realm is valid
in request in web-public queries.
2020-09-03 10:45:14 -07:00
Anders Kaseorg c5a904fb05 django_api, test_classes: Use Python 3 form of super().
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-09-02 11:31:46 -07:00
Anders Kaseorg 02725d32dd python: Rewrite list() as [].
Suggested by the flake8-comprehensions plugin.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-09-02 11:15:41 -07:00
Anders Kaseorg a276eefcfe python: Rewrite dict() as {}.
Suggested by the flake8-comprehensions plugin.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-09-02 11:15:41 -07:00
Anders Kaseorg a610bd19a1 python: Simplify away various unnecessary lists and list comprehensions.
Loosely inspired by the flake8-comprehensions plugin.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-09-02 11:15:41 -07:00
Anders Kaseorg 0e5e6d0890 email_notifications: Convert list() of generator to comprehensions.
Suggested by the flake8-comprehensions plugin.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-09-02 11:15:41 -07:00
Anders Kaseorg ab120a03bc python: Replace unnecessary intermediate lists with generators.
Mostly suggested by the flake8-comprehension plugin.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-09-02 11:15:41 -07:00
Anders Kaseorg 1ded51aa9d python: Replace list literal concatenation with * unpacking.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-09-02 11:15:41 -07:00
Anders Kaseorg a5dbab8fb0 python: Remove redundant dest for argparse arguments.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-09-02 11:04:10 -07:00
Anders Kaseorg 72d2e5df15 isort: Enable black profile.
Our isort configuration was almost Black-compatible, but we were
missing ensure_newline_before_comments.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-09-02 11:00:07 -07:00
S-Abhishek 55a23754c3 api: Fix trailing '/' at end of jitsi server url.
Some users setup zulip with trailing / at end, like 'https://meet.jit.si/
leading to extra / on clients while generating video chat link.

This commit removes trailing '/' if it exists to make it consistent. Manual
testing was done by generating jitsi url.

Fixes #16225
2020-09-01 17:29:31 -07:00
Mateusz Mandera 9b50c49ea7 streams: Mark all messages as read when deactivating a stream.
The query to finds and marks all unread UserMessages in the stream as read
can be quite expensive, so we'll move that work to the deferred_work
queue and split it into batches.

Fixes #15770.
2020-09-01 11:24:27 -07:00
Hashir Sarwar 8ebff434fb push_notifications: Return token from add_push_device_token(). 2020-08-31 17:34:23 -07:00
Hashir Sarwar b885678881 push_notifications: Simplify `if device exists` checks. 2020-08-31 17:31:41 -07:00
Vishnu KS 8a993aa7e7 hotspots: Use ugettext_lazy for i18n.
Since ALL_HOTSPOTS is a global object, it is initialized
at the time the backend server is started. Hence, the
title and description is translated only once. Using
ugettext_lazy makes sure that the strings are translated
in each and every request according to the language
of the user.

Fixes #16224
2020-08-31 11:22:54 -07:00
orientor 148c375e5b events: Add documentation and tests for `typing: stop` event.
The `typing: stop` event did not have any tests in test_events
hence its documentation wasn't added. So add tests and relevant
documentation for the typing stop event. Also edit the documentation
of `typing: start` to include the fact that servers should use
their own timeout incase `stop` event event isn't received.

Fixes #16122.
2020-08-30 16:43:44 -07:00
Tim Abbott 5548ab8b99 rest: Simplify authentication error handling.
This pure refactor removes a now unnecessarily nested else clause,
helping keep this key piece of code readable.
2020-08-30 14:51:50 -07:00
Aman fd5423a8f9 exceptions: Extract json_unauths into MissingAuthenticationError.
We raise two types of json_unauthorized when
MissingAuthenticationError is raised. Raising the one
with www_authenticate let's the client know that user needs
to be logged in to access the requested content.

Sending `www_authenticate='session'` header with the response
also stops modern web-browsers from showing a login form to the
user and let's the client handle it completely.

Structurally, this moves the handling of common authentication errors
to a single shared middleware exception handler.
2020-08-30 14:51:50 -07:00
Alex Vandiver 81893c9dbb actions: Invalid flag operation is a user error. 2020-08-29 11:38:59 -04:00
Anders Kaseorg 468c5b9a58 tests: Make tests pass with zilencer disabled.
This lets the backend tests pass if zilencer has been (manually)
removed from EXTRA_INSTALLED_APPS, by skipping the tests that require
it.  test-backend complains that some URLs are untested in this case:

ERROR: Some URLs are untested!  Here's the list of untested URLs:
   api/v1/users/me/android_gcm_reg_id
   api/v1/users/me/apns_device_token
   team/

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-08-28 11:34:09 -07:00
Anders Kaseorg 51f993e084 python: Remove unittest.mock.Mock uses from production code.
It’s somewhat expensive to import and confuses mypy.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-08-28 11:34:09 -07:00
Mohit Gupta 3390a70bcd tests: Add extra console output detection in test-backend output.
This commit adds automatic detection of extra output (other than
printed by testing library or tools) in stderr and stdout by code under
test test-backend when it is run with flag --ban-console-output.
It also prints the test that produced the extra console output.

Fixes: #1587.
2020-08-27 11:39:53 -07:00
Tim Abbott 1fddf16b73 Revert "exceptions: Extract json_unauths into MissingAuthenticationError."
This reverts commit c355f6b8d8.
2020-08-25 17:42:07 -07:00
Tim Abbott 5dceee5bd8 Revert "exceptions: Remove dead code msg_format method."
This reverts commit d6ab7ea8ff.
2020-08-25 17:42:07 -07:00
Tim Abbott fd3a43b51e Revert "rest: Simplify authentication error handling."
This reverts commit 1f90a31fa3.
2020-08-25 17:42:07 -07:00
Tim Abbott d6ab7ea8ff exceptions: Remove dead code msg_format method. 2020-08-25 17:17:15 -07:00
Tim Abbott 3b2a262b6f topic: Reorder topic history functions. 2020-08-25 17:03:48 -07:00
Tim Abbott 88a28d5470 topic: Refactor get_topic_history_for_stream.
This now uses get_topic_history_for_public_stream as a subroutine, to
avoid duplicating that large section of SQL.
2020-08-25 17:03:13 -07:00
Aman c3a8492697 topic: Rename get_topic_history_for_web_public_stream. 2020-08-25 17:01:12 -07:00
Tim Abbott 1f90a31fa3 rest: Simplify authentication error handling.
This pure refactor removes a now unnecessarily nested else clause,
helping keep this key piece of code readable.
2020-08-25 16:59:05 -07:00
Aman c355f6b8d8 exceptions: Extract json_unauths into MissingAuthenticationError.
We raise two types of json_unauthorized when
MissingAuthenticationError is raised. Raising the one
with www_authenticate let's the client know that user needs
to be logged in to access the requested content.

Sending `www_authenticate='session'` header with the response
also stops modern web-browsers from showing a login form to the
user and let's the client handle it completely.

Structurally, this moves the handling of common authentication errors
to a single shared middleware exception handler.
2020-08-25 16:52:21 -07:00
Alex Vandiver b4c2ae9cae settings: Adjust documentation and comment references to settings.py.
`zproject/settings.py` itself is mostly-empty now.  Adjust the
references which should now point to `zproject/computed_settings.py`
or `zproject/default_settings.py`.
2020-08-24 13:13:16 -07:00
orientor 372e010dbb events: Add `op` field to `update_message_flags` events.
`update_message_flags` events used `operation` instead of `op`, the
latter being the standard field used in other events. So add `op`
field to `update_message_flags` and mark `operation` as deprecated,
so that it can be removed later.
2020-08-24 12:42:03 -07:00
Steve Howell 95fe690e8d webhook tests: Avoid lots of unnecessary json dumping.
Commit c4254497b2
curiously had get_body() round tripping its data
through json load and dump.

I have seen this done for pretty-printing reasons,
but it doesn't apply here.

And if you're doing it for validation reasons,
you only need to do half the work, as my commit
here demonstrates.

We arguably don't even need the fail-fast code
here, since our fixtures are linted to be proper
json, I believe, plus downstream code probably
gives reasonably easy-to-diagnose symptoms.
2020-08-24 12:34:46 -07:00
Steve Howell 7fbe08f515 webhook tests: Introduce get_payload.
We introduce get_payload for the relatively
exceptional cases where webhooks return payloads
as dicts.

Having a simple "str" type for get_body will
allow us to extract test helpers that use
payloads from get_body() without the ugly
`Union[str, Dict[str, str]]` annotations.

I also tightened up annotations in a few places
where we now call get_payload (using Dict[str, str]
instead of Dict[str, Any]).

In the zendesk test I explicitly stringify
one of the parameters to satisfy mypy.
2020-08-24 12:34:46 -07:00
Steve Howell 025d1fa180 webhook tests: Require args for check_webhook.
We tighten up the mypy types here.  And then
once we know that expected_message and expected_topic
are never None, we don't have call the do_test_message
and do_test_topic helpers any more, so we eliminate
them, too.

Finally, we don't return a message, since no tests
use the message currently.
2020-08-24 12:34:46 -07:00
Steve Howell f74aa29a1c webhook tests: Extract assert_stream_message.
This forces us to be a bit more explicit about testing
the three key values in any stream message, and it
also de-clutters the code a bit.  I eventually want
to phase out do_test_topic and friends, since they
have the pitfall that you can call them and have them
do nothing, because they don't actually require
values to be be passed in.

I also clean up the code a bit for the tests that
have two new messages arriving.
2020-08-24 12:34:46 -07:00
Steve Howell 3a710ab996 webhook tests: Remove stream_name parameter.
Having an optional stream_name parameter makes
it confusing to read the code if you know your
webhook is sending private messages.

And then the other two callers are already
checking topics, so they might as well check
stream names, too.

We also have the two stream-oriented callers
make their own call to "subscribe".  And we
future-proof this by making sure the exception
for no-message-being-sent calls out that gotcha.

Somewhat in passing, we now assert that
self.STREAM_NAME is not None in the main
helper.  This is partly to satisfy mypy, but
it's also a good sanity check.

This also sets the stage for the next commit,
where I'll add an assert_stream_message helper.
2020-08-24 12:34:46 -07:00
Steve Howell 2fdf966d9f webhook tests: Rename helper to send_webhook_payload.
Not all webhook payloads are json, so send_json_payload was a
bit misleading.

In passing I also remove "bytes" from the Union type for
"payload" parameter.
2020-08-24 12:34:46 -07:00
Steve Howell 388053db6b webhook tests: Rename main helper to check_webhook.
Almost all webhook tests use this helper, except a few
webhooks that write to private streams.

Being concise is important here, and the name
`self.send_and_test_stream_message` always confused
me, since it sounds you're sending a stream message,
and it leaves out the webhook piece.

We should consider renaming `send_and_test_private_message`
to something like `check_webhook_private`, but I couldn't
decide on a great name, and it's very rarely used.  So
for now I just made sure the docstrings of the two
sibling functions reference each other.
2020-08-24 12:34:46 -07:00
Steve Howell 0f970c6971 minor: Avoid do_test_message.
This function is a bad idea, as it leads to a possible situation
where you aren't actually testing anything:

    def do_test_message(self, msg: Message, expected_message: Optional[str]) -> None:
        if expected_message is not None:
            self.assertEqual(msg.content, expected_message)

Unfortunately, it's called deep in the stack in some places, but
we can safely replace it with assertEqual here.
2020-08-24 12:34:46 -07:00
Steve Howell 3de2e4f034 dialogflow test: Fix nonsensical tests.
The test helper here was taking an "expected_topic"
parameter that it just ignored, and then the
dialogflow tests were passing in expected messages
in that slot, so the actual "expected_message" var
was "None" and was ignored.  So the tests weren't
testing anything.

Now we eliminate the crufty expected_topic parameter
and require an actual value for "expected_message".

I also clean up the mypy type for content_type,
and I remove the `content_type is None` check,
since all callers either pass in a str content
type or default to "application/json".
2020-08-24 12:34:46 -07:00
Alex Vandiver ad8943a64a url_preview: Only extract img tags with an `src`.
Some `<img>` tags do not have an SRC, if they are rewritten using JS
to have one later.  Attempting to access `first_image['src']` on these
will raise an exception, as they have no such attribute.

Only look for images which have a defined `src` attribute on them.  We
could instead check if `first_image.has_attr('src')`, but this seems
only likely to produce fewer valid images.
2020-08-18 14:26:21 -04:00
Alex Vandiver 5b74de7be7 markdown: Add another twitter code to retry-later.
Error code 131 is documented to be an arbitrary server error on
Twitter's side; add it to the retry list.
2020-08-18 10:32:24 -07:00
Alex Vandiver 092ed87ae3 markdown: Cache Twitter 403 responses that are semi-permanent.
03ca3afbc2 added more codes that are equivalent to 404's; this adds to
the list of cache-as-None codes a couple which are equivalent to
403's.  It does not comprise _all_ possible 403-like codes -- many of
them are "the client is not OK," which is relevant to log as an error
still.
2020-08-18 10:32:24 -07:00
sahil839 f046c9c58a streams: Add role field to Subscription objects passed to clients.
This commit adds "role" field to the Subscription objects passed to
clients.  This is important preparation for being able to work on the
frontend for this feature.
2020-08-14 16:33:11 -07:00
Tim Abbott f94a8adf9e export: Remove duplicate 'analytics' zerver_realm object.
This fixes a harmless duplication of data in the Zulip data export
format.
2020-08-14 15:45:11 -07:00
arpit551 7568f6f9a8 export: Renamed zerver_analytics to zerver_realm.
While exporting analytics data we were using wrong table name
'zerver_analytics' in analytics config. Renamed it with
correct table name 'zerver_realm'.
2020-08-14 15:45:11 -07:00
arpit551 af3a34fbca cache: Used lru_cache from functools instead of django.utils.lru_cache.
Django 3.0 removed private Python 2 compatibility APIs
so used lru_cache() directly from functools.

We cast lru_cache to Any to avoid attr-defined error in mypy since we
are adding extra field, 'key_prefix', to this object later.
2020-08-14 11:34:04 -07:00
Alex Vandiver 2b6989a40f queue: Remove a no-longer-correct comment.
This comment stopped being true in 5686821150, and very much stopped
being relevant in dd40649e04 when the middleware entirely stopped
publishing to a queue.
2020-08-14 11:30:13 -07:00
Clara Dantas 05bf72a75c attachments: Add is_web_public field.
This commit adds the is_web_public field in the AbstractAttachment
class. This is useful when validating user access to the attachment,
as otherwise we would have to make a query in the db to check if
that attachment was sent in a message in a web-public stream or not.
2020-08-12 17:26:03 -07:00
sahil839 ca1a8ac78f streams: Allow stream admin to update and deactivate streams.
The new Stream administrator role is allowed to manage a stream they
administer, including:
* Setting properties like name, description, privacy and post-policy.
* Removing subscribers
* Deactivating the stream

The access_stream_for_delete_or_update is modified and is used only
to get objects from database and further checks for administrative
rights is done by check_stream_access_for_delete_or_update.

We have also added a new exception class StreamAdministratorRequired.
2020-08-12 17:02:01 -07:00
Aman Agrawal 9f9daeea5b message_fetch: Allow access to web-public msgs for unauth users.
Via API, users can now access messages which are in web-public
streams without any authentication.

If the user is not authenticated, we assume it is a web-public
query and add `streams:web-public` narrow if not already present
to the narrow. web-public streams are also directly accessible.

Any malformed narrow which is not allowed in a web-public query
results in a 400 or 401. See test_message_fetch for the allowed
queries.
2020-08-12 16:39:02 -07:00
Anders Kaseorg d0f4af5f8c python: Catch JSONDecodeError instead of ValueError when decoding JSON.
These weren’t wrong since orjson.JSONDecodeError subclasses
json.JSONDecodeError which subclasses ValueError, but the more
specific ones express the intention more clearly.

(ujson raised ValueError directly, as did json in Python 2.)

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-08-12 11:59:59 -07:00
Alex Vandiver 153f16ee6a links: Flatten the set into a list before serializing into the queue.
orjson does not transparently do this set-to-list translation, unlike
ujson.
2020-08-12 11:42:24 -07:00
Anders Kaseorg 61d0417e75 python: Replace ujson with orjson.
Fixes #6507.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-08-11 10:55:12 -07:00
Anders Kaseorg 03d2540899 export: Post-process authentication_methods BitHandler field to list.
A BitHandler object is not JSON serializable, and orjson enforces
this.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-08-11 10:47:13 -07:00
Anders Kaseorg 2cf2547b27 export: Add missing datetime fields for post-processing.
datetime objects are not ordinarily JSON serializable.  While both
ujson and orjson have special cases to serialize datetime objects,
they do it in different ways.  So we want to fix the post-processing
code to do its job.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-08-11 10:47:13 -07:00
Anders Kaseorg 768f9f93cd docs: Capitalize Markdown consistently.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-08-11 10:23:06 -07:00
Anders Kaseorg 60a25b2721 docs: Fix spelling errors caught by codespell.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-08-11 10:23:06 -07:00
Alex Vandiver 2928bbc8bd logging: Report stack_info on logging.exception calls.
The exception trace only goes from where the exception was thrown up
to where the `logging.exception` call is; any context as to where
_that_ was called from is lost, unless `stack_info` is passed as well.
Having the stack is particularly useful for Sentry exceptions, which
gain the full stack trace.

Add `stack_info=True` on all `logging.exception` calls with a
non-trivial stack; we omit `wsgi.py`.  Adjusts tests to match.
2020-08-11 10:16:54 -07:00
Alex Vandiver 39368cad3a tornado: Extract functions called from django into one module.
This makes clearer the separation of concerns.
2020-08-10 16:55:56 -07:00
Tim Abbott e12225ace7 realm_audit_log: Fix malformed RealmAuditLog entries.
In f8bcf39014, we fixed buggy
marshalling of Streams and similar data structures where we were
including the Stream object rather than its ID in dictionaries passed
to ujson, and ujson happily wrote that large object dump into the
RealmAuditLog.extra_data field.

This commit includes a migration to fix those corrupted RealmAuditLog
entries, and because the migration loop is the same, also fixes the
format of similar RealmAuditLog entries to be in a more natural format
that doesn't weirdly nest and duplicate the "property" field.

Fixes #16066.
2020-08-09 16:51:29 -07:00
Steve Howell 6f97e9dfa9 mypy: Use object, not Any, in event_schema.
This requires a few redundant runtime isinstance
checks, but the extra assertions arguably make
the code more readable, and isinstance checks
are extremely negligible.
2020-08-08 11:30:46 -04:00
Tim Abbott ba34e2a61c help: Mark Zoom integration as stable.
We've had plenty of happy production usage at this point.
2020-08-07 16:45:30 -07:00
Anders Kaseorg 23b815bb50 queue: Fix types to reflect that Pika channels receive bytes.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-08-07 11:12:32 -07:00
Anders Kaseorg 3e7503a607 html_diff: Really return str, not bytes.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-08-07 11:12:32 -07:00
Anders Kaseorg fe5c524890 user_status: Use strings as dict keys.
JSON keys must be strings, and orjson enforces this.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-08-07 10:53:06 -07:00
Anders Kaseorg a329b538d2 test_helpers: Fix instrument_url decorator type.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-08-07 10:53:06 -07:00
Anders Kaseorg 44af001045 custom_profile_fields: Use strings as dict keys.
JSON keys must be strings, and orjson enforces this.  Mypy didn’t
catch the mismatched type of profiles_by_user_id because it doesn’t
understand CustomProfileFieldValue.field_id.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-08-07 10:53:06 -07:00
Anders Kaseorg b250e42f61 realm_user_count_by_role: Use strings as dict keys.
JSON keys must be strings, and orjson enforces this.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-08-07 10:53:06 -07:00
Anders Kaseorg f8bcf39014 actions: Do not attempt JSON serialization of a Stream object.
It doesn’t end well.  Or sometimes it doesn’t end (OverflowError:
Maximum recursion level reached).

Introduced by commits ccdf52fef6 and
94d2de8b4a (#15601).

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-08-07 10:53:06 -07:00
Steve Howell 15ffd2b666 event_schema: Extract check_stream_delete. 2020-08-06 12:29:43 -07:00
Steve Howell 059e0bb81e event_schema: Extract check_default_streams. 2020-08-06 12:29:43 -07:00
Steve Howell 36ec1571d4 event_schema: Extract check_default_stream_groups. 2020-08-06 12:29:43 -07:00
Tim Abbott 00fd9afad5 embed: Remove useless 'sender' field.
The variant `update_message` events have this extra sender field not
present in normal update_message events; this field has no purpose, so
we remove it.
2020-08-03 18:04:38 -07:00
Tim Abbott 99a54ba67e tornado: Fix ID lists leaked to the events API.
Apparently, `update_message` events unexpectedly contained what were
intended to be internal data structures about which users were
mentioned in a given message.

The bug has been present and accumulating new data structures for
years.

Fixing this should improve the performance of handling update_message
events as well as cleaning up this API's interface.

This was discovered by our automated API documentation schema checking
tooling detecting these unexpected elements in these event
definitions; that same logic should prevent future bugs like this from
being introduced in the future.
2020-08-03 17:52:39 -07:00
Clara Dantas f26d0affef bulk_update: Do updates in bulk where applicable. 2020-08-03 16:16:42 -07:00
Clara Dantas be6b2b248f Error reporting emails: Indicate user role.
Adds user role to the user's information in the error reporting
emails, as some bugs are role-dependent.

Fixes: #15344
2020-08-03 13:35:51 -07:00
Clara Dantas b4dd118aa1 refactor: Create a user object in report.
To make it easier to check if there is user information to be used
in the error report emails, we create a user object inside report.
Now, to check if we have the user's full name, email, etc, we just
need to do report['user']['user_full_name'] rather than check
each information one by one, because if the value of one key in
the report is different than None, all the others will be as well.
2020-08-03 13:35:51 -07:00
Anders Kaseorg 159641bab8 timestamp: Remove datetime_to_precise_timestamp for datetime.timestamp.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-07-31 22:10:14 -07:00
Alex Vandiver 90cdda9836 markdown: Link the twitter response code docs inline. 2020-07-31 10:35:41 -07:00
Alex Vandiver 03ca3afbc2 markdown: Treat more twitter codes as also permanent failures.
Per the API documentation[1], the following codes all correspond to
HTTP 404:

 - `34`: **Sorry, that page does not exist.**  The specified resource
   was not found.
 - `144`: **No status found with that ID.**  The requested Tweet ID is
   not found (if it existed, it was probably deleted)
 - `421`: **This Tweet is no longer available.**  The Tweet cannot be
   retrieved. This may be for a number of reasons.
 - `422`: **This Tweet is no longer available because it violated the
   Twitter Rules.**  The Tweet is not available in the API.

Treat all of these identically.

[1] https://developer.twitter.com/en/docs/basics/response-codes
2020-07-31 10:35:41 -07:00
Alex Vandiver fc141af30e markdown: Factor out twitter error code handling. 2020-07-31 10:35:41 -07:00
Clara Dantas 6b011b241b actions: Remove duplicated conditional.
In the gather_subscriptions_helper functions, there was a repeated
conditional inside one another.
2020-07-30 16:59:51 -07:00
Tim Abbott 6130a61be0 export: Only print .s with percent_callback to console.
The S3 data export tool's upload code path uses this nice boto
callback feature for showing a progress bar, which is nice for the
management command.  It's spammy/broken in production and the backend
tests, so we change percent_callback to be a parameter passed in so
that it can only be used in the contexts where it makes sense.
2020-07-30 13:14:53 -07:00
Anders Kaseorg 022c4fbfc7 Revert "digest: Support digest of web public streams for guest users."
This reverts commit c3779338c6 (part
of #14638), which incorrectly depended on commits from the future,
with the effect of either halting the flow of entropic time in an
irresolvable temporal paradox, summoning extradimensional beings to
rain destruction on the galaxy, or failing CI.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-07-29 21:05:59 -07:00
Clara Dantas c3779338c6 digest: Support digest of web public streams for guest users. 2020-07-29 17:52:36 -07:00
Clara Dantas a9af80d7a2 streams: Make /streams endpoint return also web-public streams.
This commit modifies the /streams endpoint so that the web-public
streams are included in the default list of streams that users
have access to.

This is part of PR #14638 that aims to allow guest users to
browse and subscribe themselves to web public streams.
2020-07-29 17:52:36 -07:00
Clara Dantas 0994b029d6 streams: Grant authorization to guest users to subscribe.
Modifies filter_stream_authorization so that web-public streams are
added in the list of authorized streams that a guest user can
subscribe.

This commit is part of PR #14638 that aims to allow guest users
to browse and subscribe to web-public streams.
2020-07-29 17:52:36 -07:00
Clara Dantas e22e12fe60 streams: Grant guest users access to web-public streams.
In this commit, we grant guest users access to stream history,
send message and common stream data of web-public streams.

This is part of PR #14638 that aims to allow guest users to
browse and subscribe to web-public streams.
2020-07-29 17:52:36 -07:00
Clara Dantas 523bb30f33 actions: Modify validate_user_access_to_subscribers.
This modification allows guest users to have access to web-public
streams subscribers, even if they aren't subscribed or never
subscribed to that stream.

This commit is part of PR #14638 that aims to allow guest users to
browser and subscribe to web-public streams.
2020-07-29 17:52:36 -07:00
Clara Dantas 6a1d9480bc actions: Modify gather_subscriptions() to also gather web-public streams.
Now, gather_subscriptions include web-public streams in the 3 sets
of streams that it returns, subscribed, unsubscribed and never
subscribed.

This is part of PR #14638 that aims to allow guest users to browse and
subscribe to web-public streams.
2020-07-29 17:52:36 -07:00
Clara Dantas fda94a49c1 streams: Change flow in filter_stream_authorization.
This change makes the flow more coherent by instead of checking,
in the last condition, if the user isn't authorized to access that
stream, check if they are, as it is done in the other checks. Only
if all the conditions are false, which means that the user doesn't
have access to that stream, the stream is added to the
unauthorized_streams list.
2020-07-29 17:52:36 -07:00
Gittenburg 0706de2305 docs: Make tabbed sections accessible from keyboard.
Part of #15948.
2020-07-29 11:40:39 -07:00
Hemanth V. Alluri 0e893b9045 models/drafts: Add a model for storing Draft messages.
Also add a Draft object-to-dictionary conversion method.
The following commits will provide an API around this
model using which our clients can sync drafts across each
other (if they so wish too). As of making this commit, we
haven't finalized exactly how our clients will use this.

See https://chat.zulip.org/#narrow/stream/2-general/topic/drafts
For some of the discussion around this model and in general,
around this feature.

Signed-off-by: Hemanth V. Alluri <hdrive1999@gmail.com>
2020-07-28 17:18:35 -07:00