Commit Graph

5605 Commits

Author SHA1 Message Date
Aman c355f6b8d8 exceptions: Extract json_unauths into MissingAuthenticationError.
We raise two types of json_unauthorized when
MissingAuthenticationError is raised. Raising the one
with www_authenticate let's the client know that user needs
to be logged in to access the requested content.

Sending `www_authenticate='session'` header with the response
also stops modern web-browsers from showing a login form to the
user and let's the client handle it completely.

Structurally, this moves the handling of common authentication errors
to a single shared middleware exception handler.
2020-08-25 16:52:21 -07:00
Alex Vandiver b4c2ae9cae settings: Adjust documentation and comment references to settings.py.
`zproject/settings.py` itself is mostly-empty now.  Adjust the
references which should now point to `zproject/computed_settings.py`
or `zproject/default_settings.py`.
2020-08-24 13:13:16 -07:00
orientor 372e010dbb events: Add `op` field to `update_message_flags` events.
`update_message_flags` events used `operation` instead of `op`, the
latter being the standard field used in other events. So add `op`
field to `update_message_flags` and mark `operation` as deprecated,
so that it can be removed later.
2020-08-24 12:42:03 -07:00
Steve Howell 95fe690e8d webhook tests: Avoid lots of unnecessary json dumping.
Commit c4254497b2
curiously had get_body() round tripping its data
through json load and dump.

I have seen this done for pretty-printing reasons,
but it doesn't apply here.

And if you're doing it for validation reasons,
you only need to do half the work, as my commit
here demonstrates.

We arguably don't even need the fail-fast code
here, since our fixtures are linted to be proper
json, I believe, plus downstream code probably
gives reasonably easy-to-diagnose symptoms.
2020-08-24 12:34:46 -07:00
Steve Howell 7fbe08f515 webhook tests: Introduce get_payload.
We introduce get_payload for the relatively
exceptional cases where webhooks return payloads
as dicts.

Having a simple "str" type for get_body will
allow us to extract test helpers that use
payloads from get_body() without the ugly
`Union[str, Dict[str, str]]` annotations.

I also tightened up annotations in a few places
where we now call get_payload (using Dict[str, str]
instead of Dict[str, Any]).

In the zendesk test I explicitly stringify
one of the parameters to satisfy mypy.
2020-08-24 12:34:46 -07:00
Steve Howell 025d1fa180 webhook tests: Require args for check_webhook.
We tighten up the mypy types here.  And then
once we know that expected_message and expected_topic
are never None, we don't have call the do_test_message
and do_test_topic helpers any more, so we eliminate
them, too.

Finally, we don't return a message, since no tests
use the message currently.
2020-08-24 12:34:46 -07:00
Steve Howell f74aa29a1c webhook tests: Extract assert_stream_message.
This forces us to be a bit more explicit about testing
the three key values in any stream message, and it
also de-clutters the code a bit.  I eventually want
to phase out do_test_topic and friends, since they
have the pitfall that you can call them and have them
do nothing, because they don't actually require
values to be be passed in.

I also clean up the code a bit for the tests that
have two new messages arriving.
2020-08-24 12:34:46 -07:00
Steve Howell 3a710ab996 webhook tests: Remove stream_name parameter.
Having an optional stream_name parameter makes
it confusing to read the code if you know your
webhook is sending private messages.

And then the other two callers are already
checking topics, so they might as well check
stream names, too.

We also have the two stream-oriented callers
make their own call to "subscribe".  And we
future-proof this by making sure the exception
for no-message-being-sent calls out that gotcha.

Somewhat in passing, we now assert that
self.STREAM_NAME is not None in the main
helper.  This is partly to satisfy mypy, but
it's also a good sanity check.

This also sets the stage for the next commit,
where I'll add an assert_stream_message helper.
2020-08-24 12:34:46 -07:00
Steve Howell 2fdf966d9f webhook tests: Rename helper to send_webhook_payload.
Not all webhook payloads are json, so send_json_payload was a
bit misleading.

In passing I also remove "bytes" from the Union type for
"payload" parameter.
2020-08-24 12:34:46 -07:00
Steve Howell 388053db6b webhook tests: Rename main helper to check_webhook.
Almost all webhook tests use this helper, except a few
webhooks that write to private streams.

Being concise is important here, and the name
`self.send_and_test_stream_message` always confused
me, since it sounds you're sending a stream message,
and it leaves out the webhook piece.

We should consider renaming `send_and_test_private_message`
to something like `check_webhook_private`, but I couldn't
decide on a great name, and it's very rarely used.  So
for now I just made sure the docstrings of the two
sibling functions reference each other.
2020-08-24 12:34:46 -07:00
Steve Howell 0f970c6971 minor: Avoid do_test_message.
This function is a bad idea, as it leads to a possible situation
where you aren't actually testing anything:

    def do_test_message(self, msg: Message, expected_message: Optional[str]) -> None:
        if expected_message is not None:
            self.assertEqual(msg.content, expected_message)

Unfortunately, it's called deep in the stack in some places, but
we can safely replace it with assertEqual here.
2020-08-24 12:34:46 -07:00
Steve Howell 3de2e4f034 dialogflow test: Fix nonsensical tests.
The test helper here was taking an "expected_topic"
parameter that it just ignored, and then the
dialogflow tests were passing in expected messages
in that slot, so the actual "expected_message" var
was "None" and was ignored.  So the tests weren't
testing anything.

Now we eliminate the crufty expected_topic parameter
and require an actual value for "expected_message".

I also clean up the mypy type for content_type,
and I remove the `content_type is None` check,
since all callers either pass in a str content
type or default to "application/json".
2020-08-24 12:34:46 -07:00
Alex Vandiver ad8943a64a url_preview: Only extract img tags with an `src`.
Some `<img>` tags do not have an SRC, if they are rewritten using JS
to have one later.  Attempting to access `first_image['src']` on these
will raise an exception, as they have no such attribute.

Only look for images which have a defined `src` attribute on them.  We
could instead check if `first_image.has_attr('src')`, but this seems
only likely to produce fewer valid images.
2020-08-18 14:26:21 -04:00
Alex Vandiver 5b74de7be7 markdown: Add another twitter code to retry-later.
Error code 131 is documented to be an arbitrary server error on
Twitter's side; add it to the retry list.
2020-08-18 10:32:24 -07:00
Alex Vandiver 092ed87ae3 markdown: Cache Twitter 403 responses that are semi-permanent.
03ca3afbc2 added more codes that are equivalent to 404's; this adds to
the list of cache-as-None codes a couple which are equivalent to
403's.  It does not comprise _all_ possible 403-like codes -- many of
them are "the client is not OK," which is relevant to log as an error
still.
2020-08-18 10:32:24 -07:00
sahil839 f046c9c58a streams: Add role field to Subscription objects passed to clients.
This commit adds "role" field to the Subscription objects passed to
clients.  This is important preparation for being able to work on the
frontend for this feature.
2020-08-14 16:33:11 -07:00
Tim Abbott f94a8adf9e export: Remove duplicate 'analytics' zerver_realm object.
This fixes a harmless duplication of data in the Zulip data export
format.
2020-08-14 15:45:11 -07:00
arpit551 7568f6f9a8 export: Renamed zerver_analytics to zerver_realm.
While exporting analytics data we were using wrong table name
'zerver_analytics' in analytics config. Renamed it with
correct table name 'zerver_realm'.
2020-08-14 15:45:11 -07:00
arpit551 af3a34fbca cache: Used lru_cache from functools instead of django.utils.lru_cache.
Django 3.0 removed private Python 2 compatibility APIs
so used lru_cache() directly from functools.

We cast lru_cache to Any to avoid attr-defined error in mypy since we
are adding extra field, 'key_prefix', to this object later.
2020-08-14 11:34:04 -07:00
Alex Vandiver 2b6989a40f queue: Remove a no-longer-correct comment.
This comment stopped being true in 5686821150, and very much stopped
being relevant in dd40649e04 when the middleware entirely stopped
publishing to a queue.
2020-08-14 11:30:13 -07:00
Clara Dantas 05bf72a75c attachments: Add is_web_public field.
This commit adds the is_web_public field in the AbstractAttachment
class. This is useful when validating user access to the attachment,
as otherwise we would have to make a query in the db to check if
that attachment was sent in a message in a web-public stream or not.
2020-08-12 17:26:03 -07:00
sahil839 ca1a8ac78f streams: Allow stream admin to update and deactivate streams.
The new Stream administrator role is allowed to manage a stream they
administer, including:
* Setting properties like name, description, privacy and post-policy.
* Removing subscribers
* Deactivating the stream

The access_stream_for_delete_or_update is modified and is used only
to get objects from database and further checks for administrative
rights is done by check_stream_access_for_delete_or_update.

We have also added a new exception class StreamAdministratorRequired.
2020-08-12 17:02:01 -07:00
Aman Agrawal 9f9daeea5b message_fetch: Allow access to web-public msgs for unauth users.
Via API, users can now access messages which are in web-public
streams without any authentication.

If the user is not authenticated, we assume it is a web-public
query and add `streams:web-public` narrow if not already present
to the narrow. web-public streams are also directly accessible.

Any malformed narrow which is not allowed in a web-public query
results in a 400 or 401. See test_message_fetch for the allowed
queries.
2020-08-12 16:39:02 -07:00
Anders Kaseorg d0f4af5f8c python: Catch JSONDecodeError instead of ValueError when decoding JSON.
These weren’t wrong since orjson.JSONDecodeError subclasses
json.JSONDecodeError which subclasses ValueError, but the more
specific ones express the intention more clearly.

(ujson raised ValueError directly, as did json in Python 2.)

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-08-12 11:59:59 -07:00
Alex Vandiver 153f16ee6a links: Flatten the set into a list before serializing into the queue.
orjson does not transparently do this set-to-list translation, unlike
ujson.
2020-08-12 11:42:24 -07:00
Anders Kaseorg 61d0417e75 python: Replace ujson with orjson.
Fixes #6507.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-08-11 10:55:12 -07:00
Anders Kaseorg 03d2540899 export: Post-process authentication_methods BitHandler field to list.
A BitHandler object is not JSON serializable, and orjson enforces
this.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-08-11 10:47:13 -07:00
Anders Kaseorg 2cf2547b27 export: Add missing datetime fields for post-processing.
datetime objects are not ordinarily JSON serializable.  While both
ujson and orjson have special cases to serialize datetime objects,
they do it in different ways.  So we want to fix the post-processing
code to do its job.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-08-11 10:47:13 -07:00
Anders Kaseorg 768f9f93cd docs: Capitalize Markdown consistently.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-08-11 10:23:06 -07:00
Anders Kaseorg 60a25b2721 docs: Fix spelling errors caught by codespell.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-08-11 10:23:06 -07:00
Alex Vandiver 2928bbc8bd logging: Report stack_info on logging.exception calls.
The exception trace only goes from where the exception was thrown up
to where the `logging.exception` call is; any context as to where
_that_ was called from is lost, unless `stack_info` is passed as well.
Having the stack is particularly useful for Sentry exceptions, which
gain the full stack trace.

Add `stack_info=True` on all `logging.exception` calls with a
non-trivial stack; we omit `wsgi.py`.  Adjusts tests to match.
2020-08-11 10:16:54 -07:00
Alex Vandiver 39368cad3a tornado: Extract functions called from django into one module.
This makes clearer the separation of concerns.
2020-08-10 16:55:56 -07:00
Tim Abbott e12225ace7 realm_audit_log: Fix malformed RealmAuditLog entries.
In f8bcf39014, we fixed buggy
marshalling of Streams and similar data structures where we were
including the Stream object rather than its ID in dictionaries passed
to ujson, and ujson happily wrote that large object dump into the
RealmAuditLog.extra_data field.

This commit includes a migration to fix those corrupted RealmAuditLog
entries, and because the migration loop is the same, also fixes the
format of similar RealmAuditLog entries to be in a more natural format
that doesn't weirdly nest and duplicate the "property" field.

Fixes #16066.
2020-08-09 16:51:29 -07:00
Steve Howell 6f97e9dfa9 mypy: Use object, not Any, in event_schema.
This requires a few redundant runtime isinstance
checks, but the extra assertions arguably make
the code more readable, and isinstance checks
are extremely negligible.
2020-08-08 11:30:46 -04:00
Tim Abbott ba34e2a61c help: Mark Zoom integration as stable.
We've had plenty of happy production usage at this point.
2020-08-07 16:45:30 -07:00
Anders Kaseorg 23b815bb50 queue: Fix types to reflect that Pika channels receive bytes.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-08-07 11:12:32 -07:00
Anders Kaseorg 3e7503a607 html_diff: Really return str, not bytes.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-08-07 11:12:32 -07:00
Anders Kaseorg fe5c524890 user_status: Use strings as dict keys.
JSON keys must be strings, and orjson enforces this.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-08-07 10:53:06 -07:00
Anders Kaseorg a329b538d2 test_helpers: Fix instrument_url decorator type.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-08-07 10:53:06 -07:00
Anders Kaseorg 44af001045 custom_profile_fields: Use strings as dict keys.
JSON keys must be strings, and orjson enforces this.  Mypy didn’t
catch the mismatched type of profiles_by_user_id because it doesn’t
understand CustomProfileFieldValue.field_id.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-08-07 10:53:06 -07:00
Anders Kaseorg b250e42f61 realm_user_count_by_role: Use strings as dict keys.
JSON keys must be strings, and orjson enforces this.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-08-07 10:53:06 -07:00
Anders Kaseorg f8bcf39014 actions: Do not attempt JSON serialization of a Stream object.
It doesn’t end well.  Or sometimes it doesn’t end (OverflowError:
Maximum recursion level reached).

Introduced by commits ccdf52fef6 and
94d2de8b4a (#15601).

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-08-07 10:53:06 -07:00
Steve Howell 15ffd2b666 event_schema: Extract check_stream_delete. 2020-08-06 12:29:43 -07:00
Steve Howell 059e0bb81e event_schema: Extract check_default_streams. 2020-08-06 12:29:43 -07:00
Steve Howell 36ec1571d4 event_schema: Extract check_default_stream_groups. 2020-08-06 12:29:43 -07:00
Tim Abbott 00fd9afad5 embed: Remove useless 'sender' field.
The variant `update_message` events have this extra sender field not
present in normal update_message events; this field has no purpose, so
we remove it.
2020-08-03 18:04:38 -07:00
Tim Abbott 99a54ba67e tornado: Fix ID lists leaked to the events API.
Apparently, `update_message` events unexpectedly contained what were
intended to be internal data structures about which users were
mentioned in a given message.

The bug has been present and accumulating new data structures for
years.

Fixing this should improve the performance of handling update_message
events as well as cleaning up this API's interface.

This was discovered by our automated API documentation schema checking
tooling detecting these unexpected elements in these event
definitions; that same logic should prevent future bugs like this from
being introduced in the future.
2020-08-03 17:52:39 -07:00
Clara Dantas f26d0affef bulk_update: Do updates in bulk where applicable. 2020-08-03 16:16:42 -07:00
Clara Dantas be6b2b248f Error reporting emails: Indicate user role.
Adds user role to the user's information in the error reporting
emails, as some bugs are role-dependent.

Fixes: #15344
2020-08-03 13:35:51 -07:00
Clara Dantas b4dd118aa1 refactor: Create a user object in report.
To make it easier to check if there is user information to be used
in the error report emails, we create a user object inside report.
Now, to check if we have the user's full name, email, etc, we just
need to do report['user']['user_full_name'] rather than check
each information one by one, because if the value of one key in
the report is different than None, all the others will be as well.
2020-08-03 13:35:51 -07:00
Anders Kaseorg 159641bab8 timestamp: Remove datetime_to_precise_timestamp for datetime.timestamp.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-07-31 22:10:14 -07:00
Alex Vandiver 90cdda9836 markdown: Link the twitter response code docs inline. 2020-07-31 10:35:41 -07:00
Alex Vandiver 03ca3afbc2 markdown: Treat more twitter codes as also permanent failures.
Per the API documentation[1], the following codes all correspond to
HTTP 404:

 - `34`: **Sorry, that page does not exist.**  The specified resource
   was not found.
 - `144`: **No status found with that ID.**  The requested Tweet ID is
   not found (if it existed, it was probably deleted)
 - `421`: **This Tweet is no longer available.**  The Tweet cannot be
   retrieved. This may be for a number of reasons.
 - `422`: **This Tweet is no longer available because it violated the
   Twitter Rules.**  The Tweet is not available in the API.

Treat all of these identically.

[1] https://developer.twitter.com/en/docs/basics/response-codes
2020-07-31 10:35:41 -07:00
Alex Vandiver fc141af30e markdown: Factor out twitter error code handling. 2020-07-31 10:35:41 -07:00
Clara Dantas 6b011b241b actions: Remove duplicated conditional.
In the gather_subscriptions_helper functions, there was a repeated
conditional inside one another.
2020-07-30 16:59:51 -07:00
Tim Abbott 6130a61be0 export: Only print .s with percent_callback to console.
The S3 data export tool's upload code path uses this nice boto
callback feature for showing a progress bar, which is nice for the
management command.  It's spammy/broken in production and the backend
tests, so we change percent_callback to be a parameter passed in so
that it can only be used in the contexts where it makes sense.
2020-07-30 13:14:53 -07:00
Anders Kaseorg 022c4fbfc7 Revert "digest: Support digest of web public streams for guest users."
This reverts commit c3779338c6 (part
of #14638), which incorrectly depended on commits from the future,
with the effect of either halting the flow of entropic time in an
irresolvable temporal paradox, summoning extradimensional beings to
rain destruction on the galaxy, or failing CI.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-07-29 21:05:59 -07:00
Clara Dantas c3779338c6 digest: Support digest of web public streams for guest users. 2020-07-29 17:52:36 -07:00
Clara Dantas a9af80d7a2 streams: Make /streams endpoint return also web-public streams.
This commit modifies the /streams endpoint so that the web-public
streams are included in the default list of streams that users
have access to.

This is part of PR #14638 that aims to allow guest users to
browse and subscribe themselves to web public streams.
2020-07-29 17:52:36 -07:00
Clara Dantas 0994b029d6 streams: Grant authorization to guest users to subscribe.
Modifies filter_stream_authorization so that web-public streams are
added in the list of authorized streams that a guest user can
subscribe.

This commit is part of PR #14638 that aims to allow guest users
to browse and subscribe to web-public streams.
2020-07-29 17:52:36 -07:00
Clara Dantas e22e12fe60 streams: Grant guest users access to web-public streams.
In this commit, we grant guest users access to stream history,
send message and common stream data of web-public streams.

This is part of PR #14638 that aims to allow guest users to
browse and subscribe to web-public streams.
2020-07-29 17:52:36 -07:00
Clara Dantas 523bb30f33 actions: Modify validate_user_access_to_subscribers.
This modification allows guest users to have access to web-public
streams subscribers, even if they aren't subscribed or never
subscribed to that stream.

This commit is part of PR #14638 that aims to allow guest users to
browser and subscribe to web-public streams.
2020-07-29 17:52:36 -07:00
Clara Dantas 6a1d9480bc actions: Modify gather_subscriptions() to also gather web-public streams.
Now, gather_subscriptions include web-public streams in the 3 sets
of streams that it returns, subscribed, unsubscribed and never
subscribed.

This is part of PR #14638 that aims to allow guest users to browse and
subscribe to web-public streams.
2020-07-29 17:52:36 -07:00
Clara Dantas fda94a49c1 streams: Change flow in filter_stream_authorization.
This change makes the flow more coherent by instead of checking,
in the last condition, if the user isn't authorized to access that
stream, check if they are, as it is done in the other checks. Only
if all the conditions are false, which means that the user doesn't
have access to that stream, the stream is added to the
unauthorized_streams list.
2020-07-29 17:52:36 -07:00
Gittenburg 0706de2305 docs: Make tabbed sections accessible from keyboard.
Part of #15948.
2020-07-29 11:40:39 -07:00
Hemanth V. Alluri 0e893b9045 models/drafts: Add a model for storing Draft messages.
Also add a Draft object-to-dictionary conversion method.
The following commits will provide an API around this
model using which our clients can sync drafts across each
other (if they so wish too). As of making this commit, we
haven't finalized exactly how our clients will use this.

See https://chat.zulip.org/#narrow/stream/2-general/topic/drafts
For some of the discussion around this model and in general,
around this feature.

Signed-off-by: Hemanth V. Alluri <hdrive1999@gmail.com>
2020-07-28 17:18:35 -07:00
Hemanth V. Alluri d5f42e2722 timestamp: Add function to convert a datetime to precise Unix timestamp.
Unlike the other Python datetime to Unix timestamp conversion
function (`datetime_to_timestamp`), `datetime_to_precise_timestamp`
won't drop the microseconds.

Signed-off-by: Hemanth V. Alluri <hdrive1999@gmail.com>
2020-07-28 17:18:35 -07:00
Clara Dantas 831298d387 test_classes: Modify functions to deal with web-public streams.
Modify common_subscribe_to_streams to perform subscription in
web-public streams as well, and make_stream function to create
web-public streams.
2020-07-27 18:30:06 -07:00
orientor ec40a5dda4 openapi_py: Add validation for `/events`.
Edit the function `validate_against_openapi_schema` and add some
helper functions to allow for validation of documented events.
Also add OpenAPI response validation in `verify_action` as it is
called in a large number of `/events` tests.
2020-07-27 18:08:41 -07:00
orientor 86f7b8f7d9 openapi_markdown: Add parser for events documentation.
Add parser so that documentation of the various events returned by
`/events` are rendered to the docs directly from their OpenAPI
specification.
2020-07-27 18:08:41 -07:00
Sumanth V Rao 9b6de63afe stream/docs: Add date_created to Stream.API_FIELDS.
The parameter Stream.date_created is now sent down to the clients
for both:

    - client.get_streams()
    - client.list_subscriptions()

API docs updated for stream and subscriptions.

Fixes #15410
2020-07-27 16:33:36 -07:00
orientor bdf9d912e3 test_classes: Use OpenAPI request validator for validating requests.
Expand the `validate_api_response_openapi` function to also validate
requests to ensure better testing.
2020-07-26 16:26:57 -07:00
Tim Abbott 29c66cf7c2 actions: Remove log_event and its legacy settings.
Now that we've finally converted these to use RealmAuditLog, we can
remove this ultra-legacy bit of code.
2020-07-24 12:13:16 -07:00
arpit551 0d79b55b2e audit_log: Record RealmAuditLog in do_change_notification_settings.
Removed logging with log_event and used RealmAuditLog instead.
Added tests in test_audit_log for the same.
2020-07-24 12:04:40 -07:00
arpit551 54df9290b9 audit_log: Record RealmAuditLog in do_rename_stream.
Removed logging with  log_event and used RealmAuditLog instead.
Added tests in test_audit_log for the same.
2020-07-24 12:00:31 -07:00
arpit551 94d2de8b4a audit_log: Record RealmAuditLog while changing default streams.
Removed logging with log_event and used RealmAuditLog instead.
Added tests in test_audit_log for the same.
2020-07-24 12:00:31 -07:00
arpit551 753713fde7 audit_log: Record RealmAuditLog in do_change_subscription_property.
Removed logging with log_subscription_property_change
and used RealmAuditLog instead.
Added tests in test_audit_log for the same.
2020-07-24 12:00:29 -07:00
arpit551 e6f54a3278 audit_log: Record RealmAuditLog in do_change_icon_source.
Removed logging with  log_event and used RealmAuditLog instead.
Added tests in test_audit_log for the same.
2020-07-24 11:56:40 -07:00
arpit551 12a19b0ff1 actions.py: Removed unnecessary logging in notify_subscription_added/removed.
We already record RealmAuditLog in bulk_add/remove_subscription
so there is no need to log while notifying.
2020-07-24 11:51:47 -07:00
arpit551 ccdf52fef6 audit_log: Log RealmAuditLog for realm notification properties.
Log RealmAuditLog for do_set_realm_notifications_stream and
do_set_realm_signup_notifications_stream function.
Added tests for the same.
2020-07-24 11:51:47 -07:00
arpit551 8f6a1c3f40 actions.py: Removed do_set_realm_message_deleting function.
Used do_set_realm_property function instead of
do_set_realm_message_deleting and removed it.
2020-07-24 11:51:47 -07:00
arpit551 422fa0ff7d audit_log: Log RealmAuditLog for do_set_realm_message_editing.
Log RealmAuditLog for do_set_realm_message_editing. Added tests for
same.
2020-07-24 11:51:46 -07:00
Steve Howell f03605bd73 event_schema: Support plan_type in check_realm_update. 2020-07-24 09:38:34 -07:00
Steve Howell 33f173ae1b event_schema: Use check_realm_update in two more places.
We also have the caller pass in the property name for an
additional sanity check.

Note that we don't yet handle the possibility of extra_data;
that will be a subsequent commit.

Also, the stream_id fields aren't in Realm.property_types,
so we specify their types in the checker.
2020-07-24 09:38:34 -07:00
Steve Howell 176ab66fc7 event_schema: Extract check_realm_user_update.
This a pretty big commit, but I really wanted it
to be atomic.

All realm_user/update events look the same from
the top:

    _check_realm_user_update = check_events_dict(
        required_keys=[
            ("type", equals("realm_user")),
            ("op", equals("update")),
            ("person", _check_realm_user_person),
        ]
    )

And then we have a bunch of fields for person that
are optional, and we usually only send user_id plus
one other field, with the exception of avatar-related
events:

    _check_realm_user_person = check_dict_only(
        required_keys=[
            # vertical formatting
            ("user_id", check_int),
        ],
        optional_keys=[
            ("avatar_source", check_string),
            ("avatar_url", check_none_or(check_string)),
            ("avatar_url_medium", check_none_or(check_string)),
            ("avatar_version", check_int),
            ("bot_owner_id", check_int),
            ("custom_profile_field", _check_custom_profile_field),
            ("delivery_email", check_string),
            ("full_name", check_string),
            ("role", check_int_in(UserProfile.ROLE_TYPES)),
            ("email", check_string),
            ("user_id", check_int),
            ("timezone", check_string),
        ],
    )

I would start the code review by just skimming the changes
to event_schema.py, to get the big picture of the complexity
here.  Basically the schema is just the combined superset of
all the individual schemas that we remove from test_events.

Then I would read test_events.py.

The simplest diffs are basically of this form:

    -  schema_checker = check_events_dict([
    -      ('type', equals('realm_user')),
    -      ('op', equals('update')),
    -      ('person', check_dict_only([
    -          ('role', check_int_in(UserProfile.ROLE_TYPES)),
    -          ('user_id', check_int),
    -      ])),
    -  ])

    # ...
    -  schema_checker('events[0]', events[0])
    +  check_realm_user_update('events[0]', events[0], {'role'})

Instead of a custom schema checker, we use the "superset"
schema checker, but then we pass in the set of fields that we
expect to be there.  Note that 'user_id' is always there.

So most of the heavy lifting happens in this new function
in event_schema.py:

    def check_realm_user_update(
        var_name: str, event: Dict[str, Any], optional_fields: Set[str],
    ) -> None:
        _check_realm_user_update(var_name, event)

        keys = set(event["person"].keys()) - {"user_id"}
        assert optional_fields == keys

But we still do some more custom checks in test_events.py.

custom profile fields: check keys of custom_profile_field

     def test_custom_profile_field_data_events(self) -> None:
+        self.assertEqual(
+            events[0]['person']['custom_profile_field'].keys(),
+            {"id", "value", "rendered_value"}
+        )

+        check_realm_user_update('events[0]', events[0], {"custom_profile_field"})
+        self.assertEqual(
+            events[0]['person']['custom_profile_field'].keys(),
+            {"id", "value"}
+        )

avatar fields: check more specific types, since the superset
    schema has check_none_or(check_string)

     def test_change_avatar_fields(self) -> None:
+        check_realm_user_update('events[0]', events[0], avatar_fields)
+        assert isinstance(events[0]['person']['avatar_url'], str)
+        assert isinstance(events[0]['person']['avatar_url_medium'], str)

+        check_realm_user_update('events[0]', events[0], avatar_fields)
+        self.assertEqual(events[0]['person']['avatar_url'], None)
+        self.assertEqual(events[0]['person']['avatar_url_medium'], None)

Also note that avatar_fields is a set of four fields that
are set in event_schema.

full name: no extra work!

     def test_change_full_name(self) -> None:
-        schema_checker('events[0]', events[0])
+        check_realm_user_update('events[0]', events[0], {'full_name'})

test_change_user_delivery_email_email_address_visibilty_admins:

    no extra work for delivery_email
    check avatar fields more directly

roles (several examples) -- actually check the specific role

     def test_change_realm_authentication_methods(self) -> None:
-            schema_checker('events[0]', events[0])
+            check_realm_user_update('events[0]', events[0], {'role'})
+            self.assertEqual(events[0]['person']['role'], role)

bot_owner_id: no extra work!

-        change_bot_owner_checker_user('events[1]', events[1])
+        check_realm_user_update('events[1]', events[1], {"bot_owner_id"})

-        change_bot_owner_checker_user('events[1]', events[1])
+        check_realm_user_update('events[1]', events[1], {"bot_owner_id"})

-        change_bot_owner_checker_user('events[1]', events[1])
+        check_realm_user_update('events[1]', events[1], {"bot_owner_id"})

timezone: no extra work!

-                timezone_schema_checker('events[1]', events[1])
+                check_realm_user_update('events[1]', events[1], {"email", "timezone"})
2020-07-24 09:38:34 -07:00
Steve Howell 1fa6ae1e16 refactor: Extract build_page_params_for_home_page_load. 2020-07-22 17:15:03 -07:00
Steve Howell 27072289ce refactor: Extract zerver/lib/home.py.
The two functions extracted here are mostly
copied verbatim, but we use dataclasses
to marshal the values back.
2020-07-22 17:15:02 -07:00
Vishnu KS 67bacd6e31 billing: Don't allow guest users to upgrade. 2020-07-22 16:57:49 -07:00
Steve Howell a6519e7b8f event_schema: Extract check_user_group_add. 2020-07-22 16:48:19 -07:00
Steve Howell 3f25e52667 event_schema: Extract check_user_status. 2020-07-22 16:48:19 -07:00
Steve Howell 631adc5677 event_schema: Extract check_alert_words. 2020-07-22 16:48:19 -07:00
Steve Howell 0a9a9d8258 event_schema: Extract check_custom_profile_fields. 2020-07-22 16:48:19 -07:00
Steve Howell 7176b90882 event_schema: Extract check_typing_start. 2020-07-22 16:48:19 -07:00
Steve Howell 5f3ea0a659 event_schema: Extract check_invites_changed. 2020-07-22 16:48:19 -07:00
Steve Howell ec17091521 event_schema: Extract check_submessage. 2020-07-22 16:48:19 -07:00
Steve Howell 92136d738a event_schema: Extract check_reaction. 2020-07-22 16:48:19 -07:00
Steve Howell 5209de0261 event_schema: Extract check_update_message_flags. 2020-07-22 16:48:19 -07:00
Steve Howell f2bc22e869 event_schema: Extract check_update_message*. 2020-07-22 16:48:19 -07:00
Steve Howell b81f3433d8 event_schema: Extract check_message. 2020-07-22 16:48:19 -07:00
Steve Howell 385050de20 event_schema: Extract check_realm_bot_(delete/remove).
It is strange that we have both of these events.
2020-07-22 16:48:19 -07:00
Steve Howell 96f5ab1c87 event_schema: Extract check_realm_bot_update. 2020-07-22 16:48:19 -07:00
Steve Howell f5c4ee4477 event_schema: Extract check_realm_bot_add.
Note that we use the actual integer bot_type
value now to determine how we validate
services.
2020-07-22 16:48:19 -07:00
Steve Howell 0a6ce36ac9 event_schema: Extract check_update_global_notifications. 2020-07-22 16:48:19 -07:00
Steve Howell 96b821684b event_schema: Extract check_update_display_settings. 2020-07-22 16:48:19 -07:00
Steve Howell dd5949274d event_schema: Extract check_subscription_peer_*. 2020-07-22 16:48:19 -07:00
Steve Howell 502f1b9fe2 event_schema: Extract check_subscription_remove. 2020-07-22 16:48:19 -07:00
Steve Howell 055f1a590d event_schema: Extract check_subscription_add. 2020-07-22 16:48:19 -07:00
Steve Howell b116f1e911 event_schema: Extract check_stream_update. 2020-07-22 16:48:19 -07:00
Steve Howell 14aa87a168 event_schema: Extract check_stream_create. 2020-07-22 16:48:19 -07:00
Steve Howell a6796e9e86 event_schema: Extract check_realm_update. 2020-07-22 16:48:19 -07:00
Steve Howell e49acfa637 event_schema: Extract event_schema.py.
Obviously, this file will soon grow--this
was the easiest way to start without introducing
noise into other commits.

It will soon be structurally similar
to frontend_tests/node_tests/lib/events.js--I
have some ideas there.  But this should also
help for things like API docs.
2020-07-22 16:48:19 -07:00
Vinit Singh 308cf8ac00 markdown: Inline Youtube previews instead of appending it to the end.
This change makes our handling of youtube-url previews consistent
with how we handle our inline images. This allows the previews to
render next to the paragraph that links to the youtube video.

Follow-up to PR #15773.
2020-07-22 16:11:17 -07:00
Emilio López 7b35234c7b
email_mirror: Fix exception handling unstructured headers.
This commit rewrites the way addresses are collected. If
the header with the address is not an AddressHeader (for instance,
Delivered-To and Envelope-To), we take its string representation.

Fixes: #15864 ("Error in email_mirror - _UnstructuredHeader has no attribute addresses").
2020-07-22 12:11:25 -07:00
David Wood 366cb30db1 integrations: Re-add Canarytoken integration.
This commit re-adds the integration for canarytokens.org, now separate
from the primary Thinkst integration.

Signed-off-by: David Wood <david@davidtw.co>
2020-07-20 15:34:40 -07:00
David Wood f1c9d9c2fe integrations: Fix Thinkst Canary integration.
This commit fixes the Thinkst Canary integration which - based on the
schema in upstream documentation - incorrectly assumed that some fields
would always be sent, which meant that the integration would fail. In
addition, this commit adjusts support for canarytokens to only support
the canarytoken schema with Thinkst Canaries (not Thinkst's
canarytokens.org).

Signed-off-by: David Wood <david@davidtw.co>
2020-07-20 15:34:40 -07:00
Rohitt Vashishtha fb2946aaf6 Revert "markdown: Remove paragraphs that only contain a tweet link."
This reverts commit d3770153a6.

We do not show a link to the tweet in our preview, so we should revert
to our previous behavior for now.
2020-07-17 14:30:22 -07:00
Steve Howell c44500175d database: Remove short_name from UserProfile.
A few major themes here:

    - We remove short_name from UserProfile
      and add the appropriate migration.

    - We remove short_name from various
      cache-related lists of fields.

    - We allow import tools to continue to
      write short_name to their export files,
      and then we simply ignore the field
      at import time.

    - We change functions like do_create_user,
      create_user_profile, etc.

    - We keep short_name in the /json/bots
      API.  (It actually gets turned into
      an email.)

    - We don't modify our LDAP code much
      here.
2020-07-17 11:15:15 -07:00
Steve Howell b375581f58 api: Remove (sender_)short_name from message APIs. 2020-07-17 11:15:14 -07:00
Steve Howell 3b2c881ce6 tests: Decouple test_retention and test_reactions.
We generally want to avoid having two sibling test
suites depend on each other, unless there's a real
compelling reason to share code.  (And if there is
code to share, we can usually promote it to either
test_helpers or ZulipTestCase, as I did here.)

This commit is also prep for the next commit, where
I try to simplify all of the helpers in EmojiReactionBase.

Especially now that we have f-strings, it is usually
better to just call api_post explicitly than to
obscure the mechanism with thin wrappers around
api_post.  Our url schemes are pretty stable, so it's
unlikely that the helpers are actually gonna prevent
future busywork.
2020-07-17 11:04:54 -07:00
Steve Howell 2a9cb4c8cf dead code: Remove legacy reaction actions. 2020-07-17 04:40:01 -04:00
Steve Howell 2374e25b94 import: Import AlertWord table. 2020-07-16 08:50:31 -07:00
Steve Howell 318c55e030 export: Export AlertWord table. 2020-07-16 08:50:31 -07:00
Tim Abbott 289819fb70 push_notifications: Reduce log level for weird warning.
This issue isn't something a system administrator needs to take action
on -- it's a likely minor logic bug around organization
administrators moving topics between streams.

As a result, it shouldn't send error emails to administrators.
2020-07-16 01:22:03 -07:00
Rohitt Vashishtha b64ba98e90 markdown: Use unicode ellipses for collapsing spoilers.
We had initially implemented this feature using `(...)` but `(…)` is the
better variation.
2020-07-15 23:30:28 -07:00
Rohitt Vashishtha 7d5d90b484 markdown: Format spoilers for emails.
This is a hacky fix to avoid spoiler content leaking in emails. The
general idea here is to tell people to open Zulip to view the actual
message in full.

We create a mini-markdown parser here that strips away the fence content
that has the 'spoiler' tag for the text emails.

Our handling of html emails is much better in comparison where we can
use lxml to parse the spoiler blocks.
2020-07-15 23:30:28 -07:00
Tim Abbott 2f66c825a2 push_notifications: Disable badge counts.
We include tests for the new implementation to avoid churning the
codebase too much so this can be easily reverted when we are able to
re-enable the feature.
2020-07-15 22:19:53 -07:00
Rohitt Vashishtha 78c48935ca markdown: Format spoilers for push notifications.
We now do something sensible for spoilers in notifications. A message
like:

    ```spoiler Luke's father is
    Vader. Don't tell anyone else.
    ```

would be rendered as:

    Luke's father is (...)
2020-07-15 11:17:38 -07:00
Aman Agrawal e22885a6bf push_notifications: Return if push_notify already active.
If the push_notification for the UserMessage is already active,
we don't send any push notification to the user. This may
happen due to race conditions.

Added and fixed test cases affected by this.
2020-07-14 00:35:29 -07:00
Anders Kaseorg 6189e4d0c1 python: Convert more percent formatting to "".format.
Semgrep has gotten a little more clever at applying the percent
formatting rule.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-07-13 13:16:38 -07:00
Rohitt Vashishtha d3770153a6 markdown: Remove paragraphs that only contain a tweet link.
This is similar to our behavior with image previews, and helps
reduce clutter in the final rendered html.

We add the string 'Tweet: ' to our existing tests so those tests
remain the same.
2020-07-13 12:24:32 -07:00
Rohitt Vashishtha 87e01cd1fa markdown: Inline Twitter previews instead of appending at end.
This commit makes our handling of twitter previews consistent with
how we handle our inline images so that tweets render next to the
paragraph that links to the tweet.
2020-07-13 12:24:32 -07:00
Rohitt Vashishtha a8ab745ee4 markdown: Extract get_inlining_information for link previews.
We decouple the logic of insertion rules for inline links from
image preview logic. Now, we can use this same logic for other
kinds of link previews as well.
2020-07-13 12:24:32 -07:00
Steve Howell 9662af842f export: Remove stream sanity check.
We also remove the post_process_data option.

The sanity check is just overkill at this point,
since the mechanism to find streams is very
direct due to a recent commit.
2020-07-09 11:34:00 -07:00
Steve Howell 54c596cfc4 export: Just export all streams in a realm.
Before this change we would only export streams
that had actual subscribers, which is usually
harmless, but it was mostly a relic of a one
time migration that we did when we were cleaning
up some dirty data in some of our very early
databases (circa 2016).

Now we work down the table hierarchy in a
more natural way:

    - get Streams in Realm
    - get Recipients matching above Streams
    - get Subscriptions matching above Recipients

Note that for per-user exports, I kept the
same logic (users -> subscriptions -> recipients ->
streams) we had before.

One subtle detail here is that we make our
final Config blocks--which build the final
version of Recipient/Subscription--now hang
off of realm_config.

Fixes #15146.
2020-07-09 11:33:55 -07:00
Rohitt Vashishtha 912e372c4e markdown: Remove !avatar() and !gravatar() syntax.
This particular commit has been a long time coming. For reference,
!avatar(email) was an undocumented syntax that simply rendered an
inline 50px avatar for a user in a message, essentially allowing
you to create a user pill like:

`!avatar(alice@example.com) Alice: hey!`

---

Reimplementation

If we decide to reimplement this or a similar feature in the future,
we could use something like `<avatar:userid>` syntax which is more
in line with creating links in markdown. Even then, it would not be
a good idea to add this instead of supporting inline images directly.

Since any usecases of such a syntax are in automation, we do not need
to make it userfriendly and something like the following is a better
implementation that doesn't need a custom syntax:

`![avatar for Alice](/avatar/1234?s=50) Alice: hey!`

---

History

We initially added this syntax back in 2012 and it was 'deprecated'
from the get go. Here's what the original commit had to say about
the new syntax:

> We'll use this internally for the commit bot.  We might eventually
> disable it for external users.

We eventually did start using this for our github integrations in 2013
but since then, those integrations have been neglected in favor of
our GitHub webhooks which do not use this syntax.

When we copied `!gravatar` to add the `!avatar` syntax, we also noted
that we want to deprecate the `!gravatar` syntax entirely - in 2013!

Since then, we haven't advertised either of these syntaxes anywhere
in our docs, and the only two places where this syntax remains is
our game bots that could easily do without these, and the git commit
integration that we have deprecated anyway.

We do not have any evidence of someone asking about this syntax on
chat.zulip.org when developing an integration and rightfully so- only
the people who work on Zulip (and specifically, markdown) are likely
to stumble upon it and try it out.

This is also the only peice of code due to which we had to look up
emails -> userid mapping in our backend markdown. By removing this,
we entirely remove the backend markdown's dependency on user emails
to render messages.

---

Relevant commits:

- Oct 2012, Initial commit        c31462c278
- Nov 2013, Update commit bot     968c393826
- Nov 2013, Add avatar syntax     761c0a0266
- Sep 2017, Avoid email use       c3032a7fe8
- Apr 2019, Remove from webhook   674fcfcce1
2020-07-07 10:39:44 -07:00
Aman Agrawal f1cc2ab926 delete_event_notify_user_ids: Fix UnboundLocalError. 2020-07-07 10:33:50 -07:00
arpit551 a2428017e9 audit_log: Log RealmAuditLog in do_set_realm_authentication_methods.
Log RealmAuditLog in do_set_realm_authentication_methods and added
tests for it.
2020-07-06 17:38:20 -07:00
arpit551 ba92666dbd audit_log: Log RealmAuditLog in do_set_realm_property.
Log RealmAuditLog in do_set_realm_property and do_remove_realm_domain.
Tests for the changes are written in test_events because it will save
duplicate code for test_change_realm_property.
2020-07-06 17:38:19 -07:00
arpit551 6af337c07e audit_log: Log RealmAuditLog when stream is deactivated. 2020-07-06 17:32:11 -07:00
arpit551 e0c9b0fdff audit_log: Log RealmAuditLog when Stream is created.
Added new Event Type in AbstractRealmAuditLog STREAM_CREATED.
Since we finally create streams in create_stream_if_needed function
in zerver/lib/streams.py so logged realm_audit there.
Passed acting_user when create_stream_if_needed or ensure_stream
function  is called.
Added tests in test_audit_log.
2020-07-06 17:32:11 -07:00
arpit551 8807cddc90 audit_log: Correctly add acting_user in bulk_remove_subscriptions.
Earlier we were not adding acting_user while creating RealmAuditLog
objects in bulk_remove_subscriptions. Improved tests_subscriptions
to check it.
2020-07-06 17:32:11 -07:00
arpit551 87aaa84b42 audit_log: Log acting_user in do_change_user_role. 2020-07-06 17:32:11 -07:00
arpit551 01f12b9fc2 audit_log: Log acting_user in user creation and user activation. 2020-07-06 17:32:09 -07:00
arpit551 653928bdfe audit_log: Log acting_user in do_change_avatar_fields. 2020-07-06 17:24:18 -07:00
arpit551 2279fef316 audit_log: Log acting_user in do_change_logo_source. 2020-07-06 17:24:18 -07:00
arpit551 19a8841a9e audit_log: Log acting_user in do_scrub_realm. 2020-07-06 17:24:18 -07:00
Rohitt Vashishtha 732ec3c0e6 timestamp: Change syntax to `<time:timestammp>`.
We had been using !time() syntax for timestamps so far. Since its
an unreleased feature, we can make changes without affecting many
people.

Fixes #15442.
2020-07-06 15:53:56 -07:00
Aman Agrawal 18537b63f5 move_topic_to_stream: Apply the event to the fetched state.
Fixes #14101.

We fix the unread count for the messages which were moved to
the new stream too.
2020-07-06 15:48:55 -07:00
Mohit Gupta f8d1e0f86a refactor: Rename convert to markdown_convert.
Prior to this commit whenever convert was imported from zerver.lib.markdown
it was aliased as markdown_convert for readability.
This commit rename convert function to markdown_convert so that it can be
directly import it without aliasing and without compromising readability.
2020-07-06 12:39:59 -07:00
Anders Kaseorg e363382628 push_notifications: Fix strict_optional errors.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-07-06 11:25:48 -07:00
Anders Kaseorg e24b2fdf06 markdown: Fix strict_optional errors.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-07-06 11:25:48 -07:00
Anders Kaseorg d64f036247 outgoing_webhook: Fix strict_optional errors.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-07-06 11:25:48 -07:00
Anders Kaseorg 0d7539dc50 test_classes: Fix strict_optional errors.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-07-06 11:25:48 -07:00
Anders Kaseorg 813cee3e50 test_helpers: Fix strict_optional errors.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-07-06 11:25:48 -07:00
Anders Kaseorg c0bbdbcadf test_messages: Fix strict_optional errors.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-07-06 11:25:48 -07:00
Anders Kaseorg 26fe6be16a test_service_bot_system: Fix strict_optional errors.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-07-06 11:25:48 -07:00
Anders Kaseorg 489d73f63a queue: Fix strict_optional errors.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-07-06 11:25:48 -07:00
Anders Kaseorg 3885fdadce realm: Fix strict_optional errors.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-07-06 11:25:48 -07:00
Anders Kaseorg dbd1b56362 remote_server: Fix send_to_push_bouncer type.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-07-05 11:53:45 -07:00
Steve Howell 0b65abcdf5 pointer: Remove pointer from UserProfile.
Most of the changes here are just that we no
longer need to provide a value for pointer
when we create UserProfile objects.
2020-07-03 13:08:40 +00:00
Anders Kaseorg c08ee904d8 models: Add explicit id fields for better type checking.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-07-02 13:28:10 -07:00
Anders Kaseorg 9b7c6828ec models: Annotate most field types.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-07-02 13:28:10 -07:00
Anders Kaseorg 8c64a14a31 markdown: Fix get_user_by_id type.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-07-02 13:28:10 -07:00
Tim Abbott 0c48c75920 push_notifications: Fix events with no message_ids.
This fixes our triggering a RabbitMQ event to send a push notification
to remove the empty set of push notifications, resulting from not
using the correct data structure to determine which message IDs to look at.

This was causing a lot of visible exceptions when running
the `test_messages.py` test suite.
2020-07-01 14:58:43 -07:00
Tim Abbott c6638a62dc message_edit: Clean up comments around moving topics. 2020-07-01 14:55:17 -07:00
Tim Abbott 03cf3edf42 message_edit: Fix typo in old_stream_unsubbed_guests. 2020-07-01 14:54:36 -07:00
Tim Abbott 7e11be0429 message_edit: s/subscribers_ids/subscriber_ids/. 2020-07-01 14:51:58 -07:00
Tim Abbott bda7116f86 message_edit: Reuse delete_event_notify_user_ids.
This better describes semantically what we have in mind here -- every
user should get one of the two events.
2020-07-01 14:50:32 -07:00
Aman Agrawal 3f42d15168 move_topic_to_stream: Delete UserMessage for new stream unsubs.
For users who are unsubscribed from the new stream but are in
the old stream, we delete the UserMessage.

We send the delete_message event only to guest users,
who have completely lost asses to the moved messages, for other
users we send the normal update_message event which moves
the messages to the new unsubed stream which
otherwise would look broken to the
user without reloading to the webpage.
2020-07-01 14:42:01 -07:00
orientor 529da34513 openapi: Use third-party validator for schema validation.
Our previous OpenAPI schema validator that we implemented ourselves
was useful training wheels for our understanding OpenAPI properly, and
was mostly correct.  But given that we've finally reached the point
where our OpenAPI file accurately describes the API, it makes sense to
switch to use an official OpenAPI validator.  We lose some ability to
do exclude rules for particular elements, but those were primarily
important for us when we had a lot of them.

As part of this change, we need to add `additionalProperties: false`
for all of our dictonaries/objects where we've documented every
parameter; otherwise the OpenAPI schema checker won't know that we
expect every parameter to be documented.
2020-07-01 11:21:41 -07:00
Vishnu KS 908e1b6bea 18n: Cleanup strings to be translated in do_rename_stream. 2020-07-01 11:03:29 -07:00
Vishnu KS c7ce9097d0 18n: Cleanup strings to be translated in send_pm_if_empty_stream. 2020-07-01 11:03:29 -07:00
Anders Kaseorg 1b96af2987 cache: Fix type: ignore issues.
This was hiding an actual type error in test_cache: a mismatch between
the object ID type, which is str, and the default id_fetcher, which
returns int.

Mypy’s insufficient support for default generic arguments basically
means we can’t use them without a lot of overloading, and there are
not enough callers here to justify that.

https://github.com/python/mypy/issues/3737

We avoid this being super messy where the code calls this by adding
some less generic wrappers for generic_bulk_cached_fetch.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-07-01 10:51:15 -07:00
Anders Kaseorg bb8dcb9b1e test_runner: Remove customized run_test, TestSuite, and TestLoader.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-07-01 10:48:12 -07:00
Anders Kaseorg e40cdc6c2c test_runner: Move bounce_key_prefix_for_testing to ZulipTestCase.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-07-01 10:48:12 -07:00
Anders Kaseorg a7c0cca704 test_runner: Replace full_test_name with TestCase.id.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-07-01 10:48:12 -07:00
Anders Kaseorg 3d917ddfde test_runner: Remove unused ReturnT variable.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-07-01 10:48:12 -07:00
Anders Kaseorg 92d0290dc5 tests: Remove slow tests detection.
According to @showell:

> All the slow decorators can die. That was a failed experiment of
> mine from 2014 days.  I have meaning to kill them for a couple years
> now.  I wrote this with the best of intentions, but I believe it's
> now just cruft.  We never made a "fast" mode, for one. And we kept
> writing more and more slow tests, haha.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-30 23:34:53 -07:00
Anders Kaseorg ec70aab138 test_service_bot_system: Fix type: ignore issues.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-30 18:58:23 -07:00
Anders Kaseorg 83b6884d8b test_decorators: Remove tests for conditions already excluded by mypy.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-30 18:58:23 -07:00
Anders Kaseorg 5fbccbf64d test_helpers: Fix type: ignore issues.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-30 18:58:23 -07:00
Anders Kaseorg 8a7cfd05a1 sessions: Fix type: ignore issues.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-30 18:58:23 -07:00
Anders Kaseorg bc9663d57f webhooks: Fix a type: ignore issue.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-30 18:58:23 -07:00
Anders Kaseorg f2e7076e2a decorator: Replace type: ignore with cast, avoid Any.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-30 18:58:23 -07:00
Anders Kaseorg e582bbea4a decorator: Strengthen types of signature-preserving decorators.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-30 18:58:23 -07:00
Anders Kaseorg 96cf928d27 models: Fix user_profile_id type.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-30 18:58:23 -07:00
Anders Kaseorg b655029ced rest: Strengthen default_never_cache_responses type.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-30 18:58:23 -07:00
Anders Kaseorg 8667e632c8 python: Use isinstance with a tuple for testing multiple types.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-30 18:58:23 -07:00
Aman Agrawal 5b7917da5f notifications: Remove support for unbatched push removal events.
We remove support for the old clients which required an event for
each message to clear notification.

This is justified since it has been around 1.5 years since we started
supporting the bulk operation (and so essentially nobody is using a
mobile app version so old that it doesn't support the batched
approach) and the unbatched approach has a maintenance and reliability
cost.
2020-06-30 10:12:27 -07:00
Tim Abbott f682008585 markdown: Delete unused log_markdown_error.
Its used was removed in 50ed378dd6.
2020-06-29 15:17:27 -07:00
Mohit Gupta 08e74558a9 refactor: Rename remaining bugdown word to markdown in .py files.
This commit is part of series of commits aimed at renaming bugdown to
markdown.
2020-06-29 15:03:20 -07:00
Mohit Gupta 66d54d86ad refactor: Rename remaining bugdown_convert to markdown_convert.
This commit is part of series of commits aimed at renaming bugdown to
markdown.
2020-06-29 15:03:20 -07:00
Mohit Gupta 0552936c9b refactor: Rename bugdown to markdown in message.py and test_messages.py.
This commit is part of series of commits aimed at renaming bugdown to
markdown.
2020-06-29 14:58:30 -07:00
Mohit Gupta e13691aa10 refactor: Rename log_bugdown_error to log_markdown_error.
This commit is part of series of commits aimed at renaming bugdown to
markdown.
2020-06-29 14:58:30 -07:00
Mohit Gupta c1b6fbbc7d refactor: Rename bugdown to markdown in import_realm.py.
This commit is part of series of commits aimed at renaming bugdown to
markdown.
2020-06-29 14:58:30 -07:00
Mohit Gupta bb8cafdcf1 refactor: Rename remaining bugdown refrences to markdown in actions.py.
This commit removes bugdown alias and do proper imports from markdown
module. Also remove bugdown word and replace it with markdown in
comments.
This commit is part of series of commits aimed at renaming bugdown to
markdown.
2020-06-29 14:58:30 -07:00
Mohit Gupta 6cca775557 refactor: Rename alias bugdown_verion to markdown_version.
This commit is part of series of commits aimed at renaming bugdown to
markdown.
2020-06-29 14:58:30 -07:00
Steve Howell 987408db68 test_events: Extract BaseAction. 2020-06-29 12:38:26 -07:00
Mateusz Mandera 9392e8bec7 emails: Add a custom header specifying the organization when possible.
Closes #15135.
2020-06-28 13:33:02 -07:00
Aman Agrawal 5f82e1a984 push_notif: Don't clear notif if stream notif enabled.
If a message was edited to remove a user mention, we don't
remove the push_notification for the user if the user has
stream_push_notfications enabled.
2020-06-28 11:51:50 -07:00