Commit Graph

12294 Commits

Author SHA1 Message Date
Alex Vandiver b4c2ae9cae settings: Adjust documentation and comment references to settings.py.
`zproject/settings.py` itself is mostly-empty now.  Adjust the
references which should now point to `zproject/computed_settings.py`
or `zproject/default_settings.py`.
2020-08-24 13:13:16 -07:00
orientor 372e010dbb events: Add `op` field to `update_message_flags` events.
`update_message_flags` events used `operation` instead of `op`, the
latter being the standard field used in other events. So add `op`
field to `update_message_flags` and mark `operation` as deprecated,
so that it can be removed later.
2020-08-24 12:42:03 -07:00
Steve Howell f429df3401 github tests: Extract verify_post_is_ignored. 2020-08-24 12:34:46 -07:00
Steve Howell dfe6960584 minor: Format IGNORED_EVENTS for github. 2020-08-24 12:34:46 -07:00
Steve Howell 0863874050 jira: Ignore worklog_updated event. 2020-08-24 12:34:46 -07:00
Steve Howell 7b344044b4 jira: Ignore version_released event. 2020-08-24 12:34:46 -07:00
Steve Howell 52c17fc731 minor: Format IGNORED_EVENTS for jira. 2020-08-24 12:34:46 -07:00
Steve Howell 56bfae3fac jira tests: Test ignored events. 2020-08-24 12:34:46 -07:00
Steve Howell 1a011b2adb trello: Ignore dueComplete field.
It's possible that this is a new name for the "due"
field, but it's not totally clear.

In the exception we saw in the field:

    payload['action']['data']['old']['dueComplete'] = False
    payload['action']['data']['card']['dueComplete'] = True
2020-08-24 12:34:46 -07:00
Steve Howell 1bbf5491a7 trello: Ignore changes to card "cover".
These events happen when somebody changes the
background image for a Trello card, or something
similar.
2020-08-24 12:34:46 -07:00
Steve Howell 43cc4a2c50 trello tests: Test that we ignore "pos" data.
We will extend this test soon, once we begin
ignoring other fields.
2020-08-24 12:34:46 -07:00
Steve Howell bfe04ff6cf trello: Ignore updateCheckItem/updateList actions. 2020-08-24 12:34:46 -07:00
Steve Howell 37159fc235 trello tests: Generalize test for createCheckItem.
We remove the fixture for create_check_item, which
has been bit-rotting for as long as we have ignored
this type of card data.

Our new test is more powerful, in the sense that it
shows we successfully ignore all fixtures of this
type.

If we want to handle this, we'll just need to get
new, representative fixture data from trello.
2020-08-24 12:34:46 -07:00
Steve Howell b193c22e4d trello tests: Extract verify_post_is_ignored. 2020-08-24 12:34:46 -07:00
Steve Howell 95fe690e8d webhook tests: Avoid lots of unnecessary json dumping.
Commit c4254497b2
curiously had get_body() round tripping its data
through json load and dump.

I have seen this done for pretty-printing reasons,
but it doesn't apply here.

And if you're doing it for validation reasons,
you only need to do half the work, as my commit
here demonstrates.

We arguably don't even need the fail-fast code
here, since our fixtures are linted to be proper
json, I believe, plus downstream code probably
gives reasonably easy-to-diagnose symptoms.
2020-08-24 12:34:46 -07:00
Steve Howell 6d65af1790 webhook tests: Remove needless get_body functions.
These all did essentially what the parent implementation
does, except for checking that it's valid json.
2020-08-24 12:34:46 -07:00
Steve Howell 7fbe08f515 webhook tests: Introduce get_payload.
We introduce get_payload for the relatively
exceptional cases where webhooks return payloads
as dicts.

Having a simple "str" type for get_body will
allow us to extract test helpers that use
payloads from get_body() without the ugly
`Union[str, Dict[str, str]]` annotations.

I also tightened up annotations in a few places
where we now call get_payload (using Dict[str, str]
instead of Dict[str, Any]).

In the zendesk test I explicitly stringify
one of the parameters to satisfy mypy.
2020-08-24 12:34:46 -07:00
Steve Howell f7e4cc28eb mypy: Use str for get_body in bitbucket.
We don't need to claim a union type here.
2020-08-24 12:34:46 -07:00
Steve Howell 025d1fa180 webhook tests: Require args for check_webhook.
We tighten up the mypy types here.  And then
once we know that expected_message and expected_topic
are never None, we don't have call the do_test_message
and do_test_topic helpers any more, so we eliminate
them, too.

Finally, we don't return a message, since no tests
use the message currently.
2020-08-24 12:34:46 -07:00
Steve Howell fae3535e79 webhook tests: Avoid check_webhook for multi messages.
If we're not passing in expected_topic or expected_message
to check_webhook, it's better to just call send_webhook_payload,
since we'll want to explicitly check our messages
anyway.

This preps us to always require those fields for
check_webhook, which can prevent insidious testing no-ops.
2020-08-24 12:34:46 -07:00
Steve Howell f74aa29a1c webhook tests: Extract assert_stream_message.
This forces us to be a bit more explicit about testing
the three key values in any stream message, and it
also de-clutters the code a bit.  I eventually want
to phase out do_test_topic and friends, since they
have the pitfall that you can call them and have them
do nothing, because they don't actually require
values to be be passed in.

I also clean up the code a bit for the tests that
have two new messages arriving.
2020-08-24 12:34:46 -07:00
Steve Howell 3a710ab996 webhook tests: Remove stream_name parameter.
Having an optional stream_name parameter makes
it confusing to read the code if you know your
webhook is sending private messages.

And then the other two callers are already
checking topics, so they might as well check
stream names, too.

We also have the two stream-oriented callers
make their own call to "subscribe".  And we
future-proof this by making sure the exception
for no-message-being-sent calls out that gotcha.

Somewhat in passing, we now assert that
self.STREAM_NAME is not None in the main
helper.  This is partly to satisfy mypy, but
it's also a good sanity check.

This also sets the stage for the next commit,
where I'll add an assert_stream_message helper.
2020-08-24 12:34:46 -07:00
Steve Howell 2fdf966d9f webhook tests: Rename helper to send_webhook_payload.
Not all webhook payloads are json, so send_json_payload was a
bit misleading.

In passing I also remove "bytes" from the Union type for
"payload" parameter.
2020-08-24 12:34:46 -07:00
Steve Howell 388053db6b webhook tests: Rename main helper to check_webhook.
Almost all webhook tests use this helper, except a few
webhooks that write to private streams.

Being concise is important here, and the name
`self.send_and_test_stream_message` always confused
me, since it sounds you're sending a stream message,
and it leaves out the webhook piece.

We should consider renaming `send_and_test_private_message`
to something like `check_webhook_private`, but I couldn't
decide on a great name, and it's very rarely used.  So
for now I just made sure the docstrings of the two
sibling functions reference each other.
2020-08-24 12:34:46 -07:00
Steve Howell 00001a396b webhook tests: Shorten topic constants.
The "EXPECTED_" prefix and "_EVENTS" suffix
usually provided more noise than signal.

We also use module constants to avoid the "self."
noise.  It also makes it a bit more clear which
constants actually have to be in the class (e.g.
"FIXTURE_DIR_NAME") to do their job.
2020-08-24 12:34:46 -07:00
Steve Howell 0f970c6971 minor: Avoid do_test_message.
This function is a bad idea, as it leads to a possible situation
where you aren't actually testing anything:

    def do_test_message(self, msg: Message, expected_message: Optional[str]) -> None:
        if expected_message is not None:
            self.assertEqual(msg.content, expected_message)

Unfortunately, it's called deep in the stack in some places, but
we can safely replace it with assertEqual here.
2020-08-24 12:34:46 -07:00
Steve Howell fbef6e3c03 zendesk tests: Fix anti-patterns.
We had optional parameters for expected_topic and
expected_message, which are trivial to eliminate,
since the integration is really simple.

And we were doing strange things trying to reset
class variables at the end of tests.  Now we just
set them explicitly in the tests.
2020-08-24 12:34:46 -07:00
Steve Howell 3de2e4f034 dialogflow test: Fix nonsensical tests.
The test helper here was taking an "expected_topic"
parameter that it just ignored, and then the
dialogflow tests were passing in expected messages
in that slot, so the actual "expected_message" var
was "None" and was ignored.  So the tests weren't
testing anything.

Now we eliminate the crufty expected_topic parameter
and require an actual value for "expected_message".

I also clean up the mypy type for content_type,
and I remove the `content_type is None` check,
since all callers either pass in a str content
type or default to "application/json".
2020-08-24 12:34:46 -07:00
Alex Vandiver ad8943a64a url_preview: Only extract img tags with an `src`.
Some `<img>` tags do not have an SRC, if they are rewritten using JS
to have one later.  Attempting to access `first_image['src']` on these
will raise an exception, as they have no such attribute.

Only look for images which have a defined `src` attribute on them.  We
could instead check if `first_image.has_attr('src')`, but this seems
only likely to produce fewer valid images.
2020-08-18 14:26:21 -04:00
Alex Vandiver 5b74de7be7 markdown: Add another twitter code to retry-later.
Error code 131 is documented to be an arbitrary server error on
Twitter's side; add it to the retry list.
2020-08-18 10:32:24 -07:00
Alex Vandiver 092ed87ae3 markdown: Cache Twitter 403 responses that are semi-permanent.
03ca3afbc2 added more codes that are equivalent to 404's; this adds to
the list of cache-as-None codes a couple which are equivalent to
403's.  It does not comprise _all_ possible 403-like codes -- many of
them are "the client is not OK," which is relevant to log as an error
still.
2020-08-18 10:32:24 -07:00
Alex Vandiver aa5f98907c sentry: Deal more gracefully with missing stacktraces.
Not all languages produce exceptions with context or filenames.  See
https://develop.sentry.dev/sdk/event-payloads/stacktrace/
2020-08-18 11:17:42 -04:00
Alex Vandiver 0db311ddad sentry: Stacks are returned most-recent last.
Per [1], the sentry API returns frames sorted from oldest to newest.
As such, matching against the first filename that matches is most
likely not the right frame.

Match against the last frame with the guilty filename.

[1] https://develop.sentry.dev/sdk/event-payloads/stacktrace/
2020-08-18 11:17:42 -04:00
Alex Vandiver 0078f16f06 tests: Add an explicit order_by to prevent flakes.
Without an order_by, this can fail spuriously.
2020-08-18 11:15:35 -04:00
Steve Howell 33d7a22685 test_events: Fix loop structure in settings test.
The original commit was broken here:
b553507412

The intention was to run the same loop for all
settings, but instead, we did a funny loop of
just resetting schema_checker, and then we only
actually tested the last value of the loop.
2020-08-17 13:00:47 -04:00
Steve Howell f33a314a4d openapi: Add enum for stream role. 2020-08-16 08:35:18 -04:00
sahil839 f046c9c58a streams: Add role field to Subscription objects passed to clients.
This commit adds "role" field to the Subscription objects passed to
clients.  This is important preparation for being able to work on the
frontend for this feature.
2020-08-14 16:33:11 -07:00
Tim Abbott f94a8adf9e export: Remove duplicate 'analytics' zerver_realm object.
This fixes a harmless duplication of data in the Zulip data export
format.
2020-08-14 15:45:11 -07:00
arpit551 7568f6f9a8 export: Renamed zerver_analytics to zerver_realm.
While exporting analytics data we were using wrong table name
'zerver_analytics' in analytics config. Renamed it with
correct table name 'zerver_realm'.
2020-08-14 15:45:11 -07:00
arpit551 0d6047840b decorator: Updated user_passes_test function from Django 2.2.
Since bug https://bugs.python.org/issue3445 was resolved in Python
3.3, we can avoid the use of assigned=available_attrs(view_func) in
wraps decorator (which we were only using because we'd copied code
that handled that from Django).

Also available_attrs is now depreciated from Django 3.0 onwards.
2020-08-14 11:40:13 -07:00
arpit551 af3a34fbca cache: Used lru_cache from functools instead of django.utils.lru_cache.
Django 3.0 removed private Python 2 compatibility APIs
so used lru_cache() directly from functools.

We cast lru_cache to Any to avoid attr-defined error in mypy since we
are adding extra field, 'key_prefix', to this object later.
2020-08-14 11:34:04 -07:00
Alex Vandiver 2b6989a40f queue: Remove a no-longer-correct comment.
This comment stopped being true in 5686821150, and very much stopped
being relevant in dd40649e04 when the middleware entirely stopped
publishing to a queue.
2020-08-14 11:30:13 -07:00
Tim Abbott f2c9ee8000 migrations: Fix migration 0283 with orjson.
This function now matches the copy in zerver/lib/actions.py.

This is the same migration as
b250e42f61c525029bd2b3bbb8f4ea93ece62072; orjson enforces that we
don't use integers as keys in JSON dictionaries.
2020-08-14 11:22:03 -07:00
Tim Abbott b494b16791 meta tags: Use open graph titles/descriptions.
Apparently, we were incorrectly using constants for title/description
rather than the nice non-constant values from og:title and
og:description in our meta tags.
2020-08-14 10:58:18 -07:00
Steve Howell 9b6da158ad openapi: Add enum values to role node in specs.
We will eventually want to improve descriptions here
too.
2020-08-14 10:40:29 -04:00
Steve Howell 5931ebffd2 openapi: Use enum for message_type. 2020-08-14 10:40:29 -04:00
Clara Dantas 05bf72a75c attachments: Add is_web_public field.
This commit adds the is_web_public field in the AbstractAttachment
class. This is useful when validating user access to the attachment,
as otherwise we would have to make a query in the db to check if
that attachment was sent in a message in a web-public stream or not.
2020-08-12 17:26:03 -07:00
sahil839 ca1a8ac78f streams: Allow stream admin to update and deactivate streams.
The new Stream administrator role is allowed to manage a stream they
administer, including:
* Setting properties like name, description, privacy and post-policy.
* Removing subscribers
* Deactivating the stream

The access_stream_for_delete_or_update is modified and is used only
to get objects from database and further checks for administrative
rights is done by check_stream_access_for_delete_or_update.

We have also added a new exception class StreamAdministratorRequired.
2020-08-12 17:02:01 -07:00
sahil839 78da9fd3ab subscription: Add role field to Subscription class.
This commit adds role field to the Subscription class. Currently,
there are two option of roles - STREAM_ADMINISTRATOR and MEMBER.

We also add a property 'is_stream_admin' for checking whether the
user is stream admin or not.
2020-08-12 16:47:04 -07:00
Aman Agrawal 9f9daeea5b message_fetch: Allow access to web-public msgs for unauth users.
Via API, users can now access messages which are in web-public
streams without any authentication.

If the user is not authenticated, we assume it is a web-public
query and add `streams:web-public` narrow if not already present
to the narrow. web-public streams are also directly accessible.

Any malformed narrow which is not allowed in a web-public query
results in a 400 or 401. See test_message_fetch for the allowed
queries.
2020-08-12 16:39:02 -07:00