Commit Graph

12828 Commits

Author SHA1 Message Date
Max Zawisa 0e40cc72af newrelic: Added owner field and cleaned up code.
I reformatted the tests and view to include information about who
acknowledged and closed the alert. Only includes the information about
the owner if there was an owner.

Made a few small changes to the refactored bit as requested in review.
2020-12-15 12:04:46 -08:00
Max Zawisa 57e847ab89 newrelic: refactor of time input handling.
Moved time formatting check and conversion to
zerver/lib/webhooks/common.py. Updated tests slightly to match new
output. Removed duration from the calculation because the difference
is less than the precision of output and it complicated the error
handling.
2020-12-15 12:04:46 -08:00
Max Zawisa ec00557962 docs: Updated New Relic documentation.
The docs are updated to work with the new webhook and new process on
https://one.newrelic.com.
2020-12-15 12:04:46 -08:00
Mateusz Mandera b652cc786c django3: Remove remaining postgresql_psycopg2 use.
Removed in Django 3.0.
2020-12-15 11:52:32 -08:00
angela s 64becb20b5
logging: Set decorator tests to use assertLogs.
Fixes part of #15331.
2020-12-15 11:46:25 -08:00
Alex Vandiver 438d2aa632 digests: Ensure that the teaser_data can be JSON-serialized.
Leaving this as a set means that it fails in zerver.lib.send_email
when serializing into a ScheduledEmail object.
2020-12-15 11:44:50 -08:00
Alex Vandiver 7c849fa940 slack: Check token access scopes before importing.
The Slack API always (even for failed requests) puts the access scopes
of the token passed in, into "X-OAuth-Scopes"[1], which can be used to
determine if any are missing -- and if so, which.

[1] https://api.slack.com/legacy/oauth-scopes#working-with-scopes
2020-12-15 11:33:15 -08:00
Anders Kaseorg 415897f491 api docs: Use normal async/await code in JavaScript examples.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-12-15 11:32:18 -08:00
Anders Kaseorg bf45f921a7 url_preview: Allow Beautiful Soup to get the charset from <meta>.
An HTML document sent without a charset in the Content-Type header
needs to be scanned for a charset in <meta> tags.  We need to pass
bytes instead of str to Beautiful Soup to allow it to do this.

Fixes #16843.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-12-15 11:30:57 -08:00
Siddharth Asthana daac7536f3 accounts/deactivated: Show deactivated_redirect url if present
If a user visits a realm which has been deactivated and it's
deactivated_redirect field is set, we should have a message telling the
user that the realm has moved to the deactivated_redirect url.
2020-12-14 21:04:52 -08:00
Siddharth Asthana 82f5759299 Realm: Add a deactivated_redirect URLField to Realm object.
We export a realm's data, and disable the realm, because the user
is moving from Zulip Cloud (e.g. https://example.zulipchat.com/) to
self-hosting or another platform (e.g. https://zulip.example.com/)
which we do not control. This commit adds a field in the realm object
called deactivated_redirect to store the url to which the realm has
moved.
2020-12-14 21:04:52 -08:00
Anders Kaseorg 2c5e9f65f8 eslint: Fix new-cap errors.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-12-10 19:52:22 -08:00
Puneeth Chaganti 5dc3489166 webhooks/sentry: Fix URL generated in transform_webhook_payload.
The URL incorrectly had `event` in the URL path, instead of `events`.

Closes #16783
2020-12-02 12:28:45 -08:00
Puneeth Chaganti b7a08323aa webhooks/sentry: Use received key when timestamp key is absent. 2020-12-02 12:28:45 -08:00
Sundar Guntnur cbb7fb8ac0 anchor_value: Fix parsing of large anchor values.
This handles the conditions when anchor values are larger than
LARGER_THAN_MAX_MESSAGE_ID by clamping them down to it.  Also added
tests for the function parse_anchor_value.

Fixes #16768.
2020-12-02 11:00:22 -08:00
Mateusz Mandera 43a0c60e96 exceptions: Make RateLimited into a subclass of JsonableError.
This simplifies the code, as it allows using the mechanism of converting
JsonableErrors into a response instead of having separate, but
ultimately similar, logic in RateLimitMiddleware.
We don't touch tests here because "rate limited" error responses are
already verified in test_external.py.
2020-12-01 13:40:56 -08:00
Steve Howell 92ce2d0e31 events: Fix apply_event for streams.
In 1bcb8d8ee8 I made
it so the webapp doesn't include "streams" in its
state from `fetch_initial_state_data`, but I didn't
address all the places in apply_event.
2020-12-01 13:01:38 -08:00
Steve Howell c566ecfb30 minor: Remove dead code in events test. 2020-12-01 13:01:38 -08:00
Vishnu KS dabbc3445a webhooks: Properly format the currency amount for refunds.
By default all Stripe API amounts are in the currency's smallest unit.
It's upto us to convert it to a bigger unit and show it to the end user.
And refund event used to show the currency in the smallest unit which makes
the output wrong when it comes to most currencies like USD, Europ, INR etc
which uses a bigger unit(eg Dollar instead of Cents) as the standard.
2020-11-29 18:11:24 -08:00
Max Zawisa f05a04e000
webhooks: Update NewRelic webhook for new format.
Update the New Relic webhook and tests to match the format specified
in the New Relic documentation. The new format sends a json body
instead of using url parameters. The old format is no longer supported
by New Relic according to their support staff; as a result, the fixtures for 
the old test cases were removed. Added fixtures for new test cases.

Fixes: #16393.
2020-11-18 16:19:08 -08:00
Anders Kaseorg 13e35bfa94 mypy: Use sqlalchemy-stubs.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-11-16 18:17:41 -08:00
Anders Kaseorg 8e0240300a message_fetch: Skip intermediate mutation in limit_query_to_range.
This avoids extra mypy annotations.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-11-16 18:17:41 -08:00
Anders Kaseorg d0d8c358b3 lint: Migrate typing.Text check to semgrep.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-11-16 18:17:41 -08:00
Steve Howell 99e725cbde populate_db: Simplify how we create reactions.
For 3000 messages and 400 users, this saved
about 30 seconds.

We only do two queries per batch of messages
now, and the algorithm is easier to analyze,
as it's just three nested loops.
2020-11-16 17:19:23 -08:00
Vishnu KS 5eb63ddb7a webhooks: Handle dispute events with object IDs prefixed with du.
Sometimes the dispute object IDs are prefixed with `du` instead of `dp`.

https://freenode.logbot.info/stripe/20200605#c4059469

The correct long-term fix here would be to stop using object IDs to
detect the object type of these events and instead maybe make use of
"object" key instead.

https://stripe.com/docs/api/disputes/object#dispute_object-object
2020-11-16 17:05:54 -08:00
Steve Howell e2e0f06b2a email digests: Call get_recent_topics once per batch.
Once we start processing digests in batch, this will
let us amortize the expense of the message query
over multiple users.
2020-11-16 08:59:29 -08:00
Steve Howell 428f0564a0 minor: Move context code down in the function.
This will make a subsequent diff a bit less noisy.
2020-11-16 08:59:29 -08:00
Steve Howell 1d1e45e9ec digests: Use UserActivityInterval for user activity.
Note that we are much more efficient about finding
active users here:

    - we do one query per realm (instead of per-user)
    - we pass the cutoff date to the database
    - we get back just a list of distinct ids
2020-11-16 08:59:29 -08:00
Steve Howell b52f56080e performance: Just get user_ids to queue digest emails. 2020-11-16 08:59:29 -08:00
Steve Howell e13e5d104d refactor: Only require user_id for inactive_since().
This function is going away completely soon.  It is
querying everybody's entire UserActivity history instead
of passing the cutoff date to the database!
2020-11-16 08:59:29 -08:00
Steve Howell d0260392f7 digests: Get user objects from the database.
The query counts increase here for somewhat
contrived reasons.  The tests before this
commit reflected a successful trip to the
UserProfile cache, but that's not actually
realistic in practice.
2020-11-16 08:59:29 -08:00
Steve Howell 7737413cec digest tests: Improve gather_new_streams test.
We don't need to mock the dates here.  We also
explicitly clear out all streams first, and then
we explicitly test with both the stream being
current and the stream being old.
2020-11-16 08:59:28 -08:00
Steve Howell 9538edde06 digest tests: Simplify bots test.
We can use the _enqueue_emails_for_realm helper
to avoid all the Tuesday-related logic here.

We also don't bother to create UserActivity
records, since the bot gets excluded by virtue
of its being a bot.  (Also, the date ranges
here were sketchy due to the time mocking.)
2020-11-16 08:59:28 -08:00
Steve Howell 0624833af6 digest tests: Improve Tuesday tests.
If we're mocking time, we should do it consistently.
2020-11-16 08:59:28 -08:00
Steve Howell 2f4d7a6171 tests: Fix test_inactive_users_queued_for_digest.
We can avoid all the date mocking now for all
but a couple tests that exercise the is-it-Tuesday
logic.

And this test now correctly tests that we exclude
recently active users.

And this allows us to remove the other test.
2020-11-16 08:59:28 -08:00
Steve Howell e49a482baf email digests: Make transactions atomic. 2020-11-16 08:59:28 -08:00
Steve Howell cf6bcfb84a digest emails: Exclude users who had recent digests.
This code protects us in case we ever need to re-run
email digests twice in the same day.
2020-11-16 08:59:28 -08:00
Steve Howell fb3d4c1618 digest tests: Avoid warnings about naive time. 2020-11-16 08:59:28 -08:00
Steve Howell 4271442fba email digests: Write RealmAuditLog rows. 2020-11-16 08:59:28 -08:00
Mateusz Mandera 4f47f35cb4 auth: Handle the case of invalid subdomain at /fetch_api_key endpoint. 2020-11-13 16:43:17 -08:00
Anders Kaseorg 1275613812 requirements: Upgrade mypy to 0.790.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-11-12 15:44:30 -08:00
Anders Kaseorg 8ba95063d5 test_markdown: Construct FencedBlockPreprocessor with a real Markdown.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-11-10 15:54:28 -08:00
Anders Kaseorg e7e1fde6ec fenced_code: Use immutable type for codehilite_conf.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-11-10 15:54:28 -08:00
Anders Kaseorg fbf8ce0305 markdown: Add types for extra Markdown members.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-11-10 15:54:27 -08:00
Anders Kaseorg b48bdc65b9 markdown: Fix AlertWordNotificationProcessor.run type.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-11-10 15:54:27 -08:00
Anders Kaseorg 9573f6dc00 markdown: Fix build_block_parser type.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-11-10 15:54:27 -08:00
Anders Kaseorg 4398eecd2b markdown: Use immutable type for extension config.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-11-10 15:54:27 -08:00
Anders Kaseorg 060036dfd5 markdown: Merge build_engine into Markdown constructor.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-11-10 15:54:27 -08:00
Anders Kaseorg 08c64f5cfa markdown: Fix imports for compatibility with typeshed stubs.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-11-10 15:54:27 -08:00
Anders Kaseorg 2a8a59f548 test_queue_worker: Simplify worker_queue_names computation.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-11-10 15:46:04 -08:00