Commit Graph

14071 Commits

Author SHA1 Message Date
Mateusz Mandera da815db372 test_external: Extract rate_limit_rule() decorator. 2021-08-06 12:17:44 +02:00
Mateusz Mandera ddcfd9e2ee rate_limit: Rate limit the /accounts/find/ endpoint.
Closes #19287

This endpoint allows submitting multiple addresses so we need to "weigh"
the rate limit more heavily the more emails are submitted. Clearly e.g.
a request triggering emails to 2 addresses should weigh twice as much as
a request doing that for just 1 address.
2021-08-06 12:17:44 +02:00
Mateusz Mandera 29b3e81dd4 test_external: Assert that requests don't get rate limited too soon.
This assertion was missing in do_test_hit_ratelimits, allowing a test to
pass even if the requests started being blocked sooner than expected.
2021-08-06 12:17:44 +02:00
Mateusz Mandera 0d6bb6d53b test_external: Fix incorrect clear_history() call.
This was missing the domain argument to RateLimitedIPAddr - without it,
this was clearing history on the default domain, which is api_by_ip.
2021-08-06 12:17:44 +02:00
Anders Kaseorg 7f0381d4c7 outgoing_http: Fix type annotations.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-08-05 18:34:38 -05:00
Tim Abbott cde86fbaca send_email: Move logging after dry-run abort.
Previously, the output would make it look like we sent an actual email
to the first user in the dry_run output, which is very confusing.

The `dry_run` code path already prints all the accounts that would
have been emailed at the end, so there's no reason to have this line
before the dry_run check.

Additionally, we move after the `get_connection` check because
failures at that stage shouldn't result in logging an attempt to send
an email.
2021-08-05 10:15:49 -07:00
Tim Abbott 36d15d85e0 send_custom_email: Only send to long_term_idle users. 2021-08-05 10:14:44 -07:00
tushar912 83f6557f43 integrations: Add SonarQube webhook integration.
Fixes #13395.
2021-08-05 09:53:42 -07:00
Anders Kaseorg 3a6d44b691 mention: Simplify init_user_group_data using prefetch_related.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-08-05 09:20:34 -07:00
Anders Kaseorg 806494da06 markdown: Stream and parse incrementally in fetch_open_graph_image.
This way we can stop reading as soon as we get to the body.  Also,
send an Accept header, check that the request was actually successful,
use lxml.etree.iterparse instead of a broken hand-rolled state
machine, and support XHTML, all for negative 28 lines of code.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-08-05 09:17:32 -07:00
Priyansh Garg 0a875c1c4c markdown: Fix jpeg extension in `IMAGE_EXTENSIONS`. 2021-08-05 08:54:02 -07:00
Iam-VM 285e10fe2f migrations: Fix possible 0257_fix_has_link_attribute.py failure.
While it should be an invariant that message.rendered_content is never
None for a row saved to the database, it is possible for that
invariant to be violated, likely including due to bugs in previous
versions of data import/export tools.

While it'd be ideal for such messages to be rendered to fix the
invariant, it doesn't make sense for this has_link migration to crash
because of such a corrupted row, so we apply the similar policy we
already have for rendered_content="".
2021-08-04 12:52:45 -07:00
Mateusz Mandera f1d5f3904d cache: Flush realm caches after both pre_delete and post_delete signals.
Fixes #19468.
2021-08-04 12:13:22 -07:00
Anders Kaseorg 905de1fdba email_notifications: Build HTML elements correctly.
correct_parser(f"incorrectly {generated} <html> string") is still
incorrect.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-08-04 11:29:20 -07:00
Anders Kaseorg 19424146d4 email_notifications: Fix make_emoji_img_elem type.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-08-04 11:29:20 -07:00
Tim Abbott 1685d207c8 api docs: Fix API documentation bugs issues with last commit.
* Actually bump API_FEATURE_LEVEL.
* Document changes in the endpoint documentation as well.
* Use standard linking/wording approach in API changelog.
2021-08-04 10:55:03 -07:00
Steve Howell 45f6c8d27f page load: Remove sender_ids in unread messages for streams. 2021-08-04 11:44:00 -04:00
Alya Abbott b679148788 portico: Revamp /for/companies.
We rework the landing page for companies in the same way we've
recently revamped the landing pages for other use cases.

This implementation unfortunately duplicates a lot of content from
/plans; we should clean that up at some point.
2021-08-03 21:54:38 -07:00
Aman Agrawal ae902ec421 urls: Add attribution page.
This records images we used from Creative Commons sources.

It's something of a placeholder for a nicer page.
2021-08-03 21:54:38 -07:00
Anders Kaseorg 42fa62e563 Revert "time_widget: Make the generated time string more readable."
This reverts commit 1965584eec.

This syntax has a bad interaction with table syntax and needs to be
rethought.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-08-03 16:45:31 -07:00
Gaurav Pandey 75bfdadf8c populate_db: Generate resolved topics for testing.
To try to match normal workflow, some streams have many resolved
topics and others have few.
2021-08-03 16:42:09 -07:00
Priyansh Garg 4815f6e28b data_import: Make slack bot emails unique.
Slack bot emails generated by us can be duplicate for two bots.
If such a case occur, append a counter to the email to make it
unique.

For maintaining the counter of duplicate emails and the final
email assigned to each bot, a class based approach is used with
static variables and static (class) methods. This keeps all the
data related to slack bot emails at the same place and easily
accessible from anywhere inside the module (without defining any
class object and passing it around).

Fixes: #16793
2021-08-03 16:18:14 -07:00
Lefteris Kyriazanos 2b70e88fda integrations: Add basic open collective integration.
Add basic open collective integration for the user donation
event.
Fixes #18319
2021-08-03 16:09:33 -07:00
Alex Vandiver e94b6afb00 nagios: Remove broken check_email_deliverer_* checks and related code.
These checks suffer from a couple notable problems:
 - They are only enabled on staging hosts -- where they should never
   be run.  Since ef6d0ec5ca, these supervisor processes are only
   run on one host, and never on the staging host.
 - They run as the `nagios` user, which does not have appropriate
   permissions, and thus the checks always fail.  Specifically,
   `nagios` does not have permissions to run `supervisorctl`, since
   the socket is owned by the `zulip` user, and mode 0700; and the
   `nagios` user does not have permission to access Zulip secrets to
   run `./manage.py print_email_delivery_backlog`.

Rather than rewrite these checks to run on a cron as zulip, and check
those file contents as the nagios user, drop these checks -- they can
be rewritten at a later point, or replaced with Prometheus alerting,
and currently serve only to cause always-failing Nagios checks, which
normalizes alert failures.

Leave the files installed if they currently exist, rather than
cluttering puppet with `ensure => absent`; they do no harm if they are
left installed.
2021-08-03 16:07:13 -07:00
Ganesh Pawar 1965584eec time_widget: Make the generated time string more readable.
Before: <time:2021-07-14T00:14:00-07:00>
After: <time:2021-07-14|00:14:00|UTC-07:00>

Fixes #19205
2021-08-02 23:17:01 -07:00
Anders Kaseorg 498d2b48d9 fenced_code: Use find_lexer_class_by_name.
This is more efficient than get_lexer_by_name, since we don’t need to
instantiate the class just to get its name.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-08-02 22:31:46 -07:00
Anders Kaseorg 04feadd917 mypy: Add pika-stubs.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-08-02 22:31:46 -07:00
Anders Kaseorg f87a0e912b test_queue: Fix ConnectionClosed usage.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-08-02 22:31:46 -07:00
Anders Kaseorg 9f8ba913fd queue: Fix _on_connection_open_error type to accept reason: str.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-08-02 22:31:46 -07:00
Anders Kaseorg f7e2426fc5 queue: Fix ensure_queue type to accept a callback returning any object.
channel.basic_consume actually returns str.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-08-02 22:31:46 -07:00
Anders Kaseorg 5e355abe2e queue: Add missing imports.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-08-02 22:31:46 -07:00
Anders Kaseorg 87799177b5 queue: Fix channel type for TornadoQueueClient.
The BlockingChannel annotations in TornadoQueueClient were flat-out
wrong.  BlockingChannel and Channel have no common base classes.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-08-02 22:31:46 -07:00
Anders Kaseorg 5751479932 queue: Switch TornadoQueueClient to the new base QueueClient.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-08-02 22:31:46 -07:00
Anders Kaseorg bd6a2b149c queue: Split common part of SimpleQueueClient into new base class.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-08-02 22:31:46 -07:00
Alex Vandiver 6fe67f0143 delete_realm: Allow deletion of realms with empty customers.
This is effectively a step closer to what was proposed in
https://github.com/zulip/zulip/pull/18678#discussion_r644490540 when
this code was written in #18678.

If the Customer object has neither of a Stripe id, nor any historical
plans, then there's no real billing association contained in the
existence of the Customer object, and it's safe to delete.
2021-08-02 22:29:16 -07:00
Tim Abbott 9968fb5081 send_custom_email: Fix emailing single users with TOS_VERSION set.
This code path previously threw an exception.
2021-08-02 17:57:16 -07:00
Anders Kaseorg fae92f2e3f actions: Convert TypedDict declaration to Python 3.6 style.
Generated by pyupgrade, with manual indentation adjustment for
https://github.com/asottile/pyupgrade/issues/507.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-08-02 15:53:52 -07:00
Anders Kaseorg 5483ebae37 python: Convert "".format to Python 3.6 f-strings.
Generated automatically by pyupgrade.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-08-02 15:53:52 -07:00
Anders Kaseorg 96234016f5 python: Remove obsolete directives.
Generated automatically by pyupgrade.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-08-02 15:53:52 -07:00
Anders Kaseorg ad5f0c05b5 python: Remove default "utf8" argument for encode(), decode().
Partially generated by pyupgrade.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-08-02 15:53:52 -07:00
Anders Kaseorg 1760897a8c python: Remove default "r" mode for open().
Generated automatically by pyupgrade.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-08-02 15:53:52 -07:00
Anders Kaseorg 47f795ae64 python: Remove redundant parentheses.
Generated automatically by pyupgrade.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-08-02 15:53:52 -07:00
Anders Kaseorg 3665deb93a python: Remove unnecessary intermediate lists.
Generated automatically by pyupgrade.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-08-02 15:53:52 -07:00
rht 1bbd36d181 slack_import: Remove obsolete SlackImportAttachment placeholder.
This was introduced in f4ad464d82, and
incompletely removed in e037c2f93e649c28a71c02559b5ae7a3333f42a8; here
we finish removing it.
2021-08-02 13:13:28 -07:00
rht 9441bca282 import_realm: Explain the usage of the relative_path var of attachment_path. 2021-08-02 13:13:19 -07:00
Tim Abbott 24b1439e93 email_notifications: Avoid trying to email to bot users.
This fixes a regression in de04f0ad67.

We'll do a proper test in a follow-up commit; this is a quick fix to
make sure master works.

The emails will bounce, but it'll create all sorts of infrastructure
headaches.
2021-08-02 13:13:01 -07:00
Sahil Batra 93b51088c0 api: Fix "Changes" entry for enter_sends in zulip.yaml.
This commit fixes the feature level for enter_sends in
`/regiser` response and also modifies the "Changes"
entry to match with other settings.
2021-08-02 09:23:56 -07:00
Sahil Batra c3f31bd507 api: Fix feature level for zulip_merge_base in zulip.yaml. 2021-08-02 09:23:56 -07:00
Tim Abbott d1dd34d7e0 send_custom_email: Add option for sending marketing emails. 2021-08-01 21:45:34 -07:00
Aman Agrawal c1c6674809 emails: Retheme to purple colors to match present logo. 2021-08-01 21:27:42 -07:00