Commit Graph

893 Commits

Author SHA1 Message Date
Mateusz Mandera cbeeadab16 delete_realm: Register a post_delete Realm handler.
By registering a post_delete handler to clear appropriate caches in a
nicer way, we can get rid of the ugly flush-memcached call in the
delete_realm command.
2020-10-30 11:43:03 -07:00
Alex Vandiver bff503feb4 delete_realm: Add command to completely remove realms.
This will need some tweaking in upcoming commits.
2020-10-30 11:42:40 -07:00
Anders Kaseorg aaa7b766d8 python: Use universal_newlines to get str from subprocess.
We can replace ‘universal_newlines’ with ‘text’ when we bump our
minimum Python version to 3.7.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-10-30 11:36:38 -07:00
Anders Kaseorg 1802a50cc9 python: Use requests.Response.text instead of decoding content.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-10-30 11:36:38 -07:00
Alex Vandiver f4eae83542 export: Only include real, active humans in the displayed count. 2020-10-28 18:31:06 -07:00
Alex Vandiver 97745688ca docs: Link to the new doc home of the email gateway. 2020-10-28 12:13:04 -07:00
Alex Vandiver 1f7132f50d docs: Standardize on PostgreSQL, not Postgres. 2020-10-28 11:55:16 -07:00
sahil839 a96811ab58 management: Replace knight command with change_user_role command.
We replace knight command with change_user_role command which
allows us to change role of a user to owner, admins, member and
guest. We can also give/revoke api_super_user permission using
this command.

Tweaked by tabbott to improve the logging output and update documentation.

Fixes #16586.
2020-10-25 14:55:08 -07:00
Anders Kaseorg 72d6ff3c3b docs: Fix more capitalization issues.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-10-23 11:46:55 -07:00
Anders Kaseorg efa8dd3a47 compilemessages: Sort language list with Unicode Collation Algorithm.
Right now the list of languages in Display settings → Default language
is sorted in an unintuitive order due to the varying case conventions:

British English
Chinese (Taiwan)
Deutsch
English
Hindi
Indonesian (Indonesia)
Lietuviškai
Magyar
Malayalam
Nederlands
Português
Română
Tiếng Việt
Türkçe
català
español
français
galego
italiano
polski
suomi
svenska
česky
Русский
Українська
български
српски
فارسی
தமிழ்
日本語
简体中文
繁體中文
한국어

Fix the sort to use the locale-independent Unicode Collation
Algorithm:

British English
català
česky
Chinese (Taiwan)
Deutsch
English
español
français
galego
Hindi
Indonesian (Indonesia)
italiano
Lietuviškai
Magyar
Malayalam
Nederlands
polski
Português
Română
suomi
svenska
Tiếng Việt
Türkçe
български
Русский
српски
Українська
فارسی
தமிழ்
한국어
日本語
简体中文
繁體中文

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-10-21 15:22:18 -07:00
Steve Howell 116a441bc5 refactor: Introduce SubInfo class.
This class removes a lot of the annoying tuples
we were passing around.

Also, by including the user everywhere, which
is easily available to us when we make instances
of SubInfo, it sets the stage to remove
select_related('user_profile').
2020-10-14 10:53:10 -07:00
Steve Howell 9df9934ed6 refactor: Pass realm to bulk_add_subscriptions.
I think it's important that the callers understand
that bulk_add_subscriptions assumes all streams
are being created within a single realm, so I make
it an explicit parameter.

This may be overkill--I would also be happy if we
just included the assertions from this commit.
2020-10-13 11:28:17 -07:00
Alex Vandiver f3ba227614 create_user: Strip whitespace from initial password file.
Fixes #12144.
2020-10-11 16:29:00 -07:00
Anders Kaseorg b7a94be152 python: Catch BaseException when we need to clean something up.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-10-11 16:16:16 -07:00
Anders Kaseorg 7f69c1d3d5 python: Catch specific exceptions from requests.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-10-11 16:11:41 -07:00
Anders Kaseorg 234f7245cf export_usermessage_batch: Use os.rename.
This avoids an extra stat call to check whether the target is a
directory.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-10-11 16:11:35 -07:00
Anders Kaseorg 83eca256a4 compilemessages: Use polib for get_name_from_po_file.
This also corrects the name of zh_TW from “Chinese” to
“Chinese (Taiwan)”.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-10-11 16:11:35 -07:00
Anders Kaseorg c9fec8f021 deliver_scheduled_messages: Don’t do_send_messages inside a transaction.
do_send_messages has side effects outside the database and may not
work reliably if its database effects are reordered by being inside a
transaction.

This also fixes a bug where we were doing the update incorrectly on
the Message table.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-10-11 16:09:22 -07:00
Alex Vandiver 179c387409 tornado: Switch to start_json_consumer interface. 2020-10-11 14:19:42 -07:00
Anders Kaseorg e1a3aa9c1a deliver_scheduled_messages: Exit cleanly on Ctrl+C.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-10-07 16:15:19 -07:00
Alex Vandiver d47637fa40 queue: Set a max consume timeout with SIGALRM.
SIGALRM is the simplest way to set a specific maximum duration that
queue workers can take to handle a specific message.  This only works
in non-threaded environments, however, as signal handlers are
per-process, not per-thread.

The MAX_CONSUME_SECONDS is set quite high, at 10s -- the longest
average worker consume time is embed_links, which hovers near 1s.
Since just knowing the recent mean does not give much information[1],
it is difficult to know how much variance is expected.  As such, we
set the threshold to be such that only events which are significant
outliers will be timed out.  This can be tuned downwards as more
statistics are gathered on the runtime of the workers.

The exception to this is DeferredWorker, which deals with quite-long
requests, and thus has no enforceable SLO.

[1] https://www.autodesk.com/research/publications/same-stats-different-graphs
2020-10-06 17:26:14 -07:00
sahil839 6c473ed75f message: Call build_message_send_dict from check_message.
We call build_message_send_dict from check_message instead of
do_send_messages.

This is a prep commit for adding a new setting for handling
wildcard mentions in large streams.
2020-09-29 17:18:04 -07:00
Alex Vandiver 4354386e69 tornado: Remove an unused port argument.
This was added in ec065e92ee for the WebSocket codepath, which was
subsequently removed in ea6934c26d.
2020-09-18 15:13:40 -07:00
Alex Vandiver de1db2c838 sentry: Provide more metadata in queue processors.
This allows aggregation by queue, makes the event data more readily
accessible, and clears out the breadcrumbs upon every batch that is
serviced.
2020-09-18 15:13:08 -07:00
Tim Abbott 9c0d6becc5 docs: Add comment links to i18n documentation. 2020-09-18 11:44:04 -07:00
Alex Vandiver 812af977d3 compilemessages: Switch to canonical override_language(). 2020-09-14 22:16:09 -07:00
Anders Kaseorg f91d287447 python: Pre-fix a few spots for better Black formatting.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-09-03 17:51:09 -07:00
Anders Kaseorg bb4fc3c4c7 python: Prefer --flag=option over --flag option.
For less inflation by Black.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-09-03 17:51:09 -07:00
Anders Kaseorg bb5a963086 soft_deactivate_users: Elide default=[] for users argument.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-09-03 16:17:14 -07:00
Anders Kaseorg ddcf8dd8bc create_user: Use None as default for --password, --password-file.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-09-03 16:17:14 -07:00
Anders Kaseorg a50fae89e2 python: Elide type=str from argparse arguments.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-09-03 16:17:14 -07:00
Anders Kaseorg fbfd4b399d python: Elide action="store" for argparse arguments.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-09-03 16:17:14 -07:00
Anders Kaseorg 1f2ac1962f python: Elide default=None for argparse arguments.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-09-03 16:17:14 -07:00
Anders Kaseorg 3c5b39da9c python: Elide nargs for argparse flag arguments.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-09-03 16:17:14 -07:00
Anders Kaseorg b4597a8ca8 python: Elide default for store_{true,false} argparse arguments.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-09-03 16:17:14 -07:00
Anders Kaseorg ab120a03bc python: Replace unnecessary intermediate lists with generators.
Mostly suggested by the flake8-comprehension plugin.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-09-02 11:15:41 -07:00
Anders Kaseorg 1ded51aa9d python: Replace list literal concatenation with * unpacking.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-09-02 11:15:41 -07:00
Anders Kaseorg a5dbab8fb0 python: Remove redundant dest for argparse arguments.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-09-02 11:04:10 -07:00
Alex Vandiver b23a55e669 export: Add a --deactivate flag which happens just prior to export.
This removes a manual step during export.
2020-08-30 15:26:50 -07:00
Alex Vandiver 15d7e7a6fd export: Prompt for confirmation if a consent message is involved.
We display the text of the consent message, and then continue with the
export, which will scroll the content off the screen.  Allow the
administrator time to examine the contents of the message, and decide
whether to proceed based on that and the fraction of users that have
responded so far.
2020-08-30 15:26:50 -07:00
Alex Vandiver e3ba79b9df export: Show total number of users in the realm as well.
Knowing 10 users opted in isn't useful unless one knows if the realm
has 10 or 100 total users.
2020-08-30 15:26:50 -07:00
Alex Vandiver ea65b75875 export: Move tarball and directory creation as late as possible.
This stops creating empty tarballs and directories if the option
validation fails for some reason.
2020-08-30 15:26:50 -07:00
Alex Vandiver b4c2ae9cae settings: Adjust documentation and comment references to settings.py.
`zproject/settings.py` itself is mostly-empty now.  Adjust the
references which should now point to `zproject/computed_settings.py`
or `zproject/default_settings.py`.
2020-08-24 13:13:16 -07:00
Anders Kaseorg d0f4af5f8c python: Catch JSONDecodeError instead of ValueError when decoding JSON.
These weren’t wrong since orjson.JSONDecodeError subclasses
json.JSONDecodeError which subclasses ValueError, but the more
specific ones express the intention more clearly.

(ujson raised ValueError directly, as did json in Python 2.)

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-08-12 11:59:59 -07:00
Anders Kaseorg 61d0417e75 python: Replace ujson with orjson.
Fixes #6507.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-08-11 10:55:12 -07:00
Anders Kaseorg 6dea085187 docs: Correct “login” as a verb to “log in”.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-08-11 10:25:53 -07:00
Anders Kaseorg c155403884 docs: Fix various capitalization errors.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-08-11 10:25:52 -07:00
Anders Kaseorg 768f9f93cd docs: Capitalize Markdown consistently.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-08-11 10:23:06 -07:00
Anders Kaseorg 60a25b2721 docs: Fix spelling errors caught by codespell.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-08-11 10:23:06 -07:00
Alex Vandiver 2928bbc8bd logging: Report stack_info on logging.exception calls.
The exception trace only goes from where the exception was thrown up
to where the `logging.exception` call is; any context as to where
_that_ was called from is lost, unless `stack_info` is passed as well.
Having the stack is particularly useful for Sentry exceptions, which
gain the full stack trace.

Add `stack_info=True` on all `logging.exception` calls with a
non-trivial stack; we omit `wsgi.py`.  Adjusts tests to match.
2020-08-11 10:16:54 -07:00