Mirror
/
zulip
mirror of https://github.com/zulip/zulip.git
1
0
Fork 0
Code Issues Projects Releases Wiki Activity
36,484 Commits 28 Branches 142 Tags 697 MiB
Commit Graph

922 Commits

Author SHA1 Message Date
Tim Abbott 0b93e09e72 puppet: Add nginx configuration for blog.zulip.org move. 2020-05-26 14:47:05 -07:00
Anders Kaseorg f5b33f9398 python: Further pyupgrade changes. 
Signed-off-by: Anders Kaseorg <anders@zulip.com>
 2020-05-26 11:43:40 -07:00
Anders Kaseorg 333f7d16c9 logging: Pass more format arguments to logging. 
Commit bdc365d0fe (#14852) missed this
because of https://github.com/returntocorp/semgrep/issues/831.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
 2020-05-26 11:42:23 -07:00
Anders Kaseorg 824d97987b process_fts_updates: Use cursor.execute correctly. 
Commit b501d04f6a (#14841) missed this
because of https://github.com/returntocorp/semgrep/issues/831.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
 2020-05-26 11:42:23 -07:00
arpit551 439f0d3004 install: Ad production support for Zulip on Ubuntu Focal. 
Install script now runs on Focal.  Python 2 is now installed via the
`python2` package in Focal.
 2020-05-25 16:58:42 -07:00
Tim Abbott 220620e7cf sharding: Add basic sharding configuration for Tornado. 
This allows straight-forward configuration of realm-based Tornado
sharding through simply editing /etc/zulip/zulip.conf to configure
shards and running scripts/refresh-sharding-and-restart.

Co-Author-By: Mateusz Mandera <mateusz.mandera@zulip.com>
 2020-05-20 13:47:20 -07:00
Tim Abbott cdd3b7efbc tornado: Configure upstreams for TORNADO_PROCESSES. 2020-05-20 13:43:48 -07:00
Tim Abbott c3d3324295 puppet: Add link to the sources for Zephyr patches. 2020-05-19 20:54:11 -07:00
Tim Abbott a35e71ebbc puppet: Update package name for boto-on-python3. 
The python3-boto3 package is the maintained fork that supports Python
3; it was renamed in Ubuntu Bionic from the original Ubuntu Xenial name.
 2020-05-19 20:25:11 -07:00
Tim Abbott 1c28770810 puppet: Fix apt_repo_debathena setup_file path. 
There was a typo introduced here when scripts_path was added.
 2020-05-19 20:21:30 -07:00
Tim Abbott c43b3d95e2 puppet: Switch env-wal-e to use wal-g rather than wal-e. 
wal-g is the modern reimplementation of wal-e that supports current
postgres.  It requires a bit of extra configuration to specify the AWS
region.
 2020-05-15 16:45:36 -07:00
Anders Kaseorg fcca4a38b6 puppet: Work around memcached SASL configuration path bug. 
memcached 1.5.22 in Ubuntu 20.04 has a bug where it looks for its SASL
configuration at /etc/sasl2/memcached.conf/memcached.conf instead of
/etc/sasl2/memcached.conf.

https://bugs.launchpad.net/ubuntu/+source/memcached/+bug/1878721

Signed-off-by: Anders Kaseorg <anders@zulip.com>
 2020-05-14 23:25:24 -07:00
Tim Abbott b3c5f2c13e puppet: Remove check_postgres_replication_lag hostname hardcoding. 
Since this runs on the Nagios server, which already has the relevant
hostnames defined in zulip.conf, we can just read it from there.
 2020-05-11 23:42:36 -07:00
Tim Abbott 225bbf3633 puppet: Update check_postgres_replication_lag for postgres 10. 
These functions were renamed in postgres 10.
 2020-05-11 15:59:23 -07:00
Tim Abbott d8ea649869 puppet: Cast tornado_processes to Integer. 
This is the latest mechanism in puppet for turning a string into an
integer.

We update an adjacent comment while we're at it.
 2020-05-11 00:54:48 -07:00
Tim Abbott 6319c181eb puppet: Use actual name for the bind9-host package. 
Using the `host` virtual package confused Puppet into reporting it was
doing work every time one did a puppet run, resulting in unnecessarily
spammy output.
 2020-05-11 00:51:53 -07:00
Mateusz Mandera dd40649e04 queue_processors: Remove the slow_queries queue. 
While this functionality to post slow queries to a Zulip stream was
very useful in the early days of Zulip, when there were only a few
hundred accounts, it's long since been useless since (1) the total
request volume on larger Zulip servers run by Zulip developers, and
(2) other server operators don't want real-time notifications of slow
backend queries.  The right structure for this is just a log file.

We get rid of the queue and replace it with a "zulip.slow_queries"
logger, which will still log to /var/log/zulip/slow_queries.log for
ease of access to this information and propagate to the other logging
handlers.  Reducing the amount of queues is good for lowering zulip's
memory footprint and restart performance, since we run at least one
dedicated queue worker process for each one in most configurations.
 2020-05-11 00:45:13 -07:00
Tim Abbott 21a04e2dbc puppet: Use nice to deprioritize various processes. 
Our priority hierarchy is:
(1) Tornado and base services like memcached, redis, etc.
(2) Django and message sender queue workers.
(3) Everything else.

Ideally, we'd have something a bit more fine-grained (e.g. some queue
workers are potentially in the sending path, while others aren't), but
this should have a big impact on ensuring Tornado gets the resources
it needs during load spikes.

I think this has a good chance of causing some load spikes that would
previously have resulted in a user-facing delivery delays no longer
having any significant user-facing impact.
 2020-05-10 23:28:25 -07:00
shubhamgupta2956 9cd8644c7c uploads: Add support for ".jpe" file extension. 
Currently when the user uploads files with ".jpe" file extension, the
markdown is converted to link but the image is not embedded.

This commit adds the support for ".jpe" file extension.

Fixes #14863
 2020-05-10 22:55:52 -07:00
Anders Kaseorg 8cdf2801f7 python: Convert more variable type annotations to Python 3.6 style. 
Signed-off-by: Anders Kaseorg <anders@zulip.com>
 2020-05-08 16:42:43 -07:00
Anders Kaseorg 708c6f4f11 puppet: Finally vanquish the cursed integer conversion conditional. 
We no longer support Puppet 3.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
 2020-05-08 16:42:43 -07:00
Tim Abbott 50d8d61d3c puppet: Remove unnecssary/broken ;. 
This breaks the Xenial build, which we're removing soon, but it's
unnecessary in any case.
 2020-05-07 16:23:37 -07:00
Tim Abbott 03991d098a puppet: Add optional postgres version override. 
This makes it convenient to run an alternative postgres version.
 2020-05-07 09:33:24 -07:00
Mateusz Mandera 4643e48f60 retention: Add a daily cron job. 
This will run archive_messages management command at 6am every day, 1
hour after soft_deactivate_users (which runs at 5am).
 2020-05-05 10:11:38 -07:00
Tim Abbott 4034f6f99e nagios: Fix check_postgres_replication_lag. 
This expects to be run outside a virtualenv and thus without
typing_extensions available.
 2020-05-03 00:14:54 -07:00
Tim Abbott 4f3976b917 process_fts_updates: Clean up logging output. 
This saves a couple lines of spammy output in the run-dev.py startup
experience, and will be better output in production as well.
 2020-05-01 11:51:20 -07:00
Anders Kaseorg c0ffa71fa9 nginx: Replace unanchored regexes in location directives. 
We could anchor the regexes, but there’s no need for the power (and
responsibility) of regexes here.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
 2020-04-24 16:58:19 -07:00
Anders Kaseorg 5e01a0ae8b zulip-ec2-configure-interfaces: Convert function type annotations. 
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
 2020-04-24 13:06:54 -07:00
Anders Kaseorg f8339f019d python: Convert assignment type annotations to Python 3.6 style. 
Commit split by tabbott; this has changes to scripts/, tools/, and
puppet/.

scripts/lib/hash_reqs.py, scripts/lib/setup_venv.py,
scripts/lib/zulip_tools.py, and tools/lib/provision.py are excluded so
tools/provision still gives the right error message on Ubuntu 16.04
with Python 3.5.

Generated by com2ann, with whitespace fixes and various manual fixes
for runtime issues:

-shebang_rules: List[Rule] = [
+shebang_rules: List["Rule"] = [

-trailing_whitespace_rule: Rule = {
+trailing_whitespace_rule: "Rule" = {

-whitespace_rules: List[Rule] = [
+whitespace_rules: List["Rule"] = [

-comma_whitespace_rule: List[Rule] = [
+comma_whitespace_rule: List["Rule"] = [

-prose_style_rules: List[Rule] = [
+prose_style_rules: List["Rule"] = [

-html_rules: List[Rule] = whitespace_rules + prose_style_rules + [
+html_rules: List["Rule"] = whitespace_rules + prose_style_rules + [

-    target_port: int = None
+    target_port: int

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
 2020-04-24 13:06:54 -07:00
Anders Kaseorg 09ea778db1 nginx: Listen for ACME challenges on port 80 too. 
This should make Certbot renewals more reliable.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
 2020-04-23 16:22:04 -07:00
Aman Agrawal 2dc6d09c2a python3-upgrade: Move python2 scripts to run on python3. 2020-04-22 16:13:15 -07:00
Anders Kaseorg 5901e7ba7e python: Convert function type annotations to Python 3 style. 
Generated by com2ann (slightly patched to avoid also converting
assignment type annotations, which require Python 3.6), followed by
some manual whitespace adjustment, and six fixes for runtime issues:

-    def __init__(self, token: Token, parent: Optional[Node]) -> None:
+    def __init__(self, token: Token, parent: "Optional[Node]") -> None:

-def main(options: argparse.Namespace) -> NoReturn:
+def main(options: argparse.Namespace) -> "NoReturn":

-def fetch_request(url: str, callback: Any, **kwargs: Any) -> Generator[Callable[..., Any], Any, None]:
+def fetch_request(url: str, callback: Any, **kwargs: Any) -> "Generator[Callable[..., Any], Any, None]":

-def assert_server_running(server: subprocess.Popen[bytes], log_file: Optional[str]) -> None:
+def assert_server_running(server: "subprocess.Popen[bytes]", log_file: Optional[str]) -> None:

-def server_is_up(server: subprocess.Popen[bytes], log_file: Optional[str]) -> bool:
+def server_is_up(server: "subprocess.Popen[bytes]", log_file: Optional[str]) -> bool:

-    method_kwarg_pairs: List[FuncKwargPair],
+    method_kwarg_pairs: "List[FuncKwargPair]",

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
 2020-04-18 20:42:48 -07:00
pemontto fd34bc5161
puppet: Allow /etc/zulip to be a symlink. 
This PR updates the puppet manifest to allow /etc/zulip to be a symlink. The current behaviour overwrites /etc/zulip if it is link to another directory, which is problematic with docker-zulip and 
in particular the `LINK_SETTINGS_TO_DATA` setting.
 2020-04-17 12:45:05 -07:00
Tim Abbott 777a3b6c18 puppet: Fix nagios check to not require typing_extensions. 2020-04-16 17:56:05 -07:00
Tim Abbott e1ce53ac46 puppet: Update nagios checks for disk to exclude kernel filesystems. 
The fact that we have to explicitly list these is almost certainly a
bug in check_disk, but at least this works.
 2020-04-16 17:49:29 -07:00
Tim Abbott cfbb617f5c puppet: Update nagios configuration for checking local disk. 2020-04-16 17:48:36 -07:00
Tim Abbott 9821dfa9fc puppet: The letsencrypt package is debian is now certbot. 
It was an alias starting with Ubuntu Xenial, and will eventually be
removed.
 2020-04-16 17:30:01 -07:00
Tim Abbott 8e5a866122 puppet: Update tuning for load average monitoring. 2020-04-16 16:47:05 -07:00
Tim Abbott b1ff823798 puppet: Remove old zulipbot configuration. 
We haven't used zulipbot hosted here for years.
 2020-04-16 16:18:48 -07:00
Anders Kaseorg 99242138a7 static: Serve webpack bundles from the root domain. 
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
 2020-04-10 00:48:02 -07:00
Anders Kaseorg c734bbd95d python: Modernize legacy Python 2 syntax with pyupgrade. 
Generated by `pyupgrade --py3-plus --keep-percent-format` on all our
Python code except `zthumbor` and `zulip-ec2-configure-interfaces`,
followed by manual indentation fixes.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
 2020-04-09 16:43:22 -07:00
Vishnu KS 449f7e2d4b team: Generate team page data using cron job. 
This eliminates the contributors data as a possible source of
flakiness when installing Zulip from Git.

Fixes #14351.
 2020-04-08 12:52:31 -07:00
Anders Kaseorg 15d68c40dd nginx: Set X-XSS-Protection: 1; mode=block. 
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
 2020-04-05 16:13:53 -07:00
Anders Kaseorg 79c215626e nginx: Set X-Content-Type-Options: nosniff globally. 
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
 2020-04-05 16:13:53 -07:00
Anders Kaseorg 06e7d4ec19 nginx: Don’t override HSTS, X-Frame-Options with other ‘add_header’s. 
The nginx ‘add_header’ directive doesn’t inherit the way you’d
want (https://trac.nginx.org/nginx/ticket/854), so we need to manually
simulate inheritance using ‘include’, like we previously did with
api_headers.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
 2020-04-05 16:13:53 -07:00
Mateusz Mandera 5252b081bd queue_processors: Gather statistics on queue worker operations. 2020-04-01 16:44:06 -07:00
Stefan Weil d2fa058cc1
text: Fix some typos (most of them found and fixed by codespell). 
Signed-off-by: Stefan Weil <sw@weilnetz.de>
 2020-03-27 17:25:56 -07:00
Anders Kaseorg 7ff9b22500 docs: Convert many http URLs to https. 
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
 2020-03-26 21:35:32 -07:00
Anders Kaseorg 687553a661 setup_path_on_import: Replace with setup_path function. 
isort 5 knows not to reorder imports across function calls, so this
will stop isort from breaking our code.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
 2020-02-25 15:40:21 -08:00
Anders Kaseorg 9d598d95a6 puppet: Fix puppet-lint warning. 
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
 2020-02-20 19:51:48 -08:00