Commit Graph

2678 Commits

Author SHA1 Message Date
Alex Vandiver 6f5ae8d13d puppet: wal-g backups are required for replication.
Previously, it was possible to configure `wal-g` backups without
replication enabled; this resulted in only daily backups, not
streaming backups.  It was also possible to enable replication without
configuring the `wal-g` backups bucket; this simply failed to work.

Make `wal-g` backups always streaming, and warn loudly if replication
is enabled but `wal-g` is not configured.
2022-03-11 10:09:35 -08:00
Alex Vandiver 6496d43148 puppet: Only s3_backups_bucket is required for backups.
`s3_backups_key` / `s3_backups_secret_key` are optional, as the
permissions could come from the EC2 instance's role.
2022-03-11 10:09:35 -08:00
Alex Vandiver 19beed2709 puppet: Default s3_region to the current ec2 region. 2022-03-11 10:09:35 -08:00
Alex Vandiver bfdc547b00 docs: Document s3_region setting. 2022-03-11 10:09:35 -08:00
Anders Kaseorg 646e466341 install: Desupport Ubuntu 22.04 for now.
Ubuntu 22.04 pushed a post-feature-freeze update to Python 3.10,
breaking virtual environments in a Debian patch
(https://bugs.launchpad.net/ubuntu/+source/python3.10/+bug/1962791).
Also, our antique version of Tornado doesn’t work in 3.10, and we’ll
need to do some work to upgrade that.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-03-07 11:46:07 -08:00
Alya Abbott ba1f804518 docs: Update README and installation guide.
This is a general cleanup that also aims to link to the new
self-hosting page to provide added context.
2022-03-04 13:59:17 -08:00
Alya Abbott 1572e097d9 docs: Add a section on code review communication style. 2022-03-03 16:42:23 -08:00
Anders Kaseorg 9bdf230642 docs: Fix another case of Git capitalization.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-03-01 22:26:29 -08:00
Alya Abbott d90907b006 docs: Rewrite code review documentation. 2022-03-01 21:04:42 -08:00
Anders Kaseorg 3ba444096b Revert "docs: Temporarily disable parallel building."
This reverts commit b7561049b765946d612069b52330695e8489bc7f.

The bug it worked around was fixed with the previous commit’s
upgrade.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-02-28 16:28:31 -08:00
Anders Kaseorg 75525f5b53 docs: Convert .html#fragment links to .md#fragment.
This uses the myst_heading_anchors option to automatically generate
header anchors and make Sphinx aware of them.  See
https://myst-parser.readthedocs.io/en/latest/syntax/optional.html#auto-generated-header-anchors.

Note: to be compatible with GitHub, MyST-Parser uses a slightly
different convention for .md fragment links than .html fragment links
when punctuation is involved.  This does not affect the generated
fragment links in the HTML output.

Fixes #13264.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-02-28 16:28:31 -08:00
Anders Kaseorg dacf136f21 docs: Temporarily disable parallel building.
Work around https://github.com/executablebooks/MyST-Parser/issues/411.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-02-28 16:28:31 -08:00
Anders Kaseorg 17574f3689 docs: Replace manual fragment with automatic heading fragment.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-02-28 16:28:31 -08:00
Jai soni cfa9d50901
docs: Fix a typo in GSoC project ideas. 2022-02-28 16:25:13 -08:00
Anders Kaseorg 503127c2e9 docs: Fix Git capitalization.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-02-28 16:04:35 -08:00
Tim Abbott cc5ac302ab docs: Add more metadata to GSoC project ideas and add ideas. 2022-02-25 22:05:01 -08:00
Anders Kaseorg 894a50b5c9 install: Support Ubuntu 22.04.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-02-25 14:49:07 -08:00
Anders Kaseorg f852af0709 upgrade-zulip-stage-2: Set default PostgreSQL version for Debian 11.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-02-25 14:49:07 -08:00
Alex Vandiver 2e435d2e9a version: Update version and changelog after 4.10 release. 2022-02-25 14:10:20 -08:00
Alex Vandiver 538287261d Revert "docs: Suggest running reindex-textual-data in the venv manually."
This reverts commit be7108ebca.
2022-02-25 14:04:27 -08:00
Anders Kaseorg c19d6fb3ef docs: Clean redundant relative links.
We previously had a convention of redundantly including the directory
in relative links to reduce mistakes when moving content from one file
to another.  However, these days we have a broken link checker in
test-documentation, and after #21237, MyST-Parser will check relative
links (including fragments) when you run build-docs.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-02-24 16:12:18 -08:00
Anders Kaseorg e3572894c5 docs: Clean redundant fragment links to the same page.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-02-24 16:12:18 -08:00
Anders Kaseorg 1490c91011 docs: Fix list item indentation mistake.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-02-24 15:31:01 -08:00
Anders Kaseorg 21cd1c10b3 docs: Add missing space in “time zone”.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-02-24 14:05:12 -08:00
Anders Kaseorg b3260bd610 docs: Use Debian and Ubuntu version numbers over development codenames.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-02-23 12:04:24 -08:00
Anders Kaseorg 4c80f035b1 docs: Update documentation on AddIndexConcurrently.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-02-23 11:59:45 -08:00
Anders Kaseorg 1fa2761790 upgrade-zulip-stage-2: Remove create_large_indexes optimization.
This was only used for upgrading from Zulip < 1.9.0, which is no
longer possible because Zulip < 2.1.0 had no common supported
platforms with current main.

If we ever want this optimization for a future migration, it would be
better implemented using Django merge migrations.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-02-23 11:59:45 -08:00
Tim Abbott 1ac2eaa7f9 docs: Update GSoC project ideas for 2022. 2022-02-21 08:51:50 -08:00
Palash 6bdd7c7515 docs: Update internationalization statement in new feature tutorial. 2022-02-21 06:45:53 -08:00
Tim Abbott eb3ae7c3d9 docs: Update link to mypy casts documentation. 2022-02-17 12:42:37 -08:00
Anders Kaseorg 2b3a821807 docs: Remove highlight parameters from links.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-02-16 13:15:39 -08:00
jai2201 67625d6c7b message_feed: Remove the parenthesis around edited notices.
They were found to not particularly improve the UI.

Fixes #21043.
2022-02-10 13:52:24 -08:00
Alex Vandiver b7e309c95b docs: Add checklist steps to upgrade puppet dependencies. 2022-02-10 10:00:25 -08:00
Alya Abbott f098556ce1 help: Add a Project size section to GSoC contributor doc. 2022-02-09 14:15:07 -08:00
Anders Kaseorg e1f42c1ac5 docs: Add missing space to compound verbs “back up”, “log in”, etc.
Noun: backup, login, logout, lookup, setup.

Verb: back up, log in, log out, look up, set up.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-02-07 19:20:54 -08:00
Anders Kaseorg b0ce4f1bce docs: Fix many spelling mistakes.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-02-07 18:51:06 -08:00
Anders Kaseorg d8f4e0ffb9 webhooks: Correct OpsGenie to Opsgenie.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-02-07 18:51:06 -08:00
Anders Kaseorg 3e159446f0 docs: Update “G Suite” to “Google Workspace”.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-02-07 18:51:06 -08:00
Alex Vandiver d61914e8e1 docs: Minor grammar fix in settings.py upgrade docs. 2022-02-04 15:44:48 -08:00
Tim Abbott 39dea1feaf docs: Improve closing of life-of-a-request.
Reading the previous commit reminded me of various other details we
should document here.
2022-02-04 15:24:58 -08:00
Lauryn Menard c2b6e76af7 docs: Update mentions of json_success in documentation.
Updates developer and user-facing documentation that references
json_success for the addition of the request argument.
2022-02-04 15:16:56 -08:00
Steve Howell 498b02dfb2 presence: Avoid doc/naming references to "orange".
It seems like orange is the loudest possible color to
denote a quasi-neutral-idle state, so we hope to
replace it with another color.

This commit does not change any styling.

I removed the sentences in the doc, since they are
kind of too vague to be useful. If we want to say that
the idle state is correlated with the half-orange
circles in the buddy list, then we want to say that
more specifically.
2022-02-03 15:31:21 -08:00
Alex Vandiver 487e7ccd11 docs: Fix the path to the py3 venv.
`/srv/zulip-py3-venv` only exists on development hosts; use the path
to the current venv.
2022-01-29 16:52:11 -08:00
Steve Howell 0d376126f2 docs: Tweak example testing workflow.
The node test was slightly inaccurate.

I added `# etc.` to try to make it clear that
these are just examples.

I removed the puppeteer example, so that we
don't lie about "run in just a few seconds".
2022-01-28 11:46:42 -08:00
Lauryn Menard 1cbfa7e672 docs: Update gsoc "how to" guide to use contributor vs student.
Updates `docs/contributing/summer-with-zulip.md` to use
contributor, instead of student, due to changes in gsoc
eligibility and terminology.
2022-01-28 09:47:57 -08:00
Anders Kaseorg be7108ebca docs: Suggest running reindex-textual-data in the venv manually.
Until the previous commit makes its way into a release, we can
document this workaround.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-01-26 11:56:30 -08:00
Lauryn Menard c4eede68d6 docs: Rename `user.md` to `helpcenter.md`.
Renames `/docs/documentation/user.md` to reflect the rebranding
from "user documentation" to "help center documentation".

Also, fixes any linking in the docs to that file.
2022-01-25 18:42:04 -08:00
Lauryn Menard 8f7f486061 docs: Expand help center guide for updating existing articles.
Adds a section on updating existing help center articles to
the guide for writing help center documentation.
2022-01-25 18:42:04 -08:00
Lauryn Menard 784f923dda documentation: Update tutorial for writing help center documentation.
Updates the tutorial for writing help center articles to encourage
contributors to add to or enhance the existing help center docs
before writing a new articles for new features.

Also, generally updates references to 'user documentation' to be
'help center documentation'.

Additionally, updates some headers within the tutorials for clarity
and consistency, and adds some linkifying throughout the section on
writing documentation.
2022-01-25 18:42:04 -08:00
Anders Kaseorg 5de6dd66a7 docs: Adjust modinfo command for Debian.
On Debian, /sbin isn’t in the default path for normal users.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-01-25 12:50:36 -08:00
Raghav Luthra 4b8cb0a8a9 docs: Uncapitalize the name for nginx.
This matches how nginx refers to itself on its own website and
documentation.

Fixes #20887.
2022-01-25 11:17:51 -08:00
Alex Vandiver 7ec2a2cde7 version: Update version and changelog after 4.9 release. 2022-01-24 18:46:46 -08:00
Alex Vandiver a5496f4098 CVE-2021-43799: Set a secure Erlang cookie.
The RabbitMQ docs state ([1]):

    RabbitMQ nodes and CLI tools (e.g. rabbitmqctl) use a cookie to
    determine whether they are allowed to communicate with each
    other. [...] The cookie is just a string of alphanumeric
    characters up to 255 characters in size. It is usually stored in a
    local file.

...and goes on to state (emphasis ours):

    If the file does not exist, Erlang VM will try to create one with
    a randomly generated value when the RabbitMQ server starts
    up. Using such generated cookie files are **appropriate in
    development environments only.**

The auto-generated cookie does not use cryptographic sources of
randomness, and generates 20 characters of `[A-Z]`.  Because of a
semi-predictable seed, the entropy of this password is thus less than
the idealized 26^20 = 94 bits of entropy; in actuality, it is 36 bits
of entropy, or potentially as low as 20 if the performance of the
server is known.

These sizes are well within the scope of remote brute-force attacks.

On provision, install, and upgrade, replace the default insecure
20-character Erlang cookie with a cryptographically secure
255-character string (the max length allowed).

[1] https://www.rabbitmq.com/clustering.html#erlang-cookie
2022-01-25 02:13:53 +00:00
Alex Vandiver a46f6df91e CVE-2021-43799: Write rabbitmq configuration before starting.
Zulip writes a `rabbitmq.config` configuration file which locks down
RabbitMQ to listen only on localhost:5672, as well as the RabbitMQ
distribution port, on localhost:25672.

The "distribution port" is part of Erlang's clustering configuration;
while it is documented that the protocol is fundamentally
insecure ([1], [2]) and can result in remote arbitrary execution of
code, by default the RabbitMQ configuration on Debian and Ubuntu
leaves it publicly accessible, with weak credentials.

The configuration file that Zulip writes, while effective, is only
written _after_ the package has been installed and the service
started, which leaves the port exposed until RabbitMQ or system
restart.

Ensure that rabbitmq's `/etc/rabbitmq/rabbitmq.config` is written
before rabbitmq is installed or starts, and that changes to that file
trigger a restart of the service, such that the ports are only ever
bound to localhost.  This does not mitigate existing installs, since
it does not force a rabbitmq restart.

[1] https://www.erlang.org/doc/apps/erts/erl_dist_protocol.html
[2] https://www.erlang.org/doc/reference_manual/distributed.html#distributed-erlang-system
2022-01-25 01:48:05 +00:00
Alex Vandiver 43d63bd5a1 puppet: Always set the RabbitMQ nodename to zulip@localhost.
This is required in order to lock down the RabbitMQ port to only
listen on localhost.  If the nodename is `rabbit@hostname`, in most
circumstances the hostname will resolve to an external IP, which the
rabbitmq port will not be bound to.

Installs which used `rabbit@hostname`, due to RabbitMQ having been
installed before Zulip, would not have functioned if the host or
RabbitMQ service was restarted, as the localhost restrictions in the
RabbitMQ configuration would have made rabbitmqctl (and Zulip cron
jobs that call it) unable to find the rabbitmq server.

The previous commit ensures that configure-rabbitmq is re-run after
the nodename has changed.  However, rabbitmq needs to be stopped
before `rabbitmq-env.conf` is changed; we use an `onlyif` on an `exec`
to print the warning about the node change, and let the subsequent
config change and notify of the service and configure-rabbitmq to
complete the re-configuration.
2022-01-25 01:48:02 +00:00
Alex Vandiver 694c4dfe8f puppet: Admit we leave epmd port 4369 open on all interfaces.
The Erlang `epmd` daemon listens on port 4369, and provides
information (without authentication) about which Erlang processes are
listening on what ports.  This information is not itself a
vulnerability, but may provide information for remote attackers about
what local Erlang services (such as `rabbitmq-server`) are running,
and where.

`epmd` supports an `ERL_EPMD_ADDRESS` environment variable to limit
which interfaces it binds on.  While this environment variable is set
in `/etc/default/rabbitmq-server`, Zulip unfortunately attempts to
start `epmd` using an explicit `exec` block, which ignores those
settings.

Regardless, this lack of `ERL_EPMD_ADDRESS` variable only controls
`epmd`'s startup upon first installation.  Upon reboot, there are two
ways in which `epmd` might be started, neither of which respect
`ERL_EPMD_ADDRESS`:

 - On Focal, an `epmd` service exists and is activated, which uses
   systemd's configuration to choose which interfaces to bind on, and
   thus `ERL_EPMD_ADDRESS` is irrelevant.

 - On Bionic (and Focal, due to a broken dependency from
   `rabbitmq-server` to `epmd@` instead of `epmd`, which may lead to
   the explicit `epmd` service losing a race), `epmd` is started by
   `rabbitmq-server` when it does not detect a running instance.
   Unfortunately, only `/etc/init.d/rabbitmq-server` would respects
   `/etc/default/rabbitmq-server` -- and it defers the actual startup
   to using systemd, which does not pass the environment variable
   down.  Thus, `ERL_EPMD_ADDRESS` is also irrelevant here.

We unfortunately cannot limit `epmd` to only listening on localhost,
due to a number of overlapping bugs and limitations:

 - Manually starting `epmd` with `-address 127.0.0.1` silently fails
   to start on hosts with IPv6 disabled, due to an Erlang bug ([1],
   [2]).

 - The dependencies of the systemd `rabbitmq-server` service can be
   fixed to include the `epmd` service, and systemd can be made to
   bind to `127.0.0.1:4369` and pass that socket to `epmd`, bypassing
   the above bug.  However, the startup of this service is not
   guaranteed, because it races with other sources of `epmd` (see
   below).

 - Any process that runs `rabbitmqctl` results in `epmd` being started
   if one is not currently running; these instances do not respect any
   environment variables as to which addresses to bind on.  This is
   also triggered by `service rabbitmq-server status`, as well as
   various Zulip cron jobs which inspect the rabbitmq queues.  As
   such, it is difficult-to-impossible to ensure that some other
   `epmd` process will not win the race and open the port on all
   interfaces.

Since the only known exposure from leaving port 4369 open is
information that rabbitmq is running on the host, and the complexity
of adjusting this to only bind on localhost is high, we remove the
setting which does not address the problem, and document that the port
is left open, and should be protected via system-level or
network-level firewalls.

[1]: https://bugs.launchpad.net/ubuntu/+source/erlang/+bug/1374109
[2]: https://github.com/erlang/otp/issues/4820
2022-01-25 01:46:51 +00:00
Alya Abbott 669010494e portico: Update contributor count from 700 to 1000.
Note: I did not check whether we have numbers other than 700 that also
need to be updated.
2022-01-24 12:41:49 -08:00
Anders Kaseorg a58a71ef43 Remove Ubuntu 18.04 support.
As a consequence:

• Bump minimum supported Python version to 3.7.
• Move Vagrant environment to Debian 10, which has Python 3.7.
• Move CI frontend tests to Debian 10.
• Move production build test to Debian 10.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-01-21 17:26:14 -08:00
Alex Vandiver be1c4c2bd8 docs: Mention Camo does not use a local Smokescreen in the proxies docs.
This documents the new behaviour in d328d3dd4d.
2022-01-21 15:57:27 -08:00
Eeshan Garg 9e1fd26125 docs: Make general improvements to our billing docs.
With a few wording tweaks from tabbott to the "Upgrading a Zulip
organization" section.
2022-01-21 14:21:02 -08:00
rht 42f46a78e9 docs: Fix grammar problems found by LanguageTool.
With tweaks to security-model.md by tabbott to expand the SSO acronym.

Ignored, but still needs discussion on whether we should exclude this
rule:

```
The word ‘install’ is not a noun.
  ✗ ...ble to connect to the client during the install process:  So you'll need to shut down a...
                                               ^^^^^^^
  ✓ ...ble to connect to the client during the installation process:  So you'll need to shut down a...
  A_INSTALL: a/the + install

The word ‘install’ is not a noun.
  ✗ ...detected at install time will cause the install to abort. If you already have PostgreSQ...
                                               ^^^^^^^
  ✓ ...detected at install time will cause the installation to abort. If you already have PostgreSQ...
  A_INSTALL: a/the + install
```
2022-01-21 14:02:14 -08:00
Alya Abbott ca311e83c8 docs: Fix typos in GSoC guide. 2022-01-21 13:38:30 -08:00
Alya Abbott 19154f81c0
docs: Clarify purpose of zulip-announce. 2022-01-19 15:34:24 -08:00
Eeshan Garg fd303e3b1b docs: Update the release checklist for our PyPI packages. 2022-01-19 13:59:13 -08:00
Alex Vandiver 5f237cb34e puppet: Document that upgrades from Git require 3GB.
The step of rebuilding static assets using webpack requires more than
2G of RAM.
2022-01-19 12:36:44 -08:00
Abhijeet Prasad Bodas 12eb52221b docs: Rename `contributing/gsoc-ideas` -> `contributing/gsoc`.
This page contains a lot of other material related to GSoC than
just project ideas.
We would also want to add a redirect from the old URL to the new
one from the RTD admin page.
2022-01-19 11:35:33 -08:00
Alya Abbott 3659d95092 developer docs: Update GSoC documentation. 2022-01-18 21:16:18 -08:00
Rishabh-792 177931a23d doc: Fix typos in accessibility doc.
Hyphenated open source to open-source.

Capitalized aXe to Axe.
2022-01-11 15:41:08 -08:00
Rishabh-792 6cd8e088e9 doc: Fix typo in zulipbot-usage doc.
Fix a spelling mistake in the zulipbot doc.
2022-01-11 15:40:06 -08:00
Rishabh-792 1ec018d237 doc: Fix typos in code reviewing doc.
Made some spelling and grammatical changes.
2022-01-11 15:40:05 -08:00
Alex Vandiver d328d3dd4d puppet: Allow routing camo requests through an outgoing proxy.
Because Camo includes logic to deny access to private subnets, routing
its requests through Smokescreen is generally not necessary.  However,
it may be necessary if Zulip has configured a non-Smokescreen exit
proxy.

Default Camo to using the proxy only if it is not Smokescreen, with a
new `proxy.enable_for_camo` setting to override this behaviour if need
be.  Note that that setting is in `zulip.conf` on the host with Camo
installed -- not the Zulip frontend host, if they are different.

Fixes: #20550.
2022-01-07 12:08:10 -08:00
Alex Vandiver 2c5fc1827c puppet: Standardize what values are bools, and what true is.
For `no_serve_uploads`, `http_only`, which previously specified
"non-empty" to enable, this tightens what values are true.  For
`pgroonga` and `queue_workers_multiprocess`, this broadens the
possible values from `enabled`, and `true` respectively.
2022-01-07 12:08:10 -08:00
Anders Kaseorg 1696144df7 docs: Consistently hyphenate “self-host” and “self-service”.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-01-05 16:21:35 -08:00
Alex Vandiver 6218ed91c2 puppet: Use lazy-apps and uwsgi control sockets for rolling reloads.
Restarting the uwsgi processes by way of supervisor opens a window
during which nginx 502's all responses.  uwsgi has a configuration
called "chain reloading" which allows for rolling restart of the uwsgi
processes, such that only one process at once in unavailable; see
uwsgi documentation ([1]).

The tradeoff is that this requires that the uwsgi processes load the
libraries after forking, rather than before ("lazy apps"); in theory
this can lead to larger memory footprints, since they are not shared.
In practice, as Django defers much of the loading, this is not as much
of an issue.  In a very basic test of memory consumption (measured by
total memory - free - caches - buffers; 6 uwsgi workers), both
immediately after restarting Django, and after requesting `/` 60 times
with 6 concurrent requests:

                      |  Non-lazy  |  Lazy app  | Difference
    ------------------+------------+------------+-------------
    Fresh             |  2,827,216 |  2,870,480 |   +43,264
    After 60 requests |  3,332,284 |  3,409,608 |   +77,324
    ..................|............|............|.............
    Difference        |   +505,068 |   +539,128 |   +34,060

That is, "lazy app" loading increased the footprint pre-requests by
43MB, and after 60 requests grew the memory footprint by 539MB, as
opposed to non-lazy loading, which grew it by 505MB.  Using wsgi "lazy
app" loading does increase the memory footprint, but not by a large
percentage.

The other effect is that processes may be served by either old or new
code during the restart window.  This may cause transient failures
when new frontend code talks to old backend code.

Enable chain-reloading during graceful, puppetless restarts, but only
if enabled via a zulip.conf configuration flag.

Fixes #2559.

[1]: https://uwsgi-docs.readthedocs.io/en/latest/articles/TheArtOfGracefulReloading.html#chain-reloading-lazy-apps
2022-01-05 14:48:52 -08:00
BIKI DAS 42dd58cffe
docs: Fix a few typos in documentation. 2021-12-28 09:36:59 -08:00
BIKI DAS c1134a8bda
docs: Fix "should should" typo. 2021-12-28 09:19:04 -08:00
Anders Kaseorg 1d3520db12 webhooks: Remove space from UptimeRobot.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-12-22 14:05:17 -08:00
Anders Kaseorg 68c99511a2 webhooks: Fix TeamCity capitalization.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-12-22 14:05:17 -08:00
Anders Kaseorg 65868b09eb webhooks: Add missing space in Review Board.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-12-22 14:05:17 -08:00
Anders Kaseorg c02c053ec3 webhooks: Fix Mailchimp capitalization.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-12-22 14:05:17 -08:00
Anders Kaseorg cd8a01587b webhooks: Fix Jotform capitalization.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-12-22 14:05:17 -08:00
Anders Kaseorg 3ca2f8ca1e webhooks: Fix Clubhouse capitalization.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-12-22 14:05:17 -08:00
Anders Kaseorg 517ddbc9e6 setup-advanced: Remove misleading python3 symlink suggestion.
One should never have to manually symlink things in /usr/bin,
especially with -f.  That should be managed by the system package
manager.  Indeed, on CentOS 7 and 8, one can simply install the
python3 package and get a working /usr/bin/python3.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-12-16 17:30:04 -08:00
Alya Abbott aaf1258de2 developer docs: Tweak ToS for push notifications wording. 2021-12-14 14:13:34 -08:00
Tim Abbott ee77c6365a portico: Use /help/ style pages for displaying policies.
This replaces the TERMS_OF_SERVICE and PRIVACY_POLICY settings with
just a POLICIES_DIRECTORY setting, in order to support settings (like
Zulip Cloud) where there's more policies than just those two.

With minor changes by Eeshan Garg.
2021-12-10 17:56:12 -08:00
Alex Vandiver 01e8f752a8 puppet: Use certbot package timer, not our own cron job.
The certbot package installs its own systemd timer (and cron job,
which disabled itself if systemd is enabled) which updates
certificates.  This process races with the cron job which Zulip
installs -- the only difference being that Zulip respects the
`certbot.auto_renew` setting, and that it passes the deploy hook.
This means that occasionally nginx would not be reloaded, when the
systemd timer caught the expiration first.

Remove the custom cron job and `certbot-maybe-renew` script, and
reconfigure certbot to always reload nginx after deploying, using
certbot directory hooks.

Since `certbot.auto_renew` can't have an effect, remove the setting.
In turn, this removes the need for `--no-zulip-conf` to
`setup-certbot`.  `--deploy-hook` is similarly removed, as running
deploy hooks to restart nginx is now the default; pass
`--no-directory-hooks` in standalone mode to not attempt to reload
nginx.  The other property of `--deploy-hook`, of skipping symlinking
into place, is given its own flog.
2021-12-09 13:47:33 -08:00
Eeshan Garg 3bab91079f external links: Migrate the rest of /developer-community links.
We recently changed /developer-community to /development-community.
Now that this change is in production, we can also migrate the
external links in our ReadTheDocs documentation.
2021-12-09 12:14:26 -08:00
Alex Vandiver cb2d0ff32b postgresql: Support replication on PostgreSQL >= 11, document.
PostgreSQL 11 and below used a configuration file names
`recovery.conf` to manage replicas and standbys; support for this was
removed in PostgreSQL 12[1], and the configuration parameters were
moved into the main `postgresql.conf`.

Add `zulip.conf` settings for the primary server hostname and
replication username, so that the complete `postgresql.conf`
configuration on PostgreSQL 14 can continue to be managed, even when
replication is enabled.  For consistency, also begin writing out the
`recovery.conf` for PostgreSQL 11 and below.

In PostgreSQL 12 configuration and later, the `wal_level =
hot_standby` setting is removed, as `hot_standby` is equivalent to
`replica`, which is the default value[2].  Similarly, the
`hot_standby = on` setting is also the default[3].

Documentation is added for these features, and the commentary on the
"Export and Import" page referencing files under `puppet/zulip_ops/`
is removed, as those files no longer have any replication-specific
configuration.

[1]: https://www.postgresql.org/docs/current/recovery-config.html
[2]: https://www.postgresql.org/docs/12/runtime-config-wal.html#GUC-WAL-LEVEL
[3]: https://www.postgresql.org/docs/12/runtime-config-replication.html#GUC-HOT-STANDBY
2021-12-03 16:32:41 -08:00
Emilio López baea14ee57 docs: Clarify use of `loadbalancer.ips` when using a reverse proxy.
When Zulip is run behind one or more reverse proxies, you must
configure `loadbalancer.ips` so that Zulip respects the client IP
addresses found in the `X-Forwarded-For` header. This is not
immediately clear from the documentation, so this commit makes it more
clear and augments the existing examples to showcase this need.

Fixes: #19073
2021-12-03 13:59:31 -08:00
Alex Vandiver ab8be84b36 docs: Secret reading is done using RawConfigParser, not ConfigParser.
ConfigParser makes `%` signs require escaping, which is why it is not
used in Zulip, particularly for secrets.
2021-12-02 15:25:04 -08:00
Alex Vandiver 54d037f24a version: Update version and changelog after 4.8 release. 2021-12-01 23:42:11 +00:00
AEsping f6c4f17900 dev docs: Update Jinja translation tag link.
Updates the link to Jinja i18n extension for auto-translation.
2021-11-30 14:36:29 -08:00
AEsping 828313b54a dev docs: Update Jinja translation tag link.
Updates the link to Jinja i18n extension for auto-translation.
2021-11-30 14:36:29 -08:00
AEsping 704c9609ee dev docs: Update Tig link.
Updates the link to the Tig git visualizer.
2021-11-30 14:36:29 -08:00
AEsping 11f2575c31 dev docs: Update "Solo" link.
Fixes the link to "El adveribo <<solo>> y los pronombres
demonstrativos, sin tilde."
2021-11-30 14:36:29 -08:00
AEsping 510b8867a6 dev docs: Update Neil Green link in the reading list.
Fixes the link to the Neil Green presentation on TypeScript
vs Coffee Script vs ES6.

This is a change from slides to a video becasue the slides are
no longer available.
2021-11-30 14:36:29 -08:00
AEsping 55f9178506 dev docs: Update Black link.
Updates the link to the editior integration for Black.
2021-11-30 14:36:29 -08:00
AEsping 5410009a88 prod docs: Update BBB configuration link.
Updates the Big Blue Button customization link for
extracting shared secrets.
2021-11-30 14:36:29 -08:00
Mateusz Mandera 8c1a6f4bba docs: Suggest updating settings.py in OIDC instructions.
OIDC config features a get_secret call (so it requires adding an import)
as well as having a bunch of its instructions in the form of comments on
the various keys of the config dict - thus users should really update
settings.py to fetch all of that.
2021-11-29 15:52:52 -08:00
Alex Vandiver 0ae375e0f9 ci: Test upgrades from the latest minor release. 2021-11-25 08:00:34 -08:00
AEsping 6ad1c5c8ed docs:: Update GSoC application tips.
- Add missing link for GitHub.
- Fix broken links to Matt Ringel's blog post.
- Add link to Julia Evans blog post.
- Add section heading for "Questions Are Important."
- Rearrange some content to fit with new section heading.

With additional tweaks from tabbott:
* Avoid linking to chat.zulip.org not via our documentation.
* Avoid the CZO abbreviation.
2021-11-23 16:05:33 -08:00
Alex Vandiver b982222e03 camo: Replace with go-camo implementation.
The upstream of the `camo` repository[1] has been unmaintained for
several years, and is now archived by the owner.  Additionally, it has
a number of limitations:
 - It is installed as a sysinit service, which does not run under
   Docker
 - It does not prevent access to internal IPs, like 127.0.0.1
 - It does not respect standard `HTTP_proxy` environment variables,
   making it unable to use Smokescreen to prevent the prior flaw
 - It occasionally just crashes, and thus must have a cron job to
   restart it.

Swap camo out for the drop-in replacement go-camo[2], which has the
same external API, requiring not changes to Django code, but is more
maintained.  Additionally, it resolves all of the above complaints.

go-camo is not configured to use Smokescreen as a proxy, because its
own private-IP filtering prevents using a proxy which lies within that
IP space.  It is also unclear if the addition of Smokescreen would
provide any additional protection over the existing IP address
restrictions in go-camo.

go-camo has a subset of the security headers that our nginx reverse
proxy sets, and which camo set; provide the missing headers with `-H`
to ensure that go-camo, if exposed from behind some other non-nginx
load-balancer, still provides the necessary security headers.

Fixes #18351 by moving to supervisor.
Fixes zulip/docker-zulip#298 also by moving to supervisor.

[1] https://github.com/atmos/camo
[2] https://github.com/cactus/go-camo
2021-11-19 15:58:26 -08:00
Alex Vandiver c33562f0a8 puppet: Default to installing smokescreen on application frontends.
This is an additional security hardening step, to make Zulip default
to preventing SSRF attacks.  The overhead of running Smokescreen is
minimal, and there is no reason to force deployments to take
additional steps in order to secure themselves against SSRF attacks.

Deployments which already have a different external proxy configured
will not gain a local Smokescreen installation, and running without
Smokescreen is supported by explicitly unsetting the `host` or `port`
values in `/etc/zulip/zulip.conf`.
2021-11-19 15:29:28 -08:00
Sahil Batra 5024ba98cd misc: Rename "dark mode" to "dark theme".
Fixes part of #20228.
2021-11-18 16:57:11 -08:00
Ankur c578f186b6
docs: Change chat.zulip.org links to zulip.com/developer-community.
This helps increase the probability that folks read the guidelines for how the
chat.zulip.org community works and what streams to use before arriving there.

Fixes #19827.
2021-11-17 22:25:55 -08:00
Tim Abbott c64eebcc9e docs: Mention server_name in existing server nginx docs.
As discussed in the issue, it'd be a helpful hint for folks doing this
to mention this detail.

Fixes #16705.
2021-11-17 16:53:11 -08:00
Tim Abbott cc11743047 docs: Advertise docker-zulip on rqeuirements page.
We also remove coverage of 64-bit, since it's no longer a relevant
detail in 2021.

Fixes #18534.
2021-11-17 16:48:51 -08:00
Aman Agrawal fc42568248 docs: Separate recommend and vagrant setup sections for dev setup.
This will possibly direct more users towards WSL setup on Windows.
2021-11-15 10:58:08 -08:00
Alex Vandiver 544e8c569e install: Switch default to PostgreSQL 14. 2021-11-08 18:21:46 -08:00
YashRE42 40444cf415 docs: Update all links to Django docs to point to version /3.2/.
Previously, our docs had links to various versions of the Django docs,
eg https://docs.djangoproject.com/en/1.10/topics/migrations/ and
https://docs.djangoproject.com/en/2.0/ref/signals/#post-save, opening
a link to a doc with an outdated Django version would show a warning
"This document is for an insecure version of Django that is no longer
supported. Please upgrade to a newer release!".

This commit uses a search with the regex
"docs.djangoproject.com/en/([0-9].[0-9]*)/" and replaces all matches
inside the /docs/ folder with "docs.djangoproject.com/en/3.2/".

All the new links in this commit have been generated by the above
replace and each link has then been manually checked to ensure that
(1) the page still exists and has not been moved to a new location
(and it has been found that no page has been moved like this), (2)
that the anchor that we're linking to has not been changed (and it has
been found that this happened once, for https://docs.djangoproject.com
/en/1.8/ref/django-admin/#runserver-port-or-address-port, where
/#runserver-port-or-address-port was changed to /#runserver).
2021-11-08 18:06:16 -08:00
YashRE42 87dd2a15c2 docs: Re-add accidentally deleted link definitions in advanced setup.
In commit f6c78a35a4 we accidentally
deleted these link definitions, probably thinking that the end of the
markdown file would be the same as the end of the rendered doc. This
broke the links `[cloning your fork of the Zulip
repository][zulip-rtd-git-cloning]` and `[connecting the Zulip
upstream repository][zulip-rtd-git-connect]`.

This commit fixes things by adding back the definitions.
2021-11-08 18:06:16 -08:00
Vishnu KS 6c06858e02 billing: Migrate to Stripe hosted checkout page. 2021-11-05 17:23:10 -07:00
Vishnu KS 1a1b9b28ff corporate: Store the Stripe API version. 2021-11-05 17:23:10 -07:00
Anders Kaseorg 6fa86385aa docs: Enable sphinx_rtd_theme as an extension.
This has no effect at present, but it’s documented as necessary to
enable localization of theme strings in translated output, so maybe
it’ll be relevant some day.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-11-04 11:55:15 -07:00
Anders Kaseorg 03a7d0c053 docs: Remove html_theme_path override.
This block has been obsolete since at least sphinx-rtd-theme 0.2.5.
Removing it fixes the heading permalink icon in a local build to be
consistent with the one shown on Read the Docs, and has no other
effect.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-11-04 11:55:15 -07:00
Anders Kaseorg 8d4568140c docs: Enable collapse_navigation for local builds.
This makes local builds significantly faster, while leaving the fancy
navigation enabled on Read the Docs where it’s important.

https://sphinx-rtd-theme.readthedocs.io/en/stable/configuring.html#confval-collapse_navigation

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-11-04 11:55:15 -07:00
Anders Kaseorg 177dde15eb docs: Clean Sphinx configuration file.
Delete all the boilerplate comments and unused options generated by
the ancient version of Sphinx that originally generated this file,
leaving a file that one can realistically read.  Leave some links for
those who want to read about all the options that exist.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-11-04 11:55:15 -07:00
Lauryn Menard dd5cad549c documentation: Expand documentation processes in new feature tutorial.
Expands the developer tutorial 'Writing a new application feature' to
include more detail about the documentation aspects of adding a new
feature. Adds references to specific files that will be impacted and
highlights API changes as well as writing `/help` articles.
2021-11-02 14:03:34 -07:00
AEsping 6963876e22 docs: Update instructions for Windows 10 installation.
This commit includes the following changes.
- Adds the definition of the WSL acronym.
- Adds information for changing BIOS settings
in order to enable machine virtulization.
- Fixes a broken link to Microsoft WSL installation instructions.
- Adds a reminder to create a new SSH key before connecting to
GitHub.
- Removes the step to install Ubuntu.  This step is now
included in the standard installation.
- Reminds the user to launch Ubuntu as and administrator.
- Switches the text editor in the example to nano from vim.
Nano is included with the wsl installation, and is easier for
most people to use than vim.
- Adds a separate step to fork the Zulip/Zulip repository.
- Adds the bash command to open VS Code and
reminds the user to install the relevant extensions.

With various formatting tweaks by tabbott.
2021-11-02 13:47:11 -07:00
Tim Abbott 61c0825036 docs: Extend Certbot troubleshooting documentation.
This should help folks who have problems with Certbot renewal; we had
a couple reported this week which I think were both caused by firewall
issues.
2021-11-01 18:11:31 -07:00
Mateusz Mandera 3d731de3f1 docs: Add disabling of Force POST Binding to Keyclock SLO instructions.
This needs to be disabled, because python3-saml only supports the
Redirect binding. This step was forgotten in the original writing of
this doc.
2021-11-01 11:02:51 -07:00
Mateusz Mandera 4105ccdb17 saml: Implement IdP-initated logout for Keycloak.
Fixes #13948.
2021-10-27 13:13:55 -07:00
Jonny Tran ddf4053a73 composebox_typeahead: Remove redundant options from the `/` typeahead.
Removes the `/day` and `/night` options from the typeahead menu while
still allowing the commands to be used. Typing `/day` and `/night`
will now suggest `/light` and `/dark`, respectively. Also changes the
`Dark mode` and `Light mode` popups that appear after using the
corresponding command.

Fixes #18318.
2021-10-21 10:50:27 -07:00
YashRE42 f6c78a35a4 docs: Remove AWS Cloud9 setup instructions.
Due to the fact that it's not possible to run the development
environment on a t2.micro (1 GiB RAM + 1 vCPU), which is what is
available from the free tier, the fact that signing up require a
credit/debit card and can take up to 24 hours, and that it is quite
easy to unintentionally exceed the free tier resources when expanding
or upgrading, it is no longer feasible to develop on cloud9. As such,
we should not recommend it in out setup docs.
2021-10-18 10:38:00 -07:00
YashRE42 c8353a1820 minor: Add missing word to "sending messages" doc. 2021-10-18 10:38:00 -07:00
YashRE42 94167c6db9 docs: Update wsl install guide link.
The previous link "/wsl/wsl2-install" leads to a 404 page which
recommends "/wsl/install". This commit updates the link to
"/wsl/install".

The previous link has been giving a redirect since at least May 23,
2020.
2021-10-18 10:38:00 -07:00
Eeshan Garg d312432ea8 docs: Update links to zulip-announce mailing list.
The links we have now redirect to "My groups" and not to our
Google group. Also, the RSS feed is no longer supported by Google,
so we should no longer link to it.

Fixes #19560.
2021-10-14 16:51:36 -07:00
Tim Abbott a87bbe22be docs: Document how to test the polling email gateway configuration.
We don't recommend this configuration, so it's rarely used, but it's
worth a few minutes to have nice documentation.

Fixes #19565.
2021-10-14 14:50:52 -07:00
Mateusz Mandera 16321327f9 docs: Improve documentation on syncing UserProfile bool flags via ldap.
It's better to explicitly list the possibilities. Also, the
recommendation regarding is_active should be changed to a strict
"Don't", as Subscription.is_user_active is a denormalized field and
flipping a user's is_active will cause inconsistent state by leaving
Subscriptions unupdated. Given that similar things can be introduced in
the future for any other flag not officially supported by having a
setter, the recommendation should "Don't" in general.
2021-10-06 17:38:56 -07:00
Tim Abbott f01ee5f17a lint: Fix line wrapping of new schema migrations doc. 2021-10-04 22:16:55 -07:00
Alex Vandiver 36f92d6843 docs: Document what to do when releases need migrations. 2021-10-04 22:10:24 -07:00
Alex Vandiver 3922b171a8 version: Update version and changelog after 4.7 release. 2021-10-04 21:52:52 +00:00
Tim Abbott 67fdbbe5fd docs: Inline some upgrade instructions.
It feels like the "Same as" content was unnecessarily requiring the
user to bounce around in these cases.

(I've left the "Same as" text for the Ubuntu ones, where it's two
steps in a row to follow).
2021-10-01 11:08:14 -07:00
Gaurav Pandey 77d1c4361b docs: Document upgrade steps from buster to bullseye.
Fixes #17863.
2021-10-01 11:08:14 -07:00
Anders Kaseorg 2f993f1a79 install-node: Stop using NVM.
NVM doesn’t check hashes or signatures and really just adds
complexity we don’t need.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-09-24 06:58:32 -07:00
Tim Abbott 10c47b5d6c version: Update version and changelog after 4.6 release. 2021-09-23 16:19:57 -07:00
Anders Kaseorg 6367384510 docs: Recommend Docker for all macOS users.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-09-17 21:18:38 -07:00
Tim Abbott ea6a40661c migrations: Add progress output to confirmation backfill.
This makes it a lot easier to estimate how long this migration will
take to run. Additionally, document this migration in the upgrade notes.
2021-09-17 11:23:32 -07:00
Mateusz Mandera 8ad7520180 ldap: Support alternative attrs to userAccountControl.
Fixes #17456.

The main tricky part has to do with what values the attribute should
have. LDAP defines a Boolean as
Boolean = "TRUE" / "FALSE"
so ideally we'd always see exactly those values. However,
although the issue is now marked as resolved, the discussion in
https://pagure.io/freeipa/issue/1259 shows how this may not always be
respected - meaning it makes sense for us to be more liberal in
interpreting these values.
2021-09-16 13:29:13 -07:00
Mateusz Mandera b32450f98e docs: Remove mentions of some ldap features being added in 2.0.
2.0 is old enough that explicitly mentioning when these features were
implemented isn't particularly useful and adds clutter.
2021-09-16 13:29:13 -07:00
Eeshan Garg 3b1cb0b25a docs: Indicate latest Zulip version in installation and upgrade docs.
With copy-editing from tabbott, and also a migration to use
LATEST_RELEASE_VERSION, which will be correct even on the /latest/
paths.

Fixes #19695.
2021-09-10 14:50:48 -07:00
Tim Abbott 908e1e6d19 docs: Run prettier on authentication-methods.md.
The new sections of this file were rebased past the addition of
running prettier on these files.
2021-09-10 13:23:11 -07:00
Philipp Seßner a8797b4d9f help: Document how to configure Keycloak as the SAML IdP. 2021-09-10 13:04:04 -07:00
Anders Kaseorg 5360bd76ac help: Add newline after admonition header for Prettier compatibility.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-09-10 10:39:25 -07:00
Gaurav Pandey 502697d239 docs: Add documentation for bullseye support.
The support for bullseye was added in #17951
but it was not documented as bullseye was
frozen and did not have proper configuration
files, hence wasn't documented.

Since now bullseye is released as a stable
version, it's support can be documented.
2021-09-09 11:05:16 -07:00
Anders Kaseorg 1fc801a43f docs: Add missing space in “feature set”.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-09-08 15:31:28 -07:00
Anders Kaseorg 85fec56697 docs: Update links for zulip-mobile branch rename.
GitHub redirects these, but we should use the canonical URLs.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-09-08 15:30:37 -07:00
Anders Kaseorg a4dbc1edd4 docs: Format Markdown with Prettier.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-09-08 12:06:24 -07:00
Anders Kaseorg 35c1c8d41b docs: Apply sentence single-spacing from Prettier.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-09-08 12:06:24 -07:00
Anders Kaseorg 915884bff7 docs: Apply bullet style changes from Prettier.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-09-08 12:06:24 -07:00
Anders Kaseorg 6145fdf678 docs: Rewrap to avoid line breaks in inline code spans.
This works around https://github.com/prettier/prettier/issues/11372.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-09-08 12:06:24 -07:00
Anders Kaseorg a6e01b35fc docs: Fix `pip compile` typo.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-09-08 12:06:24 -07:00
Anders Kaseorg f4d2d199e2 docs: Display `main` branch name as inline code.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-09-06 12:56:35 -07:00
Anders Kaseorg 646c04eff2 Rename default branch to ‘main’.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-09-06 12:56:35 -07:00
Tim Abbott 1c3517a5de docs: Document ReadTheDocs preview builds. 2021-09-06 12:40:36 -07:00
Anders Kaseorg 1ce12191aa docs: Update links for other repository branch renames.
GitHub redirects these, but we should use the canonical URLs.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-08-31 15:28:01 -07:00
Anders Kaseorg 0dcb84a2e8 docs: Follow Sphinx 4 rename of master_doc to root_doc.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-08-31 14:24:16 -07:00
Alex Vandiver 7d7d727865 tools: Switch to download.zulip.com from www.zulip.org. 2021-08-26 17:21:31 -07:00
Alex Vandiver 710a92f48c tools: Add a tool to upload release tarballs to S3. 2021-08-26 17:21:31 -07:00
Alex Vandiver faf71eea41 upgrade-postgresql: Do not remove other supervisor configs.
We previously used `zulip-puppet-apply` with a custom config file,
with an updated PostgreSQL version but more limited set of
`puppet_classes`, to pre-create the basic settings for the new cluster
before running `pg_upgradecluster`.

Unfortunately, the supervisor config uses `purge => true` to remove
all SUPERVISOR configuration files that are not included in the puppet
configuration; this leads to it removing all other supervisor
processes during the upgrade, only to add them back and start them
during the second `zulip-puppet-apply`.

It also leads to `process-fts-updates` not being started after the
upgrade completes; this is the one supervisor config file which was
not removed and re-added, and thus the one that is not re-started due
to having been re-added.  This was not detected in CI because CI added
a `start-server` command which was not in the upgrade documentation.

Set a custom facter fact that prevents the `purge` behaviour of the
supervisor configuration.  We want to preserve that behaviour in
general, and using `zulip-puppet-apply` continues to be the best way
to pre-set-up the PostgreSQL configuration -- but we wish to avoid
that behaviour when we know we are applying a subset of the puppet
classes.

Since supervisor configs are no longer removed and re-added, this
requires an explicit start-server step in the instructions after the
upgrades complete.  This brings the documentation into alignment with
what CI is testing.
2021-08-24 19:00:58 -07:00
Anders Kaseorg fdb7ec8c9e docs: Avoid [GitHub] as an internal Markdown link reference name.
To avoid confusing the linter later when Prettier lowercases these.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-08-20 23:14:37 -07:00
Anders Kaseorg 0e4a30daad reading-list: Inline links.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-08-20 23:14:37 -07:00
Anders Kaseorg 32da7e3eb2 docs: Adjust colon fence for Prettier compatibility.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-08-20 23:14:37 -07:00
Anders Kaseorg 7b3d4ff1de docs: Escape asterisks for Prettier compatibility.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-08-20 23:14:37 -07:00
Anders Kaseorg 0a3e022376 docs: Adjust list item indentation for Prettier compatibility.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-08-20 23:14:37 -07:00
Anders Kaseorg 4bfffc9f74 docs: Fix list item indentation mistakes.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-08-20 23:14:37 -07:00
Anders Kaseorg 962f14995e docs: Fix “sinternet” typo.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-08-20 23:14:37 -07:00
Anders Kaseorg fe3db63381 docs: Add missing blockquote.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-08-20 23:14:37 -07:00
Anders Kaseorg b53e676860 docs: Convert reST index pages to Markdown.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-08-20 06:22:42 -07:00
Anders Kaseorg b29b6f6526 docs: Add syntax highlighting languages to code blocks.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-08-20 06:14:47 -07:00
Anders Kaseorg 672358c5da docs: Migrate from recommonmark to MyST-Parser.
Recommonmark is no longer maintained, and MyST-Parser is much more
complete.

https://myst-parser.readthedocs.io/

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-08-19 00:53:32 -07:00
Anders Kaseorg 8f2827a65d docs: Remove trailing newlines from code blocks.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-08-19 00:53:32 -07:00
Anders Kaseorg cb61ea69c6 docs: Fix misaligned Markdown source indentation.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-08-19 00:53:32 -07:00
Anders Kaseorg 25c6d5c99c docs: Fix code span syntax in embedded reST block.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-08-19 00:53:32 -07:00
Anders Kaseorg c3646ec67f docs: Move authentication-methods#ldap anchor to appropriate heading.
Commit 30eaed0378 (#15001) incorrectly
inserted a different section between the anchor and the heading.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-08-19 00:53:32 -07:00
Mateusz Mandera fdbde59b07 rate_limit: Add management command to reset auth rate limit.
The auth attempt rate limit is quite low (on purpose), so this can be a
common scenario where a user asks their admin to reset the limit instead
of waiting. We should provide a tool for administrators to handle such
requests without fiddling around with code in manage.py shell.
2021-08-19 00:45:17 -07:00
Alex Vandiver ec354ee013 changelog: Add 2.1.8 release notes. 2021-08-12 01:06:15 +00:00
Alex Vandiver e94b6afb00 nagios: Remove broken check_email_deliverer_* checks and related code.
These checks suffer from a couple notable problems:
 - They are only enabled on staging hosts -- where they should never
   be run.  Since ef6d0ec5ca, these supervisor processes are only
   run on one host, and never on the staging host.
 - They run as the `nagios` user, which does not have appropriate
   permissions, and thus the checks always fail.  Specifically,
   `nagios` does not have permissions to run `supervisorctl`, since
   the socket is owned by the `zulip` user, and mode 0700; and the
   `nagios` user does not have permission to access Zulip secrets to
   run `./manage.py print_email_delivery_backlog`.

Rather than rewrite these checks to run on a cron as zulip, and check
those file contents as the nagios user, drop these checks -- they can
be rewritten at a later point, or replaced with Prometheus alerting,
and currently serve only to cause always-failing Nagios checks, which
normalizes alert failures.

Leave the files installed if they currently exist, rather than
cluttering puppet with `ensure => absent`; they do no harm if they are
left installed.
2021-08-03 16:07:13 -07:00
Tim Abbott f84703bc48 test-backend: Default to running all tests.
This switches the default of the previous --nonfatal-errors option,
and also moves to use the `-x | --stop` names used by the nose test
framework.
2021-07-30 16:18:42 -07:00
Tim Abbott 665f479f4d docs: Update changelog since 4.0 release. 2021-07-29 14:54:17 -07:00
Tim Abbott e4667c2af6 docs: Add developer documentation for Python support. 2021-07-28 11:23:27 -07:00
sahil839 9ef8727a4a docs: Fix new_feature_tutorial to replace add_emoji_by_admins_only.
This commit replaces add_emoji_by_admins_only with
add_custom_emoji_policy in new_feature_tutorial.md
as the old boolean setting is replaced by a new
integer setting.
2021-07-27 16:41:23 -07:00
Suyash Vardhan Mathur ad8d9f2133 docs: Document the new APIDocs template system.
Recently, the need for individual Markdown templates for
every endpoint's OpenAPI page was removed, as they are now
auto-generated from OpenAPI data. Further, as a part of this
migration, several new fields and Markdown extensions were added.
This commit updates the documentation to reflect the changes that
have occured as a result of the migration.

With various edits by tabbott to clarify or simplify the documentation.
2021-07-26 18:38:57 -07:00
PIG208 c7210bfc75 docs: Update client.md for the request notes change.
`request.client` is no longer valid since the ZulipRequestNotes change.
This update the documentation to reflect that. And it also makes it
recommend `check_send_webhook_message` in favor of
`check_send_stream_message`.
2021-07-26 08:40:53 -07:00
Tim Abbott 454a900487 docs: Replace chat-zulip-org page with /developer-community/. 2021-07-25 22:17:57 -07:00
Tim Abbott 361112a08f version: Update version and changelog after 4.5 release. 2021-07-25 16:05:22 -07:00
Mateusz Mandera 388932bcc4 docs: Move docs on SOCIAL_AUTH_OIDC_FULL_NAME_VALIDATED to ReadTheDocs. 2021-07-24 15:16:13 -07:00
Mateusz Mandera 9bf84a35ef docs: Mention auto_signup OIDC option in ReadTheDocs documentation. 2021-07-24 15:13:47 -07:00
Mateusz Mandera 0d6b1cd08b docs: Tweak phrasing about OIDC supporting only a single provider.
Strictly speaking, this sentence is talking about the IdP configuration,
while the backend is just GenericOpenIdConnectBackend, so the new
phrasing is more correct.
2021-07-24 15:12:06 -07:00
Alex Vandiver 34833aa795 version: Update version and changelog after 4.4 release. 2021-07-22 17:12:36 -07:00
Alex Vandiver d78723b6e8 ci: Update outdated comments, documentation and gitignore.
Use of `Dockerfile.template` and generated `tools/ci/images/` was
removed in 16067bc4fc.
2021-07-22 14:09:01 -07:00
manavdesai27 572cef9a0f provision: Add support for Fedora 34. 2021-07-20 12:10:41 -07:00
Alex Vandiver 91282ab490 reindex-textual-data: Provide a tool to reindex all text indices.
The script is added to upgrade steps for 20.04 and Buster because
those are the upgrades that cross glibc 2.28, which is most
problematic.  It will also be called out in the upgrade notes, to
catch those that have already done that upgrade.
2021-07-19 16:34:23 -07:00
Alex Vandiver 528e5adaab smokescreen: Default to only listening on 127.0.0.1.
This prevents Smokescreen from acting as an open proxy.

Fixes #19214.
2021-07-14 15:40:26 -07:00
Anders Kaseorg 47897c76a2 scripts: Use curl -f (--fail).
This makes curl exit with nonzero status on HTTP 4xx/5xx errors.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-07-13 16:47:49 -07:00
Anders Kaseorg 77169408f4 docs: Remove do-release-upgrade -d option.
It has not been necessary since Ubuntu 20.04.1 was released in August.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-07-13 11:02:44 -07:00
Suyash Vardhan Mathur 981e4f8946
openapi: Render all responses of an operation.
Previously, one needed to specifying all the HTTP status
codes that we want to render along with the operation,
but the primary use case just needs the responses of
all the status codes, and not just one.

This commit modifies the Markdown extension to render
all the responses of all status codes of a specified
operation in a loop.
2021-07-13 08:33:43 -07:00
Anders Kaseorg 93f62b999e nagios: Replace check_website_response with standard check_http plugin.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-07-09 16:47:03 -07:00
Tim Abbott 1905bb8f1a docs: Document forked change password command. 2021-07-09 12:37:55 -07:00
Mateusz Mandera 85cbdc8904 rate_limit: Add rate limiting of ZulipRemoteServer. 2021-07-08 15:55:02 -07:00