Commit Graph

649 Commits

Author SHA1 Message Date
Alex Vandiver cb2d0ff32b postgresql: Support replication on PostgreSQL >= 11, document.
PostgreSQL 11 and below used a configuration file names
`recovery.conf` to manage replicas and standbys; support for this was
removed in PostgreSQL 12[1], and the configuration parameters were
moved into the main `postgresql.conf`.

Add `zulip.conf` settings for the primary server hostname and
replication username, so that the complete `postgresql.conf`
configuration on PostgreSQL 14 can continue to be managed, even when
replication is enabled.  For consistency, also begin writing out the
`recovery.conf` for PostgreSQL 11 and below.

In PostgreSQL 12 configuration and later, the `wal_level =
hot_standby` setting is removed, as `hot_standby` is equivalent to
`replica`, which is the default value[2].  Similarly, the
`hot_standby = on` setting is also the default[3].

Documentation is added for these features, and the commentary on the
"Export and Import" page referencing files under `puppet/zulip_ops/`
is removed, as those files no longer have any replication-specific
configuration.

[1]: https://www.postgresql.org/docs/current/recovery-config.html
[2]: https://www.postgresql.org/docs/12/runtime-config-wal.html#GUC-WAL-LEVEL
[3]: https://www.postgresql.org/docs/12/runtime-config-replication.html#GUC-HOT-STANDBY
2021-12-03 16:32:41 -08:00
Emilio López baea14ee57 docs: Clarify use of `loadbalancer.ips` when using a reverse proxy.
When Zulip is run behind one or more reverse proxies, you must
configure `loadbalancer.ips` so that Zulip respects the client IP
addresses found in the `X-Forwarded-For` header. This is not
immediately clear from the documentation, so this commit makes it more
clear and augments the existing examples to showcase this need.

Fixes: #19073
2021-12-03 13:59:31 -08:00
AEsping 5410009a88 prod docs: Update BBB configuration link.
Updates the Big Blue Button customization link for
extracting shared secrets.
2021-11-30 14:36:29 -08:00
Mateusz Mandera 8c1a6f4bba docs: Suggest updating settings.py in OIDC instructions.
OIDC config features a get_secret call (so it requires adding an import)
as well as having a bunch of its instructions in the form of comments on
the various keys of the config dict - thus users should really update
settings.py to fetch all of that.
2021-11-29 15:52:52 -08:00
Alex Vandiver b982222e03 camo: Replace with go-camo implementation.
The upstream of the `camo` repository[1] has been unmaintained for
several years, and is now archived by the owner.  Additionally, it has
a number of limitations:
 - It is installed as a sysinit service, which does not run under
   Docker
 - It does not prevent access to internal IPs, like 127.0.0.1
 - It does not respect standard `HTTP_proxy` environment variables,
   making it unable to use Smokescreen to prevent the prior flaw
 - It occasionally just crashes, and thus must have a cron job to
   restart it.

Swap camo out for the drop-in replacement go-camo[2], which has the
same external API, requiring not changes to Django code, but is more
maintained.  Additionally, it resolves all of the above complaints.

go-camo is not configured to use Smokescreen as a proxy, because its
own private-IP filtering prevents using a proxy which lies within that
IP space.  It is also unclear if the addition of Smokescreen would
provide any additional protection over the existing IP address
restrictions in go-camo.

go-camo has a subset of the security headers that our nginx reverse
proxy sets, and which camo set; provide the missing headers with `-H`
to ensure that go-camo, if exposed from behind some other non-nginx
load-balancer, still provides the necessary security headers.

Fixes #18351 by moving to supervisor.
Fixes zulip/docker-zulip#298 also by moving to supervisor.

[1] https://github.com/atmos/camo
[2] https://github.com/cactus/go-camo
2021-11-19 15:58:26 -08:00
Alex Vandiver c33562f0a8 puppet: Default to installing smokescreen on application frontends.
This is an additional security hardening step, to make Zulip default
to preventing SSRF attacks.  The overhead of running Smokescreen is
minimal, and there is no reason to force deployments to take
additional steps in order to secure themselves against SSRF attacks.

Deployments which already have a different external proxy configured
will not gain a local Smokescreen installation, and running without
Smokescreen is supported by explicitly unsetting the `host` or `port`
values in `/etc/zulip/zulip.conf`.
2021-11-19 15:29:28 -08:00
Ankur c578f186b6
docs: Change chat.zulip.org links to zulip.com/developer-community.
This helps increase the probability that folks read the guidelines for how the
chat.zulip.org community works and what streams to use before arriving there.

Fixes #19827.
2021-11-17 22:25:55 -08:00
Tim Abbott c64eebcc9e docs: Mention server_name in existing server nginx docs.
As discussed in the issue, it'd be a helpful hint for folks doing this
to mention this detail.

Fixes #16705.
2021-11-17 16:53:11 -08:00
Tim Abbott cc11743047 docs: Advertise docker-zulip on rqeuirements page.
We also remove coverage of 64-bit, since it's no longer a relevant
detail in 2021.

Fixes #18534.
2021-11-17 16:48:51 -08:00
Alex Vandiver 544e8c569e install: Switch default to PostgreSQL 14. 2021-11-08 18:21:46 -08:00
YashRE42 40444cf415 docs: Update all links to Django docs to point to version /3.2/.
Previously, our docs had links to various versions of the Django docs,
eg https://docs.djangoproject.com/en/1.10/topics/migrations/ and
https://docs.djangoproject.com/en/2.0/ref/signals/#post-save, opening
a link to a doc with an outdated Django version would show a warning
"This document is for an insecure version of Django that is no longer
supported. Please upgrade to a newer release!".

This commit uses a search with the regex
"docs.djangoproject.com/en/([0-9].[0-9]*)/" and replaces all matches
inside the /docs/ folder with "docs.djangoproject.com/en/3.2/".

All the new links in this commit have been generated by the above
replace and each link has then been manually checked to ensure that
(1) the page still exists and has not been moved to a new location
(and it has been found that no page has been moved like this), (2)
that the anchor that we're linking to has not been changed (and it has
been found that this happened once, for https://docs.djangoproject.com
/en/1.8/ref/django-admin/#runserver-port-or-address-port, where
/#runserver-port-or-address-port was changed to /#runserver).
2021-11-08 18:06:16 -08:00
Tim Abbott 61c0825036 docs: Extend Certbot troubleshooting documentation.
This should help folks who have problems with Certbot renewal; we had
a couple reported this week which I think were both caused by firewall
issues.
2021-11-01 18:11:31 -07:00
Mateusz Mandera 3d731de3f1 docs: Add disabling of Force POST Binding to Keyclock SLO instructions.
This needs to be disabled, because python3-saml only supports the
Redirect binding. This step was forgotten in the original writing of
this doc.
2021-11-01 11:02:51 -07:00
Mateusz Mandera 4105ccdb17 saml: Implement IdP-initated logout for Keycloak.
Fixes #13948.
2021-10-27 13:13:55 -07:00
Eeshan Garg d312432ea8 docs: Update links to zulip-announce mailing list.
The links we have now redirect to "My groups" and not to our
Google group. Also, the RSS feed is no longer supported by Google,
so we should no longer link to it.

Fixes #19560.
2021-10-14 16:51:36 -07:00
Tim Abbott a87bbe22be docs: Document how to test the polling email gateway configuration.
We don't recommend this configuration, so it's rarely used, but it's
worth a few minutes to have nice documentation.

Fixes #19565.
2021-10-14 14:50:52 -07:00
Mateusz Mandera 16321327f9 docs: Improve documentation on syncing UserProfile bool flags via ldap.
It's better to explicitly list the possibilities. Also, the
recommendation regarding is_active should be changed to a strict
"Don't", as Subscription.is_user_active is a denormalized field and
flipping a user's is_active will cause inconsistent state by leaving
Subscriptions unupdated. Given that similar things can be introduced in
the future for any other flag not officially supported by having a
setter, the recommendation should "Don't" in general.
2021-10-06 17:38:56 -07:00
Tim Abbott 67fdbbe5fd docs: Inline some upgrade instructions.
It feels like the "Same as" content was unnecessarily requiring the
user to bounce around in these cases.

(I've left the "Same as" text for the Ubuntu ones, where it's two
steps in a row to follow).
2021-10-01 11:08:14 -07:00
Gaurav Pandey 77d1c4361b docs: Document upgrade steps from buster to bullseye.
Fixes #17863.
2021-10-01 11:08:14 -07:00
Mateusz Mandera 8ad7520180 ldap: Support alternative attrs to userAccountControl.
Fixes #17456.

The main tricky part has to do with what values the attribute should
have. LDAP defines a Boolean as
Boolean = "TRUE" / "FALSE"
so ideally we'd always see exactly those values. However,
although the issue is now marked as resolved, the discussion in
https://pagure.io/freeipa/issue/1259 shows how this may not always be
respected - meaning it makes sense for us to be more liberal in
interpreting these values.
2021-09-16 13:29:13 -07:00
Mateusz Mandera b32450f98e docs: Remove mentions of some ldap features being added in 2.0.
2.0 is old enough that explicitly mentioning when these features were
implemented isn't particularly useful and adds clutter.
2021-09-16 13:29:13 -07:00
Eeshan Garg 3b1cb0b25a docs: Indicate latest Zulip version in installation and upgrade docs.
With copy-editing from tabbott, and also a migration to use
LATEST_RELEASE_VERSION, which will be correct even on the /latest/
paths.

Fixes #19695.
2021-09-10 14:50:48 -07:00
Tim Abbott 908e1e6d19 docs: Run prettier on authentication-methods.md.
The new sections of this file were rebased past the addition of
running prettier on these files.
2021-09-10 13:23:11 -07:00
Philipp Seßner a8797b4d9f help: Document how to configure Keycloak as the SAML IdP. 2021-09-10 13:04:04 -07:00
Gaurav Pandey 502697d239 docs: Add documentation for bullseye support.
The support for bullseye was added in #17951
but it was not documented as bullseye was
frozen and did not have proper configuration
files, hence wasn't documented.

Since now bullseye is released as a stable
version, it's support can be documented.
2021-09-09 11:05:16 -07:00
Anders Kaseorg a4dbc1edd4 docs: Format Markdown with Prettier.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-09-08 12:06:24 -07:00
Anders Kaseorg 35c1c8d41b docs: Apply sentence single-spacing from Prettier.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-09-08 12:06:24 -07:00
Anders Kaseorg 915884bff7 docs: Apply bullet style changes from Prettier.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-09-08 12:06:24 -07:00
Anders Kaseorg 6145fdf678 docs: Rewrap to avoid line breaks in inline code spans.
This works around https://github.com/prettier/prettier/issues/11372.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-09-08 12:06:24 -07:00
Anders Kaseorg f4d2d199e2 docs: Display `main` branch name as inline code.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-09-06 12:56:35 -07:00
Anders Kaseorg 646c04eff2 Rename default branch to ‘main’.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-09-06 12:56:35 -07:00
Alex Vandiver 7d7d727865 tools: Switch to download.zulip.com from www.zulip.org. 2021-08-26 17:21:31 -07:00
Alex Vandiver faf71eea41 upgrade-postgresql: Do not remove other supervisor configs.
We previously used `zulip-puppet-apply` with a custom config file,
with an updated PostgreSQL version but more limited set of
`puppet_classes`, to pre-create the basic settings for the new cluster
before running `pg_upgradecluster`.

Unfortunately, the supervisor config uses `purge => true` to remove
all SUPERVISOR configuration files that are not included in the puppet
configuration; this leads to it removing all other supervisor
processes during the upgrade, only to add them back and start them
during the second `zulip-puppet-apply`.

It also leads to `process-fts-updates` not being started after the
upgrade completes; this is the one supervisor config file which was
not removed and re-added, and thus the one that is not re-started due
to having been re-added.  This was not detected in CI because CI added
a `start-server` command which was not in the upgrade documentation.

Set a custom facter fact that prevents the `purge` behaviour of the
supervisor configuration.  We want to preserve that behaviour in
general, and using `zulip-puppet-apply` continues to be the best way
to pre-set-up the PostgreSQL configuration -- but we wish to avoid
that behaviour when we know we are applying a subset of the puppet
classes.

Since supervisor configs are no longer removed and re-added, this
requires an explicit start-server step in the instructions after the
upgrades complete.  This brings the documentation into alignment with
what CI is testing.
2021-08-24 19:00:58 -07:00
Anders Kaseorg 32da7e3eb2 docs: Adjust colon fence for Prettier compatibility.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-08-20 23:14:37 -07:00
Anders Kaseorg 0a3e022376 docs: Adjust list item indentation for Prettier compatibility.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-08-20 23:14:37 -07:00
Anders Kaseorg 4bfffc9f74 docs: Fix list item indentation mistakes.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-08-20 23:14:37 -07:00
Anders Kaseorg fe3db63381 docs: Add missing blockquote.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-08-20 23:14:37 -07:00
Anders Kaseorg b53e676860 docs: Convert reST index pages to Markdown.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-08-20 06:22:42 -07:00
Anders Kaseorg b29b6f6526 docs: Add syntax highlighting languages to code blocks.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-08-20 06:14:47 -07:00
Anders Kaseorg 672358c5da docs: Migrate from recommonmark to MyST-Parser.
Recommonmark is no longer maintained, and MyST-Parser is much more
complete.

https://myst-parser.readthedocs.io/

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-08-19 00:53:32 -07:00
Anders Kaseorg 8f2827a65d docs: Remove trailing newlines from code blocks.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-08-19 00:53:32 -07:00
Anders Kaseorg cb61ea69c6 docs: Fix misaligned Markdown source indentation.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-08-19 00:53:32 -07:00
Anders Kaseorg c3646ec67f docs: Move authentication-methods#ldap anchor to appropriate heading.
Commit 30eaed0378 (#15001) incorrectly
inserted a different section between the anchor and the heading.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-08-19 00:53:32 -07:00
Mateusz Mandera fdbde59b07 rate_limit: Add management command to reset auth rate limit.
The auth attempt rate limit is quite low (on purpose), so this can be a
common scenario where a user asks their admin to reset the limit instead
of waiting. We should provide a tool for administrators to handle such
requests without fiddling around with code in manage.py shell.
2021-08-19 00:45:17 -07:00
Alex Vandiver e94b6afb00 nagios: Remove broken check_email_deliverer_* checks and related code.
These checks suffer from a couple notable problems:
 - They are only enabled on staging hosts -- where they should never
   be run.  Since ef6d0ec5ca, these supervisor processes are only
   run on one host, and never on the staging host.
 - They run as the `nagios` user, which does not have appropriate
   permissions, and thus the checks always fail.  Specifically,
   `nagios` does not have permissions to run `supervisorctl`, since
   the socket is owned by the `zulip` user, and mode 0700; and the
   `nagios` user does not have permission to access Zulip secrets to
   run `./manage.py print_email_delivery_backlog`.

Rather than rewrite these checks to run on a cron as zulip, and check
those file contents as the nagios user, drop these checks -- they can
be rewritten at a later point, or replaced with Prometheus alerting,
and currently serve only to cause always-failing Nagios checks, which
normalizes alert failures.

Leave the files installed if they currently exist, rather than
cluttering puppet with `ensure => absent`; they do no harm if they are
left installed.
2021-08-03 16:07:13 -07:00
Tim Abbott 454a900487 docs: Replace chat-zulip-org page with /developer-community/. 2021-07-25 22:17:57 -07:00
Mateusz Mandera 388932bcc4 docs: Move docs on SOCIAL_AUTH_OIDC_FULL_NAME_VALIDATED to ReadTheDocs. 2021-07-24 15:16:13 -07:00
Mateusz Mandera 9bf84a35ef docs: Mention auto_signup OIDC option in ReadTheDocs documentation. 2021-07-24 15:13:47 -07:00
Mateusz Mandera 0d6b1cd08b docs: Tweak phrasing about OIDC supporting only a single provider.
Strictly speaking, this sentence is talking about the IdP configuration,
while the backend is just GenericOpenIdConnectBackend, so the new
phrasing is more correct.
2021-07-24 15:12:06 -07:00
Alex Vandiver 91282ab490 reindex-textual-data: Provide a tool to reindex all text indices.
The script is added to upgrade steps for 20.04 and Buster because
those are the upgrades that cross glibc 2.28, which is most
problematic.  It will also be called out in the upgrade notes, to
catch those that have already done that upgrade.
2021-07-19 16:34:23 -07:00
Alex Vandiver 528e5adaab smokescreen: Default to only listening on 127.0.0.1.
This prevents Smokescreen from acting as an open proxy.

Fixes #19214.
2021-07-14 15:40:26 -07:00
Anders Kaseorg 47897c76a2 scripts: Use curl -f (--fail).
This makes curl exit with nonzero status on HTTP 4xx/5xx errors.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-07-13 16:47:49 -07:00
Anders Kaseorg 77169408f4 docs: Remove do-release-upgrade -d option.
It has not been necessary since Ubuntu 20.04.1 was released in August.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-07-13 11:02:44 -07:00
Anders Kaseorg 93f62b999e nagios: Replace check_website_response with standard check_http plugin.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-07-09 16:47:03 -07:00
Mateusz Mandera 85cbdc8904 rate_limit: Add rate limiting of ZulipRemoteServer. 2021-07-08 15:55:02 -07:00
Mateusz Mandera 3b4f8cc85b saml: Add setting to skip the "continue to registration" page.
It's a smoother Just-In-Time provisioning process to allow
creating the account and getting signed in on the first login by the
user.
2021-07-08 15:21:20 -07:00
Anders Kaseorg ee8724e436 docs: Correct Big Blue Button to BigBlueButton.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-07-05 16:59:00 -07:00
Anders Kaseorg 91bfebca7d install: Replace wget with curl.
curl uses Happy Eyeballs to avoid long timeouts on systems with broken
IPv6.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-06-25 09:05:07 -07:00
Alex Vandiver 108e7ee8af docs: Update example supervisorctl status. 2021-06-14 17:12:59 -07:00
Mateusz Mandera c54b48452d saml: Support syncing custom profile fields with SAML attributes.
Fixes #17277.

The main limitation of this implementation is that the sync happens if
the user authing already exists. This means that a new user going
through the sign up flow will not have their custom fields synced upon
finishing it. The fields will get synced on their consecutive log in via
SAML in the future. This can be addressed in the future by moving the
syncing code further down the codepaths to login_or_register_remote_user
and plumbing the data through to the user creation process.

We detail that limitation in the documentation.
2021-06-08 14:47:33 -07:00
Tim Abbott 16700fc82c docs: Remove inaccurate LDAP coming soon notice.
We implemented this feature in the series ending with the
documentation in 2abe325147.
2021-06-08 14:29:47 -07:00
Alex Vandiver 161645c83c docs: Link to install-existing-server from production/deployment. 2021-06-02 17:16:14 -07:00
Alex Vandiver 03dd44e0ad docs: Remove link to 16.04, which can be confusing.
The instructions do not just apply to 16.04; the block below describes
the settings, which are correct for all relevant Ubuntu versions.
2021-06-02 16:58:24 -07:00
Alex Vandiver 1771f61f96 docs: Add a missing close paren. 2021-06-01 14:57:42 -07:00
Alex Vandiver 663d3b780d docs: Update path to nginx.conf, as it is now a template.
Also provide the right expansion for the one embedded variable
currently in the template.
2021-06-01 14:57:42 -07:00
Mateusz Mandera e1b8315f5e docs: Give an example command for generating SAML key/cert pair. 2021-06-01 08:29:05 -07:00
Mateusz Mandera b31ef13cf1 docs: Mention that SAML private key cannot be encrypted.
The underlying xmlsec library doesn't give a very useful error when
trying to load an encrypted private key which can lead to confusion.
2021-06-01 08:29:05 -07:00
Mateusz Mandera e17758f8ad auth: Implement a generic OpenID Connect backend.
Fixes #11939.
2021-05-28 09:43:06 -07:00
Tim Abbott 8960e64443 docs: Provide more detailed instructions for requesting API key.
Some users had trouble getting an API key that was resolved by doing
this.

Fixes #18524.
2021-05-27 12:09:40 -07:00
Adam Birds 3c28fa033d docs: Update documentation for new database user and dbname flags.
Updated the install documentation to include the explanation of the
two new install options `--postgresql-database-name` and
`--postgresql-database-user`.
2021-05-25 13:56:05 -07:00
Aman Agrawal d1c385f425 docs: Clarify how to set GIPHY API key in settings. 2021-05-18 17:32:55 -07:00
Anders Kaseorg e015f3ed7d docs: Correct “webapp” to “web app”.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-05-14 13:15:38 -07:00
Anders Kaseorg e3c570401e docs: Document password hashing change from PBKDF2 to Argon2.
This changed in commit 483a351d44
(#3410).

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-05-14 13:07:51 -07:00
Alex Vandiver 653c67a2b6 docs: Explicitly list possible outgoing HTTP connections. 2021-05-13 11:06:30 -07:00
Tim Abbott 6777660e02 deployment: Document smokescreen default and configuration. 2021-05-13 11:05:47 -07:00
Alex Vandiver a9ced1d228 smokescreen: Capitalize Smokescreen in documentation. 2021-05-12 12:32:39 -07:00
Matthew Wegner c43f883699
docs: Fix typo in export docs.
"as part of a high evailability environment" => "as part of a high availability environment".
2021-05-10 17:33:21 -07:00
Mateusz Mandera 8261f7e801 commands: Add delete_user management command and document it. 2021-05-10 15:38:14 -07:00
Siddharth Asthana 6ee7d5c983 docs: Improve documentation for changing a realm's subdomain.
Add a single line for self-hosting case to point to the ReadTheDocs
section of management command to change the realm's subdomain.

Fixes: #17857
2021-05-10 12:33:43 -07:00
Anders Kaseorg 544bbd5398 docs: Fix capitalization mistakes.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-05-10 09:57:26 -07:00
Tim Abbott 35a5cd1e65 docs: Add new summary section to the release lifecycle page. 2021-05-05 18:24:17 -07:00
Alya Abbott 279f7b1c8d user docs: Create separate "Code blocks" user documentation page.
* Move the extended documentation of code blocks to a separate page.
* Merge "code playgrounds" documentation to be a section of that page.
* Document copy widget on code blocks.
* This commit changes how we refer to "```python" type syntax for code
  blocks. Instead of being called a syntax highlighting label, this is
  now referred to as a "language tag", since it serves both syntax
  highlighting and playgrounds.
* Remap all the links.
* Advertise this new page in various places that previously did not have a link.
2021-05-05 16:11:21 -07:00
Wesley Aptekar-Cassels 6b7a3fb74a markdown: Rewrite all external images to use Camo.
Requesting external images is a privacy risk, so route all external
images through Camo.

Tweaked by tabbott for better test coverage, more comments, and to fix
bugs.
2021-04-30 10:36:16 -07:00
Tim Abbott 03e155f38a docs: Add production documentation for custom management commands. 2021-04-29 12:36:13 -07:00
strifel 02c5676a67 ldap: Add advanced LDAP realm access control.
This allows access to be more configurable than just setting one
attribute.  This can be configured by setting the setting
AUTH_LDAP_ADVANCED_REALM_ACCESS_CONTROL.
2021-04-28 09:08:13 -07:00
Aman Agrawal f71b591f95 Revert "docs: Temporarily remove Giphy help center link."
This reverts commit a94b9cc9e2.
2021-04-28 07:07:36 -07:00
Alex Vandiver 4c88da8ed9 scripts: Tool to find the diff to an original settings.py prod template.
This hits the unauthenticated Github API to get the list of tags,
which is rate-limited to 60 requests per hour.  This means that the
tool can only be run 60 times per hour before it starts to exit with
errors, but that seems like a reasonable limit for the moment.
2021-04-27 21:50:33 -07:00
Adam Birds ceb4b239c4 docs: Update install docs to document the install options correctly.
Update `docs/production/install.md` and
`docs/production/deployment.md` to document the install flags that can
be used as part of the installer more clearly.

Fixes #18122.
2021-04-27 19:44:05 -07:00
Alex Vandiver bc9d20eaa2 docs: Describe the process of upgrading settings.py files.
Fixes #17782.
2021-04-27 18:01:28 -07:00
Alex Vandiver ae2c377d13 postgresql: Switch to defaulting to PostgreSQL 13. 2021-04-27 16:55:04 -07:00
Tim Abbott a615dcb83e docs: Fix double-period typo in settings.md. 2021-04-26 17:07:41 -07:00
Anders Kaseorg 6060d0d364 docs: Add missing space to compound verbs “log in”, “set up”, etc.
Noun: backup, checkout, cleanup, login, logout, setup, shutdown, signup,
timeout.

Verb: back up, check out, clean up, log in, log out, set up, shut
down, sign up, time out.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-04-26 09:31:08 -07:00
Mateusz Mandera 1b5a7a2337 docs: Clarify what the certificate is supposed to be in the saml config.
Closes #17955.

Users were sometimes confused about what this certificate is supposed to
be.
2021-04-23 15:16:58 -07:00
Alex Vandiver 476524c0c1 scripts: Add a script to stop the server.
Using `supervisorctl stop all` to stop the server is not terribly
discoverable, and may stop services which are not part of Zulip
proper.

Add an explicit tool which only stops the relevant services.  It also
more carefully controls the order in which services are stopped to
minimize lost requests, and maximally quiesce the server.

Locations which may be stopping _older_ versions of Zulip (without
this script) are left with using `supervisorctl stop all`.

Fixes #14959.
2021-04-21 10:24:08 -07:00
Mateusz Mandera 977a2f7fa0 emails: Rename "missed message email" to "message notification email". 2021-04-21 10:10:54 -07:00
Jennifer Hwang c9f5946239 puppet: Add override for queue_workers_multiprocess.
With tweaks to the documentation by tabbott.

This uses the following configuration option:

[application_server]
queue_workers_multiprocess = false
2021-04-20 14:37:15 -07:00
Tim Abbott f3abe73e72 docs: Use our Digital Ocean marketplace referral code. 2021-04-20 10:45:26 -07:00
Alex Vandiver 7264d44dd4 docs: Minor grammar tweak. 2021-04-16 14:05:42 -07:00
Alex Vandiver e683ae2d24 docs: Add a missing word. 2021-04-16 14:05:42 -07:00
Tim Abbott d58931e2b5 docs: Document situations where one needs smokescreen. 2021-04-16 14:05:42 -07:00
Tim Abbott a94b9cc9e2 docs: Temporarily remove Giphy help center link.
This will fix CI; we can restore the link after Giphy is on zulip.com.
2021-04-05 15:44:20 -07:00
Aman Agrawal 5e83965e80 giphy: Use GIPHY web SDK to allow inserting GIFs in compose box.
We use GIPHY web SDK to create popover containing GIFs in a
grid format. Simply clicking on the GIFs will insert the GIF in the compose
box.

We add GIPHY logo to compose box action icons which opens the GIPHY
picker popover containing GIFs with "Powered by GIPHY"
attribution.
2021-04-05 15:04:49 -07:00
Mateusz Mandera 82b43a8cfe streams: Use "archive stream" phrasing instead of "delete stream".
The previous phrasing was misleading in relation to what the action
actually does.
2021-04-02 22:06:48 -07:00
Tim Abbott 68be06deda docs: Clean up installer troubleshooting help guide.
* Remove references to the defunct zulip-help Google Group.
* Mention how to create a code block.
* Use nicer markdown syntax for links.
2021-04-01 12:10:30 -07:00
Alex Vandiver de46edf966 docs: Remove unused X-Client-IP header from haproxy configuration.
This is unrelated to the X-Real-IP configuration that nginx will add,
and is unused by Zulip.  Remove it, to reduce confusion.
2021-03-31 14:19:38 -07:00
Strifel 209c89be10 ldap: Add option to limit user access to certain realms.
This adds an option for restricting a ldap user
to only be allowed to login into certain realms.
This is done by configuring an attribute mapping of "org_membership"
to an ldap attribute that will contain the list of subdomains the ldap
user is allowed to access. This is analogous to how it's done in SAML.

Co-authored-by: Mateusz Mandera <mateusz.mandera@zulip.com>
2021-03-18 11:19:59 -07:00
Anders Kaseorg 129ea6dd11 nginx: Consistently listen on IPv6 and with HTTP/2.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-03-17 17:46:32 -07:00
Nikhil Maske 6b34ba048d docs: Add a note in Incoming email integration docs.
The note states the incoming emails are rate-limited and
its current limits.

Fixes #17435.
2021-03-08 12:23:10 -08:00
Alex Vandiver d938dd9d4a puppet: Document smokescreen installation, and move to puppet/zulip/.
This is more broadly useful than for just Kandra; provide
documentation and means to install Smokescreen for stand-alone
servers, and motivate its use somewhat more.
2021-03-02 17:16:38 -08:00
aryanshridhar 5aa1205d23 docs: Added reactivate_realm management command. 2021-02-23 16:25:52 -08:00
aryanshridhar 42c49e4fb1 docs: Added missing semicolon in export_single_user management command. 2021-02-23 16:25:52 -08:00
Alex Vandiver 0b736ef4cf puppet: Remove puppet_ops configuration for separate loadbalancer host. 2021-02-22 16:05:13 -08:00
Tim Abbott 04197309ae docs: Explain the export tool purpose more clearly.
The reference to "hardware platforms" was confusing -- we really mean
different logical Zulip database installations.
2021-02-19 10:05:40 -08:00
Tim Abbott 150a07cdcd docs: Add a performance design document.
This document is mainly an answer to a set of questions other
developers have been asking about Zulip's architecture and scalability
design.  It's intended for developers working on Zulip, to help with
thinking about where to prioritize further efforts to optimize
scalability.
2021-02-13 13:05:47 -08:00
Nikhil Maske 89b25556dc prod_settings_template: Move LDAP up into authentication, and merge part 1/2. 2021-02-05 17:02:27 -08:00
rht 16583fd5b4 docs: Say that certbot wildcard has been released. 2021-01-17 11:17:09 -08:00
Aman Agrawal c685d36821 hipchat_import: Remove tool from codebase.
Remove functions and scripts used by HipChat import tool and
those which will no longer be required in future.
2020-12-23 08:28:49 -08:00
Aman Agrawal 62d721e859 docs: Remove HipChat migration guide.
As of Feb 15th 2019, Hipchat Cloud and Stride
have reached End Of Life and are no longer
supported by Atlassian. Since it is almost 2 years
now we can remove the migration guides.
2020-12-23 15:43:13 +05:30
Mateusz Mandera d0dc04a093 models: Rename is_api_super_user to can_forge_sender, 2020-12-21 13:15:39 -08:00
Ben Steinberg f00dba3c81 docs: Fix repeated word. 2020-12-02 11:54:49 -08:00
Tim Abbott 067cd3a97a docs: Remove incorrect references to chat.zulip.org.
Most of these are Help Center links that should be pointing to the
production Help Center.
2020-10-29 16:46:40 -07:00
ryanreh99 dfa7ce5637 uploads: Support non-AWS S3-compatible server.
Boto3 does not allow setting the endpoint url from
the config file. Thus we create a django setting
variable (`S3_ENDPOINT_URL`) which is passed to
service clients and resources of `boto3.Session`.

We also update the uploads-backend documentation
and remove the config environment variable as now
AWS supports the SIGv4 signature format by default.
And the region name is passed as a parameter instead
of creating a config file for just this value.

Fixes #16246.
2020-10-28 21:59:07 -07:00
Alex Vandiver 6b9d7000b5 puppet: Set proxy environment variables.
These are respected by `urllib`, and thus also `requests`.  We set
`HTTP_proxy`, not `HTTP_PROXY`, because the latter is ignored in
situations which might be running under CGI -- in such cases it may be
coming from the `Proxy:` header in the request.
2020-10-28 12:17:35 -07:00
Alex Vandiver 524ffe6acc docs: Rename postgres.md to postgresql.md. 2020-10-28 11:57:03 -07:00
Alex Vandiver 2b0bbbb882 tools: Rename postgres to postgresql in tool names. 2020-10-28 11:57:02 -07:00
Alex Vandiver 5eb8064a1a install: Rename postgres options to postgresql. 2020-10-28 11:55:32 -07:00
Alex Vandiver 1f7132f50d docs: Standardize on PostgreSQL, not Postgres. 2020-10-28 11:55:16 -07:00
Alex Vandiver eaa99359b1 puppet: Rename to check_postgresql_replication_lag. 2020-10-28 11:51:52 -07:00
Alex Vandiver 53e59a0a13 puppet: Rename check_postgres_backup to check_postgresql_backup. 2020-10-28 11:51:52 -07:00
Tim Abbott 3a4b38d732 docs: Clarify discussion of no_serve_uploads. 2020-10-27 13:39:14 -07:00
Alex Vandiver a155430eb5 docs: Document all zulip.conf settings.
This provides a single reference point for all zulip.conf settings;
these mostly link out to the more complete documentation about each
setting, elsewhere.

Fixes #12490.
2020-10-27 13:31:57 -07:00
Alex Vandiver d24c571bab puppet: Automatically back up the database if we have the secrets.
This avoids folks having to manually add to the puppet_classes.
2020-10-27 13:29:19 -07:00
Alex Vandiver e7798d2797 puppet: Move zulip_ops::profile::postgres_appdb to postgresql. 2020-10-27 13:29:19 -07:00
Alex Vandiver 9f25389bff puppet: Move top-level zulip_ops deployments to zulip_ops::profile. 2020-10-27 13:29:19 -07:00
Alex Vandiver 188af57296 puppet: Rename postgres_appdb to postgresql.
There is only one PostgreSQL database; the "appdb" is irrelevant.
Also use "postgresql," as it is the name of the software, whereas
"postgres" the name of the binary and colloquial name.  This is minor
cleanup, but enabled by the other renames in the previous commit.
2020-10-27 13:29:19 -07:00
Alex Vandiver 0f25acc7b3 puppet: Rename "voyager"/"dockervoyager" to "standalone"/"docker".
The "voyager" name is non-intuitive and not significant.
`zulip::voyager` and `zulip::dockervoyager` stubs are kept for
back-compatibility with existing `zulip.conf` files.
2020-10-27 13:29:19 -07:00
Alex Vandiver c2185a81d6 puppet: Move top-level zulip deployments into "profile" directory.
This moves the puppet configuration closer to the "roles and profiles
method"[1] which is suggested for organizing puppet classes.  Notably,
here it makes clear which classes are meant to be able to stand alone
as deployments.

Shims are left behind at the previous names, for compatibility with
existing `zulip.conf` files when upgrading.

[1] https://puppet.com/docs/pe/2019.8/the_roles_and_profiles_method
2020-10-27 13:29:19 -07:00
sahil839 a96811ab58 management: Replace knight command with change_user_role command.
We replace knight command with change_user_role command which
allows us to change role of a user to owner, admins, member and
guest. We can also give/revoke api_super_user permission using
this command.

Tweaked by tabbott to improve the logging output and update documentation.

Fixes #16586.
2020-10-25 14:55:08 -07:00
Alex Vandiver c296b5d819 puppet: Allow unattended-upgrades for all but servers.
Restarting servers is what can cause service interruptions, and
increase risk.  Add all of the servers that we use to the list of
ignored packages, and uncomment the default allowed-origins in order
to enable unattended upgrades.
2020-10-23 16:46:06 -07:00
Anders Kaseorg 72d6ff3c3b docs: Fix more capitalization issues.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-10-23 11:46:55 -07:00
Alex Vandiver 50e9e2ed20 puppet: Make zulip::base include zulip::apt_repository.
There was likely more dependency complexity prior to 97766102df, but
there is now no reason to require that consumers explicitly include
zulip::apt_repository.
2020-10-22 11:30:53 -07:00
Alex Vandiver 84142df152 docs: Drop an out-of-date paragraph that talks of 2017 in the future. 2020-10-19 16:28:22 -07:00
Tim Abbott 54dd612f5c docs: Update list of files new since last release.
This fixes CI failing due to our not having released the /video-calls.md rename.
2020-10-18 17:16:31 -07:00
Tim Abbott 2ba76c8289 docs: Merge Zoom and BBB configuration pages. 2020-10-18 15:10:23 -07:00
strifel 1ed1614c9d docs: Add more specific Big Blue Button documentation. 2020-10-18 15:03:16 -07:00
Anders Kaseorg dd48dbd912 docs: Add spaces to “check out”, “log in”, “set up”, “sign up” as verbs.
“Checkout”, “login”, “setup”, and “signup” are nouns, not verbs.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-10-13 15:47:13 -07:00
Brian Lee 9685854edc docs: Add docker push notifications registration example.
Tweaked by tabbott to provide a more detailed comment.
2020-10-02 15:27:32 -07:00
Alex Vandiver 7c3903fc0d docs: Fix count of replaced lines. 2020-08-30 15:26:50 -07:00
Alex Vandiver b23a55e669 export: Add a --deactivate flag which happens just prior to export.
This removes a manual step during export.
2020-08-30 15:26:50 -07:00
Alex Vandiver b4c2ae9cae settings: Adjust documentation and comment references to settings.py.
`zproject/settings.py` itself is mostly-empty now.  Adjust the
references which should now point to `zproject/computed_settings.py`
or `zproject/default_settings.py`.
2020-08-24 13:13:16 -07:00
Tim Abbott 2d70a73902 docs: Fix spelling of audit_fix_indexes.
Fixes #16140.
2020-08-14 16:56:35 -07:00
Alex Vandiver 7a1002af96 docs: Add explicit steps to verify FTS indexes after upgrading to 3.0.
The OS upgrade paths which go through 2.1 do not call
`upgrade-zulip-stage-2` with `--audit-fts-indexes` because that flag
was added in 3.0.

Add an explicit step to do this audit after the 3.0 upgrade.  Stating
it as another command to run, rather than attempting to tell them
to add it to the `upgrade-zulip` call that we're linking to seems
easiest, since that does not dictate if they should upgrade to a
release or from the tip of git.

We do not include a step describing this for the Trusty -> Xenial
upgrade, because the last step already chains into Xenial -> Bionic,
which itself describes auditing the indexes.

Fixes #15877.
2020-08-12 12:48:50 -07:00
Alex Vandiver 8dcc0928ed docs: Be explicit about continuing with upgrades.
Strongly suggest Xenial -> Bionic, or upgrading to 3.x, at the end of
the various other upgrading steps.
2020-08-12 12:48:50 -07:00
Alex Vandiver 3364ca7899 docs: Fold "check if it is working" into the last step. 2020-08-12 12:48:50 -07:00
Alex Vandiver 450f94a1ce docs: Don't suggest --audit-fts-indexes for non-3.0 upgrades.
Only Zulip 3.0 and above support the `--audit-fts-indexes` option to
`upgrade-zulip-stage-2`; saying "same as Bionic to Focal" on other
other steps, which are for Zulip 2.1 or 2.0, will result in errors.

Provide the full text of the updated `upgrade-zulip-stage-2` call in
step 5 for all non-3.0 upgrades.  For Trusty to Xenial and Stretch to
Buster, we do not say "Same as Xenial to Bionic" , because it is
likely that readers do not notice that step does not read "Same as
Bionic to Focal."
2020-08-12 12:48:50 -07:00
Anders Kaseorg 6dea085187 docs: Correct “login” as a verb to “log in”.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-08-11 10:25:53 -07:00
Anders Kaseorg 64b2212c23 docs: Correct “setup” as a verb to “set up”.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-08-11 10:25:53 -07:00
Anders Kaseorg c155403884 docs: Fix various capitalization errors.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-08-11 10:25:52 -07:00
Anders Kaseorg 768f9f93cd docs: Capitalize Markdown consistently.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-08-11 10:23:06 -07:00
Anders Kaseorg 6698fedd1e docs: Change Zoom app type to user-managed.
Based on the Zoom documentation, a user-managed app seems more
appropriate for our use.

https://marketplace.zoom.us/docs/guides/build#account-level-user-managed-apps

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-08-10 17:00:41 -07:00
Tim Abbott d63cf04a20 docs: Clarify how manage.py backup --output works. 2020-08-09 17:44:32 -07:00
Tim Abbott 4baf5d73d9 docs: Fix incorrect path to restart-server. 2020-07-31 16:09:00 -07:00
Anders Kaseorg 7ee974729b docs: Replace Digital Ocean with DigitalOcean.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-07-30 11:30:43 -07:00
Dinesh 4afce5d94d apple_auth: Change BUNDLE_ID setting to APP_ID everywhere.
The apple developer webapp consistently refers this App ID. So,
this clears any confusion that can occur.

Since python social auth only requires us to include App ID in
_AUDIENCE(a list), we do that in computed settings making it easier for
server admin and we make it much clear by having it set to
APP_ID instead of BUNDLE_ID.
2020-07-28 17:12:49 -07:00
Dinesh 782970d9f9 apple_auth: Change where private key is stored.
Changes to a better name apple-auth-key.p8 and removes the extra
directory apple.
2020-07-28 17:12:49 -07:00
Tim Abbott 26bafa2e73 docs: Fix versions in stretch=>buster documentation. 2020-07-22 16:35:05 -07:00
Alex Vandiver 6c27f07c1d puppet: Move PostgreSQL backups to their own class.
wal-g was used in `puppet/zulip` by env-wal-g, but only installed in
`puppet/zulip_ops`.

Merge all of the dependencies of doing backups using wal-g (wal-g
installation, the pg_backup_and_purge job, the nagios plugin that
verifies it happens) into a common base class in `puppet/zulip`, since
it is generally useful.
2020-07-14 00:40:25 -07:00
Mateusz Mandera c231d88d9f upgrade: Add management command to fix FTS indexes.
Upgrading the base OS's dictionary files can corrupt our FTS
indexes. We add a command for fixing them.

Fixes #14982.
2020-07-13 12:40:44 -07:00
Tim Abbott 92d33070dd docs: Improve explanation of running master.
It's a bit annoying we haven't renamed master yet, but we'll do that
in a bulk pass.
2020-07-08 12:17:59 -07:00
Alex Vandiver 0d7dbd1b07 puppet: Apply basic PostgreSQL configuration before pg_upgradecluster.
Running `pg-upgradecluster` runs the `CREATE TEXT SEARCH DICTIONARY`
and `CREATE TEXT SEARCH CONFIGURATION` from
`zerver/migrations/0001_initial.py` on the new PostgreSQL cluster;
this requires that the stopwords file and dictionary exist _prior_
to `pg_upgradecluster` being run.

This causes a minor dependency conflict -- we do not wish to duplicate
the functionality from `zulip::postgres_appdb_base` which configures
those files, but installing all of `zulip::postgres_appdb_tuned` will
attempt to restart PostgreSQL -- which has not configured the cluster
for the new version yet.

In order to split out configuration of the prerequisites for the
application database, and the steps required to run it, we need to be
able to apply only part of the puppet configuration.  Use the
newly-added `--config` argument to provide a more limited `zulip.conf`
which only applies `zulip::postgres_appdb_base` to the new version of
Postgres, creating the required tsearch data files.

This also preserves the property that a failure at any point prior to
the `pg_upgradecluster` is easily recoverable, by re-running
`zulip-puppet-apply`.
2020-07-06 18:30:16 -07:00
arpit551 294510c68a docs: Update documentation for Bionic to Focal upgrade.
Added -d Flag in do-release-upgrade for Bionic to Focal upgrade.
The -d switch is necessary to upgrade from Ubuntu 18.04 LTS
as upgrades have not yet been enabled and will only be enabled
after the first point release of 20.04 LTS.
Source https://wiki.ubuntu.com/FocalFossa/ReleaseNotes
2020-07-03 16:51:51 -07:00
Tim Abbott dc05b5c317 docs: Highlight that Zulip now defaults to postgres 12. 2020-06-26 16:26:57 -07:00
Alex Vandiver 16d131bb17 docs: Add an explicit section about upgrading PostgreSQL.
Fixes #15415.
2020-06-26 16:16:12 -07:00
Alex Vandiver c94aed905e docs: Remove one indirection in the docs.
Pointing at Xenial to Bionic, which just points to Bionic to Focal, is
merely an extra hop.
2020-06-26 16:16:12 -07:00
Alex Vandiver 2494d4bcae docs: Repeat the post-upgrade check stanza after each section. 2020-06-26 16:16:12 -07:00
Alex Vandiver da03761871 docs: Clarify that earlier distro upgrades cannot be done with 3.0.
As alluded to in the previous commit, only 3.0 can use the new tooling
-- indeed, it requires it, as the zulip.conf entry must be changed.
Clarify that in the upgrade steps for earlier distributions.
2020-06-26 16:16:11 -07:00
Alex Vandiver f642c7c568 docs: Update Bionic -> Focal database upgrade steps for db tool.
Update the upgrade documentation for the tool added in the previous
step.  Only the Bionic -> Focal upgrade step need be updated, because
none of the other upgrade steps can be run starting from a Zulip 3.0
installation.

Fixes #15415.
2020-06-26 16:07:39 -07:00
Tim Abbott e46bbf18eb docs: Change next planned major release to 3.0.
After some discussion, everyone seems to agree that 3.0 is the more
appropriate version number for our next major release.  This updates
our documentation to reflect that we'll be using 3.0 as our next major
release.
2020-06-24 16:27:27 -07:00
Alex Vandiver 31f1f10501 installer: Halt if wrong version of PostgreSQL is already installed.
49a7a66004 and immediately previous commits began installing
PostgreSQL 12 from their apt repository.  On machines which already
have the distribution-provided version of PostgreSQL installed,
however, this leads to failure to apply puppet when restarting
PostgreSQL 12, as both attempt to claim the same port.

During installation, if we will be installing PostgreSQL, look for
other versions than what we will install, and abort if they are
found.  This is safer than attempting to automatically uninstall or
reconfigure existing databases.
2020-06-24 12:57:38 -07:00
Alex Vandiver 58cb7cecd8 installer: Remove `--remote-postgres`, redundant with `--no-init-db`.
The previous commit removed the only behavior difference between the
two flags; both of them skip user/database creation, and the tables
therein.

Of the two options `--no-init-db` is more explicit as to what it does,
as opposed to just one facet of when it might be used; remove
`--remote-postgres`.
2020-06-24 12:57:38 -07:00
Greg Price baa09f0518 docs: Clarify instructions on setting up email for Apple auth.
This comes after I read through all the linked docs, and went through
the UI to do this registration.
2020-06-23 19:04:26 -07:00
Greg Price 71c995d50d docs: Explain a bit more explicitly how to configure Apple auth.
In particular the Services ID and Bundle ID each have one of Apple's
random-looking 10-character identifiers, in addition to the Java-style
names the admin chooses.  Best to be clear about what names are
supposed to be the chosen names and which are supposed to be the
random-looking assigned names.

(I don't know of any docs elsewhere making this clear -- but I guessed
it'd be this way, and empirically it works.)

Also mention you need to enable the backend. :-)
2020-06-23 19:04:26 -07:00
Greg Price 0021dfe8e7 auth settings: Put Apple "Team ID" before the things it namespaces.
I believe the Bundle ID (aka App ID) and Services ID have meaning only
relative to a specific Team ID.  In particular, in some places in the
developer.apple.com UI, they're displayed in a fully-qualified form
like "ABCDE12345.com.example.app", where "com.example.app" is the
App ID or Services ID and ABCDE12345 is the Team ID.
2020-06-23 19:04:26 -07:00
Greg Price 9b620dfe6c docs: Break up a long sentence in Apple auth instructions.
This also helps keep the actionable part short, by separating
the actionable instruction from the longer parenthetical
explaining what it's for.
2020-06-23 19:04:26 -07:00
Brainrecursion 30eaed0378 saml: Add option to restrict subdomain access based on SAML attributes.
Adds the ability to set a SAML attribute which contains a
list of subdomains the user is allowed to access. This allows a Zulip
server with multiple organizations to filter using SAML attributes
which organization each user can access.

Cleaned up and adapted by Mateusz Mandera to fit our conventions and
needs more.

Co-authored-by: Mateusz Mandera <mateusz.mandera@zulip.com>
2020-06-23 17:14:31 -07:00
Tim Abbott 077c741ef4 docs: Simplify reviewing upgrade notes before upgrading.
This adds a convenient way to review the upgrade notes for all Zulip
releases that one is upgrading across.

I thought about moving all the upgrade notes to a common section, but
in some cases the language is clearly explaining changes in the
release that are not duplicated elsewhere, and I think it reads better
having them inline alongisde related changes.
2020-06-22 15:58:52 -07:00
Anders Kaseorg 62f1a9da26 docs: Use Mozilla recommended SSL settings for Apache.
• Specify disabled rather than enabled protocols, so as not to disable
  TLS 1.3.
• Provide an explicit cipher suite list (Mozilla intermediate config
  version 5.4).
• Respect the browser’s preferred cipher suite ordering over the
  server’s.
• Use FFDHE2048 Diffie-Hellman parameters.
• Disable SSL session tickets.

(SSL stapling is also recommended but SSLStaplingCache cannot be
configured inside a <VirtualHost> block.)

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-14 20:17:12 -07:00
Tim Abbott 365fed531a docs: Remove nginx reverse proxy websockets documentation.
Zulip no longer uses websockets in production, so this code was
unnecessary.
2020-06-14 15:17:32 -07:00
qnxor 6399bccc07 docs: Add Apache2 reverse proxy instructions and example.
Tweaked by tabbott to disable older SSL and remove websockets logic,
which isn't relevant in master.
2020-06-14 15:15:23 -07:00
Tim Abbott a361646221 docs: Fix references to removed puppet rules. 2020-06-14 12:47:22 -07:00
Alex Vandiver 4fe0444108 puppet: Install wal-g, not wal-e. 2020-06-11 15:52:43 -07:00
Tim Abbott f0d8f60b66 help: Add basic documentation of organization owners. 2020-06-10 14:07:46 -07:00
Dinesh dc90d54b08 auth: Add Sign in with Apple support.
This implementation overrides some of PSA's internal backend
functions to handle `state` value with redis as the standard
way doesn't work because of apple sending required details
in the form of POST request.

Includes a mixin test class that'll be useful for testing
Native auth flow.

Thanks to Mateusz Mandera for the idea of using redis and
other important work on this.

Documentation rewritten by tabbott.

Co-authored-by: Mateusz Mandera <mateusz.mandera@zulip.com>
2020-06-09 17:29:35 -07:00
Tim Abbott 5154ddafca docs: Update production supported releases.
Now that we have production support for Ubuntu Focal, we update the
documentation to state our support for it.

(We also drop deprecated Xenial and Stretch from supported platforms).
2020-06-08 22:11:28 -07:00
Tim Abbott 7b8ba5ebd9 docs: Update most remaining references to zulipchat.com.
In some cases, the cleanest tweak is to replace references to the
domain with Zulip Cloud, the product.
2020-06-08 18:10:45 -07:00
Tim Abbott 71078adc50 docs: Update URLs to use https://zulip.com.
We're migrating to using the cleaner zulip.com domain, which involves
changing all of our links from ReadTheDocs and other places to point
to the cleaner URL.
2020-06-08 18:10:45 -07:00
Anders Kaseorg 08ddeca8a5 docs: Fix an incorrect use of i.e.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-08 16:28:05 -07:00
Anders Kaseorg 4d04fa3118 compose: Rewrite Zoom video call integration to use OAuth.
This reimplements our Zoom video call integration to use an OAuth
application.  In addition to providing a cleaner setup experience,
especially on zulipchat.com where the server administrators can have
done the app registration already, it also fixes the limitation of the
previous integration that it could only have one call active at a time
when set up with typical Zoom API keys.

Fixes #11672.

Co-authored-by: Marco Burstein <marco@marco.how>
Co-authored-by: Tim Abbott <tabbott@zulipchat.com>
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2020-06-03 16:39:12 -07:00
Tim Abbott 4d2b1673f8 docs: Replace support@zulipchat.com with support@zulip.com.
The new address is cleaner and shorter.
2020-05-28 08:14:30 +00:00
Mateusz Mandera 501e7c44dc docs: Add instructions for SAML with Okta/OneLogin in /help/.
Tweaked by tabbott to shift how this is organized.
2020-05-28 08:14:30 +00:00
Mateusz Mandera b66dc9de50 saml: Support IdP-initiated SSO. 2020-05-25 16:09:30 -07:00
Alex Vandiver 031260573f docs: Link to section on migrating local -> S3 storage.
This section at the top was clearly written before the documentation
at the bottom existed, and hasn't been updated to point to the
now-existent docs below.

Add the link, rather than directing to #production-help.
2020-05-19 14:08:44 -07:00
Steve Howell 23c2198da3 docs: Use string_id in header. 2020-05-17 21:25:53 -07:00
Steve Howell 35df5f35d8 docs: Keep manual-restore steps sequential. 2020-05-17 21:25:53 -07:00
Tim Abbott 1c1c47b94a docs: Delete discussion of legacy push notifications signup.
This hasn't been used in months; it was only there in case of problems
with the rollout of the automated workflow, so it makes sense to
delete it now.
2020-05-05 15:59:35 -07:00
Vishnu Ks 9a8d0ca9fe docs: Recommend contact page over email wherever possible.
With a few tweaks from tabbott to preserve the enthusiasm for feedback
and de-emphasize twitter as a channel (we give better support
elsewhere).
2020-05-05 15:57:54 -07:00
Tim Abbott 9566905b81 docs: Clarify nginx reverse proxy documentation.
Fixes #14740.
2020-04-26 10:51:39 -07:00
Tim Abbott 1219a2e854 docs: Deprecate support for Xenial and Stretch.
Also make sure our documentation for upgrading is reasonable for
Stretch => Buster.

Our reasoning for deprecating support for these releases is as follows:

* Ubuntu 16.04 Xenial reached desktop EOL last year; and will reach
  EOL on the server in about a year.

* Debian Stretch will each EOL in 2020 (the precise date is unclear in
  Debian's documentation, but based on past precedent it's in the next
  few months, perhaps July 2020).
  https://wiki.debian.org/DebianReleases#Production_Releases

* Both Ubuntu 16.04 and Debian Stretch use Python 3.5 as the system
  Python, which will reach EOL in September 2020 (and we're already
  seeing various third-party dependencies that we use drop support for
  them).

* While there is LTS support for these older releases, it's not clear it's
  going to be worth the added engineering effort for us to maintain EOL
  releases of the base OSes that we support.

* We (now) have clear upgrade instructions for moving to Debian Buster
  and Ubuntu 18.04.
2020-04-16 15:36:18 -07:00
Tim Abbott 724fcc74e8 help: Improve documentation on importing exported organizations.
This should help avoid confusion where some users used the Zulip
backup tool, not the import tool, to try to import their backups.
2020-03-30 13:25:03 -07:00
thoemie 7a48bec189 docs: Document EMAIL_USE_SSL for smtp providers on port 465.
I found the solution by simply trying out EMAIL_USE_SSL and it
worked. I had problems with sending emails (did not work at all, there
wasn't even a connection going on - I checked with tcpdump. Then I
found this: To use port 465, you need to call
smtplib.SMTP_SSL(). Currently, it looks like Django only uses
smtplib.SMTP() (source: https://code.djangoproject.com/ticket/9575).

Fixes #14350.
2020-03-27 00:25:49 -07:00
Anders Kaseorg 7ff9b22500 docs: Convert many http URLs to https.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-03-26 21:35:32 -07:00
Mateusz Mandera f9db77c400 docs: Updated links to python-social-auth docs.
The URL seems to have changed.
2020-03-20 08:54:37 -04:00
Stefan Weil c220b971ae
docs: Fix some typos in documentation (most of them found and fixed by codespell).
Signed-off-by: Stefan Weil <sw@weilnetz.de>
2020-03-17 05:57:10 -07:00
Rike-Benjamin Schuppner bb49a60bd5
docs: Fix type in authentication methods. 2020-03-15 17:55:08 -07:00
Tim Abbott 6c74af4c06 docs: Document our stable release branches.
The experiment with the published 2.1.x tpye branches has been going
well, so we should document the feature.
2020-02-29 11:50:44 -08:00
Tim Abbott b775becc68 docs: Change our security contact to security@.
This has for a while been our only active Google Groups mailing list,
and given that folks will guess security@ as our security contact, we
might as well just standardize on that.

Also tweak some ambiguous text; it wouldn't be appropriate for us to
issue a CVE for e.g. an operational issue only affecting us.
2020-02-26 16:35:29 -08:00
Mateusz Mandera 7814f52d45 docs: Replace links to Django 1.11 docs with 2.2 links. 2020-02-19 11:51:18 -08:00
Tim Abbott 4a36ed6cb2 docs: Advertise support for GitLab authentication. 2020-02-11 14:13:39 -08:00
xpac1985 65fe1a9eed
docs: Add info about zulip-announce RSS feed to install docs.
The mailing list can also be subscribed to via RSS/Atom feeds, I just wanted to make that information easier accessible.
2020-01-31 17:24:43 -08:00
Tim Abbott d356622594 docs: Add link from LDAP docs to invitation docs.
This addresses confusion we had with some organizations where they
were surprised that with only LDAP enabled, the "invite more users"
feature was available.

Fixes #11685.
2020-01-25 23:41:19 -08:00
Tim Abbott d70e799466 bots: Remove FEEDBACK_BOT implementation.
This legacy cross-realm bot hasn't been used in several years, as far
as I know.  If we wanted to re-introduce it, I'd want to implement it
as an embedded bot using those common APIs, rather than the totally
custom hacky code used for it that involves unnecessary queue workers
and similar details.

Fixes #13533.
2020-01-25 22:41:39 -08:00
Anders Kaseorg ea6934c26d dependencies: Remove WebSockets system for sending messages.
Zulip has had a small use of WebSockets (specifically, for the code
path of sending messages, via the webapp only) since ~2013.  We
originally added this use of WebSockets in the hope that the latency
benefits of doing so would allow us to avoid implementing a markdown
local echo; they were not.  Further, HTTP/2 may have eliminated the
latency difference we hoped to exploit by using WebSockets in any
case.

While we’d originally imagined using WebSockets for other endpoints,
there was never a good justification for moving more components to the
WebSockets system.

This WebSockets code path had a lot of downsides/complexity,
including:

* The messy hack involving constructing an emulated request object to
  hook into doing Django requests.
* The `message_senders` queue processor system, which increases RAM
  needs and must be provisioned independently from the rest of the
  server).
* A duplicate check_send_receive_time Nagios test specific to
  WebSockets.
* The requirement for users to have their firewalls/NATs allow
  WebSocket connections, and a setting to disable them for networks
  where WebSockets don’t work.
* Dependencies on the SockJS family of libraries, which has at times
  been poorly maintained, and periodically throws random JavaScript
  exceptions in our production environments without a deep enough
  traceback to effectively investigate.
* A total of about 1600 lines of our code related to the feature.
* Increased load on the Tornado system, especially around a Zulip
  server restart, and especially for large installations like
  zulipchat.com, resulting in extra delay before messages can be sent
  again.

As detailed in
https://github.com/zulip/zulip/pull/12862#issuecomment-536152397, it
appears that removing WebSockets moderately increases the time it
takes for the `send_message` API query to return from the server, but
does not significantly change the time between when a message is sent
and when it is received by clients.  We don’t understand the reason
for that change (suggesting the possibility of a measurement error),
and even if it is a real change, we consider that potential small
latency regression to be acceptable.

If we later want WebSockets, we’ll likely want to just use Django
Channels.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-01-14 22:34:00 -08:00
Mateusz Mandera e477cae800 docs: Fix missing apostrophe in EMAIL_HOST_USER value. 2020-01-03 16:52:31 -08:00
Mateusz Mandera dc59850d15 docs: Fix incorrect path to get-django-setting script. 2020-01-03 16:52:31 -08:00
Mateusz Mandera d88494deae docs: Add some troubleshooting notes for ldap. 2020-01-03 16:52:30 -08:00
Mateusz Mandera bfb963b9aa docs: Include suggested USERNAME_ATTR in example AD ldap configs. 2020-01-03 16:46:07 -08:00
Tim Abbott e38c58e7c7 docs: Rewrite LDAP discussion of AUTH_LDAP_REVERSE_EMAIL_SEARCH.
This moves the mandatory configuration for options A/B/C into a single
bulleted list for each option, rather than split across two steps; I
think the result is significantly more readable.

It also fixes a bug where we suggested setting
AUTH_LDAP_REVERSE_EMAIL_SEARCH = AUTH_LDAP_USER_SEARCH in some cases,
whereas in fact it will never work because the parameters are
`%(email)s`, not `%(user)s`.

Also, now that one needs to set AUTH_LDAP_REVERSE_EMAIL_SEARCH, it
seems worth adding values for that to the Active Directory
instructions.  Thanks to @alfonsrv for the suggestion.
2019-12-13 13:55:52 -08:00
Vishnu KS 6901087246 install: Use crudini for storing value of POSTGRES_MISSING_DICTIONARIES.
This simplifies the RDS installation process to avoid awkwardly
requiring running the installer twice, and also is significantly more
robust in handling issues around rerunning the installer.

Finally, the answer for whether dictionaries are missing is available
to Django for future use in warnings/etc. around full-text search not
being great with this configuration, should they be required.
2019-12-13 12:05:39 -08:00
Tim Abbott 35959d43c4 docs: Clean up troubleshooting guide.
This article is definitely still below our polish goals, but this is
also definitely an improvement.
2019-12-12 22:19:12 -08:00
Mateusz Mandera 8bd2a130a9 docs: Fix some typos. 2019-12-12 17:19:10 -08:00
Tim Abbott 171c6f119d docs: Clean up upgrade/modify documentation. 2019-12-12 17:02:07 -08:00
Tim Abbott 305adc4f64 docs: Clean up requirements page. 2019-12-12 16:31:02 -08:00
Tim Abbott 080864ca44 docs: Minor edits to export and management command docs. 2019-12-12 16:06:40 -08:00
Tim Abbott ea60670c9f docs: Clean up some editing issues in export docs. 2019-12-12 15:56:23 -08:00
Vishnu KS 6c97a36355 install: Support remote database services like RDS.
Documentation and variable names edited by tabbott.
2019-12-12 12:59:45 -08:00
Tim Abbott 23e3ae1211 docs: Reorganize some SAML setup instructions.
This reads and line-wraps better.
2019-12-11 16:32:43 -08:00
Mateusz Mandera ed513f1f59 docs: Explain SAML ACS url with SOCIAL_AUTH_SUBDOMAIN. 2019-12-11 16:29:46 -08:00
Tim Abbott f5eece9b4f docs: Fix broken social authentication links. 2019-12-10 17:34:27 -08:00
Mateusz Mandera b3085f924d docs: List the backends in section explaining SOCIAL_AUTH_SUBDOMAIN.
Tweaked by tabbott to remove the reference to `python-social-auth`;
the key detail is whether a callback URL is involved.
2019-12-10 17:16:20 -08:00
Tim Abbott c6fe6cf0a4 docs: Fix recommendations to copy secrets during data import.
The previous documentation was essentially wrong, in that it
recommended copying certain settings that would cause significant
problems post-import if they were indeed copied.
2019-12-09 17:57:37 -08:00
Tim Abbott 6ca56f81f2 docs: Document RabbitMQ configuration issue when importing. 2019-12-09 17:22:04 -08:00
Tim Abbott 79604c7817 docs: Update import docs to mention RAM requirements. 2019-12-09 17:22:04 -08:00
Tim Abbott b036fa897e docs: Move unattended-upgrades docs to troubleshooting guide.
This also rewrites the text to better explain what's happening.  It's
likely further polish would be valuable, but that's true for the whole
"Troubleshooting" page.

This block of text was misplaced when we split the long
maintain-secure-update; article; we want it to be easy to find by
folks who are looking into error emails Zulip is sending.
2019-12-02 11:37:16 -08:00
David Rosa 62d555115a docs: Improve formatting of important notes.
Improves formatting using reST and Sphinx admonition directives
so that important notes stand out when viewed in ReadTheDocs.
2019-12-02 10:37:49 -08:00
Tim Abbott c9a3e4e872 docs: Reorganize and refocus section on management commands. 2019-11-22 10:48:37 -08:00
Tim Abbott 5a7b5f1337 docs: Rework text for scalability and monitoring sections.
This text is very old and hadn't been edited in a long time, in large
part because it was buried within old docs.  This change cleans it up
to give accurate and better-organized information.
2019-11-22 10:22:07 -08:00
David Rosa 87a2831b83 docs: Split maintain-secure-upgrade into dedicated docs.
* Moves "Management commands" to a top-level section.
* Moves "Scalability" as a subsection at the bottom of "Requirements".
* Moves "Monitoring" as a subsections at the bottom of "Troubleshooting".
* Replaces "API and your Zulip URL" with a link to REST API docs.  This
  documentation text has been irrelevant for some time.
* Removes maintain-secure-upgrade from the TOC but the file remains to
  avoid breaking old links from release blog posts and emails.
2019-11-22 10:21:20 -08:00
Tim Abbott 6d17fea076 docs: Document a few management commands around email. 2019-11-18 15:45:21 -08:00
rht 517534777d export-and-import: Update repo url of Zulip Archive. 2019-11-15 11:39:52 -08:00
David Rosa b041948132 docs: Reorganize auth and migrations subsystems.
- Moves "Authentication in the development environment" from subsystems
to "development/authentication.md".
- Moves "Renumbering migrations" to a section within "Schema migrations".
2019-11-07 09:42:36 -08:00