Commit Graph

456 Commits

Author SHA1 Message Date
Mateusz Mandera fdbde59b07 rate_limit: Add management command to reset auth rate limit.
The auth attempt rate limit is quite low (on purpose), so this can be a
common scenario where a user asks their admin to reset the limit instead
of waiting. We should provide a tool for administrators to handle such
requests without fiddling around with code in manage.py shell.
2021-08-19 00:45:17 -07:00
Alex Vandiver e94b6afb00 nagios: Remove broken check_email_deliverer_* checks and related code.
These checks suffer from a couple notable problems:
 - They are only enabled on staging hosts -- where they should never
   be run.  Since ef6d0ec5ca, these supervisor processes are only
   run on one host, and never on the staging host.
 - They run as the `nagios` user, which does not have appropriate
   permissions, and thus the checks always fail.  Specifically,
   `nagios` does not have permissions to run `supervisorctl`, since
   the socket is owned by the `zulip` user, and mode 0700; and the
   `nagios` user does not have permission to access Zulip secrets to
   run `./manage.py print_email_delivery_backlog`.

Rather than rewrite these checks to run on a cron as zulip, and check
those file contents as the nagios user, drop these checks -- they can
be rewritten at a later point, or replaced with Prometheus alerting,
and currently serve only to cause always-failing Nagios checks, which
normalizes alert failures.

Leave the files installed if they currently exist, rather than
cluttering puppet with `ensure => absent`; they do no harm if they are
left installed.
2021-08-03 16:07:13 -07:00
Tim Abbott 454a900487 docs: Replace chat-zulip-org page with /developer-community/. 2021-07-25 22:17:57 -07:00
Mateusz Mandera 388932bcc4 docs: Move docs on SOCIAL_AUTH_OIDC_FULL_NAME_VALIDATED to ReadTheDocs. 2021-07-24 15:16:13 -07:00
Mateusz Mandera 9bf84a35ef docs: Mention auto_signup OIDC option in ReadTheDocs documentation. 2021-07-24 15:13:47 -07:00
Mateusz Mandera 0d6b1cd08b docs: Tweak phrasing about OIDC supporting only a single provider.
Strictly speaking, this sentence is talking about the IdP configuration,
while the backend is just GenericOpenIdConnectBackend, so the new
phrasing is more correct.
2021-07-24 15:12:06 -07:00
Alex Vandiver 91282ab490 reindex-textual-data: Provide a tool to reindex all text indices.
The script is added to upgrade steps for 20.04 and Buster because
those are the upgrades that cross glibc 2.28, which is most
problematic.  It will also be called out in the upgrade notes, to
catch those that have already done that upgrade.
2021-07-19 16:34:23 -07:00
Alex Vandiver 528e5adaab smokescreen: Default to only listening on 127.0.0.1.
This prevents Smokescreen from acting as an open proxy.

Fixes #19214.
2021-07-14 15:40:26 -07:00
Anders Kaseorg 47897c76a2 scripts: Use curl -f (--fail).
This makes curl exit with nonzero status on HTTP 4xx/5xx errors.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-07-13 16:47:49 -07:00
Anders Kaseorg 77169408f4 docs: Remove do-release-upgrade -d option.
It has not been necessary since Ubuntu 20.04.1 was released in August.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-07-13 11:02:44 -07:00
Anders Kaseorg 93f62b999e nagios: Replace check_website_response with standard check_http plugin.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-07-09 16:47:03 -07:00
Mateusz Mandera 85cbdc8904 rate_limit: Add rate limiting of ZulipRemoteServer. 2021-07-08 15:55:02 -07:00
Mateusz Mandera 3b4f8cc85b saml: Add setting to skip the "continue to registration" page.
It's a smoother Just-In-Time provisioning process to allow
creating the account and getting signed in on the first login by the
user.
2021-07-08 15:21:20 -07:00
Anders Kaseorg ee8724e436 docs: Correct Big Blue Button to BigBlueButton.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-07-05 16:59:00 -07:00
Anders Kaseorg 91bfebca7d install: Replace wget with curl.
curl uses Happy Eyeballs to avoid long timeouts on systems with broken
IPv6.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-06-25 09:05:07 -07:00
Alex Vandiver 108e7ee8af docs: Update example supervisorctl status. 2021-06-14 17:12:59 -07:00
Mateusz Mandera c54b48452d saml: Support syncing custom profile fields with SAML attributes.
Fixes #17277.

The main limitation of this implementation is that the sync happens if
the user authing already exists. This means that a new user going
through the sign up flow will not have their custom fields synced upon
finishing it. The fields will get synced on their consecutive log in via
SAML in the future. This can be addressed in the future by moving the
syncing code further down the codepaths to login_or_register_remote_user
and plumbing the data through to the user creation process.

We detail that limitation in the documentation.
2021-06-08 14:47:33 -07:00
Tim Abbott 16700fc82c docs: Remove inaccurate LDAP coming soon notice.
We implemented this feature in the series ending with the
documentation in 2abe325147.
2021-06-08 14:29:47 -07:00
Alex Vandiver 161645c83c docs: Link to install-existing-server from production/deployment. 2021-06-02 17:16:14 -07:00
Alex Vandiver 03dd44e0ad docs: Remove link to 16.04, which can be confusing.
The instructions do not just apply to 16.04; the block below describes
the settings, which are correct for all relevant Ubuntu versions.
2021-06-02 16:58:24 -07:00
Alex Vandiver 1771f61f96 docs: Add a missing close paren. 2021-06-01 14:57:42 -07:00
Alex Vandiver 663d3b780d docs: Update path to nginx.conf, as it is now a template.
Also provide the right expansion for the one embedded variable
currently in the template.
2021-06-01 14:57:42 -07:00
Mateusz Mandera e1b8315f5e docs: Give an example command for generating SAML key/cert pair. 2021-06-01 08:29:05 -07:00
Mateusz Mandera b31ef13cf1 docs: Mention that SAML private key cannot be encrypted.
The underlying xmlsec library doesn't give a very useful error when
trying to load an encrypted private key which can lead to confusion.
2021-06-01 08:29:05 -07:00
Mateusz Mandera e17758f8ad auth: Implement a generic OpenID Connect backend.
Fixes #11939.
2021-05-28 09:43:06 -07:00
Tim Abbott 8960e64443 docs: Provide more detailed instructions for requesting API key.
Some users had trouble getting an API key that was resolved by doing
this.

Fixes #18524.
2021-05-27 12:09:40 -07:00
Adam Birds 3c28fa033d docs: Update documentation for new database user and dbname flags.
Updated the install documentation to include the explanation of the
two new install options `--postgresql-database-name` and
`--postgresql-database-user`.
2021-05-25 13:56:05 -07:00
Aman Agrawal d1c385f425 docs: Clarify how to set GIPHY API key in settings. 2021-05-18 17:32:55 -07:00
Anders Kaseorg e015f3ed7d docs: Correct “webapp” to “web app”.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-05-14 13:15:38 -07:00
Anders Kaseorg e3c570401e docs: Document password hashing change from PBKDF2 to Argon2.
This changed in commit 483a351d44
(#3410).

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-05-14 13:07:51 -07:00
Alex Vandiver 653c67a2b6 docs: Explicitly list possible outgoing HTTP connections. 2021-05-13 11:06:30 -07:00
Tim Abbott 6777660e02 deployment: Document smokescreen default and configuration. 2021-05-13 11:05:47 -07:00
Alex Vandiver a9ced1d228 smokescreen: Capitalize Smokescreen in documentation. 2021-05-12 12:32:39 -07:00
Matthew Wegner c43f883699
docs: Fix typo in export docs.
"as part of a high evailability environment" => "as part of a high availability environment".
2021-05-10 17:33:21 -07:00
Mateusz Mandera 8261f7e801 commands: Add delete_user management command and document it. 2021-05-10 15:38:14 -07:00
Siddharth Asthana 6ee7d5c983 docs: Improve documentation for changing a realm's subdomain.
Add a single line for self-hosting case to point to the ReadTheDocs
section of management command to change the realm's subdomain.

Fixes: #17857
2021-05-10 12:33:43 -07:00
Anders Kaseorg 544bbd5398 docs: Fix capitalization mistakes.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-05-10 09:57:26 -07:00
Tim Abbott 35a5cd1e65 docs: Add new summary section to the release lifecycle page. 2021-05-05 18:24:17 -07:00
Alya Abbott 279f7b1c8d user docs: Create separate "Code blocks" user documentation page.
* Move the extended documentation of code blocks to a separate page.
* Merge "code playgrounds" documentation to be a section of that page.
* Document copy widget on code blocks.
* This commit changes how we refer to "```python" type syntax for code
  blocks. Instead of being called a syntax highlighting label, this is
  now referred to as a "language tag", since it serves both syntax
  highlighting and playgrounds.
* Remap all the links.
* Advertise this new page in various places that previously did not have a link.
2021-05-05 16:11:21 -07:00
Wesley Aptekar-Cassels 6b7a3fb74a markdown: Rewrite all external images to use Camo.
Requesting external images is a privacy risk, so route all external
images through Camo.

Tweaked by tabbott for better test coverage, more comments, and to fix
bugs.
2021-04-30 10:36:16 -07:00
Tim Abbott 03e155f38a docs: Add production documentation for custom management commands. 2021-04-29 12:36:13 -07:00
strifel 02c5676a67 ldap: Add advanced LDAP realm access control.
This allows access to be more configurable than just setting one
attribute.  This can be configured by setting the setting
AUTH_LDAP_ADVANCED_REALM_ACCESS_CONTROL.
2021-04-28 09:08:13 -07:00
Aman Agrawal f71b591f95 Revert "docs: Temporarily remove Giphy help center link."
This reverts commit a94b9cc9e2.
2021-04-28 07:07:36 -07:00
Alex Vandiver 4c88da8ed9 scripts: Tool to find the diff to an original settings.py prod template.
This hits the unauthenticated Github API to get the list of tags,
which is rate-limited to 60 requests per hour.  This means that the
tool can only be run 60 times per hour before it starts to exit with
errors, but that seems like a reasonable limit for the moment.
2021-04-27 21:50:33 -07:00
Adam Birds ceb4b239c4 docs: Update install docs to document the install options correctly.
Update `docs/production/install.md` and
`docs/production/deployment.md` to document the install flags that can
be used as part of the installer more clearly.

Fixes #18122.
2021-04-27 19:44:05 -07:00
Alex Vandiver bc9d20eaa2 docs: Describe the process of upgrading settings.py files.
Fixes #17782.
2021-04-27 18:01:28 -07:00
Alex Vandiver ae2c377d13 postgresql: Switch to defaulting to PostgreSQL 13. 2021-04-27 16:55:04 -07:00
Tim Abbott a615dcb83e docs: Fix double-period typo in settings.md. 2021-04-26 17:07:41 -07:00
Anders Kaseorg 6060d0d364 docs: Add missing space to compound verbs “log in”, “set up”, etc.
Noun: backup, checkout, cleanup, login, logout, setup, shutdown, signup,
timeout.

Verb: back up, check out, clean up, log in, log out, set up, shut
down, sign up, time out.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-04-26 09:31:08 -07:00
Mateusz Mandera 1b5a7a2337 docs: Clarify what the certificate is supposed to be in the saml config.
Closes #17955.

Users were sometimes confused about what this certificate is supposed to
be.
2021-04-23 15:16:58 -07:00
Alex Vandiver 476524c0c1 scripts: Add a script to stop the server.
Using `supervisorctl stop all` to stop the server is not terribly
discoverable, and may stop services which are not part of Zulip
proper.

Add an explicit tool which only stops the relevant services.  It also
more carefully controls the order in which services are stopped to
minimize lost requests, and maximally quiesce the server.

Locations which may be stopping _older_ versions of Zulip (without
this script) are left with using `supervisorctl stop all`.

Fixes #14959.
2021-04-21 10:24:08 -07:00
Mateusz Mandera 977a2f7fa0 emails: Rename "missed message email" to "message notification email". 2021-04-21 10:10:54 -07:00
Jennifer Hwang c9f5946239 puppet: Add override for queue_workers_multiprocess.
With tweaks to the documentation by tabbott.

This uses the following configuration option:

[application_server]
queue_workers_multiprocess = false
2021-04-20 14:37:15 -07:00
Tim Abbott f3abe73e72 docs: Use our Digital Ocean marketplace referral code. 2021-04-20 10:45:26 -07:00
Alex Vandiver 7264d44dd4 docs: Minor grammar tweak. 2021-04-16 14:05:42 -07:00
Alex Vandiver e683ae2d24 docs: Add a missing word. 2021-04-16 14:05:42 -07:00
Tim Abbott d58931e2b5 docs: Document situations where one needs smokescreen. 2021-04-16 14:05:42 -07:00
Tim Abbott a94b9cc9e2 docs: Temporarily remove Giphy help center link.
This will fix CI; we can restore the link after Giphy is on zulip.com.
2021-04-05 15:44:20 -07:00
Aman Agrawal 5e83965e80 giphy: Use GIPHY web SDK to allow inserting GIFs in compose box.
We use GIPHY web SDK to create popover containing GIFs in a
grid format. Simply clicking on the GIFs will insert the GIF in the compose
box.

We add GIPHY logo to compose box action icons which opens the GIPHY
picker popover containing GIFs with "Powered by GIPHY"
attribution.
2021-04-05 15:04:49 -07:00
Mateusz Mandera 82b43a8cfe streams: Use "archive stream" phrasing instead of "delete stream".
The previous phrasing was misleading in relation to what the action
actually does.
2021-04-02 22:06:48 -07:00
Tim Abbott 68be06deda docs: Clean up installer troubleshooting help guide.
* Remove references to the defunct zulip-help Google Group.
* Mention how to create a code block.
* Use nicer markdown syntax for links.
2021-04-01 12:10:30 -07:00
Alex Vandiver de46edf966 docs: Remove unused X-Client-IP header from haproxy configuration.
This is unrelated to the X-Real-IP configuration that nginx will add,
and is unused by Zulip.  Remove it, to reduce confusion.
2021-03-31 14:19:38 -07:00
Strifel 209c89be10 ldap: Add option to limit user access to certain realms.
This adds an option for restricting a ldap user
to only be allowed to login into certain realms.
This is done by configuring an attribute mapping of "org_membership"
to an ldap attribute that will contain the list of subdomains the ldap
user is allowed to access. This is analogous to how it's done in SAML.

Co-authored-by: Mateusz Mandera <mateusz.mandera@zulip.com>
2021-03-18 11:19:59 -07:00
Anders Kaseorg 129ea6dd11 nginx: Consistently listen on IPv6 and with HTTP/2.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-03-17 17:46:32 -07:00
Nikhil Maske 6b34ba048d docs: Add a note in Incoming email integration docs.
The note states the incoming emails are rate-limited and
its current limits.

Fixes #17435.
2021-03-08 12:23:10 -08:00
Alex Vandiver d938dd9d4a puppet: Document smokescreen installation, and move to puppet/zulip/.
This is more broadly useful than for just Kandra; provide
documentation and means to install Smokescreen for stand-alone
servers, and motivate its use somewhat more.
2021-03-02 17:16:38 -08:00
aryanshridhar 5aa1205d23 docs: Added reactivate_realm management command. 2021-02-23 16:25:52 -08:00
aryanshridhar 42c49e4fb1 docs: Added missing semicolon in export_single_user management command. 2021-02-23 16:25:52 -08:00
Alex Vandiver 0b736ef4cf puppet: Remove puppet_ops configuration for separate loadbalancer host. 2021-02-22 16:05:13 -08:00
Tim Abbott 04197309ae docs: Explain the export tool purpose more clearly.
The reference to "hardware platforms" was confusing -- we really mean
different logical Zulip database installations.
2021-02-19 10:05:40 -08:00
Tim Abbott 150a07cdcd docs: Add a performance design document.
This document is mainly an answer to a set of questions other
developers have been asking about Zulip's architecture and scalability
design.  It's intended for developers working on Zulip, to help with
thinking about where to prioritize further efforts to optimize
scalability.
2021-02-13 13:05:47 -08:00
Nikhil Maske 89b25556dc prod_settings_template: Move LDAP up into authentication, and merge part 1/2. 2021-02-05 17:02:27 -08:00
rht 16583fd5b4 docs: Say that certbot wildcard has been released. 2021-01-17 11:17:09 -08:00
Aman Agrawal c685d36821 hipchat_import: Remove tool from codebase.
Remove functions and scripts used by HipChat import tool and
those which will no longer be required in future.
2020-12-23 08:28:49 -08:00
Aman Agrawal 62d721e859 docs: Remove HipChat migration guide.
As of Feb 15th 2019, Hipchat Cloud and Stride
have reached End Of Life and are no longer
supported by Atlassian. Since it is almost 2 years
now we can remove the migration guides.
2020-12-23 15:43:13 +05:30
Mateusz Mandera d0dc04a093 models: Rename is_api_super_user to can_forge_sender, 2020-12-21 13:15:39 -08:00
Ben Steinberg f00dba3c81 docs: Fix repeated word. 2020-12-02 11:54:49 -08:00
Tim Abbott 067cd3a97a docs: Remove incorrect references to chat.zulip.org.
Most of these are Help Center links that should be pointing to the
production Help Center.
2020-10-29 16:46:40 -07:00
ryanreh99 dfa7ce5637 uploads: Support non-AWS S3-compatible server.
Boto3 does not allow setting the endpoint url from
the config file. Thus we create a django setting
variable (`S3_ENDPOINT_URL`) which is passed to
service clients and resources of `boto3.Session`.

We also update the uploads-backend documentation
and remove the config environment variable as now
AWS supports the SIGv4 signature format by default.
And the region name is passed as a parameter instead
of creating a config file for just this value.

Fixes #16246.
2020-10-28 21:59:07 -07:00
Alex Vandiver 6b9d7000b5 puppet: Set proxy environment variables.
These are respected by `urllib`, and thus also `requests`.  We set
`HTTP_proxy`, not `HTTP_PROXY`, because the latter is ignored in
situations which might be running under CGI -- in such cases it may be
coming from the `Proxy:` header in the request.
2020-10-28 12:17:35 -07:00
Alex Vandiver 524ffe6acc docs: Rename postgres.md to postgresql.md. 2020-10-28 11:57:03 -07:00
Alex Vandiver 2b0bbbb882 tools: Rename postgres to postgresql in tool names. 2020-10-28 11:57:02 -07:00
Alex Vandiver 5eb8064a1a install: Rename postgres options to postgresql. 2020-10-28 11:55:32 -07:00
Alex Vandiver 1f7132f50d docs: Standardize on PostgreSQL, not Postgres. 2020-10-28 11:55:16 -07:00
Alex Vandiver eaa99359b1 puppet: Rename to check_postgresql_replication_lag. 2020-10-28 11:51:52 -07:00
Alex Vandiver 53e59a0a13 puppet: Rename check_postgres_backup to check_postgresql_backup. 2020-10-28 11:51:52 -07:00
Tim Abbott 3a4b38d732 docs: Clarify discussion of no_serve_uploads. 2020-10-27 13:39:14 -07:00
Alex Vandiver a155430eb5 docs: Document all zulip.conf settings.
This provides a single reference point for all zulip.conf settings;
these mostly link out to the more complete documentation about each
setting, elsewhere.

Fixes #12490.
2020-10-27 13:31:57 -07:00
Alex Vandiver d24c571bab puppet: Automatically back up the database if we have the secrets.
This avoids folks having to manually add to the puppet_classes.
2020-10-27 13:29:19 -07:00
Alex Vandiver e7798d2797 puppet: Move zulip_ops::profile::postgres_appdb to postgresql. 2020-10-27 13:29:19 -07:00
Alex Vandiver 9f25389bff puppet: Move top-level zulip_ops deployments to zulip_ops::profile. 2020-10-27 13:29:19 -07:00
Alex Vandiver 188af57296 puppet: Rename postgres_appdb to postgresql.
There is only one PostgreSQL database; the "appdb" is irrelevant.
Also use "postgresql," as it is the name of the software, whereas
"postgres" the name of the binary and colloquial name.  This is minor
cleanup, but enabled by the other renames in the previous commit.
2020-10-27 13:29:19 -07:00
Alex Vandiver 0f25acc7b3 puppet: Rename "voyager"/"dockervoyager" to "standalone"/"docker".
The "voyager" name is non-intuitive and not significant.
`zulip::voyager` and `zulip::dockervoyager` stubs are kept for
back-compatibility with existing `zulip.conf` files.
2020-10-27 13:29:19 -07:00
Alex Vandiver c2185a81d6 puppet: Move top-level zulip deployments into "profile" directory.
This moves the puppet configuration closer to the "roles and profiles
method"[1] which is suggested for organizing puppet classes.  Notably,
here it makes clear which classes are meant to be able to stand alone
as deployments.

Shims are left behind at the previous names, for compatibility with
existing `zulip.conf` files when upgrading.

[1] https://puppet.com/docs/pe/2019.8/the_roles_and_profiles_method
2020-10-27 13:29:19 -07:00
sahil839 a96811ab58 management: Replace knight command with change_user_role command.
We replace knight command with change_user_role command which
allows us to change role of a user to owner, admins, member and
guest. We can also give/revoke api_super_user permission using
this command.

Tweaked by tabbott to improve the logging output and update documentation.

Fixes #16586.
2020-10-25 14:55:08 -07:00
Alex Vandiver c296b5d819 puppet: Allow unattended-upgrades for all but servers.
Restarting servers is what can cause service interruptions, and
increase risk.  Add all of the servers that we use to the list of
ignored packages, and uncomment the default allowed-origins in order
to enable unattended upgrades.
2020-10-23 16:46:06 -07:00
Anders Kaseorg 72d6ff3c3b docs: Fix more capitalization issues.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-10-23 11:46:55 -07:00
Alex Vandiver 50e9e2ed20 puppet: Make zulip::base include zulip::apt_repository.
There was likely more dependency complexity prior to 97766102df, but
there is now no reason to require that consumers explicitly include
zulip::apt_repository.
2020-10-22 11:30:53 -07:00
Alex Vandiver 84142df152 docs: Drop an out-of-date paragraph that talks of 2017 in the future. 2020-10-19 16:28:22 -07:00
Tim Abbott 54dd612f5c docs: Update list of files new since last release.
This fixes CI failing due to our not having released the /video-calls.md rename.
2020-10-18 17:16:31 -07:00