This document is mainly an answer to a set of questions other
developers have been asking about Zulip's architecture and scalability
design. It's intended for developers working on Zulip, to help with
thinking about where to prioritize further efforts to optimize
scalability.
As of Feb 15th 2019, Hipchat Cloud and Stride
have reached End Of Life and are no longer
supported by Atlassian. Since it is almost 2 years
now we can remove the migration guides.
Boto3 does not allow setting the endpoint url from
the config file. Thus we create a django setting
variable (`S3_ENDPOINT_URL`) which is passed to
service clients and resources of `boto3.Session`.
We also update the uploads-backend documentation
and remove the config environment variable as now
AWS supports the SIGv4 signature format by default.
And the region name is passed as a parameter instead
of creating a config file for just this value.
Fixes#16246.
These are respected by `urllib`, and thus also `requests`. We set
`HTTP_proxy`, not `HTTP_PROXY`, because the latter is ignored in
situations which might be running under CGI -- in such cases it may be
coming from the `Proxy:` header in the request.
This provides a single reference point for all zulip.conf settings;
these mostly link out to the more complete documentation about each
setting, elsewhere.
Fixes#12490.
There is only one PostgreSQL database; the "appdb" is irrelevant.
Also use "postgresql," as it is the name of the software, whereas
"postgres" the name of the binary and colloquial name. This is minor
cleanup, but enabled by the other renames in the previous commit.
The "voyager" name is non-intuitive and not significant.
`zulip::voyager` and `zulip::dockervoyager` stubs are kept for
back-compatibility with existing `zulip.conf` files.
This moves the puppet configuration closer to the "roles and profiles
method"[1] which is suggested for organizing puppet classes. Notably,
here it makes clear which classes are meant to be able to stand alone
as deployments.
Shims are left behind at the previous names, for compatibility with
existing `zulip.conf` files when upgrading.
[1] https://puppet.com/docs/pe/2019.8/the_roles_and_profiles_method
We replace knight command with change_user_role command which
allows us to change role of a user to owner, admins, member and
guest. We can also give/revoke api_super_user permission using
this command.
Tweaked by tabbott to improve the logging output and update documentation.
Fixes#16586.
Restarting servers is what can cause service interruptions, and
increase risk. Add all of the servers that we use to the list of
ignored packages, and uncomment the default allowed-origins in order
to enable unattended upgrades.
There was likely more dependency complexity prior to 97766102df, but
there is now no reason to require that consumers explicitly include
zulip::apt_repository.
`zproject/settings.py` itself is mostly-empty now. Adjust the
references which should now point to `zproject/computed_settings.py`
or `zproject/default_settings.py`.
The OS upgrade paths which go through 2.1 do not call
`upgrade-zulip-stage-2` with `--audit-fts-indexes` because that flag
was added in 3.0.
Add an explicit step to do this audit after the 3.0 upgrade. Stating
it as another command to run, rather than attempting to tell them
to add it to the `upgrade-zulip` call that we're linking to seems
easiest, since that does not dictate if they should upgrade to a
release or from the tip of git.
We do not include a step describing this for the Trusty -> Xenial
upgrade, because the last step already chains into Xenial -> Bionic,
which itself describes auditing the indexes.
Fixes#15877.
Only Zulip 3.0 and above support the `--audit-fts-indexes` option to
`upgrade-zulip-stage-2`; saying "same as Bionic to Focal" on other
other steps, which are for Zulip 2.1 or 2.0, will result in errors.
Provide the full text of the updated `upgrade-zulip-stage-2` call in
step 5 for all non-3.0 upgrades. For Trusty to Xenial and Stretch to
Buster, we do not say "Same as Xenial to Bionic" , because it is
likely that readers do not notice that step does not read "Same as
Bionic to Focal."
The apple developer webapp consistently refers this App ID. So,
this clears any confusion that can occur.
Since python social auth only requires us to include App ID in
_AUDIENCE(a list), we do that in computed settings making it easier for
server admin and we make it much clear by having it set to
APP_ID instead of BUNDLE_ID.
wal-g was used in `puppet/zulip` by env-wal-g, but only installed in
`puppet/zulip_ops`.
Merge all of the dependencies of doing backups using wal-g (wal-g
installation, the pg_backup_and_purge job, the nagios plugin that
verifies it happens) into a common base class in `puppet/zulip`, since
it is generally useful.
Running `pg-upgradecluster` runs the `CREATE TEXT SEARCH DICTIONARY`
and `CREATE TEXT SEARCH CONFIGURATION` from
`zerver/migrations/0001_initial.py` on the new PostgreSQL cluster;
this requires that the stopwords file and dictionary exist _prior_
to `pg_upgradecluster` being run.
This causes a minor dependency conflict -- we do not wish to duplicate
the functionality from `zulip::postgres_appdb_base` which configures
those files, but installing all of `zulip::postgres_appdb_tuned` will
attempt to restart PostgreSQL -- which has not configured the cluster
for the new version yet.
In order to split out configuration of the prerequisites for the
application database, and the steps required to run it, we need to be
able to apply only part of the puppet configuration. Use the
newly-added `--config` argument to provide a more limited `zulip.conf`
which only applies `zulip::postgres_appdb_base` to the new version of
Postgres, creating the required tsearch data files.
This also preserves the property that a failure at any point prior to
the `pg_upgradecluster` is easily recoverable, by re-running
`zulip-puppet-apply`.
Added -d Flag in do-release-upgrade for Bionic to Focal upgrade.
The -d switch is necessary to upgrade from Ubuntu 18.04 LTS
as upgrades have not yet been enabled and will only be enabled
after the first point release of 20.04 LTS.
Source https://wiki.ubuntu.com/FocalFossa/ReleaseNotes
As alluded to in the previous commit, only 3.0 can use the new tooling
-- indeed, it requires it, as the zulip.conf entry must be changed.
Clarify that in the upgrade steps for earlier distributions.
Update the upgrade documentation for the tool added in the previous
step. Only the Bionic -> Focal upgrade step need be updated, because
none of the other upgrade steps can be run starting from a Zulip 3.0
installation.
Fixes#15415.
After some discussion, everyone seems to agree that 3.0 is the more
appropriate version number for our next major release. This updates
our documentation to reflect that we'll be using 3.0 as our next major
release.
49a7a66004 and immediately previous commits began installing
PostgreSQL 12 from their apt repository. On machines which already
have the distribution-provided version of PostgreSQL installed,
however, this leads to failure to apply puppet when restarting
PostgreSQL 12, as both attempt to claim the same port.
During installation, if we will be installing PostgreSQL, look for
other versions than what we will install, and abort if they are
found. This is safer than attempting to automatically uninstall or
reconfigure existing databases.
The previous commit removed the only behavior difference between the
two flags; both of them skip user/database creation, and the tables
therein.
Of the two options `--no-init-db` is more explicit as to what it does,
as opposed to just one facet of when it might be used; remove
`--remote-postgres`.
In particular the Services ID and Bundle ID each have one of Apple's
random-looking 10-character identifiers, in addition to the Java-style
names the admin chooses. Best to be clear about what names are
supposed to be the chosen names and which are supposed to be the
random-looking assigned names.
(I don't know of any docs elsewhere making this clear -- but I guessed
it'd be this way, and empirically it works.)
Also mention you need to enable the backend. :-)
I believe the Bundle ID (aka App ID) and Services ID have meaning only
relative to a specific Team ID. In particular, in some places in the
developer.apple.com UI, they're displayed in a fully-qualified form
like "ABCDE12345.com.example.app", where "com.example.app" is the
App ID or Services ID and ABCDE12345 is the Team ID.
Adds the ability to set a SAML attribute which contains a
list of subdomains the user is allowed to access. This allows a Zulip
server with multiple organizations to filter using SAML attributes
which organization each user can access.
Cleaned up and adapted by Mateusz Mandera to fit our conventions and
needs more.
Co-authored-by: Mateusz Mandera <mateusz.mandera@zulip.com>
This adds a convenient way to review the upgrade notes for all Zulip
releases that one is upgrading across.
I thought about moving all the upgrade notes to a common section, but
in some cases the language is clearly explaining changes in the
release that are not duplicated elsewhere, and I think it reads better
having them inline alongisde related changes.
• Specify disabled rather than enabled protocols, so as not to disable
TLS 1.3.
• Provide an explicit cipher suite list (Mozilla intermediate config
version 5.4).
• Respect the browser’s preferred cipher suite ordering over the
server’s.
• Use FFDHE2048 Diffie-Hellman parameters.
• Disable SSL session tickets.
(SSL stapling is also recommended but SSLStaplingCache cannot be
configured inside a <VirtualHost> block.)
Signed-off-by: Anders Kaseorg <anders@zulip.com>
This implementation overrides some of PSA's internal backend
functions to handle `state` value with redis as the standard
way doesn't work because of apple sending required details
in the form of POST request.
Includes a mixin test class that'll be useful for testing
Native auth flow.
Thanks to Mateusz Mandera for the idea of using redis and
other important work on this.
Documentation rewritten by tabbott.
Co-authored-by: Mateusz Mandera <mateusz.mandera@zulip.com>
Now that we have production support for Ubuntu Focal, we update the
documentation to state our support for it.
(We also drop deprecated Xenial and Stretch from supported platforms).
We're migrating to using the cleaner zulip.com domain, which involves
changing all of our links from ReadTheDocs and other places to point
to the cleaner URL.
This reimplements our Zoom video call integration to use an OAuth
application. In addition to providing a cleaner setup experience,
especially on zulipchat.com where the server administrators can have
done the app registration already, it also fixes the limitation of the
previous integration that it could only have one call active at a time
when set up with typical Zoom API keys.
Fixes#11672.
Co-authored-by: Marco Burstein <marco@marco.how>
Co-authored-by: Tim Abbott <tabbott@zulipchat.com>
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
This section at the top was clearly written before the documentation
at the bottom existed, and hasn't been updated to point to the
now-existent docs below.
Add the link, rather than directing to #production-help.
Also make sure our documentation for upgrading is reasonable for
Stretch => Buster.
Our reasoning for deprecating support for these releases is as follows:
* Ubuntu 16.04 Xenial reached desktop EOL last year; and will reach
EOL on the server in about a year.
* Debian Stretch will each EOL in 2020 (the precise date is unclear in
Debian's documentation, but based on past precedent it's in the next
few months, perhaps July 2020).
https://wiki.debian.org/DebianReleases#Production_Releases
* Both Ubuntu 16.04 and Debian Stretch use Python 3.5 as the system
Python, which will reach EOL in September 2020 (and we're already
seeing various third-party dependencies that we use drop support for
them).
* While there is LTS support for these older releases, it's not clear it's
going to be worth the added engineering effort for us to maintain EOL
releases of the base OSes that we support.
* We (now) have clear upgrade instructions for moving to Debian Buster
and Ubuntu 18.04.
I found the solution by simply trying out EMAIL_USE_SSL and it
worked. I had problems with sending emails (did not work at all, there
wasn't even a connection going on - I checked with tcpdump. Then I
found this: To use port 465, you need to call
smtplib.SMTP_SSL(). Currently, it looks like Django only uses
smtplib.SMTP() (source: https://code.djangoproject.com/ticket/9575).
Fixes#14350.
Alex Vandiver
Tim Abbott
Nikhil Maske
rht
Aman Agrawal
Mateusz Mandera
Ben Steinberg
ryanreh99
sahil839
Anders Kaseorg
strifel
Brian Lee
Dinesh
arpit551
Greg Price
Brainrecursion
qnxor
Anders Kaseorg
Mateusz Mandera
Alex Vandiver
Steve Howell
Tim Abbott
Vishnu Ks
thoemie