This is more broadly useful than for just Kandra; provide
documentation and means to install Smokescreen for stand-alone
servers, and motivate its use somewhat more.
This commit adds about Remote - SSH extension (in VS Code), which
helps us develop remotely by providing a similar interface as if we
are developing locally. We also simplify the documentation for RMate
to use the new standards.
Tweaked by tabbott for simplification/formatting.
Go to Recent Topics on "#", no hash and "#recent_topics".
Go to Recent Topics as the last destination for escape key.
Map `a` key to All messages and change its hash to
`#all_messages`.
These aren't relevant to most users, but in the interest of
transparency, we also don't want the existence of these to feel like a
secret. And maybe publishing their existence will result in folks who
we forget to add to these private streams asking about them.
This document is mainly an answer to a set of questions other
developers have been asking about Zulip's architecture and scalability
design. It's intended for developers working on Zulip, to help with
thinking about where to prioritize further efforts to optimize
scalability.
This commit adds the Zulip logo to the readthedocs
documentation by adding an .svg file (taken from
zulip.com) to the `docs/images` directory.
Morever, it removes the name `Zulip` which was
written at the top of the sidebar because the
logo already has that.
Also fix incorrect 2011 copyright years; the original Zulip, Inc. that
was merged into Dropbox, Inc. was incorporated in 2012, not 2011.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
We now require keywords, so that there is no
pitfall for mixing up boolean parameters.
Positional parameters are basically evil
when you have a bunch of bools.
I also make user_profile the first argument.
Finally, the code is more diff-friendly.
Support for saving it in the session is dropped in django3, the cookie
is the mechanism that needs to be used. The relevant i18n code doesn't
have access to the response objects and thus needs to delegate setting
the cookie to LocaleMiddleware.
Fixes the LocaleMiddleware point of #16030.
As of Feb 15th 2019, Hipchat Cloud and Stride
have reached End Of Life and are no longer
supported by Atlassian. Since it is almost 2 years
now we can remove the migration guides.
It seems the Ubuntu base image we use now has a new enough VirtualBox
Guest Additions to trigger the ETXTBSY bug even when it’s not upgraded
by the vagrant-vbguest plugin. Provide and document a way to
downgrade it.
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
We seem to periodically get contributors who rebase upstream commits
onto their pull request rather than the other way around, resulting in
a lot of GitHub noise. The PRs where this happens were made from
branches named master. We have always documented that you should work
on a feature branch, but not from this page; maybe this will help
reduce that kind of confusion.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
We add navigating to user home inside WSL virtual disk as another
step as many users clone Zulip inside a mounted windows disk and
run into permission issues when running provision.
Steve asked me to remove this, since the tictactoe game was always
intended as a proof of concept. Now that we have poll and todo
widgets, the sample code for tictactoe has much less value.
We replace the content and type in test_widgets.py to maintain
coverage.
Boto3 does not allow setting the endpoint url from
the config file. Thus we create a django setting
variable (`S3_ENDPOINT_URL`) which is passed to
service clients and resources of `boto3.Session`.
We also update the uploads-backend documentation
and remove the config environment variable as now
AWS supports the SIGv4 signature format by default.
And the region name is passed as a parameter instead
of creating a config file for just this value.
Fixes#16246.
These are respected by `urllib`, and thus also `requests`. We set
`HTTP_proxy`, not `HTTP_PROXY`, because the latter is ignored in
situations which might be running under CGI -- in such cases it may be
coming from the `Proxy:` header in the request.
This provides a single reference point for all zulip.conf settings;
these mostly link out to the more complete documentation about each
setting, elsewhere.
Fixes#12490.
There is only one PostgreSQL database; the "appdb" is irrelevant.
Also use "postgresql," as it is the name of the software, whereas
"postgres" the name of the binary and colloquial name. This is minor
cleanup, but enabled by the other renames in the previous commit.
The "voyager" name is non-intuitive and not significant.
`zulip::voyager` and `zulip::dockervoyager` stubs are kept for
back-compatibility with existing `zulip.conf` files.
This moves the puppet configuration closer to the "roles and profiles
method"[1] which is suggested for organizing puppet classes. Notably,
here it makes clear which classes are meant to be able to stand alone
as deployments.
Shims are left behind at the previous names, for compatibility with
existing `zulip.conf` files when upgrading.
[1] https://puppet.com/docs/pe/2019.8/the_roles_and_profiles_method
We replace knight command with change_user_role command which
allows us to change role of a user to owner, admins, member and
guest. We can also give/revoke api_super_user permission using
this command.
Tweaked by tabbott to improve the logging output and update documentation.
Fixes#16586.
Restarting servers is what can cause service interruptions, and
increase risk. Add all of the servers that we use to the list of
ignored packages, and uncomment the default allowed-origins in order
to enable unattended upgrades.
There was likely more dependency complexity prior to 97766102df, but
there is now no reason to require that consumers explicitly include
zulip::apt_repository.
In this commit I attempt to make the docs for
writing `test_events` a little more oriented
toward telling the developer that they need to
write a test in `test_events`; in other words,
the tone is a bit more of how-to-do-it vs.
explaining the innards.
So I now start with a concrete example.
Then much of the copy related to `verify_action`
is the same as before this commit, but I rework
it a bit to be a bit more step-by-step, and I refer
back to the concrete example.
Finally, I explain the schema checking step, which
is all new copy, although for that, I mostly
instruct the developer to read event_schema.py
itself.
css-loader@4 broke @import statements referencing files with
extensions other than .css, unless those @import statements are
compiled away by another loader. Upstream is more interested in
arguing that such @import statements are semantically incorrect than
applying the one line fix.
https://github.com/webpack-contrib/css-loader/issues/1164
Signed-off-by: Anders Kaseorg <anders@zulip.com>
This was a broken abstraction that returned to its caller within
multiple forked processes on exceptions, and encouraged ignoring the
error code (as all of its callers did).
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
We don't modify bootstrap.js here but override its popover and
tooltip plugins. In future we will not import these plugins
via npm. We also copy all the popover code from bootstrap.css v2.1.1
to popovers.scss since all the code in bootstrap-tooltip.js is
based upon this css or vice versa.
Update THIRDPARTY info about bootstrap libraries.
There were 4 types of changes to bootstrap.js - bugfixes, file
moves, changes to typeahead plugin and changes to tooltip +
popover plugin.
Bugfixes were automatically fixed when upgrading to v2.3.2, file
moves are irrelevant to this upgrade and the plugins were
extracted into separate files.
46e562f - POPOVER
8779e55 - POPOVER
66c6423 - POPOVER
21ccf45 - POPOVER
cb9b526 - TYPEAHEAD EXTRACTED
3079cf8 - TYPEAHEAD
9ea4f50 - TYPEAHEAD
b961093 - TYPEAHEAD
0e2c509 - TYPEAHEAD
28589c5 - TYPEAHEAD
70a14d8 - TYPEAHEAD
0c42e4a - TYPEAHEAD
213b8ce - FIXED IN 2.3.2
0bac986 - TYPEAHEAD
0e3332d - FIXED IN V2.3.2
eaa777b - TYPEAHEAD
f944a8e - TYPEAHEAD
546ae10 - TYPEAHEAD
3bba0cc - FILE MOVED
b8794e1 - TYPEAHEAD
6217c1a - TYPEAHEAD
dc85fa7 - TYPEAHEAD
d329317 - TYPEAHEAD
b3ef776 - TYPEAHEAD
fcb3999 - TYPEAHEAD
0975cfa - TYPEAHEAD
fbed3e2 - TYPEAHEAD
0fa857d - POPOVER
68b890a - TYPEAHEAD
b5cadec - typeahead
441e429 - copyright
22ce2c0 Fixed In v2.3.2
d78d761- typeahead
bff933e- typeahead
ef585cf- typeahead
7e35369 - typeahead
8f1cee0 - Files moving around
1490ae1 - add file
We merge bootstrap-responsive.css into bootsrap.css since that is
how bootstrap distributes it from this version onwards.
bootstrap.js has a lot of changes to it which completely breaks
our typeaheads and popovers, so we will have to override these
plugins with our version of these plugins. In future versions
of bootstrap when we use npm, we can just choose not to
import them.
This commit removes the part which mentions specifying the property field
type in new feauture tutorial as it is no longer required to specify the
type.
`zproject/settings.py` itself is mostly-empty now. Adjust the
references which should now point to `zproject/computed_settings.py`
or `zproject/default_settings.py`.
c2f9227892 switched from a long hard-coded list in `TEST_FUNCTIONS` to
using a new `@openapi_test_function` decorator; update the
documentation about writing API docs accordingly.
There are file sharing issues with the macOS 10.15.6 and
vagrant. var/remote_cache_prefix was an empty file when using
VirtualBox and Docker on macOS.
Using parallels as a provider for vagrant fixes the issue.
Use --watch-poll which makes webpack to recompile
automatically on file changes, since inotify is not
working here too.
The OS upgrade paths which go through 2.1 do not call
`upgrade-zulip-stage-2` with `--audit-fts-indexes` because that flag
was added in 3.0.
Add an explicit step to do this audit after the 3.0 upgrade. Stating
it as another command to run, rather than attempting to tell them
to add it to the `upgrade-zulip` call that we're linking to seems
easiest, since that does not dictate if they should upgrade to a
release or from the tip of git.
We do not include a step describing this for the Trusty -> Xenial
upgrade, because the last step already chains into Xenial -> Bionic,
which itself describes auditing the indexes.
Fixes#15877.
Only Zulip 3.0 and above support the `--audit-fts-indexes` option to
`upgrade-zulip-stage-2`; saying "same as Bionic to Focal" on other
other steps, which are for Zulip 2.1 or 2.0, will result in errors.
Provide the full text of the updated `upgrade-zulip-stage-2` call in
step 5 for all non-3.0 upgrades. For Trusty to Xenial and Stretch to
Buster, we do not say "Same as Xenial to Bionic" , because it is
likely that readers do not notice that step does not read "Same as
Bionic to Focal."
This section referenced the old Puppet path for
puppet/zulip/templates/nginx/upstreams.conf.template.erb, and overall
felt focused on naming files that developers never look at.
This commit adds EMAIL_PORT setting for explicitly specifying the
port of SMTP provider in dev_settings.py.
We also change email_backends.send_email_smtp to pass EMAIL_PORT
along with EMAIL_HOST to smtplib.SMTP.
After this change, we will not need to include the port along with
host in EMAIL_HOST.
Also updated the email.md docs accordingly for this change.
The apple developer webapp consistently refers this App ID. So,
this clears any confusion that can occur.
Since python social auth only requires us to include App ID in
_AUDIENCE(a list), we do that in computed settings making it easier for
server admin and we make it much clear by having it set to
APP_ID instead of BUNDLE_ID.
When you post to /json/users, we no longer
require or look at the short_name parameter,
since we don't use it in any meaningful way.
An upcoming commit will eliminate it from the
database.
wal-g was used in `puppet/zulip` by env-wal-g, but only installed in
`puppet/zulip_ops`.
Merge all of the dependencies of doing backups using wal-g (wal-g
installation, the pg_backup_and_purge job, the nagios plugin that
verifies it happens) into a common base class in `puppet/zulip`, since
it is generally useful.
Running `pg-upgradecluster` runs the `CREATE TEXT SEARCH DICTIONARY`
and `CREATE TEXT SEARCH CONFIGURATION` from
`zerver/migrations/0001_initial.py` on the new PostgreSQL cluster;
this requires that the stopwords file and dictionary exist _prior_
to `pg_upgradecluster` being run.
This causes a minor dependency conflict -- we do not wish to duplicate
the functionality from `zulip::postgres_appdb_base` which configures
those files, but installing all of `zulip::postgres_appdb_tuned` will
attempt to restart PostgreSQL -- which has not configured the cluster
for the new version yet.
In order to split out configuration of the prerequisites for the
application database, and the steps required to run it, we need to be
able to apply only part of the puppet configuration. Use the
newly-added `--config` argument to provide a more limited `zulip.conf`
which only applies `zulip::postgres_appdb_base` to the new version of
Postgres, creating the required tsearch data files.
This also preserves the property that a failure at any point prior to
the `pg_upgradecluster` is easily recoverable, by re-running
`zulip-puppet-apply`.
Log RealmAuditLog in do_set_realm_property and do_remove_realm_domain.
Tests for the changes are written in test_events because it will save
duplicate code for test_change_realm_property.
Added -d Flag in do-release-upgrade for Bionic to Focal upgrade.
The -d switch is necessary to upgrade from Ubuntu 18.04 LTS
as upgrades have not yet been enabled and will only be enabled
after the first point release of 20.04 LTS.
Source https://wiki.ubuntu.com/FocalFossa/ReleaseNotes
I also fix the code formatting so it's more
considerate of folks that have smaller monitors
or do side-by-side editing. And it's more
diff friendly as well.
Rename rest of function names, classes and comments containing bugdoown
to markdown in test_markdown.py. Also change the refactored classes and
functions occurences in other files.
This commit is part of series of commits aimed at renaming bugdown to
markdown.
Rename the file and all the refrences to file and module test_bugdown.py
to test_markdown.py.
This commit is part of series of commit that renames bugdown to markdown.
This commit is first of few commita which aim to change all the
bugdown references to markdown. This commits rename the files,
file path mentions and change the imports.
Variables and other references to bugdown will be renamed in susequent
commits.
As alluded to in the previous commit, only 3.0 can use the new tooling
-- indeed, it requires it, as the zulip.conf entry must be changed.
Clarify that in the upgrade steps for earlier distributions.
Update the upgrade documentation for the tool added in the previous
step. Only the Bionic -> Focal upgrade step need be updated, because
none of the other upgrade steps can be run starting from a Zulip 3.0
installation.
Fixes#15415.
After some discussion, everyone seems to agree that 3.0 is the more
appropriate version number for our next major release. This updates
our documentation to reflect that we'll be using 3.0 as our next major
release.
49a7a66004 and immediately previous commits began installing
PostgreSQL 12 from their apt repository. On machines which already
have the distribution-provided version of PostgreSQL installed,
however, this leads to failure to apply puppet when restarting
PostgreSQL 12, as both attempt to claim the same port.
During installation, if we will be installing PostgreSQL, look for
other versions than what we will install, and abort if they are
found. This is safer than attempting to automatically uninstall or
reconfigure existing databases.
The previous commit removed the only behavior difference between the
two flags; both of them skip user/database creation, and the tables
therein.
Of the two options `--no-init-db` is more explicit as to what it does,
as opposed to just one facet of when it might be used; remove
`--remote-postgres`.
In particular the Services ID and Bundle ID each have one of Apple's
random-looking 10-character identifiers, in addition to the Java-style
names the admin chooses. Best to be clear about what names are
supposed to be the chosen names and which are supposed to be the
random-looking assigned names.
(I don't know of any docs elsewhere making this clear -- but I guessed
it'd be this way, and empirically it works.)
Also mention you need to enable the backend. :-)
I believe the Bundle ID (aka App ID) and Services ID have meaning only
relative to a specific Team ID. In particular, in some places in the
developer.apple.com UI, they're displayed in a fully-qualified form
like "ABCDE12345.com.example.app", where "com.example.app" is the
App ID or Services ID and ABCDE12345 is the Team ID.
Adds the ability to set a SAML attribute which contains a
list of subdomains the user is allowed to access. This allows a Zulip
server with multiple organizations to filter using SAML attributes
which organization each user can access.
Cleaned up and adapted by Mateusz Mandera to fit our conventions and
needs more.
Co-authored-by: Mateusz Mandera <mateusz.mandera@zulip.com>
This adds a convenient way to review the upgrade notes for all Zulip
releases that one is upgrading across.
I thought about moving all the upgrade notes to a common section, but
in some cases the language is clearly explaining changes in the
release that are not duplicated elsewhere, and I think it reads better
having them inline alongisde related changes.
We only use this in a few places, but they're really important places
for understanding the types in the codebase, and so it's worth having
a bit of expository documentation explaining how we use it.
(And I expect we'll add more with time).
We have migrated ellipsis-v from chevron for sidebars,
it would be better to define it in glossary as it is gonna replace
"chevron" during conversations.
Replace word 'chevron' with 'ellipsis'(because of the
recent migration to ellipsis for sidebars menu icons)
and mention where to click more specifically for sidebar
popover testing.
65774e1c4f switched from using the bundled check_postgres.pl to using
the version from packages; the file itself remained, however.
Remove it, and clean up references to it.
Fixes#15389.
• Specify disabled rather than enabled protocols, so as not to disable
TLS 1.3.
• Provide an explicit cipher suite list (Mozilla intermediate config
version 5.4).
• Respect the browser’s preferred cipher suite ordering over the
server’s.
• Use FFDHE2048 Diffie-Hellman parameters.
• Disable SSL session tickets.
(SSL stapling is also recommended but SSLStaplingCache cannot be
configured inside a <VirtualHost> block.)
Signed-off-by: Anders Kaseorg <anders@zulip.com>
This is a prep commit for replacement of chevron
from sidebars.
This commit will add ellipsis-v icon in svg format downloaded
from font-awesome 5. This has to be done because font-awesome 4.7
(the version we are using) does not have this icon with
circular dots.
And font-awesome 5 as a whole doesn't make sense to upgrade to because
it's intentionally semi-crippled as part of their business plan.
Also include entry in THIRDYPARTY and Licence details.
Fixes#2665.
Regenerated by tabbott with `lint --fix` after a rebase and change in
parameters.
Note from tabbott: In a few cases, this converts technical debt in the
form of unsorted imports into different technical debt in the form of
our largest files having very long, ugly import sequences at the
start. I expect this change will increase pressure for us to split
those files, which isn't a bad thing.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
Automatically generated by the following script, based on the output
of lint with flake8-comma:
import re
import sys
last_filename = None
last_row = None
lines = []
for msg in sys.stdin:
m = re.match(
r"\x1b\[35mflake8 \|\x1b\[0m \x1b\[1;31m(.+):(\d+):(\d+): (\w+)", msg
)
if m:
filename, row_str, col_str, err = m.groups()
row, col = int(row_str), int(col_str)
if filename == last_filename:
assert last_row != row
else:
if last_filename is not None:
with open(last_filename, "w") as f:
f.writelines(lines)
with open(filename) as f:
lines = f.readlines()
last_filename = filename
last_row = row
line = lines[row - 1]
if err in ["C812", "C815"]:
lines[row - 1] = line[: col - 1] + "," + line[col - 1 :]
elif err in ["C819"]:
assert line[col - 2] == ","
lines[row - 1] = line[: col - 2] + line[col - 1 :].lstrip(" ")
if last_filename is not None:
with open(last_filename, "w") as f:
f.writelines(lines)
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
This implementation overrides some of PSA's internal backend
functions to handle `state` value with redis as the standard
way doesn't work because of apple sending required details
in the form of POST request.
Includes a mixin test class that'll be useful for testing
Native auth flow.
Thanks to Mateusz Mandera for the idea of using redis and
other important work on this.
Documentation rewritten by tabbott.
Co-authored-by: Mateusz Mandera <mateusz.mandera@zulip.com>
Now that we have production support for Ubuntu Focal, we update the
documentation to state our support for it.
(We also drop deprecated Xenial and Stretch from supported platforms).
We're migrating to using the cleaner zulip.com domain, which involves
changing all of our links from ReadTheDocs and other places to point
to the cleaner URL.
Calling jwt.decode without an algorithms list raises a
DeprecationWarning. This is for protecting against
symmetric/asymmetric key confusion attacks.
This is a backwards-incompatible configuration change.
Fixes#15207.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
Previously, it was unnecessarily difficult to parse the sentence to
determine that "HTTP response" and "internal state of the server
following the request" are the coherent ideas. Even if length wasn't
an issue, e.g. "... and checking both the A and B are correct", the
sentence still feels a bit fragile without a "that".
Since the second phrase is indeed relatively long, and "internal state
of the server" is a reasonable guess for the second coherent idea, the
"the" helps to reset the reader's expectation about where the next
coherent idea starts, and ends.
Lastly, having "both" in front of the two phrases encourages an
assumption that they're shorter (which is especially problematic for
the second phrase), while having it at the end of the sentence helps
to anchor the end of the second phrase; this is especially true since
the absence of "both" before that point encourages an assumption that
you haven't finished reading yet, given that two things have been
mentioned.
datetime.timezone is available in Python ≥ 3.2. This also lets us
remove a pytz dependency from the PostgreSQL scripts.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
Fixes this warning:
/srv/zulip/docs/conf.py:337: RemovedInSphinx40Warning: The app.add_stylesheet() is deprecated. Please use app.add_css_file() instead.
app.add_stylesheet('theme_overrides.css') # path relative to _static
https://www.sphinx-doc.org/en/3.x/extdev/deprecated.html
Signed-off-by: Anders Kaseorg <anders@zulip.com>
This reimplements our Zoom video call integration to use an OAuth
application. In addition to providing a cleaner setup experience,
especially on zulipchat.com where the server administrators can have
done the app registration already, it also fixes the limitation of the
previous integration that it could only have one call active at a time
when set up with typical Zoom API keys.
Fixes#11672.
Co-authored-by: Marco Burstein <marco@marco.how>
Co-authored-by: Tim Abbott <tabbott@zulipchat.com>
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
This commit replaces fa-file-text-o with fa-file-code-o which is a
better signal for the "view source" action. It also deletes a single
line comment that had suggested the change once we moved into font
awesome 4, which Aditya Bansal <adi.bansal241996@gmail.com> helped
out in doing, first via
91962aa6ab and most recently via
75ae94e459 with several commits in
between.
This should help ensure everyone uses the SSH key approach for Git
authentication; the HTTPS one is basically unusable as one has to
provide one's GitHub password after every command.
This section at the top was clearly written before the documentation
at the bottom existed, and hasn't been updated to point to the
now-existent docs below.
Add the link, rather than directing to #production-help.
The problem is not the list comprehension, as the previous wording
implied, but rather the fact that data is needed from the linked
table.
Be explicit about _what_ in the QuerySet API is helpful for addressing
this -- namely, use of `select_related`.
The new tools now have more concise, more parallel names:
- rebuild-dev-database
- rebuild-test-database
The actual implementations are still pretty different:
rebuild-dev-database:
mostly delegates to 5 management scripts
rebuild-test-database:
is a very thin wrapper for generate-fixtures
We'll try to clean that up a bit soon.
We no longer need to maintain duplicate code
related to where we set up the emoji
cache directory.
And we no longer need two extra steps for
people doing advanced (i.e. manual) setup.
There was no clear benefit to having provision
build the cache directory for `build_emoji`,
when it was easy to make `build_emoji` more
self-sufficient. The `build_emoji` tool
was already importing the library that has
`run_as_root`, and it was already responsible
for 99% of the create-directory kind of tasks.
(We always call `build_emoji` unconditionally from
`provision`, so there's no rationale in terms
of avoiding startup time or something.)
ASIDE:
Its not completely clear to me why we need
to put this directory in "/srv", instead of
somewhere more local (like we already do for
Travis), but maybe it's just to be like
its siblings in "/srv":
node_modules
yarn.lock
zulip-emoji-cache
zulip-npm-cache
zulip-py3-venv
zulip-thumbor-venv
zulip-venv-cache
zulip-yarn
I guess the caches that we keep in var are
dev-only, although I think some of what's under
`zulip-emoji-cache` is also dev-only in nature?
./var/webpack-cache
./var/mypy-cache
In `docs/subsystems/emoji.md` we say this:
```
The `build_emoji` tool generates the set of files under
`static/generated/emoji` (or really, it generates the
`/srv/zulip-emoji-cache/<sha1>/emoji` tree, and
`static/generated/emoji` is a symlink to that tree;we do this in
order to cache old versions to make provisioning and production
deployments super fast in the common case that we haven't changed the
emoji tooling). [...]
```
I don't really understand that rationale for the development
case, since `static/generated` is as much ignored by `git` as
'/srv' is, without the complications of needing `sudo` to create it.
And in production, I'm not sure how much time we're really saving,
as it takes me about 1.4s to fully rebuild the cache in dev, not to
mention we're taking on upgrade risk by sharing files between versions.
Also make sure our documentation for upgrading is reasonable for
Stretch => Buster.
Our reasoning for deprecating support for these releases is as follows:
* Ubuntu 16.04 Xenial reached desktop EOL last year; and will reach
EOL on the server in about a year.
* Debian Stretch will each EOL in 2020 (the precise date is unclear in
Debian's documentation, but based on past precedent it's in the next
few months, perhaps July 2020).
https://wiki.debian.org/DebianReleases#Production_Releases
* Both Ubuntu 16.04 and Debian Stretch use Python 3.5 as the system
Python, which will reach EOL in September 2020 (and we're already
seeing various third-party dependencies that we use drop support for
them).
* While there is LTS support for these older releases, it's not clear it's
going to be worth the added engineering effort for us to maintain EOL
releases of the base OSes that we support.
* We (now) have clear upgrade instructions for moving to Debian Buster
and Ubuntu 18.04.
This defends against cross-origin session fixation attacks. Renaming
the cookies means this one-time upgrade will have the unfortunate side
effect of logging everyone out, but they’ll get more secure sessions
in return.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
Instructions were added by doing the setup on Ubuntu 18.04 WSL 2.
While the setup should be similar for other distributions supported by
our `./tools/provision` script inside WSL, it has not been tested.
Polished by tabbott.
Generated by `pyupgrade --py3-plus --keep-percent-format` on all our
Python code except `zthumbor` and `zulip-ec2-configure-interfaces`,
followed by manual indentation fixes.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
This setting is being overridden by the frontend since the last
commit, and the security model is clearer and more robust if we don't
make it appear as though the markdown processor is handling this
issue.
Co-authored-by: Tim Abbott <tabbott@zulipchat.com>
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
I found the solution by simply trying out EMAIL_USE_SSL and it
worked. I had problems with sending emails (did not work at all, there
wasn't even a connection going on - I checked with tcpdump. Then I
found this: To use port 465, you need to call
smtplib.SMTP_SSL(). Currently, it looks like Django only uses
smtplib.SMTP() (source: https://code.djangoproject.com/ticket/9575).
Fixes#14350.
This returns us to a consistent logging format regardless of whether
the request is authenticated.
We also update some log examples in docs to be consistent with the new
style.
Those docs were outdated and no longer represantative of how things
work. upgrade-python-dependencies and unupgradable.json are no longer a
thing, so the entire paragraph about them should be removed.
Then the requirements/README.md file is refreshed a bit to be more
accurate.
For historical reasons we were creating Recipient
objects at some point in the typing-notifications
codepath. Now we just work with UserProfiles.
This removes some queries, as indicated by
the change to `len(queries)` in a couple of the
tests.
The one subtle thing that changes here is huddles.
If user 10 sends a typing notification that they
are talking to users 20 and 30, there might not
actually be a huddle for users 10/20/30, but
we were actually creating huddles on the fly!
There is no need to create huddles just for
typing notifications, since we don't even
share huddle ids with our clients. The clients
just infer the huddles.
Some of the code that gets killed off here as
somewhat "collateral damage" is some
defensive code related to formerly supporting streams
in typing indicators. The support for streams
was killed off almost as soon as we released
the feature, and the codepath is pretty clearly
user-centric at this point.
The only clients that should use the typing
indicators endpoint are our internal clients,
and they should send a JSON-formatted list
of user_ids.
Unfortunately, we still have some older versions
of mobile that still send emails.
In this commit we fix non-user-facing things
like docs and tests to promote the user_ids
interface that has existed since about version
2.0 of the server.
One annoyance is that we documented the
typing endpoint with emails, instead of the
more modern user_ids, which may have delayed
mobile converting to user_ids (and which
certainly caused confusion). It's trivial
to update the docs, but we need to short
circuit one assertion in the openapi tests.
We also clean up the test structure for the
typing tests:
TypingHappyPathTest.test_start_to_another_user
TypingHappyPathTest.test_start_to_multiple_recipients
TypingHappyPathTest.test_start_to_self
TypingHappyPathTest.test_start_to_single_recipient
TypingHappyPathTest.test_stop_to_another_user
TypingHappyPathTest.test_stop_to_self
TypingValidateOperatorTest.test_invalid_parameter
TypingValidateOperatorTest.test_missing_parameter
TypingValidateUsersTest.test_argument_to_is_not_valid_json
TypingValidateUsersTest.test_bogus_user_id
TypingValidateUsersTest.test_empty_array
TypingValidateUsersTest.test_missing_recipient
TypingValidationHelpersTest.test_recipient_for_user_ids
TypingValidationHelpersTest.test_recipient_for_user_ids_non_existent_id
TypingLegacyMobileSupportTest.test_legacy_email_interface
This has for a while been our only active Google Groups mailing list,
and given that folks will guess security@ as our security contact, we
might as well just standardize on that.
Also tweak some ambiguous text; it wouldn't be appropriate for us to
issue a CVE for e.g. an operational issue only affecting us.
isort 5 knows not to reorder imports across function calls, so this
will stop isort from breaking our code.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
- I fixed a typo with "lowerecase"
- I elaborated on the prefix before elaborating
on the rest of the message (i.e. went in correct
order).
- I split out the provision example (since we
talk about it some depth).
- I added more positive examples.
- I removed the distracting italics around the
good commit messages.
- I moved the "gather_subscriptions" commit to
the bottom of the list, since we elaborate
on that below the list.
This includes an experiment of having a draft of the 2.1.3 changelog,
which is helpful in avoiding duplication with the 2.2.0 changelog for
items we're planning to backport.
Django 2.2.x is the next LTS release after Django 1.11.x; I expect
we'll be on it for a while, as Django 3.x won't have an LTS release
series out for a while.
Because of upstream API changes in Django, this commit includes
several changes beyond requirements and:
* urls: django.urls.resolvers.RegexURLPattern has been replaced by
django.urls.resolvers.URLPattern; affects OpenAPI code and related
features which re-parse Django's internals.
https://code.djangoproject.com/ticket/28593
* test_runner: Change number to suffix. Django changed the name in this
ticket: https://code.djangoproject.com/ticket/28578
* Delete now-unnecessary SameSite cookie code (it's now the default).
* forms: urlsafe_base64_encode returns string in Django 2.2.
https://docs.djangoproject.com/en/2.2/ref/utils/#django.utils.http.urlsafe_base64_encode
* upload: Django's File.size property replaces _get_size().
https://docs.djangoproject.com/en/2.2/_modules/django/core/files/base/
* process_queue: Migrate to new autoreload API.
* test_messages: Add an extra query caused by .refresh_from_db() losing
the .select_related() on the Realm object.
* session: Sync SessionHostDomainMiddleware with Django 2.2.
There's a lot more we can do to take advantage of the new release;
this is tracked in #11341.
Many changes by Tim Abbott, Umair Waheed, and Mateusz Mandera squashed
are squashed into this commit.
Fixes#10835.
webpack optimizes JSON modules using JSON.parse("{…}"), which is
faster than the normal JavaScript parser.
Update the backend to use emoji_codes.json too instead of the three
separate JSON files.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
This is a fairly involved set of changes, including changes that:
* Delete various legacy or semi-duplicated sections of testing.md.
Nobody needs to manually delete the postgres datbase anymore, as
reflected in the fact that the docs still mention postgres 9.1 from
Ubuntu Precise.
* Simplify the distracting heading section at the top of testing.md.
* Move content on manual testing to docs/development/using.md.
* Moves some content related to managing the database to
schema-migrations.md. (Resulting in some cleanups to that page as
well).
I ideally would have split this into smaller pieces.
This addresses confusion we had with some organizations where they
were surprised that with only LDAP enabled, the "invite more users"
feature was available.
Fixes#11685.
This legacy cross-realm bot hasn't been used in several years, as far
as I know. If we wanted to re-introduce it, I'd want to implement it
as an embedded bot using those common APIs, rather than the totally
custom hacky code used for it that involves unnecessary queue workers
and similar details.
Fixes#13533.
This correct various inaccuracies and adds a bulleted list structure
for better clarity.
I think there's a lot more that could be done here in the form of
linking to other pages, discussing restarting `run-dev.py`, etc.
Added a link from docs/development/using ("Using the Development
Environment") to ./authentication ("Authentication in the development
environment") to help people working on the authentication systems
or anyone who needs an API key.
Separate using.html into Server/Web/Mobile sections so that readers
will find what they're looking for more quickly. Server is at the top
because it contains information relevant to web and mobile developers,
e.g., that the `run-dev.py` console output will provide useful errors.
Fixes#13655.
Zulip has had a small use of WebSockets (specifically, for the code
path of sending messages, via the webapp only) since ~2013. We
originally added this use of WebSockets in the hope that the latency
benefits of doing so would allow us to avoid implementing a markdown
local echo; they were not. Further, HTTP/2 may have eliminated the
latency difference we hoped to exploit by using WebSockets in any
case.
While we’d originally imagined using WebSockets for other endpoints,
there was never a good justification for moving more components to the
WebSockets system.
This WebSockets code path had a lot of downsides/complexity,
including:
* The messy hack involving constructing an emulated request object to
hook into doing Django requests.
* The `message_senders` queue processor system, which increases RAM
needs and must be provisioned independently from the rest of the
server).
* A duplicate check_send_receive_time Nagios test specific to
WebSockets.
* The requirement for users to have their firewalls/NATs allow
WebSocket connections, and a setting to disable them for networks
where WebSockets don’t work.
* Dependencies on the SockJS family of libraries, which has at times
been poorly maintained, and periodically throws random JavaScript
exceptions in our production environments without a deep enough
traceback to effectively investigate.
* A total of about 1600 lines of our code related to the feature.
* Increased load on the Tornado system, especially around a Zulip
server restart, and especially for large installations like
zulipchat.com, resulting in extra delay before messages can be sent
again.
As detailed in
https://github.com/zulip/zulip/pull/12862#issuecomment-536152397, it
appears that removing WebSockets moderately increases the time it
takes for the `send_message` API query to return from the server, but
does not significantly change the time between when a message is sent
and when it is received by clients. We don’t understand the reason
for that change (suggesting the possibility of a measurement error),
and even if it is a real change, we consider that potential small
latency regression to be acceptable.
If we later want WebSockets, we’ll likely want to just use Django
Channels.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
I added this tool a few years ago, and I did have
a vision for how it would improve our codebase, but
I can't remember exactly where I was going with it.
At this point the tool is just a little too noisy
to be helpful. An example of it creating confusion
was a recent PR where somebody was patching
user_circle_class in the PM list, and we already
had similar code in the buddy list, because they
use the same CSS. I mean, there was possibly a way
that the code could have been structured to remove
some of the duplication, but it probably would have
just moved the complexity around.
I just don't think it's worth maintaining the tool
at this point.
The well-known rowanj/gitx repository hasn't been updated since 2014.
Preferentially direct new contributors to gitx/gitx instead.
(We retain the rowanj repo as a fallback, since it has precompiled
releases available.)
This moves the mandatory configuration for options A/B/C into a single
bulleted list for each option, rather than split across two steps; I
think the result is significantly more readable.
It also fixes a bug where we suggested setting
AUTH_LDAP_REVERSE_EMAIL_SEARCH = AUTH_LDAP_USER_SEARCH in some cases,
whereas in fact it will never work because the parameters are
`%(email)s`, not `%(user)s`.
Also, now that one needs to set AUTH_LDAP_REVERSE_EMAIL_SEARCH, it
seems worth adding values for that to the Active Directory
instructions. Thanks to @alfonsrv for the suggestion.
This simplifies the RDS installation process to avoid awkwardly
requiring running the installer twice, and also is significantly more
robust in handling issues around rerunning the installer.
Finally, the answer for whether dictionaries are missing is available
to Django for future use in warnings/etc. around full-text search not
being great with this configuration, should they be required.
The previous documentation was essentially wrong, in that it
recommended copying certain settings that would cause significant
problems post-import if they were indeed copied.
Adds an explicit explanation to help contributors avoid common mistakes
like capitalization errors, missing trailing periods, and incorrectly
prefixing the name of a subsystem.
Fixes#1535.
This also rewrites the text to better explain what's happening. It's
likely further polish would be valuable, but that's true for the whole
"Troubleshooting" page.
This block of text was misplaced when we split the long
maintain-secure-update; article; we want it to be easy to find by
folks who are looking into error emails Zulip is sending.
This text is very old and hadn't been edited in a long time, in large
part because it was buried within old docs. This change cleans it up
to give accurate and better-organized information.
* Moves "Management commands" to a top-level section.
* Moves "Scalability" as a subsection at the bottom of "Requirements".
* Moves "Monitoring" as a subsections at the bottom of "Troubleshooting".
* Replaces "API and your Zulip URL" with a link to REST API docs. This
documentation text has been irrelevant for some time.
* Removes maintain-secure-upgrade from the TOC but the file remains to
avoid breaking old links from release blog posts and emails.
Updates the message editing process to do a local 'echo'.
On slow connections, now there is visual confirmation of the edit,
similar to when sending messages. The contains_backend_only_syntax
logic and check are the same as there.
We showing "(SAVING)" until the edit is completed, and on successful
edit, the word "(EDITED)" appears. There's likely useful future work
to do on making the animation experience nicer.
Substantially rewritten by tabbott to better handle corner cases and
communicate more clearly about what's happening.
Fixes: #3530.
We'll be soon documenting a production workflow that involves using
it, and that means it needs to live under scripts/ (since tools/ isn't
present in release tarballs).
- Moves "Authentication in the development environment" from subsystems
to "development/authentication.md".
- Moves "Renumbering migrations" to a section within "Schema migrations".
Merges the "Upgrades" section from production/maintain-secure-upgrade.md
with production/modifying-zulip.md.
Contains significant textual changes by tabbott to read more clearly.
- Merges the "Backups" section from production/maintain-secure-upgrade.md
with existing "Backups" section in production/export-and-import.md.
- Cleans up and makes content more clear/explicit.
- Adds short missing section on how to use wal-e configuration.
- Removes a lot of previously duplicate text explaining the difference between
the tools.
- Various textual tweaks by tabbott.
Fixes#13184 and resolves#293.
Documentation for tab switcher added under User documentation
features as a section at the bottom of "Tips and Warnings" article.
Text substantially rewritten by tabbott.
Fixes#12505
Fixes#9576.
Initial realm creation now works fine with the LDAP backend, so the
part of the docs about needing to create the first realm with the
email backend is unnecessary and just complicates the setup process,
so it seems best to just remove it.
With webpack, variables declared in each file are already file-local
(Global variables need to be explicitly exported), so these IIFEs are
no longer needed.
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
This fixes a collection of bugs surrounding LDAP configurations A and
C (i.e. LDAP_APPEND_DOMAIN=None) with EmailAuthBackend also enabled.
The core problem was that our desired security model in that setting
of requiring LDAP authentication for accounts managed by LDAP was not
implementable without a way to
Now admins can configure an LDAPSearch query that will find if there
are users in LDAP that have the email address and
email_belongs_to_ldap() will take advantage of that - no longer
returning True in response to all requests and thus blocking email
backend authentication.
In the documentation, we describe this as mandatory configuration for
users (and likely will make it so soon in the code) because the
failure modes for this not being configured are confusing.
But making that change is pending work to improve the relevant error
messages.
Fixes#11715.
This adds the general machinery required, and sets it up for the file
`typing_status.js` as a first use case.
Co-authored-by: Anders Kaseorg <anders@zulipchat.com>
There are a few outstanding issues that we expect to resolve beforce
including this in a release, but this is good checkpoint to merge.
This PR is a collaboration with Tim Abbott.
Fixes#716.
- Updated 260+ links from ".html" to ".md" to reduce the number of issues
reported about hyperlinks not working when viewing docs on Github.
- Removed temporary workaround that suppressed all warnings reported
by sphinx build for every link ending in ".html".
Details:
The recent upgrade to recommonmark==0.5.0 supports auto-converting
".md" links to ".html" so that the resulting HTML output is correct.
Notice that links pointing to a heading i.e. "../filename.html#heading",
were not updated because recommonmark does not auto-convert them.
These links do not generate build warnings and do not cause any issues.
However, there are about ~100 such links that might still get misreported
as broken links. This will be a follow-up issue.
Background:
docs: pip upgrade recommonmark and CommonMark #13013
docs: Allow .md links between doc pages #11719Fixes#11087.
Previously, we were hardcoding the domain s3.amazonaws.com. Given
that we already have an interface for configuring the host in
/etc/zulip/boto.cfg (which in turn, automatically configures boto), we
just need to actually use the value configured in boto for what S3
hostname to use.
We don't have tests for this new use case, in part because they're
likely annoying to write with `moto` and there hasn't been a huge
amount of demand for it. Since this doesn't regress existing S3
backend support, it seems worth merging.
Add a specific command to restart Vagrant to adopt the new
configuration.
(When naïvely using only `vagrant halt` + `vagrant up --provision`,
external devices remained unable to connect; per `netstat -nltp`, the
host IP of forwarded ports remained `127.0.0.1`.)
This sidesteps tricky escaping issues, and will make it easier to
build a strict Content-Security-Policy.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
We have lots of documentation for Zulip developers; but previously
didn't have a nice top-level page for Zulip server administrators to
learn how to manage patches to Zulip.
We no longer use tsearch_extras, and the camo patch is irrelevant on
systemd systems (Xenial and newer). So we no longer need to
provide/install a PPA at all.
Closes#13027.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
Now that we're implemented tsearch_extras in pure postgres, we no
longer need a custom extension. This should help us considerably, as
it means we no longer need to ship custom apt packages at all.
Fixes#467.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
- Added warning block to dev docs using _templates/layout.html
- Removed copy-pasted warning from install.md and requirements.md
- Removed unreleased tag in docs/conf.py that's no longer used
Useful ref: https://www.sphinx-doc.org/en/master/templating.htmlFixes#13056.
This requires part 1 (which can take hours to run but generally
doesn't require downtime) to be completed first.
This portion of the migration will require the server to be completely
down for a brief period; for chat.zulip.org with 250M UserMessage
rows, it took about 60s to run; that time will vary depending on
hardware details like whether the server has an SSD, but fundamentally
shouldn't be long.
Our upgrade-zulip and upgrade-zulip-from-git tools can apply this
migration correctly; nothing special needs to be done.
Fixes#13040.
mypy in daemon mode takes some 400 MiB of memory, and cannot follow
imports of type-annotated third-party packages; meanwhile, non-daemon
mode is no longer nearly as slow as it once was.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
Previously, Google might take you to /latest rather than /stable, and
you might get information about the next release, not the current one.
Fixes#13056.
Apparently, the ordering matters.
This command now matches what we use for installing single-purpose
installations; I should have checked this directly.
One needs to explicitly request zulip::base, it appears. Ideally,
we'd just have every ruleset depend on zulip::base, but I seem to
recall Puppet didn't like your including the same module from multiple
places. Worth testing as a follow-up investigation.
This was missed because we use the zulip_ops rules internally, which
include zulip::base via zulip_ops::base.
This commit finishes adding end-to-end support for the install script
on Debian Buster (making it production ready). Some support for this
was already added in prior commits such as
99414e2d96.
We plan to revert the postgres hunks of this once we've built
tsearch_extras for our packagecloud archive.
Fixes#9828.
This plugin automatically updates the VirtualBox Guest Additions in
the guest, which makes shared folders work more reliably.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
In this section of the docs, two tools for testing openapi
documentation are mentioned. But for the second one, we
forgot to mention that the tool also resides in the tools/
folder (like the first one which explicitly mentions it).
This commit fixes that.
Delete trailing newlines from all files, except
tools/ci/success-http-headers.txt and tools/setup/dev-motd, where they
are significant, and static/third, where we want to stay close to
upstream.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
As a result of dropping support for trusty, we can remove our old
pattern of putting `if False` before importing the typing module,
which was essential for Python 3.4 support, but not required and maybe
harmful on newer versions.
cron_file_helper
check_rabbitmq_consumers
hash_reqs
check_zephyr_mirror
check_personal_zephyr_mirrors
check_cron_file
zulip_tools
check_postgres_replication_lag
api_test_helpers
purge-old-deployments
setup_venv
node_cache
clean_venv_cache
clean_node_cache
clean_emoji_cache
pg_backup_and_purge
restore-backup
generate_secrets
zulip-ec2-configure-interfaces
diagnose
check_user_zephyr_mirror_liveness
Outgoing email documentation should mention that the SMTP server needs
to allow emails originating from ZULIP_ADMINISTRATOR as well as the
noreply email addresses; previously, one would typically find this out
using the test tool, which was not ideal.
Substantially tweaked by tabbott for organization and content.
This replaces the two custom Google authentication backends originally
written in 2012 with using the shared python-social-auth codebase that
we already use for the GitHub authentication backend. These are:
* GoogleMobileOauth2Backend, the ancient code path for mobile
authentication last used by the EOL original Zulip Android app.
* The `finish_google_oauth2` code path in zerver/views/auth.py, which
was the webapp (and modern mobile app) Google authentication code
path.
This change doesn't fix any known bugs; its main benefit is that we
get to remove hundreds of lines of security-sensitive semi-duplicated
code, replacing it with a widely trusted, high quality third-party
library.
We had an organization with engineers for whom English was not their
native language think they needed to go through this laborious
process, which was definitely counterproductive for them.
Many Linux distros like Debian Buster, Fedora 29, etc. have a policy
where any services installed are disabled by default. So we should let
the developer know that they should explicitly enable and start the
Docker daemon.
Also, we should be verbose on how to do vagrant up using Docker as a
provider specifically (and similar for Virtualbox).
This was rewritten by tabbott to use markdown steps, avoid
distractions about things you don't need to change, and remove the
screenshots, which I felt were more clutter than value.
We had several patches to spectrum, but the only essential one
(0ea770fc18) had already been fixed upstream,
and another was just handling jQuery deprecation warnings for not yet removed features.
See #12749 for details.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
It's not actually a reasonable experience to use Zulip without working
full-text search, so we should more strongly direct folks to not do
it.
Fixes#12724.
Alex Vandiver
Tim Abbott
Riken-Shah
Mateusz Mandera
Aman Agrawal
nikhilmaske-2001
aryanshridhar
Sumanth V Rao
Anders Kaseorg
Steve Howell
Nikhil Maske
Atharva Parkale
IgnisDa
marchantia
Aditya Mitra
Ganesh Pawar
Vishnu KS
Harsh kumar
rht
marchantia
Anders Kaseorg
Ben Steinberg
YashRE42
ryanreh99
sahil839
the-rdt
strifel
areebbeigh
Brian Lee
Anders Kaseorg
Thomas Güttler
Dinesh
Rohitt Vashishtha
Priyansh Garg
arpit551
Vinit Singh
Priyank Patel
Mohit Gupta
Greg Price
Brainrecursion
apxwn
SiddharthVarshney
Onur Temiz
qnxor
Sharif Naas
Mateusz Mandera
Alex Vandiver
Steve Howell
Puneeth Chaganti
Alex Vandiver
Tim Abbott
Steve Howell
Aman Agrawal
Ray Kraesig
thoemie
shubhamgupta2956
Stefan Weil
Rike-Benjamin Schuppner
Yashwant
Vaibhav Raj Singh
Chris Bobbe
Dinesh
xpac1985
Rodriq
Rodriq
David Rosa
Jack Tiggleman
Hashir Sarwar
Tlazypanda
YashRE42
Shikhar Varshney
Chris Bobbe
Thomas Ashish Cherian
Wyatt Hoodes
Hemanth V. Alluri
Tej Pochiraju
Harshit Bansal
Rishi Gupta
Amal Rajan
Tyler B. Thrailkill