It seems the Ubuntu base image we use now has a new enough VirtualBox
Guest Additions to trigger the ETXTBSY bug even when it’s not upgraded
by the vagrant-vbguest plugin. Provide and document a way to
downgrade it.
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
We add navigating to user home inside WSL virtual disk as another
step as many users clone Zulip inside a mounted windows disk and
run into permission issues when running provision.
There are file sharing issues with the macOS 10.15.6 and
vagrant. var/remote_cache_prefix was an empty file when using
VirtualBox and Docker on macOS.
Using parallels as a provider for vagrant fixes the issue.
Use --watch-poll which makes webpack to recompile
automatically on file changes, since inotify is not
working here too.
The apple developer webapp consistently refers this App ID. So,
this clears any confusion that can occur.
Since python social auth only requires us to include App ID in
_AUDIENCE(a list), we do that in computed settings making it easier for
server admin and we make it much clear by having it set to
APP_ID instead of BUNDLE_ID.
This implementation overrides some of PSA's internal backend
functions to handle `state` value with redis as the standard
way doesn't work because of apple sending required details
in the form of POST request.
Includes a mixin test class that'll be useful for testing
Native auth flow.
Thanks to Mateusz Mandera for the idea of using redis and
other important work on this.
Documentation rewritten by tabbott.
Co-authored-by: Mateusz Mandera <mateusz.mandera@zulip.com>
We're migrating to using the cleaner zulip.com domain, which involves
changing all of our links from ReadTheDocs and other places to point
to the cleaner URL.
This should help ensure everyone uses the SSH key approach for Git
authentication; the HTTPS one is basically unusable as one has to
provide one's GitHub password after every command.
The new tools now have more concise, more parallel names:
- rebuild-dev-database
- rebuild-test-database
The actual implementations are still pretty different:
rebuild-dev-database:
mostly delegates to 5 management scripts
rebuild-test-database:
is a very thin wrapper for generate-fixtures
We'll try to clean that up a bit soon.
We no longer need to maintain duplicate code
related to where we set up the emoji
cache directory.
And we no longer need two extra steps for
people doing advanced (i.e. manual) setup.
There was no clear benefit to having provision
build the cache directory for `build_emoji`,
when it was easy to make `build_emoji` more
self-sufficient. The `build_emoji` tool
was already importing the library that has
`run_as_root`, and it was already responsible
for 99% of the create-directory kind of tasks.
(We always call `build_emoji` unconditionally from
`provision`, so there's no rationale in terms
of avoiding startup time or something.)
ASIDE:
Its not completely clear to me why we need
to put this directory in "/srv", instead of
somewhere more local (like we already do for
Travis), but maybe it's just to be like
its siblings in "/srv":
node_modules
yarn.lock
zulip-emoji-cache
zulip-npm-cache
zulip-py3-venv
zulip-thumbor-venv
zulip-venv-cache
zulip-yarn
I guess the caches that we keep in var are
dev-only, although I think some of what's under
`zulip-emoji-cache` is also dev-only in nature?
./var/webpack-cache
./var/mypy-cache
In `docs/subsystems/emoji.md` we say this:
```
The `build_emoji` tool generates the set of files under
`static/generated/emoji` (or really, it generates the
`/srv/zulip-emoji-cache/<sha1>/emoji` tree, and
`static/generated/emoji` is a symlink to that tree;we do this in
order to cache old versions to make provisioning and production
deployments super fast in the common case that we haven't changed the
emoji tooling). [...]
```
I don't really understand that rationale for the development
case, since `static/generated` is as much ignored by `git` as
'/srv' is, without the complications of needing `sudo` to create it.
And in production, I'm not sure how much time we're really saving,
as it takes me about 1.4s to fully rebuild the cache in dev, not to
mention we're taking on upgrade risk by sharing files between versions.
Instructions were added by doing the setup on Ubuntu 18.04 WSL 2.
While the setup should be similar for other distributions supported by
our `./tools/provision` script inside WSL, it has not been tested.
Polished by tabbott.
This returns us to a consistent logging format regardless of whether
the request is authenticated.
We also update some log examples in docs to be consistent with the new
style.
This is a fairly involved set of changes, including changes that:
* Delete various legacy or semi-duplicated sections of testing.md.
Nobody needs to manually delete the postgres datbase anymore, as
reflected in the fact that the docs still mention postgres 9.1 from
Ubuntu Precise.
* Simplify the distracting heading section at the top of testing.md.
* Move content on manual testing to docs/development/using.md.
* Moves some content related to managing the database to
schema-migrations.md. (Resulting in some cleanups to that page as
well).
I ideally would have split this into smaller pieces.
This correct various inaccuracies and adds a bulleted list structure
for better clarity.
I think there's a lot more that could be done here in the form of
linking to other pages, discussing restarting `run-dev.py`, etc.
Added a link from docs/development/using ("Using the Development
Environment") to ./authentication ("Authentication in the development
environment") to help people working on the authentication systems
or anyone who needs an API key.
Separate using.html into Server/Web/Mobile sections so that readers
will find what they're looking for more quickly. Server is at the top
because it contains information relevant to web and mobile developers,
e.g., that the `run-dev.py` console output will provide useful errors.
Fixes#13655.
Zulip has had a small use of WebSockets (specifically, for the code
path of sending messages, via the webapp only) since ~2013. We
originally added this use of WebSockets in the hope that the latency
benefits of doing so would allow us to avoid implementing a markdown
local echo; they were not. Further, HTTP/2 may have eliminated the
latency difference we hoped to exploit by using WebSockets in any
case.
While we’d originally imagined using WebSockets for other endpoints,
there was never a good justification for moving more components to the
WebSockets system.
This WebSockets code path had a lot of downsides/complexity,
including:
* The messy hack involving constructing an emulated request object to
hook into doing Django requests.
* The `message_senders` queue processor system, which increases RAM
needs and must be provisioned independently from the rest of the
server).
* A duplicate check_send_receive_time Nagios test specific to
WebSockets.
* The requirement for users to have their firewalls/NATs allow
WebSocket connections, and a setting to disable them for networks
where WebSockets don’t work.
* Dependencies on the SockJS family of libraries, which has at times
been poorly maintained, and periodically throws random JavaScript
exceptions in our production environments without a deep enough
traceback to effectively investigate.
* A total of about 1600 lines of our code related to the feature.
* Increased load on the Tornado system, especially around a Zulip
server restart, and especially for large installations like
zulipchat.com, resulting in extra delay before messages can be sent
again.
As detailed in
https://github.com/zulip/zulip/pull/12862#issuecomment-536152397, it
appears that removing WebSockets moderately increases the time it
takes for the `send_message` API query to return from the server, but
does not significantly change the time between when a message is sent
and when it is received by clients. We don’t understand the reason
for that change (suggesting the possibility of a measurement error),
and even if it is a real change, we consider that potential small
latency regression to be acceptable.
If we later want WebSockets, we’ll likely want to just use Django
Channels.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
We'll be soon documenting a production workflow that involves using
it, and that means it needs to live under scripts/ (since tools/ isn't
present in release tarballs).
- Moves "Authentication in the development environment" from subsystems
to "development/authentication.md".
- Moves "Renumbering migrations" to a section within "Schema migrations".
- Updated 260+ links from ".html" to ".md" to reduce the number of issues
reported about hyperlinks not working when viewing docs on Github.
- Removed temporary workaround that suppressed all warnings reported
by sphinx build for every link ending in ".html".
Details:
The recent upgrade to recommonmark==0.5.0 supports auto-converting
".md" links to ".html" so that the resulting HTML output is correct.
Notice that links pointing to a heading i.e. "../filename.html#heading",
were not updated because recommonmark does not auto-convert them.
These links do not generate build warnings and do not cause any issues.
However, there are about ~100 such links that might still get misreported
as broken links. This will be a follow-up issue.
Background:
docs: pip upgrade recommonmark and CommonMark #13013
docs: Allow .md links between doc pages #11719Fixes#11087.
Add a specific command to restart Vagrant to adopt the new
configuration.
(When naïvely using only `vagrant halt` + `vagrant up --provision`,
external devices remained unable to connect; per `netstat -nltp`, the
host IP of forwarded ports remained `127.0.0.1`.)
Now that we're implemented tsearch_extras in pure postgres, we no
longer need a custom extension. This should help us considerably, as
it means we no longer need to ship custom apt packages at all.
Fixes#467.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
This commit finishes adding end-to-end support for the install script
on Debian Buster (making it production ready). Some support for this
was already added in prior commits such as
99414e2d96.
We plan to revert the postgres hunks of this once we've built
tsearch_extras for our packagecloud archive.
Fixes#9828.
This plugin automatically updates the VirtualBox Guest Additions in
the guest, which makes shared folders work more reliably.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
Many Linux distros like Debian Buster, Fedora 29, etc. have a policy
where any services installed are disabled by default. So we should let
the developer know that they should explicitly enable and start the
Docker daemon.
Also, we should be verbose on how to do vagrant up using Docker as a
provider specifically (and similar for Virtualbox).
This also means the default dev environment is now based on Ubuntu
18.04 (bionic), part of our overall effort to migrate off Ubuntu
Trusty.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
Two variables were declared and assigned the respective values of the
default settings for the system. If the keyword is used in the
~/.zulip-vagrant-config file, the value is assigned to the variable.
There is no straightforward way to customize the virtual machine's
number of cpus or memory, this commit addresses that fact.
Sphinx/ReadTheDocs supports automatically translating links written as
to `.md` files to point to the corresponding `.html` files, so this
migration does not change the resulting HTML output in ReadTheDocs.
But it does fix apparent broken links on GitHub.
This doesn't prevent people from reading the documentation on GitHub
(so doesn't mitigate the fact that some rtd-specific syntax does not
render properly on GH), but it will prevent us from getting erroneous
issues reported about the hyperlinks not working.
Fixes: #11087.
The docs specify passing hostname with the --hostname flag, which
doesn't match the actual usage in scripts/setup/setup-certbot. This
change fixes the docs to match the actual usage.
Add the recommendation in docs for not using GitHub Desktop client for
zulip code as it has a bug of keeping the symlinks disabled and ignoring
any configuration done to re-enable it resulting into failure to identify
the received symlinks leading to failing test cases in the zulip development
environment.
This documentation had a tendency to bitrot, and in any case now that
we have tooling for doing Fedora (etc.) from provision, it's likely
the case that adding other Linux/UNIX distros we care about to
provision would not be difficult and is a better path than maintaining
this manually-curated duplicate of `tools/provision`.
Further, even if this documentation was maintained, one would still
end up wanting to run `provision` after rebasing a branch, so it was
never particularly practical for extended development.
The history of this was that there was a period where half of Vagrant
releases were broken (for everyone, e.g. downloading a base container
didn't work). It seems Vagrant has cleaned up their act at this point.
We really just want 2.0.x, not a specific version, and the direct
links we had are now quite old.
Thanks to Jonathon Hinchley for reporting this.
Fixes#11836.
help.github.com seems to have a bug where HEAD on a redirected page
returns 404. This causes tools/test-documentation to fail. Fix it by
skipping the redirects.
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
Otherwise this causes an error
```
AttributeError: type object 'Callable' has no attribute '_abc_registry'
```
on 3.7. While the error is specific to 3.7, it is safer to uninstall
typing for all the versions that don't require a pip-provided typing
library.
Older versions of virtual box were giving installation error in new
MacOS Mojave. While originally we considered changing the docs to
point to the current version 5.2.20, it seems better to just not pin a
version.
Significantly tweaked by tabbott.
On OSX, the user id and group id don't match. So while the previous
code was always wrong, it produced incorrect output there. We can fix
this by replacing `whoami` with `id -g` for finding the current user's
group ID.
Tweaked by tabbott to move most of the content into the remote
development documentation, both for brevity in the main docs as well
as clarity.
Fixes#10694.
It appears Luke's Dropbox folder/shared link that used to host some
tsearch_extras binaries was removed. It wasn't very high-value
regardless, because most of the platforms involved are deprecated, the
ones that don't generally have a PPA, and building from source is
pretty easy. So, we just remove these options from the documentation.
While we're at it, make clear we only support direct installation on
Ubuntu LTS.
Fixes#9863.
This didn't really make much sense - it presented a quickstart summary
of the instructions as an item in the list of differences from the
Ubuntu case, plus that quickstart only incorporated one of the two
actual differences so it didn't work anyway. Given the networking
stuff, an actual quickstart isn't in the cards, so just cut that.
Also fix some small things while we're here.
Update version of Vagrant from 1.8.6 to 2.0.2.
Update version of VirtualBox from 5.1.8 to 5.2.6.
We needed to update Vagrant because older versions have networking
problems on modern Macs, resulting in weird `apt` errors when
provisioning.
Commit message and some text tweaked by tabbott.
This is what the Sphinx docs recommend when you actually don't want
the page to be included in navigation:
http://www.sphinx-doc.org/en/stable/markup/toctree.html
And now that we have `eval_rst`, we're able to take advantage of it!
One difference between doing this and the old way of making "hidden"
toctree entries is that with the latter, the "previous" and "next"
links at the bottom of each page would thread through the hidden
entries; which gets kind of confusing when they don't appear in the nav.
Previous versions of these links were removed earlier today because
Docker upstream had broken them.
To see what the links had been intended to point to, I used the
Wayback Machine (web.archive.org/web), the super-handy project of the
Internet Archive. These are the current equivalents.
Sphinx was displaying "WARNING: document isn't included in any toctree"
for files we just don't want in the TOC. We can hide them from the index,
but the rtd theme defaults to display hidden index entries in the nav bar.
This commit excludes these files from such warnings, and patches layout.html
so that hidden index entries stay hidden from the navigation sidebar.
This commit also moves password-strength.md under docs/production and
adds it as a hidden entry in production/index.rst.
Fixes#7417.
This commit renames various source requirements files like `dev.txt`,
`mypy.txt` etc to `dev.in`, `mypy.in` etc and various locked requirements
files like `dev_lock.txt`, `mypy_lock.txt` etc to `dev.txt`, `mypy.txt`
etc. This will help in emphasizing to the user that *.in are actually
input to `update-locked-requirements` tool which should be run after
updating any of these.
This updates the developer documentation to recommend `git clone -c`
to set the git pull mode to be rebase by default in the repository
configuration.
The hope is that this will help minimize how often new contributors
end up accidentally messing up their PRs by using `git pull`.
Tweaked by tabbott to fix broken links and optimize the language.
Fixes: #7022.
This commit helps reduce clutter on the navigation sidebar.
Creates new directories and moves relevant files into them.
Modifies index.rst, symlinks, and image paths accordingly.
This commit also enables expandable/collapsible navigation items,
renames files in docs/development and docs/production,
modifies /tools/test-documentation so that it overrides a theme setting,
Also updates links to other docs, file paths in the codebase that point
to developer documents, and files that should be excluded from lint tests.
Note that this commit does not update direct links to
zulip.readthedocs.io in the codebase; those will be resolved in an
upcoming follow-up commit (it'll be easier to verify all the links
once this is merged and ReadTheDocs is updated).
Fixes#5265.
Aditya Mitra
Harsh kumar
Anders Kaseorg
Aman Agrawal
Alex Vandiver
Anders Kaseorg
Thomas Güttler
Dinesh
Tim Abbott
Steve Howell
Aman Agrawal
Tim Abbott
Anders Kaseorg
arpit551
Mateusz Mandera
Puneeth Chaganti
Dinesh
Chris Bobbe
David Rosa
YashRE42
Ray Kraesig
Hemanth V. Alluri
Amal Rajan
Vinit Singh
YashRE42
Thomas Ip
Kanishk Kakar
Mayank Madan
Wyatt Hoodes
Samuel Searles-Bryant
Zach Wener-Fligner
AsociTon
Dominik Grybos
Priyanshu Singh
Paurakh Sharma Humagain
rht
Vishnu Ks
Sumanth V Rao
ruchit2801
Scott Lee
Abhigyan Khaund
Hendy Irawan
Yago González
gooca
feorlen
cPhost
knovokresch
Greg Price
Shivam Gera
Jeremy Zhou
itstakenalr
Harshit Bansal
Mohd Ali Rizwi
KiranS