The new tools now have more concise, more parallel names:
- rebuild-dev-database
- rebuild-test-database
The actual implementations are still pretty different:
rebuild-dev-database:
mostly delegates to 5 management scripts
rebuild-test-database:
is a very thin wrapper for generate-fixtures
We'll try to clean that up a bit soon.
We no longer need to maintain duplicate code
related to where we set up the emoji
cache directory.
And we no longer need two extra steps for
people doing advanced (i.e. manual) setup.
There was no clear benefit to having provision
build the cache directory for `build_emoji`,
when it was easy to make `build_emoji` more
self-sufficient. The `build_emoji` tool
was already importing the library that has
`run_as_root`, and it was already responsible
for 99% of the create-directory kind of tasks.
(We always call `build_emoji` unconditionally from
`provision`, so there's no rationale in terms
of avoiding startup time or something.)
ASIDE:
Its not completely clear to me why we need
to put this directory in "/srv", instead of
somewhere more local (like we already do for
Travis), but maybe it's just to be like
its siblings in "/srv":
node_modules
yarn.lock
zulip-emoji-cache
zulip-npm-cache
zulip-py3-venv
zulip-thumbor-venv
zulip-venv-cache
zulip-yarn
I guess the caches that we keep in var are
dev-only, although I think some of what's under
`zulip-emoji-cache` is also dev-only in nature?
./var/webpack-cache
./var/mypy-cache
In `docs/subsystems/emoji.md` we say this:
```
The `build_emoji` tool generates the set of files under
`static/generated/emoji` (or really, it generates the
`/srv/zulip-emoji-cache/<sha1>/emoji` tree, and
`static/generated/emoji` is a symlink to that tree;we do this in
order to cache old versions to make provisioning and production
deployments super fast in the common case that we haven't changed the
emoji tooling). [...]
```
I don't really understand that rationale for the development
case, since `static/generated` is as much ignored by `git` as
'/srv' is, without the complications of needing `sudo` to create it.
And in production, I'm not sure how much time we're really saving,
as it takes me about 1.4s to fully rebuild the cache in dev, not to
mention we're taking on upgrade risk by sharing files between versions.
Instructions were added by doing the setup on Ubuntu 18.04 WSL 2.
While the setup should be similar for other distributions supported by
our `./tools/provision` script inside WSL, it has not been tested.
Polished by tabbott.
This returns us to a consistent logging format regardless of whether
the request is authenticated.
We also update some log examples in docs to be consistent with the new
style.
This is a fairly involved set of changes, including changes that:
* Delete various legacy or semi-duplicated sections of testing.md.
Nobody needs to manually delete the postgres datbase anymore, as
reflected in the fact that the docs still mention postgres 9.1 from
Ubuntu Precise.
* Simplify the distracting heading section at the top of testing.md.
* Move content on manual testing to docs/development/using.md.
* Moves some content related to managing the database to
schema-migrations.md. (Resulting in some cleanups to that page as
well).
I ideally would have split this into smaller pieces.
This correct various inaccuracies and adds a bulleted list structure
for better clarity.
I think there's a lot more that could be done here in the form of
linking to other pages, discussing restarting `run-dev.py`, etc.
Added a link from docs/development/using ("Using the Development
Environment") to ./authentication ("Authentication in the development
environment") to help people working on the authentication systems
or anyone who needs an API key.
Separate using.html into Server/Web/Mobile sections so that readers
will find what they're looking for more quickly. Server is at the top
because it contains information relevant to web and mobile developers,
e.g., that the `run-dev.py` console output will provide useful errors.
Fixes#13655.
Zulip has had a small use of WebSockets (specifically, for the code
path of sending messages, via the webapp only) since ~2013. We
originally added this use of WebSockets in the hope that the latency
benefits of doing so would allow us to avoid implementing a markdown
local echo; they were not. Further, HTTP/2 may have eliminated the
latency difference we hoped to exploit by using WebSockets in any
case.
While we’d originally imagined using WebSockets for other endpoints,
there was never a good justification for moving more components to the
WebSockets system.
This WebSockets code path had a lot of downsides/complexity,
including:
* The messy hack involving constructing an emulated request object to
hook into doing Django requests.
* The `message_senders` queue processor system, which increases RAM
needs and must be provisioned independently from the rest of the
server).
* A duplicate check_send_receive_time Nagios test specific to
WebSockets.
* The requirement for users to have their firewalls/NATs allow
WebSocket connections, and a setting to disable them for networks
where WebSockets don’t work.
* Dependencies on the SockJS family of libraries, which has at times
been poorly maintained, and periodically throws random JavaScript
exceptions in our production environments without a deep enough
traceback to effectively investigate.
* A total of about 1600 lines of our code related to the feature.
* Increased load on the Tornado system, especially around a Zulip
server restart, and especially for large installations like
zulipchat.com, resulting in extra delay before messages can be sent
again.
As detailed in
https://github.com/zulip/zulip/pull/12862#issuecomment-536152397, it
appears that removing WebSockets moderately increases the time it
takes for the `send_message` API query to return from the server, but
does not significantly change the time between when a message is sent
and when it is received by clients. We don’t understand the reason
for that change (suggesting the possibility of a measurement error),
and even if it is a real change, we consider that potential small
latency regression to be acceptable.
If we later want WebSockets, we’ll likely want to just use Django
Channels.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
We'll be soon documenting a production workflow that involves using
it, and that means it needs to live under scripts/ (since tools/ isn't
present in release tarballs).
- Moves "Authentication in the development environment" from subsystems
to "development/authentication.md".
- Moves "Renumbering migrations" to a section within "Schema migrations".
- Updated 260+ links from ".html" to ".md" to reduce the number of issues
reported about hyperlinks not working when viewing docs on Github.
- Removed temporary workaround that suppressed all warnings reported
by sphinx build for every link ending in ".html".
Details:
The recent upgrade to recommonmark==0.5.0 supports auto-converting
".md" links to ".html" so that the resulting HTML output is correct.
Notice that links pointing to a heading i.e. "../filename.html#heading",
were not updated because recommonmark does not auto-convert them.
These links do not generate build warnings and do not cause any issues.
However, there are about ~100 such links that might still get misreported
as broken links. This will be a follow-up issue.
Background:
docs: pip upgrade recommonmark and CommonMark #13013
docs: Allow .md links between doc pages #11719Fixes#11087.
Add a specific command to restart Vagrant to adopt the new
configuration.
(When naïvely using only `vagrant halt` + `vagrant up --provision`,
external devices remained unable to connect; per `netstat -nltp`, the
host IP of forwarded ports remained `127.0.0.1`.)
Now that we're implemented tsearch_extras in pure postgres, we no
longer need a custom extension. This should help us considerably, as
it means we no longer need to ship custom apt packages at all.
Fixes#467.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
This commit finishes adding end-to-end support for the install script
on Debian Buster (making it production ready). Some support for this
was already added in prior commits such as
99414e2d96.
We plan to revert the postgres hunks of this once we've built
tsearch_extras for our packagecloud archive.
Fixes#9828.
This plugin automatically updates the VirtualBox Guest Additions in
the guest, which makes shared folders work more reliably.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
Many Linux distros like Debian Buster, Fedora 29, etc. have a policy
where any services installed are disabled by default. So we should let
the developer know that they should explicitly enable and start the
Docker daemon.
Also, we should be verbose on how to do vagrant up using Docker as a
provider specifically (and similar for Virtualbox).
Steve Howell
Aman Agrawal
Tim Abbott
Anders Kaseorg
arpit551
Mateusz Mandera
Puneeth Chaganti
Dinesh
Chris Bobbe
David Rosa
YashRE42
Ray Kraesig
Hemanth V. Alluri
Amal Rajan
Vinit Singh
YashRE42
Thomas Ip
Kanishk Kakar