The new tools now have more concise, more parallel names:
- rebuild-dev-database
- rebuild-test-database
The actual implementations are still pretty different:
rebuild-dev-database:
mostly delegates to 5 management scripts
rebuild-test-database:
is a very thin wrapper for generate-fixtures
We'll try to clean that up a bit soon.
This defends against cross-origin session fixation attacks. Renaming
the cookies means this one-time upgrade will have the unfortunate side
effect of logging everyone out, but they’ll get more secure sessions
in return.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
This includes an experiment of having a draft of the 2.1.3 changelog,
which is helpful in avoiding duplication with the 2.2.0 changelog for
items we're planning to backport.
Zulip has had a small use of WebSockets (specifically, for the code
path of sending messages, via the webapp only) since ~2013. We
originally added this use of WebSockets in the hope that the latency
benefits of doing so would allow us to avoid implementing a markdown
local echo; they were not. Further, HTTP/2 may have eliminated the
latency difference we hoped to exploit by using WebSockets in any
case.
While we’d originally imagined using WebSockets for other endpoints,
there was never a good justification for moving more components to the
WebSockets system.
This WebSockets code path had a lot of downsides/complexity,
including:
* The messy hack involving constructing an emulated request object to
hook into doing Django requests.
* The `message_senders` queue processor system, which increases RAM
needs and must be provisioned independently from the rest of the
server).
* A duplicate check_send_receive_time Nagios test specific to
WebSockets.
* The requirement for users to have their firewalls/NATs allow
WebSocket connections, and a setting to disable them for networks
where WebSockets don’t work.
* Dependencies on the SockJS family of libraries, which has at times
been poorly maintained, and periodically throws random JavaScript
exceptions in our production environments without a deep enough
traceback to effectively investigate.
* A total of about 1600 lines of our code related to the feature.
* Increased load on the Tornado system, especially around a Zulip
server restart, and especially for large installations like
zulipchat.com, resulting in extra delay before messages can be sent
again.
As detailed in
https://github.com/zulip/zulip/pull/12862#issuecomment-536152397, it
appears that removing WebSockets moderately increases the time it
takes for the `send_message` API query to return from the server, but
does not significantly change the time between when a message is sent
and when it is received by clients. We don’t understand the reason
for that change (suggesting the possibility of a measurement error),
and even if it is a real change, we consider that potential small
latency regression to be acceptable.
If we later want WebSockets, we’ll likely want to just use Django
Channels.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
Merges the "Upgrades" section from production/maintain-secure-upgrade.md
with production/modifying-zulip.md.
Contains significant textual changes by tabbott to read more clearly.
- Updated 260+ links from ".html" to ".md" to reduce the number of issues
reported about hyperlinks not working when viewing docs on Github.
- Removed temporary workaround that suppressed all warnings reported
by sphinx build for every link ending in ".html".
Details:
The recent upgrade to recommonmark==0.5.0 supports auto-converting
".md" links to ".html" so that the resulting HTML output is correct.
Notice that links pointing to a heading i.e. "../filename.html#heading",
were not updated because recommonmark does not auto-convert them.
These links do not generate build warnings and do not cause any issues.
However, there are about ~100 such links that might still get misreported
as broken links. This will be a follow-up issue.
Background:
docs: pip upgrade recommonmark and CommonMark #13013
docs: Allow .md links between doc pages #11719Fixes#11087.
Now that we're implemented tsearch_extras in pure postgres, we no
longer need a custom extension. This should help us considerably, as
it means we no longer need to ship custom apt packages at all.
Fixes#467.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
This requires part 1 (which can take hours to run but generally
doesn't require downtime) to be completed first.
This portion of the migration will require the server to be completely
down for a brief period; for chat.zulip.org with 250M UserMessage
rows, it took about 60s to run; that time will vary depending on
hardware details like whether the server has an SSD, but fundamentally
shouldn't be long.
Our upgrade-zulip and upgrade-zulip-from-git tools can apply this
migration correctly; nothing special needs to be done.
Fixes#13040.
As of commit cff40c557b (#9300), these
files are no longer served directly to the browser. Disentangle them
from the static asset pipeline so we can refactor it without worrying
about them.
This has the side effect of eliminating the accidental duplication of
translation data via hash-naming in our release tarballs.
This reverts commit b546391f0b (#1148).
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
Move docs/tutorials/documenting-api-endpoint.md to
docs/documentation/api.md.
This makes it easier to find when browsing the complete set of
materials on writing Zulip documentation.
"file:/srv/zulip/docs/_build/html/git/troubleshooting.html":266.5-266.12: error: Element "commit" not allowed as child of element "em" in this context. (Suppressing further errors from this subtree.)
"file:/srv/zulip/docs/_build/html/git/troubleshooting.html":266.13-266.17: error: End tag "em" violates nesting rules.
"file:/srv/zulip/docs/_build/html/overview/architecture-overview.html":211.69-211.136: error: Start tag "a" seen but an element of the same type was already open.
"file:/srv/zulip/docs/_build/html/overview/architecture-overview.html":211.171-211.174: error: Stray end tag "a".
"file:/srv/zulip/docs/_build/html/translating/hindi.html":252.36-252.48: info warning: Text run is not in Unicode Normalization Form C.
"file:/srv/zulip/docs/_build/html/tutorials/writing-views.html":469.34-469.41: error: Element "method" not allowed as child of element "p" in this context. (Suppressing further errors from this subtree.)
"file:/srv/zulip/docs/_build/html/tutorials/writing-views.html":471.122-471.125: error: End tag "p" seen, but there were open elements.
"file:/srv/zulip/docs/_build/html/tutorials/writing-views.html":469.34-469.41: error: Unclosed element "method".
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
Add references to TypeScript in documentation where appropriate, such
as in example bash commands and discussions of the file structure.
Add a new section to the Reading List with TypeScript resources.
Also update `.editorconfig` to support ".ts" files.
Fix part of #12000.
Sphinx/ReadTheDocs supports automatically translating links written as
to `.md` files to point to the corresponding `.html` files, so this
migration does not change the resulting HTML output in ReadTheDocs.
But it does fix apparent broken links on GitHub.
This doesn't prevent people from reading the documentation on GitHub
(so doesn't mitigate the fact that some rtd-specific syntax does not
render properly on GH), but it will prevent us from getting erroneous
issues reported about the hyperlinks not working.
Fixes: #11087.
Visually, #zoom_help_text acts like
.organization-settings-parent div:first-of-type when the Zoom option
is selected, but isn't treated as such.
No visual change with the #google_hangouts_domain change; just there to make
the code more readable/defensible.
The testing section is more appropriate, since it's fundamentally part
of our CI system.
While we're at it, fix the fact that we were linking to GitHub, not
ReadTheDocs, in the run-mypy output.
A bunch of details were out of date about the current state of Zulip
and thus what projects make sense; while this isn't likely to be our
final 2019 ideas list, this should help avoid confusion about what's
happening.
Due to copyright issues with potentially displaying Apple emojisets on
non-apple devices, as well as iamcal dropping support for the emojione
emojiset (see https://github.com/iamcal/emoji-data/pull/142), we are
dropping (perhaps temporarily) support for allowing users to switch
emojisets in Zulip.
This commit just hides the feature from the user but leaves most of
the infrastructure in place so that in the future if we decide to
re-enable the support we will not need to redo the infrastructure work
(some JS-side code is deleted, mostly because we'll want to re-add the
feature using the do_settings_change infrastructure anyway).
The most likely emoji set to add is the legacy "blobs" Google emoji
set, since it seems popular with some users.
Tweaked by tabbott to remove some additional JS code and update the
changelog.
The is_private flag is intended to be set if recipient type is
'private'(1) or 'huddle'(3), otherwise i.e if it is 'stream'(2), it
should be unset.
This commit adds a database index for the is_private flag (which we'll
need to use it). That index is used to reset the flag if it was
already set. The already set flags were due to a previous removal of
is_me_message flag for which the values were not cleared out.
For now, the is_private flag is always 0 since the really hard part of
this migration is clearing the unspecified previous state; future
commits will fully implement it actually doing something.
History: Migration rewritten significantly by tabbott to ensure it
runs in only 3 minutes on chat.zulip.org. A key detail in making that
work was to ensure that we use the new index for the queries to find
rows to update (which currently requires the `order_by` and `limit`
clauses).
This adds a new settings, SOCIAL_AUTH_SUBDOMAIN, which specifies which
domain should be used for GitHub auth and other python-social-auth
backends.
If one is running a single-realm Zulip server like chat.zulip.org, one
doesn't need to use this setting, but for multi-realm servers using
social auth, this fixes an annoying bug where the session cookie that
python-social-auth sets early in the auth process on the root domain
ends up masking the session cookie that would have been used to
determine a user is logged in. The end result was that logging in
with GitHub on one domain on a multi-realm server like zulipchat.com
would appear to log you out from all the others!
We fix this by moving python-social-auth to a separate subdomain.
Fixes: #9847.
I only renamed references that I thought were absolutely necessary
and only if the resulting sentence structure wasn't awkward.
If the renaming resulted in awkward structure, I replaced the term
"webhook" with "integration" (but only in some very obvious cases).
Fixes#9500.
This should make it easier to find the templates that are actually
part of the core webapp, instead of having them all mixed together
with the portico pages.
These aren't perfect -- in particular "core chat experience" can
probably be broken up -- but I think they help in making a quick skim
work for getting some sense of what the changes are.
This change just reorders and adds headings, with virtually no wording
changes.
Steve Howell
Tim Abbott
Anders Kaseorg
Ryan Rehman
Stefan Weil
Yashwant
Rohitt Vashishtha
Mateusz Mandera
David Rosa
Puneeth Chaganti
Rafid Aslam
Kanishk Kakar
Marco Burstein
Sahil Batra
Samuel Searles-Bryant
Rishi Gupta
Vishnu Ks
Abhigyan Khaund
Priyanshu Singh
Harshit Bansal
Shubham Padia
Aditya Bansal
Eeshan Garg
Abhigyan Khaund
Greg Price