Adds the ability to set a SAML attribute which contains a
list of subdomains the user is allowed to access. This allows a Zulip
server with multiple organizations to filter using SAML attributes
which organization each user can access.
Cleaned up and adapted by Mateusz Mandera to fit our conventions and
needs more.
Co-authored-by: Mateusz Mandera <mateusz.mandera@zulip.com>
This adds a convenient way to review the upgrade notes for all Zulip
releases that one is upgrading across.
I thought about moving all the upgrade notes to a common section, but
in some cases the language is clearly explaining changes in the
release that are not duplicated elsewhere, and I think it reads better
having them inline alongisde related changes.
We only use this in a few places, but they're really important places
for understanding the types in the codebase, and so it's worth having
a bit of expository documentation explaining how we use it.
(And I expect we'll add more with time).
We have migrated ellipsis-v from chevron for sidebars,
it would be better to define it in glossary as it is gonna replace
"chevron" during conversations.
Replace word 'chevron' with 'ellipsis'(because of the
recent migration to ellipsis for sidebars menu icons)
and mention where to click more specifically for sidebar
popover testing.
65774e1c4f switched from using the bundled check_postgres.pl to using
the version from packages; the file itself remained, however.
Remove it, and clean up references to it.
Fixes#15389.
• Specify disabled rather than enabled protocols, so as not to disable
TLS 1.3.
• Provide an explicit cipher suite list (Mozilla intermediate config
version 5.4).
• Respect the browser’s preferred cipher suite ordering over the
server’s.
• Use FFDHE2048 Diffie-Hellman parameters.
• Disable SSL session tickets.
(SSL stapling is also recommended but SSLStaplingCache cannot be
configured inside a <VirtualHost> block.)
Signed-off-by: Anders Kaseorg <anders@zulip.com>
This is a prep commit for replacement of chevron
from sidebars.
This commit will add ellipsis-v icon in svg format downloaded
from font-awesome 5. This has to be done because font-awesome 4.7
(the version we are using) does not have this icon with
circular dots.
And font-awesome 5 as a whole doesn't make sense to upgrade to because
it's intentionally semi-crippled as part of their business plan.
Also include entry in THIRDYPARTY and Licence details.
Fixes#2665.
Regenerated by tabbott with `lint --fix` after a rebase and change in
parameters.
Note from tabbott: In a few cases, this converts technical debt in the
form of unsorted imports into different technical debt in the form of
our largest files having very long, ugly import sequences at the
start. I expect this change will increase pressure for us to split
those files, which isn't a bad thing.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
Automatically generated by the following script, based on the output
of lint with flake8-comma:
import re
import sys
last_filename = None
last_row = None
lines = []
for msg in sys.stdin:
m = re.match(
r"\x1b\[35mflake8 \|\x1b\[0m \x1b\[1;31m(.+):(\d+):(\d+): (\w+)", msg
)
if m:
filename, row_str, col_str, err = m.groups()
row, col = int(row_str), int(col_str)
if filename == last_filename:
assert last_row != row
else:
if last_filename is not None:
with open(last_filename, "w") as f:
f.writelines(lines)
with open(filename) as f:
lines = f.readlines()
last_filename = filename
last_row = row
line = lines[row - 1]
if err in ["C812", "C815"]:
lines[row - 1] = line[: col - 1] + "," + line[col - 1 :]
elif err in ["C819"]:
assert line[col - 2] == ","
lines[row - 1] = line[: col - 2] + line[col - 1 :].lstrip(" ")
if last_filename is not None:
with open(last_filename, "w") as f:
f.writelines(lines)
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
This implementation overrides some of PSA's internal backend
functions to handle `state` value with redis as the standard
way doesn't work because of apple sending required details
in the form of POST request.
Includes a mixin test class that'll be useful for testing
Native auth flow.
Thanks to Mateusz Mandera for the idea of using redis and
other important work on this.
Documentation rewritten by tabbott.
Co-authored-by: Mateusz Mandera <mateusz.mandera@zulip.com>
Now that we have production support for Ubuntu Focal, we update the
documentation to state our support for it.
(We also drop deprecated Xenial and Stretch from supported platforms).
We're migrating to using the cleaner zulip.com domain, which involves
changing all of our links from ReadTheDocs and other places to point
to the cleaner URL.
Calling jwt.decode without an algorithms list raises a
DeprecationWarning. This is for protecting against
symmetric/asymmetric key confusion attacks.
This is a backwards-incompatible configuration change.
Fixes#15207.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
Previously, it was unnecessarily difficult to parse the sentence to
determine that "HTTP response" and "internal state of the server
following the request" are the coherent ideas. Even if length wasn't
an issue, e.g. "... and checking both the A and B are correct", the
sentence still feels a bit fragile without a "that".
Since the second phrase is indeed relatively long, and "internal state
of the server" is a reasonable guess for the second coherent idea, the
"the" helps to reset the reader's expectation about where the next
coherent idea starts, and ends.
Lastly, having "both" in front of the two phrases encourages an
assumption that they're shorter (which is especially problematic for
the second phrase), while having it at the end of the sentence helps
to anchor the end of the second phrase; this is especially true since
the absence of "both" before that point encourages an assumption that
you haven't finished reading yet, given that two things have been
mentioned.
datetime.timezone is available in Python ≥ 3.2. This also lets us
remove a pytz dependency from the PostgreSQL scripts.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
Fixes this warning:
/srv/zulip/docs/conf.py:337: RemovedInSphinx40Warning: The app.add_stylesheet() is deprecated. Please use app.add_css_file() instead.
app.add_stylesheet('theme_overrides.css') # path relative to _static
https://www.sphinx-doc.org/en/3.x/extdev/deprecated.html
Signed-off-by: Anders Kaseorg <anders@zulip.com>
This reimplements our Zoom video call integration to use an OAuth
application. In addition to providing a cleaner setup experience,
especially on zulipchat.com where the server administrators can have
done the app registration already, it also fixes the limitation of the
previous integration that it could only have one call active at a time
when set up with typical Zoom API keys.
Fixes#11672.
Co-authored-by: Marco Burstein <marco@marco.how>
Co-authored-by: Tim Abbott <tabbott@zulipchat.com>
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
This commit replaces fa-file-text-o with fa-file-code-o which is a
better signal for the "view source" action. It also deletes a single
line comment that had suggested the change once we moved into font
awesome 4, which Aditya Bansal <adi.bansal241996@gmail.com> helped
out in doing, first via
91962aa6ab and most recently via
75ae94e459 with several commits in
between.
This should help ensure everyone uses the SSH key approach for Git
authentication; the HTTPS one is basically unusable as one has to
provide one's GitHub password after every command.
This section at the top was clearly written before the documentation
at the bottom existed, and hasn't been updated to point to the
now-existent docs below.
Add the link, rather than directing to #production-help.
The problem is not the list comprehension, as the previous wording
implied, but rather the fact that data is needed from the linked
table.
Be explicit about _what_ in the QuerySet API is helpful for addressing
this -- namely, use of `select_related`.
The new tools now have more concise, more parallel names:
- rebuild-dev-database
- rebuild-test-database
The actual implementations are still pretty different:
rebuild-dev-database:
mostly delegates to 5 management scripts
rebuild-test-database:
is a very thin wrapper for generate-fixtures
We'll try to clean that up a bit soon.
We no longer need to maintain duplicate code
related to where we set up the emoji
cache directory.
And we no longer need two extra steps for
people doing advanced (i.e. manual) setup.
There was no clear benefit to having provision
build the cache directory for `build_emoji`,
when it was easy to make `build_emoji` more
self-sufficient. The `build_emoji` tool
was already importing the library that has
`run_as_root`, and it was already responsible
for 99% of the create-directory kind of tasks.
(We always call `build_emoji` unconditionally from
`provision`, so there's no rationale in terms
of avoiding startup time or something.)
ASIDE:
Its not completely clear to me why we need
to put this directory in "/srv", instead of
somewhere more local (like we already do for
Travis), but maybe it's just to be like
its siblings in "/srv":
node_modules
yarn.lock
zulip-emoji-cache
zulip-npm-cache
zulip-py3-venv
zulip-thumbor-venv
zulip-venv-cache
zulip-yarn
I guess the caches that we keep in var are
dev-only, although I think some of what's under
`zulip-emoji-cache` is also dev-only in nature?
./var/webpack-cache
./var/mypy-cache
In `docs/subsystems/emoji.md` we say this:
```
The `build_emoji` tool generates the set of files under
`static/generated/emoji` (or really, it generates the
`/srv/zulip-emoji-cache/<sha1>/emoji` tree, and
`static/generated/emoji` is a symlink to that tree;we do this in
order to cache old versions to make provisioning and production
deployments super fast in the common case that we haven't changed the
emoji tooling). [...]
```
I don't really understand that rationale for the development
case, since `static/generated` is as much ignored by `git` as
'/srv' is, without the complications of needing `sudo` to create it.
And in production, I'm not sure how much time we're really saving,
as it takes me about 1.4s to fully rebuild the cache in dev, not to
mention we're taking on upgrade risk by sharing files between versions.
Also make sure our documentation for upgrading is reasonable for
Stretch => Buster.
Our reasoning for deprecating support for these releases is as follows:
* Ubuntu 16.04 Xenial reached desktop EOL last year; and will reach
EOL on the server in about a year.
* Debian Stretch will each EOL in 2020 (the precise date is unclear in
Debian's documentation, but based on past precedent it's in the next
few months, perhaps July 2020).
https://wiki.debian.org/DebianReleases#Production_Releases
* Both Ubuntu 16.04 and Debian Stretch use Python 3.5 as the system
Python, which will reach EOL in September 2020 (and we're already
seeing various third-party dependencies that we use drop support for
them).
* While there is LTS support for these older releases, it's not clear it's
going to be worth the added engineering effort for us to maintain EOL
releases of the base OSes that we support.
* We (now) have clear upgrade instructions for moving to Debian Buster
and Ubuntu 18.04.
This defends against cross-origin session fixation attacks. Renaming
the cookies means this one-time upgrade will have the unfortunate side
effect of logging everyone out, but they’ll get more secure sessions
in return.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
Instructions were added by doing the setup on Ubuntu 18.04 WSL 2.
While the setup should be similar for other distributions supported by
our `./tools/provision` script inside WSL, it has not been tested.
Polished by tabbott.
Generated by `pyupgrade --py3-plus --keep-percent-format` on all our
Python code except `zthumbor` and `zulip-ec2-configure-interfaces`,
followed by manual indentation fixes.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
This setting is being overridden by the frontend since the last
commit, and the security model is clearer and more robust if we don't
make it appear as though the markdown processor is handling this
issue.
Co-authored-by: Tim Abbott <tabbott@zulipchat.com>
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
I found the solution by simply trying out EMAIL_USE_SSL and it
worked. I had problems with sending emails (did not work at all, there
wasn't even a connection going on - I checked with tcpdump. Then I
found this: To use port 465, you need to call
smtplib.SMTP_SSL(). Currently, it looks like Django only uses
smtplib.SMTP() (source: https://code.djangoproject.com/ticket/9575).
Fixes#14350.
This returns us to a consistent logging format regardless of whether
the request is authenticated.
We also update some log examples in docs to be consistent with the new
style.
Those docs were outdated and no longer represantative of how things
work. upgrade-python-dependencies and unupgradable.json are no longer a
thing, so the entire paragraph about them should be removed.
Then the requirements/README.md file is refreshed a bit to be more
accurate.
For historical reasons we were creating Recipient
objects at some point in the typing-notifications
codepath. Now we just work with UserProfiles.
This removes some queries, as indicated by
the change to `len(queries)` in a couple of the
tests.
The one subtle thing that changes here is huddles.
If user 10 sends a typing notification that they
are talking to users 20 and 30, there might not
actually be a huddle for users 10/20/30, but
we were actually creating huddles on the fly!
There is no need to create huddles just for
typing notifications, since we don't even
share huddle ids with our clients. The clients
just infer the huddles.
Some of the code that gets killed off here as
somewhat "collateral damage" is some
defensive code related to formerly supporting streams
in typing indicators. The support for streams
was killed off almost as soon as we released
the feature, and the codepath is pretty clearly
user-centric at this point.
The only clients that should use the typing
indicators endpoint are our internal clients,
and they should send a JSON-formatted list
of user_ids.
Unfortunately, we still have some older versions
of mobile that still send emails.
In this commit we fix non-user-facing things
like docs and tests to promote the user_ids
interface that has existed since about version
2.0 of the server.
One annoyance is that we documented the
typing endpoint with emails, instead of the
more modern user_ids, which may have delayed
mobile converting to user_ids (and which
certainly caused confusion). It's trivial
to update the docs, but we need to short
circuit one assertion in the openapi tests.
We also clean up the test structure for the
typing tests:
TypingHappyPathTest.test_start_to_another_user
TypingHappyPathTest.test_start_to_multiple_recipients
TypingHappyPathTest.test_start_to_self
TypingHappyPathTest.test_start_to_single_recipient
TypingHappyPathTest.test_stop_to_another_user
TypingHappyPathTest.test_stop_to_self
TypingValidateOperatorTest.test_invalid_parameter
TypingValidateOperatorTest.test_missing_parameter
TypingValidateUsersTest.test_argument_to_is_not_valid_json
TypingValidateUsersTest.test_bogus_user_id
TypingValidateUsersTest.test_empty_array
TypingValidateUsersTest.test_missing_recipient
TypingValidationHelpersTest.test_recipient_for_user_ids
TypingValidationHelpersTest.test_recipient_for_user_ids_non_existent_id
TypingLegacyMobileSupportTest.test_legacy_email_interface
This has for a while been our only active Google Groups mailing list,
and given that folks will guess security@ as our security contact, we
might as well just standardize on that.
Also tweak some ambiguous text; it wouldn't be appropriate for us to
issue a CVE for e.g. an operational issue only affecting us.
isort 5 knows not to reorder imports across function calls, so this
will stop isort from breaking our code.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
- I fixed a typo with "lowerecase"
- I elaborated on the prefix before elaborating
on the rest of the message (i.e. went in correct
order).
- I split out the provision example (since we
talk about it some depth).
- I added more positive examples.
- I removed the distracting italics around the
good commit messages.
- I moved the "gather_subscriptions" commit to
the bottom of the list, since we elaborate
on that below the list.
This includes an experiment of having a draft of the 2.1.3 changelog,
which is helpful in avoiding duplication with the 2.2.0 changelog for
items we're planning to backport.
Django 2.2.x is the next LTS release after Django 1.11.x; I expect
we'll be on it for a while, as Django 3.x won't have an LTS release
series out for a while.
Because of upstream API changes in Django, this commit includes
several changes beyond requirements and:
* urls: django.urls.resolvers.RegexURLPattern has been replaced by
django.urls.resolvers.URLPattern; affects OpenAPI code and related
features which re-parse Django's internals.
https://code.djangoproject.com/ticket/28593
* test_runner: Change number to suffix. Django changed the name in this
ticket: https://code.djangoproject.com/ticket/28578
* Delete now-unnecessary SameSite cookie code (it's now the default).
* forms: urlsafe_base64_encode returns string in Django 2.2.
https://docs.djangoproject.com/en/2.2/ref/utils/#django.utils.http.urlsafe_base64_encode
* upload: Django's File.size property replaces _get_size().
https://docs.djangoproject.com/en/2.2/_modules/django/core/files/base/
* process_queue: Migrate to new autoreload API.
* test_messages: Add an extra query caused by .refresh_from_db() losing
the .select_related() on the Realm object.
* session: Sync SessionHostDomainMiddleware with Django 2.2.
There's a lot more we can do to take advantage of the new release;
this is tracked in #11341.
Many changes by Tim Abbott, Umair Waheed, and Mateusz Mandera squashed
are squashed into this commit.
Fixes#10835.
webpack optimizes JSON modules using JSON.parse("{…}"), which is
faster than the normal JavaScript parser.
Update the backend to use emoji_codes.json too instead of the three
separate JSON files.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
This is a fairly involved set of changes, including changes that:
* Delete various legacy or semi-duplicated sections of testing.md.
Nobody needs to manually delete the postgres datbase anymore, as
reflected in the fact that the docs still mention postgres 9.1 from
Ubuntu Precise.
* Simplify the distracting heading section at the top of testing.md.
* Move content on manual testing to docs/development/using.md.
* Moves some content related to managing the database to
schema-migrations.md. (Resulting in some cleanups to that page as
well).
I ideally would have split this into smaller pieces.
This addresses confusion we had with some organizations where they
were surprised that with only LDAP enabled, the "invite more users"
feature was available.
Fixes#11685.
This legacy cross-realm bot hasn't been used in several years, as far
as I know. If we wanted to re-introduce it, I'd want to implement it
as an embedded bot using those common APIs, rather than the totally
custom hacky code used for it that involves unnecessary queue workers
and similar details.
Fixes#13533.
This correct various inaccuracies and adds a bulleted list structure
for better clarity.
I think there's a lot more that could be done here in the form of
linking to other pages, discussing restarting `run-dev.py`, etc.
Added a link from docs/development/using ("Using the Development
Environment") to ./authentication ("Authentication in the development
environment") to help people working on the authentication systems
or anyone who needs an API key.
Separate using.html into Server/Web/Mobile sections so that readers
will find what they're looking for more quickly. Server is at the top
because it contains information relevant to web and mobile developers,
e.g., that the `run-dev.py` console output will provide useful errors.
Fixes#13655.
Zulip has had a small use of WebSockets (specifically, for the code
path of sending messages, via the webapp only) since ~2013. We
originally added this use of WebSockets in the hope that the latency
benefits of doing so would allow us to avoid implementing a markdown
local echo; they were not. Further, HTTP/2 may have eliminated the
latency difference we hoped to exploit by using WebSockets in any
case.
While we’d originally imagined using WebSockets for other endpoints,
there was never a good justification for moving more components to the
WebSockets system.
This WebSockets code path had a lot of downsides/complexity,
including:
* The messy hack involving constructing an emulated request object to
hook into doing Django requests.
* The `message_senders` queue processor system, which increases RAM
needs and must be provisioned independently from the rest of the
server).
* A duplicate check_send_receive_time Nagios test specific to
WebSockets.
* The requirement for users to have their firewalls/NATs allow
WebSocket connections, and a setting to disable them for networks
where WebSockets don’t work.
* Dependencies on the SockJS family of libraries, which has at times
been poorly maintained, and periodically throws random JavaScript
exceptions in our production environments without a deep enough
traceback to effectively investigate.
* A total of about 1600 lines of our code related to the feature.
* Increased load on the Tornado system, especially around a Zulip
server restart, and especially for large installations like
zulipchat.com, resulting in extra delay before messages can be sent
again.
As detailed in
https://github.com/zulip/zulip/pull/12862#issuecomment-536152397, it
appears that removing WebSockets moderately increases the time it
takes for the `send_message` API query to return from the server, but
does not significantly change the time between when a message is sent
and when it is received by clients. We don’t understand the reason
for that change (suggesting the possibility of a measurement error),
and even if it is a real change, we consider that potential small
latency regression to be acceptable.
If we later want WebSockets, we’ll likely want to just use Django
Channels.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
I added this tool a few years ago, and I did have
a vision for how it would improve our codebase, but
I can't remember exactly where I was going with it.
At this point the tool is just a little too noisy
to be helpful. An example of it creating confusion
was a recent PR where somebody was patching
user_circle_class in the PM list, and we already
had similar code in the buddy list, because they
use the same CSS. I mean, there was possibly a way
that the code could have been structured to remove
some of the duplication, but it probably would have
just moved the complexity around.
I just don't think it's worth maintaining the tool
at this point.
The well-known rowanj/gitx repository hasn't been updated since 2014.
Preferentially direct new contributors to gitx/gitx instead.
(We retain the rowanj repo as a fallback, since it has precompiled
releases available.)
This moves the mandatory configuration for options A/B/C into a single
bulleted list for each option, rather than split across two steps; I
think the result is significantly more readable.
It also fixes a bug where we suggested setting
AUTH_LDAP_REVERSE_EMAIL_SEARCH = AUTH_LDAP_USER_SEARCH in some cases,
whereas in fact it will never work because the parameters are
`%(email)s`, not `%(user)s`.
Also, now that one needs to set AUTH_LDAP_REVERSE_EMAIL_SEARCH, it
seems worth adding values for that to the Active Directory
instructions. Thanks to @alfonsrv for the suggestion.
This simplifies the RDS installation process to avoid awkwardly
requiring running the installer twice, and also is significantly more
robust in handling issues around rerunning the installer.
Finally, the answer for whether dictionaries are missing is available
to Django for future use in warnings/etc. around full-text search not
being great with this configuration, should they be required.
The previous documentation was essentially wrong, in that it
recommended copying certain settings that would cause significant
problems post-import if they were indeed copied.
Adds an explicit explanation to help contributors avoid common mistakes
like capitalization errors, missing trailing periods, and incorrectly
prefixing the name of a subsystem.
Fixes#1535.
This also rewrites the text to better explain what's happening. It's
likely further polish would be valuable, but that's true for the whole
"Troubleshooting" page.
This block of text was misplaced when we split the long
maintain-secure-update; article; we want it to be easy to find by
folks who are looking into error emails Zulip is sending.
This text is very old and hadn't been edited in a long time, in large
part because it was buried within old docs. This change cleans it up
to give accurate and better-organized information.
* Moves "Management commands" to a top-level section.
* Moves "Scalability" as a subsection at the bottom of "Requirements".
* Moves "Monitoring" as a subsections at the bottom of "Troubleshooting".
* Replaces "API and your Zulip URL" with a link to REST API docs. This
documentation text has been irrelevant for some time.
* Removes maintain-secure-upgrade from the TOC but the file remains to
avoid breaking old links from release blog posts and emails.
Updates the message editing process to do a local 'echo'.
On slow connections, now there is visual confirmation of the edit,
similar to when sending messages. The contains_backend_only_syntax
logic and check are the same as there.
We showing "(SAVING)" until the edit is completed, and on successful
edit, the word "(EDITED)" appears. There's likely useful future work
to do on making the animation experience nicer.
Substantially rewritten by tabbott to better handle corner cases and
communicate more clearly about what's happening.
Fixes: #3530.
We'll be soon documenting a production workflow that involves using
it, and that means it needs to live under scripts/ (since tools/ isn't
present in release tarballs).
- Moves "Authentication in the development environment" from subsystems
to "development/authentication.md".
- Moves "Renumbering migrations" to a section within "Schema migrations".
Merges the "Upgrades" section from production/maintain-secure-upgrade.md
with production/modifying-zulip.md.
Contains significant textual changes by tabbott to read more clearly.
- Merges the "Backups" section from production/maintain-secure-upgrade.md
with existing "Backups" section in production/export-and-import.md.
- Cleans up and makes content more clear/explicit.
- Adds short missing section on how to use wal-e configuration.
- Removes a lot of previously duplicate text explaining the difference between
the tools.
- Various textual tweaks by tabbott.
Fixes#13184 and resolves#293.
Documentation for tab switcher added under User documentation
features as a section at the bottom of "Tips and Warnings" article.
Text substantially rewritten by tabbott.
Fixes#12505
Brainrecursion
Felix
Tim Abbott
apxwn
SiddharthVarshney
Onur Temiz
Alex Vandiver
Anders Kaseorg
qnxor
Anders Kaseorg
Dinesh
Sharif Naas
arpit551
Mateusz Mandera
Mateusz Mandera
sahil839
YashRE42
Alex Vandiver
Steve Howell
Puneeth Chaganti
Alex Vandiver
Tim Abbott
Vishnu Ks
Rohitt Vashishtha
Steve Howell
Aman Agrawal
Ray Kraesig
thoemie
shubhamgupta2956
Ryan Rehman
Stefan Weil
Rike-Benjamin Schuppner
Yashwant
Vaibhav Raj Singh
rht
Chris Bobbe
Dinesh
xpac1985
Rodriq
Rodriq
David Rosa
Jack Tiggleman
Hashir Sarwar
Tlazypanda