Match the order of the variables between `default_settings.py` and
`settings.py`, and move the defaults into `default_settings.py` so
the section does not require any uncommented lines in `settings.py` if
LDAP is not in use.
Merge the two "misc" sections into one and place all the service
configurations next to each other. Place the TERMS_OF_SERVICE
and PRIVACY_POLICY at the bottom of the "misc" section.
Boto3 does not allow setting the endpoint url from
the config file. Thus we create a django setting
variable (`S3_ENDPOINT_URL`) which is passed to
service clients and resources of `boto3.Session`.
We also update the uploads-backend documentation
and remove the config environment variable as now
AWS supports the SIGv4 signature format by default.
And the region name is passed as a parameter instead
of creating a config file for just this value.
Fixes#16246.
The apple developer webapp consistently refers this App ID. So,
this clears any confusion that can occur.
Since python social auth only requires us to include App ID in
_AUDIENCE(a list), we do that in computed settings making it easier for
server admin and we make it much clear by having it set to
APP_ID instead of BUNDLE_ID.
Use the default configuration, which catches Error logging and
exceptions. This is placed in `computed_settings.py` to match the
suggested configuration from Sentry[1], which places it in `settings.py`
to ensure it is consistently loaded early enough.
It is placed behind a check for SENTRY_DSN soas to not incur the
additional overhead of importing the `sentry_sdk` modules if Sentry is
not configured.
[1] https://docs.sentry.io/platforms/python/django/
I believe the Bundle ID (aka App ID) and Services ID have meaning only
relative to a specific Team ID. In particular, in some places in the
developer.apple.com UI, they're displayed in a fully-qualified form
like "ABCDE12345.com.example.app", where "com.example.app" is the
App ID or Services ID and ABCDE12345 is the Team ID.
Adds the ability to set a SAML attribute which contains a
list of subdomains the user is allowed to access. This allows a Zulip
server with multiple organizations to filter using SAML attributes
which organization each user can access.
Cleaned up and adapted by Mateusz Mandera to fit our conventions and
needs more.
Co-authored-by: Mateusz Mandera <mateusz.mandera@zulip.com>
This prevents memcached from automatically appending the hostname to
the username, which was a source of problems on servers where the
hostname was changed.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
Automatically generated by the following script, based on the output
of lint with flake8-comma:
import re
import sys
last_filename = None
last_row = None
lines = []
for msg in sys.stdin:
m = re.match(
r"\x1b\[35mflake8 \|\x1b\[0m \x1b\[1;31m(.+):(\d+):(\d+): (\w+)", msg
)
if m:
filename, row_str, col_str, err = m.groups()
row, col = int(row_str), int(col_str)
if filename == last_filename:
assert last_row != row
else:
if last_filename is not None:
with open(last_filename, "w") as f:
f.writelines(lines)
with open(filename) as f:
lines = f.readlines()
last_filename = filename
last_row = row
line = lines[row - 1]
if err in ["C812", "C815"]:
lines[row - 1] = line[: col - 1] + "," + line[col - 1 :]
elif err in ["C819"]:
assert line[col - 2] == ","
lines[row - 1] = line[: col - 2] + line[col - 1 :].lstrip(" ")
if last_filename is not None:
with open(last_filename, "w") as f:
f.writelines(lines)
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
This implementation overrides some of PSA's internal backend
functions to handle `state` value with redis as the standard
way doesn't work because of apple sending required details
in the form of POST request.
Includes a mixin test class that'll be useful for testing
Native auth flow.
Thanks to Mateusz Mandera for the idea of using redis and
other important work on this.
Documentation rewritten by tabbott.
Co-authored-by: Mateusz Mandera <mateusz.mandera@zulip.com>
This reimplements our Zoom video call integration to use an OAuth
application. In addition to providing a cleaner setup experience,
especially on zulipchat.com where the server administrators can have
done the app registration already, it also fixes the limitation of the
previous integration that it could only have one call active at a time
when set up with typical Zoom API keys.
Fixes#11672.
Co-authored-by: Marco Burstein <marco@marco.how>
Co-authored-by: Tim Abbott <tabbott@zulipchat.com>
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
Through the limit_to_subdomains setting on IdP dicts it's now possible
to limit the IdP to only allow authenticating to the specified realms.
Fixes#13340.
"Zulip Voyager" was a name invented during the Hack Week to open
source Zulip for what a single-system Zulip server might be called, as
a Star Trek pun on the code it was based on, "Zulip Enterprise".
At the time, we just needed a name quickly, but it was never a good
name, just a placeholder. This removes that placeholder name from
much of the codebase. A bit more work will be required to transition
the `zulip::voyager` Puppet class, as that has some migration work
involved.
This legacy cross-realm bot hasn't been used in several years, as far
as I know. If we wanted to re-introduce it, I'd want to implement it
as an embedded bot using those common APIs, rather than the totally
custom hacky code used for it that involves unnecessary queue workers
and similar details.
Fixes#13533.
The url scheme is now /accounts/login/social/saml/{idp_name} to initiate
login using the IdP configured under "idp_name" name.
display_name and display_logo (the name and icon to show on the "Log in
with" button) can be customized by adding the apprioprate settings in
the configured IdP dictionaries.
This fixes a collection of bugs surrounding LDAP configurations A and
C (i.e. LDAP_APPEND_DOMAIN=None) with EmailAuthBackend also enabled.
The core problem was that our desired security model in that setting
of requiring LDAP authentication for accounts managed by LDAP was not
implementable without a way to
Now admins can configure an LDAPSearch query that will find if there
are users in LDAP that have the email address and
email_belongs_to_ldap() will take advantage of that - no longer
returning True in response to all requests and thus blocking email
backend authentication.
In the documentation, we describe this as mandatory configuration for
users (and likely will make it so soon in the code) because the
failure modes for this not being configured are confusing.
But making that change is pending work to improve the relevant error
messages.
Fixes#11715.
There are a few outstanding issues that we expect to resolve beforce
including this in a release, but this is good checkpoint to merge.
This PR is a collaboration with Tim Abbott.
Fixes#716.
Fixes#9401.
This adds a FAKE_EMAIL_DOMAIN setting, which should be used if
EXTERNAL_HOST is not a valid domain, and something else is needed to
form bot and dummy user emails (if email visibility is turned off).
It defaults to EXTERNAL_HOST.
get_fake_email_domain() should be used to get this value. It validates
that it's correctly set - that it can be used to form valid emails.
If it's not set correctly, an exception is raised. This is the right
approach, because it's undesirable to have the server seemingly
peacefully operating with that setting misconfigured, as that could
mask some hidden sneaky bugs due to UserProfiles with invalid emails,
which would blow up the moment some code that does validate the emails
is called.
This replaces the two custom Google authentication backends originally
written in 2012 with using the shared python-social-auth codebase that
we already use for the GitHub authentication backend. These are:
* GoogleMobileOauth2Backend, the ancient code path for mobile
authentication last used by the EOL original Zulip Android app.
* The `finish_google_oauth2` code path in zerver/views/auth.py, which
was the webapp (and modern mobile app) Google authentication code
path.
This change doesn't fix any known bugs; its main benefit is that we
get to remove hundreds of lines of security-sensitive semi-duplicated
code, replacing it with a widely trusted, high quality third-party
library.
Jitsi Meet is the correct name for the product we integrate with. There is
one other reference to Jitsi, but it's in the db and will require a
migration.
Fixes#12132.
Realm setting to disable avatar changes is already present.
The `AVATAR_CHANGES_DISABLED` setting now follows the same
2-setting model as `NAME_CHANGES_DISABLED`.
The main point here is that you should use a symlink rather than
changing it, since it's more maintenance work to update our nginx
configuration to use an alternative path than to just create a
symbolic link.
Fixes#12157.
These previously lived in Optional settings, which generally caused
users to not read it.
(Also do a bit of reorganization of the "optional settings" area).
This renames references to user avatars, bot avatars, or organization
icons to profile pictures. The string in the UI are updated,
in addition to the help files, comments, and documentation. Actual
variable/function names, changelog entries, routes, and s3 buckets are
left as-is in order to avoid introducing bugs.
Fixes#11824.
This setting splits away part of responsibility from THUMBOR_URL.
Now on, this setting will be responsible for controlling whether
we thumbnail images or not by asking bugdown to render image links
to hit our /thumbnail endpoint. This is irrespective of what
THUMBOR_URL is set to though ideally THUMBOR_URL should be set
to point to a running thumbor instance.
This should make life a lot more convenient for organizations that use
the LDAP integration and have their avatars in LDAP already.
This hasn't been end-to-end tested against LDAP yet, so there may be
some minor revisions, but fundamentally, it works, has automated
tests, and should be easy to maintain.
Fixes#286.
The term "username" confusingly refers both to the Django concept of
"username" (meaning "the name the user types into the login form") and
a concept the admin presumably already has in their existing
environment; which may or may not be the same thing, and in fact this
is where we document the admin's choice of whether and how they should
correspond. The Django concept in particular isn't obvious, and is
counterintuitive when it means something like an email address.
Explicitly explain the Django "username" concept, under the name of
"Zulip username" to take responsibility for our choice of how it's
exposed in the settings interface. Then use an explicit qualifier,
like "LDAP username", whenever referring to some other notion of
username. And make a pass over this whole side of the instructions,
in particular for consistent handling of these concepts.
Expand on a few things that tend to confuse people (especially the
`%(user)s` thing); move the `LDAPSearchUnion` example out to docs;
adjust the instructions to fit a bit better in their new docs/ home.
This makes it easier to iterate on these, and to expand supplemental
information (like troubleshooting, or unusual configurations) without
further straining the already-dauntingly-long settings.py.
It also makes it easier to consult the instructions while editing the
secrets file, or testing things, etc. -- most admins will find it more
natural to keep a browser open somewhere than a second terminal.
Various pieces of our thumbor-based thumbnailing system were already
merged; this adds the remaining pieces required for it to work:
* a THUMBOR_URL Django setting that controls whether thumbor is
enabled on the Zulip server (and if so, where thumbor is hosted).
* Replaces the overly complicated prototype cryptography logic
* Adds a /thumbnail endpoint (supported both on web and mobile) for
accessing thumbnails in messages, designed to support hosting both
external URLs as well as uploaded files (and applying Zulip's
security model for access to thumbnails of uploaded files).
* Modifies bugdown to, when THUMBOR_URL is set, render images with the
`src` attribute pointing /thumbnail (to provide a small thumbnail
for the image), along with adding a "data-original" attribute that
can be used to access the "original/full" size version of the image.
There are a few things that don't work quite yet:
* The S3 backend support is incomplete and doesn't work yet.
* The error pages for unauthorized access are ugly.
* We might want to rename data-original and /thumbnail?size=original
to use some other name, like "full", that better reflects the fact
that we're potentially not serving the original image URL.
This commits adds the necessary puppet configuration and
installer/upgrade code for installing and managing the thumbor service
in production. This configuration is gated by the 'thumbor.pp'
manifest being enabled (which is not yet the default), and so this
commit should have no effect in a default Zulip production environment
(or in the long term, in any Zulip production server that isn't using
thumbor).
Credit for this effort is shared by @TigorC (who initiated the work on
this project), @joshland (who did a great deal of work on this and got
it working during PyCon 2017) and @adnrs96, who completed the work.
This adds a new settings, SOCIAL_AUTH_SUBDOMAIN, which specifies which
domain should be used for GitHub auth and other python-social-auth
backends.
If one is running a single-realm Zulip server like chat.zulip.org, one
doesn't need to use this setting, but for multi-realm servers using
social auth, this fixes an annoying bug where the session cookie that
python-social-auth sets early in the auth process on the root domain
ends up masking the session cookie that would have been used to
determine a user is logged in. The end result was that logging in
with GitHub on one domain on a multi-realm server like zulipchat.com
would appear to log you out from all the others!
We fix this by moving python-social-auth to a separate subdomain.
Fixes: #9847.
This moves the documentation for this feature out of
prod_settings_template.py, so that we can edit it more easily.
We also add a bucket policy, which is part of what one would want to
use this in production.
This addresses much, but not all, of #9361.
This should help a bit more in making this file navigable.
I think there's further work that could be done to organize the
settings better: e.g., group LDAP with the auth section; separate
resource limits, from debugging and error reporting, from configuring
service dependencies like Redis and Rabbit. That'd require reordering
many settings, and also taking a closer look at many settings one by
one in order to do a good job. Leaving that for another day.
I've found this file hard to navigate for a while. We actually have a
little hierarchy of section headings which applies to a lot of the
file already; make the boundaries bolder.
Add a clear heading, and use fewer words and simpler sentences. Also
explain the password thing a bit more, and put that more inline next
to the username.
Also, on checking the Django docs, the default for EMAIL_USE_TLS
is False and for EMAIL_PORT is 25. So most admins, certainly any that
are using an SMTP service on the public Internet (that is at all
decently run), will need to set those settings. Mention that.
This comment didn't really explain things unless you were looking at
the code, and/or had a strong enough association for what "cn" means
that it was obvious what must be meant. Maybe this will be clearer.
There is one other meaningful key, which is optional: "short_name",
for which I guess a typical value if supplied would be "uid" or
"userid". I'm not sure we really do much with a user's `short_name`,
though, so didn't add a comment for it. When this key is omitted,
we set the user's `short_name` to the same thing as `full_name`.
Different formats for configuration files have a wide variety of ways
of representing lists; so if you're not accustomed to Python syntax,
or aren't thinking of this file as Python code, the syntax for several
ALLOWED_HOSTS entries may not be obvious. And this setting is one
that an admin is likely to want to touch quite early in using Zulip.
So, demonstrate a multi-element list.
For similar reasons, demonstrate an IP address. This one is in a
range reserved for documentation (by RFC 5737), like `example.com`.
Gmail is a bad example for outbound email; use a generic example.
Also leave the `= None` default out of the config file, as it's
redundant with DEFAULT_SETTINGS in our internal settings.py ; and
explain in the latter why we don't mention the other SMTP settings.
The warning here means that the admin can't really act on this yet if
they want to disable email auth, which is likely among admins that
want to make any changes here. And for admins who don't, this is an
extra thing to read and make a decision about before they can get a
server running. See #6985.
Conversely, we already discuss auth backends right at the top of the
`prod-customize` doc, which is linked under "Next steps" at the end of
these instructions.
The warning about EmailAuthBackend is important; but we can move it to
the config file right next to the setting, and then it's available
right when it's actionable, which is if the admin is actually thinking
about changing the setting.
For some historical reason we'd had the Postgres documentation on
valid SSL modes copied into the Zulip settings.py template file. This
fixes that historical artifact.
The control panel on the Google side doesn't seem to match the
instructions we have; it looks pretty 2017 to me, so I imagine
it's had a redesign since the instructions were written.
Also, in dev, EXTERNAL_HOST is now a port on zulipdev.com, not on
localhost.
Update these instructions for those developments, and edit lightly.
In dev, recommend setting in `dev_settings` instead of in
`prod_settings_template`; that feels to me a little more reflective of
the actual intent, and the effect should be equivalent.
This means one fewer thing the admin typically needs to read, absorb,
and make a decision about at install time.
The one way this change could hypothetically cause trouble is if the
admin wants to keep subdomains of EXTERNAL_HOST out of ALLOWED_HOSTS.
But while the subdomains often won't exist as domain names, it's hard
to imagine the situation in which they would exist but be under
someone else's control, or be doing something other than serving
Zulip realms.
The rules here are fuzzy, and it's quite possible none of Zulip's emails
need an address at all. Every country has its own rules though, which makes
it hard to tell. In general, transactional emails do not need an address,
and marketing emails do.
The original "quality score" was invented purely for populating
our password-strength progress bar, and isn't expressed in terms
that are particularly meaningful. For configuration and the core
accept/reject logic, it's better to use units that are readily
understood. Switch to those.
I considered using "bits of entropy", defined loosely as the log
of this number, but both the zxcvbn paper and the linked CACM
article (which I recommend!) are written in terms of the number
of guesses. And reading (most of) those two papers made me
less happy about referring to "entropy" in our terminology.
I already knew that notion was a little fuzzy if looked at
too closely, and I gained a better appreciation of how it's
contributed to confusion in discussing password policies and
to adoption of perverse policies that favor "Password1!" over
"derived unusual ravioli raft". So, "guesses" it is.
And although the log is handy for some analysis purposes
(certainly for a graph like those in the zxcvbn paper), it adds
a layer of abstraction, and I think makes it harder to think
clearly about attacks, especially in the online setting. So
just use the actual number, and if someone wants to set a
gigantic value, they will have the pleasure of seeing just
how many digits are involved.
(Thanks to @YJDave for a prototype that the code changes in this
commit are based on.)
Anders Kaseorg
Alex Vandiver
Nikhil Maske
Sourabh Rana
ryanreh99
Tim Abbott
savish
Dinesh
Greg Price
Brainrecursion
Felix
Anders Kaseorg
Mateusz Mandera
Dinesh
Tim Abbott
Vishnu KS
Wyatt Hoodes
Harshit Bansal
Puneeth Chaganti
vinitS101
Rishi Gupta
Shubham Padia
Ben Muschol
Anders Kaseorg
Aditya Bansal
seresheim
Luis Silva
Joshua Schmidlkofer
Reid Barton
Maarten Rijke