Some admins setting up Zulip's LDAP auth against Active Directory see
a rather baffling error message: "In order to perform this operation a
successful bind must be completed on the connection". This happens
despite AUTH_LDAP_BIND_DN and auth_ldap_bind_password being set
perfectly correctly, and on a query that the `ldapsearch` command-line
tool performs quite happily.
Empirically, adding a setting like this to /etc/zulip/settings.py
resolves the issue:
AUTH_LDAP_CONNECTION_OPTIONS = {
ldap.OPT_REFERRALS: 0
}
Some useful, concise background on the LDAP "referral" concept is here:
https://docs.oracle.com/javase/jndi/tutorial/ldap/referral/overview.html
and a pertinent bit of docs for the underlying Python `ldap` client:
https://www.python-ldap.org/en/latest/faq.html
and some very helpful documentation for Active Directory:
https://docs.microsoft.com/en-us/windows/desktop/ad/referrals
Based on the docs above, the story appears to be something like this:
* This server has the information for part of the scope of our query
-- in particular it happens to have the information we actually want.
* But there are other areas ("subordinate domains") that our query is
in principle asking about, and this server doesn't know if there are
matches there, so it gives us a referral.
* And by default, python-ldap lets `libldap` run ahead and attempt to
bind to those referrals and do those queries too -- which raises an
error because, unlike Microsoft's "LDAP API", it doesn't reuse the
credentials.
So if we simply skip trying to follow the referrals, there's no
error... and we already have, from the original response, the answer
we actually need. That's what the `ldap.OPT_REFERRALS` option does.
There may be more complex situations where the referral really is
relevant, because the desired user info is split across servers. Even
then, unless an anonymous query will be acceptable, there's no point
in letting `libldap` follow the referral and setting this option is
still the right thing. When someone eventually comes to this bridge,
some code will be required to cross it, by following the referrals.
That code might look a bit like this (unfinished) example:
https://bugs.launchpad.net/nav/+bug/1209178
Manually tested by tabbott.
Fixes#343, which was effectively a report of the need for this
OPT_REFERRALS setting.
Fixes#349, since with this change, we no longer require tricky manual
configuration to get Active Directory up and running.
The term "username" confusingly refers both to the Django concept of
"username" (meaning "the name the user types into the login form") and
a concept the admin presumably already has in their existing
environment; which may or may not be the same thing, and in fact this
is where we document the admin's choice of whether and how they should
correspond. The Django concept in particular isn't obvious, and is
counterintuitive when it means something like an email address.
Explicitly explain the Django "username" concept, under the name of
"Zulip username" to take responsibility for our choice of how it's
exposed in the settings interface. Then use an explicit qualifier,
like "LDAP username", whenever referring to some other notion of
username. And make a pass over this whole side of the instructions,
in particular for consistent handling of these concepts.
Expand on a few things that tend to confuse people (especially the
`%(user)s` thing); move the `LDAPSearchUnion` example out to docs;
adjust the instructions to fit a bit better in their new docs/ home.
This makes it easier to iterate on these, and to expand supplemental
information (like troubleshooting, or unusual configurations) without
further straining the already-dauntingly-long settings.py.
It also makes it easier to consult the instructions while editing the
secrets file, or testing things, etc. -- most admins will find it more
natural to keep a browser open somewhere than a second terminal.
Fixes part of #10297.
Use FAKE_LDAP_NUM_USERS which specifies the number of LDAP users
instead of FAKE_LDAP_EXTRA_USERS which specified the number of
extra users.
Also use name for selecting form in casper tests
as form with action=new is present in both /new
and /accounts/new/send_confirm/ which breaks
test in CircleCI as
waitWhileVisible('form[action^="/new/"]) never stops
waiting.
This uses the recently introduced active_mobile_push_notification
flag; messages that have had a mobile push notification sent will have
a removal push notification sent as soon as they are marked as read.
Note that this feature is behind a setting,
SEND_REMOVE_PUSH_NOTIFICATIONS, since the notification format is not
supported by the mobile apps yet, and we want to give a grace period
before we start sending notifications that appear as (null) to
clients. But the tracking logic to maintain the set of message IDs
with an active push notification runs unconditionally.
This is designed with at-least-once semantics; so mobile clients need
to handle the possibility that they receive duplicat requests to
remove a push notification.
We reuse the existing missedmessage_mobile_notifications queue
processor for the work, to avoid materially impacting the latency of
marking messages as read.
Fixes#7459, though we'll need to open a follow-up issue for
using these data on iOS.
This uses the MockLDAP class of fakeldap to fake a ldap server, based
on the approach already used in the tests in `test_auth_backends.py`.
Adds the following settings:
- FAKE_LDAP_MODE: Lets user choose out of three preset configurations.
The default mode if someone erases the entry in settings is 'a'. The
fake ldap server is disable if this option is set to None.
- FAKE_LDAP_EXTRA_USERS: Number of extra users in LDAP directory beyond
the default 8.
Fixes#9934.
Generates ldap_dir based on the mode and the no. of extra users.
It supports three modes, 'a', 'b' and 'c', description for which
can be found in prod_settings_templates.py.
That value is necessary to configure anonymous binds in
django-auth-ldap, which are useful when we're using LDAP just to
populate the user database.
Fixes#10257.
This lets us have a slightly cleaner interface for cases where we need
a custom default value than an `or None` in the definition (which
might cause issues with other falsey values).
The autenticate function now follows the signature of
Django 2.0 https://github.com/django-auth-ldap/
django-auth-ldap/commit/27a8052b26f1d3a43cdbcdfc8e7dc0322580adae
Also AUTH_LDAP_CACHE_GROUPS is depricated in favor of
AUTH_LDAP_CACHE_TIMEOUT.
We already had a setting for whether these logs were enabled; now it
also controls which stream the messages go to.
As part of this migration, we disable the feature in dev/production by
default; it's not useful for most environments.
Fixes the proximal data-export issue reported in #10078 (namely, a
stream with nobody ever subscribed to having been created).
As part of our effort to change the data model away from each user
having a single API key, we're eliminating the couple requests that
were made from Django to Tornado (as part of a /register or home
request) where we used the user's API key grabbed from the database
for authentication.
Instead, we use the (already existing) internal_notify_view
authentication mechanism, which uses the SHARED_SECRET setting for
security, for these requests, and just fetch the user object using
get_user_profile_by_id directly.
Tweaked by Yago to include the new /api/v1/events/internal endpoint in
the exempt_patterns list in test_helpers, since it's an endpoint we call
through Tornado. Also added a couple missing return type annotations.
Input pills require a contenteditable div with a class named input
to fall inside the pill container. On converting the input tag into
a div, the size of the input decreases which is compensated by a
line-height of 40px. Comment above letter-spacing:normal was removed
as chrome and firefox do not change the letter-spacing to normal
for a div via the default browser stylesheet.
NOTE: Currently writing something into the div will call the action
corresponding to that key in the keyboard shortcuts. The input will
work fine once the pills have been initiated.
For the casper tests, for now, we just use the legacy search code.
When we change that, $.val() cannot be used on contenteditable div, so
$.html() will need to be used instead in select_item_via_typeahead.
This setting isn't intended to exist long term, but instead to make it
possible to merge our search pills code before we're ready to cut over
production environments to use it.
Various pieces of our thumbor-based thumbnailing system were already
merged; this adds the remaining pieces required for it to work:
* a THUMBOR_URL Django setting that controls whether thumbor is
enabled on the Zulip server (and if so, where thumbor is hosted).
* Replaces the overly complicated prototype cryptography logic
* Adds a /thumbnail endpoint (supported both on web and mobile) for
accessing thumbnails in messages, designed to support hosting both
external URLs as well as uploaded files (and applying Zulip's
security model for access to thumbnails of uploaded files).
* Modifies bugdown to, when THUMBOR_URL is set, render images with the
`src` attribute pointing /thumbnail (to provide a small thumbnail
for the image), along with adding a "data-original" attribute that
can be used to access the "original/full" size version of the image.
There are a few things that don't work quite yet:
* The S3 backend support is incomplete and doesn't work yet.
* The error pages for unauthorized access are ugly.
* We might want to rename data-original and /thumbnail?size=original
to use some other name, like "full", that better reflects the fact
that we're potentially not serving the original image URL.
In this commit we add a new endpoint so as to have a way of fetching
topic history for a given stream id without having to be logged in.
This can only happen if the said stream is web public otherwise we
just return an empty topics list. This endpoint is quite analogous
to get_topics_backend which is used by our main web app.
In this commit we also do a bit of duplication regarding the query
responsible for fetching all the topics from DB. Basically this
query is exactly the same as what we have in the
get_topic_history_for_stream function in actions.py. Basically
duplicating now is the right thing to do because this query is
really gonna change when we add another criteria for filtering
messages which is:
Only topics for messages which were sent during the period the
corresponding stream was web public should be returned.
Now when we will do this, the query will change and thus it won't
really be a code duplication!
This commits adds the necessary puppet configuration and
installer/upgrade code for installing and managing the thumbor service
in production. This configuration is gated by the 'thumbor.pp'
manifest being enabled (which is not yet the default), and so this
commit should have no effect in a default Zulip production environment
(or in the long term, in any Zulip production server that isn't using
thumbor).
Credit for this effort is shared by @TigorC (who initiated the work on
this project), @joshland (who did a great deal of work on this and got
it working during PyCon 2017) and @adnrs96, who completed the work.
This adds a new settings, SOCIAL_AUTH_SUBDOMAIN, which specifies which
domain should be used for GitHub auth and other python-social-auth
backends.
If one is running a single-realm Zulip server like chat.zulip.org, one
doesn't need to use this setting, but for multi-realm servers using
social auth, this fixes an annoying bug where the session cookie that
python-social-auth sets early in the auth process on the root domain
ends up masking the session cookie that would have been used to
determine a user is logged in. The end result was that logging in
with GitHub on one domain on a multi-realm server like zulipchat.com
would appear to log you out from all the others!
We fix this by moving python-social-auth to a separate subdomain.
Fixes: #9847.
In this commit we are fixing a kinda serious un-noticed bug with
the way run_db_migrations worked for test db.
Basically run_db_migrations runs new migrations on db (dev or test).
When we talk about the dev platform this process is straight forward.
We have a single DB zulip which was once created and now has some data.
Introduction of new migration causes a schema change or does something
else but bottom line being we just migrate the zulip DB and stuff works
fine.
Now coming to zulip test db (zulip_test) situation is a bit complex
in comparision to dev db. Basically this is because we make use of
what we call zulip_test_template to make test fixture restoration
after tests run fast. Now before we introduced the performance
optimisation of just doing migrations when possible, introduction of
a migration would ideally result in provisioning do a full rebuild of
the test database. When that used to happen sequence of events used to
be something like this:
* Create a zulip_test db from zulip_test_base template (An absolute
basic schema holding)
* Migrate and populate the zulip_test db.
* Create/Re-create zulip_test_template from the latest zulip_test.
Now after we introduced just do migrations instead of full db rebuild
when possible, what used to happen was that zulip_test db got
successfully migrated but when test suites would run they would try to
create zulip_test from zulip_test_template (so that individual tests
don't affect each other on db level).
This is where the problem resides; zulip_test_template wasn't migrated
and we just scrapped zulip_test and re-created it using
zulip_test_template as a template and hence zulip_test will not hold the
latest schema.
This is what we fix in this commit.
This commit moves all files previously under the 'app' bundle in
the Django pipeline to being compiled by webpack under the 'app'
entry point. In the process, it moves assets under the app entry
to a file called app.js that consumes all relevant css and js files.
This commit also edits the webpack config to be able to expose certain
variables for third party libraries that are currently required by
some modules. This is bad coding form and should be refactored to
requiring whatever dependencies a module may have; we're just
deferring that to the future to simplify the series of transitions we
need to do here. The variable exposure is done using expose-loader in
webpack.
The app/index.html template is edited to override the newly introduced
'commonjs' block in the base template. This is done as a temporary
measure so as not to disrupt other pages on the app during the transition.
It also fixes the value of the 'this' context that was being inferred
as window by third party libraries. This is done using imports-loader
in the webpack config. This is also messy and probably isn't how we
want things to work long term.
We need to do a small monkey-patching of python-social-auth to ensure
that it doesn't 500 the request when a user does something funny in
their browser (e.g. using the back button in the auth flow) that is
fundamentally a user error, not a server error.
This was present in the pre-rewrite version of our Social auth
codebase, without clear documentation; I've fixed the explanation
part here.
It's perhaps worth investigating with the core social auth team
whether there's a better way to do this.
It's possible to make GitHub social authentication support letting the
user pick which of their verified email addresses to pick, using the
python-social-auth pipeline feature. We need to add an additional
screen to let the user pick, so we're not adding support for that now,
but this at least migrates this to use the data set of all emails that
have been verified as associated with the user's GitHub account (and
we just assume the user wants their primary email).
This also fixes the inability for very old GitHub accounts (where the
`email` field in the details might be a string the user wanted on
their GitHub profile page) to using GitHub auth to login.
Fixes#9127.
Adds search_pill.js to the static asset pipeline. The items
for search pill contain 2 keys, display_value and search_string.
Adding all the operator information i.e the operator, operand and
negated fields along with the search_string and description was tried out.
It was dropped because it didn't provide any advantage as one had to
always calculate the search_string and the description from the operator.
This fixes an important issue where the realm_users cache could grow
beyond 1MB when a Zulip server had more than about 10K users. The
result was that Zulip would start 500ing with that size of userbase.
There are probably better long-term fixes, but because the realm_users
data set caches well, this change should be sufficient to let us
handle to 50-100K users or more on that metric (though at some point,
we'll start having other problems interacting with the realm_users
data set).
This new implementation model is a lot cleaner and should extend
better to the non-oauth backend supported by python-social-auth (since
we're not relying on monkey-patching `do_auth` in the OAuth backend
base class).
This adds a /ping command that will be useful for users
to see what the round trip to the Zulip server is (including
only a tiny bit of actual server time to basically give a
200).
It also introduce the "/zcommand" endpoint and zcommand.js
module.
The previous logic made it look like catching ZulipLDAPException on
the authenticate() line was possible, but it isn't, because that
exception is actually being handled inside django-auth-ldap's
authenticate method.
Previously, if you had LDAPAuthBackend enabled, we basically blocked
any other auth backends from working at all, by requiring the user's
login flow include verifying the user's LDAP password.
We still want to enforce that in the case that the account email
matches LDAP_APPEND_DOMAIN, but there's a reasonable corner case:
Having effectively guest users from outside the LDAP domain.
We don't want to allow creating a Zulip-level password for a user
inside the LDAP domain, so we still verify the LDAP password in that
flow, but if the email is allowed to register (due to invite or
whatever) but is outside the LDAP domain for the organization, we
allow it to create an account and set a password.
For the moment, this solution only covers EmailAuthBackend. It's
likely that just extending the list of other backends we check for in
the new conditional on `email_auth_backend` would be correct, but we
haven't done any testing for those cases, and with auth code paths,
it's better to disallow than allow untested code paths.
Fixes#9422.
Previously, if both EmailAuthBackend and LDAPAuthBackend were enabled,
LDAP users could set a password using EmailAuthBackend and continue to
use that password, even if their LDAP account was later deactivated.
That configuration wasn't supported at all before, so this doesn't fix
a pre-existing security issue, but now that we're making that a valid
configuration, we need to cover this case.
A "zform" knows how to render data that follows our
schema for widget messages with form elements like
buttons and choices.
This code won't be triggered until a subsequent
server-side commit takes widget_content from
API callers such as the trivial chat bot and
creates submessages for us.
This starts the concept of a schema checker, similar to
zerver/lib/validator.py on the server. We can use this
to validate incoming data. Our server should filter most
of our incoming data, but it's useful to have client-side
checking to defend against things like upgrade
regressions (i.e. what if we change the name of the field
on the server side without updating all client uses).
The only purpose of this commit is to make the django templates
of Two Factor Auth work. We probably won't need this commit once
we upgrade the admin backend of Two Factor Auth to use handlebar
templates.
This commit improves the output that blueslip produces while
showing error stack traces on the front-end. This is done by
using a library called error-stack-parser to format the stack
traces.
This commit also edits the webpack config to use a different
devtool setting since the previous one did not support sourcemaps
within stack traces. It also removes a plugin that was obviated
by this change.
We add conditional infinite sleep to this delivery job as a means to
handle case of multiple servers in service to a realm running this
job. In such a scenerio race conditions might arise leading to
multiple deliveries for same message. This way we try to match the
behaviour of what other jobs do in such a case.
Note: We should eventually do something to make such jobs work
while being running on multiple servers.
Slow queries during backend tests sends messages to Error Bot
which affects the database state causing the tests to fail.
This fixes the occasional flakes due to that.
This commit lays the foundation to handle submessages for
plugin widgets. Right now it just logs events, but subsequent
commits will add widget functionality.
This moves the documentation for this feature out of
prod_settings_template.py, so that we can edit it more easily.
We also add a bucket policy, which is part of what one would want to
use this in production.
This addresses much, but not all, of #9361.
We don't reference this anymore (it was only ever used by the Dropbox
integration, which was hardcoded-off for years before being removed in
e6833b6427)
This fixes exceptions when sending PMs in development (where we were
trying to connect to the localhost push bouncer, which we weren't
authorized for, but even if we were, it wouldn't work, since there's
no APNS/GCM certs).
At the same time, we also set and order of operations that ensures one
has the opportunity to adjust the server URL before submitting
anything to us.
Most of this was straightforward.
Most functions that were grabbed verbatim and whole from
the original class still have one-line wrappers.
Many functions are just-the-data versions of functions that
remain in MessageList: see add, append, prepend, remove as
examples. In a typical pattern the MessageList code becomes
super simple:
prepend: function MessageList_prepend(messages) {
var viewable_messages = this.data.prepend(messages);
this.view.prepend(viewable_messages);
},
Two large functions had some minor surgery:
triage_messages =
top half of add_messages +
API to pass three lists back
change_message_id =
original version +
two simple callbacks to list
For the function update_muting_and_rerender(), we continue
to early-exit if this.muting_enabled is false, and we copied
that same defensive check to the new function
named update_items_for_muting(), even though it's technically
hidden from that codepath by the caller.
This commit moves the stylesheets under the archive bundle in
the Django pipeline to being compiled by webpack instead. It
also removes a remaining call to a portico stylesheet that no
longer exists.
This commit transitions landing-page.css from the Django pipeline
to being compiled by webpack as landing-page.scss under the
'landing-page' and 'integration' bundles.
This commit transitions all styles in app.css in the Django pipeline
to being compiled by webpack in an app-styles bundle, and renames the
various files to now be processed as SCSS.
To implement this transition, we move the old CSS file refernces in
settings.py and replace them with a bundle declared in
`webpack.assets.json` and includedn in the index.html template
Tweaked by tabbott to keep the list of files in `app.css` in
`webpack.assets.json`, and to preserve the ordering from the old
`settings.py`.
This commit removes the need for portico.css to be generated
by the Django pipeline and makes the error page use the css
file compiled by webpack instead.
We haven't seen significant traffic from the legacy desktop app in
over a year, and users using it get a warning to upgrade since last
summer, so it's probably OK to stop providing special fonts for it.
This introduces a generic class called list_cursor to handle the
main details of navigating the buddy list and wires it into
activity.js. It replaces some fairly complicated code that
was coupled to stream_list and used lots of jQuery.
The new code interacts with the buddy_list API instead of jQuery
directly. It also persists the key across redraws, so we don't
lose our place when a focus ping happens or we type more characters.
Note that we no longer cycle to the top when we hit the bottom, or
vice versa. Cycling can be kind of an anti-feature when you want to
just lay on the arrow keys until they hit the end.
The changes to stream_list.js here do not affect the left sidebar;
they only remove code that was used for the right sidebar.
This was a bit more than moving code. I extracted the
following things:
$widget (and three helper methods)
$input
text()
empty()
expand_column
close_widget
activity.clear_highlight
There was a minor bug before this commit, where we were inconsistent
about trimming spaces. The introduction of text() and empty() should
prevent bugs where users type the space bar into search.
static/styles/scss/portico.scss is now compiled by webpack
and supports SCSS syntax.
Changed the server-side templates to render the portico-styles
bundle instead of directly requiring the portico stylesheet. This
allows webpack to handle stylesheet compilation and minification.
We use the mini-css-extract-plugin to extract out css from the
includes in webpack and let webpacks production mode handle
minification. Currently we're not able to use it for dev mode
because it does not support HMR so we use style-loader instead.
Once the plugin supports HMR we can go on to use it for both
dev and prod.
The downside of this is that when reloading pages in the development
environment, there's an annoying flash of unstyled content :(.
It is now possible to make a change in any of the styles included
by static/styles/scss/portico.scss and see the code reload live
in the browser. This is because style-loader which we currently
use has the module.accept code built-in.
This shouldn't have a material performance impact, since we don't
query these except during login, and meanwhile this fixes an issue
where users needed to restart memcached (which usually manifested as
"needing to reboot the whole server" after updating their LDAP
configuration before a user who was migrated from one OU to another
could login).
Fixes#9057.
The code in maybe_send_to_registration incorrectly used the
`get_realm_from_request` function to fetch the subdomain. This usage
was incorrect in a way that should have been irrelevant, because that
function only differs if there's a logged-in user, and in this code
path, a user is never logged in (it's the code path for logged-out
users trying to sign up).
This this bug could confuse unit tests that might run with a logged-in
client session. This made it possible for several of our GitHub auth
tests to have a totally invalid subdomain value (the root domain).
Fixing that bug in the tests, in turn, let us delete a code path in
the GitHub auth backend logic in `backends.py` that is impossible in
production, and had just been left around for these broken tests.
This is done mainly because this backend has the simplest code path
for calling login_or_register_remote_user, more than because we expect
this case to come up. It'll make it easier to write unit tests for
the `invalid_subdomain` corner case.
This is a mobile-specific endpoint used for logging into a dev server.
On mobile without this realm_uri it's impossible to send a login request
to the corresponding realm on the dev server and proceed further; we can
only guess, which doesn't work for using multiple realms.
Also rename the endpoint to reflect the additional data.
Testing Plan:
Sent a request to the endpoint, and inspected the result.
[greg: renamed function to match, squashed renames with data change,
and adjusted commit message.]
This should help a bit more in making this file navigable.
I think there's further work that could be done to organize the
settings better: e.g., group LDAP with the auth section; separate
resource limits, from debugging and error reporting, from configuring
service dependencies like Redis and Rabbit. That'd require reordering
many settings, and also taking a closer look at many settings one by
one in order to do a good job. Leaving that for another day.
I've found this file hard to navigate for a while. We actually have a
little hierarchy of section headings which applies to a lot of the
file already; make the boundaries bolder.
Add a clear heading, and use fewer words and simpler sentences. Also
explain the password thing a bit more, and put that more inline next
to the username.
Also, on checking the Django docs, the default for EMAIL_USE_TLS
is False and for EMAIL_PORT is 25. So most admins, certainly any that
are using an SMTP service on the public Internet (that is at all
decently run), will need to set those settings. Mention that.
This is a pretty thin abstraction to prevent having to put
magic numbers in code, doing the which/keyCode hack, and remembering
to all preventDefault.
Hopefully we'll expand it to handle things like shift/alt keys
for components that want their own keyboard handlers (vs. going
through hotkey.js).
We don't indend for this server-level setting to exist in the long
term; the purpose of this is just to make it easy to test this code
path for development purposes.
This implements much of the Message side part of #2745.
This should eliminate the occasional problems that some installations
have had with RabbitMQ trying to load-balance requests between
127.0.0.1 and ::1 (the ipv6 version of localhost).
There are several ways we open help for keyboard shortcuts,
markdown help, and search operators.
- from the gear menu
- from the compose box
- from the search box
- hitting ? for keyboard help
- arrowing/clicking through the tabs
This just moves the relevant code into a module and changes a
bunch of one-line calls in various places.
Now that we have support for displaying custom profile fields, this
adds administrator-level support for creating them.
Tweaked by tabbott to fix a few small bugs and clean up the commit message.
Fixes#1760.
This bot was basically a duplicate of NOTIFICATION_BOT for some
specific corner cases, and didn't add much value. It's better to just
eliminate it, which also removes some ugly corner cases around what
happens if the user account doesn't exist.
In 18e43895ff we replaced
stream subscribe buttons with stream links. The new feature
has been well tested and well received for over a year now,
so it's safe to remove the older feature at this point.
Older sites will have super old messages that still have the
rendered markup; this commit does not attempt to address those
situations. Most likely, clicking on an old button in the old
message will either do nothing or look like a message reply.
This feature isn't really ready yet -- the relevance isn't good, so
the emails aren't a great experience. More work needed; pending that,
just don't send them.
There's already a per-realm setting, which doesn't have a control in
the org settings UI but does suppress it in the per-user settings UI.
Piggyback on that to suppress that UI control when the feature is
disabled at the server level too.
Also cut a comment that hasn't really made sense since the logic was
changed months ago -- the comment originally explained why we sent
digests on Tuesday, Wednesday, and Thursday, and doesn't correspond to
why we dialled back to weekly on Tuesdays.
This is nicer than the "For manual testing ..." comment. :-)
Also as a proper setting we can have it control some logging I
added locally while testing my recent changes to pika logging.
In this commit we start to support redirects to urls supplied as a
'next' param for the following two backends:
* GoogleOAuth2 based backend.
* GitHubAuthBackend.
Update perfect-scrollbar to fix stutter space-scrolling in #8544. Also
reworked deprecated `element.perfectScrollbar` to `new
PerfectScrollbar(element)`. Lastly, updated provision version and
changed node module path to new path.
This also refactors perfect-scrollbar in help.js to work with updated
version of perfect-scrollbar. Because the update also changed
perfect-scrollbar's css selectors for all scrollbars in zulip, we
update those too.
Fixes#8544.
In some environments, these exceptions happen regularly on upgrading
Zulip. They're harmless because we reconnect, so avoid complaining
noisily about them.
This applies only on a server open for anyone to create a realm.
Moreover, if the server admins have granted any given realm a
max_invites greater than the default, that realm is exempt too.
This makes this value much easier for a server admin to change than it
was when embedded directly in the code. (Note this entire mechanism
already only applies on a server open for anyone to create a realm.)
Doing this also means getting the default out of the database.
Instead, we make the column nullable, and when it's NULL in the
database, treat that as whatever the current default is. This better
matches anyway the likely model where there are a few realms with
specially-set values, and everything else should be treated uniformly.
The migration contains a `RenameField` step, which sounds scary
operationally -- but it really does mean just the *field*, in
the model within the Python code. The underlying column's name
doesn't change.
This mostly moves code from ui.js.
We change the arguments to `message_fetch.load_more_messages()`
to be `opts` with callbacks for `show_loading` and `hide_loading`.
We also defer starting the scroll handler until `message_fetch.js`
has been initialized.
@brockwhittaker wrote the original prototype for having
pills in the recipient box when users compose PMs (either
1:1 or huddle). The prototype was test deloyed on our
main realm for several weeks.
This commit includes all the original CSS and HTML from
the prototype.
After some things changed with the codebase after the initial
test deployment, I made the following changes:
* In prior commits I refactored out a module called
`user_pill.js` that implemented some common functions
against a more streamlined version of `input_pill.js`,
and this commit largely integrates with that.
* I made changes in a prior commit to handle Zephyr
semantics (emails don't get validated) and tested
this commit with zephyr.
* I fixed a reload bug by extracting code out to
`compose_pm_pill.js` and re-ordering some
calls to `initialize`.
There are still two flaws related to un-pill-ified text in the
input:
* We could be more aggressive about trying to pill-ify
emails when you blur or tab away.
* We only look at the pills when you send the message,
instead of complaining about the un-pill-ified text.
(Some folks may consider that a feature, but it's
probably surprising to others.)
The main point of this change is to streamline the core
code for input pills, and we use also modify user groups.
The main change to input_pill.js is that you now
configure a function called `create_item_from_text`, and
that can return an arbitrary object, and it just needs
a field called `display_value`.
Other changes:
* You now call `input.create(opts)` to create the
widget.
* There is no longer a cache, because we can
write smarter code in typeahead `source` functions
that exclude ids up front.
* There is no value/optinalKey complexity, because
the calling code can supply arbitrary objects and
do their own external data management on the pill
items.
* We eliminate `prependPill`.
* We eliminate `data`, `keys`, and `values`, and just
have `items`.
This is necessary for mobile apps to do the right thing when only
RemoteUserBackend is enabled, namely, directly redirect to the
third-party SSO auth site as soon as the user enters the server URL
(no need to display a login form, since it'll be useless).
Previously, we used to raise an exception if the direct dev login code
path was attempted when:
* we were running under production environment.
* dev. login was not enabled.
Now we redirect to an error page and give an explanatory message to the
user.
Fixes#8249.
We now isolate the code to transmit messages into transmit.js.
It is stable code that most folks doing UI work in compose.js don't
care about the details of, so it's just clutter there. Also, we may
soon have other widgets than the compose box that send messages.
This change mostly preserves test coverage, although in some cases
we stub at a higher level for the compose path (this is a good thing).
Extracting out transmit.js allows us to lock down 100% coverage on that
file.
This uses an actual query to the backend to check if the subdomain is
available, using the same logic we would use to check when the
subdomain is in fact created.
This adds button under "Organization profile" settings, which
deactivates the organization and sends an "event" to all the
active user and log out them.
Fixes: #8212.
From here on we start to authenticate uploaded file request before
serving this files in production. This involves allowing NGINX to
pass on these file requests to Django for authentication and then
serve these files by making use on internal redirect requests having
x-accel-redirect field. The redirection on requests and loading
of x-accel-redirect param is handled by django-sendfile.
NOTE: This commit starts to authenticate these requests for Zulip
servers running platforms either Ubuntu Xenial (16.04) or above.
Fixes: #320 and #291 partially.
This comment didn't really explain things unless you were looking at
the code, and/or had a strong enough association for what "cn" means
that it was obvious what must be meant. Maybe this will be clearer.
There is one other meaningful key, which is optional: "short_name",
for which I guess a typical value if supplied would be "uid" or
"userid". I'm not sure we really do much with a user's `short_name`,
though, so didn't add a comment for it. When this key is omitted,
we set the user's `short_name` to the same thing as `full_name`.
It's too easy to go over the rate limits when using the webapp.
The correct fix for this probably involves some changes to which
routes get covered by what sort of rate limit, but for now, just
increase the limits.
The original code made a 3/4-hearted effort to generically accommodate
more banners/"panels" later, but named itself after the first one made.
[greg: expanded commit message.]
Different formats for configuration files have a wide variety of ways
of representing lists; so if you're not accustomed to Python syntax,
or aren't thinking of this file as Python code, the syntax for several
ALLOWED_HOSTS entries may not be obvious. And this setting is one
that an admin is likely to want to touch quite early in using Zulip.
So, demonstrate a multi-element list.
For similar reasons, demonstrate an IP address. This one is in a
range reserved for documentation (by RFC 5737), like `example.com`.
For now, this does nothing in a production environment, but it should
simplify the process of doing testing on the Thumbor implementation,
by integrating a lot of dependency management logic.
Gmail is a bad example for outbound email; use a generic example.
Also leave the `= None` default out of the config file, as it's
redundant with DEFAULT_SETTINGS in our internal settings.py ; and
explain in the latter why we don't mention the other SMTP settings.
Since we need KaTeX to be available for zerver/lib/tex.py and
static/third/katex/cli.js to be able to shell out to it. However, for
some reason, the KaTeX we bundle using Webpack doesn't seem to be
importable by Node (and it's also kinda a pain to find its filename
from `cli.js`).
So, we work around this by just using the legacy system for KaTeX.
Something similar is needed for zxcvbn.js, in order to support the
settings_account.js use case (basically deferred loading of this
file); that requires JS code to have access to the correct path for
zxcvbn.
Stripe Checkout means using JS code provided by Stripe to handle
almost all of the UI, which is great for us.
There are more features we should add to this page and changes we
should make, but this gives us an MVP.
[greg: expanded commit message; fixed import ordering and some types.]
There are two different things you need to patch in order to get error
emails (at `/emails`) in dev. Flip one of them in dev all the time,
and make the comment on the other a bit more explicit.
This name hasn't been right since f7f2ec0ac back in 2013; this handler
sends the log record to a queue, whose consumer will not only maybe
send a Zulip message but definitely send an email. I found this
pretty confusing when I first worked on this logging code and was
looking for how exception emails got sent; so now that I see exactly
what's actually happening here, fix it.
This line was added in e8ab7cd1a as a desperate measure to get the
`set_loglevel` helper to successfully suppress the `zulip.send_email`
logs.
In fact, the reason that just setting `level` there wasn't doing the
job had nothing to do with `propagate`, which doesn't interact with
`level`. (See the long block comment in `zproject/settings.py`,
searching for "Python logging module", for my attempt at concisely
explaining these semantics.) Rather, our setting for `level` was
getting clobbered by the use of `create_logger`; and setting
`propagate` to False worked by completely suppressing all logs to
`zulip.send_email` and descendants from reaching the normal handlers,
regardless of the log messages' levels.
Now that the `create_logger` issue is fixed (see a few commits before
this one), drop that.
Because calls to `create_logger` generally run after settings are
configured, these would override what we have in `settings.LOGGING` --
which in particular defeated any attempt to set log levels in
`test_settings.py`. Move all of these settings to the same place in
`settings.py`, so they can be overridden in a uniform way.
These aren't of much use without the corresponding secrets (despite us
calling one of them a "key", both are public IDs), which are in my
personal dev-secrets.conf. So, remove them.
I had this commit applied in order to test what became 90944983f...
and then I accidentally included it in what I pushed, as 0fa9a489d.
Oops.
This adds custom CSS through JavaScript for things that do not
scope well and will override other inherited styles.
This should ONLY be used for problematic CSS that has no obvious
or easy CSS-only solution.
(Specifically, we need this for the "default link" styling, which is
hard to override because we don't want to start winning ties due to
specificity that we would not have won in the light theme).
[Modified by greg to (1) keep `USERNAME_FIELD = 'email'`,
(2) silence the corresponding system check, and (3) ban
reusing a system bot's email address, just like we do in
realm creation.]
Originally this used signals, namely SIGRTMIN. But in prod, the
signal handler never fired; I debugged fruitlessly for a while, and
suspect uwsgi was foiling it in a mysterious way (which is kind of
the only way uwsgi does anything.)
So, we listen on a socket. Bit more code, and a bit trickier to
invoke, but it works.
This was developed for the investigation of memory-bloating on
chat.zulip.org that led to a331b4f64 "Optimize query_all_subs_by_stream()".
For usage instructions, see docstring.
Tweaked by tabbott to move changes from the next commit that are
required for this to pass tests into this commit.
Note that this exports a few items that were not previously exported.
This fixes some subtle JavaScript exceptions we've been getting in
zulipchat.com, caused by the system bot realm there not being "zulip"
interacting with get_cross_realm_users.
I probably should have just done this in the original implementation;
there's only a small downside in the form of an extra database query
when trying to authenticate a user who doesn't exist.
Structurally, the main change here is replacing the `clean_username`
function, which would get called when one accessed
self.cleaned_data['username'] with code in the main `clean` function.
This is important because only in `clean` do we have access to the
`realm` object.
Since I recently added full test coverage on this form, we know each
of the major cases have a test; the error messages are unchanged.
This deletes the old mock-covered test for this, which was mostly
useless. We have a much less messy test, which we extend to provide
the same test coverage the old one did.
While the result was the same before, this makes it more obvious.
This uses the to-markdown.js library to do all the hard work of
parsing HTML and turning it into markdown and not e.g. uploaded files.
Tweaked by tabbott to better scope when it activates to just include
pastes of HTML content.
Fixes#5853.
We created this redundant pair of conditionals in a preceding commit,
in order to match the indentation of an `except` block so as to slice
the diffs extra finely as we're refactoring auth code.
This lets us delete some duplicate code, since common_get_active_user
handles an account in the wrong subdomain for us.
Also lets us delete the now-unused common_get_active_user_by_email.
This way, we don't attempt to evaluate whether the user's account is
active (etc.) until after we've checked the backend is enabled. This
won't change the result of actual auth, but feels more readable.
This is a behavior change, though we don't check the value in the
caller regardless. It just seems more logical for us to correctly
report to the caller whether the Google auth itself was valid
unconditionally.
This code path was only required because we had remote_user set as a
positional argument here, and thus we'd be running this auth backend's
code when actually using another auth backend (due to how Django auth
backends are selected based on argument signature).
This require some care to ensure we still provide the same nice error
messages for the case of a user who has an account, just not with this
organization.
Also, we fix the fact that the docstring was (and I think always has
been) at best confusing and perhaps even inaccurate.
Now we have moved the `do_auth` function to `SocialAuthMixin`. Instead
of overriding `do_auth`, derived class is now expected to override
`get_authenticated_user`.
`do_auth` now contains code which is expected by all backends.
Storage limititations are only set on the value of
a config entry, since this is the only user-accessible
part of the schema. Keys are statically set by each
embedded bot.
This endpoint will allow us to add/delete emoji reactions whose emoji
got renamed during various emoji infra changes. This was also a
required change for realm emoji migration.
This commit was tweaked significantly by tabbott for greater clarity
(with no changes to the actual logic).
Note from tabbott: While this initial version is experimental and
definitely incomplete, we expect to have a solid version done over the
next few weeks (after more refactoring). We're merging this now to
make it easy to test both versions when refactoring our CSS.
Fixes#267.
These are new:
new-user-bot
emailgateway
Our cross-realm bots are hard coded to have email addresses
in the `zulip.com` domain, and they're not part of ordinary
realms.
These have always been cross-realm, but new enforcement in the
frontend code of all messages having been sent by a known user means
that it's important to add these properly.
The warning here means that the admin can't really act on this yet if
they want to disable email auth, which is likely among admins that
want to make any changes here. And for admins who don't, this is an
extra thing to read and make a decision about before they can get a
server running. See #6985.
Conversely, we already discuss auth backends right at the top of the
`prod-customize` doc, which is linked under "Next steps" at the end of
these instructions.
The warning about EmailAuthBackend is important; but we can move it to
the config file right next to the setting, and then it's available
right when it's actionable, which is if the admin is actually thinking
about changing the setting.
For some historical reason we'd had the Postgres documentation on
valid SSL modes copied into the Zulip settings.py template file. This
fixes that historical artifact.
This change prepares us to have the server send avatar_url
of None when somebody wants a gravatar avatar (as opposed
to a user-uploaded one).
Subsequent commits will change behavior on both the server
and client to have this happen. So this commit has no-op
code for now, but it will soon use the fallback-to-gravatar
logic.
The control panel on the Google side doesn't seem to match the
instructions we have; it looks pretty 2017 to me, so I imagine
it's had a redesign since the instructions were written.
Also, in dev, EXTERNAL_HOST is now a port on zulipdev.com, not on
localhost.
Update these instructions for those developments, and edit lightly.
In dev, recommend setting in `dev_settings` instead of in
`prod_settings_template`; that feels to me a little more reflective of
the actual intent, and the effect should be equivalent.
The main limitation of this version is that it's controlled entirely
from settings, with nothing in the database and no web UI or even
management command to control it. That makes it a bit more of a
burden for the server admins than it'd ideally be, but that's fine
for now.
Relatedly, the web flow for realm creation still requires choosing a
subdomain even if the realm is destined to live at an alias domain.
Specific to the dev environment, there is an annoying quirk: the
special dev login flow doesn't work on a REALM_HOSTS realm. Also,
in this version the `add_new_realm` and `add_new_user` management
commands, which are intended for use in development environments only,
don't support this feature.
In manual testing, I've confirmed that a REALM_HOSTS realm works for
signup and login, with email/password, Google SSO, or GitHub SSO.
Most of that was in dev; I used zulipstaging.com to also test
* logging in with email and password;
* logging in with Google SSO... far enough to correctly determine
that my email address is associated with some other realm.
This means one fewer thing the admin typically needs to read, absorb,
and make a decision about at install time.
The one way this change could hypothetically cause trouble is if the
admin wants to keep subdomains of EXTERNAL_HOST out of ALLOWED_HOSTS.
But while the subdomains often won't exist as domain names, it's hard
to imagine the situation in which they would exist but be under
someone else's control, or be doing something other than serving
Zulip realms.
This commit allows for the /api-new/ page to rendered similarly to our
/help pages. It's based on the old content for /api, but we're not
replacing the old content yet, to give a bit of time to restructure
things reasonably.
Tweaked by eeshangarg and tabbott.
This was part of the logic to handle EXTERNAL_API_PATH varying.
But also it was already no longer used -- it was only ever passed
into template contexts, as `external_api_uri`, and it'd been
overtaken there by `external_api_uri_subdomain`.
So, update our dev docs to reflect that, and eliminate the variable.
This setting isn't documented at all, and I believe nobody has used it
since the end of api.zulip.com in 2016. So we get to complete the
cleanup of this logic.
The HelpView class will render a directory as markdown with an index HTML
page. This however can also be used for other generics and applied to
the API pages as well, so change the class to a generic class and
specify the path templates and names.
Tweaked by tabbott and Eeshan Garg.
Our set_loglevel tool didn't set propagate to False, so just using it
directly wouldn't work unless the logger is explcitly declared in
zproject/settings.py, which this one isn't.
The cookie mechanism only works when passing the login token to a
subdomain. URLs work across domains, which is why they're the
standard transport for SSO on the web. Switch to URLs.
Tweaked by tabbott to add a test for an expired token.
Most of these have more to do with authentication in general than with
registering a new account. `create_preregistration_user` could go
either way; we move it to `auth` so we can make the imports go only in
one direction.
Lets administrators view a list of open(unconfirmed) invitations and
resend or revoke a chosen invitation.
There are a few changes that we can expect for the future:
* It is currently possible to invite an email that you have already
invited, it might make sense to change this behavior.
* Resend currently sends an invite reminder instead of resending the
original invite, this is because 'custom_body' was not stored when
the first invite was sent.
Tweaked in various minor ways, primarily in the backend, by tabbott,
mostly for style consistency with the rest of the codebase.
Fixes: #1180.
Now that every call site of check_subdomain produces its second
argument in exactly the same way, push that shared bit of logic
into a new wrapper for check_subdomain.
Also give that new function a name that says more specifically what
it's checking -- which I think is easier to articulate for this
interface than for that of check_subdomain.
Adds support to add "Embedded bot" Service objects. This service
handles every embedded bot.
Extracted from "Embedded bots: Add support to add embedded bots from
UI" by Robert Honig.
Tweaked by tabbott to be disabled by default.
This should help prevent confusion where new users find themselves on
the LDAP login form and click "register" because they know they don't
have an account. Whereas in fact, their account will be auto-created
if they just login, so there's no need for them to access it.
This fixes a problem we've seen where LDAP users were not getting this
part of the onboarding process, and a similar problem for human users
created via the API.
Ideally, we would have put these fixes in process_new_human_user, but
that would cause import loop problems.
The rules here are fuzzy, and it's quite possible none of Zulip's emails
need an address at all. Every country has its own rules though, which makes
it hard to tell. In general, transactional emails do not need an address,
and marketing emails do.
This is a two-step notifications process that will ask a user
to enable notifications and if they click exit give them three
options:
1. Enable notifications.
2. Ask later.
3. Never ask on this computer again.
The first two are self-explanatory (ask later = next session it
asks again). The third is captured and stored in localStorage and
a check is done on page load to see whether or not notifications
should be displayed.
Commit modified heavily by Brock Whittaker <brock@zulipchat.com>.
Fixes#1189.
By default, Django sets up two handlers on this logger, one of them
its AdminEmailHandler. We have our own handler for sending email on
error, and we want to stick to that -- we like the format somewhat
better, and crucially we've given it some rate-limiting through
ZulipLimiter.
Since we cleaned out our logging config in e0a5e6fad, though, we've
been sending error emails through both paths. The config we'd had
before that for `django` was redundant with the config on the root --
but having *a* config there was essential for causing
`logging.config.dictConfig`, when Django passes it our LOGGING dict,
to clear out that logger's previous config. So, give it an empty
config.
Django by default configures two loggers: `django` and
`django.server`. We have our own settings for `django.server`
anyway, so this is the only one we need to add.
The stdlib `logging` and `logging.config` docs aren't 100% clear, and
while the source of `logging` is admirably straightforward the source
of `logging.config` is a little twisty, so it's not easy to become
totally confident that this has the right effect just by reading.
Fortunately we can put some of that source-diving to work in writing
a test for it.
This works around a bug in Django in handling the error case of a
client sending an inappropriate HTTP `Host:` header. Various
internal Django machinery expects to be able to casually call
`request.get_host()`, which will attempt to parse that header, so an
exception will be raised. The exception-handling machinery attempts
to catch that exception and just turn it into a 400 response... but
in a certain case, that machinery itself ends up trying to call
`request.get_host()`, and we end up with an uncaught exception that
causes a 500 response, a chain of tracebacks in the logs, and an email
to the server admins. See example below.
That `request.get_host` call comes in the midst of some CSRF-related
middleware, which doesn't even serve any function unless you have a
form in your 400 response page that you want CSRF protection for.
We use the default 400 response page, which is a 26-byte static
HTML error message. So, just send that with no further ado.
Example exception from server logs (lightly edited):
2017-10-08 09:51:50.835 ERR [django.security.DisallowedHost] Invalid HTTP_HOST header: 'example.com'. You may need to add 'example.com' to ALLOWED_HOSTS.
2017-10-08 09:51:50.835 ERR [django.request] Internal Server Error: /loginWithSetCookie
Traceback (most recent call last):
File ".../django/core/handlers/exception.py", line 41, in inner
response = get_response(request)
File ".../django/utils/deprecation.py", line 138, in __call__
response = self.process_request(request)
File ".../django/middleware/common.py", line 57, in process_request
host = request.get_host()
File ".../django/http/request.py", line 113, in get_host
raise DisallowedHost(msg)
django.core.exceptions.DisallowedHost: Invalid HTTP_HOST header: 'example.com'. You may need to add 'example.com' to ALLOWED_HOSTS.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File ".../django/core/handlers/exception.py", line 109, in get_exception_response
response = callback(request, **dict(param_dict, exception=exception))
File ".../django/utils/decorators.py", line 145, in _wrapped_view
result = middleware.process_view(request, view_func, args, kwargs)
File ".../django/middleware/csrf.py", line 276, in process_view
good_referer = request.get_host()
File ".../django/http/request.py", line 113, in get_host
raise DisallowedHost(msg)
django.core.exceptions.DisallowedHost: Invalid HTTP_HOST header: 'example.com'. You may need to add 'example.com' to ALLOWED_HOSTS.
Apparently, this sockjs.tornado logging code resulted in a lot of
buggy error emails whenever a Zulip browser tried to reconnect on a
new IP. I don't see an obvious way to suppress them from within
sockjs, but that might be a good follow-up issue.
Fixes#6959.
The comment is pretty self-explanatory. The fact that Google Compute
Engine has this problem does not impress confidence about their
product, but hopefully this is the only really dumb thing they do.
Fixes#4839.
The original "quality score" was invented purely for populating
our password-strength progress bar, and isn't expressed in terms
that are particularly meaningful. For configuration and the core
accept/reject logic, it's better to use units that are readily
understood. Switch to those.
I considered using "bits of entropy", defined loosely as the log
of this number, but both the zxcvbn paper and the linked CACM
article (which I recommend!) are written in terms of the number
of guesses. And reading (most of) those two papers made me
less happy about referring to "entropy" in our terminology.
I already knew that notion was a little fuzzy if looked at
too closely, and I gained a better appreciation of how it's
contributed to confusion in discussing password policies and
to adoption of perverse policies that favor "Password1!" over
"derived unusual ravioli raft". So, "guesses" it is.
And although the log is handy for some analysis purposes
(certainly for a graph like those in the zxcvbn paper), it adds
a layer of abstraction, and I think makes it harder to think
clearly about attacks, especially in the online setting. So
just use the actual number, and if someone wants to set a
gigantic value, they will have the pleasure of seeing just
how many digits are involved.
(Thanks to @YJDave for a prototype that the code changes in this
commit are based on.)
This endpoint is part of the old tutorial, which we've removed, and
has some security downsides as well.
This includes a minor refactoring of the tests.
Create a new custom email backend which would automatically
logs the emails that are send in the dev environment as
well as print a friendly message in console to visit /emails
for accessing all the emails that are sent in dev environment.
Since django.core.mail.backends.console.EmailBackend is no longer
userd emails would not be printed to the console anymore.
In 1.2.15 version of django-auth-ldap, the authenticate() function of
LDAPBackend takes username and password as keyword arguments. This
commit updates the code to match this change.
Fixes#6588
Vishnu Ks
Greg Price
Tim Abbott
Luis Silva
Shubham Padia
Roman Godov
Lyla Fischer
Harshit Bansal
Rishi Gupta
Aditya Bansal
Priyank Patel
Joshua Schmidlkofer
Armaan Ahluwalia
Yashashvi Dave
Steve Howell
Raymond Akornor
Sampriti Panda
Eeshan Garg
Umair Khan
Aman Ghebreweldi
Angelika Serwa
Vishwesh Jainkuniya
Gooca
Nikhil Kumar Mishra
feorlen
Shubham Dhama
Aman Jain
Brock Whittaker
rht
Reid Barton
Alena Volkova
Cynthia Lin
Rhea Parekh
Tommy Ip
derAnfaenger
Garvit
neiljp (Neil Pilgrim)
Henrik Pettersson
Abhijeet Kaur
darshanime
Maarten Rijke
Aastha Gupta
