From here on we start to authenticate uploaded file request before
serving this files in production. This involves allowing NGINX to
pass on these file requests to Django for authentication and then
serve these files by making use on internal redirect requests having
x-accel-redirect field. The redirection on requests and loading
of x-accel-redirect param is handled by django-sendfile.
NOTE: This commit starts to authenticate these requests for Zulip
servers running platforms either Ubuntu Xenial (16.04) or above.
Fixes: #320 and #291 partially.
It's too easy to go over the rate limits when using the webapp.
The correct fix for this probably involves some changes to which
routes get covered by what sort of rate limit, but for now, just
increase the limits.
The original code made a 3/4-hearted effort to generically accommodate
more banners/"panels" later, but named itself after the first one made.
[greg: expanded commit message.]
For now, this does nothing in a production environment, but it should
simplify the process of doing testing on the Thumbor implementation,
by integrating a lot of dependency management logic.
Gmail is a bad example for outbound email; use a generic example.
Also leave the `= None` default out of the config file, as it's
redundant with DEFAULT_SETTINGS in our internal settings.py ; and
explain in the latter why we don't mention the other SMTP settings.
Since we need KaTeX to be available for zerver/lib/tex.py and
static/third/katex/cli.js to be able to shell out to it. However, for
some reason, the KaTeX we bundle using Webpack doesn't seem to be
importable by Node (and it's also kinda a pain to find its filename
from `cli.js`).
So, we work around this by just using the legacy system for KaTeX.
Something similar is needed for zxcvbn.js, in order to support the
settings_account.js use case (basically deferred loading of this
file); that requires JS code to have access to the correct path for
zxcvbn.
Stripe Checkout means using JS code provided by Stripe to handle
almost all of the UI, which is great for us.
There are more features we should add to this page and changes we
should make, but this gives us an MVP.
[greg: expanded commit message; fixed import ordering and some types.]
There are two different things you need to patch in order to get error
emails (at `/emails`) in dev. Flip one of them in dev all the time,
and make the comment on the other a bit more explicit.
This name hasn't been right since f7f2ec0ac back in 2013; this handler
sends the log record to a queue, whose consumer will not only maybe
send a Zulip message but definitely send an email. I found this
pretty confusing when I first worked on this logging code and was
looking for how exception emails got sent; so now that I see exactly
what's actually happening here, fix it.
Because calls to `create_logger` generally run after settings are
configured, these would override what we have in `settings.LOGGING` --
which in particular defeated any attempt to set log levels in
`test_settings.py`. Move all of these settings to the same place in
`settings.py`, so they can be overridden in a uniform way.
This adds custom CSS through JavaScript for things that do not
scope well and will override other inherited styles.
This should ONLY be used for problematic CSS that has no obvious
or easy CSS-only solution.
(Specifically, we need this for the "default link" styling, which is
hard to override because we don't want to start winning ties due to
specificity that we would not have won in the light theme).
[Modified by greg to (1) keep `USERNAME_FIELD = 'email'`,
(2) silence the corresponding system check, and (3) ban
reusing a system bot's email address, just like we do in
realm creation.]
Originally this used signals, namely SIGRTMIN. But in prod, the
signal handler never fired; I debugged fruitlessly for a while, and
suspect uwsgi was foiling it in a mysterious way (which is kind of
the only way uwsgi does anything.)
So, we listen on a socket. Bit more code, and a bit trickier to
invoke, but it works.
This was developed for the investigation of memory-bloating on
chat.zulip.org that led to a331b4f64 "Optimize query_all_subs_by_stream()".
For usage instructions, see docstring.
Tweaked by tabbott to move changes from the next commit that are
required for this to pass tests into this commit.
Note that this exports a few items that were not previously exported.
This fixes some subtle JavaScript exceptions we've been getting in
zulipchat.com, caused by the system bot realm there not being "zulip"
interacting with get_cross_realm_users.
This uses the to-markdown.js library to do all the hard work of
parsing HTML and turning it into markdown and not e.g. uploaded files.
Tweaked by tabbott to better scope when it activates to just include
pastes of HTML content.
Fixes#5853.
Storage limititations are only set on the value of
a config entry, since this is the only user-accessible
part of the schema. Keys are statically set by each
embedded bot.
Note from tabbott: While this initial version is experimental and
definitely incomplete, we expect to have a solid version done over the
next few weeks (after more refactoring). We're merging this now to
make it easy to test both versions when refactoring our CSS.
Fixes#267.
These are new:
new-user-bot
emailgateway
Our cross-realm bots are hard coded to have email addresses
in the `zulip.com` domain, and they're not part of ordinary
realms.
These have always been cross-realm, but new enforcement in the
frontend code of all messages having been sent by a known user means
that it's important to add these properly.
This change prepares us to have the server send avatar_url
of None when somebody wants a gravatar avatar (as opposed
to a user-uploaded one).
Subsequent commits will change behavior on both the server
and client to have this happen. So this commit has no-op
code for now, but it will soon use the fallback-to-gravatar
logic.
The main limitation of this version is that it's controlled entirely
from settings, with nothing in the database and no web UI or even
management command to control it. That makes it a bit more of a
burden for the server admins than it'd ideally be, but that's fine
for now.
Relatedly, the web flow for realm creation still requires choosing a
subdomain even if the realm is destined to live at an alias domain.
Specific to the dev environment, there is an annoying quirk: the
special dev login flow doesn't work on a REALM_HOSTS realm. Also,
in this version the `add_new_realm` and `add_new_user` management
commands, which are intended for use in development environments only,
don't support this feature.
In manual testing, I've confirmed that a REALM_HOSTS realm works for
signup and login, with email/password, Google SSO, or GitHub SSO.
Most of that was in dev; I used zulipstaging.com to also test
* logging in with email and password;
* logging in with Google SSO... far enough to correctly determine
that my email address is associated with some other realm.
This means one fewer thing the admin typically needs to read, absorb,
and make a decision about at install time.
The one way this change could hypothetically cause trouble is if the
admin wants to keep subdomains of EXTERNAL_HOST out of ALLOWED_HOSTS.
But while the subdomains often won't exist as domain names, it's hard
to imagine the situation in which they would exist but be under
someone else's control, or be doing something other than serving
Zulip realms.
This was part of the logic to handle EXTERNAL_API_PATH varying.
But also it was already no longer used -- it was only ever passed
into template contexts, as `external_api_uri`, and it'd been
overtaken there by `external_api_uri_subdomain`.
So, update our dev docs to reflect that, and eliminate the variable.
This setting isn't documented at all, and I believe nobody has used it
since the end of api.zulip.com in 2016. So we get to complete the
cleanup of this logic.
Lets administrators view a list of open(unconfirmed) invitations and
resend or revoke a chosen invitation.
There are a few changes that we can expect for the future:
* It is currently possible to invite an email that you have already
invited, it might make sense to change this behavior.
* Resend currently sends an invite reminder instead of resending the
original invite, this is because 'custom_body' was not stored when
the first invite was sent.
Tweaked in various minor ways, primarily in the backend, by tabbott,
mostly for style consistency with the rest of the codebase.
Fixes: #1180.
Adds support to add "Embedded bot" Service objects. This service
handles every embedded bot.
Extracted from "Embedded bots: Add support to add embedded bots from
UI" by Robert Honig.
Tweaked by tabbott to be disabled by default.
This should help prevent confusion where new users find themselves on
the LDAP login form and click "register" because they know they don't
have an account. Whereas in fact, their account will be auto-created
if they just login, so there's no need for them to access it.
The rules here are fuzzy, and it's quite possible none of Zulip's emails
need an address at all. Every country has its own rules though, which makes
it hard to tell. In general, transactional emails do not need an address,
and marketing emails do.
This is a two-step notifications process that will ask a user
to enable notifications and if they click exit give them three
options:
1. Enable notifications.
2. Ask later.
3. Never ask on this computer again.
The first two are self-explanatory (ask later = next session it
asks again). The third is captured and stored in localStorage and
a check is done on page load to see whether or not notifications
should be displayed.
Commit modified heavily by Brock Whittaker <brock@zulipchat.com>.
Fixes#1189.
By default, Django sets up two handlers on this logger, one of them
its AdminEmailHandler. We have our own handler for sending email on
error, and we want to stick to that -- we like the format somewhat
better, and crucially we've given it some rate-limiting through
ZulipLimiter.
Since we cleaned out our logging config in e0a5e6fad, though, we've
been sending error emails through both paths. The config we'd had
before that for `django` was redundant with the config on the root --
but having *a* config there was essential for causing
`logging.config.dictConfig`, when Django passes it our LOGGING dict,
to clear out that logger's previous config. So, give it an empty
config.
Django by default configures two loggers: `django` and
`django.server`. We have our own settings for `django.server`
anyway, so this is the only one we need to add.
The stdlib `logging` and `logging.config` docs aren't 100% clear, and
while the source of `logging` is admirably straightforward the source
of `logging.config` is a little twisty, so it's not easy to become
totally confident that this has the right effect just by reading.
Fortunately we can put some of that source-diving to work in writing
a test for it.
Apparently, this sockjs.tornado logging code resulted in a lot of
buggy error emails whenever a Zulip browser tried to reconnect on a
new IP. I don't see an obvious way to suppress them from within
sockjs, but that might be a good follow-up issue.
Fixes#6959.
The comment is pretty self-explanatory. The fact that Google Compute
Engine has this problem does not impress confidence about their
product, but hopefully this is the only really dumb thing they do.
Fixes#4839.
The original "quality score" was invented purely for populating
our password-strength progress bar, and isn't expressed in terms
that are particularly meaningful. For configuration and the core
accept/reject logic, it's better to use units that are readily
understood. Switch to those.
I considered using "bits of entropy", defined loosely as the log
of this number, but both the zxcvbn paper and the linked CACM
article (which I recommend!) are written in terms of the number
of guesses. And reading (most of) those two papers made me
less happy about referring to "entropy" in our terminology.
I already knew that notion was a little fuzzy if looked at
too closely, and I gained a better appreciation of how it's
contributed to confusion in discussing password policies and
to adoption of perverse policies that favor "Password1!" over
"derived unusual ravioli raft". So, "guesses" it is.
And although the log is handy for some analysis purposes
(certainly for a graph like those in the zxcvbn paper), it adds
a layer of abstraction, and I think makes it harder to think
clearly about attacks, especially in the online setting. So
just use the actual number, and if someone wants to set a
gigantic value, they will have the pleasure of seeing just
how many digits are involved.
(Thanks to @YJDave for a prototype that the code changes in this
commit are based on.)
Create a new custom email backend which would automatically
logs the emails that are send in the dev environment as
well as print a friendly message in console to visit /emails
for accessing all the emails that are sent in dev environment.
Since django.core.mail.backends.console.EmailBackend is no longer
userd emails would not be printed to the console anymore.
It turns out that very little code change is required to support
GitHub auth on mobile. Ideally, this would come with tests, though
the complicated part of the code path is covered by the Google auth
version. But writing a test for this would take a long time, and I
think it's worth having the feature now, so I'll be doing tests as a
follow-up project.
The main change here is moving SOCIAL_AUTH_FIELDS_STORED_IN_SESSION to
be with the other hardcoded settings, since it's not something that
makes sense for a sysadmin to change. But while we're at it, we also
group the overall social auth settings separately from the
GitHub-specific settings.
This isn't something that a user can ever modify, so it doesn't belong
in DEFAULT_SETTINGS. While we're at it, we align the appearance of
the email gateway in the docs with whether this setting in the docs
will be valid.
This will help identify the settings that need attention: either
to remove, or to document for server admins, or to just add a
comment to explain.
Identified with the following shell "one-liner" (one 313-char line
as I originally ran it; indentation added here for clarity):
perl -lne 'next unless (/^DEFAULT_SETTINGS/../\}\)?$/);
next unless (/'\''(.*?)'\''/);
print $1' \
zproject/settings.py \
| while read var; do \
echo -n "$var: "; \
(grep -lw "$var" zproject/{prod_settings_template,{dev,test}_settings}.py \
|| echo none) \
| sed s,zproject/,,g \
| fmt -w1000; \
done
This doesn't yet do much, but it gives us a suitable place to
add code to customize how log messages are displayed, beyond what
a format string passed to the default formatter can do.
This should make it a little easier to understand our logging config
and make changes to it with confidence.
Many of these items that are now redundant used to be required when we
were setting disable_existing_loggers to True (before 500d81bf2), in
order to exempt those loggers from being cleared out. Now they're not.
One bit of test code needed a tweak to how it got its hands on the
AdminZulipHandler instance; it can do it from the list on the root
logger just as well as on the `django` logger.
This was giving a couple of lines of logs on every normal,
successful connection -- clearly a job for DEBUG, but emitted
on INFO. Quiet it down.
Fixes#6674.
Aditya Bansal
Aman Jain
Greg Price
Tim Abbott
Brock Whittaker
Vishnu Ks
Cynthia Lin
Umair Khan
Tommy Ip
Garvit
rht
derAnfaenger
Rishi Gupta
Steve Howell
Harshit Bansal
Henrik Pettersson
Abhijeet Kaur
darshanime
Maarten Rijke
Aastha Gupta