Commit Graph

1598 Commits

Author SHA1 Message Date
Tim Abbott 20a13b42e6 auth: Automatically maintain list of OAuth backends.
Here, we take advantage of the useful issubclass function to determine
which of our social backends are based on OAuth2.
2018-10-11 17:11:04 -07:00
Tim Abbott 1244a5077e auth: Automate social auth backends in AUTH_BACKEND_NAME_MAP.
This saves a line of manual code every time we add a new social auth
backend.
2018-10-11 17:11:03 -07:00
Tim Abbott b51633efb7 auth: Allow '-' characters in social auth login urls.
This is important for backends like azuread-oauth2.
2018-10-11 16:44:33 -07:00
Tim Abbott 8cf104b643 avatar: Allow API authentication for /avatar/ routes.
This makes it feasibly for the mobile apps to correctly render user
avatars generated by the `!avatar()` syntax.
2018-10-11 15:52:29 -07:00
Steve Howell f1e0152d71 minor: Fix indentation from recent ldap commit.
The linter was complaining.
2018-10-11 15:28:58 -07:00
Vishnu Ks 646ba439f7 settings: Remove unused EMAIL_LOG_DIR setting from dev_settings.
This settings is not used anywhere.
2018-10-11 15:12:08 -07:00
Vishnu Ks 6aa4b64dc0 emails: Don't log emails while running test suite.
Modified the tests to ensure 100% coverage.
2018-10-11 15:12:08 -07:00
Vishnu Ks d8c19cb003 models: Move billing models from zilencer to corporate. 2018-10-11 14:54:29 -07:00
Greg Price d77a4c776d ldap: Skip following dubious referrals.
Some admins setting up Zulip's LDAP auth against Active Directory see
a rather baffling error message: "In order to perform this operation a
successful bind must be completed on the connection".  This happens
despite AUTH_LDAP_BIND_DN and auth_ldap_bind_password being set
perfectly correctly, and on a query that the `ldapsearch` command-line
tool performs quite happily.

Empirically, adding a setting like this to /etc/zulip/settings.py
resolves the issue:
  AUTH_LDAP_CONNECTION_OPTIONS = {
      ldap.OPT_REFERRALS: 0
  }

Some useful, concise background on the LDAP "referral" concept is here:
  https://docs.oracle.com/javase/jndi/tutorial/ldap/referral/overview.html
and a pertinent bit of docs for the underlying Python `ldap` client:
  https://www.python-ldap.org/en/latest/faq.html
and some very helpful documentation for Active Directory:
  https://docs.microsoft.com/en-us/windows/desktop/ad/referrals

Based on the docs above, the story appears to be something like this:

 * This server has the information for part of the scope of our query
   -- in particular it happens to have the information we actually want.

 * But there are other areas ("subordinate domains") that our query is
   in principle asking about, and this server doesn't know if there are
   matches there, so it gives us a referral.

 * And by default, python-ldap lets `libldap` run ahead and attempt to
   bind to those referrals and do those queries too -- which raises an
   error because, unlike Microsoft's "LDAP API", it doesn't reuse the
   credentials.

So if we simply skip trying to follow the referrals, there's no
error... and we already have, from the original response, the answer
we actually need.  That's what the `ldap.OPT_REFERRALS` option does.

There may be more complex situations where the referral really is
relevant, because the desired user info is split across servers.  Even
then, unless an anonymous query will be acceptable, there's no point
in letting `libldap` follow the referral and setting this option is
still the right thing.  When someone eventually comes to this bridge,
some code will be required to cross it, by following the referrals.
That code might look a bit like this (unfinished) example:
  https://bugs.launchpad.net/nav/+bug/1209178

Manually tested by tabbott.

Fixes #343, which was effectively a report of the need for this
OPT_REFERRALS setting.

Fixes #349, since with this change, we no longer require tricky manual
configuration to get Active Directory up and running.
2018-10-11 14:52:13 -07:00
Vishnu Ks d8ddb7f69b settings: Remove reminder-bot from REALM_INTERNAL_BOTS. 2018-10-02 11:12:07 -07:00
Vishnu Ks 6914ee126c billing: Move billing-related views and urls to corporate. 2018-10-01 15:43:21 -07:00
Tim Abbott 923ecb1851 docs: Clean up documentation for fakeldap testing in development. 2018-09-27 13:38:14 -07:00
Greg Price a612049ac4 docs: Expand on LDAP config; try to clarify "username" concept.
The term "username" confusingly refers both to the Django concept of
"username" (meaning "the name the user types into the login form") and
a concept the admin presumably already has in their existing
environment; which may or may not be the same thing, and in fact this
is where we document the admin's choice of whether and how they should
correspond.  The Django concept in particular isn't obvious, and is
counterintuitive when it means something like an email address.

Explicitly explain the Django "username" concept, under the name of
"Zulip username" to take responsibility for our choice of how it's
exposed in the settings interface.  Then use an explicit qualifier,
like "LDAP username", whenever referring to some other notion of
username.  And make a pass over this whole side of the instructions,
in particular for consistent handling of these concepts.
2018-09-27 13:20:10 -07:00
Greg Price 6f23d2f691 docs: Make a pass over LDAP config/setup docs.
Expand on a few things that tend to confuse people (especially the
`%(user)s` thing); move the `LDAPSearchUnion` example out to docs;
adjust the instructions to fit a bit better in their new docs/ home.
2018-09-27 13:20:10 -07:00
Greg Price e7737c6342 docs: Move prose LDAP instructions out of settings template.
This makes it easier to iterate on these, and to expand supplemental
information (like troubleshooting, or unusual configurations) without
further straining the already-dauntingly-long settings.py.

It also makes it easier to consult the instructions while editing the
secrets file, or testing things, etc. -- most admins will find it more
natural to keep a browser open somewhere than a second terminal.
2018-09-27 13:20:10 -07:00
Luis Silva 5cc0fb5c40 auth: Add example using django LDAPSearchUnion function.
This makes it clear how to do a union search, without having to look
anything up.
2018-09-21 11:04:40 -07:00
Vishnu Ks 9489ce0efc plans: Redirect to login if /plans is accessed from realm subdomain. 2018-09-05 13:44:35 -07:00
Shubham Padia 69bfa8c432 auth: Use different defaults for name and email for fakeldap.
Fixes part of #10297.
Use FAKE_LDAP_NUM_USERS which specifies the number of LDAP users
instead of FAKE_LDAP_EXTRA_USERS which specified the number of
extra users.
2018-08-31 17:07:02 +05:30
Roman Godov a2f407c11b api: Add an endpoint for listing the user groups in realm.
Adds endpoint for listing the user groups in realm.
Provides "description", "members", "name" and "id" for each user group.

Fixes #10298
2018-08-27 16:56:23 -07:00
Lyla Fischer 1efcdfdb5c templates: Merge Help and API doc main into documentation_main.
With minor fixes by eeshangarg.
2018-08-27 20:19:24 -02:30
Vishnu Ks 62b93d551a signup: Fix broken resend email link in realm creation flow.
Also use name for selecting form in casper tests
as form with action=new is present in both /new
and /accounts/new/send_confirm/ which breaks
test in CircleCI as
waitWhileVisible('form[action^="/new/"]) never stops
waiting.
2018-08-26 22:54:05 -07:00
Vishnu Ks d2e4417a72 urls: Separate endpoint for signup and new realm email confirm.
This is preparation for the next commit.
2018-08-26 22:53:57 -07:00
Harshit Bansal 96f00d1dfa settings: Remove unused emoji sprite CSS entries in static assets pipeline. 2018-08-26 22:16:23 -07:00
Vishnu Ks 9bb338be11 models: Add plan_type to Realm. 2018-08-21 12:39:06 -07:00
Tim Abbott 3cfb2000cc ldap: Improve error message for username/LDAP domain mismatches. 2018-08-20 10:39:34 -07:00
Rishi Gupta 27d4b1a5f0 billing: Add RealmAuditLogProcessor. 2018-08-13 17:12:50 -07:00
Tim Abbott da8f4bc0e9 push notifications: Add support for removing GCM push notifications.
This uses the recently introduced active_mobile_push_notification
flag; messages that have had a mobile push notification sent will have
a removal push notification sent as soon as they are marked as read.

Note that this feature is behind a setting,
SEND_REMOVE_PUSH_NOTIFICATIONS, since the notification format is not
supported by the mobile apps yet, and we want to give a grace period
before we start sending notifications that appear as (null) to
clients.  But the tracking logic to maintain the set of message IDs
with an active push notification runs unconditionally.

This is designed with at-least-once semantics; so mobile clients need
to handle the possibility that they receive duplicat requests to
remove a push notification.

We reuse the existing missedmessage_mobile_notifications queue
processor for the work, to avoid materially impacting the latency of
marking messages as read.

Fixes #7459, though we'll need to open a follow-up issue for
using these data on iOS.
2018-08-10 13:58:39 -07:00
Tim Abbott bf8557f4c2 mypy: Suppress mypy error with fakeldap configuration. 2018-08-09 14:33:50 -07:00
Shubham Padia 30c3d55cb3 auth: Add fakeldap based authentication method in development environment.
This uses the MockLDAP class of fakeldap to fake a ldap server, based
on the approach already used in the tests in `test_auth_backends.py`.

Adds the following settings:
- FAKE_LDAP_MODE: Lets user choose out of three preset configurations.
The default mode if someone erases the entry in settings is 'a'. The
fake ldap server is disable if this option is set to None.
- FAKE_LDAP_EXTRA_USERS: Number of extra users in LDAP directory beyond
the default 8.

Fixes #9934.
2018-08-09 13:51:38 -07:00
Shubham Padia d409555b2f auth: Add function for generating test ldap_dir to backends.py.
Generates ldap_dir based on the mode and the no. of extra users.
It supports three modes, 'a', 'b' and 'c', description for which
can be found in prod_settings_templates.py.
2018-08-09 13:46:44 -07:00
Tim Abbott e5668da879 settings: Fix a grammar typo in LDAP documentation. 2018-08-09 13:46:44 -07:00
Tim Abbott 49528a98eb settings: Set empty string default for AUTH_LDAP_BIND_PASSWORD.
That value is necessary to configure anonymous binds in
django-auth-ldap, which are useful when we're using LDAP just to
populate the user database.

Fixes #10257.
2018-08-09 10:02:40 -07:00
Tim Abbott bfb3dcec7e settings: Add a default value parameter to get_secret.
This lets us have a slightly cleaner interface for cases where we need
a custom default value than an `or None` in the definition (which
might cause issues with other falsey values).
2018-08-09 10:00:27 -07:00
Tim Abbott 65aa3d0848 backends: Import AppIdentityError inside GoogleMobileOauth2Backend.
We only need to import this for GoogleMobileOauth2Backend, so the same
performance reasoning for 271c7fbe65
applies here as well.
2018-08-08 14:19:42 -07:00
Tim Abbott 271c7fbe65 backends: Import googleapiclient lazily.
This saves about 30-50ms in the startup time of a Zulip manage.py
command.
2018-08-08 09:21:14 -07:00
Vishnu Ks c0ed2283d3 requirements: Upgrade django-auth-ldap to 1.6.0.
The autenticate function now follows the signature of
Django 2.0 https://github.com/django-auth-ldap/
django-auth-ldap/commit/27a8052b26f1d3a43cdbcdfc8e7dc0322580adae

Also AUTH_LDAP_CACHE_GROUPS is depricated in favor of
AUTH_LDAP_CACHE_TIMEOUT.
2018-08-02 15:53:11 -07:00
Aditya Bansal 3dd1613fff auth: Add 'auth' subdomain to ROOT_SUBDOMAIN_ALIASES list.
This makes it easy to use the recommended/default value of
SOCIAL_AUTH_SUBDOMAIN without having to edit ROOT_SUBDOMAIN_ALIASES.
2018-08-02 09:53:38 -07:00
Tim Abbott 7ea5987e5d errors: Use a setting to control the stream for slow-query logs.
We already had a setting for whether these logs were enabled; now it
also controls which stream the messages go to.

As part of this migration, we disable the feature in dev/production by
default; it's not useful for most environments.

Fixes the proximal data-export issue reported in #10078 (namely, a
stream with nobody ever subscribed to having been created).
2018-07-30 17:40:20 -07:00
Tim Abbott 02ae71f27f api: Stop using API keys for Django->Tornado authentication.
As part of our effort to change the data model away from each user
having a single API key, we're eliminating the couple requests that
were made from Django to Tornado (as part of a /register or home
request) where we used the user's API key grabbed from the database
for authentication.

Instead, we use the (already existing) internal_notify_view
authentication mechanism, which uses the SHARED_SECRET setting for
security, for these requests, and just fetch the user object using
get_user_profile_by_id directly.

Tweaked by Yago to include the new /api/v1/events/internal endpoint in
the exempt_patterns list in test_helpers, since it's an endpoint we call
through Tornado. Also added a couple missing return type annotations.
2018-07-30 12:28:31 -07:00
Tim Abbott 07af59d4cc tornado: Split get_events_backend into two functions.
The lower-layer function, now called get_events_backend, is intended
to be called by multiple code paths (including the upcoming
get_events_internal).
2018-07-30 12:28:31 -07:00
Priyank Patel f8398238a9 webpack: Migrate archive bundle to webpack.
Tested by visiting localhost:9991/archive/streams/1/topics/denmark3.
2018-07-23 15:19:44 -07:00
Shubham Padia e228687094 search: Make search query a contenteditable div for search pill.
Input pills require a contenteditable div with a class named input
to fall inside the pill container. On converting the input tag into
a div, the size of the input decreases which is compensated by a
line-height of 40px. Comment above letter-spacing:normal was removed
as chrome and firefox do not change the letter-spacing to normal
for a div via the default browser stylesheet.

NOTE: Currently writing something into the div will call the action
corresponding to that key in the keyboard shortcuts. The input will
work fine once the pills have been initiated.

For the casper tests, for now, we just use the legacy search code.
When we change that, $.val() cannot be used on contenteditable div, so
$.html() will need to be used instead in select_item_via_typeahead.
2018-07-23 11:29:10 -07:00
Tim Abbott 8dca8f1087 search: Add setting to control whether search pills are enabled.
This setting isn't intended to exist long term, but instead to make it
possible to merge our search pills code before we're ready to cut over
production environments to use it.
2018-07-23 11:29:10 -07:00
Aditya Bansal 529e4e76c8 thumbnaling: Support S3 upload backend in thumbor.
Credits for doing this goes to @sampritipanda.
2018-07-15 00:57:59 +05:30
Aditya Bansal 98a4e87e1d thumbor: Complete implementation of thumbnailing.
Various pieces of our thumbor-based thumbnailing system were already
merged; this adds the remaining pieces required for it to work:

* a THUMBOR_URL Django setting that controls whether thumbor is
  enabled on the Zulip server (and if so, where thumbor is hosted).

* Replaces the overly complicated prototype cryptography logic

* Adds a /thumbnail endpoint (supported both on web and mobile) for
  accessing thumbnails in messages, designed to support hosting both
  external URLs as well as uploaded files (and applying Zulip's
  security model for access to thumbnails of uploaded files).

* Modifies bugdown to, when THUMBOR_URL is set, render images with the
  `src` attribute pointing /thumbnail (to provide a small thumbnail
  for the image), along with adding a "data-original" attribute that
  can be used to access the "original/full" size version of the image.

There are a few things that don't work quite yet:
* The S3 backend support is incomplete and doesn't work yet.
* The error pages for unauthorized access are ugly.
* We might want to rename data-original and /thumbnail?size=original
  to use some other name, like "full", that better reflects the fact
  that we're potentially not serving the original image URL.
2018-07-15 00:39:41 +05:30
Aditya Bansal 2e837b1407 archives: Change the topic/<topic_name> code path to use topics prefix.
We do this for maintaining consistency in naming of the endpoints.
2018-07-14 10:39:35 +05:30
Aditya Bansal 63ec8b08b8 archives: Add endpoint to fetch topic history of web public streams.
In this commit we add a new endpoint so as to have a way of fetching
topic history for a given stream id without having to be logged in.
This can only happen if the said stream is web public otherwise we
just return an empty topics list. This endpoint is quite analogous
to get_topics_backend which is used by our main web app.

In this commit we also do a bit of duplication regarding the query
responsible for fetching all the topics from DB. Basically this
query is exactly the same as what we have in the
get_topic_history_for_stream function in actions.py. Basically
duplicating now is the right thing to do because this query is
really gonna change when we add another criteria for filtering
messages which is:
Only topics for messages which were sent during the period the
corresponding stream was web public should be returned.
Now when we will do this, the query will change and thus it won't
really be a code duplication!
2018-07-14 09:51:37 +05:30
Joshua Schmidlkofer b1a57d144f thumbor: Add production installer/puppet support.
This commits adds the necessary puppet configuration and
installer/upgrade code for installing and managing the thumbor service
in production.  This configuration is gated by the 'thumbor.pp'
manifest being enabled (which is not yet the default), and so this
commit should have no effect in a default Zulip production environment
(or in the long term, in any Zulip production server that isn't using
thumbor).

Credit for this effort is shared by @TigorC (who initiated the work on
this project), @joshland (who did a great deal of work on this and got
it working during PyCon 2017) and @adnrs96, who completed the work.
2018-07-12 20:37:34 +05:30
Aditya Bansal 9b485f3ef4 auth: Fix bug with subdomains and GitHub auth causing apparent logouts.
This adds a new settings, SOCIAL_AUTH_SUBDOMAIN, which specifies which
domain should be used for GitHub auth and other python-social-auth
backends.

If one is running a single-realm Zulip server like chat.zulip.org, one
doesn't need to use this setting, but for multi-realm servers using
social auth, this fixes an annoying bug where the session cookie that
python-social-auth sets early in the auth process on the root domain
ends up masking the session cookie that would have been used to
determine a user is logged in.  The end result was that logging in
with GitHub on one domain on a multi-realm server like zulipchat.com
would appear to log you out from all the others!

We fix this by moving python-social-auth to a separate subdomain.

Fixes: #9847.
2018-07-10 16:49:00 +05:30
Aditya Bansal 872e8c1d7b test_fixtures: Fix bug with run_db_migrations for test platform.
In this commit we are fixing a kinda serious un-noticed bug with
the way run_db_migrations worked for test db.
Basically run_db_migrations runs new migrations on db (dev or test).
When we talk about the dev platform this process is straight forward.
We have a single DB zulip which was once created and now has some data.
Introduction of new migration causes a schema change or does something
else but bottom line being we just migrate the zulip DB and stuff works
fine.

Now coming to zulip test db (zulip_test) situation is a bit complex
in comparision to dev db. Basically this is because we make use of
what we call zulip_test_template to make test fixture restoration
after tests run fast. Now before we introduced the performance
optimisation of just doing migrations when possible, introduction of
a migration would ideally result in provisioning do a full rebuild of
the test database. When that used to happen sequence of events used to
be something like this:
* Create a zulip_test db from zulip_test_base template (An absolute
basic schema holding)
* Migrate and populate the zulip_test db.
* Create/Re-create zulip_test_template from the latest zulip_test.

Now after we introduced just do migrations instead of full db rebuild
when possible, what used to happen was that zulip_test db got
successfully migrated but when test suites would run they would try to
create zulip_test from zulip_test_template (so that individual tests
don't affect each other on db level).
This is where the problem resides; zulip_test_template wasn't migrated
and we just scrapped zulip_test and re-created it using
zulip_test_template as a template and hence zulip_test will not hold the
latest schema.

This is what we fix in this commit.
2018-07-09 14:17:53 +05:30
Armaan Ahluwalia de40a7c2f6 webpack: Remove app files from settings.py in django.
Removes the app files from being included in the Django pipeline
as these files are now being compiled and included via webpack.
2018-07-07 08:00:34 +02:00
Armaan Ahluwalia 54d3d8e8b3 webpack: Transition app.js to be compiled by webpack.
This commit moves all files previously under the 'app' bundle in
the Django pipeline to being compiled by webpack under the 'app'
entry point. In the process, it moves assets under the app entry
to a file called app.js that consumes all relevant css and js files.

This commit also edits the webpack config to be able to expose certain
variables for third party libraries that are currently required by
some modules. This is bad coding form and should be refactored to
requiring whatever dependencies a module may have; we're just
deferring that to the future to simplify the series of transitions we
need to do here. The variable exposure is done using expose-loader in
webpack.

The app/index.html template is edited to override the newly introduced
'commonjs' block in the base template. This is done as a temporary
measure so as not to disrupt other pages on the app during the transition.

It also fixes the value of the 'this' context that was being inferred
as window by third party libraries. This is done using imports-loader
in the webpack config.  This is also messy and probably isn't how we
want things to work long term.
2018-07-05 11:03:08 +02:00
Tim Abbott 5a99118b3e auth: Restore a minimal SocialAuthMixin.
We need to do a small monkey-patching of python-social-auth to ensure
that it doesn't 500 the request when a user does something funny in
their browser (e.g. using the back button in the auth flow) that is
fundamentally a user error, not a server error.

This was present in the pre-rewrite version of our Social auth
codebase, without clear documentation; I've fixed the explanation
part here.

It's perhaps worth investigating with the core social auth team
whether there's a better way to do this.
2018-07-03 18:53:59 +02:00
Tim Abbott c9b0c0add4 github: Refactor email extraction to use the full emails data set.
It's possible to make GitHub social authentication support letting the
user pick which of their verified email addresses to pick, using the
python-social-auth pipeline feature.  We need to add an additional
screen to let the user pick, so we're not adding support for that now,
but this at least migrates this to use the data set of all emails that
have been verified as associated with the user's GitHub account (and
we just assume the user wants their primary email).

This also fixes the inability for very old GitHub accounts (where the
`email` field in the details might be a string the user wanted on
their GitHub profile page) to using GitHub auth to login.

Fixes #9127.
2018-07-03 18:35:29 +02:00
Yashashvi Dave 2b478268f3 third: Add sortablejs library. 2018-07-01 02:05:00 -07:00
Vishnu Ks 74351fb250 send_email: Add function to generate tokenized noreply in FromAddress. 2018-06-23 12:03:30 -07:00
Shubham Padia 6853530eba search: Add search_pill.js with basic operations.
Adds search_pill.js to the static asset pipeline. The items
for search pill contain 2 keys, display_value and search_string.
Adding all the operator information i.e the operator, operand and
negated fields along with the search_string and description was tried out.
It was dropped because it didn't provide any advantage as one had to
always calculate the search_string and the description from the operator.
2018-06-22 09:05:36 -04:00
Aditya Bansal d8e0c959bc cache: Enable compression in pylibmc for cache stored in memcached.
This fixes an important issue where the realm_users cache could grow
beyond 1MB when a Zulip server had more than about 10K users.  The
result was that Zulip would start 500ing with that size of userbase.

There are probably better long-term fixes, but because the realm_users
data set caches well, this change should be sufficient to let us
handle to 50-100K users or more on that metric (though at some point,
we'll start having other problems interacting with the realm_users
data set).
2018-06-16 05:24:42 -07:00
Aditya Bansal 998fb038a9 django-pylibmc: Add dependancy and use as cache backend for memcached. 2018-06-16 05:24:19 -07:00
Yashashvi Dave f7f039e772 custom fields: Add endpoint to delete value of custom profile data.
Add delete endpoint at `users/me/profile_data` to delete value of
custom profile data.

Fixes #9669
2018-06-07 08:35:04 -07:00
Steve Howell 8ce8e93645 refactor: Extract settings_panel_menu.js. 2018-06-06 09:42:33 -07:00
Tim Abbott 5ce69b3ecb backends: Fix unnecessary duplicate query to realm in social auth.
This is just a small cleanup to the social auth backend code.
2018-06-06 00:31:59 -07:00
Tim Abbott 35c4a9f1d2 auth: Rewrite our social auth integration to use pipeline.
This new implementation model is a lot cleaner and should extend
better to the non-oauth backend supported by python-social-auth (since
we're not relying on monkey-patching `do_auth` in the OAuth backend
base class).
2018-06-05 23:24:48 -07:00
Raymond Akornor 0335da7e05 python: Use python3 style super() in 2FA code paths. 2018-06-04 20:55:42 -07:00
Steve Howell ad7e3ddd9f casper: Show casper debug files in the dev server.
Example:
    http://localhost:9991/casper/casper-failure1.png
2018-06-04 09:02:58 -07:00
Tim Abbott 714ad67db1 storage: Move library code into zerver/lib.
There's no particularly good reason for this to have been a top-level
file.
2018-06-03 16:02:02 -07:00
Steve Howell 4b2e8b83c4 slash commands: Add /ping command (via zcommand).
This adds a /ping command that will be useful for users
to see what the round trip to the Zulip server is (including
only a tiny bit of actual server time to basically give a
200).

It also introduce the "/zcommand" endpoint and zcommand.js
module.
2018-06-02 09:40:12 -07:00
Tim Abbott 2d6993f6dc settings: Enable caching with webpack-loader. 2018-05-31 16:45:51 -07:00
Tim Abbott 47824a97a4 ldap: Add return_data for the ldap_missing_attribute property.
This should make it possible in the future to do better error output
for this case.
2018-05-31 14:16:03 -07:00
Tim Abbott ecb3a2ccef ldap: Clarify outside_ldap_domain exception logic.
The previous logic made it look like catching ZulipLDAPException on
the authenticate() line was possible, but it isn't, because that
exception is actually being handled inside django-auth-ldap's
authenticate method.
2018-05-31 14:12:06 -07:00
Sampriti Panda 46711a43f0 pgroonga: Upgrade to PGroonga 2 API. 2018-05-31 13:00:34 -07:00
Yashashvi Dave 07b63e4886 jquery-form: Remove jquery-form plugin dependency.
Fixes #9430.
2018-05-29 15:44:40 -07:00
Tim Abbott 91ec0aba09 auth: Improve interactions between LDAPAuthBackend and EmailAuthBackend.
Previously, if you had LDAPAuthBackend enabled, we basically blocked
any other auth backends from working at all, by requiring the user's
login flow include verifying the user's LDAP password.

We still want to enforce that in the case that the account email
matches LDAP_APPEND_DOMAIN, but there's a reasonable corner case:
Having effectively guest users from outside the LDAP domain.

We don't want to allow creating a Zulip-level password for a user
inside the LDAP domain, so we still verify the LDAP password in that
flow, but if the email is allowed to register (due to invite or
whatever) but is outside the LDAP domain for the organization, we
allow it to create an account and set a password.

For the moment, this solution only covers EmailAuthBackend.  It's
likely that just extending the list of other backends we check for in
the new conditional on `email_auth_backend` would be correct, but we
haven't done any testing for those cases, and with auth code paths,
it's better to disallow than allow untested code paths.

Fixes #9422.
2018-05-28 22:47:47 -07:00
Tim Abbott 8119670da1 user_settings: Prevent LDAP users from setting a Zulip password.
Previously, if both EmailAuthBackend and LDAPAuthBackend were enabled,
LDAP users could set a password using EmailAuthBackend and continue to
use that password, even if their LDAP account was later deactivated.

That configuration wasn't supported at all before, so this doesn't fix
a pre-existing security issue, but now that we're making that a valid
configuration, we need to cover this case.
2018-05-28 22:47:47 -07:00
Eeshan Garg 0a1b429d04 settings: Remove PRIVATE_STREAM_HISTORY_FOR_SUBSCRIBERS.
Now that we have a UI for this feature, we don't need this setting
anymore.
2018-05-24 17:18:14 -07:00
Vishnu Ks 54a002c2e2 requirements: Upgrade pyflakes to 2.0.0.
We fix a few errors that only the new version finds.
2018-05-24 11:31:36 -07:00
Steve Howell 1b57e568ff Add "zform" functionality to the web client.
A "zform" knows how to render data that follows our
schema for widget messages with form elements like
buttons and choices.

This code won't be triggered until a subsequent
server-side commit takes widget_content from
API callers such as the trivial chat bot and
creates submessages for us.
2018-05-24 09:30:22 -07:00
Steve Howell 435719c65b Add a schema checking concept to the web app.
This starts the concept of a schema checker, similar to
zerver/lib/validator.py on the server.  We can use this
to validate incoming data.  Our server should filter most
of our incoming data, but it's useful to have client-side
checking to defend against things like upgrade
regressions (i.e. what if we change the name of the field
on the server side without updating all client uses).
2018-05-24 09:30:22 -07:00
Umair Khan 9502cbbfab 2FA: Enable Django template backend.
The only purpose of this commit is to make the django templates
of Two Factor Auth work. We probably won't need this commit once
we upgrade the admin backend of Two Factor Auth to use handlebar
templates.
2018-05-23 15:46:56 -07:00
Armaan Ahluwalia 1525e92058 logging/errors/webpack: Improve error stack traces with blueslip.
This commit improves the output that blueslip produces while
showing error stack traces on the front-end. This is done by
using a library called error-stack-parser to format the stack
traces.

This commit also edits the webpack config to use a different
devtool setting since the previous one did not support sourcemaps
within stack traces. It also removes a plugin that was obviated
by this change.
2018-05-22 14:36:41 -07:00
Umair Khan f38d6ac6fe ldap: Make Zulip compatible with django-auth-ldap==1.5.
In version 1.5, get_or_create_user method is not used. It exists just
for the compatibility. The main function to use now is
get_or_build_user.

See the changelog:
https://django-auth-ldap.readthedocs.io/en/latest/changes.html#id1

Fixes #9307
2018-05-22 08:13:41 -07:00
Aditya Bansal 81a677e02b reminders: Refactor relevant code to live in a separate reminder.js. 2018-05-21 09:03:31 -07:00
Aditya Bansal 57f247283d scheduledmessages: Handle multiple servers running this delivery job.
We add conditional infinite sleep to this delivery job as a means to
handle case of multiple servers in service to a realm running this
job. In such a scenerio race conditions might arise leading to
multiple deliveries for same message. This way we try to match the
behaviour of what other jobs do in such a case.
Note: We should eventually do something to make such jobs work
while being running on multiple servers.
2018-05-21 08:53:14 -07:00
Sampriti Panda 3f4200db3c tests: Disable slow query messages in test environment.
Slow queries during backend tests sends messages to Error Bot
which affects the database state causing the tests to fail.
This fixes the occasional flakes due to that.
2018-05-20 10:16:53 -07:00
Yashashvi Dave 4162e61f33 /json/users: Replace email with user_id in API to reactivate user. 2018-05-18 15:20:43 -07:00
Yashashvi Dave 06e7e933cc /json/users: Replace email with user_id in API to update/remove users. 2018-05-18 15:20:43 -07:00
Steve Howell aa7c031f36 Add tictactoe widget.
Thanks to @amanagr for a few refinements to my original
version, which was really raw.  (It's still raw.)
2018-05-16 15:13:33 -07:00
Steve Howell d0e70a9700 Add poll widget. 2018-05-16 15:13:33 -07:00
Steve Howell b48f052b0c Add basic widgets framework (JS side). 2018-05-16 15:13:33 -07:00
Steve Howell a84bd6da57 Add client code for handling submessages.
This commit lays the foundation to handle submessages for
plugin widgets.  Right now it just logs events, but subsequent
commits will add widget functionality.
2018-05-16 15:13:33 -07:00
Steve Howell de47eeb6f1 Add /submessage endpoint. 2018-05-16 15:13:33 -07:00
Steve Howell 33d753d174 Add setting for ALLOW_SUB_MESSAGES. 2018-05-16 15:13:33 -07:00
Tim Abbott e63541e616 docs: Extract documentation for file upload backends.
This moves the documentation for this feature out of
prod_settings_template.py, so that we can edit it more easily.

We also add a bucket policy, which is part of what one would want to
use this in production.

This addresses much, but not all, of #9361.
2018-05-16 15:00:54 -07:00
Aman Ghebreweldi a781b89ec2 third: Use Underscore throttle function instead of jQuery.
Since we already have _.js and it has this feature, there's no need
for a separate library.
2018-05-16 11:56:49 -07:00
Steve Howell d2ee76896c Extract local_message.js.
We will eventually want these functions to support things
other than local echo, such as ephemeral messages from chat
bots.
2018-05-16 09:30:49 -07:00
Yashashvi Dave b949d10592 /json/bots: Replace email with user_id in API to generate bot_api_key.
Fixes #3643.
2018-05-15 16:37:06 -07:00
Yashashvi Dave d6e2f9fc88 /json/bots: Replace email with user_id in API to update bots. 2018-05-15 16:34:17 -07:00
Aditya Bansal 83d422d5bc zproject: Change use of typing.Text to str. 2018-05-10 14:19:49 -07:00
Raymond Akornor 1a215d5504 zproject: Migrate away from six. 2018-05-09 18:57:01 +00:00
Tim Abbott c7b44d44e4 third: Remove now-unused LazyLoad library.
We don't reference this anymore (it was only ever used by the Dropbox
integration, which was hardcoded-off for years before being removed in
e6833b6427)
2018-05-08 14:38:27 -07:00
Steve Howell 27b7461e0a Revert "js: Implement DynamicText class."
This reverts commit 6e7305f784.

We never ending up using this class.
2018-05-08 12:13:55 -07:00
Tim Abbott cfd22c6832 zilencer: Clean up logic around mobile push notifications signup.
This fixes exceptions when sending PMs in development (where we were
trying to connect to the localhost push bouncer, which we weren't
authorized for, but even if we were, it wouldn't work, since there's
no APNS/GCM certs).

At the same time, we also set and order of operations that ensures one
has the opportunity to adjust the server URL before submitting
anything to us.
2018-05-05 16:42:00 -07:00
Steve Howell 264dcb6f40 refactor: Extract MessageListData class.
Most of this was straightforward.

Most functions that were grabbed verbatim and whole from
the original class still have one-line wrappers.

Many functions are just-the-data versions of functions that
remain in MessageList:  see add, append, prepend, remove as
examples.  In a typical pattern the MessageList code becomes
super simple:

    prepend: function MessageList_prepend(messages) {
        var viewable_messages = this.data.prepend(messages);
        this.view.prepend(viewable_messages);
    },

Two large functions had some minor surgery:

    triage_messages =
            top half of add_messages +
            API to pass three lists back

    change_message_id =
            original version +
            two simple callbacks to list

For the function update_muting_and_rerender(), we continue
to early-exit if this.muting_enabled is false, and we copied
that same defensive check to the new function
named update_items_for_muting(), even though it's technically
hidden from that codepath by the caller.
2018-05-05 06:31:51 -07:00
Tim Abbott 43098a6f7c zilencer: Add automated signup system for push notifications.
Based on an initial version by Rishi Gupta.

Fixes #7325.
2018-05-03 21:27:49 -07:00
Armaan Ahluwalia cca10beb78 css/webpack: Moved archive styles to webpack bundle.
This commit moves the stylesheets under the archive bundle in
the Django pipeline to being compiled by webpack instead. It
also removes a remaining call to a portico stylesheet that no
longer exists.
2018-05-03 19:15:49 -07:00
Armaan Ahluwalia 54bf2a6231 css/webpack: Transition landing-page.css to webpack.
This commit transitions landing-page.css from the Django pipeline
to being compiled by webpack as landing-page.scss under the
'landing-page' and 'integration' bundles.
2018-05-03 19:15:23 -07:00
Armaan Ahluwalia 1c016e990d css/webpack: Transition common.css to webpack.
This commit transitions common.css from the Django pipeline
to being compiled by webpack under the common bundle.
2018-05-03 19:15:23 -07:00
Umair Khan 9a3b310db9 2FA: Enable apps and middlewares unconditionally. 2018-05-03 11:59:32 -07:00
Priyank Patel 1e1b72f6c8 webpack: Transition stats.css to use webpack.
The stats.scss file is added to activity bundle.
2018-05-03 08:08:23 -07:00
Priyank Patel c780bc33ba webpack: Transition activity.css to use webpack.
The activity.scss file is added to activity bundle.
2018-05-03 07:53:24 -07:00
Armaan Ahluwalia 5f7b47e20c css: Transition 'app.css' to SCSS.
This commit transitions all styles in app.css in the Django pipeline
to being compiled by webpack in an app-styles bundle, and renames the
various files to now be processed as SCSS.

To implement this transition, we move the old CSS file refernces in
settings.py and replace them with a bundle declared in
`webpack.assets.json` and includedn in the index.html template

Tweaked by tabbott to keep the list of files in `app.css` in
`webpack.assets.json`, and to preserve the ordering from the old
`settings.py`.
2018-05-02 17:13:16 -07:00
Aditya Bansal 583f50179c public_archives: Add floating recipient bar. 2018-05-02 15:23:33 -07:00
Aditya Bansal 898c281692 public_archives: Add styling to the public topics. 2018-05-02 15:23:33 -07:00
Aditya Bansal 5d7907b59f public_archives: Add basic infra for displaying topics.
We add very basic infra so that we can view any discussion which
happened under a topic of a global public stream without
authorization.
2018-05-02 15:23:33 -07:00
Umair Khan 29e3a1d576 two_factor: Add templates for 2-factor-auth setup.
Note from Tim: We'll likely need to do some work on the strings in
these before translating, so I removed some translation tags.
2018-05-02 15:20:49 -07:00
Umair Khan e8b0ae821b two_factor: Fix namespace error.
If you specify namespace, Django throws an error.
2018-05-02 14:59:14 -07:00
Armaan Ahluwalia fce6882eb9 webpack: Move the styles consumed by 5xx.html to webpack.
This commit removes the need for portico.css to be generated
by the Django pipeline and makes the error page use the css
file compiled by webpack instead.
2018-05-02 09:45:01 -07:00
Tim Abbott d92edb8ea5 css: Remove support for legacy desktop app.
We haven't seen significant traffic from the legacy desktop app in
over a year, and users using it get a warning to upgrade since last
summer, so it's probably OK to stop providing special fonts for it.
2018-05-01 09:49:28 -07:00
Angelika Serwa f4f64243dd custom_profile_fields: Support changing the sort order of the fields.
Tweaked by tabbott for variable naming and the URL.

Closes #8879.
2018-04-30 18:17:41 -07:00
Tim Abbott 7d6bb3dcb4 settings: Remove obsolete default_desktop_notifications setting.
This actually hasn't been hooked up to do anything in years.

While we're at it, we remove the entire "Zulip Labs" settings page.
2018-04-28 13:46:07 -07:00
Steve Howell 97b9367d20 refactor: Extract scroll_util.js.
We now use scroll_element_into_container() in two different
places, so it's worth extracting.
2018-04-28 11:15:14 -07:00
Steve Howell fb712027bf buddy list: Fix and simplify up/down navigation.
This introduces a generic class called list_cursor to handle the
main details of navigating the buddy list and wires it into
activity.js.  It replaces some fairly complicated code that
was coupled to stream_list and used lots of jQuery.

The new code interacts with the buddy_list API instead of jQuery
directly.  It also persists the key across redraws, so we don't
lose our place when a focus ping happens or we type more characters.

Note that we no longer cycle to the top when we hit the bottom, or
vice versa.  Cycling can be kind of an anti-feature when you want to
just lay on the arrow keys until they hit the end.

The changes to stream_list.js here do not affect the left sidebar;
they only remove code that was used for the right sidebar.
2018-04-28 11:15:14 -07:00
Steve Howell 65d8eb3189 buddy list: Extract user_search.js.
This was a bit more than moving code.  I extracted the
following things:

        $widget (and three helper methods)
        $input
        text()
        empty()
        expand_column
        close_widget
        activity.clear_highlight

There was a minor bug before this commit, where we were inconsistent
about trimming spaces.  The introduction of text() and empty() should
prevent bugs where users type the space bar into search.
2018-04-28 11:15:14 -07:00
Armaan Ahluwalia 93ac40105f CSS: Move portico styles to webpack compilation.
static/styles/scss/portico.scss is now compiled by webpack
and supports SCSS syntax.

Changed the server-side templates to render the portico-styles
bundle instead of directly requiring the portico stylesheet. This
allows webpack to handle stylesheet compilation and minification.

We use the mini-css-extract-plugin to extract out css from the
includes in webpack and let webpacks production mode handle
minification. Currently we're not able to use it for dev mode
because it does not support HMR so we use style-loader instead.
Once the plugin supports HMR we can go on to use it for both
dev and prod.

The downside of this is that when reloading pages in the development
environment, there's an annoying flash of unstyled content :(.

It is now possible to make a change in any of the styles included
by static/styles/scss/portico.scss and see the code reload live
in the browser. This is because style-loader which we currently
use has the module.accept code built-in.
2018-04-27 09:04:50 -07:00
Tim Abbott e4ca6e947b ldap: Disable django-auth-ldap caching of users.
This shouldn't have a material performance impact, since we don't
query these except during login, and meanwhile this fixes an issue
where users needed to restart memcached (which usually manifested as
"needing to reboot the whole server" after updating their LDAP
configuration before a user who was migrated from one OU to another
could login).

Fixes #9057.
2018-04-24 09:57:55 -07:00
Aditya Bansal ae398dc48b csp_nonce: Add nonce to script tags loading minified JS. 2018-04-24 06:13:21 +05:30
Steve Howell 3f1930f9c5 buddy list: Extract buddy_data.js. 2018-04-22 20:08:08 -07:00
Steve Howell 536236d9b1 buddy list: Extract buddy_list.js. 2018-04-22 20:08:08 -07:00
Tim Abbott a9fb02b712 test_auth_backends: Add a test for GitHub auth mobile_flow_otp. 2018-04-22 19:55:05 -07:00
Tim Abbott 64023fc563 auth: Fix incorrect use of get_realm_from_request.
The code in maybe_send_to_registration incorrectly used the
`get_realm_from_request` function to fetch the subdomain.  This usage
was incorrect in a way that should have been irrelevant, because that
function only differs if there's a logged-in user, and in this code
path, a user is never logged in (it's the code path for logged-out
users trying to sign up).

This this bug could confuse unit tests that might run with a logged-in
client session.  This made it possible for several of our GitHub auth
tests to have a totally invalid subdomain value (the root domain).

Fixing that bug in the tests, in turn, let us delete a code path in
the GitHub auth backend logic in `backends.py` that is impossible in
production, and had just been left around for these broken tests.
2018-04-22 16:24:43 -07:00
Tim Abbott 65025e8327 auth: Add return_data for RemoteUserBackend.
This is done mainly because this backend has the simplest code path
for calling login_or_register_remote_user, more than because we expect
this case to come up.  It'll make it easier to write unit tests for
the `invalid_subdomain` corner case.
2018-04-22 14:44:06 -07:00
Tim Abbott fb6cc4cb65 secrets: Make it a bit easier to do GitHub auth in development. 2018-04-22 14:44:05 -07:00
Steve Howell f73bfd2a5c refactor: Extract settings_toggle.js.
This mostly moves code, and we also abandon the lookup mechanism
for finding our toggler.
2018-04-20 13:45:58 -07:00
Tim Abbott aa4b067e68 avatars: Fix 500 with the /avatar/ URL to be a 404.
Apparently, we had a somewhat sloppy regular expression for the URL
for this endpoint.
2018-04-18 12:43:57 -07:00
Tim Abbott e6833b6427 cleanup: Remove the legacy Dropbox file upload integration.
This has been hard-disabled for years, we have no plans to re-enable
it, and it has some hacky code in it.
2018-04-11 11:39:48 -07:00
Aditya Bansal 6c1a50da76 csp_reports: Add endpoint to handle logging of reports sent by clients. 2018-04-11 23:01:13 +05:30
Vishwesh Jainkuniya 383c62fb03 dev_login: Identify each user's realm when listing them.
This is a mobile-specific endpoint used for logging into a dev server.
On mobile without this realm_uri it's impossible to send a login request
to the corresponding realm on the dev server and proceed further; we can
only guess, which doesn't work for using multiple realms.

Also rename the endpoint to reflect the additional data.

Testing Plan:
Sent a request to the endpoint, and inspected the result.

[greg: renamed function to match, squashed renames with data change,
 and adjusted commit message.]
2018-04-10 17:03:36 -07:00
Greg Price 9a0fdf5b8d settings: Add a few more section headings.
This should help a bit more in making this file navigable.

I think there's further work that could be done to organize the
settings better: e.g., group LDAP with the auth section; separate
resource limits, from debugging and error reporting, from configuring
service dependencies like Redis and Rabbit.  That'd require reordering
many settings, and also taking a closer look at many settings one by
one in order to do a good job.  Leaving that for another day.
2018-04-05 21:24:48 -07:00
Greg Price 9a9d3097be settings: Add some visual weight to the section headings.
I've found this file hard to navigate for a while.  We actually have a
little hierarchy of section headings which applies to a lot of the
file already; make the boundaries bolder.
2018-04-05 21:24:48 -07:00
Greg Price f597f0b52e settings: Revise block comment at top of file.
Hopefully this is a bit clearer to read.
2018-04-05 21:24:48 -07:00
Greg Price 9956d61e20 settings: Revise comments on SMTP / outgoing email settings.
Add a clear heading, and use fewer words and simpler sentences.  Also
explain the password thing a bit more, and put that more inline next
to the username.

Also, on checking the Django docs, the default for EMAIL_USE_TLS
is False and for EMAIL_PORT is 25.  So most admins, certainly any that
are using an SMTP service on the public Internet (that is at all
decently run), will need to set those settings.  Mention that.
2018-04-05 21:24:48 -07:00
Steve Howell bd591424e2 Add keydown_util.js module.
This is a pretty thin abstraction to prevent having to put
magic numbers in code, doing the which/keyCode hack, and remembering
to all preventDefault.

Hopefully we'll expand it to handle things like shift/alt keys
for components that want their own keyboard handlers (vs. going
through hotkey.js).
2018-04-04 16:37:39 -07:00
Tim Abbott bec71d7a50 messages: Add a server-level setting to control private stream history.
We don't indend for this server-level setting to exist in the long
term; the purpose of this is just to make it easy to test this code
path for development purposes.

This implements much of the Message side part of #2745.
2018-04-04 16:18:46 -07:00
Tim Abbott 938c4cee08 settings: Add option to control Jitsi video chat integration.
Fixes #8922.
2018-04-02 16:55:16 -07:00
Tim Abbott af004fa6f5 rabbitmq: Connect to rabbitmq via 127.0.0.1 rather than localhost.
This should eliminate the occasional problems that some installations
have had with RabbitMQ trying to load-balance requests between
127.0.0.1 and ::1 (the ipv6 version of localhost).
2018-04-01 16:11:12 -07:00
Steve Howell 9ce9c2f9db Extract info_overlay.js.
There are several ways we open help for keyboard shortcuts,
markdown help, and search operators.

    - from the gear menu
    - from the compose box
    - from the search box
    - hitting ? for keyboard help
    - arrowing/clicking through the tabs

This just moves the relevant code into a module and changes a
bunch of one-line calls in various places.
2018-03-30 09:07:23 -07:00
Umair Khan c14cefc24c settings: Add support for adding/removing custom profile fields.
Now that we have support for displaying custom profile fields, this
adds administrator-level support for creating them.

Tweaked by tabbott to fix a few small bugs and clean up the commit message.

Fixes #1760.
2018-03-29 13:59:16 -07:00
Tim Abbott 656f882a44 bots: Eliminate NEW_USER_BOT.
This bot was basically a duplicate of NOTIFICATION_BOT for some
specific corner cases, and didn't add much value.  It's better to just
eliminate it, which also removes some ugly corner cases around what
happens if the user account doesn't exist.
2018-03-29 12:01:21 -07:00
Tim Abbott 013a8886b2 styles: Rename dark.css to night_mode.css. 2018-03-28 10:42:28 -07:00
Steve Howell 4f05cefb32 Remove obsolete code for stream subscribe buttons.
In 18e43895ff we replaced
stream subscribe buttons with stream links.  The new feature
has been well tested and well received for over a year now,
so it's safe to remove the older feature at this point.

Older sites will have super old messages that still have the
rendered markup; this commit does not attempt to address those
situations.  Most likely, clicking on an old button in the old
message will either do nothing or look like a message reply.
2018-03-24 12:31:36 -04:00