Commit Graph

2069 Commits

Author SHA1 Message Date
Tim Abbott 65025e8327 auth: Add return_data for RemoteUserBackend.
This is done mainly because this backend has the simplest code path
for calling login_or_register_remote_user, more than because we expect
this case to come up.  It'll make it easier to write unit tests for
the `invalid_subdomain` corner case.
2018-04-22 14:44:06 -07:00
Tim Abbott f806526551 auth: Remove unused request.verified_email variable.
This has never been used since it was introduced in the very first
version of our Google auth code in April 2013.
2018-04-22 14:44:06 -07:00
Shubham Dhama 26d2ffa821 populate_db: Add new user of "guest" type.
The purpose of this user is to act as a guest.
(This is a preliminary step in adding the guest type of user
and is a part of #8385.)
2018-04-20 16:20:00 -07:00
Umair Khan 0420b89468 profile: Add hint attribute to custom profile fields.
This is the model for #8876.
2018-04-19 11:32:35 -07:00
Aditya Bansal 6c1a50da76 csp_reports: Add endpoint to handle logging of reports sent by clients. 2018-04-11 23:01:13 +05:30
Vishwesh Jainkuniya 383c62fb03 dev_login: Identify each user's realm when listing them.
This is a mobile-specific endpoint used for logging into a dev server.
On mobile without this realm_uri it's impossible to send a login request
to the corresponding realm on the dev server and proceed further; we can
only guess, which doesn't work for using multiple realms.

Also rename the endpoint to reflect the additional data.

Testing Plan:
Sent a request to the endpoint, and inspected the result.

[greg: renamed function to match, squashed renames with data change,
 and adjusted commit message.]
2018-04-10 17:03:36 -07:00
Vishnu Ks e92838a31f registration: Catch email validation error and show error message. 2018-04-06 15:18:32 -07:00
Tim Abbott ad861c5fae messages: Improve comment on need_messages. 2018-04-06 08:57:46 -07:00
Tim Abbott b0b134cb4c help: Clean up settings_html and subscriptions_html.
After some thinking, I don't think there's any actual value to doing
the ../ style relative links here, whereas there is actual harm from
the links being slightly broken in the current model.  We fix this by
just using /#settings as the URL.

Fixes #8978.
2018-04-05 14:48:26 -07:00
Steve Howell 36844418e9 bug fix: Respect include_history for certain queries.
For certain queries where both include_history and
use_first_unread_anchor are set to True, we were excluding
historical rows.  Now we only use the use_first_unread_anchor
flag to filter rows that we use to find the anchor, without
having it filter the actual search results.

The bug went unreported for a long time, because it only
affected mobile users who had newly subscribed to streams.

Note that we make a small change to the test called
test_use_first_unread_anchor_with_muted_topics, which has
a very scary comment about being "arcane" and "be
absolutely sure you know what you're doing."  I think it's
fine.

Also, the new test code would fail before this fix, so it
should help prevent future regressions.

Fixes #8958
2018-04-05 17:16:41 -04:00
Steve Howell b64117d872 refactor: Build query in find_first_unread_anchor().
This is a bit more than a pure refactor, because we duplicate a
chunk of code to calculate a query inside of
find_first_unread_anchor(), so we're doing a bit more work
than before.

We need this refactoring to start decoupling find_first_unread_anchor
from get_messages_backend for the case where include_history is
True.  This will happen in a subsequent commit.

The only test that changes here is a direct test on
find_first_unread_anchor().  All other tests pass without
modification, and we have decent coverage on get_messages_backend.
2018-04-05 17:16:41 -04:00
Steve Howell 345d44b5f1 Extract get_base_query_for_search(). 2018-04-05 17:16:41 -04:00
Steve Howell 59a9b69c25 Simplify search code for keyword searches.
We use an array now to build up the list of search operands and
then consolidate the special search handling after the loop (which
means setting the flag, putting two more columns in the query, and
using ' '.join to build the string).
2018-04-05 13:27:31 -07:00
Steve Howell d521906fb6 search refactor: Extract add_narrow_conditions().
This code was basically pulled from two near-to-each-other
sections of get_messages_backend, and it does an early
return for narrow-is-None.
2018-04-05 13:27:31 -07:00
Steve Howell 3ac660d972 minor: Show `narrow` for UnicodeDecodeError.
We have a debugging statement for some obscure errors we get
when narrows have search terms.  We now show all the narrow
operators.  This isn't really to improve debugging; it's more
to make it easier in the next commit to extract a function
that would make search_term have to be passed back in a tuple.
But it shouldn't hurt debugging either.
2018-04-05 13:27:31 -07:00
Tim Abbott 98889608a2 help: Fix structure of markdown context logic.
Refactoring in this file had resulted in the logic for
html_settings_link being duplicated and extra logic being needed to
ensure these variables were set where they were needed.

This fixes subscriptions_html not being rendered properly in the /help
and /api pages, in addition to removing duplicate code.
2018-04-05 12:22:41 -07:00
Steve Howell 4cba679d38 Move code for find_first_unread_anchor().
This is a pure refactoring and just pulls the function out
to the top level of the module.  (The prior commit extracted
it inside a larger function to make a nicer diff.)
2018-04-05 09:55:54 -07:00
Steve Howell d8a95c6517 Extract find_first_unread_anchor().
This is a pure refactoring.
2018-04-05 09:55:54 -07:00
Tim Abbott 228f41e916 messages: Pass UserProfile to is_public_stream_by_name and rename.
The new name can_access_stream_history_by_name gets to the point of
what this function actually does.  And passing in a user object lets
us define what this does based on the user subscribed.
2018-04-04 15:13:11 -07:00
Tim Abbott 5e82d750c5 get_messages: Refactor ok_to_include_history to accept a UserProfile.
If we make history accessible to some stream subscribers of private
streams, we'll need the UserProfile to be available here.
2018-04-04 15:06:53 -07:00
Tim Abbott 758d7b9146 bots: Clean up editing bots impacting non-bot users.
This fixes a bug where the endpoint for editing bot users would allow
an organization administrator to edit the full name of a bot user.

A combination of this an another recently fixed bug made it possible
for this process to set a `bot_owner` for a non-bot user; so we also
include a migration to fix that for any users that might have had our
model invariants corrupted in that way.
2018-04-02 18:58:26 -07:00
Tim Abbott 938c4cee08 settings: Add option to control Jitsi video chat integration.
Fixes #8922.
2018-04-02 16:55:16 -07:00
Shubham Dhama b7aae89029 org settings: Add realm level default time format setting. 2018-03-31 16:23:26 -07:00
Tim Abbott 656f882a44 bots: Eliminate NEW_USER_BOT.
This bot was basically a duplicate of NOTIFICATION_BOT for some
specific corner cases, and didn't add much value.  It's better to just
eliminate it, which also removes some ugly corner cases around what
happens if the user account doesn't exist.
2018-03-29 12:01:21 -07:00
Tarun Kumar 86eaa553a6 user-groups: Add backend enforcing for new modification settings.
Add function in user-groups.py for getting member ids
for a group.
Update view to enforce checks for modifying user-groups.
Only admins and user group members can modify user-groups.
2018-03-23 14:44:47 -07:00
Tim Abbott fcd265304c message editing: Fix buggy timestamp check.
This somehow got messed up in a final refactoring before merging
91197fa4f1.
2018-03-22 16:26:18 -07:00
Sarah 91197fa4f1 org settings: Add logic for applying allow_community_topic_editing.
Applies the logic to allow community members to edit topics
of others' messages if this setting is True. Otherwise,
only administrators can update the topic of others' messages.

This logic includes a 24-hour time limit for community topic editing.
2018-03-22 16:13:36 -07:00
Sarah f5c2fb8438 org settings: Create backend api for allow_community_topic_editing.
Adds the code for updating the allow_community_topic_editing
setting.
2018-03-22 16:02:27 -07:00
Jack Weatherilt 8535625341 parse_user_agent: Assert user agent is not None.
This commit asserts that parse_user_agent never returns None. The
RegEx will match any string, so that `match` is never None. This
brings test coverage of lib/user_agent.py to 100%. Changes were also
made in test/test_decorators.py and views/compatibility.py to reflect
that parse_user_agent cannot return None.

Improves: #7089.
Fixes: #8779.
2018-03-22 14:29:29 -07:00
Aditya Bansal 2a2df0ef5e auth: Make redirects to next work for REMOTE_USER based Apache SSO.
It's possible that this won't work with some versions of the
third-party backend, but tabbott has tested carefully that it does
work correctly with the Apache basic auth backend in our test
environment.
2018-03-21 14:01:05 -07:00
Aditya Bansal 1e48dac8f3 auth.py: Make redirects to 'next' url work for google and github.
In this commit we start to support redirects to urls supplied as a
'next' param for the following two backends:
* GoogleOAuth2 based backend.
* GitHubAuthBackend.
2018-03-21 13:35:44 -07:00
Aditya Bansal 9a100b1f55 auth.py: Make redirects to 'next' url work for dev environment.
This makes these redirects work for the local authentication
backend.
2018-03-21 13:35:44 -07:00
Harshit Bansal a49655e0d4 emoji: Migrate realm emoji to be addressed by `id` rather than `name`.
This commit migrates realm emoji to be addressed by their `id` rather
than their name. This fixes a long standing issue which was causing
an error on uploading an emoji with same name as a deactivated realm
emoji.

Fixes: #6977.
2018-03-20 22:24:44 -07:00
Tim Abbott 0e5c954393 users: Fix error string for disposable email addresses.
This should not use the term "realm", and doesn't need to name the
organization, either.
2018-03-16 16:59:01 -07:00
Tim Abbott 149f3efe5a realm: Clean up use of "Realm" in more user-facing strings.
We should be talking about organization names.
2018-03-16 16:59:01 -07:00
Vishnu Ks b13150a438 models: Do the check for disposable email in email_allowed_for_realm. 2018-03-15 14:35:24 -07:00
Vishnu Ks 951b88dd30 models: Make email_allowed_for_realm raise exception. 2018-03-15 14:35:24 -07:00
Steve Howell a4a8527ec5 search: Return info flags in payload.
We now return these:
    anchor
    found_anchor
    found_oldest
    found_newest

Fixes #8639
2018-03-15 12:36:06 -07:00
Steve Howell c6839e07c0 search: Fix num_after/num_before semantics precisely.
We now post-process query results so that you never get
more than `num_after` rows with id < `anchor`, and likewise
for `num_before`.
2018-03-15 12:36:06 -07:00
Steve Howell 6f3ebf6c4c Add post_process_limited_query(). 2018-03-15 12:36:06 -07:00
Steve Howell bd95b37d67 search: Make `num_after`/`num_after` more consistent.
We now consistently set our query limits so that we get at
least `num_after` rows such that id > anchor.  (Obviously, the
caveat is that if there aren't enough rows that fulfill the
query, we'll return the full set of rows, but that may be less
than `num_after`.)  Likewise for `num_before`.

Before this change, we would sometimes return one too few rows
for narrow queries.

Now, we're still a bit broken, but in a more consistent way.  If
we have a query that does not match the anchor row (which could
be true even for a non-narrow query), but which does match lots
of rows after the anchor, we'll return `num_after + 1` rows
on the right hand side, whether or not the query has narrow
parameters.

The off-by-one semantics here have probably been moot all along,
since our windows are approximate to begin with.  If we set
num_after to 100, its just a rough performance optimization to
begin with, so it doesn't matter whether we return 99 or 101 rows,
as long as we set the anchor correctly on the subsequent query.

We will make the results more rigorous in a follow up commit.
2018-03-15 12:36:06 -07:00
Steve Howell ec5299b1f2 search refactor: Rename query_result to rows.
The name `query_result` seems to suggest something more complex.
2018-03-15 12:36:06 -07:00
Aditya Bansal d4360e2287 uploads: Make django-sendfile to force downloading attachments.
We start to force downloads for the attachment files. We do this
for all files except images or pdf's. We would like images or pdf's
to open up in browser itself.

Tweaked by tabbott for comment clarity and correctness.
2018-03-14 11:22:10 -07:00
YJDave 93ee0aace7 stream settings: Allow realm admins to remove others from any stream.
This will allow realm admins to remove others from private stream to
which the realm administrator is not subscribed; this is important for
managing those streams, because previously nobody could remove users
from private streams that didn't have any realm administrators
subscribed.
2018-03-13 14:59:09 -07:00
YJDave 2031118545 stream settings: Allow realm admins to access all private stream subs.
This will allow realm admins to access subscribers of unsubscribed
private stream.  This is a preparatory commit for letting realm admins
remove those users.
2018-03-13 14:59:09 -07:00
YJDave 37f9d5c193 stream settings: Allow realm admins to update any stream name & description.
This will allow realm admins to update the names and descriptions of
private streams even if they are not subscribed, which fixes the buggy
behavior that previously nobody could(!).
2018-03-13 14:59:09 -07:00
Steve Howell 1ff653c986 search refactor: Extract `limit_query_to_range`.
This generic function isolates the before/after logic that really
is independent of Message and doesn't need to clutter up
`get_messages_backend`.  Also, introducing a new namespace
reduces some shadowing/mutation with variables like `query`.

It's a pure code move, with some very minor renaming (e.g.
inner_msg_id_col -> id_col).
2018-03-13 13:51:22 -07:00
Steve Howell 74a4a69333 search refactor: Move code that increments num_after/num_before.
We move the code that increments num_after/num_before to be closer
to where we do all the before/after logic.
2018-03-13 13:51:22 -07:00
Steve Howell 63c21707ee search refactor: Tighten up before/after logic.
If anchor is 0, there is no sense doing a before_query.

Likewise, if anchor is `LARGER_THAN_MAX_MESSAGE_ID`, there is
no sense doing an after_query.

We introduce variables called `need_before_query` and
`need_after_query` to enforce those conditions.

This also adds some comments explaining the fallthrough case
where neither query makes sense.
2018-03-13 13:51:22 -07:00
Steve Howell 129faa2c21 search refactor: Avoid `message_id >= 0` in queries.
We don't need things like `AND message_id >= 0` in our queries.

We can short circuit the syntax when `anchor` or
`first_message_visible` are zero.
2018-03-13 13:51:22 -07:00
Steve Howell e232a0bd57 search refactor: Streamline queries for caught-up users.
If use_first_unread_anchor is set and we don't have any unread
messages, then our anchor is effectively "positive infinity" and
we can streamline queries.

In the past we'd have clauses like `message_id <= 999999999999999`
in the query that were harmless but crufty.
2018-03-13 13:51:22 -07:00
Steve Howell e77350dc8e search refactor: Replace confusing `!= 0` idiom in search code.
We want to say `if num_after > 0` when we expect num_after to be
a positive integer.  We don't want any confusion that we will
execute the blocks for values of -7 or None.
2018-03-13 13:51:22 -07:00
neiljp (Neil Pilgrim) 9e1dbde82d mypy: Final small migrations to python3.5 annotations in many files. 2018-03-12 11:23:30 -07:00
neiljp (Neil Pilgrim) 6b9671c95c mypy: Migrate views/storage.py to python3.5 type annotations. 2018-03-12 11:23:30 -07:00
neiljp (Neil Pilgrim) 6bda59f3d7 mypy: Almost migrate views/streams.py to python3.5 annotations.
One remaining issue with FuncKwargPair use.
2018-03-12 11:23:30 -07:00
Vishnu Ks a44255eedb emails: Add backend for disallowing disposable email addresses. 2018-03-11 22:05:58 -07:00
Tim Abbott ef92fcbe2b topic history: Fix fetching topic history of public streams.
Apparently, we did essentially all the work to support showing full
topic history to newly subscribed users from a data flow perspective,
but didn't actually enable this feature by having the topic history
endpoint grant access to historical topics.  This fixes that gap.

I'm not altogether happy with how the code and tests read for this
feature; the code itself has more duplication than I'd like, and the
tests do too, but it works.
2018-03-11 20:59:20 -07:00
YJDave c94b21e9ac settings: Add setting to disable message content in missed message emails.
Fixes #6938.
2018-03-09 21:16:02 -08:00
Umair Khan 54e56481e6 auth: Retain email value if login fails.
Fixes #7795
2018-03-09 14:51:24 -08:00
Shubham Dhama 777b6de689 org settings: Add setting to prevent users from adding bots.
Fixes: #7908.
2018-03-09 13:21:55 -08:00
Robert Hönig 649e76e932 Display error when creating embedded bot with incorrect config data.
"incorrect" here means rejected by a bot's validate_config() method.
A common scenario for this is validating API keys before the bot is
created. If validate_config() fails, the bot will not be created.
2018-03-08 15:05:42 -08:00
Tim Abbott c3964dff6e i18n: Fix a last few strings mentioning realms. 2018-03-07 17:15:29 -08:00
Tim Abbott c47403b024 i18n: Fix message-send error messages using 'realm'.
We also do some small quality improvements to these strings, while
we're at it.
2018-03-07 17:15:29 -08:00
Tim Abbott e2a6541133 i18n: Fix use of 'realm administrator' in translated strings.
These are user-facing and thus should refer to being an "organization
administrator".
2018-03-07 17:15:29 -08:00
Tim Abbott 342d8cd4e0 i18n: Fix use of realm to refer to an organization. 2018-03-07 17:15:29 -08:00
Marco Burstein bdb86f1b5e emoji: Add support for translating emoticons.
Add `translate_emoticons` to `prop_types` and `expected_keys`.
Furthermore, create a emoji-translating Markdown inline pattern.

Also use a JavaScript version of `translate_emoticons` and then use
this function during Markdown previews and as a preprocessor. This
is only needed for previews, because usually emoticon translation
happens on the backend after sending.

Add tests for emoticon translation, a settings UI, and a /help/ page
as well.

Tweaked by tabbott to fix various test failurse as well as how this
handles whitespace, requiring emoticons to not have adjacent
characters.

Fixes #1768.
2018-03-04 15:37:24 -08:00
YJDave ce46cd914a settings: Allow admin to change email/name even if it is disabled in realm.
Allow realm admin users to change their email or name even,
changing name or email is disabled in realm.
2018-03-02 12:17:00 -08:00
Vishnu Ks f0d651f9fe invites: Only admins should be able to create multiuse invites. 2018-03-02 11:59:32 -08:00
Vishnu Ks 56e54262c3 api: Create api for creating multiuse invites. 2018-03-02 11:28:36 +00:00
rht 71ff8c370e django-2.0: Don't assign directly to Many-to-Many field.
The old pattern of setting the value and then using .save() here has
been deprecated.  set() also saves the record.
2018-03-01 08:49:35 -08:00
Robert Hönig 48b2e4eb66 backend: Add bot config data patching. 2018-03-01 08:25:43 -08:00
Aastha Gupta d124597f4b org-settings: Add setting to turn off welcome emails.
This adds an organization-level setting to provide an option to turn
off the welcome emails.

Fixes #8000.
2018-02-28 12:39:01 -08:00
Tim Abbott 710f5f7c97 auth: Add support for mobile_flow_otp for RemoteUserBackend.
Because we have a pretty good framework for the existing
mobile_flow_otp system, this requires very little new code.

Fixes #8291.
2018-02-24 08:14:17 -08:00
Tim Abbott 34efab9157 auth: Report to mobile apps the availability of RemoteUserBackend.
This is necessary for mobile apps to do the right thing when only
RemoteUserBackend is enabled, namely, directly redirect to the
third-party SSO auth site as soon as the user enters the server URL
(no need to display a login form, since it'll be useless).
2018-02-24 08:14:17 -08:00
Eeshan Garg df4c645587 integrations: Make sure settings.EXTERNAL_URI_SCHEME renders.
This got broken at some point when we moved around the context
processing logic for integrations/webhooks. Thankfully, the
context value for external_uri_scheme was only used in a couple of
our less popular integration docs. It should render perfectly now.
2018-02-24 06:21:02 -05:00
Umair Khan d22639717c auth: Redirect to an error page instead of 500.
Previously, we used to raise an exception if the direct dev login code
path was attempted when:

* we were running under production environment.
* dev. login was not enabled.

Now we redirect to an error page and give an explanatory message to the
user.

Fixes #8249.
2018-02-20 22:34:53 -08:00
Vishnu Ks d34dd4cd02 signup: Show subdomain availability during signup.
This uses an actual query to the backend to check if the subdomain is
available, using the same logic we would use to check when the
subdomain is in fact created.
2018-02-19 10:45:17 -08:00
neiljp (Neil Pilgrim) 3e98e77638 mypy: Amend typing of storage parameter to update_storage to be non-Optional. 2018-02-19 09:24:50 -08:00
neiljp (Neil Pilgrim) 2176f4789b mypy: Rewrite conditional to clarify message-sending defer_until code.
This actually fixes a small bug.
2018-02-19 09:24:28 -08:00
Shubham Dhama 9feae472f8 org settings: Add button to deactivate organization.
This adds button under "Organization profile" settings, which
deactivates the organization and sends an "event" to all the
active user and log out them.

Fixes: #8212.
2018-02-18 10:20:38 -08:00
Shubham Padia a4b686297a api: Return anchor in get_messages when use_first_unread_anchor=True.
This may be helpful for some API clients, since it avoids them needed
to do somewhat messy post-processing on the results (the data was
always available via scanning for the first unread message in the result).

Fixes #6244.
2018-02-16 10:06:20 -08:00
Aditya Bansal efe8545303 local-uploads: Start running authentication checks on file requests.
From here on we start to authenticate uploaded file request before
serving this files in production. This involves allowing NGINX to
pass on these file requests to Django for authentication and then
serve these files by making use on internal redirect requests having
x-accel-redirect field. The redirection on requests and loading
of x-accel-redirect param is handled by django-sendfile.

NOTE: This commit starts to authenticate these requests for Zulip
servers running platforms either Ubuntu Xenial (16.04) or above.

Fixes: #320 and #291 partially.
2018-02-16 05:06:37 +05:30
neiljp (Neil Pilgrim) b8f81801c1 mypy: Annotate existing_user_profile in accounts_register(). 2018-02-13 11:40:52 -08:00
neiljp (Neil Pilgrim) 16b939a964 REQ: Ensure only initial parameter is positional & adjust views/messages.py. 2018-02-13 11:40:52 -08:00
Shubham Dhama 03e4026c62 bots: Check bot owner for deactivated users and bots.
We should not allow deactivated users and bots as a
bot owner.
2018-02-13 09:24:42 -08:00
Shubham Dhama adfc905c3f bots: Handle exception on changing bot owner to invalid user.
It catches the `UserProfile.DoesNotExist` exception and
hence prevent internal server error.
Also remove option to select empty bot owner.
Fixes: #8334.
2018-02-13 09:23:13 -08:00
Greg Price ecbc72b857 push notifs: Add a diagnostic in API of whether push notifs enabled.
When the answer is False, this will allow the mobile app to show a
warning that push notifications will not work and the server admin
should set them up.

Based partly on Kunal's PR #7810.  Provides the necessary backend API
for zulip/zulip-mobile#1507.
2018-02-12 14:34:59 -08:00
Robert Hönig c0a0e6fd70 Set add_bot_backend() config_data param default to {}.
Previously, the default was None. This caused 500s when
adding a bot without adding a config_data argument.
2018-02-09 12:30:24 -08:00
Robert Hönig 146b833454 marketing: Update integration count.
This also automizes the integrations count update for
the /integrations page.
2018-02-06 08:50:41 -08:00
Roman Godov b875fe07eb settings: Added setting to turn on and off realm name in email subject.
Users having only account in one realm will not be distracted by realm
name in subject lines of every email.  Users who have multiple
accounts in realms can turn this setting on and receive a
corresponding realm name in email's subject.

Tweaked by tabbott to rebase and address a few small issues.

Fixes #5489.
2018-02-05 18:01:54 -08:00
Brock Whittaker f485abf522 page_params: Pass through settings.WARN_NO_EMAIL, as `warn_no_email`.
[greg: fixed test]
2018-02-05 16:00:33 -08:00
Greg Price 48791b731e create_realm: Let server admin skip confirming email.
This will let us defer configuring outbound email to the end of the
install procedure, so we can greatly simplify it by consolidating
several scripted steps.

The new flow could be simplified further by giving the user the full
form in the first place, rather than first a form for just their
email address and then a form with the other details.  We'll leave
that improvement for a separate change.
2018-02-05 12:59:12 -08:00
Greg Price 0dceeebd05 create_realm: Refactor to deal ASAP with key record, not string.
Now, there's just one spot at the beginning of the function where we
inspect the string key the user gave us; and after that point, we not
only have validated that string but in fact are working from our own
record that it pointed to, not the string itself.

This simplifies the code a bit, e.g. by not repeatedly searching the
database for the key (and hoping everything agrees so that we keep
getting the same row), and it will simplify adding logic to inspect
row attributes like `presume_email_valid`.
2018-02-05 12:59:12 -08:00
Greg Price f766ac6c10 create_realm: Reject invalid key even if no key required.
There's no use case for presenting a key that's invalid; if we haven't
given the user a valid key, we needn't send them to a URL that
presents an invalid one.  And the code is simpler to think about if
the only keys that can exist (after the validation at the top of the
function) are valid ones.

Apart from the case where creation_key is None, but invalid, and
settings.OPEN_REALM_CREATION is True so that we'd previously let the
invalid key slide, this is a pure refactor.
2018-02-05 12:59:12 -08:00
rht fae8d23039 zerver/views: Remove u prefix from strings. 2018-02-05 12:11:33 -08:00
rht 9a8d2244ca django-2.0: Shift to resolvers from urlresolvers.
The old name is deprecated.
2018-01-30 10:53:54 -08:00
Vishnu Ks 43a6439b3b upload: Enforce per-realm quota. 2018-01-29 16:06:11 -08:00
Greg Price 55cf54c087 upload: Remove old per-user quota feature.
We'll replace this primarily with per-realm quotas (plus the simple
per-file limit of settings.MAX_FILE_UPLOAD_SIZE, 25 MiB by default).

We do want per-user quotas too, but they'll need some more management
apparatus around them so an admin has a practical way to set them
differently for different users.  And the error handling in this
existing code is rather confused.  Just clear this feature out
entirely for now; then we'll build the per-realm version more cleanly,
and then we can later add back per-realm quotas modelled after that.

The migration to actually remove the field is in a subsequent commit.

Based in part on work by Vishnu Ks (hackerkid).
2018-01-29 16:06:11 -08:00
Greg Price 0f99610478 signup: Small refactor to isolate actually sending mail.
This is a little cleaner in that the try/except blocks for
SMTPException are a lot narrower; and it'll facilitate an upcoming
change to sometimes skip sending mail.
2018-01-29 10:27:11 -08:00
Greg Price 4bdecb7150 create_realm: Small refactor to check key just once. 2018-01-24 14:34:30 -08:00
YJDave 15dd94035c org settings: Handle NEW_USER_BOT not being set.
If NEW_USER_BOT is not configured in organization, then disable setting
for selecting stream for new user notification.
2018-01-23 15:23:25 -05:00
Brock Whittaker 849d63dd1c settings: Change error for "no data" to something more friendly. 2018-01-23 14:38:59 -05:00
Brock Whittaker ffe3a6c127 settings: Remove password confirmation in modal.
This removes the requirement to confirm your new password.
It isn't necessary and can be fixed easily with an email
reset if messed up.
2018-01-23 14:38:59 -05:00
Robert Hönig 9a15c4e3ff Add bot services to page_params.
This is the first step for allowing users
to edit a bot's service entries, name the
outgoing webhook configuration entries. The
chosen data structures allow for a future
with multiple services per bot; right now,
only one service per bot is supported.
2018-01-23 07:29:00 -05:00
Robert Hönig 06fb868482 Send an event to bot owners when a bot is created. 2018-01-23 07:29:00 -05:00
Eeshan Garg 00d3f19c0a integrations/email: Render settings.EMAIL_GATEWAY_EXAMPLE correctly. 2018-01-22 18:05:20 -05:00
Aditya Bansal f272ea9087 scheduledmessages: Start using/expecting delivery_type as a param. 2018-01-19 11:33:11 -05:00
Aditya Bansal c770bdaa3a reminder_bot: Add infra for adding reminder bot to every realm. 2018-01-19 11:33:11 -05:00
Aditya Bansal f46d098558 schedulemessages: Add handle_deferred_message() to handle requests.
This is responsible for:
1.) Handling all the incoming requests at the
messages endpoint which have defer param set. This is similar to
send_message_backend apart from the fact that instead of really
sending a message it schedules one to be sent later on.
2.) Does some preliminary checks such as validating timestamp for
scheduling a message, prevent scheduling a message in past, ensure
correct format of message to be scheduled.
3.) Extracts time of scheduled delivery from message.
4.) Add tests for the newly introduced function.
5.) timezone: Add get_timezone() to obtain tz object from string.
This helps in obtaining a timezone (tz) object from a timezone
specified as a string. This string needs to be a pytz lib defined
timezone string which we use to specify local timezones of the
users.
2018-01-19 11:33:11 -05:00
Robert Hönig d1d8365a6b embedded bots: Add config data UI.
This adds UI fields in the bot settings for specifying
configuration values like API keys for a bot. The names
and placeholder values for each bot's config fields are
fetched from the bot's <bot>.conf template file in the
zulip_bots package. This also adds giphy and followup
as embedded bots.
2018-01-09 11:50:54 -05:00
Alena Volkova 9d1063d362 urls: Move the json/fetch_api_key endpoint to be an API-style route. 2018-01-08 13:15:52 -05:00
Eeshan Garg 479bc41ca5 api docs: Remove old and outdated endpoint docs. 2018-01-04 10:17:29 -05:00
Alena Volkova 45f0c76c44 settings: Limit the creation of generic bots.
This commit adds a setting to limit creation of generic bots
to admins for realms that want that restriction.  (Generic
bots, apart from being considered spammy on some realms,
have less locked down permissions than webhook bots).

Fixes #7066.
2018-01-02 18:12:22 -05:00
Aastha Gupta daf86eb664 settings: Add "text" option to emoji_set model.
We no longer have a special UI setting and model
field ("emoji_alt_code") for saying users want text-only
emojis.  We now instead make "text" be a fifth choice
for "emojiset".

Fixes #7406
2018-01-02 14:55:01 -05:00
Vishnu Ks 277d463d23 messages: Add option to limit users message history. 2018-01-02 13:45:01 -05:00
YJDave fa44d2ea69 settings: Remove autoscroll_forever setting.
Fixes #6845
2018-01-02 10:35:49 -05:00
rht 1324d9f000 mypy: Use Python 3 type syntax in zerver/views. 2017-12-30 07:34:05 -05:00
YJDave 29905ad5d6 create stream: Before sending notificaion, check if NOTIFICATION_BOT exists.
There might be case that NOTIFICATION_BOT is none, so before sending stream
announce notification, check first if settings.NOTIFICATION_BOT is not none.
2017-12-29 08:59:55 -05:00
Kiy4h 6fa51be822 mypy: Use Python 3 type syntax in zerver/views/push_notifications.py. 2017-12-28 07:26:44 -05:00
Kiy4h 64413e3b91 mypy: Use Python 3 type syntax in zerver/views/email_mirror.py. 2017-12-28 07:26:44 -05:00
Kiy4h 238cd2b132 mypy: Use Python 3 type syntax in zerver/views/alert_words.py. 2017-12-28 07:26:44 -05:00
Xavier Cooney 36b58ff6fc mypy: Use Python 3 type syntax in 'views/users.py'. 2017-12-26 08:34:38 -05:00
nyan-salmon 0fd0332a2c mypy: Use Python 3 type syntax in zerver/views/custom_profile_fields.py. 2017-12-26 08:34:06 -05:00
Viraat Chandra 0494902e6d mypy: Use Python 3 syntax for typing in `zerver/views/auth.py`. 2017-12-26 08:31:43 -05:00
Xavier Cooney 61704dc20b mypy: Use Python 3 type syntax in views/pointer.py. 2017-12-26 08:29:30 -05:00
Callum Fraser b0c5eb5482 mypy: Use Python 3 type syntax in zerver/views/typing.py. 2017-12-18 11:32:52 -05:00
Greg Price 17a6632c43 zerver/views: Revert to Python 2 typing syntax for now (storage, streams).
This reverts commit 620b2cd6e.

Contributors setting up a new development environment were getting
errors like this:
```
++ dirname tools/do-destroy-rebuild-database
[...]
+ ./manage.py purge_queue --all
Traceback (most recent call last):
[...]
  File "/home/zulipdev/zulip/zproject/legacy_urls.py", line 3, in <module>
    import zerver.views.streams
  File "/home/zulipdev/zulip/zerver/views/streams.py", line 187, in <module>
    method_kwarg_pairs: List[FuncKwargPair]) -> HttpResponse:
  File "/usr/lib/python3.5/typing.py", line 1025, in __getitem__
    tvars = _type_vars(params)
[...]
  File "/usr/lib/python3.5/typing.py", line 277, in _get_type_vars
    for t in types:
TypeError: 'ellipsis' object is not iterable
```

The issue appears to be that we're using the `typing` module from the
3.5 stdlib, rather than the `typing=3.6.2` in our requirements files,
and that doesn't understand the `Callable[..., HttpResponse]` that
appears in the definition of `FuncKwargPair`.

Revert for now to get provision working again; at least one person
reports that reverting this sufficed.  We'll need to do more testing
before putting this change back in.
2017-12-13 10:20:11 -08:00
rht 620b2cd6e2 zerver/views: Use Python 3 syntax for typing (storage, streams). 2017-12-12 17:40:50 -08:00
Rhea Parekh 8ed7f73199 IRC mirror: No longer require IRC stream names to start with #. 2017-12-12 10:04:31 -06:00
Xavier Cooney 7887857c81 mypy: Use Python 3 type syntax in user_settings.py 2017-12-11 20:33:00 -08:00
picapi_ 2adeaab6ab mypy: Use Python 3 type syntax in zerver/views/invite.py. 2017-12-11 20:30:19 -08:00
Shreyansh Dwivedi 47fcb27e39 invitations: Remove custom_body.
Fixes #7672
2017-12-11 19:23:54 -08:00
Callum Fraser 00178d114b mypy: Use Python 3 type syntax in zerver/views/streams.py.
[greg: small formatting tweaks]
2017-12-11 18:43:24 -08:00
Rishi Gupta d9ce33dcf0 registration: Clarify realm_creation conditionals in accounts_register.
No change in behavior.
2017-12-11 18:36:14 -08:00
Rishi Gupta e545a681fc registration: Use standard pattern for returning link_does_not_exist.
Eventually this check for the realm will be done in get_object_from_key
itself. Rewriting this to fit the pattern in get_object_from_key.

No change to behavior.
2017-12-11 18:36:14 -08:00
Rishi Gupta dbe3576706 registration: Enforce realm is None only if realm_creation.
Commit d4ee3023 and its parent have the history behind this code.

Since d4ee3023^, all new PreregistrationUser objects, except those for
realm creation, have a non-None `realm`.  Since d4ee3023, any legacy
PreregistrationUsers, with a `realm` of None despite not being for
realm creation, are treated as expired.  Now, we ignore them
completely, and remove any that exist from the database.

The user-visible effect is to change the error message for
registration (or invitation) links created before d4ee3023^ to be
"link does not exist", rather than "link expired".

This change will at most affect users upgrading straight from 1.7 or
earlier to 1.8 (rather than from 1.7.1), but I think that's not much
of a concern (such installations are probably long-running
installations, without many live registration or invitation links).

[greg: tweaked commit message]
2017-12-11 18:36:14 -08:00
Arseny Chekanov e5c05f128a mypy: Use Python 3 type syntax in zerver/views/realm_filters.py. 2017-12-09 16:50:12 -08:00
Xavier Cooney aa37b50d1e mypy: User Python 3 syntax for typing in realm.py. 2017-12-08 18:34:26 -08:00
Xavier Cooney 63e793a8d1 mypy: Use Python 3 syntax for typing in views/tutorial.py. 2017-12-08 18:34:26 -08:00
Rishi Gupta 100167fcf8 invitations: Change error messages for resend/revoke views.
We don't have a concept of invitation ID on the frontend (or backend).
2017-12-06 20:35:50 -08:00
Rishi Gupta cb5eb47470 invitations: Move resend/revoke error handling out of actions.py. 2017-12-06 20:35:50 -08:00
picapi_ e2c14724d9 mypy: Use Python 3 type syntax in zerver/views/messages.py. 2017-12-03 13:48:18 -08:00
Alena Volkova 3be8e95d13 settings: Send bot types from the backend instead of hardcoding them. 2017-11-30 15:28:53 -08:00
Rishi Gupta 7c9694077e confirmation: Move check_prereg_key_and_redirect to registration.py. 2017-11-29 22:18:05 -08:00
Rishi Gupta 331a9bee6b confirmation: Add Confirmation.REALM_CREATION.
Does not change any behavior. Groundwork for adding a realm field to
Confirmation.
2017-11-29 22:03:09 -08:00
Sarah ba70b542d3 notification settings: Add stream email notifications setting.
Adds a field to UserProfile for enabling emails about stream
messages and a field to Subscriptions about email
notification preferences.
2017-11-28 17:51:18 -08:00
Greg Price 0ec2a9d259 auth: Try switching to register even if user exists on another realm.
For example, this means that if a user already has an account on one
realm and they try to make an account on another by hitting "Sign in
with Google" (rather than following the little "Register" link to a
"Sign up with Google" button instead), they'll get to make an account
instead of getting an error.

Until very recently, if the user existed on another realm, any attempt
to register with that email address had to fail in the end, so this
logic gave the user a useful error message early.  We introduced it in
c23aaa178 "GitHub: Show error on login page for wrong subdomain"
back in 2016-10 for that purpose.  No longer!  We now support reusing
an email on multiple realms, so we let the user proceed instead.

This function's interface is kind of confusing, but I believe when its
callers use it properly, `invalid_subdomain` should only ever be true
when `user_profile` is None -- in which case the revised
`invalid_subdomain` condition in this commit can never actually fire,
and the `invalid_subdomain` parameter no longer has any effect.  (At
least some unit tests call this function improperly in that respect.)
I've kept this commit to a minimal change, but it would be a good
followup to go through the call sites, verify that, eliminate the use
of `invalid_subdomain`, then remove it from the function entirely.
2017-11-28 16:38:41 -08:00
Tim Abbott 586e18b237 auth: Allow accounts with the same email in different realms.
[Modified by greg to (1) keep `USERNAME_FIELD = 'email'`,
(2) silence the corresponding system check, and (3) ban
reusing a system bot's email address, just like we do in
realm creation.]
2017-11-28 16:23:10 -08:00
Tim Abbott e6f460f511 auth: Replace user_email_is_unique validator.
As we migrate to allow reuse of the same email with multiple realms,
we need to replace the old "no email reuse" validators.  Because
stealing the email for a system bot would be problematic, we still ban
doing so.

This commit only affects the realm creation logic, not registering an
account in an existing realm.
2017-11-28 16:23:10 -08:00
Greg Price 5f38b374b6 register: Activate existing mirror dummy only if on same realm.
The one thing this bit of logic is used for is to decide whether
there's an existing user which is a mirror dummy that we should
activate.  This change causes us to ignore such an existing user if
it's on some other realm, and go straight into `do_create_user`.
2017-11-28 16:23:10 -08:00
Tim Abbott 4f5a5a8547 auth: Extract add_dev_login_context.
This simple refactor cleans up the code for the dev_auth_enabled code
path to be a bit less deeply in the login_page() logic.
2017-11-28 15:17:33 -08:00
derAnfaenger c8a5ae753c embedded bots: Consistently use 'storage' instead of 'state.' 2017-11-27 21:05:34 -08:00
rht e538f4dd44 zerver/views: Use Python 3 syntax for typing.
Edited by tabbott to remove state.py and streams.py, because of
problems with the original PR's changes, and wrap some long lines.
2017-11-27 17:10:39 -08:00
Tim Abbott bd8196c3a5 lint: Fix too-long line in auth.py. 2017-11-27 15:28:46 -08:00
Vishnu Ks d4ee3023b2 registration: Require an explicit realm on PreregistrationUser.
This completes the last commit's work to fix CVE-2017-0910, applying
to any invite links already created before the fix was deployed.  With
this change, all new-user registrations must match an explicit realm
in the PreregistrationUser row, except when creating a new realm.

[greg: rewrote commit message]
2017-11-27 14:59:34 -08:00
Vishnu Ks 985768b2fd registration: Check realm against PreregistrationUser realm.
We would allow a user with a valid invitation for one realm to use it
on a different realm instead.  On a server with multiple realms, an
authorized user of one realm could use this (by sending invites to
other email addresses they control) to create accounts on other
realms. (CVE-2017-0910)

With this commit, when sending an invitation, we record the inviting
user's realm on the PreregistrationUser row; and when registering a
user, we check that the PregistrationUser realm matches the realm the
user is trying to register on.  This resolves CVE-2017-0910 for
newly-sent invitations; the next commit completes the fix.

[greg: rewrote commit message]
2017-11-27 14:58:26 -08:00
Tim Abbott 2210f627a5 signup: Switch active mirror-dummy users to an AssertionError.
Previously, this was a ValidationError, but that doesn't really make
sense, since this condition reflects an actual bug in the code.

Because this happened to be our only test coverage the ValidationError
catch on line 84 of registration.py, we add nocoverage there for now.
2017-11-26 16:13:45 -08:00
Vishnu Ks 9f225360dc register: Make /register redirect to find accounts in root domain without realm.
If there is no realm in the root domain don't show
registration form. Instead just redirect to find
accounts page.
2017-11-26 15:32:17 -08:00
Robert Hönig 0e0a8a2b14 queue processor tests: Call consume by default.
This significantly improves the API for queue_json_publish to not be
overly focused on what the behavior of this function should be in our
unit tests.
2017-11-26 11:45:34 -08:00
rht e169bb0954 zerver: Remove simplejson dependency.
Modified by tabbott to put the third-party code in a new file.

Fixes #6970.
2017-11-21 21:51:48 -08:00
derAnfaenger e526d0c144 embedded bots: Add views to access state. 2017-11-21 21:10:39 -08:00
Eeshan Garg 73d0f1db81 lib/integrations: Remove Integration.doc_context.
An Integration object doesn't need access to the context dict used
to render its doc.md, since the context dict is just passed directly to
render_markdown_path.
2017-11-21 20:39:30 -08:00
Eeshan Garg 600ee90a38 integrations: Add add_context_for_single_integration function.
Previously, when rendering a single integration, we tacked on the
following information to the context dict that was redundant:

* An OrderedDict containing all of the Integration objects for
  all integrations.
* An OrderedDict containing all of the integration categories.

The context dict for rendering a particular integration doc would
contain 4 OrderedDicts, 2 for categories, 2 for Integration objects
because of how many times add_integrations_context had been called.

This was very wasteful, since an Integration object doesn't need
to access any other Integration object (or itself for that matter)
to render its documentation. This commit adds a function that
allows us to only pass in the context values that are necessary.
2017-11-21 20:39:30 -08:00
Tim Abbott 3bfb19b5f3 Convert EmailAuthBackend and LDAPAuthBackend to accept a realm. 2017-11-21 18:23:50 -08:00
Tim Abbott 1b95b098dd auth: Clarify comments explaining the GoogleMobileOauth2Backend. 2017-11-21 18:23:50 -08:00
Tim Abbott a7d51127fb auth: Convert GoogleMobileOAuth2Backend to accept a realm object. 2017-11-21 18:23:49 -08:00
Tim Abbott 07bc31f818 auth: Convert DevAuthBackend to accept a realm object. 2017-11-21 18:23:49 -08:00
Tim Abbott 4968631d1b auth: Convert DevAuthBackend to use a unique argument pattern.
This helps ensure that we won't accidentally activate this backend on
other code paths.
2017-11-21 18:23:49 -08:00
Tim Abbott 387c9109ec auth: Convert RemoteUserBackend to accept a realm object. 2017-11-21 18:23:49 -08:00
Tim Abbott f17974ab32 DummyAuthBackend: Require being passed a realm object.
We should now always know the realm in our auth code paths.
2017-11-21 18:22:37 -08:00
Vishnu Ks 34689370cd settings: Create UI and backend for setting signups stream. 2017-11-21 17:39:51 -08:00
Rishi Gupta 42652713c8 onboarding: Refactor away setup_initial_private_stream.
It is a confusing name, now that the stream creation part has been taken
away.
2017-11-21 17:39:50 -08:00
Tim Abbott 29408d24ff emoji: Rework reactions validation to have a single function.
This feels more readable to me, and also identified a couple cases
where we were missing test coverage.
2017-11-20 15:27:21 -08:00
Tim Abbott 2d4c7b771a reactions: Move legacy endpoint code further down in file.
This helps ensure that new contributors working on the reactions code
read the stuff we use now first.
2017-11-20 15:27:21 -08:00
Tim Abbott 80a2a36dd1 reactions: Rename legacy reactions for greater clarity. 2017-11-20 15:27:21 -08:00
Tim Abbott 25ddba99f6 lint: Ban general use of user_profile.save().
This often can cause minor caching problems.

Obviously, it'd be better if we had access to the AST and thus could
do this rule for UserProfile objects in general.
2017-11-20 10:57:08 -08:00
Vishnu Ks 432c9d3449 email_log: Reset the value of email after email change process.
This commit actually needs only user_profile.save() line
but it's more readable with both the lines.
2017-11-20 10:45:16 -08:00
Tim Abbott c8edbae21c password reset: Fix error message for invalid realm.
This is a lot cleaner than the previous model.

Basically rewritten by Vishnu Ks to actually work :).
2017-11-20 10:34:55 -08:00
Eeshan Garg 9138d3b8d7 integrations: Move integration-specific context to integration_doc.
Instead of populating the context dict with integration-specific
information in render_markdown_path, we now do that in
zerver.views.integrations.integration_doc instead.

Fixes #7401.

Tweaked by tabbott to use cast to handle the typing issues here.
2017-11-19 20:48:46 -08:00
Tim Abbott 18369cb5ad JWT: Fetch the subdomain earlier in the auth code path. 2017-11-17 17:33:28 -08:00
Harshit Bansal d9c2f613e3 api: Add new endpoint for reactions.
This endpoint will allow us to add/delete emoji reactions whose emoji
got renamed during various emoji infra changes. This was also a
required change for realm emoji migration.

This commit was tweaked significantly by tabbott for greater clarity
(with no changes to the actual logic).
2017-11-16 20:52:15 -08:00
Sampriti Panda a696141a25 search: Add more extensive logging for Unicode errors in highlight_string.
Tweaked by tabbott to provide finer-grained logging.
2017-11-16 15:08:16 -08:00
Tim Abbott 054952a44a docs: Update links from codebase to point to ReadTheDocs. 2017-11-16 10:53:49 -08:00
Umair Khan cc76f7deac user-groups: Create check_delete_user_group. 2017-11-15 22:35:43 -08:00
Umair Khan 66179ea295 bulk_add_members_to_user_group: Pass user group.
Previously we passed user group id. If we pass user group, send event is
easier.
2017-11-15 22:35:23 -08:00
Tim Abbott c7a975e4df users: Move check_change_full_name to actions.py.
This avoids an import loop in the next commit, and better matches our
usual code structure.
2017-11-15 17:39:09 -08:00
Tim Abbott 03b4a0240c user_settings: Fix mypy annotations.
While we're at it, convert this to use the Python 3 style, which isn't
prone to this sort of bug.
2017-11-15 17:38:19 -08:00
Brock Whittaker f9f0f356be settings: Add backend storage and interface for night mode.
This allows the night mode setting to be stored in the backend.
2017-11-15 16:36:06 -08:00
Tim Abbott 85844107fd integrations: Remove code for legacy Hubot lozenges.
We no longer display the only model anyway.
2017-11-15 14:04:19 -08:00
rht bafba95eb7 Text-wrap the remaining long lines exceeding 110. 2017-11-15 10:58:03 -08:00
Tim Abbott 5aa6751e7a lint: Fix long lines in newly added features. 2017-11-15 10:58:03 -08:00
Vishnu Ks 36f29764cb emails: Add option to forward mails send in dev env to external email.
Fixes #7085.
2017-11-14 15:27:29 -08:00
Vishnu Ks d2555e8769 api: Add support to change name of default stream groups. 2017-11-14 14:41:42 -08:00
Vishnu Ks 2af249dd5d api: Add description attribute to default stream group. 2017-11-14 14:41:42 -08:00
Vishnu Ks f45ba7de93 api: Pass group id instead of name to default stream group api. 2017-11-14 14:41:42 -08:00
rht a93b7c40a7 zerver: Text-wrap long lines exceeding 110. 2017-11-10 16:29:32 -08:00
Vishnu Ks f44b60a150 Implement API for default stream groups. 2017-11-10 16:05:36 -08:00
Umair Khan b1603d289c user-groups: Add update members API endpoint.
Significantly modified by tabbott to fix some bugs.
2017-11-09 17:35:37 -08:00
Umair Khan a1b894a786 user-groups: Add delete API endpoint.
Significantly modified by tabbott to fix some bugs.
2017-11-09 17:35:30 -08:00
Umair Khan eeaf0f1742 user-groups: Add edit API endpoint.
Significantly modified by tabbott to fix some bugs.
2017-11-09 17:35:22 -08:00
Umair Khan 1bbe84af49 user-groups: Add create API endpoint.
Significantly modified by tabbott for better security structure.
2017-11-09 17:26:14 -08:00
Steve Howell c38996ddde Remove unused force_text import. 2017-11-09 10:32:14 -08:00
Steve Howell 786e90d774 Removed unused imports of force_bytes. 2017-11-09 10:32:14 -08:00
YJDave de67bf81fd org settings: Allow users to delete their message.
Fixes #7049.
2017-11-09 10:20:34 -08:00
rht 88a828dd0c Remove six.moves.urllib usage. 2017-11-09 10:00:00 -08:00
derAnfaenger 19bc55aa45 Fix various typos.
The typos and their corrections were found with the
aid of https://github.com/lucasdemarchi/codespell.
2017-11-09 16:26:38 +01:00
Tim Abbott a8c735e1d2 /api/: Fix some headings. 2017-11-08 13:08:23 -08:00
Eeshan Garg 7c113f0447 api-docs: Render api_url in /api docs.
In templates/zerver/api/main.html, since the current context isn't
passed to render_markdown_path when rendering an article,
render_markdown_path doesn't have the context to render values such
as api_url. This commit makes sure that it does by passing a dict
called api_uri_context to render_markdown_path when rendering an
article.
2017-11-08 01:29:49 -03:30
Steve Howell 4108797218 Eliminate the parse_usermessage_flags interface.
This commit puts the guts of parse_usermessage_flags into
UserMessage.flags_list_for_flags, since it was slightly faster
than the old implementation and produced the same results.
(Both algorithms were super fast, actually.)

And then all callers use the model method now.
2017-11-07 17:48:27 -08:00
Steve Howell ae0b27a7ed Extract messages_for_ids. 2017-11-07 17:48:27 -08:00
Steve Howell 70f2cd2860 De-duplicate is_search logic in get_messages_backend().
The logic to set search_fields was essentially the same for both
sides of the include_history conditional.

Now we have just one code block that sets search_fields, and we
can quickly short-circuit the loop when is_search is False.
2017-11-07 17:48:27 -08:00
Steve Howell c62588c359 minor: Use dict comprehension in get_messages_backend(). 2017-11-07 17:48:27 -08:00
Tim Abbott a8a159c5d2 mypy: Fix variable reuse in streams.py. 2017-11-07 17:48:27 -08:00
rht 296835351d zerver/views: Text-wrap long lines exceeding 110.
Tweaked by tabbott to fix various minor issues.
2017-11-07 17:24:09 -08:00
Rishi Gupta 15b3a8e4ff email change: Use confirmation object user to determine changeability.
Seems like the more logical check. Also, the previous code makes it feel
like there is a potential vulnerability where one could get an email change
object in a realm where email changes are disabled, and then open that link
while logged in to a different realm.

While we're at it, remove the unnecessary check that the user is
logged in when clicking the confirmation link; that creates
unnecessary trouble for users who use multiple browsers.
2017-11-07 16:26:27 -08:00
Rishi Gupta 4fc8aa2b53 email change: Refactor confirmation path for readability.
Removes an assert, which at this point is there just for readability, since
the second argument to
get_object_from_key(confirmation_key, Confirmation.EMAIL_CHANGE)
ensures that the returned object is of the correct type.
2017-11-07 16:02:49 -08:00
Rishi Gupta 6fbb7e8046 email change: Remove confirmation_key.lower() from confirm_email_change.
Not sure why it is here, and it adds confusion.
2017-11-07 16:02:49 -08:00
Rishi Gupta 1823d46236 unsubscribe: Use a custom confirmation error page.
Reverts a bit of 6e02ce8.
2017-11-07 16:02:49 -08:00
rht 14671fc7b9 refactor: Remove six.moves.zip_longest import. 2017-11-07 10:51:44 -08:00
rht 80a8d4f9f3 refactor: Remove six.moves.map import. 2017-11-07 10:46:42 -08:00
rht 549a26860f refactor: Remove six.moves.range import. 2017-11-07 10:46:42 -08:00
rht ec5120e807 refactor: Remove six.moves.zip import. 2017-11-07 10:46:42 -08:00
rht 19bd335cbb Change urllib import to be Python 3-specific. 2017-11-07 10:46:42 -08:00
Steve Howell 8450f0ec69 Register client_gravar=True for the webapp. 2017-11-07 10:42:18 -08:00
Steve Howell 2bbfda041a Support client_gravatar field for event registration.
This commit allows clients to register client_gravatar=True, and
then we recognize that flag for message events.  If the flag is
True, we will not calculate gravatar URLs and let the clients do
it themselves.  (Clients can calculate gravatar URLs based on
emails with just a little bit of code.)
2017-11-07 10:42:17 -08:00
rht 8990b1046d zerver: Remove inheritance from object. 2017-11-06 08:53:48 -08:00
neiljp (Neil Pilgrim) 42f5eea61f mypy: Improve typing of request.pyi (REQ).
This expands request.pyi to type-check the arguments passed into REQ.

Tweaked by tabbott to fix the report.py annotations.
2017-11-04 20:26:03 -07:00
neiljp (Neil Pilgrim) 186181c2a0 mypy: Specify type parameters in convert_term in narrow_parameter. 2017-11-04 19:47:45 -07:00
neiljp (Neil Pilgrim) da97de3370 mypy: Use Dict[Any,Any] for update_realm parameter. 2017-11-04 19:47:45 -07:00
Greg Price 68b0a419ec decorator: Cut a bunch of dead imports of two view decorators.
Saw these when grepping for these two decorators; they're actually
more numerous than the surviving use sites are.  Cut out the noise.
2017-11-04 19:27:00 -07:00
Greg Price 6b9d294873 auth: Set user_activity `query` nicely for several auth views.
This gets used when we call `process_client`, which we generally do at
some kind of login; and in particular, we do in the shared auth
codepath `login_or_register_remote_user`.  Add a decorator to make it
easy, and use it on the various views that wind up there.

In particular, this ensures that the `query` is some reasonable
constant corresponding to the view, as intended.  When not set, we
fall back in `update_user_activity` on the URL path, but in particular
for `log_into_subdomain` that can now contain a bunch of
request-specific data, which makes it (a) not aggregate properly, and
(b) not even fit in the `CHARACTER VARYING(50)` database field we've
allotted it.
2017-11-04 19:27:00 -07:00
Rishi Gupta 5d04e29ac2 confirmation: Add comment for registration.confirmation_key.
I remember being really confused by this function in the past, and I finally
figured it out. It should be removed, and the dev_url added by
00-realm-creation should call a function that just gets the confirmation_key
from outbox like all of the backend tests, but until then this comment
should help.
2017-11-03 15:33:16 -07:00
Rishi Gupta 6e02ce8344 unsubscribe: Use get_object_from_key for confirmation.
This is the only one of the confirmation pathways that wasn't using this
idiom, I think.
2017-11-03 15:33:16 -07:00
Rishi Gupta fdbe36644e confirmation: Add confirmation_type to get_object_from_key.
This change:

* Prevents weird potential attacks like taking a valid confirmation link
  (say an unsubscribe link), and putting it into the URL of a multiuse
  invite link. I don't know of any such attacks one could do right now, but
  reasoning about it is complicated.

* Makes the code easier to read, and in the case of confirmation/views.py,
  exposes something that needed refactoring anyway (USER_REGISTRATION and
  INVITATION should have different endpoints, and both of those endpoints
  should be in zerver/views/registration, not this file).
2017-11-03 15:33:16 -07:00
Harshit Bansal 44bff43c86 actions.py: Rename `do_remove_reaction()` to `do_remove_reaction_legacy()`. 2017-10-31 15:42:35 -07:00
Harshit Bansal 233595b2f7 actions.py: Rename `do_add_reaction()` to `do_add_reaction_legacy()`. 2017-10-31 15:42:35 -07:00
Steve Howell 9689168bf6 Simplify highlight_string logic.
This change avoids some complicated byte arithmetic by
extracting some local variables and being more explicit
about using bytes vs. using strings.
2017-10-31 12:45:58 -07:00
Brock Whittaker 425738fa24 landing-page: Extract /team/ from /about/.
This extracts and creates a /team/ page using the existing
'team.html' template and applying the why-page styling to it.
2017-10-31 11:37:42 -07:00
Sampriti Panda ee61e1be2c search: Fix misaligned highlights due to unicode characters.
tsearch_extras returns search offsets in bytes but our highlight
function treated them as character offsets.  Added a check to subtract
extra bytes if the tsearch search backend is being used.

Fixes #4084.
Fixes #7021.
2017-10-31 09:06:46 -07:00
Greg Price 22e5110789 subdomains: Dedupe out the last copy of Realm.host_for_subdomain.
This makes the affected code somewhat easier to read, and paves the
way for changing this logic to add support for alias domains.
2017-10-30 18:29:29 -07:00
Greg Price eb55a3a1ba template context: Give better names to the URLs for the API.
The "subdomain" label is redundant, to the extent it's even
accurate -- this is really just the URL we want to display,
which may or may not involve a subdomain.  Similarly "external".

The former `external_api_path_subdomain` was never a path -- it's a
host, followed by a path, which together form a scheme-relative URL.
I'm not quite convinced that value is actually the right thing in
2 of the 3 places we use it, but fixing that can start by giving an
accurate name to the thing we have.
2017-10-30 18:29:29 -07:00
Greg Price 7366a4ca7a settings: Eliminate the disused EXTERNAL_API_PATH.
This setting isn't documented at all, and I believe nobody has used it
since the end of api.zulip.com in 2016.  So we get to complete the
cleanup of this logic.
2017-10-30 18:29:29 -07:00
rht c4fcff7178 refactor: Replace super(.*self) with Python 3-specific super().
We change all the instances except for the `test_helpers.py`
TimeTrackingCursor monkey-patching, which actually needs to specify
the base class.
2017-10-30 14:30:25 -07:00
Brock Whittaker 6933d51c0f views/integrations: Change non-generic HelpView to MarkdownDirectoryView.
The HelpView class will render a directory as markdown with an index HTML
page. This however can also be used for other generics and applied to
the API pages as well, so change the class to a generic class and
specify the path templates and names.

Tweaked by tabbott and Eeshan Garg.
2017-10-29 18:15:36 -07:00
Steve Howell 8b012c6210 Extract get_personal_recipient(). 2017-10-28 17:57:39 -07:00
Steve Howell a28841e8aa Extract get_stream_recipient().
Do you call get_recipient(Recipient.STREAM, stream_id) or
get_recipient(stream_id, Recipient.STREAM)?  I could never
remember, and it was not very type safe, since both parameters
are integers.
2017-10-28 17:57:39 -07:00
Steve Howell d132c30c24 Remove `active` parameter from `do_create_user()`.
Almost all callers to do_create_user were trying to
create active users, except for one test.  The
active=False codepath was kind of broken (things
like sending welcome messages had sort of undefined
behavior there), so instead of trying to maintain it,
we just update the one test (`test_people`) to flip the
`is_active` flag manually.

Fixes #7197
2017-10-28 10:32:01 -07:00
Tim Abbott 5b5dffebe7 get_messages_backend: Tweak how search fields are added.
This is a no-op, but makes mypy happier, since previously it wanted to
know the type of the {} we were passing to `.update()`.
2017-10-28 09:58:51 -07:00
Tim Abbott 80a9253f89 attachments: Remove unnecessary use of REQ.
It didn't do anything, because that third argument is actually a
positional argument coming from the URL parsing logic.
2017-10-27 15:18:44 -07:00
Tim Abbott 5467296f60 decorators: Use human_users_only more aggressively. 2017-10-27 15:16:13 -07:00
Tim Abbott 1cd017288d views: Fix imports of REQ/has_request_variables from the wrong place.
These were never in zerver/decorator.py, and so it makes sense to
import them zerver/lib/request.py, mostly for ease of finding things.
2017-10-27 15:07:31 -07:00
Greg Price d9cb606804 oauth: Find a better name for redirect_to_main_site.
This name was way too broad for the rather specific logic
in the actual function.
2017-10-27 14:42:24 -07:00
Greg Price ad551427ed auth: Factor out some uses of EXTERNAL_HOST with ROOT_DOMAIN_URI.
Apart from being less verbose, this makes it more manifest (on e.g. grep)
that we aren't using EXTERNAL_HOST here to construct subdomains.
2017-10-27 14:42:24 -07:00
Greg Price 318682fd52 auth: Use URL rather than cookie to pass signed data cross-domain.
The cookie mechanism only works when passing the login token to a
subdomain.  URLs work across domains, which is why they're the
standard transport for SSO on the web.  Switch to URLs.

Tweaked by tabbott to add a test for an expired token.
2017-10-27 14:42:04 -07:00
Greg Price fad3d56810 views: Move some login code from `registration` to `auth`.
Most of these have more to do with authentication in general than with
registering a new account.  `create_preregistration_user` could go
either way; we move it to `auth` so we can make the imports go only in
one direction.
2017-10-27 14:28:38 -07:00
Henrik Pettersson 09cd47c6ad Add UI for viewing and cancelling open Zulip invitations.
Lets administrators view a list of open(unconfirmed) invitations and
resend or revoke a chosen invitation.

There are a few changes that we can expect for the future:

  * It is currently possible to invite an email that you have already
    invited, it might make sense to change this behavior.

  * Resend currently sends an invite reminder instead of resending the
    original invite, this is because 'custom_body' was not stored when
    the first invite was sent.

Tweaked in various minor ways, primarily in the backend, by tabbott,
mostly for style consistency with the rest of the codebase.

Fixes: #1180.
2017-10-27 13:07:43 -07:00
Tim Abbott 237e3cdca1 registration: Clarify comment on do_activate_user code path. 2017-10-27 11:22:19 -07:00
Vishnu Ks 8c68a167fe backend: Allow Administrators to invite new users as admins.
Tweaked by tabbott to have the field before the invitation is
completed be called invite_as_admins, not invited_as_admins, for
readability.

Fixes #6834.
2017-10-27 11:19:38 -07:00
Steve Howell 65b6676844 minor: Remove unused import (internal_send_message). 2017-10-27 10:48:11 -07:00
Steve Howell 655f37a34b Rename subject_name in send_message_backend(). 2017-10-27 10:48:11 -07:00
rht 45e8ce559d zerver/views: Use python 3 syntax for typing. 2017-10-26 21:58:22 -07:00
Tim Abbott 8e2cdedf9a lint: Fix lines in Python codebase longer than 120 characters. 2017-10-26 17:47:30 -07:00
Tim Abbott b936e8c24b lint: Fix lines in Python codebase longer than 125 characters. 2017-10-26 17:36:54 -07:00
Steve Howell df93a99b50 Cache only one row per message.
Before this change, we populated two cache entries for each
message that we sent.  The entries were largely redundant,
with the only difference being whether we sent the content
as raw markdown or as the rendered HTML.

This commit makes it so we only have one cache entry per
message, and it includes both content and rendered_content.

One legacy source on confusion here is that `content`
changes meaning when you're on the front end.  Here is the
situation going forward:

    database:
        content = raw
        rendered_contented = rendered

    cache entry:
        content = raw
        rendered_contented = rendered

    payload for the frontend:
        content = raw (for apply_markdown=False)
        content = rendered (for apply_markdown=True)
2017-10-26 16:35:28 -07:00
derAnfaenger 1792dcbd09 tests: Call real consume method of queue processors.
This switches to more real tests for a first batch of
queue_json_publish() calls that don't cause trouble when
used with call_consume_tests=True.
2017-10-26 14:58:03 -07:00
Tim Abbott b8658c6901 default stream groups: Use cleaner system for error handling.
Wherever possible, we always want to move checking for error
conditions to the views code, so that we don't need to worry about
handling failures with (in this case) a user that's half-created
because a DefaultStreamGroup doesn't exist.
2017-10-26 11:38:27 -07:00
Vishnu Ks eecdc5bb61 registration: Add option to choose default stream groups.
This effectively implements the feature of default stream groups,
except for a UI, nice styling, etc.

Note that we're careful to not have this do anything in an
organization that doesn't have any default stream groups.
2017-10-26 11:34:06 -07:00
Greg Price 093bae4bc5 subdomains: Fix some implicit uses of "" for the root subdomain.
These are just instances that jumped out at me while working on the
subdomains code, mostly while grepping for get_subdomain call sites.
I haven't attempted a comprehensive search, and there are likely
still others left.
2017-10-26 10:29:17 -07:00
Abhijeet Kaur a88178afaf Embedded bots: Add support for creating embedded bots via the API.
Adds support to add "Embedded bot" Service objects. This service
handles every embedded bot.

Extracted from "Embedded bots: Add support to add embedded bots from
UI" by Robert Honig.

Tweaked by tabbott to be disabled by default.
2017-10-25 16:13:29 -07:00
derAnfaenger d0c0b148b3 bot services: Rename add_outgoing_webhook_service to add_service. 2017-10-25 15:46:07 -07:00
darshanime a14ec17473 dev_login: Don't redirect to realm if it doesn't exist.
Fixes #7132.
2017-10-25 15:13:20 -07:00
Vishnu Ks 1d94119d31 actions: Call send_initial_pms from process_new_human_user. 2017-10-25 14:14:59 -07:00
Tim Abbott 4439dbf781 log_into_subdomain: Fix realm creation on servers using an IP address.
While it's totally fine to put a leading '.' before the cookie domain
for normal hostnames and browsers will just strip them, if you're
using an IP address, it doesn't work, because .127.0.0.1 (for example)
is just invalid, and the cookie won't be set.

This fixes an issue where after installing with an IP address, realm
creation would end with being stuck at a blank page for
/accounts/login/subdomain/.
2017-10-24 14:13:31 -07:00
Tim Abbott 3ee53d5de3 auth: Don't offer password reset links when useless.
If an organization doesn't have the EmailAuthBackend (which allows
password auth) enabled, then our password reset form doesn't do
anything, so we should hide it in the UI.
2017-10-24 12:07:43 -07:00
Tim Abbott b81bde3a90 ldap: Prevent creating accounts with Zulip/EmailAuthBackend passwords.
While our recent changing to hide /register means we don't need a nice
pretty error message here, eventually we'll want to clean up the error
message.

Fixes #7047.
2017-10-24 11:21:49 -07:00
Tim Abbott 914e6a38fe registration: Extract login_and_go_to_home.
This new function extractions the bit of logic we use after creating a
new user account to log them in and send them to the home page,
without emailing the user about their new login.
2017-10-24 11:21:49 -07:00
Tim Abbott 70d509196d backends: Call send_initial_pms on other user creation paths.
This fixes a problem we've seen where LDAP users were not getting this
part of the onboarding process, and a similar problem for human users
created via the API.

Ideally, we would have put these fixes in process_new_human_user, but
that would cause import loop problems.
2017-10-24 09:03:53 -07:00
Steve Howell a1d887c950 Use check_bool validator in get_messages_backend().
This is the more modern way in our codebase to validate
bool fields.
2017-10-20 15:51:57 -07:00
Steve Howell 0e106a2488 Add client_gratavar support to GET /messages.
Clients fetching messages can now specify that they are able
to compute their avatar, and if they set client_gratavar to
True in the request (w/our normal encoding scheme), then the
backend will not compute it, and the payload will be smaller.

The fix starts with get_messages_backend.  The flag gets
passed down through these functions:

    * MessageDict.post_process_dicts.
    * MessageDict.set_sender_avatar.

We also fix up the callers for post_process_dicts to explicitly
pass in the client_gravatar path, but for now they all just hard
code the value to False.
2017-10-20 15:49:21 -07:00
Umair Khan 2172185d5a i18n: Store language name in session; not locale.
In the UI we use locale as the code for the language. Django expects
language code. For Simplified Chinese, 'zh_Hans' is the locale which
maps to a directaory under static/locale, and 'zh-hans' is the language
code, which is used in settings.LANGUAGES setting found in Django.
2017-10-19 17:56:49 -07:00
Tim Abbott 642086ff65 subdomains: Pass whether root domain is available to registration.
This data is necessary to determine whether to offer the root domain
in the realm creation form.
2017-10-18 22:53:10 -07:00
Tim Abbott 1ab2ca5986 subdomains: Extract zerver.lib.subdomains library.
These never really belonged with the rest of zerver.lib.utils.py, and
having a separate library makes it easier to enforce full test
coverage.
2017-10-18 22:27:48 -07:00
Tim Abbott f998550fb4 reactions: Add missing mypy import for Message. 2017-10-18 08:59:29 -07:00
Harshit Bansal 31ce96ff2a views/reactions.py: Extract `create_historical_message()` function. 2017-10-17 22:24:34 -07:00
Alena Volkova 5515a075ec urls: Move the report endpoints to be API-style routes. 2017-10-17 22:05:56 -07:00
Steve Howell d909355dc2 refactor: Move methods from models.py -> lib/messages.py.
Message.get_raw_db_rows is moved to MessageDict, since its
implementation details are highly coupled to other methods
in MessageDict.

And then sew_messages_and_reactions comes along for the
ride.

We eventually want to move Reaction.get_raw_db_rows to there
as well.
2017-10-16 23:37:10 -07:00
Tim Abbott a466bf8a47 report: Fix handling of reports with no more_info. 2017-10-12 17:38:28 -07:00
Steve Howell a6c3621f55 Support client_gravatar in /json/users endpoint. 2017-10-12 14:00:41 -07:00
derAnfaenger 7deb698e02 email_log.py: Remove redundant shebang. 2017-10-12 11:18:45 -07:00
Tim Abbott 339e206c90 highlight_html_differences: Improve logging output.
Now at least it will give the message ID, and thus be possible to
debug.
2017-10-11 23:38:29 -07:00
Tim Abbott 065033e864 auth: Fix mypy annotations in authenticate_remote_user. 2017-10-11 23:32:11 -07:00
Tim Abbott 57b8bfe84b authenticate_remote_user: Accept a realm object.
This paves the path for further code cleanup migrations.
2017-10-11 23:27:00 -07:00
Tim Abbott db645f5825 finish_google_oauth2: Depend on a realm existing. 2017-10-11 23:27:00 -07:00
Tim Abbott e48abcfcf1 auth: Fix confusing realm variable name in JWT auth. 2017-10-11 23:27:00 -07:00
Tim Abbott 6176d0fbca json: Replace most use of simplejson with json.
This is progress towards removing simplejson as a dependency.
2017-10-11 22:55:35 -07:00
Tim Abbott b3b5d5b7cd report: Avoid sending raw message content in error reporting.
This fixes a violation of Zulip's privacy policies (that error
reporting never contain message content) in the previous commit.
2017-10-11 17:44:05 -07:00
Steve Howell 3910448b1d Extract MessageDict.post_process_dicts().
Introduce MessageDict.post_process_dicts() will allow us
the ability to do the following:

    * use less memory in the cache for repeated data
    * prevent cache invalidation
    * format data according to different client needs

The first use of this function is pretty inconsequential, but
it sets us up for more consequential changes.

In this commit we defer the MessageDict.hydrate_recipient_info
step until after we pull data out of the cache.  This impacts
cache size as follows:

    * streams - negligibly bigger
    * PMs/huddles - slimmer due to not needing to repeat
                    sender data like email/full_name

Again, the main point of this change is to start setting up
the infrastructure to do post-processing.
2017-10-11 11:37:16 -07:00
Vishnu Ks 63ed4110fe auth: Submit dev login form to the correct subdomain. 2017-10-11 10:16:55 -07:00
Alena Volkova 79560e21bf urls: Move the json/tutorial_status endpoint to be an API-style route. 2017-10-09 15:13:33 -07:00
Steve Howell 75db4f5187 Avoid O(N) query pitfall for /users endpoint.
We now use a `.values` query to get just the fields we need
in order to fulfill '/json/users' requests.

The main benefit is that we don't do O(N) queries for bot
owners, but we also have less data on UserProfile to process.
2017-10-09 14:08:07 -07:00