Commit Graph

10992 Commits

Author SHA1 Message Date
Tim Abbott df90533eef home: Switch invalid pointer logging to by user ID. 2019-11-15 17:07:52 -08:00
Tim Abbott 093e9394dd find_account: Use delivery_email when searching emails.
Previously, this code wouldn't work with
EMAIL_ADDRESS_VISIBILITY_ADMINS.
2019-11-15 17:07:52 -08:00
Tim Abbott 3bc7ba1767 digest: Switch from emails to user IDs for logging.
This is better practice.
2019-11-15 17:07:52 -08:00
Tim Abbott bb64b0fa4d queue processors: Switch SignupWorker to logging user IDs.
This is a better setup than logging emails, especially with
EMAIL_ADDRESS_VISIBILITY_ADMINS.
2019-11-15 17:07:24 -08:00
Tim Abbott b14b18b76f soft_deactivation: Remove 'email' from logging.
The value wasn't correct with EMAIL_ADDRESS_VISIBILITY_ADMINS, and in
any case we really just need the user ID.
2019-11-15 17:06:51 -08:00
Tim Abbott 8b45756dd2 onboarding: Fix welcome emails not using delivery_email.
For organizations with EMAIL_ADDRESS_VISIBILITY_ADMINS, we were using
the wrong email address in the notice telling the user how to login in
the future.
2019-11-15 16:49:28 -08:00
Tim Abbott e124837cdc management commands: Fix incorrect use of user_profile.email.
All of these management commands should be interacting with
.delivery_email; this results in buggy behavior with
EMAIL_ADDRESS_VISIBILITY_ADMINS.
2019-11-15 16:43:51 -08:00
Vishnu Ks 0632d26e5e tests: Remove update-stream from curl test exclude_list. 2019-11-15 15:53:31 -08:00
Vishnu Ks be49dd4f30 tests: Remove delete-user-group from curl test exclude_list. 2019-11-15 15:53:31 -08:00
Vishnu KS ee2d20ff3c tests: Remove update-user-group from curl test exclude_list. 2019-11-15 15:53:31 -08:00
Vishnu Ks 4b212efba6 tests: Remove create-user-group from curl test exclude_list. 2019-11-15 15:53:31 -08:00
Vishnu Ks 473d178fbc tests: Remove delete-stream from curl test exclude_list. 2019-11-15 15:53:31 -08:00
Vishnu Ks 4e15dabcfa tests: Remove remove-subscriptions from curl test exclude_list. 2019-11-15 15:53:31 -08:00
Vishnu KS 3efe35e35b tests: Remove delete-message from curl test exclude_list. 2019-11-15 15:53:31 -08:00
Vishnu Ks 00455df7f9 tests: Use admin client for curl examples test. 2019-11-15 15:53:31 -08:00
Tim Abbott a41a4fcc33 onboarding: Use delivery_email in "new login" notifications.
Since we implemented EMAIL_ADDRESS_VISIBILITY_ADMINS, the intent is
that `delivery_only` should be used for accessing a user's actual
email address; with `email` used only in the Zulip API where we
haven't migrated to interacting with other users by ID.

This fixes a place we neglected to migrate.
2019-11-14 12:19:47 -08:00
Anders Kaseorg 0d20145b93 mypy: Upgrade from 0.730 to 0.740.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-11-13 12:38:45 -08:00
Anders Kaseorg 40f4ead738 mypy: Upgrade from 0.720 to 0.730.
Fixes #13269.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-11-13 12:38:45 -08:00
Anders Kaseorg 6fe5e44b35 settings: Define logging paths with, like, normal human variables.
This makes these variables available for type-checking.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-11-13 12:38:35 -08:00
Anders Kaseorg cafac83676 request: Tighten type checking on REQ.
Then, find and fix a predictable number of previous misuses.

With a small change by tabbott to preserve backwards compatibility for
sending `yes` for the `forged` field.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-11-13 12:35:55 -08:00
Anders Kaseorg b0a7b33f9b push_notifications: Declare token of type str, not bytes.
Declaring a CharField of type bytes made no sense.

Signed-off-by: Anders Kaseorg <andersk@zulipchat.com>
2019-11-12 23:21:20 -08:00
Gloria Elston f8855ca179 api: Remove legacy emoji reactions endpoint.
The original/legacy emoji reactions endpoints made use of HTTP PUT and
didn't have an API that could correctly handle situations where the
emoji names change over time.  We stopped using the legacy endpoints
some time ago, so we can remove them now.

This requires straightforward updates to older tests that were still
written against the legacy API.

Fixes #12940.
2019-11-12 13:07:06 -08:00
Tim Abbott ddd1a0eb00 actions: Convert do_delete_messages to take a Realm.
The function only used the user's realm anyway, so this is a cleaner
API.

This should also make it more convenient to permanently delete
messages manually, since one doesn't have to fetch a random user in
the realm in order to delete a message using the management shell.

No functional change.
2019-11-12 12:20:31 -08:00
Tlazypanda 6945ced76f slack import: Map Slack guest users to Zulip guests.
Slack's Single-User Guest and Multi-User Guest users should be
imported as Zulip guests during data import.

Fixes #13255.
2019-11-12 12:12:59 -08:00
Anders Kaseorg cce85f6ec7 dependencies: Upgrade katex from 0.10.2 to 0.11.1.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-11-11 16:26:31 -08:00
Tim Abbott 44f9ce92e9 bots: Fix rendering of bot owner fields in admin settings.
This fixes two regressions in 1946692f9a.

The first bug was actually introduced much earlier, namely that we
were not sending a `bot_owner_id` field at all for bot users without
an owner.  The correct behavior would have been send `None` for the
owner field.

The second bug was simply that we needed to update the webapp to look
for the `bot_owner_id` field, rather than an old email-address format
`bot_owner` field.

Thanks to Vinit Singh for reporting this bug.
2019-11-08 15:09:44 -08:00
Mateusz Mandera ed40d37e44 ldap: Fix realm_creation=True registration flow.
When creating realm with the ldap backend, the registration flow didn't
properly handle some things - the user wouldn't be set as realm admin,
initial subscriptions and messages weren't created, and the redirect
wasn't happening properly in the case of subdomains.
2019-11-08 14:01:45 -08:00
Mateusz Mandera 9ca26e91e0 ldap registration: Refactor, renaming auth_result to user_profile. 2019-11-08 14:01:45 -08:00
Mateusz Mandera fc4ad44dae registration: Refactor, reordering some conditional blocks. 2019-11-08 14:01:45 -08:00
Mateusz Mandera 071038f111 test_signup: Check that user is admin in RealmCreationTest. 2019-11-08 14:01:45 -08:00
Mateusz Mandera 3daec7783a ldap: Fix development environment configuration.
The state of the FAKELDAP setup for the dev env has fallen behind the
backend changes and updates to fakeldap (which implemented
SCOPE_ONELEVEL searches), as well as having some other minor issues.
This commit restore it to a working state and now all three config modes
work properly.
2019-11-08 14:00:24 -08:00
Tim Abbott 9692df4024 test_classes: Add sending_client argument to send_*_message.
This makes it possible to simlulate messages sent by specific clients,
rather than just "test suite".  Relevant for sending messages where
`message.sent_by_human()` is True.
2019-11-06 17:52:53 -08:00
Tim Abbott 509da6c9a6 get_recipient_info: Simplify user_ids_muting_topic handling.
Rather than subtracting sets in multiple places, it's simpler/cleaner
to just check which users are in the set when processing them.

This refactoring be helpful when we extend the get_recipient_info
logic to handle wildcard mentions as well.
2019-11-06 16:25:34 -08:00
Tim Abbott 43a965ff19 test_event_queue: Remove a stray print statement. 2019-11-06 16:25:34 -08:00
Mateusz Mandera b05a0d0177 social_backends: If no icon is to be displayed, set display_icon to None. 2019-11-05 15:44:07 -08:00
Mateusz Mandera c228f7ebcb email_notifs: Take full advantage of django_to_ldap_username.
django_to_ldap_username is now able to find the correct ldap username in
every supported type of configuration, so we can remove these
conditionals and use django_to_ldap_username in a straight-forward
manner.
2019-11-05 15:32:00 -08:00
Mateusz Mandera 8edbbe7b3c ldap: Make email search config obligatory without LDAP_APPEND_DOMAIN.
Having to account everywhere for both cases of having and not
having email search configured makes things needlessly complicated.
It's better to make the setting obligatory in configurations other than
LDAP_APPEND_DOMAIN.
2019-11-05 15:25:58 -08:00
Mateusz Mandera bfe800b11a ldap tests: Move test_ldap into test_auth_backends.
These tests belong more in test_auth_backends rather than deserving
their own separate file.
2019-11-05 15:25:58 -08:00
Mateusz Mandera bb3ddb9576 ldap tests: Put django_to_ldap tests in DjangoToLDAPUsernameTests.
test_auth_backends had a few random django_to_ldap_username tests laying
around, they belong in DjangoToLDAPUsernameTests.
2019-11-05 15:25:58 -08:00
Tim Abbott 54e357e154 avatars: Fix gravatar URLs with EMAIL_ADDRESS_VISIBILITY_ADMINS.
Previously, we were using user_profile.email rather than
user_profile.delivery_email in all calculations involving Gravatar
URLs, which meant that all organizations with the new
EMAIL_ADDRESS_VISIBILITY_ADMINS setting enabled had useless gravatars
not based on the `user15@host.domain` type fake email addresses we
generate for the API to refer to users.

The fix is to convert these calculations to use the user's
delivery_email.  Some refactoring is required to ensure the data is
passed through to the parts of the codebase that do the check;
fortunately, our automated tests of schemas are effective in verifying
that the new `sender_delivery_email` field isn't visible to the API.

Fixes #13369.
2019-11-05 12:33:20 -08:00
Tim Abbott fb59f98304 test_events: Fix handling of client_gravatar flag.
Previously, we weren't properly passing through the value of the
client_gravatar flag from the caller, resulting in buggy results if
the caller passed client_gravatar=False to do_test().

We happened to not have any uses of this before, but we're about to
add one.
2019-11-05 12:33:20 -08:00
Tim Abbott 399acea755 outgoing webhooks: Call finalize_payload on payloads.
Apparently, the refactor months ago that introduced finalize_payload
wasn't applied to the outgoing webhook code path, resulting in message
dicts with an unexpected format with no avatar_url and some extra
values that were intended to be internal details not relevant to
external clients.

Because this API is not widely used, we expect there to be little to
no impact of converting this back to matching the `get_messages`
interface, as it once was and has always been intended to be.

The one somewhat tricky detail is that we include both the `content`
and `rendered_content` fields, rather than asking the client to pick
which they want via the `apply_markdown` flag, because there is no
place for the client to configure that setting.
2019-11-05 12:26:33 -08:00
Tim Abbott 9bf947846e tests: Improve outgoing webhook testing strategy. 2019-11-05 11:49:00 -08:00
Tim Abbott aad99ce951 mattermost import: Fix handling of channels with no subscribers.
Previously, we skipped setting the list of subscribers to the channel,
which could result in problems if any messages had been posted there
in the past (e.g. because the channel used to have members, but now
doesn't).  It could be correct to skip importing dead channels
altogether, but probably simpler is to just set an empty subscriber list.
2019-11-04 18:10:37 -08:00
Tim Abbott dc682da47a mattermost: Handle replies to private messages.
Previously, our logic to handle Mattermost's "replies" feature didn't
copy the right fields for private messages, where `channel_members` is
included on the message body rather than a `channel` name.
2019-11-04 18:10:37 -08:00
Tim Abbott 47cf99a970 realm: Prevent setting invalid enum policy values.
As discussed in the comment, ideally these checks should be added
completely automatically, rather than needing to be manually added
every time we add a new setting.  But hopefully the example code for
all of the similar enums that this provides will at least provide some
help.
2019-11-03 17:11:56 -08:00
Tim Abbott 4066b58faa send_message: Remove long-obsolete comment about @require_login.
All of our view functions work this way, so it doesn't need a special
comment.
2019-11-03 16:52:28 -08:00
Matheus Melo c96762b7a9 settings: Add setting for who can edit user groups.
Fixes #12380.
2019-11-03 16:45:13 -08:00
Mateusz Mandera 849c925dc9 social_auth tests: Test registration with ldap populator enabled.
Finishes testing for the last remaining case and thus closes #13298.
2019-11-03 16:15:49 -08:00
Mateusz Mandera c1095474c6 social_auth tests: Extract some common logic in the long signup tests. 2019-11-03 16:15:49 -08:00
Mateusz Mandera 5aded51b73 register: Pre-populate Name in social backend flow.
By adding some additional plumbing (through PreregistrationUser) of the
full_name and an additional full_name_validated option, we
pre-populate the Full Name field in the registration form when coming
through a social backend (google/github/saml/etc.) and potentially skip
the registration form (if the user would have nothing to do there other
than clicking the Confirm button) and just create the account and log
the user in.
2019-11-03 16:15:48 -08:00
Mateusz Mandera 34a540bacb context: Rename social_backends to external_authentication_methods.
The main purpose of this is to make that name change happen in
/server_settings. external_authentication_methods is a much better, more
descriptive name than social_backends from API perspective.
2019-11-03 15:55:44 -08:00
Mateusz Mandera a62d084247 social_backends: Rename display_logo to display_icon. 2019-11-03 15:54:05 -08:00
Mateusz Mandera 5a39e70bce social_backends: Remove sort_order from social backend dicts.
These are returned through the API, at the /server_settings
endpoint. It's better to just return the list of dicts with a guarantee
of being sorted in the correct order, than to clutter things with the
sort_order field.
2019-11-03 15:51:49 -08:00
Mateusz Mandera 339f742578 test_signup: Add test for a non-covered case in accounts_register. 2019-11-03 15:51:19 -08:00
Anders Kaseorg f7245e9ec6 unminify: Update for webpack chunk splitting.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-11-01 18:25:30 -07:00
Mateusz Mandera 2e1529e879 registration: Fix logic disabling password requirement in the form.
This small block of code was over-indented. It should be run in this
part of the function unconditionally, not inside an "else" block.
We obviously want it to run regardless of whether
request.POST.get('from_confirmation')
is True or not.
2019-11-01 16:59:06 -07:00
Mateusz Mandera a4838d8b97 urls: Reorder where the accounts/login/google/ endpoint is defined.
Needed so that the google entry in social_backends in /server_settings
shows the new url rather than the legacy accounts/login/google/ url as
the login url.
2019-11-01 16:30:52 -07:00
Mateusz Mandera 29314f3195 api: Remove unused /get_auth_backends endpoint.
This legacy endpoint was designed for the original native Zulip mobile
apps, which were deprecated years ago in favor of the React Native
app.

It was replaced by /server_settings for active use years ago, so it's
safe to remove it now.
2019-11-01 12:30:59 -07:00
Vishnu Ks 2f5f23ecc6 tests: Remove get-presence from curl test exclude_list. 2019-10-30 16:49:27 -07:00
Vishnu Ks 7aabbe2f35 tests: Remove delete-queue from curl test exclude_list. 2019-10-30 16:49:27 -07:00
Vishnu Ks a99b96d9db tests: Remove get-events-from-queue from curl test exclude_list. 2019-10-30 16:49:27 -07:00
Vishnu Ks 4aae633d59 tests: Remove get-stream-topics from curl test exclude_list. 2019-10-30 16:49:27 -07:00
Vishnu Ks db294072be tests: Remove update-subscription-data from curl test exclude_list. 2019-10-30 16:49:27 -07:00
Vishnu Ks eb567abcec tests: Remove mark-as-read-bulk from curl test exclude_list. 2019-10-30 16:49:27 -07:00
Vishnu KS 39f7d250bd tests: Remove update-message-flags from curl test exclude_list. 2019-10-30 16:49:27 -07:00
Vishnu KS 31a50753f3 tests: Remove get-message-history from curl test exclude_list. 2019-10-30 16:49:27 -07:00
Vishnu KS 93c003d2aa tests: Remove update-message from curl test exclude_list. 2019-10-30 16:49:27 -07:00
Vishnu KS 8e9ccdf376 tests: Remove get-raw-message from curl test exclude_list. 2019-10-30 16:49:26 -07:00
Andrew Szeto b312001fd9 rabbitmq: Set a short TCP keepalive idle time on BlockingConnection.
The code comment explains this issue in some detail, but essentially
in Kubernetes and Docker Swarm systems, the container overlayer
network has a relatively short TCP idle lifetime (about 15 minutes),
which can lead to it killing the connection between Tornado and
RabbitMQ.

We fix this by setting a TCP keepalive on that connection shorter than
15 minutes.

Fixes #10776.
2019-10-30 16:15:44 -07:00
Rafid Aslam 447f74ae63 Upgrade pika to 1.1.*.
Upgrade pika to 1.1.* and make some changes accordingly
to comply with the new version.

Fixes #12899.
2019-10-29 17:01:12 -07:00
Mateusz Mandera f229839022 docs: Document social_backends in /server_settings.
This updates the docs for the /server_settings endpoint to document the
newly added social_backends structure.
2019-10-29 15:57:29 -07:00
Hemanth V. Alluri ac9008f564 openapi: Update the test and python code example for get-all-users.
This is following the change to the /users endpoint where we allow
an optional parameter "include_custom_profile_fields" which would
allow the client to request for users' custom profile fields along
with their other standard data.
2019-10-29 15:41:35 -07:00
Hemanth V. Alluri c1370547d5 events: Only send bot_type for bots and thus remove the for_api param. 2019-10-29 15:41:35 -07:00
Hemanth V. Alluri fa6bd42f4c docs: Update the example for /users for custom profile fields.
The previous example no longer gives a good enough idea of what the user
can expect when the `include_custom_profile_fields` boolean parameter is
set to true.
2019-10-29 15:41:35 -07:00
Hemanth V. Alluri 5b3e346369 users: Send custom profile fields with the /profile endpoint. 2019-10-29 15:41:35 -07:00
Mateusz Mandera b870816a75 saml: Sanity-check configuration in both login and signup codepaths. 2019-10-28 15:11:19 -07:00
Mateusz Mandera db29fcbbc4 auth: Add social_backends to /server_settings. 2019-10-28 15:11:19 -07:00
Mateusz Mandera 892d25faa1 auth: Change SAML login url scheme, enabling multiple IdP support.
The url scheme is now /accounts/login/social/saml/{idp_name} to initiate
login using the IdP configured under "idp_name" name.

display_name and display_logo (the name and icon to show on the "Log in
with" button) can be customized by adding the apprioprate settings in
the configured IdP dictionaries.
2019-10-28 15:09:42 -07:00
Mateusz Mandera 28dd1b34f2 auth: Refactor social login rendering.
login_context now gets the social_backends list through
get_social_backend_dicts and we  move display_logo customization
to backend class definition.

This prepares for easily adding multiple IdP support in SAML
authentication - there will be a social_backend dict for each configured
IdP, also allowing display_name and icon customization per IdP.
2019-10-28 15:06:26 -07:00
Jan Koscisz 12bd3c04c0 integration: Fix missing taiga test action.
Previously testing webhook from taiga site would result in error.
This adds missing 'test' action to taiga webhooks.

Fixes #13317.
2019-10-28 14:58:24 -07:00
Mateusz Mandera 8c065d1fcd ldap: Ensure django_to_ldap_username returns username that is in ldap.
This changes the way django_to_ldap_username works to make sure the ldap
username it returns actually has a corresponding ldap entry and raise an
exception if that's not possible. It seems to be a more sound approach
than just having it return its best guess - which was the case so far.
Now there is a guarantee that what it returns is the username of an
actual ldap user.

This allows communicating to the registration flow when the email being
registered doesn't belong to ldap, which then will proceed to register
it via the normal email backend flow - finally fixing the bug where you
couldn't register a non-ldap email even with the email backend enabled.

These changes to the behavior of django_to_ldap_username require small
refactorings in a couple of other functions that call it, as well as
adapting some tests to these changes. Finally, additional tests are
added for the above-mentioned registration flow behavior and some
related corner-cases.
2019-10-25 12:14:51 -07:00
Anders Kaseorg 453919bbc7 version: Only let `git describe` match tags beginning with a digit.
This will let us use other tags for things like `@zulip/shared`
versions.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-10-24 14:54:45 -07:00
Hemanth V. Alluri 1946692f9a users: Refactor get_members_backend endpoint to use get_raw_user_data.
Modify the get_raw_user_data method for use by the /users API endpoint
and then modify the /users endpoint to use it.
2019-10-23 14:50:26 -07:00
Hemanth V. Alluri dca990d60a events: Extract custom profile field logic from get_raw_user_data.
This is a simple refactor to move the code which gets custom profile
field values in the get_raw_user method to a separate function.
2019-10-23 14:36:07 -07:00
Hemanth V. Alluri 34849d7355 test_events: Fix a small typo for test_realm_bots_admin. 2019-10-23 14:35:32 -07:00
David Rosa bd3a8f1e96 docs: Merge front-end-build-process with html-css.
This merges the "Static asset pipeline" article as a new section
at the bottom of the "HTML and CSS" article.
2019-10-23 13:08:36 -07:00
Mateusz Mandera 869b57b7f7 ldap tests: Change TestQueryLDAP tests to use the test ldap directory.
Instead of mocking the _LDAPUser class, these tests can now take
advantage of the test directory that other ldap are using. After these
changes, test_query_email_attr also verifies that query_ldap can
successfully be used to query by user email, if email search is
configured.
2019-10-22 16:03:58 -07:00
Mateusz Mandera 1be2779515 tests: Add ldap_username() and ldap_password() method. 2019-10-22 16:03:58 -07:00
Mateusz Mandera 68f4cd1e94 ldap: Extract ldap user -> django username mapping logic to a function.
Fixes #11878

Instead of a confusing mix of django_auth_backed applying
ldap_to_django_username in its internals for one part of the
translation, and then custom logic for grabbing it from the email
attribute of the ldapuser in ZulipLDAPAuthBackend.get_or_build_user
for the second part of the translation,
we put all the logic in a single function user_email_from_ldapuser
which will be used by get_or_build of both ZulipLDAPUserPopulator and
ZulipLDAPAuthBackend.

This, building on the previous commits with the email search feature,
fixes the ldap sync bug from issue #11878.

If we can get upstream django-auth-ldap to merge
https://github.com/django-auth-ldap/django-auth-ldap/pull/154, we'll
be able to go back to using the version of ldap_to_django_username
that accepts a _LDAPUser object.
2019-10-22 16:02:23 -07:00
Mateusz Mandera 3699fe28f8 ldap: Use email search in django_to_ldap_username.
With this, django_to_ldap_username can take an email and find the ldap
username of the ldap user who has this email - if email search is
configured.

This allows successful authenticate() with ldap email and ldap password,
instead of ldap username. This is especially useful because when
a user wants to fetch their api key, the server attempts authenticate
with user_profile.email - and this used to fail if the user was an ldap
user (because the ldap username was required to authenticate
succesfully). See issue #9277.
2019-10-22 15:57:52 -07:00
Mateusz Mandera fea4d0b2be ldap: Do a proper search for email in email_belongs_to_ldap.
This fixes a collection of bugs surrounding LDAP configurations A and
C (i.e. LDAP_APPEND_DOMAIN=None) with EmailAuthBackend also enabled.

The core problem was that our desired security model in that setting
of requiring LDAP authentication for accounts managed by LDAP was not
implementable without a way to

Now admins can configure an LDAPSearch query that will find if there
are users in LDAP that have the email address and
email_belongs_to_ldap() will take advantage of that - no longer
returning True in response to all requests and thus blocking email
backend authentication.

In the documentation, we describe this as mandatory configuration for
users (and likely will make it so soon in the code) because the
failure modes for this not being configured are confusing.

But making that change is pending work to improve the relevant error
messages.

Fixes #11715.
2019-10-22 15:53:39 -07:00
Tim Abbott 87b43fdac0 tests: Fix whitespace around INVALID_STREAM_ID. 2019-10-21 22:14:46 -07:00
Vishnu KS e080b42fe5 tests: Move get_stream_id function to ZulipTestCase. 2019-10-21 16:56:48 -07:00
Vishnu KS 23036a9f40 confirmation: Set confirmation object realm attribute in realm reactivation.
The value of realm attribute in confirmation object used to be empty
before. We are not currently using the realm attribute of reactivation
links anywhere. The value of realm stored in content_object is currently
used.
2019-10-21 16:52:46 -07:00
Vishnu KS 139ebf387b support: Pass various realm functions as template context.
We currently have code to calculate the value of realm_icon_url,
admin_emails and default_discount in two diffrent places. With
the addition of showing confirmation links it would become three.
The easiest way to deduplicate the code and make the view cleaner
is by doing the calculations in template. Alternatively one can
write a function that takes users, realms and confirmations as
arguments and sets the value of realm_icon_url, admin_emails and
default_discount appropriately in realm object according to the
type of the confirmation. But that seems more messy than passing
the functions directly to template approach.
2019-10-21 16:52:46 -07:00
Tim Abbott b12d3d54c6 events: Fix documentation testing for /events.
Most of the failures were due to parameters that are not intended to
be used by third-party code, so the correct fix for those was the set
intentionally_undocumented=True.

Fixes #12969.
2019-10-21 16:50:10 -07:00
Tim Abbott c35472e025 test_openapi: Fix confused error messages for missing parameters.
These conditionals were backwards.
2019-10-21 16:26:08 -07:00
Tim Abbott c120ce6ecc events: Make it easy for test_openapi to find /events. 2019-10-21 16:26:00 -07:00
chgl bea9e41fbd webhooks: Add Harbor webhook integration. 2019-10-21 15:51:35 -07:00
Pragati Agrawal 857470c5ca events: Add delivery_email to user initial state data.
For admins, this adds `delivery_email` in addition to `email` for the
case when email address visibility is set to admins only.
2019-10-21 15:43:49 -07:00
Vishnu Ks 37c5f9b9c5 tests: Move add-linkifiers to correct position in exclude list. 2019-10-21 15:30:57 -07:00
Vishnu KS f2242c1487 docs: Make update-notification-settings use curl example system. 2019-10-21 17:10:35 +05:30
Mateusz Mandera bbf2474bd0 tests: setUp overrides should call super().setUp().
MigrationsTestCase is intentionally omitted from this, since migrations
tests are different in their nature and so whatever setUp()
ZulipTestCase may do in the future, MigrationsTestCase may not
necessarily want to replicate.
2019-10-19 17:27:01 -07:00
Vishnu KS 0af7aa8db3 docs: Make update-stream use curl example system.
new_name and description params should be valid JSON
strings. The format of these params are marked as
json so that the curl example genenrator can convert
them into json strings.
2019-10-18 13:45:30 -07:00
Vishnu Ks e96d96b6e4 docs: Add curl example to upload-file. 2019-10-18 13:45:30 -07:00
Vishnu Ks 9ac77a8734 docs: Make upload-custom-emoji use curl example system. 2019-10-18 13:45:30 -07:00
Mateusz Mandera 438c4b2935 tests: tearDown overrides should call super().tearDown().
ZulipTestCase subclasses that override tearDown() should call the parent
class tearDown() too, as it does some clean up of its own.
2019-10-18 13:36:59 -07:00
Mateusz Mandera 72dd834088 test_signup: Migrate ldap tests to the new format. 2019-10-17 17:10:39 -07:00
Mateusz Mandera 33050c733d test_email_notifications: Migrate ldap tests to the new format. 2019-10-17 16:49:53 -07:00
Mateusz Mandera e9df17884a test_settings: Migrate ldap tests to the new format. 2019-10-17 16:49:53 -07:00
Mateusz Mandera efba7290fd test_auth_backends: Migrate ldap tests to the new format. 2019-10-17 16:49:53 -07:00
Mateusz Mandera 8cbcf8cf45 tests: Add some logic to ZulipTestCase to prepare to migrate ldap tests. 2019-10-17 16:49:53 -07:00
Tim Abbott d364891894 ldap: Fix password prompt when configured only to populate data.
Previously, the logic for determining whether to provide an LDAP
password prompt on the registration page was incorrectly including it
if any LDAP authentication was backend enabled, even if LDAP was
configured with the populate-only backend that is not responsible for
authentication (just for filling in name and custom profile fields).

We fix this by correcting the conditional, and add a test.

There's still follow-up work to do here: We may still end up
presenting a registration form in situations where it's useless
because we got all the data from SAML + LDAP.  But that's for a future
issue.

This fixes a bug reported in #13275.
2019-10-17 14:46:46 -07:00
Rohitt Vashishtha 5fd0a121ea tests: Clean up and extend topic links tests in test_messages.
This is a follow-up to b69213808a.
We now actually send messages from the notification_bot, which
is the real usecase for this code.

Also, this cleans up the code and removes needless asserts like
`assertNotEqual(zulip_realm, lear_realm)` making the test easier
to read.
2019-10-16 17:39:09 -07:00
Vishnu Ks 1a0a282242 docs: Make delete-queue use curl example system. 2019-10-15 15:53:03 -07:00
Vishnu Ks f7add25cc2 docs: Make get-events-from-queue use curl example system. 2019-10-15 15:53:03 -07:00
Vishnu Ks 9d73578930 openapi: Fix narrow param example in /register POST endpoint. 2019-10-15 15:53:03 -07:00
Vishnu Ks 28e8df15a0 openapi: Fix fetch_event_types param example in /register POST endpoint. 2019-10-15 15:53:03 -07:00
Vishnu Ks 626c536b5e openapi: Fix event_types param example in /register POST endpoint. 2019-10-15 15:53:03 -07:00
Vishnu Ks 0b297fc092 docs: Make remove-linkifiers use curl example system. 2019-10-15 15:53:03 -07:00
Vishnu Ks 616502fd37 docs: Remove realm_logo and realm_night_logo from server-settings response.
They were removed in 16123c9a58
2019-10-15 15:53:03 -07:00
Vishnu Ks 6a5fdcabbd docs: Make delete-user-group use curl example system. 2019-10-15 15:53:03 -07:00
Vishnu Ks d1540f7e10 docs: Make update-user-group use curl example system. 2019-10-15 15:53:03 -07:00
Vishnu Ks c5d6be4df7 docs: Make create-user-group use curl example system. 2019-10-15 15:53:03 -07:00
Vishnu KS 70e8f5ce8a docs: Make get-user-groups use curl example system. 2019-10-15 15:53:03 -07:00
Vishnu KS 519b24061d docs: Update the response schema of get-presence.
The response no longer contains pushable and client keys.
2019-10-15 15:53:03 -07:00
Vishnu KS 503e65079a docs: Make get-presence use curl example system. 2019-10-15 15:53:03 -07:00
Vishnu Ks af83884f6f docs: Make create-user use curl example system. 2019-10-15 15:53:02 -07:00
Vishnu Ks ea06dc6ab3 docs: Make delete-stream use curl example system. 2019-10-15 15:40:44 -07:00
Vishnu Ks 13bdfc0ef8 docs: Make get-stream-topics use curl example system. 2019-10-15 15:40:44 -07:00
Vishnu KS bd6b41cb9d tests: Remove unused variable in test_curl_examples. 2019-10-15 15:40:44 -07:00
Vishnu Ks ad16c55c77 docs: Make remove-subscriptions use curl example system. 2019-10-15 15:40:44 -07:00
Vishnu Ks 4364e31d69 docs: Make update-subscription-properties use curl example system. 2019-10-15 15:40:44 -07:00
Vishnu Ks 89d82ab827 openapi: Improve description of subscriptions in /users/me/subscriptions. 2019-10-15 15:40:44 -07:00
Vishnu Ks c35ef2d390 bugdown: Add option to include only selected fields in generate_curl_example. 2019-10-15 15:40:44 -07:00
Vishnu Ks 3f9835ed07 docs: Make mark-as-read-bulk use curl example system. 2019-10-15 15:40:44 -07:00
Vishnu Ks f0b099dbc0 docs: Make update-message-flags use curl example system. 2019-10-15 15:40:44 -07:00
Vishnu Ks 19ee518f2e docs: Make get-message-history use curl example system. 2019-10-15 15:40:44 -07:00
Vishnu Ks ed2e89f167 docs: Make delete-message use curl example system. 2019-10-15 15:40:44 -07:00
Vishnu Ks 0dccebbe39 docs: Make update-message use curl example system. 2019-10-15 15:40:44 -07:00
Vishnu KS 9bc3594f75 docs: Make get-raw-message use curl example system. 2019-10-15 15:40:44 -07:00
Vishnu Ks cb64fc8732 bugdown: Show example value for path params in curl example. 2019-10-15 15:40:44 -07:00
Vishnu Ks a03765bbe2 bugdown: Extract out function for getting openapi example value. 2019-10-15 15:40:44 -07:00
Rafid Aslam 718b70ec8b Rename `not_subscribed key` to `not_removed` in users/me/subscriptions.
Rename `not_subscibed_key` to `not_removed` in
`users/me/subscriptions` DELETE response.

Fixes #13277.
2019-10-13 10:30:34 +07:00
Mateusz Mandera a94b04dede test_mattermost_importer: Fix leftover references to message.pub_date.
The field is now called date_sent.
2019-10-11 04:07:12 +02:00
Tim Abbott d2970a56c2 lint: Remove some unused imports.
These were introduced in ae5bc92602.
2019-10-10 18:06:30 -07:00
Vishnu KS 1585ad7bf4 mattermost: Add support for exporting DMs and huddles. 2019-10-10 16:37:03 -07:00
Vishnu KS ae5bc92602 queue: Don't create confirmation objects twice during invite.
A confirmation object is already created when
do_send_confirmation_email is called just above.

Tweaked by tabbott to remove an unnecessary somewhat hacky database
query.
2019-10-10 16:19:42 -07:00
Mateusz Mandera 4dc3ed36c3 auth: Add initial SAML authentication support.
There are a few outstanding issues that we expect to resolve beforce
including this in a release, but this is good checkpoint to merge.

This PR is a collaboration with Tim Abbott.

Fixes #716.
2019-10-10 15:44:34 -07:00
Rohitt Vashishtha b69213808a messages: Render topic links in context of stream realm.
Priviously, we rendered the topic links using the msg.sender.realm.
This resulted in issues with Zulip's internal bots not having access
to the realm_filters of the destination stream's realm. For example,
sending a message via the email gateway or notification would not
linkify any realm filters that a user would expect them to.
2019-10-10 14:33:35 -07:00
Anders Kaseorg f25968f0ff exceptions: Fix AbstractEnum typing.
This will be needed for mypy 0.730.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-10-08 17:49:28 -07:00
Rishi Gupta 360cd7f147 remote data: Send RealmAuditLog data. 2019-10-08 17:27:29 -07:00
Rishi Gupta b86142089b billing: Sync realm activation events for on-prem billing. 2019-10-08 17:18:30 -07:00
Rishi Gupta 01bf99ec65 billing: Keep track of role count changes in RealmAuditLog. 2019-10-08 17:18:30 -07:00
Anders Kaseorg cc791afc54 requirements: Upgrade django-sendfile2.
django-sendfile2 now always sends a Content-Disposition header even if
it’s inline.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-10-08 17:15:00 -07:00
Vishnu KS cd06e0ab79 tests: Remove /messages/{message_id} from buggy endpoints. 2019-10-07 11:45:11 -07:00
Vishnu KS bfc9089124 tests: Remove /users/me/subscriptions/muted_topics from buggy endpoints.
Even though required attribute of stream and stream_id params is marked
false in openapi specification, the API expects atleast one of the
params to be set. There is no way to specify relationships like this
openapi and they dont seem to have any plan to implement this in future.

https://github.com/OAI/OpenAPI-Specification/issues/256
2019-10-07 11:42:52 -07:00
Vishnu KS e7419f815a tests: Remove /settings/notifications from buggy endpoints. 2019-10-07 11:42:52 -07:00
Vishnu KS 21ab8c542a bugdown: Cast enum elements to string in APIArgumentsTablePreprocessor.
So that enums other than of type string gets rendered without any error.
2019-10-07 11:42:52 -07:00
overide dd67b1b3bb topic: Change before bound limit from 2 to 7 days.
This limit was introduced in c588c79 as a part of the
feature and not due to performance crisis. So we are
increasing this limit to 7 days. Since topics tends to
naturally fizzle after day or two so 7 days limit
would be good enough.
2019-10-06 20:12:59 -07:00
Rishi Gupta 48dc1d1128 remote data: Refactor remote_server_post_analytics to be more generic.
One small change in behavior is that this creates an array with all the
row_objects at once, rather than creating them 1000 at a time.

That should be fine, given that the client batches these in units of
10000 anyway, and so we're just creating 10K rows of a relatively
small data structure in Python code here.
2019-10-06 16:55:41 -07:00
Rishi Gupta e10361a832 models: Replace is_guest and is_realm_admin with UserProfile.role.
This new data model will be more extensible for future work on
features like a primary administrator.
2019-10-06 16:24:37 -07:00
Rishi Gupta 4256ee61cf billing: Change RealmAuditLog.event_type from str to int.
This is a more robust long-term model for storing these data.
2019-10-06 15:55:56 -07:00
Shikhar Varshney fe5d975b33 settings: Add support for overriding APNS_TOPIC and ZULIP_IOS_APP_ID.
Documentation added by tabbott.
2019-10-05 22:53:59 -07:00
Mateusz Mandera dbe508bb91 models: Migration of Message.pub_date to date_sent, part 2.
Fixes #1727.

With the server down, apply migrations 0245 and 0246. 0246 will remove
the pub_date column, so it's essential that the previous migrations
ran correctly to copy data before running this.
2019-10-05 19:01:34 -07:00
Mateusz Mandera f198475281 models: Migration of Message.pub_date to date_sent, part 1.
1. Apply migration 0243 to add date_sent column.
2. Apply migration 0244 to copy pub_date over to date_sent. Can be done
with the server running.
3. With the server down (for consistency between memory and
database state of Django objects), verify consistency with
Message.objects.exclude(date_sent=F("pub_date")).count() == 0
2019-10-05 19:01:34 -07:00
Vishnu Ks 30ad650afb bugdown: Don't show path params as data in curl example.
With test added by tabbott.
2019-10-05 18:17:36 -07:00
Vishnu Ks bfd5f4066a bugdown: Remove unecessary call to get_openapi_parameters. 2019-10-05 18:10:55 -07:00
Vishnu Ks dff62a240b bugdown: Rename openapi_example_params to openapi_params in generate_curl_example. 2019-10-05 18:10:55 -07:00
Vishnu Ks f4e3e6068f bugdown: Rename pocket to param in generate_curl_example. 2019-10-05 18:10:55 -07:00
Mateusz Mandera 94d5ca838f social auth tests: Extract prepare_login_url_and_headers method.
It will be reused in SAML tests in upcoming commits.
2019-10-01 16:39:12 -07:00
Mateusz Mandera 0e5f964363 social auth tests: Consistently refer to hamlet's name via self.name. 2019-10-01 16:38:50 -07:00
Tim Abbott c869a3bf82 upload: Fix browser caching of uploads with local uploads backend.
Apparently, our change in b8a1050fc4 to
stop caching responses on API endpoints accidentally ended up
affecting uploaded files as well.

Fix this by explicitly setting a Cache-Control header in our Sendfile
responses, as well as changing our outer API caching code to only set
the never cache headers if the view function didn't explicitly specify
them itself.

This is not directly related to #13088, as that is a similar issue
with the S3 backend.

Thanks to Gert Burger for the report.
2019-10-01 15:15:17 -07:00
Ivan Mitev 16c9d63056 emoji: Fix sort order of emoji choices.
The previous code for ensuring the sort order of emoji choices was
correct relied on an OrderedDict structure, which isn't guaranteed to
be preserved when passed to the frontend via JSON (in fact, it isn't,
since we converted the way page_params is passed to use
sort_keys=True).  Switch it to a list of dictionaries to correct this.

Fixes #13220.
2019-10-01 13:54:55 -07:00
Mateusz Mandera 4166c901ef do_update_user_custom_profile_data: Rename to ..._if_changed.
This adds clarity to the fact that the function no longer does
anything if the field values haven't changed.
2019-10-01 13:52:43 -07:00
Mateusz Mandera d66cbd2832 do_update_user_custom_profile_data: Don't notify if value not changed. 2019-10-01 13:50:14 -07:00
Tim Abbott 2b227d6b63 models: Add comments documenting our name field decisions. 2019-09-27 18:01:56 -07:00
Tim Abbott bf088519a7 api: Add avatar URL to the GET /profile endpoint.
This endpoint is legacy in a lot of ways, but it seems reasonable that
it should have these data.
2019-09-27 12:07:03 -07:00
Vishnu Ks 1c81538570 mattermost: Read fixures inside test cases instead of setUp. 2019-09-25 20:07:00 +05:30
Vishnu KS a21856c569 mattermost: Rename user_id to sender_user_id in process_raw_message_batch. 2019-09-25 20:06:47 +05:30
Vishnu KS 23d70bb685 mattermost: Rename get_recipient_id to get_recipient_id_from_receiver_name. 2019-09-25 20:06:04 +05:30
Vishnu KS a743991935 mattermost: Fix mirror dummy fixture data.
The channel where Snape sends message should be gryffindor-quidditch-team
and not slytherin-quidditch-team since he is mirror dummy in gryffindor.
2019-09-25 19:03:43 +05:30
Tim Abbott 7e0ea61b00 upload: Support S3-compatible S3 hosting providers.
Previously, we were hardcoding the domain s3.amazonaws.com.  Given
that we already have an interface for configuring the host in
/etc/zulip/boto.cfg (which in turn, automatically configures boto), we
just need to actually use the value configured in boto for what S3
hostname to use.

We don't have tests for this new use case, in part because they're
likely annoying to write with `moto` and there hasn't been a huge
amount of demand for it.  Since this doesn't regress existing S3
backend support, it seems worth merging.
2019-09-24 17:17:21 -07:00
Tim Abbott b8b0ae362c uploads: Only initialize S3 connection once in __init__.
This should be a mild performance optimization for the S3
authentication backend, since we aren't initializing unnecessary
duplicate connections.
2019-09-24 17:15:44 -07:00
Hemanth V. Alluri 98a33c0fda page_params: Fix list of config_options sent for incoming webhook bots.
This patches an issue in f37535044 where we mistakenly tried to send
the function as part of the page_params. Instead, we should just try
to send the list of configuration options (in their user displayable
form).
2019-09-24 16:04:51 -07:00
Laurel Michaels a67769091d docs: Add OpenAPI data for `/messages/{message_id}/reactions`.
This change adds the OpenAPI data needed to document the POST and
DELETE methods associated with this endpoint.

Descriptions edited slightly by tabbott.
2019-09-24 15:57:51 -07:00
Tim Abbott 96726c00ce export: Fix broken URLs in UI with S3 backend.
Apparently, the Zulip notifications (and resulting emails) were
correct, but the download links inside the Zulip UI were incorrectly
not including S3 prefix on the URL, making them not work.

While we're at this, we rewrite the somewhat convoluted previous
system for formatting the data export output.
2019-09-24 13:56:49 -07:00
Tim Abbott 8024b1179a bots: Fix bot email addresses with EMAIL_ADDRESS_VISIBILITY_ADMINS.
When using our EMAIL_ADDRESS_VISIBILITY_ADMINS feature, we were
apparently creating bot users with different email and delivery_email
properties, due to effectively an oversight in how the code was
written (the initial migration handled bots correctly, but not bots
created after the transition).

Following the refactor in the last commit, the fix for this is just
adding the missing conditional, a test, and a database migration to
fix any incorrectly created bots leaked previously.
2019-09-23 15:53:53 -07:00
Tim Abbott 6e5c99328a models: Extract a helper function for whether emails are realm-public.
This is also a useful preparatory refactor for having a user setting
controlling whether one's own email address is publicly available
within the organization.
2019-09-23 15:53:53 -07:00
Anders Kaseorg 9339402541 messages: Sort IDs in the by_streams query deterministically.
This fixes a test flake after upgrading SQLAlchemy.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-09-23 11:56:52 -07:00
Anders Kaseorg cb0652071f zulip_otp_required: Avoid _user_is_authenticated compatibility shim.
This private shim is not needed with Django ≥ 1.10, and was removed
from django-otp 0.7.0.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-09-23 11:56:52 -07:00
Rishi Gupta 3646936cfb integrations: Advertise our native Zapier integration. 2019-09-22 18:47:42 -07:00
Tim Abbott 767a1b9f0c user_activity: Skip an unnecessary database query for new entries.
While new entries are an overwhelming minority of calls to this
function, it's still worth having a clean implementation.
2019-09-21 11:49:43 -07:00
Tim Abbott 1c73ce2450 user_activity: Use LoopQueueProcessingWorker strategy.
This should dramatically improve the queue processor's performance in
cases where there's a very high volume of requests on a given endpoint
by a given user, as described in the new docstring.

Until we test this more broadly in production, we won't know if this
is a full solution to the problem, but I think it's likely.  We've
never seen the UserActivityInterval worker end up backlogged without a
total queue processor outage, and it should have a similar workload.

Fixes #13180.
2019-09-21 11:48:24 -07:00
Tim Abbott 52a2b2c6a8 tests: Move user_activity test next to slow_query test.
This will conveniently put all the LoopQueueProcessingWorkers
together.
2019-09-21 11:47:14 -07:00
Tim Abbott f0d8951035 do_update_user_activity: Refactor to support passing a count.
We'll use this in upcoming commits.
2019-09-21 11:47:14 -07:00
Tim Abbott 5c960b3e0f user_activity: Make the queue processor a bit more efficient.
We don't actually need to go to the memcached (falling back to the
database) to fetch either user or client objects on every event.  For
user objects, we actually can just pass through the user ID
transparently; for client objects, we can use an in-process cache,
since the mapping of string to ID never changes.
2019-09-21 11:47:14 -07:00
Mateusz Mandera 3271235200 email_mirror: Ignore missed message email if the user isn't active. 2019-09-20 17:58:10 -07:00
Mateusz Mandera 21459ed193 test_email_mirror: Test missed message emails in a single ZulipTestCase.
With the way these tests are, it's unnecessary to have 3 separate
classes, and it makes it confusing to decide where to add a potential
additional mm email test.
2019-09-20 17:58:10 -07:00
Tim Abbott c19b7ec89d test_events: Remove unnecessary check_none validator.
This can be done cleanly using the existing equals(None).
2019-09-20 11:51:50 -07:00
Anders Kaseorg cf5a00d94b bulk_get_subscriber_user_ids: Sort each user list by ID.
This simple backwards-compatible change saves approximately 12% in the
compressed size of the chat.zulip.org page_params.  We can do much,
much better by changing the format, but this seems like a good
intermediate step.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-09-19 20:17:43 -07:00
Rishi Gupta 74d5d3db26 integration docs: Remove angle brackets from trello bash command.
If you accidentally leave them in, you'll get a confusing error message.
2019-09-19 17:48:16 -07:00
Yago González e1fbf6dddb actions: Make do_regenerate_api_key return the new key.
This way, the new API key can be fetched without needing to read it from
the UserProfile object.
2019-09-19 13:03:56 -07:00
Anders Kaseorg fbc2de157e templates: Move page_params to a <div> at the bottom of <body>.
In a gigantic realm where we send several MB of `page_params`, it’s
slightly better to have the rest of the `<body>` available to the
browser earlier, so it can show the “Loading…” spinner and start
fetching subresources.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-09-19 12:38:10 -07:00
Mateusz Mandera 935702b6ec auth: Rename any_oauth_backend_enabled to any_social_backend_enabled.
any_oauth_backend_enabled is all about whether we will have extra
buttons on the login/register pages for logging in with some non-native
backends (like Github, Google etc.). And this isn't about specifically
oauth backends, but generally "social" backends - that may not rely
specifically rely on Oauth. This will have more concrete relevance when
SAML authentication is added - which will be a "social" backend,
requiring an additional button, but not Oauth-based.
2019-09-19 12:35:27 -07:00
Anders Kaseorg fc0554fcb8 emoji: Finish removing leftover code from banned Emoji sets.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-09-19 12:08:52 -07:00
Vishnu Ks c4af0b7bc4 mattermost: Support importing messages without team name.
Mattermost doesn't place private messages within a particular team,
which is what this is needed for.
2019-09-18 11:57:37 -07:00
Vishnu Ks bf5f531e90 import_util: Support huddles in SubscriberHandler. 2019-09-18 11:53:13 -07:00
Rishi Gupta 2646d70cbd emails: Add realm name to footer of missed message emails.
Missed message emails for mentions come from streams and possibly orgs you
don't normally get missed message emails for, so they can be hard to place.
2019-09-17 17:45:59 -07:00
Anders Kaseorg fe7d814e8d team: Move contributors_list into page_params.
This sidesteps tricky escaping issues, and will make it easier to
build a strict Content-Security-Policy.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-09-17 16:06:33 -07:00
Anders Kaseorg 7494f1600c templates: Move page_params from an inline script to the <body> dataset.
This sidesteps tricky escaping issues, and will make it easier to
build a strict Content-Security-Policy.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-09-17 16:06:33 -07:00
Wyatt Hoodes d2c3ddc765 test_events: Use stdout_suppressed in export test. 2019-09-13 11:54:14 -07:00
Wyatt Hoodes e37c91d195 test_realm_export: Use stdout_suppressed in tests.
From a testing perspective, calling `export_realm_wrapper`
prints unnecessary data to stdout.
2019-09-13 11:54:14 -07:00
Wyatt Hoodes e3595e4b3f test_slack_message_conversion: Use subTest for fixtures test. 2019-09-13 11:54:14 -07:00
Wyatt Hoodes dbaf6ac7e7 test_push_notifications: Remove fixtures print statement. 2019-09-13 11:54:14 -07:00
Wyatt Hoodes 4a64a487d3 bugdown: Remove print statement from fixtures test. 2019-09-13 11:54:14 -07:00
Wyatt Hoodes 698b0f1b91 test_management_commands: Use subTest with help tests.
We remove the print statement, and instead use
subTest to improve debuggability.
2019-09-13 11:54:14 -07:00
Anders Kaseorg 78b018989e export: Refuse to overwrite an existing directory or tarball.
Previously, incorrectly passing an existing directory to the
`manage.py export --output` option would remove its contents without
warning.  Abort instead.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-09-12 16:04:55 -07:00
Anders Kaseorg 780ecb672b CVE-2019-16216: Fix MIME type validation.
* Whitelist a small number of image/ types to be served as
  non-attachments.
* Serve the file using the type that we validated rather than relying
  on an independent guess to match.

This issue can lead to a stored XSS security vulnerability for older
browsers that don't support Content-Security-Policy.

It primarily affects servers using Zulip's local file uploads backend
for servers running Ubuntu 16.04 Xenial or newer; the legacy local
file upload backend for (now EOL) Ubuntu 14.04 Trusty was not affected
and it has limited impact for the S3 upload backend (which uses an
unprivileged S3 bucket domain to serve files).

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-09-11 15:46:36 -07:00
Thomas Ip 574c35c0b8 markdown: Render ordered lists using <ol> markup.
This brings us in line, and also allows us to style these more like
unordered lists, which is visually more appealing.

On the backend, we now use the default list blockprocessor + sane list
extension of python-markdown to get proper list markup; on the
frontend, we mostly return to upstream's code as they have followed
CommonMark on this issue.

Using <ol> here necessarily removes the behaviour of not renumbering
on lists written like 3, 4, 7; hopefully users will be OK with the
change.

Fixes #12822.
2019-09-08 16:42:20 -07:00
Wyatt Hoodes 4a176714e1 actions: Remove leftover print statement. 2019-09-08 15:36:30 -07:00
Tim Abbott 7e75f987df ldap: Fix logging of warning for deactivated users.
Also cleans up the interface between the management command and the
LDAP backends code to not guess/recompute under what circumstances
what should be logged.

Co-authored-by: mateuszmandera <mateusz.mandera@protonmail.com>
2019-09-08 09:35:23 -07:00
Tim Abbott d1a2784d52 ldap: Fix attempting to sync data for deactivated users.
The order of operations for our LDAP synchronization code wasn't
correct: We would run the code to sync avatars (etc.) even for
deactivated users.

Thanks to niels for the report.

Co-authored-by: mateuszmandera <mateusz.mandera@protonmail.com>
2019-09-08 09:35:23 -07:00
Mateusz Mandera 2ce2024bd7 ldap: Fix unintended user deactivation in case of connection failure.
Fixes #13130.

django_auth_ldap doesn't give any other way of detecting that LDAPError
happened other than catching the signal it emits - so we have to
register a receiver. In the receiver we just raise our own Exception
which will properly propagate without being silenced by
django_auth_ldap. This will stop execution before the user gets
deactivated.
2019-09-05 11:59:20 -07:00
Mateusz Mandera dfd6771237 stream settings: Use the .show-sender version of email address.
As requested in #13134.
2019-09-05 11:48:32 -07:00
Mateusz Mandera 307d8d8758 test_email_mirror: Better document the various address formats. 2019-09-05 11:48:32 -07:00
Vishnu KS 30440cf466 mattermost: Fix test flake in test_convert_channel_data.
So the reason 38f8cf612c seems
to be flaking is because the value of harry id switches between
1 and 2 in Xenial while in Bionic it would be fixed at 2. The
reason behind this is that Bionic ships with Python3.6 which
preserves dict insert order while Python3.5 that ships with Xenial
dont preserve the order. In initialize_stream_membership_dicts
we iterate user_data_map dict and the order in which the iteration
happens affects the ID of the users.
2019-09-05 11:20:33 -07:00
Puneeth Chaganti df134be235 webhooks/papertrail: Requests from Papertrail are not JSON requests.
Papertrail sends requests with the content type
`application/x-www-form-urlencoded`, with the payload parameter holding the
JSON body. This commit fixes the papertrail integration to use the payload
parameter in the request's POST data instead of trying to parse the
request's entire body as JSON.

Papertrail documentation here:
https://help.papertrailapp.com/kb/how-it-works/web-hooks#encoding
2019-09-04 13:16:17 -07:00
Wyatt Hoodes fc2574fdfd test_mattermost_importer: Add test coverage.
This is a rebase from a recent revert. A test
was failing non-deterministically relating to
`convert_channel_data` and use of an incorrect
`realm_id`.
2019-09-03 12:36:41 -07:00
Mateusz Mandera a7ec1a089c errors: Fix infinite recursion during memcached outages.
We have a very useful piece of code, _RateLimitFilter, which is
designed to avoid sending us a billion error emails in the event that
a Zulip production server is down in a way that throws the same
exception a lot. The code uses memcached to ensure we send each
traceback roughly once per Zulip server per 10 minutes (or if
memcached is unavailable, at most 1/process/10 minutes, since we use
memcached to coordinate between processes)

However, if memcached is down, there is a logging.error call internal
to the Django/memcached setup that happens inside the cache.set() call,
and those aren't caught by the `except Exception` block around it.

This ends up resulting in infinite recursion, eventually leading to
Fatal Python error: Cannot recover from stack overflow., since this
handler is configured to run for logging.error in addition to
logging.exception.

We fix this using a thread-local variable to detect whether we are
being called recursively.

This change should prevent some nasty failure modes we've had in the
past where memcached being down resulted in infinite recursion
(resulting in extra resources being consumed by our error
notifications code, and most importantly, the error notifications not
being sent).

Fixes #12595.
2019-09-03 11:44:39 -07:00
Tim Abbott 70c513a640 analytics: Fix logging for errors connecting to push bouncer.
There's no reason for this to be a category of error that emails the
server administrator, since there's a good chance that fixing it will
need to be done in the Zulip codebase, not administrator action.
2019-09-02 18:47:10 -07:00
Mateusz Mandera d70e1bcdb7 settings: Add FAKE_EMAIL_DOMAIN setting.
Fixes #9401.

This adds a FAKE_EMAIL_DOMAIN setting, which should be used if
EXTERNAL_HOST is not a valid domain, and something else is needed to
form bot and dummy user emails (if email visibility is turned off).
It defaults to EXTERNAL_HOST.

get_fake_email_domain() should be used to get this value. It validates
that it's correctly set - that it can be used to form valid emails.

If it's not set correctly, an exception is raised. This is the right
approach, because it's undesirable to have the server seemingly
peacefully operating with that setting misconfigured, as that could
mask some hidden sneaky bugs due to UserProfiles with invalid emails,
which would blow up the moment some code that does validate the emails
is called.
2019-08-30 14:59:00 -07:00
Tim Abbott 62c9ea7cf9 linkifiers: Fix problems with capture groups called "name".
Apparently, due to poor naming of the outer capture group we use to
separate the actual match from the surrounding whitespace (etc.) we
use to determine if the syntax is a possible linkifier start/end, if
you created a linkifier using "name" as the capture group, we'd try to
compile a pattern with two capture groups called "name", which would
500, preventing anyone from accessing the organization.
2019-08-30 09:36:14 -07:00
Anders Kaseorg c3e395b7d8 search: Reimplement ts_locs_array in pure PostgreSQL.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-28 17:59:12 -07:00
Anders Kaseorg d2d4800b1c test_narrow: Compile queries with the current SQLAlchemy dialect.
We’re about to start using PostgreSQL-specific syntax that can’t be
stringified without a specified dialect.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-28 17:58:01 -07:00
rht 07808e35be parse_lsb_release: Use /etc/os-release instead of /etc/lsb-release. 2019-08-28 17:53:27 -07:00
Yashashvi Dave e05429b6a9 custom fields: Add default external account custom fields. 2019-08-28 15:35:53 -07:00
Anders Kaseorg e8ee71e9a2 restore-backup, test_docs: Import settings from django.conf.
A settings module isn’t supposed to be imported directly.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-28 15:33:16 -07:00
Mateusz Mandera 6bd34afa0d models: Migration of UserMessage.id to bigint, part 2.
This requires part 1 (which can take hours to run but generally
doesn't require downtime) to be completed first.

This portion of the migration will require the server to be completely
down for a brief period; for chat.zulip.org with 250M UserMessage
rows, it took about 60s to run; that time will vary depending on
hardware details like whether the server has an SSD, but fundamentally
shouldn't be long.

Our upgrade-zulip and upgrade-zulip-from-git tools can apply this
migration correctly; nothing special needs to be done.

Fixes #13040.
2019-08-26 21:25:19 -07:00
Mateusz Mandera b008515d63 models: Migration of UserMessage.id to bigint, part 1.
As part of adding support for more than 2B UserMessage rows in a Zulip
server, we need to change UserMessage.id (a field we don't access but
is needed by Django) from an int to a bigint.  This commit is a series
of migrations which create a `bigint_id` column and populates it correctly.

This migration will take a long time to run; on chat.zulip.org (a
server with a lot of history), it took about 4 hours to complete.

How to migrate with minimal downtime:

1. Run `upgrade-zulip-from-git` through this commit.  It will install
migration 0238 and then more or less hang while applying migration
0239.  Once migration 0238 is completed, however, your server should
be able to be started back up safely while migration 0239 is running.

2. Run `/home/zulip/deployments/next/scripts/restart-server` in a
separate terminal to get Zulip running again.

3. When the `upgrade-zulip-from-git` command finishes, it will
automatically re-restart the Zulip server, leaving you in a consistent
state and ready to do part 2 of the migration.

A useful `manage.py shell` query for checking the state after this
commit is consistent is this:

    assert UserMessage.objects.exclude(bigint_id=F("id")).count() == 0

Part of #13040.
2019-08-26 21:14:15 -07:00
Tim Abbott 89aeefed76 urls: Tighten many unintentionally broad URL patterns.
Previously, several of our URL patterns accidentally did not end with
`$`, and thus ended up controlling just the stated URL, but actually a
much broader set of URLs starting with it.

I did an audit and fixed what I believe are all instances of this URL
pattern behavior.  In the process, I fixed a few tests that were
unintentionally relying on the behavior.

Fixes #13082.
2019-08-26 20:51:49 -07:00
Tim Abbott e0f8228d6e auth: Add a test for the legacy /accounts/login/google/ mobile flow.
This is needed to return 100% URL coverage, and should also help
ensure we don't accidentally break the fix from
a43b231f90.
2019-08-26 20:51:49 -07:00
Tim Abbott c037ba49a0 i18n: Tag push notification subtitles for translation. 2019-08-26 15:02:04 -07:00
Tim Abbott 0ed0bb6828 messages: Add email/push notifications for wildcard mentions.
Historically, Zulip's implementation of wildcard mentions never
triggered either email or push notifications, instead being limited to
desktop notifications and the "mentions" counter.

We fix this just by plumbing the "wildcard_mentioned" flag through our
system.

Implements much of
https://github.com/zulip/zulip/issues/6040#issuecomment-510157264.
We're also now ready to seriously work on #3750.
2019-08-26 14:39:53 -07:00
Vishnu KS d434c0ee88 slack: Remove unnecessary comments.
Remove comments that tries to explain code that is already readable.
Also remove some todo comments that has been already taken care of.
2019-08-26 14:10:19 -07:00
Vishnu KS 99d34fd11d slack: Rename default_channels to slack_default_channels. 2019-08-26 14:10:19 -07:00
Vishnu KS b919514f7f slack: Rename customprofilefield_id to custom_profile_field_id. 2019-08-26 14:10:19 -07:00
Vishnu KS c31355f9c1 slack: Rename custom_field_id_count to custom_profile_field_value_id_count. 2019-08-26 14:10:19 -07:00
Vishnu KS 138c659c97 slack: Rename slack_custom_field_name_to_zulip_custom_field_id.
Rename custom_field_map to
slack_custom_field_name_to_zulip_custom_field_id.
2019-08-26 14:10:19 -07:00
Vishnu KS 9560736d86 slack: Rename slack_user_id_to_custom_profile_fields.
Renames slack_user_custom_field_map to
slack_user_id_to_custom_profile_fields for readability.
2019-08-26 14:10:19 -07:00
Vishnu KS 01a51c8f4e slack: Rename added_recipient to slack_recipient_name_to_zulip_recipient_id. 2019-08-26 14:10:19 -07:00
Vishnu KS 9d51a1b527 slack: Rename added_users to slack_user_id_to_zulip_user_id. 2019-08-26 14:10:19 -07:00
Vishnu KS 3650f19692 slack: Lookup dir_name key in dict instead of in dict_keys.
No reason to do the lookup in O(n) when we can do
it in average O(1) time complexity.
2019-08-26 14:10:19 -07:00
Vishnu Ks 1e5c49ad82 slack: Support importing shared channels. 2019-08-26 14:10:19 -07:00
Vishnu Ks e09a29f4d3 slack: Refactor get_slack_api_data to accept multiple query params. 2019-08-26 14:10:19 -07:00
Tim Abbott 73258efb16 development: Accept ToS for create user buttons.
We added default ToS for the development environment a few months
back; as a side effect, we now need to accept ToS when going through
the development environment registration flow, including for our
one-click account creation buttons.
2019-08-26 13:55:55 -07:00
Tim Abbott b8bfdebea1 onboarding: Mark the most recent 20 messages as unread.
After a new user joins an active organization, it isn't obvious what
to do next; this change causes there to be recent unread messages in
the stream sidebar for the user to click on to get a feel for what's
happening in the organization and experiment with Zulip.

Fixes #6512.
2019-08-26 13:51:52 -07:00
Rohitt Vashishtha 8b443a25b8 markdown: Show link href if title is empty.
Fixes #6221.
2019-08-25 21:36:42 -07:00
Rohitt Vashishtha abe2dab88c markdown: Upgrade to use InlineProcessor for links.
This commit wraps up the major work that we held back when upgrading
py-markdown 2.6.11 to 3.0.1. Since we were making our custom changes
to the link syntax, at the time we stuck to using the old method of
parsing links. This lays the groundwork for further changes to our
link and image link handling, and brings us on par with upstream.

Also, we now better document the ways in which our link handling is
different from upstream.
2019-08-25 21:36:42 -07:00
Kanishk Kakar e4f0d3d79b notifications: Add 'none' to unread count options. 2019-08-25 21:29:10 -07:00
Yashashvi Dave 330d065a69 zerver/tests/test_custom_profile_data: Group tests cases. 2019-08-25 20:59:13 -07:00
Yashashvi Dave ed5cb6c28b custom profile fields: Extract field validation function. 2019-08-25 20:59:08 -07:00
Tim Abbott 5e19546c14 messages: Fix unread_msgs accounting of wildcard mentions.
Previously, the unread_msgs data structure accounting (used for both
the web and mobile apps to determine the "Unread mentions" count
displayed in the UI) did not include wildcard mentions at all.

We fix this by adding the logic required to include properly that
data, with tests.  As discussed in #6040, it makes sense to include
muted streams and topics for the purpose of this calculation.

Fixes part of #6040.
2019-08-25 20:32:12 -07:00
Tim Abbott 625eb53b08 actions: Fetch presence idle data for non-basic mentions.
Apparently, get_active_presence_idle_user_ids, which is carefully
optimized to only fetch data for users who might actually need
notification processing, was only considering PMs and direct mentions,
not wildcard mentions or alert words.

This caused some pretty weird failure modes when working on adding
support for broader mention notifications, because users who had one
of these types of notifications would be treated as never
presence-idle, which was just confusing.

This is part of adding support for notifications for wildcard mentions
and alert words; it's worth merging this as an early commit because
the consequence of not doing this are very difficult to debug.
2019-08-25 19:58:33 -07:00
Tim Abbott 4e65f1dd2b test_event_queue: Clean up testing setup.
Rather than continually resetting the contents of an existing event
queue, we allocate a new one for each subtest.

We also fix a rather confusing bundle of comments.
2019-08-25 19:37:08 -07:00
Mateusz Mandera 16479fa606 populate_db: Extract internal realm creation into a function. 2019-08-23 12:57:24 -07:00
Mateusz Mandera 8e1a7cfb52 python_examples: Specify which user the hard-coded user ids refer to.
We add an ensure_users function and use it in tests which have
hard-coded user ids, to make it clear to which users the ids refer to
and have it verified.
This makes it easier to see what's happening
in these tests and to keep track of any renumberings of user ids due to
changes in how we populate the database.
2019-08-23 12:57:24 -07:00
Rishi Gupta e058558a52 emails: Send invitation reminder email two days before expiry.
Hopefully this does a better job of spurring people to action, and also
suggests a self-service fix if they don't (i.e. contacting the person that
invited them).
2019-08-23 12:53:11 -07:00
Rishi Gupta 2d260031ed emails: Use referrer.delivery_email in invitation emails. 2019-08-23 12:53:11 -07:00
Mohit Gupta e5482adec0 search: Add streams:public to search entire history of public streams.
Add ability to search entire message history of all public streams at
once. It includes all subscibed, non subscribed public streams messages
and even historical public stream messages sent before user had joined
an organization or stream.

Fixes #8859.
2019-08-22 13:40:49 -07:00
Rishi Gupta e1848e83bc integrations: Update instructions for AppFollow. 2019-08-22 13:17:06 -07:00
Rishi Gupta 02cfb47315 exports: Update wording on settings page, /help and /features. 2019-08-22 13:17:03 -07:00
Mateusz Mandera c437fd4344 test_users: Eliminate hard-coded gravatar urls.
Instead of having a hard-coded url, it seems better to replace it with
get_gravatar_url - which returns the correct url, without breaking if
the email/id of the example user changes.
2019-08-21 21:28:09 -07:00
Mateusz Mandera 55bf13db88 test_user_groups: Eliminate hard-coded user ids. 2019-08-21 21:28:09 -07:00
Mateusz Mandera 52d4583987 test_push_notifs: Eliminate hard-coded user ids. 2019-08-21 21:28:09 -07:00
Mateusz Mandera 0edc7d74ff test_events: Eliminate hard-coded user id. 2019-08-21 21:28:09 -07:00
Mateusz Mandera afc9650ebb test_email_change: Eliminate hard-coded user id. 2019-08-21 21:28:09 -07:00
Tim Abbott 7844f87d65 actions: Tag already has an account string for translation.
And also simplify an another validation string.

Fixed #9473.
2019-08-21 21:16:47 -07:00
Rohitt Vashishtha 400d0367dc tests: Improve logging for fixture tests in push_notifications. 2019-08-21 16:34:40 -07:00
Tim Abbott d0d16ed81c Revert "test_mattermost_importer: Fix missing coverage."
This reverts commit c48e53f587.

This new test code had nondeterministic failures; reverting so we can
debug without disruption.
2019-08-21 11:20:20 -07:00
Anders Kaseorg 5aeb562700 migrations.0001_initial: Expand \n in long SQL string.
One occasionally finds that a 1580 character string of SQL queries
might not most readably be presented on a single line.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-21 11:00:38 -07:00
Hemanth V. Alluri f375350446 page_params: Send the list of config_options for incoming webhook bots.
Send the config_options for each supported incoming webhook bot along
with the initial state (not present in apply_events since this is
mostly just static data).
2019-08-20 17:00:48 -07:00
Hemanth V. Alluri d73a37726d bots: Allow incoming webhook bots to be configured via /bots.
Without disturbing the flow of the existing code for configuring
embedded bots too much, we now use the config_options feature to
allow incoming webhook type bot to be configured via. the "/bots"
endpoint of the API.
2019-08-20 17:00:48 -07:00
Hemanth V. Alluri 94c351ead4 bot_config: Have check_valid_bot_config also take the bot_type.
This is a prep commit to allow us to validate user provided bot
config data using the same function for incoming webhook type
bots alongside embedded bots (as opposed to creating a new
function just for incoming webhook bots).
2019-08-20 16:44:56 -07:00
Hemanth V. Alluri f5c57361d6 bots: Specify config options for an incoming webhook integration.
In integrations.py we have a class called Integration which we then usually
subclass and then use to define the meta-data for all of our integrations.
Now, we want to allow all of our bots, specifically incoming webhook bots,
to be configured (i.e. we should let the user provide BotConfigData).

For this we create a new instance member of the Integration class called
config_options which will be a list of tuples containing the displayable
integration name, the configuration key form of the integration name and
the validator that it's value is supposed to adhere to.
2019-08-20 16:44:56 -07:00
Wyatt Hoodes c48e53f587 test_mattermost_importer: Fix missing coverage. 2019-08-20 15:23:41 -07:00
Mateusz Mandera 26f7dd09ce display_recipient: Eliminate raw_display_recipient from message dicts.
This was used as a helper to construct the final display_recipient when
fetching messages. With the new mechanism of constructing
display_recipient by fetching appropriate users/streams from the
database and cache, this shouldn't be needed anymore.
2019-08-20 12:22:52 -07:00
Tim Abbott b25abf3de9 display_recipient: Deduplicate list of display_recipient fields. 2019-08-20 12:22:37 -07:00
Mateusz Mandera d200cb921d display_recipient: Only fetch the necessary fields from the database.
There is no need to fetch the entire Stream or UserProfile objects, as
only several fields are needed. We use Django's .values() method to only
get what's needed.

For UserProfiles, it means that we get from the queries are dictionaries
already in the display_recipient form (UserDisplayRecipient type) - so
we can remove the user_profile_to_display_recipient_dict function, as
there's no need for this UserProfile -> UserDisplayRecipient conversion
anymore.
2019-08-20 12:15:30 -07:00
Mateusz Mandera 87ccb4f6c7 types: Consistently use DisplayRecipientT for display_recipient objects.
Instead of having the rather unclear type Union[str,
List[UserDisplayRecipient]] where display_recipient of message dicts was
involved, we use DisplayRecipientT (renamed from DisplayRecipientCacheT
- since there wasn't much reason to have the word Cache in there), which
makes it clearer what is the actual nature of the objects and gets rid
of this pretty big type declaration.
2019-08-20 12:15:30 -07:00
Mateusz Mandera 3ba0a37a92 types: Define UserDisplayRecipient type using TypedDict.
Since the display_recipients dictionaries corresponding to users are
always dictionaries with keys email, full_name, short_name, id,
is_mirror_dummy - instead of using the overly general Dict[str, Any]
type, we can define a UserDisplayRecipient type,
using an appropriate TypedDict.

The type definitions are moved from display_recipient.py to types.py, so
that they can be imported in models.py.

Appropriate type adjustments are made in various places in the code
where we operate on display_recipients.
2019-08-20 12:15:30 -07:00
Mateusz Mandera c779bb1959 messages: Don't use display_recipient values from cached message_dicts.
The user information in display_recipient in cached message_dicts
becomes outdated if the information is changed in any way.

In particular, since we don't have a way to find all the message
objects that might contain PMs after an organization toggles the
setting to hide user email addresses from other users, we had a
situation where client might see inaccurate cached data from before
the transition for a period of up to hours.

We address this by using our generic_bulk_cached_fetch toolchain to
ensure we always are fetching display_recipient data from the database
(and/or a special recipient_id -> display_recipient cache, which we
can flush easily).

Fixes #12818.
2019-08-20 12:15:30 -07:00
Pragati Agrawal 48efd46bc6 realm_logo: Remove redundant `realm_logo_url` function.
This function is nothing but a redundant layer of call to
`get_realm_logo_url`.
2019-08-20 12:07:20 -07:00
Pragati Agrawal 7326acdc2b realm_logo: Refactor `get_realm_logo_url` function. 2019-08-20 12:07:20 -07:00
Pragati Agrawal 1da5001875 realm_logo: Refactor/Deduplicate the code to sent the events.
This also creates a function to get source and url for logo.
2019-08-20 12:07:20 -07:00
Tim Abbott 69505c30a4 mattermost: Handle users who aren't on any team correctly.
It's not clear to me how this is intended to work in Mattermost's
system in that they don't document this behavior, but some users have
`null` as their list of teams, and presumably are not meant to be
included in any team at all.
2019-08-19 16:06:39 -07:00
Tim Abbott e02806ccc5 request: Fix other syntax error on Xenial Python. 2019-08-19 15:58:12 -07:00
Tim Abbott 0435556fd5 request: Fix syntax error on Xenial Python. 2019-08-19 15:47:40 -07:00
Tim Abbott 3781a0dae2 urls: Reformat how we handle API-only mobile endpoints.
This restructures the API endpoints that we currently have implemented
more or less for exclusive use by the mobile and desktop apps (things
like checking what authentication methods are supported) to use a
system that can be effectively parsed by our test_openapi
documentation.

This brings us close to being able to eliminate
`buggy_documentation_endpoints` as a persistently nonempty list.
2019-08-19 15:34:40 -07:00
Tim Abbott 27bfed7896 openapi: Remove presence from untestable endpoints.
This add some regular expression manipulation hacks to make it
possible for us to validate the documentation for the presence
endpoint with a slightly more complex regular expression capture
group.
2019-08-19 15:17:28 -07:00
Tim Abbott 44db59bbe0 openapi: Fix handling of parameters passed via the URL/path.
Previously, our OpenAPI documentation validation was failing for some
endpoints because it didn't account for the `in: path` type of
parameter, resulting in a mismatch between what was declared via REQ
and what was declared in the OpenAPI docs.

We fix this by excluding the path type parameters in both places from
what's considered by documentation using the `path_only` flag.

I doubt this is the correct long-term fix; in particular, I don't
think we're actually running the validators for these path-only
parameters.  The examples that exist today are all IDs with validators
for being non-negative numbers, but longer-term I think we'll want to
do something different (possibly at the REQ layer, see the TODO).
2019-08-19 15:06:08 -07:00
Hemanth V. Alluri 2e20dcc053 webhooks/jira: Update webhook to support improved comment events.
Atlassian announced that it will no longer provide information about
comments along with their "issue" type event payloads for Jira. So
we must now update the Jira integration to appropriately respond to
"comment" type events (the reason why we didn't do this before was
that initially, the "comment" type event payloads didn't contain
sufficient information about their issues, but this payload has
since been improved).

Note: This commit does *not* remove support for the older "issue"
type event payloads where information about comments was included.
This way we can maintain compatibility with old self-hosted versions
of self hosted Jira (2016 and before).

Source:
https://developer.atlassian.com/cloud/jira/platform/change-notice-
removal-of-comments-from-issue-webhooks/

Fixes #13012
2019-08-19 14:22:42 -07:00
Wyatt Hoodes 92dd5e2c4f test_queue_worker: Remove timing from test_slow_queries_worker.
Testing a specific query time was causing spurious failures
in CI.
2019-08-18 12:32:45 -07:00
okmanl 2a1305de9f lint: Add a rule to avoid msgid as a Python variable name.
This is for consistency with our usual patterns, see #12995.  We will
need a similar commit for JavaScript to complete #12995.
2019-08-17 12:47:13 -07:00
Wyatt Hoodes dc191b0be4 test_queue_worker: Test actual code path for slow_queries.
Instead of just mocking some fake events, we use the code
path that generates slow query events and publishes them
to SlowQueryWorker.

This test improvement would have got a recent potential regression
caught in code review.
2019-08-17 12:38:02 -07:00
Brutus5000 a386e3aebb github webhook: For release events show actions and tag name. 2019-08-17 12:35:16 -07:00
Hemanth V. Alluri 243d8ffc51 openapi: Add a test to ensure generated curl examples work.
This new test runs each generated curl example against the Zulip API,
checking whether it returns successfully without errors.

Significantly modified by tabbott for simplicity.
2019-08-17 11:35:08 -07:00
Tim Abbott 26ac3ebd3e openapi: Suggest the -sS options to curl.
These options prevent curl from doing the downloading progress bar
(which is clutter).
2019-08-17 11:35:08 -07:00
Tim Abbott 2366490ffc openapi: Pass api_url to curl example generation.
Our new curl example generation logic was broken, in that it hardcoded
localhost:9991 (without an HTTP method or anything) as the API URL.

It requires a bit of plumbing to make this possible.
2019-08-17 11:35:08 -07:00
Vishnu KS f32382f7b7 emails: Show preheader block only if preheader is present. 2019-08-17 11:32:28 -07:00
Priyank Patel 3680393b47 messages: Support passing user ID for stream operator.
ok_to_include_history fuction was updated to expect stream ID.

Fixes part of #9474.
2019-08-17 11:19:12 -07:00
Priyank Patel 5eab5bbc3e streams: Refactor can_access_stream_history.
This refactor extracts the code logic of checking if user can access
stream history into it's own function: can_access_stream_history
that takes in user_profile and stream. Then we make seperate function
can_access_stream_by_name that takes in stream_name and retrives stream
and pass it to can_access_stream_history. This will make it easily to later
add a function that does the same thing with stream ID.
2019-08-17 11:10:00 -07:00
Priyank Patel 0307b6d684 refactor: Make get_stream_name_from_narrow return stream.
This function was used only once in exclude_muting_conditions where
it returned stream name so the function to fetch stream id.
Since we exepect the narrow to also include stream id we refactor it to
return a stream object and since we use get_stream_by_narrow_operand_access_unchecked
we don't need to worry about handling cases where stream id is passed since
the function handles it.
2019-08-17 11:10:00 -07:00
Priyank Patel 463ecb375f refactor: Add get_stream_by_narrow_operand_access_unchecked.
This let's us clean up the linter that excludes the use of get_stream
and by adding the access_unchecked in the name we make it clear that
it should be used with caution.

Refactoring idea by Tim Abbott.
2019-08-17 11:10:00 -07:00
Priyank Patel 1f8f8867cd message_fetch: Rename handle_user_ids_supported_operators.
This renames handle_user_ids_supported_operators to
handle_operators_supporting_id_based_api.
2019-08-17 11:10:00 -07:00
Priyank Patel 1edde4a989 Rename user_id(s)_supported_operators -> operators_supporting_id(s). 2019-08-17 11:10:00 -07:00
Priyank Patel 99450f336c messages: Extract mypy typing for narrow list to variable.
Makes it obvious and readable and as an added bonus take up less space.

Note, some types used Iterable instead of List that were change
to used List since the narrow_paramter converter return a List.
2019-08-17 11:10:00 -07:00
Gloria Elston 092662d83e api: Use new endpoint for handling reactions. 2019-08-16 14:36:15 -07:00
Anders Kaseorg a5596011a0 queue_processors, python_examples: Fix mypy errors.
zerver/openapi/python_examples.py:105: error: Argument 1 to "get_user_presence" of "Client" has incompatible type "str"; expected "Dict[str, Any]"
    zerver/openapi/python_examples.py:563: error: Argument 1 to "add_reaction" of "Client" has incompatible type "Dict[str, object]"; expected "Dict[str, str]"
    zerver/openapi/python_examples.py:576: error: Argument 1 to "remove_reaction" of "Client" has incompatible type "Dict[str, object]"; expected "Dict[str, str]"
    zerver/worker/queue_processors.py:587: error: Argument "client" to "extract_query_without_mention" has incompatible type "EmbeddedBotHandler"; expected "ExternalBotHandler"

These were only missed because mypy daemon mode requires us to set
`follow_imports = skip` for the `zulip` package.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-16 14:13:40 -07:00
Mateusz Mandera c6b3d0212d models: Move some display_recipient code to display_recipient.py. 2019-08-15 17:15:32 -07:00
Mateusz Mandera cb2c9b04b3 generic_bulk_cached_fetch: Only call query_function if necessary. 2019-08-15 17:14:02 -07:00
Mateusz Mandera 898bc52538 models: Add bulk_get_huddle_user_ids function. 2019-08-15 17:03:17 -07:00
Mateusz Mandera f70a0bba82 display_recipient: Extract user_profile_to_display_recipient_dict function. 2019-08-15 16:59:28 -07:00
Tim Abbott 27a0e307b6 cache: Fix typing for generic_bulk_cached_fetch.
The typing for generic_bulk_cached_fetch is complicated, and was
recorded incorrectly previously for the case where a cache_transformer
function is required.  We fix this by adding the new CacheItemT, and
additionally add comments explaining what's going on with these types
for future reference.

Thanks to Mateusz Mandera for raising this issue.
2019-08-14 11:00:40 -07:00
Tim Abbott 2ada0a9bad management: Fix password reset emails being sent to deactivated users.
Apparently, the filters written for the send_password_reset_email (and
some other management commands) didn't correctly consider the case of
deactivated users.

While some commands, like syncing LDAP data (which can include whether
a user should be deactivated) want to process all users, other
commands generally only want to interact with active users.  We fix
this and add some tests.
2019-08-14 10:50:45 -07:00
Wyatt Hoodes 001ff72861 data exports: Add upload and message limit test. 2019-08-12 19:52:46 -07:00
Tim Abbott 8ac8247dbe data export: Add limits on exporting large organizations.
It seems possible that attempting to export large organizations could
result in high resource consumption that justifies having a technician
manage the exports manually.
2019-08-12 18:21:09 -07:00
Tim Abbott 1d63f129bf data export: Add summary logging of runtime.
This should help us investigate cases where this runs a very long
time.
2019-08-12 18:21:08 -07:00
Wyatt Hoodes 896886b412 realm_export: Add endpoint for export deletion. 2019-08-12 17:51:46 -07:00
Wyatt Hoodes 777dcdd272 test_realm_export: Clean up export testing.
* Added comments.
* Refactored `export_object` to `audit_log_entry`.
* Cleaned up white space.
* Removed uses of `getattr`.
2019-08-12 17:51:46 -07:00
Wyatt Hoodes 7c118096e7 test_events: Remove print statement from self.client_post.
Leftover from prior debugging.
2019-08-12 17:51:46 -07:00
Wyatt Hoodes 7d178bbb0f queue_processors: Clean up the extra_data dict code.
We don't want to add a `deleted_timestamp` key until
the export is actually deleted.
2019-08-12 17:51:46 -07:00
Wyatt Hoodes 6f6efa516d exports: Refactor extra_data to export_data. 2019-08-12 17:51:46 -07:00
Wyatt Hoodes 7a2a1f29ad exports: Refactor event_time to export_time timestamp.
The time of the event was incorrectly being sent
as a datetime object.
2019-08-12 17:51:46 -07:00
Anders Kaseorg e0cf3d0e76 zerver: Accept HEAD requests wherever GET requests are accepted.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-12 16:47:41 -07:00
Hemanth V. Alluri d425ab5ae5 test_templates: Add a verbose error messages for rendering failures.
If a template fails to render in test_templates because of a missing
context variable, we should better explain the reason for the failure.
2019-08-12 16:24:08 -07:00
Anders Kaseorg 72655611ce requirements: Use maintained fork django-sendfile2 of django-sendfile
The original seems to be unmaintained
(johnsensible/django-sendfile#65).  Notably, this fixes a bug in the
filename parameter, which perviously showed the Python 3 repr of a
byte string (johnsensible/django-sendfile#49).

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-12 15:40:08 -07:00
Anders Kaseorg 73330f3136 rest_dispatch: Remove invalid Content-Length: 0 from 204 responses.
There’s an apparent contradiction between RFC 7230 §3.3.2
Content-Length:

“A server MUST NOT send a Content-Length header field in any response
with a status code of 1xx (Informational) or 204 (No Content).”

and RFC 7231 §4.3.7 OPTIONS:

“A server MUST generate a Content-Length field with a value of "0" if
no payload body is to be sent in the response.”

The only resolution within the existing language would be to disallow
all 204 responses to OPTIONS requests.  However, I don’t think that
was the intention, so I submitted this erratum report:

https://www.rfc-editor.org/errata/eid5806

and updated the code accordingly.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-12 15:34:42 -07:00
Tim Abbott 7953e23d49 event_queue: Actually fix missing copy for edit-message events.
Apparently, our edit-message events did not guarantee that the outer
wrapper dictionary, which is intended to be unique for each client,
was unique for every client (instead only ensuring it was unique for
each user).

This led to clients unexpectedly getting last_event_id validation
errors in this code path when a user had multiple connected clients,
because the linear ordering of event IDs within a given queue was
corrupted.

In fd2a63b049, we accidentally fixed
this issue with a different set of userdata events, without fixing the
edit-message event bug.  This commit fixes the remaining issue.
2019-08-12 15:17:10 -07:00
Anders Kaseorg 7bf09067d1 event_queue: Clean up type ignores.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-09 17:42:33 -07:00
Anders Kaseorg 7cd11d0559 test_users: Clean up type ignores.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-09 17:42:33 -07:00
Anders Kaseorg 75a01810e3 test_outgoing_webhook_interfaces: Clean up type ignores.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-09 17:42:33 -07:00
Anders Kaseorg 1ffbc0009d test_openapi: Clean up type ignores.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-09 17:42:33 -07:00
Anders Kaseorg 9d33200fb2 test_messages: Clean up type ignores.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-09 17:42:33 -07:00
Anders Kaseorg 4671e4ca5c test_i18n: Clean up type ignores.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-09 17:42:33 -07:00
Anders Kaseorg c453c4b602 test_events: Clean up type ignores.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-09 17:42:33 -07:00
Anders Kaseorg d05578817e soft_deactivate_users: Clean up type ignores.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-09 17:42:33 -07:00
Anders Kaseorg 6d875dd211 topic: Clean up type ignores.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-09 17:42:33 -07:00
Priyank Patel 72bf7e2181 test_narrow: Rename stream_id -> stream_recipient_id.
It looks like this isn't actually a stream id, doing
get_stream_by_id_in_realm(stream_id, stream_messages[0].get_realm())
throws error.
2019-08-09 17:31:53 -07:00
Anders Kaseorg a0f4c99dad test_runner: Clean up type ignores.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-09 16:39:16 -07:00
Anders Kaseorg aa384c03a6 push_notifications: Clean up type ignores.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-09 16:39:16 -07:00
Anders Kaseorg 95658a2a54 logging_util: Clean up type ignores.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-09 16:39:16 -07:00
Anders Kaseorg df1fbbfee3 emoji: Clean up type ignores.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-09 16:39:16 -07:00
Anders Kaseorg bb5507009f camo: Clean up type ignores.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-09 16:39:16 -07:00
Anders Kaseorg 9b392917c5 addressee: Clean up type ignores.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-09 16:39:16 -07:00
Anders Kaseorg e417d3a040 slack_message_conversion: Clean up type ignores.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-09 16:39:16 -07:00
meiomorphism 1af72a2745 subscriptions: Avoid sending unneeded subscriber information.
The `users/me/subscriptions` endpoint accidentally started returning
subscriber information for each stream.  This is convenient, but
unnecessarily costly for those clients which either don't need it
(most API apps) or already acquire this information via /register
(including Zulip's apps).

This change removes that data set from the default response.  Clients
which had come to rely on it, or would like to rely on it in future,
may still access it via an additional documented API parameter.

Fixes #12917.
2019-08-09 16:03:21 -07:00
Tim Abbott fb01f2208d api: Verify that all registered OpenAPI examples are called.
This should help make this test suite robust against potential
mistakes when updating it.
2019-08-08 11:45:05 -07:00
Hemanth V. Alluri c2f9227892 openapi/python_examples: Replace TEST_FUNCTIONS with a decorator.
Use a decorator called openapi_test_function instead of the hard-coded
TEST_FUNCTIONS list for increased readability and maintainablity (at
the cost of performance).
2019-08-08 11:45:05 -07:00
Alexandra Ciobica e5e45c9a25 auth: Change page title and add description for the list.
I changed the class of the title in order to use the same styling as the
 other similar pages (like `/accounts/go` or `/login`).

Changed the related test.
2019-08-08 11:12:51 -07:00
Alexandra Ciobica f109dcce9c auth: Add logic for avatars to the GitHub auth email selection page.
For the emails that are associated to an existing account in an
organisation, the avatars will be displayed in the email selection
page.  This includes avatar data in what is passed to the page.

Added `avatar_urls` to the context in `test_templates.py`.
2019-08-08 11:12:51 -07:00
Alexandra Ciobica d4ccd73ae3 auth: Remove `@users.noreply.github.com` from the email selection list.
Apparently GitHub changed the email address for these; we need to
update our code accordingly.

One cannot receive emails on the username@users.noreply.github.com, so
if someone tries creating an account with this email address, that
person would not be able to verify the account.
2019-08-08 11:12:51 -07:00
Anders Kaseorg 7e92ff9d0a request: Delete request.pyi and make request.py type check.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-07 15:37:21 -07:00
Anders Kaseorg 13a2ac7b8e request: Remove ExtractRecipients type safety hole on REQ.
It was allowing us to get away with wrong types on a few functions:
`check_send_typing_notification` and `send_notification_backend` can be
(and are) called with a list of `int` as `notification_to`, not just a
list of `str`.

The problem it was working around already had a better solution using
the dummy `type` argument.  Use that.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-07 15:26:59 -07:00
Tim Abbott 4e631876f4 test_events: Fix realm export test to actually work.
The previous iteration still had the failure mode of not actually
testing anything, because it didn't trigger the data export code path
(and in fact was getting an HTTP 401 authentication denied error).
2019-08-07 14:18:27 -07:00
Wyatt Hoodes 22842dab34 events: Rename notify_export_completed.
notify_realm_export is more reasonable for the context of doing
deletion events as well.
2019-08-07 14:18:27 -07:00
Wyatt Hoodes 7853e9d2de test_events: Fix broken test case.
This test was broken due to using an empty `RealmAuditLog`
table.  We fix this by mocking the creation of an export,
thus creating an entry, similar to what we do in our other
tests.
2019-08-07 12:05:44 -07:00
Wyatt Hoodes f93efcb89c test_helpers: Refactor dummy file creation for broader use. 2019-08-07 12:05:44 -07:00
Wyatt Hoodes 11db0c23fb exports: Update extra_data field to a JSON structure.
We add the `deleted_timestamp` key to the new `extra_data`
dictionary.
2019-08-07 12:04:28 -07:00
Anders Kaseorg 0962393933 cleanup: Delete trailing newlines.
Delete trailing newlines from all files, except
tools/ci/success-http-headers.txt and tools/setup/dev-motd, where they
are significant, and static/third, where we want to stay close to
upstream.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-06 23:29:11 -07:00
Anders Kaseorg becef760bf cleanup: Delete leading newlines.
Previous cleanups (mostly the removals of Python __future__ imports)
were done in a way that introduced leading newlines.  Delete leading
newlines from all files, except static/assets/zulip-emoji/NOTICE,
which is a verbatim copy of the Apache 2.0 license.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-06 23:29:11 -07:00
neiljp (Neil Pilgrim) 12f3ed12c4 mypy: Refactor extract_recipients in actions.py to remove type ignore. 2019-08-06 23:24:56 -07:00
neiljp (Neil Pilgrim) 5ab64daecc mypy: Remove type ignore by defining ProfileDataElement using TypedDict. 2019-08-06 23:24:56 -07:00
neiljp (Neil Pilgrim) accf4411f0 mypy: Remove type ignore on MissedMessageWorker.stop_timer. 2019-08-06 23:24:56 -07:00
neiljp (Neil Pilgrim) 62a1c3443d mypy: Remove type ignores for python 3 stubs now present in typeshed.
Namely:
* posix stubs
* readfp stub
2019-08-06 23:24:56 -07:00
neiljp (Neil Pilgrim) 5f673f5820 mypy: Remove type ignores after boto stub improvements. 2019-08-06 23:24:56 -07:00
neiljp (Neil Pilgrim) ba54e20def mypy: Remove type ignore for merged typeshed PR 1762. 2019-08-06 23:24:56 -07:00
Tim Abbott 6b4300904a models: Add block comment explaining per-request cache. 2019-08-06 23:22:48 -07:00
Tim Abbott fd2a63b049 event_queue: Fix missing copy for edit-message events.
Apparently, our edit-message events did not guarantee that the outer
wrapper dictionary, which is intended to be unique for each client,
was unique for every client (instead only ensuring it was unique for
each user).

This led to clients unexpectedly getting last_event_id validation
errors in this code path when a user had multiple connected clients,
because the linear ordering of event IDs within a given queue was
corrupted.
2019-08-06 13:40:30 -07:00
Hemanth V. Alluri ac69717604 openapi: Migrate /messages to use the new cURL example system.
Now that we can create cURL examples based on the OpenAPI
documentation. We can begin using simple one line tags in
the documentation instead of manually creating cURL examples.

Fixes part of #12878.
2019-08-05 21:27:24 -07:00
Hemanth V. Alluri 5af753d940 api_code_examples: Parse kwargs/mods for example code generation.
Now we can also include extra keyword arguments to specify
modifications in how the example code should be generated
in the generate_code_example template tag.

E.g. generate_code_example(curl, exclude=["param1", "param2"])
2019-08-05 21:27:23 -07:00
Hemanth V. Alluri b20cf095e7 curl: Add code to auto generate cURL examples from OpenAPI docs.
This commit extends api_code_examples.py to support automatically
generating cURL examples from the OpenAPI documentation. This way
work won't have to be repeated and we can also drastically reduce
the chance of introducing faulty cURL examples (via. an automated
test which can now be easily created).
2019-08-05 21:13:14 -07:00
Hemanth V. Alluri f280e9cf84 lib: Rename lib/api_test_helpers.py to openapi/python_examples.py
This will make the contained code easier to find.
2019-08-05 21:06:19 -07:00
Hemanth V. Alluri 0682f4f12a openapi: Extend type checking to check the elements of lists.
Now that the type of the elements of each array/list are also
specified in the OpenAPI documentation, we can extend our type
checking even further.
2019-08-05 21:06:19 -07:00
Hemanth V. Alluri 748192d58c openapi: Replace add-subscriptions from TEST_FUNCTIONS.
Migrate the add-subscriptions line to the OpenAPI equivalent.
2019-08-05 21:06:19 -07:00
Hemanth V. Alluri 14c0a387cf openapi: Remove /users/me from pending endpoints.
This commit progress our efforts to reduce pending_endpoints
as well as to migrate away from templates/zerver/api/fixtures
and towards our OpenAPI documentation.

Similar to commit d62b75fc.
2019-08-05 21:06:19 -07:00
Rishi Gupta 2c80224e2d custom profile: Rename check_url_pattern.
validator.py has validators for the whole project, so we need a more
specific name.
2019-08-05 20:59:47 -07:00
Rishi Gupta b37c6a8644 custom profile: Update a few error messages. 2019-08-05 20:59:47 -07:00
Rishi Gupta 8167b5b645 email: Remove special integration doc behavior when gateway not set.
The current code looks like it's trying to redirect /integrations/doc/email
to /integrations when EMAIL_GATEWAY_PATTERN is not set.

I think it doesn't currently do this. The test for that pathway has a bug:
self.get_doc('integrations/doc-html/email', subdomain='zulip') needs a
leading slash, and putting the slash back in results in the test failing.

This redirection is not really desired behavior -- better is to
unconditionally show that the email integration exists, and just point the
user to https://zulip.readthedocs.io/en/latest/production/email-gateway.html
(this is done in a child commit).
2019-08-05 17:33:05 -07:00
Anders Kaseorg 68dd8e4ec8 mypy: Migrate from mypy_extensions to typing_extensions.
This gives us access to typing_extensions.Deque, which was not added
to typing until 3.5.4.

(PROVISION_VERSION is not bumped because the transitive dependency set
in dev.txt hasn’t changed.)

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-05 17:24:09 -07:00
Anders Kaseorg 86a7fdddd7 events: Check last_event_id for validity, take 2.
This verifies that the client passed a last_event_id that actually
came from the queue instead of making up an ID from the future.  It
turns out one of our tests was making up such an ID, but legitimate
clients are expected not to do so.

The previous version of this commit (commit
e00d4be6d5, #12888) had to be reverted
(commit b86c5cc490) because it was
missing the `to_dict`/`from_dict` migration code.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-05 17:18:49 -07:00
Tim Abbott 441bb844d2 models: Improve comments about get_user_profile_by_*_email.
This should help avoid folks accidentally using methods they should
not.
2019-08-05 17:10:58 -07:00
Tim Abbott ba67a9b76a message: Use RawUnreadMessageResult typing more.
This is a cleaner implementation given we have the TypedDict defined.
2019-08-02 17:34:27 -07:00
Tim Abbott 8f7a86f5ec events: Move remove_message_id_from_unread_mgs to message.py.
It belongs there, next to the related code for adding an event.
2019-08-02 17:26:13 -07:00
Tim Abbott 2c7511023d events: Simplify arguments for remove_message_id_from_unread_mgs.
This should make it reusable by the event_queue.py code path.
2019-08-02 17:26:13 -07:00
Rohitt Vashishtha a7f2bedb15 markdown: Enable hashheadings syntax.
Our implementation requires at least 1 space after the
'#' not not break existing linkifiers like '#123', etc.
that generally follow the convention we show in linkifier
examples.

- [valid]  : # Hello
- [valid]  : #  Hello
- [invalid]: #Hello

For the frontend, we have taken the code from v0.7.0 of
upstream marked and made minor changes to avoid having
to refactor a significant part of our marked code.

For the backend, we merely have to change the regex to
force require spaces after #, and add hashheader to our
list of blockparsers.

Fixes #11418.
2019-08-02 15:15:34 -07:00
Tim Abbott b86c5cc490 Revert "events: Check last_event_id for validity."
This isn't correct without a proper migration for existing queues,
which may not be implementable.

This reverts commit e00d4be6d5.
2019-08-02 14:44:35 -07:00
Tim Abbott 2738e909f8 event_queue: Expand testing of event queue save/restore.
This verifies that every valid state of our event queue system will be
properly save/restored via the from_dict methods.
2019-08-02 14:43:47 -07:00
Tim Abbott 3a6e5dad53 tests: Move EventQueueTest to test_event_queue.py.
This seems more appropriate, especially as we expand this library.
2019-08-02 14:43:12 -07:00
Tim Abbott 8be3df0e29 tornado: Fix bugs in Tornado autoreload library.
This fixes two issues:

* The syntax check logic we had for zerver.tornado.autoreload would
  end up clearing _reload_hooks if one of the files that had changed
  was zerver.tornado.autoreload itself (because we'd had re-imported
  the current module), which could be incredibly confusing when trying
  to test the autoreload logic.  It seems better to just not run the
  syntax check for syntax errors in this file.

  Similarly, because reloading event_queue.py would destroy the state
  in the queues, we avoid that as well.

* We make sure to flush stdout after running and reload hooks, to make
  sure their output reaches the user.
2019-08-02 12:47:49 -07:00
Tim Abbott a43b2f7a43 tornado: Fix incorrect import of autoreload library.
We were apparently not running our own forked Tornado autoreload
library when adding reload hooks, which meant that our autoreload
hooks didn't run at all.

This fixes an issue that made dump_event_queues never run and thus the
local development environment difficult to use for testing event queues.
2019-08-02 12:47:49 -07:00
Mateusz Mandera d3d62f99e4 tests: Adjust to new changes in populate_db. 2019-07-31 16:01:37 -07:00
Mateusz Mandera 9b2b24c617 production: Rename the 'zulip' system bot realm to 'zulipinternal'. 2019-07-31 16:01:36 -07:00
Tim Abbott e1d9fe84ad clubhouse: Type payload as Optional.
Apparently if you post json.dumps(None), one will end up with this
None case being possible.
2019-07-31 15:31:57 -07:00
Tim Abbott 78df15ef08 Revert "clubhouse: Remove dead code."
This reverts commit 48ea427e87.
2019-07-31 15:31:57 -07:00
Wyatt Hoodes b005c7567a messages: Fix typing errors.
We remove a redundant term conversion test, as well as
add appropriate typing for the `anchor` variable in
`get_messages_backend`.
2019-07-31 12:42:29 -07:00
Tim Abbott 48ea427e87 clubhouse: Remove dead code. 2019-07-31 12:40:01 -07:00
Tim Abbott 9599d311a5 logging_util: Add an extra type: ignore. 2019-07-31 12:40:01 -07:00
Wyatt Hoodes c787f0a550 webhooks/jira: Fix useless check for subject being None.
This wasn't possible since a recent rewrite of the webhook code.
2019-07-31 12:27:04 -07:00
Tim Abbott acd52be820 pivotal: Remove useless conditionals.
It isn't actually possible for these values to be None.
2019-07-31 12:26:21 -07:00
Wyatt Hoodes b807c4273e middleware: Fix exception typing.
Mypy seems to have trouble understanding `Exception` inheritance
here, so we create a `Union` for the only `Exception` we are
looking for.
2019-07-31 12:23:20 -07:00
Wyatt Hoodes 167d0bad61 test_runner: Unionize suite type.
Mypy cannot see the class hierarchy pertaining to django's
ParallelTestSuite. To deal with this, we simply create a
`Union` of our two types.
2019-07-31 12:20:12 -07:00
Wyatt Hoodes 4beec5c6b9 typing: Use TYPE_CHECKING when dealing with cyclic dependencies. 2019-07-31 12:19:39 -07:00
Wyatt Hoodes fa227d79af typing: Fix options typing in management/commands.
We simply state that certain options are `Optional`.

The following files are affected:

add_users_to_mailing_list
send_to_email_mirror
fill_memcached_caches
client_activity
2019-07-31 12:13:56 -07:00
Wyatt Hoodes 457842ccd6 add_users_to_mailing_list: Prepare for using Optional typing.
When typing `**options` as an `Optional[str]` we will see errors
in the from of `None type has no attribute 'split'`.  This change
allows mypy to effectively handle the `None` case.
2019-07-31 12:13:56 -07:00
Tim Abbott 997fd714d1 openapi: Fix some errors caught by online editor.
This fixes several values that had invalid types recorded.
2019-07-30 16:36:50 -07:00
Tim Abbott 782cb2e13c openapi: Remove duplicate announce section. 2019-07-30 15:08:59 -07:00
Tim Abbott 9827801569 slack import: Improve readability of user recipient object allocation.
This loop management tweak makes it a bit more obvious what's
happening in this block of code.
2019-07-30 14:46:14 -07:00
Vishnu KS ff3871fc63 slack_import: Clean up return values of channels_to_zerver_stream.
This commits reduces the number of values returned by
channel_to_zerver_stream function by setting the values
directly in realm dict and returning it instead.
2019-07-30 14:46:14 -07:00
Vishnu Ks 6110f495df slack_import: Support importing pms. 2019-07-30 14:46:14 -07:00
Mateusz Mandera 8f6f78b912 requirements: Upgrade redis from 2.10.6 to 3.2.1.
Fixes #11209.

This requires changing how zadd is used in rate_limiter.py:
In redis-py >= 3.0 the pairs to ZADD need to be passed as a dictionary,
not as *args or **kwargs, as described at
https://pypi.org/project/redis/3.2.1/ in the section
"Upgrading from redis-py 2.X to 3.0".

The rate_limiter change has to be in one commit with the redis upgrade,
because the dict format is not supported before redis-py 3.0.
2019-07-29 15:34:05 -07:00
Wyatt Hoodes ecb1b1a1e1 openapi: Fix typing for self.data. 2019-07-29 15:24:51 -07:00
Wyatt Hoodes b01ba48369 migration_0060: Remove unreachable return statement.
This was introduced in a past refactoring.
2019-07-29 15:23:34 -07:00
Wyatt Hoodes f22b2c7e9d archive: Fix typing for prev_sender variable.
We know that via the `AbstractMessage` class that `sender`
is of the type `UserProfile`.  We type this as `Optional`
to tell mypy that the operands to the right of the first
`or` can indeed be evaluated within the following `for` loop.
2019-07-29 15:23:10 -07:00
Wyatt Hoodes 99bd55917e zephyr: Fix typing for cred parameter. 2019-07-29 15:23:10 -07:00
Wyatt Hoodes 0834514668 auth: Fix typing for authenticate_remote_user. 2019-07-29 15:23:10 -07:00
Wyatt Hoodes 4149bac696 actions: Fix typing of query_filter. 2019-07-29 15:23:10 -07:00
Wyatt Hoodes 1706e06884 bugdown/init: Fix typing for fence variable. 2019-07-29 15:23:10 -07:00
Wyatt Hoodes 22481f63bf upload: Fix typing for key variable. 2019-07-29 15:23:10 -07:00
Wyatt Hoodes a75abe096b migration_0064: Fix typing for file_key. 2019-07-29 15:23:10 -07:00
Wyatt Hoodes e106f95c8b migration_0064: Remove usage of Text type. 2019-07-29 15:23:10 -07:00
Wyatt Hoodes 8107230b35 domains: Fix typing for validate_domain. 2019-07-29 15:18:24 -07:00
Wyatt Hoodes a109508e34 typing: Remove now-unnecessary conditional import.
As a result of dropping support for trusty, we can remove our old
pattern of putting `if False` before importing the typing module,
which was essential for Python 3.4 support, but not required and maybe
harmful on newer versions.

cron_file_helper
check_rabbitmq_consumers
hash_reqs
check_zephyr_mirror
check_personal_zephyr_mirrors
check_cron_file
zulip_tools
check_postgres_replication_lag
api_test_helpers
purge-old-deployments
setup_venv
node_cache
clean_venv_cache
clean_node_cache
clean_emoji_cache
pg_backup_and_purge
restore-backup
generate_secrets
zulip-ec2-configure-interfaces
diagnose
check_user_zephyr_mirror_liveness
2019-07-29 15:18:22 -07:00
Anders Kaseorg e00d4be6d5 events: Check last_event_id for validity.
This verifies that the client passed a last_event_id that actually
came from the queue instead of making up an ID from the future.  It
turns out one of our tests was making up such an ID, but legitimate
clients are expected not to do so.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-07-26 17:18:28 -07:00
Wyatt Hoodes bbbea9ec87 events: Rewrite system for managing realm exports.
This feature is intended to cover all of our ways of exporting a
realm, not just the initial "public export" feature, so we should name
things appropriately for that goal.

Additionally, we don't want to include data exports in page_params;
the original implementation was actually buggy and would have.
2019-07-26 16:38:52 -07:00
Wyatt Hoodes b1900c406a public_export: Add logic for deleting the export tarball.
The path to the uploaded tarball is reconstructed via the relative url and
removed with the canonical methods in `upload.py`.
2019-07-26 15:52:03 -07:00
Wyatt Hoodes ef02de4834 public_export: Add endpoint for returning all REALM_EXPORTED objects. 2019-07-26 15:52:02 -07:00
Wyatt Hoodes d070f27359 queue_processors: Change the extra_data field to a relative url path.
A better approach as compared to saving the full public url.
2019-07-26 15:50:02 -07:00
Wyatt Hoodes 266ed9ff75 public_export: Add acting_user to RealmAuditLog object.
An acting_user attribute is required to correctly reference who
initiated a realm_exported event.
2019-07-26 15:49:47 -07:00
Tim Abbott 1a7fd33ddb webhooks: Fix typo in a trello error message. 2019-07-26 15:12:45 -07:00
Hemanth V. Alluri ede73ee4cd streams: New stream messages should be auto marked as read by creator.
When a person creates a new realm, they'll likely want to create a
bunch of initial streams at once. When doing so, it could be annoying
to have to mark all of the new stream notification messages as read.

Thus to make this process smoother, we should automatically mark
the messages generated by the Notification Bot in the notifications
(announcements) stream, as well as in the newly created stream itself
as read by the stream creator.

Fixes #12765.
2019-07-26 15:10:09 -07:00
Hemanth V. Alluri 3528c454c0 openapi: Update tests to ensure that request parameter types match.
This commit add an pretty elaborate extension to the existing
openapi documentation validation test: test_openapi_arguments.

This does a metacode analysis, comparing the openapi documentation
with the appropriate function's declaration, default values etc.

While it has some limitations, it is able to catch various common
classes of mistakes in the types declared for our OpenAPI
documentation.
2019-07-26 12:49:59 -07:00
Tim Abbott 866b3566d4 api: Fix OpenAPI type declarations for events_register.
A few fields that are actually arrays were incorrectly declared as
strings.
2019-07-26 12:31:53 -07:00
Wyatt Hoodes b3323366b3 migration_0077: Migrate from Text to str.
In `force_str` we assume that python 2 strings should be
considered.  This is no longer the case, so we replace all
occurences of `Text` with `str`, and remove the unreachable
condition.

(Probably further cleanup is possible, but this code shouldn't be
modified again in any case).
2019-07-25 17:54:26 -07:00
Wyatt Hoodes 1c8106fc67 timezone: Fix return type in get_timezone.
While it's true `datetime` is implicit via `pytz`, it makes sense
that mypy should now complain about the semantics of calling our
return type `pytz.datetime.tzinfo`, when such a type doesn't
actually exist.
2019-07-25 17:41:01 -07:00
Hemanth V. Alluri 48526b1fd6 webhooks: Fix small bug in the fixture_to_headers method of bitbucket3.
We should return an empty dictionary instead of None.
2019-07-25 16:14:14 -07:00
Anders Kaseorg 0bcae0be55 write_log_line: Fix logging of 4xx error data.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-07-25 14:42:52 -07:00
Tim Abbott b8a1050fc4 rest: Disable caching for all REST API endpoints.
Investigation into #12876, a mysterious bug where users were seeing
messages reappear as unread, determined that the root cause was
missing headers to disable client-side caching for Zulip's REST API
endpoints.

This manifested, in particular, for `GET /messages`, which is
essentially the only API GET endpoint used by the webapp at all.  When
using the `Ctrl+Shift+T` feature of browsers to restore a recently
closed tab (and potentially other code paths), the browser would
return from its disk cache a cached copy of the GET /messages results.

Because we include message flags on messages fetched from the server,
this in particular meant that those tabs would get a stale version of
the unread flag for the batches of the most recent ~1200 messages that
Zulip fetches upon opening a new browser tab.

The issue took same care to reproduce as well, in large part because
the arguments to those initial GET /messages requests will vary as one
reads messages (because the `pointer` moves forward) and then enters
the "All messages" view; the disk cache is only used for GET requests
with the exact same URL parameters.

We will probably still want to merge the events error-handling changes
we had previously proposed for this, but the conclusion of this being
a straightforward case of missing cache-control headers is much more
satisfying than the "badly behaving Chrome" theory discussed in the
issue thread.

Fixes #12876.
2019-07-25 12:42:46 -07:00
Anders Kaseorg 2b33822de9 storage: Stop using django-pipeline.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-07-24 17:40:31 -07:00
Anders Kaseorg fd7803e7f4 settings: Unset STATIC_ROOT in development.
Django’s default FileSystemFinder disallows STATICFILES_DIRS from
containing STATIC_ROOT (by raising an ImproperlyConfigured exception),
because STATIC_ROOT is supposed to be the result of collecting all the
static files in the project, not one of the potentially many sources
of static files.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-07-24 17:40:31 -07:00
Anders Kaseorg a97a2612bb integrations: Get logo URLs from staticfiles.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-07-24 17:40:31 -07:00
Mateusz Mandera 1e107ef349 test_messages: Prepare for moving system bots to zulipinternal. 2019-07-24 16:44:16 -07:00
Mateusz Mandera 8c0e5c5fde test_push_notifs: Prepare for moving system bots to zulipinternal. 2019-07-24 16:44:16 -07:00
Mateusz Mandera 2e74d7b97f test_email_notifs: Prepare for moving system bots to zulipinternal. 2019-07-24 16:44:16 -07:00
Mateusz Mandera 6b20006aea test_bugdown: Prepare for moving system bots to zulipinternal. 2019-07-24 16:44:16 -07:00
Mateusz Mandera a953240b5e test_subs: Prepare for moving system bots to zulipinternal. 2019-07-24 16:44:15 -07:00
Mateusz Mandera 0198736e77 test_upload: Prepare for moving system bots to zulipinternal. 2019-07-24 16:26:10 -07:00
Tim Abbott daca742e9f email_gateway: Use html2text directly.
In the rare case that Zulip receives an email with only an HTML
format, we originally (code dating to 2013) shelled out to
html2markdown/python-html2text in order to convert the HTML into
markdown.

We long since added html2text as a reasonably managed Python
dependency of Zulip; we should just use it here.
2019-07-24 12:42:17 -07:00
Mateusz Mandera bf919fd354 test_management_command: Prepare for moving system bots to zulipinternal. 2019-07-23 15:37:37 -07:00
Mateusz Mandera 0e3b7aab67 test_home: Prepare for moving system bots to zulipinternal. 2019-07-23 15:30:30 -07:00
Mateusz Mandera 11862e5ce0 test_classes: Use subdomain kwarg in api_* functions instead of realm.
It's more appropriate for the kwarg to be named subdomain. We also
update the functions, so that this kwarg is used in all of them
consistently.
2019-07-23 15:05:39 -07:00
Mateusz Mandera 7f24e27610 test_classes: Rename first argument of api_* functions to "identifier".
This was the name in api_post already - we rename it in the other
functions, for consistency.
2019-07-23 15:05:39 -07:00
Eeshan Garg 1579dc2fee webhooks/github: Add support for star events. 2019-07-22 18:20:53 -07:00
Eeshan Garg badaf28f21 webhooks/github: Ignore 'meta' events.
This event occurs when its corresponding webhook is deleted. This
isn't very useful information, so we should ignore it.
2019-07-22 18:20:53 -07:00
Eeshan Garg 8e10ab282a webhooks: Log unexpected webhook events separately.
This change serves to declutter webhook-errors.log, which is
filled with too many UnexpectedWebhookEventType exceptions.

Keeping UnexpectedWebhookEventType in zerver/lib/webhooks/common.py
led to a cyclic import when we tried to import the exception in
zerver/decorators.py, so this commit also moves this exception to
another appropriate module. Note that our webhooks still import
this exception via zerver/lib/webhooks/common.py.
2019-07-22 18:20:53 -07:00
Conner Bondurant 4d4b6b8319 email: Convert send_email to use formataddr.
Changed the requirements for UserProfile in order to allow use of
the formataddr function in send_mail.py.

Converted send_email to use formataddr in conjunction with the commit
that strengthened requirements for full_name, such that they can now be
used in the to field of emails.

Fixes #4676.
2019-07-22 18:15:58 -07:00
Conner Bondurant c25dcf048d models: Enforce stricter requirements on the full_name field.
This changes the requirements for UserProfile to disallow some
additional characters, with the overall goal of being able to use
formataddr in send_mail.py.

We don't need to be particularly careful in the database migration,
because user full_names are not required to be unique.
2019-07-22 18:13:34 -07:00
Tim Abbott 47d80977ac plans: Make usable in the development environment for testing. 2019-07-22 17:43:43 -07:00
neiljp (Neil Pilgrim) ba7a0934e3 requirements: Upgrade mypy to 0.711.
This comes with it a big performance improvement; mypy is now only
barely our slowest linter even if it wasn't previously running.

Fixes: #12058
2019-07-22 17:12:50 -07:00
neiljp (Neil Pilgrim) 80a4504784 mypy: Clean up typing for CustomProfileField code. 2019-07-22 17:10:50 -07:00
Wyatt Hoodes e4133e687f test_runner: Ignore inheritance issues in TestSuite.
Again, `result` of type `TestResult` is considered by mypy to
not have these _attributes acquired through inheritance. So,
we ignore them.
2019-07-22 17:06:41 -07:00
Wyatt Hoodes 5686821150 middleware: Change write_log_line to publish as a dict.
We were seeing errors when pubishing typical events in the form of
`Dict[str, Any]` as the expected type to be a `Union`.  So we instead
change the only non-dictionary call, to pass a dict instead of `str`.
2019-07-22 17:06:41 -07:00
Wyatt Hoodes dd19e2df1b test_runner: Ignore self.stream in TextTestResult class.
Per the import line:

`from unittest import loader, runner  # type: ignore  # Mypy cannot pick
these up.`

Because `TextTestResult` inherits from `runner.TextTestResult`, mypy
doesn't see `self` as having an attribute `stream`, so we ignore these
instead of cluttering with `casts` or `isinstances`.
2019-07-22 16:45:21 -07:00
Tim Abbott d3f59db8ce test_runner: Cleanup typing for TestResult objects.
There's two possible types here.
2019-07-22 16:45:21 -07:00
Wyatt Hoodes 76e8dc1657 test_runner: Change type to unittest.TestSuite.
In `deserialize_suite`, it passes a test as being of the
super type, where just having TestSuite assumes the type of
`zerver.Runner.TestSuite`.
2019-07-22 16:27:39 -07:00
Wyatt Hoodes 3509fe07b5 test_runner: Fix incorrect type for enforce_timely_test_completion.
Our TestResult objects are always actually TextTestResults.
2019-07-22 16:27:39 -07:00
Wyatt Hoodes a2fa1a6f25 handlers: Remove duplicate type annotation.
`self._request_middleware` is already typed in the `__init__` method.
2019-07-22 16:27:39 -07:00
Wyatt Hoodes 97fbcf020f test_tornado: Remove duplicate type annotation.
`self.session_cookie` is already typed in the `setUp` method.
2019-07-22 16:27:39 -07:00
Mateusz Mandera a50524762c populate_db: Refactor message.pub_date generating code.
The code generating pub_dates for messages would fail to distribute them
across days if tot_messages was too large.
We refactor this code as a separate function (for clarity and to unit
test for the bug we're fixing), and change the structure and naming to a
form that more clearly describes what's happening. We also shift away
from the approach of all the float-to-int conversions as this is in
general tricky and bug prone - django's timedelta() handles floats as
arguments, so we take advantage of that.
2019-07-22 15:55:28 -07:00
Mateusz Mandera be0a024355 integration docs: Document new email integration behaviors.
Tweaked by tabbott for cleaner, more precise English.
2019-07-22 15:52:08 -07:00
Hemanth V. Alluri fecf6a529e webhooks: Migrate gitlab tests to the headers fixtures system.
We use the same approach as was used for the GitHub integration.
2019-07-22 12:12:50 -07:00
Tim Abbott 02d55928ea import: Fix importing slack avatars into S3_UPLOAD_BACKEND.
Apparently, a subtle mismatch between the filename/URL formats for our
upload codebases meant that importing Slack avatars into systems using
S3_UPLOAD_BACKEND would end up with the avatars having the wrong URLs.
2019-07-21 21:25:31 -07:00
Harshit Bansal bf14a0af4d auth: Migrate google auth to python-social-auth.
This replaces the two custom Google authentication backends originally
written in 2012 with using the shared python-social-auth codebase that
we already use for the GitHub authentication backend.  These are:

* GoogleMobileOauth2Backend, the ancient code path for mobile
  authentication last used by the EOL original Zulip Android app.

* The `finish_google_oauth2` code path in zerver/views/auth.py, which
  was the webapp (and modern mobile app) Google authentication code
  path.

This change doesn't fix any known bugs; its main benefit is that we
get to remove hundreds of lines of security-sensitive semi-duplicated
code, replacing it with a widely trusted, high quality third-party
library.
2019-07-21 20:51:34 -07:00
Tim Abbott 15da425c54 auth: Fix camel case name for register_extra_endpoints. 2019-07-21 19:31:13 -07:00
Tim Abbott 95a1827db0 auth: Move GitHub auth tests out of SocialAuthBase.
During the time between when we refactored the GitHub authentication
backend to use SocialAuthBase and now (when we're about to migrate
GoogleAuthBackend to use that code path as well), we accidentally
added some GitHub-specific authentication backend tests to the common
test class.

Fix this by moving them to the GitHub-specific subclass.
2019-07-21 19:26:47 -07:00
Hemanth V. Alluri 1b8009837b openapi: Rename variable function to function_name in validation test.
This is a prep commit for adding validation of the request variable
types since then we would need to actually analyze the code of the
actual function itself and we would need a variable storing the
function itself.
2019-07-21 18:46:48 -07:00
Hemanth V. Alluri d62b75fc4c openapi: Remove /users/me/subscriptions from pending_endpoints.
In commit 7c71e98, we added a special exception for the
/users/me/subscriptions endpoint in the automatic validation test.
By adding some extra documentation, we now remove this extra code,
as well as the endpoint from the list of pending endpoints.
2019-07-21 18:39:53 -07:00
Hemanth V. Alluri ff73c45fa3 openapi: Improve exception message for pending_endpoint with docs.
In the validation test, we now use a different message for when there
is an endpoint in pending_endpoints with some documentation already.
This change is a bit hackish, but it's okay since we'll be removing it
once we've resolved all pending endpoints (which is bound to happen).
2019-07-21 18:39:53 -07:00
Rishi Gupta 1a6f8cdf17 emails: Remove unsubscribe link from missed message emails.
This allows the footer to be two lines long instead of three.
2019-07-21 15:10:48 -07:00
Tim Abbott 21c8a7ec36 i18n: Fix strings in send_pm_if_empty_stream.
The previous iteration did not properly handle languages with a
different word order than English.

Discovered via warning output in `manage.py makemessages`.
2019-07-21 14:43:50 -07:00
Rishi Gupta 7d8d0b2284 settings: Update upgrade text and styling.
When we add Plus, the first sentence should change to "Available on Zulip
Standard and Plus".

I copied the styling of .tip out of expediency, but it's also possible that
long term we'll want only 1 tip-like box styling.

The hover styling is a bit random, but I tried to copy other hover styles I
found in settings.scss.

Note that this renames .upgrade_realm_plan_type_suggestion to .upgrade-tip.
2019-07-21 14:32:36 -07:00
Mateusz Mandera f1b135bd16 email_mirror: Rename include-quotations to include-quotes. 2019-07-20 15:53:43 -07:00
Mateusz Mandera 519ed41803 email_mirror: Display stream email address with . as the separator. 2019-07-20 15:53:43 -07:00
Mateusz Mandera 8bbd1d005d email_mirror: Support both + and . as separators in the address. 2019-07-20 15:53:42 -07:00
Wyatt Hoodes e331a758c3 python: Migrate open statements to use with.
This is low priority, but it's nice to be consistently using the best
practice pattern.

Fixes: #12419.
2019-07-20 15:48:52 -07:00
Tim Abbott e97179fc87 test_docs: Improve error handling for API doc tests. 2019-07-20 15:15:10 -07:00
Mateusz Mandera 4646c7550c test_retention: Prepare for moving system bots to zulipinternal. 2019-07-20 15:08:08 -07:00
Mateusz Mandera 76883d5da9 test_audit_log: Prepare for moving system bots to zulipinternal. 2019-07-20 15:08:06 -07:00
Mateusz Mandera 0299d6f5ef test_new_users: Prepare for moving system bots to zulipinternal. 2019-07-20 14:55:22 -07:00