Commit Graph

11560 Commits

Author SHA1 Message Date
Steve Howell d5cc29755e typing: Accept only user_ids in typing endpoint.
The only clients that should use the typing
indicators endpoint are our internal clients,
and they should send a JSON-formatted list
of user_ids.

We now enforce this, which removes some
complexity surrounding legacy ways of sending
users, such as emails and comma-delimited
strings of user_ids.

There may be a very tiny number of mobile
clients that still use the old emails API.

This won't have any user-facing effect on
the mobile users themselves, but if you type
a message to your friend on an old mobile
app, the friend will no longer see typing
indicators.

Also, the mobile team may see some errors
in their Sentry logs from the server rejecting
posts from the old mobile clients.

The error messages we report here are a bit
more generic, since we now just use REQ
to do validation with this code:

    validator=check_list(check_int)

This also allows us to remove a test hack
related to the API documentation.  (We changed
the docs to reflect the modern API in an
earlier commit, but the tests couldn't be
fixed while we still had the more complex
semantics for the "to" parameter.)
2020-06-10 17:50:32 -07:00
sahil839 81c28c1d3e realm: Allow only organization owners to deactivate a realm.
We now allow only organization owners to deactivate a realm.

'require_realm_owner' decorator has been added for this purpose.
2020-06-10 17:33:02 -07:00
sahil839 86b52ef7bf users: Owners can only be deactivated by other organization owners.
This commit adds restriction on deactivation of organization owners.
Only owners can deactivate other organization owners.
2020-06-10 17:33:02 -07:00
orientor e989c87a18 openapi: Fix response validation bug.
Currently there are no checks in validate_against_openapi_schema
to check whether the `content` it received actually matched with
the `response code`. For example during testing if a certain endpoint
was returning 400 but it was expected to return 200, then it would
pass schema validation as it would only have `msg` and `result` keys.
Add this validation and fix the wrong response returning points.
2020-06-10 16:45:36 -07:00
orientor 03ef5e0b31 openapi: Add response example validation.
Add test to validate example responses in zulip.yaml. Also change
zulip.yaml for some wrong examples or for cases which were not
covered by `test-api`. Also enhance `validate_against_openapi_schema`.
2020-06-10 15:45:50 -07:00
Anders Kaseorg 67e7a3631d python: Convert percent formatting to Python 3.6 f-strings.
Generated by pyupgrade --py36-plus.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-10 15:02:09 -07:00
Anders Kaseorg 6480deaf27 python: Convert more "".format to Python 3.6 f-strings.
Generated by pyupgrade --py36-plus --keep-percent-format, with more
restrictions patched out.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-10 14:48:09 -07:00
Hemanth V. Alluri 7b2255597c outgoing-webhook: Fix check for empty content.
Prevent `JsonableError(_("Missing content"))` from
ever being triggered.

That error wasn't handle by anything, and thus just threw a 500, as
it's not a response to an HTTP request.

The right fix is to adjust the caller to ban the empty string in
content (or content that strips to the empty string).

Closes #15145.
2020-06-10 14:46:04 -07:00
Dinesh 0c45e0118f tests: Clear remove/remock approach for ACCESS_TOKEN_URL endpoint.
ACCESS_TOKEN_URL works a different for apple authentication, so,
we removed and remocked the ACCESS_TOKEN_URL mock in
`register_extra_endpoints` override of apple auth test class.
It is cleaner to have it as generic feature of `social_auth_test`.

So, this commit adds a function that returns token_data_dict that
we had earlier and is called in the ACCESS_TOKEN_URL mock.
This function is overriden in apple auth test class to generate
payload of the format that apple auth expects.

Thanks to Mateusz Mandera for the simple idea.

Co-authored-by: Mateusz Mandera <mateusz.mandera@zulip.com>
2020-06-10 14:38:49 -07:00
Tim Abbott 94d0c330e4 management: Display organization owners in show_admins. 2020-06-10 14:07:38 -07:00
sahil839 22a188f12e migration: Add migration to set all realm admins as realm owners.
This migration is added to set all the realm admins as realm owners
and also creates RealmAuditLog entries for modified users.
2020-06-10 14:07:38 -07:00
sahil839 5cc937f670 events: Add 'is_owner' to page_params and raw_users dict of page_params.
This commit adds 'is_owner' field to page_params and also to the
raw_users dict of page_params.
2020-06-10 14:07:38 -07:00
sahil839 6d667dbe53 realm owner: Add ability to change realm owner status of user.
This commit adds some basic checks while adding or removing
realm owner status of a user and adds code to change owner
status of a user using update_user_backend.

This also adds restriction on removing owner status of the
last owner of realm. This restriction was previously on
revoking admin status, but as we have added a more privileged
role of realm owner, we now have this restriction on owner
instead of admin.

We need to apply that restriction both in the role change code path
and the deactivate code path.
2020-06-10 12:49:37 -07:00
sahil839 2c8c641556 realm owner: Make first user as realm owner on realm creation.
This commit sets the role of the user creating the realm as
realm owner after the realm is created.

Previously, the role of user creating the realm was set as admin.
But now we want it to be owner because owners have the highest
privilege level.
2020-06-10 12:44:52 -07:00
Anders Kaseorg d3e8af4ad2 python: Replace silly uses of filter().
The test_management_commands use in particular was causing pickling
errors when the test failed, because Python 3 filter returns an
iterator, not a list.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-09 23:31:59 -07:00
Anders Kaseorg 5c9d56d2f7 get_safe_redirect_to: Remove incorrect allowed_hosts setting.
set(redirect_host) is a set of characters, so the only non-relative
redirects being allowed were to certain one-character hostnames, which
certainly isn’t what was intended.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-09 22:51:52 -07:00
Anders Kaseorg 9824405842 dev_settings: Set ZULIP_ADMINISTRATOR.
Fixes this error in the dev environment:

$ ./manage.py checkconfig
Error: You must set ZULIP_ADMINISTRATOR in /etc/zulip/settings.py.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-09 22:15:34 -07:00
Anders Kaseorg 04919528e7 models: Parameterize .extra(where=["… IN …"]) with tuple/list adapter.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-09 21:12:43 -07:00
Anders Kaseorg de22f7a378 migrations: Use cursor.execute correctly.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-09 21:12:43 -07:00
Anders Kaseorg 859c8b169e stream_recipient: Use cursor.execute correctly.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-09 21:12:43 -07:00
Anders Kaseorg 68e78c809f get_recent_private_conversations: Use cursor.execute correctly.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-09 21:12:43 -07:00
Anders Kaseorg 3aab9c03a9 fix_unreads: Use cursor.execute correctly.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-09 21:12:43 -07:00
Anders Kaseorg 14bbfe6ffb bulk_get_subscriber_user_ids: Use cursor.execute correctly.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-09 21:12:43 -07:00
Anders Kaseorg 2604ebba38 import_message_data: Use psycopg2.extras.execute_values.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-09 21:12:43 -07:00
Anders Kaseorg 6649c8e35f bulk_insert_ums: Use psycopg2.extras.execute_values.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-09 21:12:43 -07:00
Dinesh dc90d54b08 auth: Add Sign in with Apple support.
This implementation overrides some of PSA's internal backend
functions to handle `state` value with redis as the standard
way doesn't work because of apple sending required details
in the form of POST request.

Includes a mixin test class that'll be useful for testing
Native auth flow.

Thanks to Mateusz Mandera for the idea of using redis and
other important work on this.

Documentation rewritten by tabbott.

Co-authored-by: Mateusz Mandera <mateusz.mandera@zulip.com>
2020-06-09 17:29:35 -07:00
David Wood f0f42f7a94 integrations: Add Thinkst Canary integration.
This commit adds an integration for Thinkst Canaries - physical, VM and
cloud-based canaries for detecting attackers to a network. Thinkst
Canaries can send webhook alerts when canaries have been tripped, and
this integration will post Zulip messages when these webhooks are
received.

Signed-off-by: David Wood <david@davidtw.co>
2020-06-09 16:28:28 -07:00
Tim Abbott e242ddc848 page_params: Add community_topic_editing_limit_seconds.
This was previously hardcoded with agreement between the Zulip backend
and frontend as 86400 seconds (1 day).  Now, it's still hardcoded in
the backend, but arranged in a way where we could add a setting
without any changes to the mobile and terminal apps to update logic.

Fixes #15278.
2020-06-09 14:40:12 -07:00
Tim Abbott cb8290ac57 test_home: Stop checking for removed string. 2020-06-08 23:09:21 -07:00
Tim Abbott 2beaf2cab2 pointer: Remove have_initial_messages code.
This was implemented in 2012 to avoid showing a loading indicator for
fetching messages for users with no message history.  However, the
Zulip onboarding UI always creates some message history, and fetching
history is fast, so this is likely clutter more than a useful
optimization.
2020-06-08 22:36:18 -07:00
Anders Kaseorg 47b4e45931 markdown_test_cases: Update encoded zulipchat.com links too.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-08 19:47:07 -07:00
Tim Abbott 7b8ba5ebd9 docs: Update most remaining references to zulipchat.com.
In some cases, the cleanest tweak is to replace references to the
domain with Zulip Cloud, the product.
2020-06-08 18:10:45 -07:00
Tim Abbott 95be7dcbab portico: Remove mystery hunt landing page.
This was written by Rishi for a very brief purpose a few years ago,
and it doesn't serve much purpose now other than to be a place we
update in code sweeps.
2020-06-08 18:10:45 -07:00
Tim Abbott 71078adc50 docs: Update URLs to use https://zulip.com.
We're migrating to using the cleaner zulip.com domain, which involves
changing all of our links from ReadTheDocs and other places to point
to the cleaner URL.
2020-06-08 18:10:45 -07:00
Dinesh d30f11888a logging: Set up a different logger for each backend.
Adds a top-level logger in `settings.LOGGING` `zulip.auth`
with the default handlers `DEFAULT_ZULIP_HANDLERS` and
an extra hanlder that writes to `/var/log/zulip/auth.log`.

Each auth backend uses it's own logger, `self.logger` which
is in form 'zulip.auth.<backend name>'.

This way it's clear which auth backend generated the log
and is easier to look for all authentication logs in one file.

Besides the above mentioned changes, `name` attribute is added to
`ZulipAuthMixin` so that these logging kind of calls wouldn't raise
any issues when logging is tried in a class without `name` attribute.

Also in the tests we use a new way to check if logger calls are made
i.e. we use `assertLogs` to test if something is logged.

Thanks to Mateusz Mandera for the idea of having a seperate logger
for auth backends and suggestion of using `assertLogs`.
2020-06-08 17:42:07 -07:00
Anders Kaseorg b1b34b6f6a dev_login: Move owners to the top.
Since owners have the highest privilege level, it made little sense to
sandwich them between administrators and guests.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-08 17:26:34 -07:00
sahil839 7de23b8b5c users: Remove short_name and client_id from get_profile_backend.
This commit removes short_name and client_id fields from the user
objects returned by get_profile_backend because neither of them
had a purpose.

* short_name hasn't been present anywhere else in the Zulip API for
  several years, and isn't set through any coherent algorithm.
* client_id was a forgotten 2013-era predecessor to the queue_id field
  returned by the register_event_queue process.

The combination of these changes gets us close to having `get_profile`
have the exact same format as other endpoints fetching a user object.
2020-06-08 17:01:08 -07:00
sahil839 1f5778bad7 users: Refactor get_profile_backend to be based on format_user_row.
This commit changes get_profile_backend to be based on format_user_row
such that it's a superset of the fields for our other endpoints for
getting data on a user.

To be clear, this does not removes any of the exisiting fields, that
were returned by this endpoint.

This change adds some fields to the User object returned by the
endpoint. API docs are updated accordingly for the added fields.
2020-06-08 16:57:44 -07:00
sahil839 2f7d684a84 slack_import: Map slack owners to zulip realm owners.
Slack owners and primary owners will be mapped to zulip
realm owners on import.

Previously, we mapped the owner and primary owner roles of slack
to realm admins in zulip. As we have added ROLE_REALM_OWNER in
8bbc074, we now map slack owners and primary owners to owners in
zulip.

Tests are modified for checking all the 3 cases-
 - Slack workspace primary owner
 - Slack workspace owner
 - Slack workspace admin

This commit also has docs changes in 'import-from-slack.md'.
2020-06-08 16:22:54 -07:00
Anders Kaseorg 8e4f22c184 auth: Require algorithms setting for JWT auth.
Calling jwt.decode without an algorithms list raises a
DeprecationWarning.  This is for protecting against
symmetric/asymmetric key confusion attacks.

This is a backwards-incompatible configuration change.

Fixes #15207.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-08 16:22:25 -07:00
Anders Kaseorg 8dd83228e7 python: Convert "".format to Python 3.6 f-strings.
Generated by pyupgrade --py36-plus --keep-percent-format, but with the
NamedTuple changes reverted (see commit
ba7906a3c6, #15132).

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-08 15:31:20 -07:00
Anders Kaseorg 8c15081069 python: Further pyupgrade changes.
Generated by pyupgrade --py3-plus --keep-percent-format.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-08 15:23:10 -07:00
Anders Kaseorg 444fbbf964 python: Whitespace fixes from autopep8.
Generated by autopep8.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-08 15:21:30 -07:00
Anders Kaseorg 549fbeac72 test_narrow: Convert a triple-quoted format string to single-quoted.
This works around a semgrep bug following the conversion to f-strings.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-08 15:19:59 -07:00
Anders Kaseorg 139cb8026f auth: Accept next as POST parameter in POST requests.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-08 11:07:32 -07:00
Johannes Hochreiter d78d9099d1 gitlab: added pipeline number to event notification
- link to pipeline now also features the pipeline number
- updated tests accordingly
2020-06-07 14:01:04 -07:00
Johannes Hochreiter 60077f3097 gitlab: Added artifact link to pipeline event body
- if artifact was built file name, download link and browse link will be added to pipeline event body
- added test for pipeline with built artifacts
2020-06-07 14:01:04 -07:00
Aman Agrawal e2ed155310 do_update_message: Send old topic with stream edit event.
Old topic of the msg edit event can be used to help the client
calculate useful information such as if a change
in current narrow is required.

This fixes our re narrow logic after a stream edit of a topic, with
no change in topic name itself, since the original topic was not
present in the event received and hence the `orig_topic` was
undefined in this case.
2020-06-07 11:20:34 -07:00
sahil839 c394337118 tests: Refactor 'attempt_unsubscribe_of_principal` to accept list of users.
This adds 'target_users' parameter to 'attempt_unsubscribe_of_principal`
function in test_subs.py, which accepts list of UserProfile objects to be
unsubscribed, instead of defining users in the function itself.

This change makes the code cleaner and more readable.

Also, 'other_user_subbed' parameter is changed to 'target_users_subbed'
to clearly depict the use of this parameter.
2020-06-07 11:18:37 -07:00
Anders Kaseorg 19cc22e5ab queue: Fix types to reflect that Pika channels transmit bytes.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-07 11:09:24 -07:00