Fixes#14828.
Giving the /subdomain/<token>/ url there could feel buggy if the user
ended up using the token in the desktop app, and then tried clicking the
"continue in browser" link - which had the same token that would now be
expired. It's sufficient to simply link to /login/ instead.
This adds support for a "spoiler" syntax in Zulip's markdown, which
can be used to hide content that one doesn't want to be immediately
visible without a click.
We use our own spoiler block syntax inspired by Zulip's existing quote
and math block markdown extensions, rather than requiring a token on
every line, as is present in some other markdown spoiler
implementations.
Fixes#5802.
Co-authored-by: Dylan Nugent <dylnuge@gmail.com>
This adds a new function `get_apns_badge_count()` to
fetch count value for a user push notification and
then sends that value with the APNs payload.
Once a message is read from the web app, the count is
decremented accordingly and a push notification with
`event: remove` is sent to the iOS clients.
Fixes#10271.
Mocking `get_base_payload()` verifies the wrong output
when the code is actually correct. So, its better that
we call the real function here, especially when we are
adding the Apple case.
This line was effectively hardcoding a specific stream_post_policy,
overriding the value already present in the event, to no purpose.
(I believe it got here via cargo-culting induced by #13787.)
This commit removes is_old_stream property from the stream objects
returned by the API. This property was unnecessary and is essentially
equivalent to 'stream_weekly_traffic != null'.
We compute sub.is_old_stream in stream_data.update_calculated_fields
in frontend code and it is used to check whether we have a non-null
stream_weekly_traffic or not.
Fixes#15181.
This likely fix a bug that can leak thousands of messages into the
invalid state where:
* user_message.flags.active_mobile_push_notification is True
* user_message.flags.read is True
which is intended to be impossible except during the transient process
between marking messages as read sending the "remove push
notifications" event.
The bug is that if a user who is declaring bankruptcy with 10,000s of
unreads ends up having the database query to mark all of those as read
take 60s, the Django/uwsgi request will time out and kill the process.
If the postgres transaction still completes, we'll end up with the
second half of this function never being run.
A safer ordering is to do the smaller queries first.
We do this in a loop for correctness in the unlikely event there are
more than 10,000 of these.
Fixes#15285
This event will be used more now for guest users when moving
topic between streams (See #15277). So, instead of deleting
messages in the topic as part of different events which is
very slow and a bad UX, we now handle the messages to delete in
bulk which is a much better UX.
After merging PR #15355 we found that CircleCI may send
super minimal payloads. This does not seem to depend on
.circleci/config.yml since we received two different
types of payloads off of the same configuration.
Signed-off-by: Hemanth V. Alluri <hdrive1999@gmail.com>
This is designed to have no user-facing change unless the client
declares bulk_message_deletion in its client_capabilities.
Clients that do so will receive a single bulk event for bulk deletions
of messages within a single conversation (topic or PM thread).
Backend implementation of #15285.
This commits adds restriction on admins to set message retention policy.
We now only allow only organization owners to set message retention
policy.
Dropdown for changing retention policy is disabled in UI for admins also.
Whenever we use API queries to mark messages as read we now increment
two new LoggingCount stats, messages_read::hour and
messages_read_interactions::hour.
We add an early return in do_increment_logging_stat function if there
are no changes (increment is 0), as an optimization to avoid
unnecessary database queries.
We also log messages_read_interactions::hour Logging stat
as the number of API queries to mark messages as read.
We don't include tests for the case where do_update_pointer is called
because do_update_pointer will most likely be removed from the
codebase in the near future.
This uses `where_active_push_notification()` instead of
the flag condition to keep it consistent with the app
code as we use the same function to filter active push
notifications everywhere else.
Overrides some of internal functions of python-social-auth
to handle native flow.
Credits to Mateusz Mandera for the overridden functions.
Co-authored-by: Mateusz Mandera <mateusz.mandera@zulip.com>
This adds a powerful end-to-end test for Zulip's API documentation:
For every documented API endpoint (with a few declared exceptions that
we hope to remove), we verify that every API response received by our
extensive backend test suite matches the declared schema.
This is a critical step towards being able to have complete, high
quality API documentation.
Fixes#15340.
When doing query for same topic names in a stream, we should do a
case-insensitive exact match for the topic, since that's the data
model for topics in Zulip.
This is a test case verifying the current codebase produces incorrect
results. All the messages should have been edited in this case
regardless of the topic's case unless they belong to a different
stream.
Generated by pyupgrade --py36-plus --keep-percent-format.
Now including %d, %i, %u, and multi-line strings.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
The Python 3.6 style does support non-total and even partially-total
TypedDict, but total gives us better guarantees.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
The narrow parameter was incorrectly documented as a one-level array
rather than an array of arrays, and the tests incorrectly expected
this due to a combination of design and implementation bugs.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
Use read-only types (List ↦ Sequence, Dict ↦ Mapping, Set ↦
AbstractSet) to guard against accidental mutation of the default
value.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
There seems to have been a confusion between two different uses of the
word “optional”:
• An optional parameter may be omitted and replaced with a default
value.
• An Optional type has None as a possible value.
Sometimes an optional parameter has a default value of None, or None
is otherwise a meaningful value to provide, in which case it makes
sense for the optional parameter to have an Optional type. But in
other cases, optional parameters should not have Optional type. Fix
them.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
We triage on the event's "op" field instead
of using the type check.
And then we still get the implicit assertion
that we any "add" event produces a dict
for "subscriptions", since we dereference it
in the assertion for "subscribers".
Fixes#2665.
Regenerated by tabbott with `lint --fix` after a rebase and change in
parameters.
Note from tabbott: In a few cases, this converts technical debt in the
form of unsorted imports into different technical debt in the form of
our largest files having very long, ugly import sequences at the
start. I expect this change will increase pressure for us to split
those files, which isn't a bad thing.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
Automatically generated by the following script, based on the output
of lint with flake8-comma:
import re
import sys
last_filename = None
last_row = None
lines = []
for msg in sys.stdin:
m = re.match(
r"\x1b\[35mflake8 \|\x1b\[0m \x1b\[1;31m(.+):(\d+):(\d+): (\w+)", msg
)
if m:
filename, row_str, col_str, err = m.groups()
row, col = int(row_str), int(col_str)
if filename == last_filename:
assert last_row != row
else:
if last_filename is not None:
with open(last_filename, "w") as f:
f.writelines(lines)
with open(filename) as f:
lines = f.readlines()
last_filename = filename
last_row = row
line = lines[row - 1]
if err in ["C812", "C815"]:
lines[row - 1] = line[: col - 1] + "," + line[col - 1 :]
elif err in ["C819"]:
assert line[col - 2] == ","
lines[row - 1] = line[: col - 2] + line[col - 1 :].lstrip(" ")
if last_filename is not None:
with open(last_filename, "w") as f:
f.writelines(lines)
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
This commit adds three `.pysa` model files: `false_positives.pysa`
for ruling out false positive flows with `Sanitize` annotations,
`req_lib.pysa` for educating pysa about Zulip's `REQ()` pattern for
extracting user input, and `redirects.pysa` for capturing the risk
of open redirects within Zulip code. Additionally, this commit
introduces `mark_sanitized`, an identity function which can be used
to selectively clear taint in cases where `Sanitize` models will not
work. This commit also puts `mark_sanitized` to work removing known
false postive flows.
This commit adds 'add_query_to_redirect_url' to one additional
function which had not yet been written when
'add_query_to_redirect_url' was introduced. This helper centralizes
URL manipulation for redirects, making it easier to add Pysa
sanitization in subsequent commits.
This new endpoint returns a 'user' dictionary which, as of now,
contains a single key 'is_subscribed' with a boolean value that
represents whether the user with the given 'user_id' is subscribed
to the stream with the given 'stream_id'.
Fixes#14966.
The only clients that should use the typing
indicators endpoint are our internal clients,
and they should send a JSON-formatted list
of user_ids.
We now enforce this, which removes some
complexity surrounding legacy ways of sending
users, such as emails and comma-delimited
strings of user_ids.
There may be a very tiny number of mobile
clients that still use the old emails API.
This won't have any user-facing effect on
the mobile users themselves, but if you type
a message to your friend on an old mobile
app, the friend will no longer see typing
indicators.
Also, the mobile team may see some errors
in their Sentry logs from the server rejecting
posts from the old mobile clients.
The error messages we report here are a bit
more generic, since we now just use REQ
to do validation with this code:
validator=check_list(check_int)
This also allows us to remove a test hack
related to the API documentation. (We changed
the docs to reflect the modern API in an
earlier commit, but the tests couldn't be
fixed while we still had the more complex
semantics for the "to" parameter.)
Currently there are no checks in validate_against_openapi_schema
to check whether the `content` it received actually matched with
the `response code`. For example during testing if a certain endpoint
was returning 400 but it was expected to return 200, then it would
pass schema validation as it would only have `msg` and `result` keys.
Add this validation and fix the wrong response returning points.
Add test to validate example responses in zulip.yaml. Also change
zulip.yaml for some wrong examples or for cases which were not
covered by `test-api`. Also enhance `validate_against_openapi_schema`.
Prevent `JsonableError(_("Missing content"))` from
ever being triggered.
That error wasn't handle by anything, and thus just threw a 500, as
it's not a response to an HTTP request.
The right fix is to adjust the caller to ban the empty string in
content (or content that strips to the empty string).
Closes#15145.
ACCESS_TOKEN_URL works a different for apple authentication, so,
we removed and remocked the ACCESS_TOKEN_URL mock in
`register_extra_endpoints` override of apple auth test class.
It is cleaner to have it as generic feature of `social_auth_test`.
So, this commit adds a function that returns token_data_dict that
we had earlier and is called in the ACCESS_TOKEN_URL mock.
This function is overriden in apple auth test class to generate
payload of the format that apple auth expects.
Thanks to Mateusz Mandera for the simple idea.
Co-authored-by: Mateusz Mandera <mateusz.mandera@zulip.com>
This commit adds some basic checks while adding or removing
realm owner status of a user and adds code to change owner
status of a user using update_user_backend.
This also adds restriction on removing owner status of the
last owner of realm. This restriction was previously on
revoking admin status, but as we have added a more privileged
role of realm owner, we now have this restriction on owner
instead of admin.
We need to apply that restriction both in the role change code path
and the deactivate code path.
This commit sets the role of the user creating the realm as
realm owner after the realm is created.
Previously, the role of user creating the realm was set as admin.
But now we want it to be owner because owners have the highest
privilege level.
The test_management_commands use in particular was causing pickling
errors when the test failed, because Python 3 filter returns an
iterator, not a list.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
set(redirect_host) is a set of characters, so the only non-relative
redirects being allowed were to certain one-character hostnames, which
certainly isn’t what was intended.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
Fixes this error in the dev environment:
$ ./manage.py checkconfig
Error: You must set ZULIP_ADMINISTRATOR in /etc/zulip/settings.py.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
This implementation overrides some of PSA's internal backend
functions to handle `state` value with redis as the standard
way doesn't work because of apple sending required details
in the form of POST request.
Includes a mixin test class that'll be useful for testing
Native auth flow.
Thanks to Mateusz Mandera for the idea of using redis and
other important work on this.
Documentation rewritten by tabbott.
Co-authored-by: Mateusz Mandera <mateusz.mandera@zulip.com>
This commit adds an integration for Thinkst Canaries - physical, VM and
cloud-based canaries for detecting attackers to a network. Thinkst
Canaries can send webhook alerts when canaries have been tripped, and
this integration will post Zulip messages when these webhooks are
received.
Signed-off-by: David Wood <david@davidtw.co>
This was previously hardcoded with agreement between the Zulip backend
and frontend as 86400 seconds (1 day). Now, it's still hardcoded in
the backend, but arranged in a way where we could add a setting
without any changes to the mobile and terminal apps to update logic.
Fixes#15278.
This was implemented in 2012 to avoid showing a loading indicator for
fetching messages for users with no message history. However, the
Zulip onboarding UI always creates some message history, and fetching
history is fast, so this is likely clutter more than a useful
optimization.
This was written by Rishi for a very brief purpose a few years ago,
and it doesn't serve much purpose now other than to be a place we
update in code sweeps.
We're migrating to using the cleaner zulip.com domain, which involves
changing all of our links from ReadTheDocs and other places to point
to the cleaner URL.
Adds a top-level logger in `settings.LOGGING` `zulip.auth`
with the default handlers `DEFAULT_ZULIP_HANDLERS` and
an extra hanlder that writes to `/var/log/zulip/auth.log`.
Each auth backend uses it's own logger, `self.logger` which
is in form 'zulip.auth.<backend name>'.
This way it's clear which auth backend generated the log
and is easier to look for all authentication logs in one file.
Besides the above mentioned changes, `name` attribute is added to
`ZulipAuthMixin` so that these logging kind of calls wouldn't raise
any issues when logging is tried in a class without `name` attribute.
Also in the tests we use a new way to check if logger calls are made
i.e. we use `assertLogs` to test if something is logged.
Thanks to Mateusz Mandera for the idea of having a seperate logger
for auth backends and suggestion of using `assertLogs`.
Since owners have the highest privilege level, it made little sense to
sandwich them between administrators and guests.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
This commit removes short_name and client_id fields from the user
objects returned by get_profile_backend because neither of them
had a purpose.
* short_name hasn't been present anywhere else in the Zulip API for
several years, and isn't set through any coherent algorithm.
* client_id was a forgotten 2013-era predecessor to the queue_id field
returned by the register_event_queue process.
The combination of these changes gets us close to having `get_profile`
have the exact same format as other endpoints fetching a user object.
This commit changes get_profile_backend to be based on format_user_row
such that it's a superset of the fields for our other endpoints for
getting data on a user.
To be clear, this does not removes any of the exisiting fields, that
were returned by this endpoint.
This change adds some fields to the User object returned by the
endpoint. API docs are updated accordingly for the added fields.
Slack owners and primary owners will be mapped to zulip
realm owners on import.
Previously, we mapped the owner and primary owner roles of slack
to realm admins in zulip. As we have added ROLE_REALM_OWNER in
8bbc074, we now map slack owners and primary owners to owners in
zulip.
Tests are modified for checking all the 3 cases-
- Slack workspace primary owner
- Slack workspace owner
- Slack workspace admin
This commit also has docs changes in 'import-from-slack.md'.
Calling jwt.decode without an algorithms list raises a
DeprecationWarning. This is for protecting against
symmetric/asymmetric key confusion attacks.
This is a backwards-incompatible configuration change.
Fixes#15207.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
Generated by pyupgrade --py36-plus --keep-percent-format, but with the
NamedTuple changes reverted (see commit
ba7906a3c6, #15132).
Signed-off-by: Anders Kaseorg <anders@zulip.com>
Old topic of the msg edit event can be used to help the client
calculate useful information such as if a change
in current narrow is required.
This fixes our re narrow logic after a stream edit of a topic, with
no change in topic name itself, since the original topic was not
present in the event received and hence the `orig_topic` was
undefined in this case.
This adds 'target_users' parameter to 'attempt_unsubscribe_of_principal`
function in test_subs.py, which accepts list of UserProfile objects to be
unsubscribed, instead of defining users in the function itself.
This change makes the code cleaner and more readable.
Also, 'other_user_subbed' parameter is changed to 'target_users_subbed'
to clearly depict the use of this parameter.
This commit extracts removing_someone_else logic to the function
'check_if_removing_someone_else' for better readability.
This is a follow up to 2187c84.
Option to disable breadcrumb messages were given in both message edit
form and topic edit stream popover.
User now has the option to select which stream to send the notification
of stream edit of a topic via checkboxes in the UI.
We pipe realm_id through functions where it is available,
this helps us avoid doing query for realm_id in loop when
multiple messages are being processed.
Fixes warnings like this:
/srv/zulip-py3-venv/lib/python3.8/site-packages/django/db/models/fields/__init__.py:1424: RuntimeWarning: DateTimeField MutedTopic.date_muted received a naive datetime (2020-01-01 00:00:00) while time zone support is active.
warnings.warn("DateTimeField %s received a naive datetime (%s)"
Signed-off-by: Anders Kaseorg <anders@zulip.com>
datetime.timezone is available in Python ≥ 3.2. This also lets us
remove a pytz dependency from the PostgreSQL scripts.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
Fixes warnings like these with python -Wd:
/home/circleci/zulip/zerver/lib/bugdown/__init__.py:327: DeprecationWarning: This method will be removed in future versions. Use 'list(elem)' or iteration over elem instead.
for child in currElementPair.value.getchildren():
/home/circleci/zulip/zerver/lib/bugdown/__init__.py:328: DeprecationWarning: This method will be removed in future versions. Use 'list(elem)' or iteration over elem instead.
if child.getchildren():
/home/circleci/zulip/zerver/lib/bugdown/__init__.py:282: DeprecationWarning: This method will be removed in future versions. Use 'list(elem)' or iteration over elem instead.
for child in currElement.getchildren():
/home/circleci/zulip/zerver/lib/bugdown/__init__.py:283: DeprecationWarning: This method will be removed in future versions. Use 'list(elem)' or iteration over elem instead.
if child.getchildren():
https://docs.python.org/3.8/library/xml.etree.elementtree.html#xml.etree.ElementTree.Element.getchildren
Signed-off-by: Anders Kaseorg <anders@zulip.com>
Fixes this warning with python -Wd:
/home/circleci/zulip/zerver/lib/bot_config.py:69: DeprecationWarning: This method will be removed in future versions. Use 'parser.read_file()' instead.
config.readfp(conf)
https://docs.python.org/3/library/configparser.html#configparser.ConfigParser.readfp
Signed-off-by: Anders Kaseorg <anders@zulip.com>
This has been a deprecated alias for django.db.backends.postgresql
since Django 1.9, removed in Django 3.0.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
url_to_a returns Union[Element, str], but str cannot be appended to
Element; that would raise TypeError at runtime.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
zerver/lib/i18n.py:34:28: E741 ambiguous variable name 'l'
zerver/lib/webhooks/common.py:103:34: E225 missing whitespace around operator
zerver/tests/test_queue_worker.py:563:9: E306 expected 1 blank line before a nested definition, found 0
Signed-off-by: Anders Kaseorg <anders@zulip.com>
The previous code for this event was using a key that's not actually
a part of the payload. So here we simple remove the usage of that key
and add a (previously missing) test for this event.
Signed-off-by: Hemanth V. Alluri <hdrive1999@gmail.com>
This reimplements our Zoom video call integration to use an OAuth
application. In addition to providing a cleaner setup experience,
especially on zulipchat.com where the server administrators can have
done the app registration already, it also fixes the limitation of the
previous integration that it could only have one call active at a time
when set up with typical Zoom API keys.
Fixes#11672.
Co-authored-by: Marco Burstein <marco@marco.how>
Co-authored-by: Tim Abbott <tabbott@zulipchat.com>
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
Objects whose properties are not described were validated by the
current validator. Edit it so that objects with no `properties`
or `additionalProperties` attribute i.e. opaque objects get
invalidated.
Also make changes in zulip.yaml to fix any opaque objects (tweaked by
tabbott to edit the documentation for better clarity).
Currently, `validate_against_openapi_schema` checks only the top
level of the response dictionary. Improve it so that it can
validate objects and arrays at all levels. Also edit zulip.yaml
accordingly. And for new response keys which were not defined
before add VERY basic documentation.
We change do_create_user and create_user to accept
role as a parameter instead of 'is_realm_admin' and 'is_guest'.
These changes are done to minimize data conversions between
role and boolean fields.
request_retry and notify_bot_owner don't use request_data so might
as well not send it to them at all.
Signed-off-by: Hemanth V. Alluri <hdrive1999@gmail.com>
Using the Python Standard Library's abc library and NotImplementedError
we can better define interfaces (this is mainly to improve readability
and consistency).
Signed-off-by: Hemanth V. Alluri <hdrive1999@gmail.com>
Integrations can be supplied a logo parameter which is used to contruct
their `logo_url`. It would be useful to store this parameter, instead of
computing the path from the URL.
This commit fixes the tests to use role instead of is_admin in
update user endpoint. These changes got missed in the original
commit 9fa6067 which included the change of using role in update user
endpoint and were also not caught in tests.
Tim Abbott
Mateusz Mandera
Sara Gulotta
Aman Agrawal
Hashir Sarwar
sahil839
Anders Kaseorg
Hemanth V. Alluri
arpit551
Dinesh
orientor
Steve Howell
Anders Kaseorg
Graham Bleaney
Kartik Srivastava
Steve Howell
David Wood
Tim Abbott
Johannes Hochreiter
anshgit101
Sahil Batra
Puneeth Chaganti