Unlike `receiver_is_off_zulip`, fetching from a dict is pretty cheap,
so we can calculate `online_push_enabled` along with the other
variables.
This is a prep change to start using the dataclass introduced in the
earlier commit in this code.
We will in later commits, extend this class to contain methods
to determine if a message is notifiable or not, but for now
we only turn it into a dict and pass it on.
This gives us a single place where all user data for the message
send event is calculated, and is a prep change for introducing
a TypedDict or dataclass to keep this data toghether.
We have already calculated these values, so storing them should not cause
significant performance degradation.
This is a prep chenge for sending a few more flags through internal_data,
namely if `sender_is_muted`.
For this extraction, we need to move some context
parameter (from home_real in `views/home.py`) to extra
page_params parameter (of
build_page_params_for_home_page_load in
`lib/home.py`) so handlebars template can access them.
While moving I confirmed that these parameters are not
used elsewhere if some parameter is used elsewhere
(like `apps_page_url`) then I didn't remove it from the
context list, I just added it to the page_params list.
Fixes: #18795.
This is a prep commit to extract the gear menu as a
handlebars template.
We are renaming `enable_marketing_emails_enabled` to
`corporate_enabled` as it will be also used in the
handlebars template of the gear menu.
This is a prep commit to extract the gear menu as a
handlebars template.
We are extracting it as a function so we can also use it as
a parameter in `page_params`.
This results in moving the `zulip_merge_base` parameter to
page_params, so that it's available to JavaScript.
Since this is technically a tiny overlay, it needs to be initialized
before hashchange.js.
We convert the `clean-unused-caches` script to a
python file so we can run it in provision by importing it
instead of running the script, hence saving some time.
These tests didn't configure ldap settings correctly and as a result,
the user involved in these tests wasn't actually hamlet@zulip.com, but a
new, malformed user with email "hamlet" that was being created by the
ldap auto-registration codepath. This wasn't caught because the codepath
didn't validate the email address and thus created such a malformed user
silently.
* In `event_queue.py`, only the sender and recipient users who have muted
the sender will have the "read" flag set.
* We already skip enqueueing notifications for users who've muted the sender
after 58da384da3.
* The queue consume functions for email and push notifications already
check filter messages which have been read before sending notifications.
* So, the "read" logic in `event_queue.py` is unnecessary, and the
processing power saved from not enqueueing notifications for a single
user should be insignificant, so we remove these checks all toghether.
This logic was peviously untested. This is a prep change for us
to completely depend on the logic here for the "read" flag, and
not on the `event_queue` code.
It was unclear what the original test was testing, and more
importantly, the test passed even if one removed the `read` flag
check in the `handle_push_notifications` function, so we fix it
to be more comprehensive.
The javascript tab in .md templates can be
generated along with the line that adds js
example.
Further, as a part of the effort of moving
towards a single template, the markdown extension
for javascrit examples is modified to return empty
string if javascript example doesn't exist for that
endpoint. This would make it possible to cover more
endpoints with a single template.
The js example tabs are now automatically generated
during generation of javascript code, and so need to
be removed. Also, the markdown function to render js
examples can be added in all templates, since it is
parses and returns an empty string if the examples
don't exist, and allows us to move towards a common
template.
The pages have been verified to be correct
by using diff between old and new pages' raw HTML.
This module deals with the testing of /activity, /realm_activity
and /user_activity. All these pages reside in analytics module.
Keeping these tests in zerver/tests is kind is not appropriate
since person who makes changes to /activity pages would not think
it is necessary to run tests in zerver. So better to keep them
in the analytics module.
The headings for return values were currently hardcoded
in cases where they occur, but they can be rendered directly
in the markdown extension if the return values exist.
Currently, in the FAQ on our /plans page, when the user clicks on
the sponsorship link in the answer for the first question, they
are always taken to /accounts/go, causing them to have to input
their organization URL even if they are on a subdomain page.
This commit makes it so that when the user is on a subdomain page,
they are taken to /upgrade#sponsorship directly. On the other
hand, when they are on a root domain (/) page, they have to go
through /accounts/go and specify their organization's name.
Because the payload of V3 will no longer include the description,
We replace the ":" by "." in the message and create the new string
template for trigger messages.
The Hubot project looks to be abandoned; it hasn’t been updated in
years and its own installation instructions don’t work anymore.
Remove our special placement of Hubot alongside Zapier and IFTTT.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
Currently, the descriptions for API responses were
hardcoded in the templates. However, they now exist
in YAML data as well, and so can be fetched from there.
Also, as a part of moving towards a common template, it is
beneficial to show return response description along with
the code example directly. Also, for the same reason, the
need for mentioning subschemas for the response has been
removed, and ALL responses for that response code should
be returned in the proper format.
This also includes returning empty string if that response
code doesn't exist. This allows us to directly mention both
200 and 400 responses in all templates, and they are parsed
automatically.
A followup is necessary to remove the descriptions in the template
as they are duplicated just by this commit
Currently, the descriptions of API responses were
hardcoded in the template. However, as a part of the
goal of moving towards a common template, these should
be auto-generated.
This commit copies these descriptions into the `description`
parameter in the OpenAPI data for that effect.
This causes avatars and emoji which are hosted by Zulip in S3 (or
compatible) servers to no longer go through camo. Routing these
requests through camo does not add any privacy benefit (as the request
logs there go to the Zulip admins regardless), and may break emoji
imported from Slack before 1bf385e35f,
which have `application/octet-stream` as their stored Content-Type.
Earlier, the notification-blocking for messages from muted senders
was a side-effect of we never sending notifications for messages
with the "read" flag.
This commit decouples these two things, as a prep for having new
settings which will allow users to **always** receive email
notifications, including when/if they read the message during the
time the notifications is in the queue.
We still mark muted-sender messages as "read" when they are sent,
because that's desirable anyways.
Fixes#17277.
The main limitation of this implementation is that the sync happens if
the user authing already exists. This means that a new user going
through the sign up flow will not have their custom fields synced upon
finishing it. The fields will get synced on their consecutive log in via
SAML in the future. This can be addressed in the future by moving the
syncing code further down the codepaths to login_or_register_remote_user
and plumbing the data through to the user creation process.
We detail that limitation in the documentation.
The old type in default_settings wasn't right - limit_to_subdomains is a
List[str]. We define a TypeDict for capturing the typing of the settings
dict more correctly and to allow future addition of configurable
attributes of other non-str types.
The `message_id` and `user_profile_id` values don't really matter for
our testing here, so we might as well set these dummy values in the
main function.
This is a prep change for importing (and using) `dataclasses.field`
elsewhere in the same file, because pyflakes would throw "Import
module shodowed" errors otherwise.
Rename poll_timeout to event_queue_longpoll_timeout_seconds
and change its value from 90000 ms to 90 sec. Expose its
value in register api response when realm data is fetched.
Bump API_FEATURE_LEVEL to 74.
Expose the boolean value server_needs_upgrade in the
responses for register api so that it can be used
by mobile and terminal clients as well.
Highlighted in api changelog as part of
feature level 74 in commit fb93c96
(next commit).
Shift functions used for compatibility from
zerver.lib.home (is_outdated_server) and
zerver.view.compatibility (pop_numerals,
version_lt, find_mobile_os,
is_outdated_desktop_app, is_unsupported_browser)
to zerver.lib.compatibility module.
This commit adds support for testing of
those endpoints whose .md files would
be deleted in favour of their pages
to be generated automatically by the template.
curl examples for such endpoints must exist
in accordance to the pattern of template, which
can be used to run the tests for them.
The list curl_commands_to_test gets filled
already and so, the code to test each command
in the list can be moved out of the block that
opens the file. The only change in this commit
is reducing an indentation for the entire block.
This has been done to reuse the whole block
in case the file does not exist.
This commit refactors the code to find the
lines to be first, and then test all the
lines that contained commands.
This was done to avoid duplication of
the code for the other case, where the
.md file won't exist, as those are soon
to be deleted in favour of a common template.
Soon, the .md files of each endpoint would
be removed and be auto-generated from OpenAPI
data.
So, instead of using the files directly,
we should check from the list of endpoints and
open the files from there.
A follow-up to change logic when the .md files
get deleted should be done.
This locks the message row while a reaction is being added/removed,
which will handle race conditions caused by deleting the message
at the same time.
We make sure that events work happens outside the transaction,
so that in case there's some problem with the queue processor, the
locks aren't held for too long.
As a nice side-effect, we also handle race conditions from double
adding reactions, because once the message is locked, a duplicate
request will wait till the earlier transaction commits, and hence
will not throw `IntegrityErrors`s (rather, will be handled in our
safety check in the /views code itself), which earlier had to be
handled explicitly.
This locks the message while creating a submessage, which
will handle race conditions caused by deleting the message
simultaneously.
We make sure that events work happens outside the transaction,
so that in case there's a problem with the queue processor,
the locks aren't held for too long.
Further commits will start locking the message rows while
adding related fields like reactions or submessages,
to handle races caused by deleting the message itself at the
same time.
The message locking implemented then will create a possibility
of deadlocks, where the related field transaction holds a lock
on the message row, and the message-delete transaction holds a
lock on the database row of the related field (which will also
need to be deleted when the message is deleted), and both
transactions wait for each other.
To prevent such a deadlock, we lock the message itself while
it is being deleted, so that the message-delete transaction
will have to wait till the other transaction (which is about
to delete the related field, and also holds a lock on the
message row) commits.
https://chat.zulip.org/#narrow/near/1185943 has more details.
Further commits will hook some `send_event` calls to `on_commit`.
With those changes, these will never be executed in tests, because
transactions never get commited with `TestCase`, which the
`ZulipTestCase` is a subclass of.
We want to make sure that these events are actually sent for testing
purposes, hence this change.
There's no need to actually capture the callbacks, because the
events are already thoroughly tested.
Further commits will hook `send_event` calls to `on_commit`
in some cases. This change will make it easier to test such
situations.
We don't need to actually capture the callbacks, because the
events sent are already tested via the list in which they are
captured by `tornado_redirected_to_list`.
This commit fixes a bug where moving messages between streams was
not allowed for non-admins when allow_community_topic_editing was
set to false and move_messages_between_streams_policy was set to
Realm.POLICY_MEMBERS_ONLY.
The bug is fixed by calling can_edit_content_or_topic only when
topic or content edit is there and not in the case where only
message is moved from one stream to another.
This commit extracts the logic of checking the message edit permissions,
like whether the sender is same as user, whether it is a (no topic)
message or whether community topic editing is allowed, into a separate
function.
This is a prep commit for fixing a bug where permission to move messages
between streams is affected by permission of editing topics.
Previously when enforcing the check to do not allow editing topics
after a certain time, we were checking whether 'content is None' and
considering it as that if content is None then there must be topic
edit.
But after adding support for moving messages between streams it can be
the case that we are neither changing topic nor content and just moving
streams, and the original code raises error if this is done after the
time limit of editing topics, which is wrong.
This commit fixes this by actually checking 'topic_name is not None'.
Soon, each endpoint won't necessarily have a .md
file, but would generate API doc directly from
OpenAPI data using a template.
So, the lists of endpoints to be tested should not
be taken from the .md files, but from the REST
endpoints available in the sidebar.
This commit also adds a missing test for an invalid
article being accessed in the URL of an API page.
The current logic to get API pages' title using
OperationID should be used when the first line
of the file explicitly mentions so.
In cases where the files didn't begin with `#` but also
didn't need to get title from OpenAPI summary,
this logic fails and causes Server error.
This particularly happens when the article is invalid,
and the `missing.md` file doesn't need title to be
generated, but doesn't start with `#` either.
This commit fixes the logic of using the generated title and covers the bug.
This should help with #17425, where messages with lots of LaTeX are
lost, due to the large expansion factor.
This isn't a total fix for this - large messages with lots of LaTeX
can still end up larger than 1MB, and rendering could timeout, but
this fix should help significantly.
1MB is still small enough that I don't expect we'll run into any DOS
problems - my testing didn't show any problems rendering messages that
contain ~1MB of LaTeX.
This will offer users who are self-hosting to adjust
this value. Moreover, this will help to reduce the
overall time taken to test `test_markdown.py` (since
this can be now overridden with `override_settings`
Django decorator).
This is done as a prep commit for #18641.
Checked the email looked OK in `/emails` for both creating realm and
registering within an existing one.
Not sure zerver/tests/test_i18n.py test has been suppressed correctly.
Fixes#17786.
This is a bit hacky, but will make these tests more readable,
in that the reader would not have to remember the order or parameter
names.
Python 3.8 introduced `mock.call_args.kwargs`, and once we upgrade,
we can use those to assert actual dictionaries instead of this hack.
d66cbd2832 added these mentioning
"always_notify" for some reason, but always_notify clearly isn't a real
thing in this context so the comments need to be fixed to eliminate this
potential source of confusion.
Our current logic only allows S3 block storage providers whose
upload URL matches with the format used by AWS. This also allows
other styles such as the "virtual host" format used by Oracle cloud.
Fixes#17762.
These checks are more related to the API than the editability
or permissions logic, so it makes sense to handle them first
before further processing the request.
Also split the main test class to separate out the tests for
this logic.
This also simplifies some tests by reducing the data setup
required to reach failure.
Tweaked by tabbott to avoid losing the topic_name.strip().
Since caa08d76b5, we no longer have
a common component for stream IDs in zulip.yaml, so we might as
well change the description to be specific and clear.
Sometimes the Slack import zip file we get isn't quite the canonical
form that Slack produces -- often because the user has unzip'd it,
looked at it, and re-zip'd it, resulting in extra nested directories
and the like.
For such cases, support passing in a path to an unpacked Slack export
tree.
modified_user=sub_info.user and modified_stream=sub_info.stream, added
by commit 6d1f9de7d3 (#16553), were
always coming from the last entry in the loop above, not from the
enclosing list comprehension.
Found by the Pylint rule undefined-loop-variable.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
The current logic of geneerating HTML titles requires the title to be
present as a heading in the first line of .md file. However, this will
shortly be no longer true for /api pages where these are
auto-generated from OpenAPI data. Modified the code to fetch the
title from OpenAPI data in case of such pages.
Currently, the title of each endpoint are hardcoded in .md
files, but these are also added in summary parameter of
openapi. Added a markdown processor to insert the title
of a given endpoint.
As a part of effort of removing .md files for /api pages, the
title of each page can be added into the summary OpenAPI
parameter. This also adds a nice summary visible in Swagger
and enhances the documentation. Added the parameter for all endpoints.
The old name `push_notify_user_ids` was misleading, because
it does not contain user ids which should be notified for
the current message, but rather user ids who have the online
push notifications setting enabled.
When the Tornado server is restarted during an upgrade, if
server has old events with the `push_notify_user_ids` fields,
the server will throw error after this rename. Hence, we need
to explicitly handle such cases while processing the event.
We should only show the referrer name in subject of invitation emails,
and show only 'Zulip' in the 'From' header. This helps in preventing
the email from being marked as suspicious by the detection systems
when they see an employee's name as sender of an email sent from an
unrelated domain.
The behavior is already the same for reminder invitation emails where
we do not show name and only 'Zulip' in the 'From' header.
Fixes#18256.
?dl=1 causes Dropbox to send Content-Type: application/binary, which
can’t be interpreted by Camo. Use ?raw=1 instead.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
Commit 1a7ddd9ea3 “Fix
UserActivityInterval overlap bug” introduced a mathematically
incorrect assertion about how intervals work. There’s a third way two
intervals could overlap: both the start and end of the old interval
could be inside the new interval. This probably can’t happen here
because the old interval should be at least as long as the new
interval. However, a correct overlap test can be formulated in a
simpler way anyway.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
This is will make it easier to systematically use Django's
`capturOnCommitCallbacks` in tests outside of the main
`test_events` file which involve assertions on events.
Having the alert state in the message body is useful when alert topics
are not defined by alert description but encoded in the url.
E.g. in large environments having a topic for each alert [alerting] and [ok] would
make it harder to properly track if an alert has been resolved.
When each alert is in a single topic, so far, the alert state has been missing.
This change will add the current alert state and a fitting icon in front
of the alert name.(Similar to the prometheus alertmanager integration)
The test cases have been amended to cover all possible alert states, even
though realistically grafana only fires the ok and alerting states via
webhook.
We should only keep tests for changing email_address_visibility in
test_realm.py. The tests for checking the value of delivery_email
and email in the user dicts returned by 'GET users/{user_id}'
endpoint according to email_address_visibility should be in
test_users.py and not test_realm.py.
The tests of other realm settings are also arranged in the same way.
This commit makes the test more robust by checking for all
possible values of email_address_visibility and checking
emails and delivery emails values received by different
user roles.
Convert this function that absolutely makes a stream web public.
We already have do_change_stream_invite_only to convert
streams to public and private streams.
We also update all the fields that should be set when a stream
is made web public.
Since this is currently only useful to interpret presence data, we
send this only if presence is requested.
I'm not sure that server_timestamp is the right name for this field,
but ultimately it should match the main presence API format.
After re-assignment, mypy will still think the type of
`widget_content` to be `str`, not `Dict`. So we need to
create a new variable.
This is a prep change for stronger type checking in this
code.
The `create_user` API and data import tools can result in our having
active users in the database who haven't intentionally created a Zulip
account or agreed to the ToS; we should never email such users.
The check for `TOS_VERSION is not None` is necessary for the
development environment, which has `TERMS_OF_SERVICE` set but not
`TOS_VERSION`.
It's likely that we will want this check in other places as well.
This parameter has never been used, and causes an unnecessary database
query.
We keep the num_push_devices_for_user function, since we may have uses
for it down the line.
Fixes part of #14166.
The command:
codespell --skip='./locale,*.svg,./docs/translating,postgresql.conf.template.erb,.*fixtures,./yarn.lock,./docs/THIRDPARTY,./tools/setup/emoji/emoji_names.py,./tools/setup/emoji/emoji_map.json,./zerver/management/data/unified_reactions.json' --ignore-words=codespell_ignore_words.txt .
The content of codespell_ignore_words:
```
te
ans
pullrequest
ist
cros
wit
nwe
circularly
ned
ba
ressemble
ser
sur
hel
fpr
alls
nd
ot
```
Prior to this, we only supported direct mention to
the user groups. This commit extends that support
to silent mention for the user groups.
A related test case is also added.
Fixes: #11711.
Earlier, USER_GROUP_MENTIONS_RE was:
r"(?<![^\s\'\"\(,:<])@(\*[^\*]+\*)"
For the syntax: *foo*, this was unnecessarily capturing it as
*foo* and the extraction of `foo` was done using another helper
function: `extract_user_group`.
This is now changed as:
r"(?<![^\s\'\"\(,:<])@(\*(?P<match>[^\*]+)\*)"
and extraction of `foo` can be done just by using the named capture
group `match`.
This change also helps to simplify its related code path.
Earlier, MENTIONS_RE was:
r"(?<![^\s\'\"\(,:<])@(?P<silent>_?)(?P<match>\*\*[^\*]+\*\*)"
For the syntax: **foo**, this was unnecessarily capturing it as
**foo** and adding extra operation for the extraction of `foo`.
This is now changed as:
r"(?<![^\s\'\"\(,:<])@(?P<silent>_?)(\*\*(?P<match>[^\*]+)\*\*)"
and extraction of `foo` can be done just by using the named capture
group `match`.
This change also helps to simplify its related code path.
Earlier wildcard mentions were used as: @all, @everyone, @stream.
This syntax is deprecated and we will no longer support
this syntax in future. See the commits:
1. 7a4c3c1a5c
2. b650b6b38c
When we started to use these syntaxes for wildcard mentions.
Following the convention, we use uppercase for
regex. Also, `user_group_mentions` is given a
conventional name ending with `*_RE`: `USER_GROUP_MENTIONS_RE`.
`deliver_scheduled_emails` and `deliver_scheduled_messages` use their
respective tables like a queue, but do not have guarantees that there
was only one consumer (besides the EMAIL_DELIVERER_DISABLED setting),
and could send duplicate messages if multiple consumers raced in
reading rows.
Use database locking to ensure that the database only feeds a given
ScheduledMessage or ScheduledEmail row to a single consumer. A second
consumer, if it exists, will block until the first consumer commits
the transaction.
We record Git details about the merge-base with upstream branches in
the zulip-git-version file, if the upstream repository is available.
Note that the first Git upgrade after merging the parent commit will
not include the merge-base details, since the upstream repository will
not have been available.
Co-authored-by: Tim Abbott <tabbott@zulip.com>
Signed-off-by: Anders Kaseorg <anders@zulip.com>
These logs were pretty spammy, and there have long been much better
ways to communicate to system administrators that the incoming email
gateway is great, including, most importantly, in the section of the
emails themselves that explains how replying works.
Previously only admins were allowed to move messages between streams
and admins are allowed to post in any stream irresepctive of stream
post policy, so there was no need to check for stream post policy.
But as we now allow other members to also move messages, we need
to check whether the user who is moving the message is allowed
to post to the target stream (i.e. stream to which the messages
are being moved) and thus we allow moving messages only if the
user is allowed to post in target stream.
b7b1ec0aeb made our checks of the response
format stronger, to enforce that the json translates to a valid dict.
However, old client code (zulip_botserver) was using "" as equivalent to
response_not_required - so we need to keep backward-compatibility to not
break things built on it.
Currently, moving messages between streams is an action limited to
organization administrators. A big part of the motivation for that
restriction was to prevent users from moving messages from a private
stream without shared history as a way to access messages they should
not have access to.
Organization administrators can already just make the stream have
shared history if they want to access its messages, but allowing
non-administrators to move messages between would have
introduced a security bug without this change.
This completes the effort to make it possible to use
bulk_access_message in contexts where there are more than a handful of
messages without creating performance issues.
Cleaning up test_realm_domains.RealmDomainTest.test_list_realm_domains,
test_subs.StreamAdminTest.test_private_stream_live_updates,
test_subs.StreamAdminTest.test_realm_admin_can_update_unsub_private_stream
and test_subs.StreamAdminTest.test_non_admin_cannot_access_unsub_private_stream.
This new function optimizes how we fetch subscriptions
for streams. Basically, it excludes most long-term-idle
users from the query.
With 8k users, of which all but 400 are long term idle,
this speeds up get_recipient_info from about 150ms
to 50ms.
Overall this change appears to save a factor of 2-3 in the backend
processing time for sending or editing a message in large, public
streams in chat.zulip.org (at 18K users today).
If the caller has already fetched the Stream or subscription details
for the user, those can be passed to has_message_access to avoid extra
database queries.
When the format of the response received from the outgoing webhook
server is invalid (unparsable json, or just wrong format that doesn't
translate into a dictionary etc.), a message with the error is sent to
the bot owner. We should include the actual payload to make reasonable
debugging possible.
In notify_bot_owner we have to move the `if response_content` block to
append the payload to the message whenever it was specified as an
argument to the function. It shouldn't be nested inside
`elif status_code` as before.
This makes it parallel with deliver_scheduled_messages, and clarifies
that it is not used for simply sending outgoing emails (e.g. the
`email_senders` queue).
This also renames the supervisor job to match.
The screenshot generating mechanism doesn't work for newrelic and
causes error because its configuration file doesn't exist. This
commit fixes the configuration and re-generate the screenshots.
Also link to it from the API documentation page,
other help pages, and the confirmation dialog for
muting a user.
With substantial edits by tabbott and alya.
A message containing wildcard mention when quoted (which
is turned into a silent mention) or message with silent
wildcard mention notifies the users by sending desktop,
sound, and missed message email notifications. This
is clearly a bug which is fixed by this commit.
Fixes: #18354.
* Remove unnecessary json_validator for string parameters.
* Update frontend to pass right parameter.
Bump api feature level and highlight the fix for `emojiset`
parameter of `settings/display` endpoint in zulip.yaml file.
Fixes part of #18035.
* Remove unnecessary json validator for string validator.
* Update frontend to pass right validator.
* Update zulip.yaml to pass right parameter for curl request
in openapi.
* Update python_examples to pass right paramater.
Fixes part of #18035.
Note that the documentation cannot fully use our macros, because
Uptime Robot requires an & of the end of the URL, because of how it
passes its payload.
Fixes#13854. Fixes#13939.
We combine the two loops into one, so that we
can check our flags before creating the
UserMessageList object.
And we lift a few calculations out of the loop.
For 8k users, with 95% long-term-idle, this was
about a 10x speedup for me. (~30ms -> 3ms)
This removes unnecessary json_validator for string parameters in the
BigBlueButton video calls endpoints. Note that this breaks links to
video meetings sent before the upgrade; there's not much we can do
about that.
Since this is the last commit in this series, we update the
ZULIP_FEATURE_LEVEL for this batch of changes.
Fixes part of #18035.
* Remove unnecessary json_validator for string parameters.
* Remove unnecessary JSON encoding in frontend calls. Structurally,
JavaScript does correct encoding without explicit JSON encoding.
Fixes part of #18035.
Remove unnecessary json_validator for string parameters. This change
does not modify JavaScript because we don't have a frontend for these
API endpoints yet.
Fixes part of #18035.
The previous hashers mirrored the ones used in production, but that was
non-ideal because those are slow. Replacing them with quick hashers is a
performance improvement for those tests.
Raising jsonableError in the authentication form was non-ideal because
it took the user to an ugly page with the returned json.
We also add logging of this rare occurence of the scenario being
handled here.
user_profile.check_password(password) in authenticate of
EmailAuthBackend can raise PasswordTooWeakError; this happens when the
user's password is weaker than the current required policies and needs
to be rehashed (E.g. because, as in Django 3.2, the minimum salt
entropy increased).
This is a very rare case, but still needs a good user-facing error
message. We raise a json error to handle this with a user-facing error
message.
See this comment by Mateusz Mandera for a detailed explanation
about this case along with a traceback it generates.
https://github.com/zulip/zulip/pull/15449#discussion_r448308614
Support for the timeouts, and tests for them, was added in
53a8b2ac87 -- though no code could have set them after 31597cf33e.
Add a 10-second default timeout. Observationally, p99 is just about
5s, with everything else being previously being destined to meet the
30s worker timeout; 10s provides a sizable buffer between them.
Fixes#17742.
Thumbor and tc-aws have been dragging their feet on Python 3 support
for years, and even the alphas and unofficial forks we’ve been running
don’t seem to be maintained anymore. Depending on these projects is
no longer viable for us.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
Leave the Intel build as the prominent default, since it will run on
both platforms. (I would have liked to detect the appropriate
platform, but Apple seems to have put significant effort into making
that impossible for anti-fingerprinting reasons, which is probably an
overall good.)
Signed-off-by: Anders Kaseorg <anders@zulip.com>
Move `get_setup_webhook_message` to
`zerver/lib/webhooks/common.py` so multiple integrations can use this
rather than just those which import `zerver/lib/webhooks/git.py`. Also
added the documentation for this.
Since the invariant we're trying to protect is that every realm has an
active owner, we should check precisely that.
The root bug here, which the parent commit failed to fix properly, is
that we were doing a "greater than" check when we clearly originally
meant a "less than" check -- lower role numbers have more permissions.
Django's default SMTP implementation can raise various exceptions
when trying to send an email. In order to allow Zulip calling code
to catch fewer exceptions to handle any cause of "email not
sent", we translate most of them into EmailNotDeliveredException.
The non-translated exceptions concern the connection with the
SMTP server. They were not merged with the rest to keep some
details about the nature of these.
Tests are implemented in the test_send_email.py module.
* Move the extended documentation of code blocks to a separate page.
* Merge "code playgrounds" documentation to be a section of that page.
* Document copy widget on code blocks.
* This commit changes how we refer to "```python" type syntax for code
blocks. Instead of being called a syntax highlighting label, this is
now referred to as a "language tag", since it serves both syntax
highlighting and playgrounds.
* Remap all the links.
* Advertise this new page in various places that previously did not have a link.
As discussed in the comment, this is a critical scalability
optimization for organizations with thousands of users.
With substantial comment updates by tabbott.
Linked the Help Center document in places like
- zulip.yaml (/events, /register/, realm/playgrounds,
/realm/playgrounds/{playground_id})
- /help/format-your-message-using-markdown (Linked to make
users reading the markdown code block style, aware of this
feature)
- /templates/settings/playground_settings_admin.hbs (Linked
as a reference to read more about playgrounds before
configuring one)
Also showcase the feature on /features and /for/open-source.
This was accidentally removed in 27e4f5da92
as the initial idea was to have another parameter `stream_id`
for stream type messages instead of sending it in this `to` parameter.
Also fixed a description of `to` for `stream` type.
This is a prep refactor, instead of creating Confirmation
object and using `confirmation_url` for generating confirmation
link/url, using `create_confirmation_link` would be a cleaner
approach, also this can help us avoid failing test in case
Confirmation model is changed.
Part of #16359.
This will stop dropping events in the case that the background
`maybe_send_batched_email` thread takes longer than 30s. However, see
also #15280 and the TODO comment about how we lose events upon
restart; this worker is still lossy.
This commit modifies the test_wildcard_mention_restrictions test
for checking that moderators are not allowed to send messages
with wildcard mention if wildcard_mention_policy is set to
WILDCARD_MENTION_POLICY_ADMINS. Previously, we were checking
for members, but it is better to check for moderators.
Apparently, after upgrading to Django 3.2, mutating is_staff and then
saving can result in a user's session being destroyed.
In any case, this test is probably better written using two different
users with the different roles, which we have in our initial database
anyway.
In Django 3.2 slugify strips trailing dashes and underscores:
0382ecfe02
sanitize_name doesn't so this difference should be documented like the
others.
Underscore character is already covered by \w, so _ in the regex is
redundant. Also the docstring is mildly incorrect - underscore already
is an allowed character by django's slugify (and always was) for the
aforementioned reason.
This is a straightforward upgrade in terms of changes needed.
Necessary changes were:
- Set `DEFAULT_AUTO_FIELD`
https://docs.djangoproject.com/en/3.2/releases/3.2/#customizing-type-of-auto-created-primary-keys
- `The default_app_config application configuration variable is deprecated, due
to the now automatic AppConfig discovery.`
https://docs.djangoproject.com/en/3.2/releases/3.2/#automatic-appconfig-discovery
To handle this one, we can remove default_app_config from
zerver/__init__.py because it satisfies what release notes describe in
https://docs.djangoproject.com/en/3.2/releases/3.2/#automatic-appconfig-discovery:
"Most pluggable applications define an AppConfig subclass in an apps.py
submodule. Many define a default_app_config variable pointing to this
class in their __init__.py. When the apps.py submodule exists and
defines a single AppConfig subclass, Django now uses that configuration
automatically, so you can remove default_app_config."
An important note is that rebuild-test-database needs to be run after
this upgrade in dev environment - if tests are run with test db that was
built on the previous version, they will fail due to a mysterious bug
(?), where changing attributes of a user and .save()ing after logging in
in the test via self.login_user, causes getting logged out - the next
requests via self.client_get etc. are unauthed for some reason,
unless self.login_user is called again. This behavior is no longer
exhibited upon rebuilding the test db - and I can't reproduce it in
production or dev db. So this can likely be reasonably dismissed as some
quirk of the test client system that won't be relevant in the future and
doesn't impact production.
Now that we are passing source realm's id instead of string_id in
source realm selector, it makes sense to rename the "source_realm" field
to "source_realm_id".
In the source realm selector, when we select a realm from which we want
to import the data, we pass the source realm's string_id. The problem
with this approach is that the string_id can be an empty string. This
commit makes the source_realm pass the realm's id instead of string_id.
Now, the source_realm's value will either be an integer or "" (empty
string) when we don't want to import settings from any realm.
Currently only enabled in development, since the exact details don't
seem right..
Co-Author-By: Signior-X <b19188@students.iitmandi.ac.in>
Co-Author-By: Aman Agrawal <amanagr@zulip.com>
Implements UI for #8005.
The comment mentioned the values in this dict to be consistent
wth values in settings_config.invited_as_values.
But settings_config.invited_as_values was replaced by
settings_config.user_role_values in 136c005f3f, so
updated the comment accordingly.
This commit adds both frontend and backend code to invite a user as
moderator. We allow only existing owners and admins to invite a user
as a moderator.
The function get_role_for_new_user was added to get role from the
invited_as value, as invited_as values were one of (1,2,3,4)
previously, but it was then changed to be the actual role value,
i.e. one of (100, 200, 400, 600), in 1f8f227444.
So, we can safely remove this function now and use invited_as value
directly and handle realm_creation case by using an if condition.
Requesting external images is a privacy risk, so route all external
images through Camo.
Tweaked by tabbott for better test coverage, more comments, and to fix
bugs.
As of now, editing a widget doesn't update the rendered content.
It's important to ensure that existing votes or options added later on
don't get deleted when rendered.
This seems more complex than it's worth.
For now, we just prevent edits to widgets.
This commit makes the UI clearer that editing widgets isn't allowed.
See also:
https://github.com/zulip/zulip/issues/14229https://github.com/zulip/zulip/issues/14799Fixes#17156
`ensure_basic_avatar_image` and `ensure_medium_avatar_image` are
essentially the same thing, except a size parameter.
So, refactor them into a single function.
This doesn't introduce any functional changes.
Abhijeet Prasad Bodas
Riken Shah
Ganesh Pawar
Steve Howell
Vishnu KS
sahil839
akshatdalton
Mateusz Mandera
Suyash Vardhan Mathur
Tim Abbott
Eeshan Garg
PIG208
Anders Kaseorg
Alex Vandiver
Gaurav Pandey
Wesley Aptekar-Cassels
seiwailai
Gilbert Bishop-White
Adam Birds
ryanreh99
Sumanth V Rao
AdamVB
Aman Agrawal
rht
Dinesh
Siddharth Asthana
pletinckxc
Alya Abbott
shanukun
m-e-l-u-h-a-n
pilgrim2308