Commit Graph

49586 Commits

Author SHA1 Message Date
Alex Vandiver cc9b028312 uploads: Set X-Accel-Redirect manually, without using django-sendfile2.
The `django-sendfile2` module unfortunately only supports a single
`SENDFILE` root path -- an invariant which subsequent commits need to
break.  Especially as Zulip only runs with a single webserver, and
thus sendfile backend, the functionality is simple to inline.

It is worth noting that the following headers from the initial Django
response are _preserved_, if present, and sent unmodified to the
client; all other headers are overridden by those supplied by the
internal redirect[^1]:
 - Content-Type
 - Content-Disposition
 - Accept-Ranges
 - Set-Cookie
 - Cache-Control
 - Expires

As such, we explicitly unset the Content-type header to allow nginx to
set it from the static file, but set Content-Disposition and
Cache-Control as we want them to be.

[^1]: https://www.nginx.com/resources/wiki/start/topics/examples/xsendfile/
2023-01-09 18:23:58 -05:00
Alex Vandiver 43fe24a5a0 uploads: Make realm_avatar_and_logo_path non-abstract. 2023-01-09 18:23:58 -05:00
Alex Vandiver 8e68d68f32 uploads: Be consistent about first arguments to write_local_file.
Enforcing a consistent `type` helps us double-check that we're not
playing fast-and-loose with any file paths for local files.  As noted
in the comment, this is purely for defense-in-depth.

Passing `write_local_file` a consistent `type` requires removing the
"avatars" out of `realm_avatar_and_logo_path` -- which makes it
consistent across upload backends.

This, in turn, requires a compensatory change to zerver.lib.export, to
be explicit that the realm icons are exported from the avatars
directory.  This clarity is likely an improvement.
2023-01-09 18:23:58 -05:00
Alex Vandiver 679fb76acf uploads: Provide our own Content-Disposition header.
sendfile already applied a Content-Disposition header, but the
algorithm may provide both `filename=` and `filename*=` values (which
is potentially confusing to clients) and incorrectly slash-escapes
quotes in Unicode strings.

Django provides a correct implementation, but it is only accessible to
FileResponse objects.  Since the entire point is to offload the
filehandle handling, we cannot use a FileResponse.

Django 4.2 will make the function available outside of FileResponse.
Until then, extract our own Content-Disposition handling, based on
Django's.

We remove the very verbose comment added in d4360e2287, describing
Content-Disposition headers, as it does not add much.
2023-01-09 18:23:58 -05:00
Alex Vandiver 83fd807885 uploads: Remove unncessary return in create_attachment. 2023-01-09 18:23:58 -05:00
Alex Vandiver 7c0d414aff uploads: Split out S3 and local file backends into separate files.
The uploads file is large, and conceptually the S3 and local-file
backends are separable.
2023-01-09 18:23:58 -05:00
Alex Vandiver 497abc2e48 nginx: Move uploads handling into app_frontend_base.
As uploads are a feature of the application, not of a generic nginx
deployment, move them into the `zulip::app_frontend_base` class.  This
is purely for organizational clarity -- we do not support deployments
with has `zulip::nginx` but not `zulip::app_frontend_base`.
2023-01-09 18:23:58 -05:00
Lauryn Menard 79fb2c38fb api-docs: Add `missing.md` file specifically for API documentation.
Previously, `templates/zerver/api/missing.md` was a symlink to the
file in the help center documentation.

Prep commit for moving the help center documentation to a top level
directory.
2023-01-09 11:17:00 -08:00
Abhijeet Prasad Bodas 932ce6cee4 MissedMessageHookTest: Add more tests.
Add more tests analogous to existing ones but for different scenarios.

This is mostly boring text, but is important for completeness, since the
notificability logic underneath is subtle.
2023-01-06 17:08:20 -08:00
Abhijeet Prasad Bodas 6288d7ae3e test_event_queue: Split missed message hook tests.
Split the one giant `test_end_to_end_missedmessage_hook` into many
smaller tests.

This allows us to not worry about resetting database state after each
test case and also allows extracting a lot of common stuff into setUp
and tearDown.

There is probably even more scope of deduplication here (for example,
the mock and the `assert_maybe_enqueue_notifications_call_args` call are
same for all test cases) but that might not be worth the added
complexity.

We also change a few
```
user_profile.<setting> = <value>
user_profile.save()
```
expressions to instead use the `do_change_user_setting` function.
2023-01-06 17:08:20 -08:00
Abhijeet Prasad Bodas 04e6b04213 message send: Ignore alert words for presence idle users.
For alert words, we currently don't send email/push notifications --
only desktop notifications. Thus, we don't need to consider alert words
here, since desktop notifications do not utilize the presence status
calculated at this stage.

Tested manually that alert word desktop notifications work as expected.

When we implement email/push notifications for alert words (issues #5137
and #13127), we can add new fields like
`notifications_data.alert_word_email_notify`, similar to the existing
`notifications_data.wildcard_mention_email_notify`, which will allow us
to keep the alert word notifiability check inside the dataclass, similar
to how the mentions checks are done currently. So, even when that
feature is implemented, the code which this commit removes would be
unnecessary.
2023-01-06 17:08:20 -08:00
Abhijeet Prasad Bodas 75cb401f0b test_messages: Rename helper for clarity. 2023-01-06 17:08:20 -08:00
N-Shar-ma d88ce61801 typeahead: Make the typeahead completions undo friendly.
We now use the undo supporting `insert` function from the
`text-field-edit` module to update the text after autocompletion,
instead of just resetting the value of the textarea / input, which was
not undo-able.

So that the undo history seems sensible, we replace only the minimal
diff between the text before and after autocompletion. This ensures
that only this diff, and not the entire text, is highlighted when
undoing, as would be ideal.

Fixes: #14311.
2023-01-06 16:53:57 -08:00
N-Shar-ma 95b77c0c1a util: Add function `get_string_diff`.
The new function specifies the single minimal diff between 2 strings.
It takes in 2 string parameters and returns an array with 3 integers
where the 1st is the index where the difference starts and the 2nd and
3rd are the indices of the end of the diff in the 2 strings.
2023-01-06 16:53:57 -08:00
Lauryn Menard d1714b0671 help-docs: Update articles for restricting email and avatar changes.
Updates the help center articles on restricting email and avatar
changes so that it is clear that administrators can always change
these settings for themselves.

Also, fixes a broken link and a few small text/formatting errors.
2023-01-06 16:41:11 -08:00
Mateusz Mandera f4024f6e9d help center: Remove deprecated mmctl flag from instructions.
--attachments is redundant and should not be used:
```
$ mmctl export create --attachments
Flag --attachments has been deprecated, the tool now includes attachments by default. The flag will be removed in a future version.
```
2023-01-06 16:36:01 -08:00
Zixuan James Li 33716f6156 decorator: Do not send HEAD response with non-empty body.
An HTTP HEAD response with a non-empty message body is not compliant
with the standard.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2023-01-06 13:32:47 -08:00
Zixuan James Li 1e36d79b83 intercom: Return success on HEAD requests.
Intercom sends a HEAD request to validate the webhook URL on their side,
which was not expected in the previous implementation.
This fixes the problem that we send out a confusing error message for it.

Fixes #23912.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2023-01-06 13:32:47 -08:00
Alya Abbott 6fb4f10abb docs: Add communication guidance to mentor guide.
Also link to GSoC's mentor guide.
2023-01-06 11:05:51 -08:00
Alya Abbott 358a0dda5b docs: Clarify --email installation option. 2023-01-05 16:10:34 -08:00
Alya Abbott 17ee697e37 contributor docs: Link to GitHub's blog post on commit discipline. 2023-01-05 16:09:43 -08:00
N-Shar-ma 88603bd34b todo: Fix bug preventing toggling todos with content in composebox.
Uptil now, with the composebox expanded and some content in it, when a
task in a todo widget was clicked, it did not toggle as expected but
focused the composebox, selecting all the text in it.

This is fixed by ensuring the task is a clickable message element and
clicking it doesn't collapse the composebox. This has the added benefit
of fixing the bug where toggling a task expanded a collapsed composebox.

Fixes: #22928.
2023-01-05 12:59:58 -08:00
Alex Vandiver a1151aef8b create-database: Hide harmless "non-existant database" warnings.
During installation on a new host, `create-database` attempts to
verify that there isn't a bunch of data already in the database which
is it about to drop and recreate.  In the most common case, this
statement emits a scary-looking warning, since the database does not
exist yet:

```
+ /home/zulip/deployments/current/scripts/setup/create-database
+ POSTGRES_USER=postgres
++ crudini --get /etc/zulip/zulip.conf postgresql database_name
++ echo zulip
+ DATABASE_NAME=zulip
++ crudini --get /etc/zulip/zulip.conf postgresql database_user
++ echo zulip
+ DATABASE_USER=zulip
++ cd /
++ su postgres -c 'psql -v ON_ERROR_STOP=1 -Atc '\''SELECT COUNT(*) FROM zulip.zerver_message;'\'' zulip'
psql: error: connection to server on socket "/var/run/postgresql/.s.PGSQL.5432" failed: FATAL:  database "zulip" does not exist
```

Because we are attempting to gracefully handle the case where the
database does not exist yet, we also continue (and drop the database)
in other, less expected cases -- for instance, if database contains a
schema we do not expect.

Explicitly check for the database existence first, and once we verify
that, allow any further failures in the `SELECT COUNT(*)` to abort
`create-database`.  This serves the dual purpose of hiding the "FATAL"
error for the common case when the database does not exist, as well as
preventing dropping the database if anything else goes awry.
2023-01-05 11:39:23 -08:00
Anders Kaseorg 778275575e ruff: Ignore RUF002 for consistency.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-01-04 23:06:04 -08:00
Sahil Batra 677b26025a settings_org: Refactor code for dropdown list widget settings.
This commit refactors the code for dropdown list widget settings
such that we can reuse the added functions for further settings
that will use dropdown_list_widget.

This change will be helpful when we will add group-based settings
and change some of the existing role-based settings to be group-based.

We add a new function "get_widget_for_dropdown_list_settings" to
get the widget variable from setting name. The functions to get
and set the setting value use "get_widget_for_dropdown_list_settings"
function to get the widget and then gets or sets the setting value
accordingly.

We also add "data-setting-widget-type" and "data-setting-value-type"
attributes to the element like other settings.

The "data-setting-widget-type" attr is used by get_input_element_value
to use "get_dropdown_list_widget_setting_value" function for
dropdown-list-widget settings.

The "data-setting-value-type" attribute is used to parse the setting
value to correct types in "get_dropdown_list_widget_setting_value"
function.
2023-01-04 17:51:42 -08:00
Lauryn Menard e7c6f35482 user-events: Update account settings display if `is_admin` is changed.
There are a few account settings that are possibly deactivated if
a user is not an administrator (email, name and avatar changes),
depending on the organization's policy for changing those settings.

When a user's role is updated to become (or no longer be) an admin,
the display for these account settings may need to be updated.

Adds `settings_account.update_account_settings_display` to the
functions called in `user_events.update_person` if the active
user's role is changed to or from an administrator role.
2023-01-04 16:26:21 -08:00
Anders Kaseorg 2afdb46095 ruff: Enable new lints DTZ, ISC, PIE, PLW, Q, S, SIM.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-01-04 16:25:07 -08:00
Anders Kaseorg 17300f196c ruff: Fix ISC003 Explicitly concatenated string.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-01-04 16:25:07 -08:00
Anders Kaseorg 2c5e114f8b ruff: Fix ISC001 Implicitly concatenated string literals on one line.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-01-04 16:25:07 -08:00
Anders Kaseorg 234d628fee ruff: Fix PIE807 Prefer `list()` over useless lambda.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-01-04 16:25:07 -08:00
Anders Kaseorg 46cdcd3f33 ruff: Fix PIE790 Unnecessary `pass` statement.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-01-04 16:25:07 -08:00
Anders Kaseorg d9f2f60f4b ruff: Fix DTZ011 `datetime.date.today()`.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-01-04 16:25:07 -08:00
Anders Kaseorg 3e10ceb022 ruff: Fix DTZ007 `datetime.datetime.strptime()` without %z.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-01-04 16:25:07 -08:00
Anders Kaseorg b5cad938b8 ruff: Fix DTZ006 `datetime.datetime.fromtimestamp()` without `tz` argument.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-01-04 16:25:07 -08:00
Anders Kaseorg ba78bee8c4 ruff: Fix DTZ005 `datetime.datetime.now()` without `tz` argument.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-01-04 16:25:07 -08:00
Anders Kaseorg 705460ff58 ruff: Fix DTZ004 `datetime.datetime.utcfromtimestamp()`.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-01-04 16:25:07 -08:00
Anders Kaseorg 7216ba4813 ruff: Fix DTZ001 `datetime.datetime()` without `tzinfo` argument.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-01-04 16:25:07 -08:00
Anders Kaseorg 1451dbffb9 sentry: Interpret event timestamp in UTC.
Force the precision to microseconds to appease our other code that
naively splits this on ‘.’.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-01-04 16:25:07 -08:00
Anders Kaseorg 3025d9a63a ruff: Fix Q002 Single quote docstring found.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-01-04 16:25:07 -08:00
Anders Kaseorg e1ed44907b ruff: Fix SIM118 Use `key in dict` instead of `key in dict.keys()`.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-01-04 16:25:07 -08:00
Anders Kaseorg a6442288cf ruff: Fix PLW0120 Else clause on loop without a break statement.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-01-04 16:25:07 -08:00
Anders Kaseorg f7e97b1180 ruff: Fix PLW0602 Using global but no assignment is done.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-01-04 16:25:07 -08:00
Alya Abbott 1ac5526023 portico: Emphasize cover letter requirement for job applications. 2023-01-04 16:24:11 -08:00
Anders Kaseorg 7afb12ee26 README: Add Ruff badge.
Because Ruff is awesome.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-01-04 16:22:12 -08:00
Alex Vandiver 8ba51f90e6 puppet: Go's website is officially go.dev, not golang.org. 2023-01-04 14:33:37 -08:00
Alex Vandiver 1b692984ce slack_incoming: Support "fields" in "section"s.
This is a best-effort rendering of the "fields" of Slack incoming
hooks, which Slack renders in two columns.  We approximate them in a
Markdown table, with some minor in-place replacements.

Fixes #22228.
2023-01-04 14:03:47 -08:00
Alex Vandiver 4dc57dadd6 slack_incoming: Make check_text_block return its input.
`check_text_block` transformed its input, making the object it
returned not the same object it was passed; this invalidated it for
use in `check_list`.  It is also, in general, unlike all other
validators.

Make it return a TypedDict cast of its input.
2023-01-04 14:03:47 -08:00
Anders Kaseorg cd66448311 install-shfmt: Upgrade shfmt to 3.6.0.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-01-04 14:01:34 -08:00
Anders Kaseorg 986bad8f90 install-shellcheck: Upgrade ShellCheck to 0.9.0.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-01-04 14:01:34 -08:00
Anders Kaseorg 8a8e538af7 install-aws-server: Use correct fd and exit code for usage message.
On --help, write to stdout and exits with 0; on error, write to stderr
and exit with 1.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-01-04 14:01:34 -08:00