Commit Graph

4832 Commits

Author SHA1 Message Date
Alex Vandiver 1eb31f4f54 upload-release: Update SHA256SUM for existing files if changed.
In the very infrequent case that we re-upload a file, the logic for
upload-release would have the existing SHA256SUM take precedence over
the new one -- despite uploading the new file.

Skip existing filenames (i.e. from the new file) when gathering
SHA256SUM metadata used to assemble `SHA256SUMS.txt`.
2023-01-23 17:41:29 -08:00
Anders Kaseorg 7a7513f6e0 ruff: Fix SIM201 Use `… != …` instead of `not … == …`.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-01-23 11:18:36 -08:00
Anders Kaseorg 6303ebfc2f ruff: Fix SIM115 Use context handler for opening files.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-01-23 11:18:36 -08:00
Anders Kaseorg c6b6004432 ruff: Fix SIM111 Use `return all(…)` instead of `for` loop.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-01-23 11:18:36 -08:00
Anders Kaseorg ff1971f5ad ruff: Fix SIM105 Use `contextlib.suppress` instead of try-except-pass.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-01-23 11:18:36 -08:00
Anders Kaseorg b0e569f07c ruff: Fix SIM102 nested `if` statements.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-01-23 11:18:36 -08:00
Lalit 112df91fbd provision: Do not require that no other yarn precedes us in $PATH.
`check_version` in `install-yarn` had the rather careful check that
the yarn it installed into `/usr/bin/yarn` was the yarn which was
first in the user's `$PATH`.  This caused problems when the user had a
pre-existing `/usr/local/bin/yarn`; however, those problems are
limited to the `install-yarn` script itself, since the nearly all
calls to yarn from Zulip's code already hardcode the `/srv/zulip-yarn`
location, and do not depend on what is in `$PATH`.

Remove the checks in `install-yarn` that depend on the local `$PATH`,
and stop installing our `yarn` into it.  We also adjust the two
callsites which did not specify the full path to `yarn`, so use
`/srv/zulip-yarn`.

Fixes: #23993

Co-authored-by: Alex Vandiver <alexmv@zulip.com>
2023-01-19 17:51:52 -05:00
Anders Kaseorg 8f7a7877fe python: Clean up janky URL matching code with urlsplit.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-01-18 17:25:46 -05:00
Josh Klar 70b30e7792 i18n: Unescape Unicode sequences in JSON.
This greatly improves the readability of the diffs and in-codebase
translation strings over using ASCII sequences for unicode in the JSON.

We've previously noticed [^1] some JSON translation files ending up with
escaped Unicode sequences on disk, which Transifex indicates is expected
behavior [^2], though it is sometimes fixed by `manage.py
compilemessages` [^3]. Further, as noted in #23932 [^4], some JSON
translation files include HTML-escaped entities like quotation marks.

This script will ingest valid JSON files and output them as proper UTF-8
files with appropriately unescaped (unless otherwise necessary, like
double quotes being backslash-escaped) sequences, except when the key
itself contains HTML escape sequences (as it's presumed the value of
such entries must be pre-escaped before being passed to consumers).

[^1]: https://chat.zulip.org/#narrow/stream/58-translation/topic/Transifex.20client/near/1479205

[^2]: https://chat.zulip.org/#narrow/stream/58-translation/topic/an.20email.20for.20Transifex.20support/near/1481287

[^3]: https://chat.zulip.org/#narrow/stream/58-translation/topic/an.20email.20for.20Transifex.20support/near/1481908

[^4]: Which is not end-to-end fixed yet by this commit: that will
require a new release of Zulip Server.

gitlint-ignore: B1, title-trailing-punctuation, body-min-length, body-is-missing
2023-01-17 13:19:45 -08:00
Alex Vandiver 7c0d414aff uploads: Split out S3 and local file backends into separate files.
The uploads file is large, and conceptually the S3 and local-file
backends are separable.
2023-01-09 18:23:58 -05:00
Anders Kaseorg 2afdb46095 ruff: Enable new lints DTZ, ISC, PIE, PLW, Q, S, SIM.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-01-04 16:25:07 -08:00
Anders Kaseorg 17300f196c ruff: Fix ISC003 Explicitly concatenated string.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-01-04 16:25:07 -08:00
Anders Kaseorg 2c5e114f8b ruff: Fix ISC001 Implicitly concatenated string literals on one line.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-01-04 16:25:07 -08:00
Anders Kaseorg d9f2f60f4b ruff: Fix DTZ011 `datetime.date.today()`.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-01-04 16:25:07 -08:00
Anders Kaseorg e1ed44907b ruff: Fix SIM118 Use `key in dict` instead of `key in dict.keys()`.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-01-04 16:25:07 -08:00
Anders Kaseorg cd66448311 install-shfmt: Upgrade shfmt to 3.6.0.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-01-04 14:01:34 -08:00
Anders Kaseorg 986bad8f90 install-shellcheck: Upgrade ShellCheck to 0.9.0.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-01-04 14:01:34 -08:00
Anders Kaseorg 8a8e538af7 install-aws-server: Use correct fd and exit code for usage message.
On --help, write to stdout and exits with 0; on error, write to stderr
and exit with 1.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-01-04 14:01:34 -08:00
Anders Kaseorg bd884c88ed Fix typos caught by typos.
https://github.com/crate-ci/typos

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-01-03 11:09:50 -08:00
Yuri Sulyma 911b3b5365 ts: Convert util.js to TypeScript; add (Raw)Message type in types.ts.
Co-authored-by: Zixuan James Li <p359101898@gmail.com>
2022-12-13 15:23:29 -08:00
Josh Klar 0acb96ea9d provision: Respect proxy settings when installing shellcheck+shfmt. 2022-12-13 12:34:08 -08:00
Josh Klar 0c44b933c8 provision: Replace transifex-client with new transifex-cli.
transifex-client went EOL on November 30, 2022, replaced by
transifex/cli [^1].

Swap this in-place, since per the upstream README [^2]:

> The current version of the client maintains backwards compatibility
> for the tx push and tx pull commands. So, if you have a CI setup that
> uses them, you should not have to change anything.

As the mobile team found out, this is a partial truth if one previously
used some of the more advanced CLI flags, but all workflows referenced
in tools/ and docs/ use forwards-compatible flags to the new version.

[^1]: https://github.com/transifex/transifex-client/

[^2]: a0f28a1cf3/README.md
2022-12-13 12:34:08 -08:00
Greg Price 214eec007a tools: Add script apns/prep-cert.
This prepares server certificates for talking to APNs.
2022-12-07 22:07:36 -08:00
Anders Kaseorg 8b6bd961e6 dependencies: Upgrade JavaScript dependencies.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-12-07 09:55:05 -08:00
Anders Kaseorg 872f4b41c1 ci: Check that non-scripts aren’t marked executable.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-12-07 09:54:01 -08:00
Tim Abbott 158352c339 lint: Remove stale compose_error linter rule.
The module has been renamed, but also the original linter rule was
likely ineffective because of the length of compose_banner calls
anyway.

I'm not particularly worried about future internationalization bugs
here, so removing seems better than replacing it.
2022-12-06 15:48:22 -08:00
evykassirer 606258e75b template_parser: Handle handlebar partial blocks. 2022-12-06 15:40:40 -08:00
evykassirer 8848359887 template_parser: Rename handlebars_singleton for consistency. 2022-12-06 15:40:40 -08:00
evykassirer 4b175514aa emoji: Remove variation selector when generating emoji.
Fixes #23625.

The U+FE0F character was appearing in the data supplied by
emoji-datasource-google, but isn't in the CLDR data. By
removing that variation selector character, all remaining
emoji (that were previously marked as "not found in custom
emoji name maps, but also not found in CLDR data. Skipping.")
are now supported.
2022-12-06 12:33:59 -08:00
Aman Agrawal eb45925b1a actions_popover: Use tippyjs instead of bootstrap to display popover.
Fixes #23494
Popover now automatically displays on left when there is not
enough space top or bottom of the reference element.
2022-12-06 09:58:40 -08:00
Aman Agrawal 4593fcf8d1 actions_popover: Move data to display the popover to a new module.
Since we are switching to tippyjs to display popovers, extracting
this data in a commit of its own, makes further changes easier to
review and merge.

Also, we could add tests with full coverage on the data
provided to popovers, possibly a good first issue for beginners.
2022-12-06 09:58:40 -08:00
Josh Klar 3a0620a40c tools: Reimplement CI failure script without using CircleCI endpoint.
Using curl to POST to the CircleCI workflow endpoint on CZO:

- Doesn't work on zulip/zulip@main (CZO runs a revert)
- Sets a bad example for other orgs
- Robs us of an opportunity to dogfood our own zulip/github-actions-zulip

Refactor the Actions workflows in this repo to report failure states
using the Zulip Action, and reimplement the related helper scripts in
Python, since they'd previously mostly shelled out to Python anyway.
2022-12-05 14:33:15 -05:00
Anders Kaseorg 4898fdb89d ruff: Enable some new errors.
PGH001 forbids eval().

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-12-04 22:11:24 -08:00
Anders Kaseorg c850a006d9 emoji: Import modules relative to ZULIP_PATH.
Newer mypy is less confused this way.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-12-04 22:11:24 -08:00
Anders Kaseorg 84bdd6371d emoji: Remove old broken CSV import/export scripts.
These have been broken at least since commit
e331a758c3 (#12787), so clearly nobody
cares.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-12-04 22:11:24 -08:00
Sahil Batra 00ad7e5aeb development: Add CSS file for emails page in dev server.
Previously, CSS rules were added inline for emails page in
dev server. This commit adds a new file which contains
CSS rules for the emails page in dev server. This will
also help us in adding focus for the radio buttons in
the page, which cannot be added inline to the elements,
when we remove the use of bootstrap for this page.
2022-11-30 17:35:22 -08:00
evykassirer 512705f267 emoji: Remove commas from emoji names.
Fixes part of #23626.

This was preventing emoji like `:family_man,_woman,_girl,_boy:`
from displaying.
2022-11-29 16:26:15 -08:00
Mateusz Mandera 51ed2442fa oneclick: Wait in a loop until the image build droplet gets an IP.
The script will fail if we process with a droplet object with no
.ip_address, which can sometimes happen if we just droplet.load().
Sometimes that gives droplet.ip_address set to None. If we wait in a
loop, we eventuall get the .ip_address set.
2022-11-18 12:45:08 -08:00
Anders Kaseorg 9972fcc85b custom_check: Remove exclusions for comment annotations check.
Commit 2762121162 (#21552) made them
unnecessary.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-11-18 10:08:24 -08:00
Anders Kaseorg 4dce3c54d9 custom_check: Remove assertEquals check.
Ruff checks this as U005 (and even auto-fixes it).

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-11-18 10:08:24 -08:00
Anders Kaseorg 73c4da7974 ruff: Fix N818 exception name should be named with an Error suffix.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-11-17 16:52:00 -08:00
Anders Kaseorg 9a8a2bd345 ruff: Enable import sorting, replacing isort.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-11-16 09:29:11 -08:00
Alex Vandiver f3fd0b6975 release: Support -rc1 style suffixes for releases.
These suffixes suppress some checks in the process, but still generate
and upload a tarball, push a tag, and make a Github prerelease.
`upload-release` already understands that anything with a suffix never
becomes the "latest" release.
2022-11-15 12:50:41 -08:00
Alex Vandiver 1f1e1e4ec2 build-release-tarball: Override merge-base for full official releases. 2022-11-15 12:50:41 -08:00
Alex Vandiver e95e8b985f release: Be explicit about which branch was expected. 2022-11-15 12:50:41 -08:00
Alex Vandiver abb7a63b62 release: Be consistent about TZ-setting. 2022-11-15 12:50:41 -08:00
rht 5ce2103b87 Slack import: Cache emoji.json in static/generated/emoji.
Previously, emoji.json was read from
"$ZULIP_PATH/node_modules/emoji-datasource-google/emoji.json".
This path doesn't exist in production when installing from scratch from
a release tarball. And so, we ensure emoji.json exists by copying it to
`static/generated/emoji`.

With tweaks to comments by tabbott.

Fixes: #23469
2022-11-15 10:43:11 -08:00
Lauryn Menard 53df35e698 tools: Remove obsolete `_set_error_state` method in linter spider.
In 2f547ea, the custom `has_error` logic in BaseDocumentationSpider
was removed in favor of checking whether any errors were logged.

One reference to that method was missed in UnusedImagesLinterSpider
and went unremarked as no images were added to the documentation
that were not used.

Replaces that reference with an error log.
2022-11-09 11:54:19 -08:00
David Rosa 233e690fb4 tools: Upgrade scrapy's settings.py to the latest version.
Fixes a warning caused by using the deprecated `2.6` value
of the `REQUEST_FINGERPRINTER_IMPLEMENTATION` setting.

Upgrades `settings.py` to what would have been generated
by starting the documentation_crawler project using the
`startproject` command of scrapy v2.7.
2022-11-09 11:40:07 -08:00
Aman Agrawal d15b29f056 list_widget: Add functions to add and remove a rendered item in the list.
This a refactoring commit as a followup from #23385.
2022-11-08 13:27:48 -08:00