This raises an error when we forget to make migration non-atomic when
doing it in batches. BATCH_SIZE seems to be a good indicator for when
we do batching with few early exceptions.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
This in-progress feature was started in 2018 and hasn't
been worked on much since. It's already in a broken state,
which makes it hard to iterate on the existing search bar
since it's hard to know how those changes will affect search
pills.
We do still want to add search pills eventually, and when
we work on that, we can refer to this diff to readd the
changes back.
When using a start date before the first commit to a repo, we should
include the repo's entire history (up to the end date) in our totals.
Instead, we were using a range like "..{upper_version}", which in Git
revision-range syntax means the start of the range is HEAD -- so the
range was empty.
Fix that by leaving out the ".." when we want no left endpoint.
This prevents `get_user_profile_by_api_key` from doing a sequential
scan.
Doing this requires moving the generation of initial api_key values
into the column definition, so that even bare calls to
`UserProfile.objects.create` (e.g. from tests) call appropriately
generate a random initial value.
Creates a custom linter rule for `zerver/openapi/zulip.yaml` to
only allow lowercase versions of "true", "false" and "null".
Updates existing documentation for new rules.
This commit adds a new test to check how the visibility policy updates
when moving messages to a topic that didn't exist previously.
This test also helps us adding coverage for the code which just
skips setting visibility_policy if there is no need to update the
value because both previous and new value of visibility policy
is INHERIT. The "actions/message_edit.py" file has 100% coverage
now and thus is removed from "not_yet_fully_covered" list.
Adds test coverage for the error sent for editing a scheduled
message that was successfully sent.
`zerver/actions/scheduled_messages.py` now has 100% test coverage
again.
Test coverage for `zerver/actions/message_delete.py`.
Both callers of this function would already return if there were
no Messages specified to delete, which is why existing tests did
not cover this.
This commit migrates `scroll_util.js` to typescript. I made a helper
type `JQueryOrZJQuery` for the argument of `get_scroll_element` instead
of adding a `__zjquery` property to the global JQuery object because it
isn't being used anywhere outside of this function and hence it makes
sense to have a little helper type than to add the property to the global
object.
Created the new `compose_textarea` and moved event handlers for `#compose_textarea`
from `ui` to this new module so that it is now responsible for initializing event handlers
for compose_textarea instead of `ui` module.
This commit renames 'settings_muted_topics.js'
to 'settings_user_topics.js' because the file now
supports the settings for topics with any visibility_policy,
not just MUTED.
It also renames the corresponding test file.
Added function parameter types, return type, and types of local
varaibles. Added a `null` check for `raw_data` before `JSON.parse`.
Created a type `FormData` and an export type `LocalStorage` to
imporve conciseness and clearity.
Type `LocalStorage` is exported because it might be used in other
files based on an observation that many files have imported `localstorage`.
This helps reduce the amount of import cycles we have in the compose
code path following the migration to a fancier stream input.
`compose_closed_ui.initialize()` was moved further down in the
initialization order because it relies on the dropdown widget
to be defined.
This removes the validator argument for 0423_realmfilter_url_template,
which do not really alter the database schema. It otherwise fails
the migration because the filter_format_validator function is removed.
Migration 0094_realm_filter_url_validator is modified because we can no
longer refer to filter_format_validator.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
This swaps out url_format_string from all of our APIs and replaces it
with url_template. Note that the documentation changes in the following
commits will be squashed with this commit.
We change the "url_format" key to "url_template" for the
realm_linkifiers events in event_schema, along with updating
LinkifierDict. "url_template" is the name chosen to normalize
mixed usages of "url_format_string" and "url_format" throughout
the backend.
The markdown processor is updated to stop handling the format string
interpolation and delegate the task template expansion to the uri_template
library instead.
This change affects many test cases. We mostly just replace "%(name)s"
with "{name}", "url_format_string" with "url_template" to make sure that
they still pass. There are some test cases dedicated for testing "%"
escaping, which aren't relevant anymore and are subject to removal.
But for now we keep most of them as-is, and make sure that "%" is always
escaped since we do not use it for variable substitution any more.
Since url_format_string is not populated anymore, a migration is created
to remove this field entirely, and make url_template non-nullable since
we will always populate it. Note that it is possible to have
url_template being null after migration 0422 and before 0424, but
in practice, url_template will not be None after backfilling and the
backend now is always setting url_template.
With the removal of url_format_string, RealmFilter model will now be cleaned
with URL template checks, and the old checks for escapes are removed.
We also modified RealmFilter.clean to skip the validation when the
url_template is invalid. This avoids raising mulitple ValidationError's
when calling full_clean on a linkifier. But we might eventually want to
have a more centric approach to data validation instead of having
the same validation in both the clean method and the validator.
Fixes#23124.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
This will later be used to expand matching linkifier patterns.
Making it nullable for now, but we will make it required in
the APIs.
As a part of this transition, we temporarily make url_format_string
nullable as well, which will be later removed. This allows us to
switch to populating url_template without caring about passing
url_format_string.
Note that the validators are imported in the migration because Django
otherwise diffs it and considers the schema to be different, generating
a migration, failing the "tools/test-migrations" test.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
If the ID of the scheduled message is passed by the client, we
edit the existing scheduled message instead of creating a new one.
However, this will soon be moved into its own API endpoint.
Removed functions mute_topic and unmute_topic in muted_topics_ui.js
which were using old API endpoints to mute/unmute topics.
In user_topics.js added new function set_user_topic_visibility_policy
which uses new API endpoint `/user_topics` to update the topic's
visibility_policy to visibility_policy passed to
set_user_topic_visibility_policy function as a parameter.
In functions toggle_topic_mute and mute_or_unmute_topic, replaced
the calls of mute_topic and unmute_topic with the new
user_topics.set_user_topic_visibility_policy function.
Added "web/src/user_topics.js" in EXEMPT_FILES in test-js-with-node
as adding function `set_user_topic_visibility_policy` resulted in
user_topics.js losing 100% test coverage.
Fixes#24244
This commit refactors get_realm_user_groups_for_dropdown_list_widget
function to use "group_permission_settings" module instead of
passing variables like "allow_internet_group", etc. individually
for each settings.
We lose some test coverage due to this commit, but that would be
fixed when we add some more group-based settings and allow groups
other than system groups in the group based settings.
This commit adds a new module group_permission_settings.ts
which is used to define configuration options like
allow_owners_group, allow_nobody_group, etc. for group-based
settings similar to GroupPermissionSetting dataclass we have
in backend.
Added type annotations to function parameters, function return
values and local variables. Added neccessary `if` statements to
enforce the objects having proper type before executing the later
operations.
An error type check in the catch block in `update_favicon`
function is added because the statement in the catch block requires
error to be a type `Error` in order to access `error.stack`.
Previously, we had an architecture where CSS inlining for emails was
done at provision time in inline_email_css.py. This was necessary
because the library we were using for this, Premailer, was extremely
slow, and doing the inlining for every outgoing email would have been
prohibitively expensive.
Now that we've migrated to a more modern library that inlines the
small amount of CSS we have into emails nearly instantly, we are able
to remove the complex architecture built to work around Premailer
being slow and just do the CSS inlining as the final step in sending
each individual email.
This has several significant benefits:
* Removes a fiddly provisioning step that made the edit/refresh cycle
for modifying email templates confusing; there's no longer a CSS
inlining step that, if you forget to do it, results in your testing a
stale variant of the email templates.
* Fixes internationalization problems related to translators working
with pre-CSS-inlined emails, and then Django trying to apply the
translators to the post-CSS-inlined version.
* Makes the send_custom_email pipeline simpler and easier to improve.
Signed-off-by: Daniil Fadeev <fadeevd@zulip.com>
This reverts commit 5f4bdfe6bc (#24630).
This was causing Webpack to trigger type checking with the TypeScript
compiler, which is very expensive.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
Primary goal of library replacement is improving execution speed.
This commit should not affect the functionality of the system
or make any changes to it.
Created `WidgetConfig` type which has all the parameter that can be
passed launch(). Except html_heading, html_body, and on_click all
other fields are optional. Therefore, removed checking of undefined
mandatory fields as it will be done by typescript.
Co-authored-by: Hardik Dharmani <Ddharmani99@gmail.com>
Refactored some code as well to avoid unneccesaary `undefined`
checks in `overlays`. To be exact created an action_overlay
object of type `Overlay` such that the `reset_state` is just
`action_overlay = undefined` and having attributes like `$element`
and `close_handler`.
This would ensure that if you have an `active_overlay`, there will
be a registered `close handler` attached to it without needing to
write additional checks for `close_handler` just to satisfy type
checker.
TypeScript doesn't understand Array.prototype.slice.call; rather than dealing with the
type-checking issues around it, we remove it; it was only necessary for ancient browsers where
NodeList isn’t directly iterable.
Remove the winjs specific `getScrollbarWidth` property while we're at it;
WinJS isn't relevant anymore.
Reorders methods due to TypeScript files enforcing define-before-use when
this commit was drafted.
This commit extracts realm creation form html in a new file
realm_creation_form.html as we are reworking the organization
flow as per #24307 and this change would allow us to avoid
duplicating the code.
Some well-intentioned adblockers also block Sentry client-side error
reporting. Provide an endpoint on the Zulip server which forwards to
the Sentry server, so that these requests are not blocked.
Use the built-in HTML escaping of Markup("…{var}…").format(), in order
to allow Semgrep to detect mistakes like Markup("…{var}…".format())
and Markup(f"…{var}…").
Signed-off-by: Anders Kaseorg <anders@zulip.com>
This commit adds 'zerver/lib/email_notifications.py'
to the FILES_WITH_LEGACY_SUBJECT set.
Because the file can have 'subject' in the email sense,
it should be exempted from the 'avoid subject as a var' lint rule.
Updated the title and description in the 'enable-emoticon-translation'
file and renamed the file accordingly. Added a new bullet point for
'time format' in the 'configure-new-user-settings.md' file and updated
the sidebar index by replacing the title 'Use 24-hour time' with
'Change the time format'.
The Django convention is for __repr__ to include the type and __str__
to omit it. In fact its default __repr__ implementation for models
automatically adds a type prefix to __str__, which has resulted in the
type being duplicated:
>>> UserProfile.objects.first()
<UserProfile: <UserProfile: emailgateway@zulip.com <Realm: zulipinternal 1>>>
Signed-off-by: Anders Kaseorg <anders@zulip.com>
Zulip already has integrations for server-side Sentry integration;
however, it has historically used the Zulip-specific `blueslip`
library for monitoring browser-side errors. However, the latter sends
errors to email, as well optionally to an internal `#errors` stream.
While this is sufficient for low volumes of users, and useful in that
it does not rely on outside services, at higher volumes it is very
difficult to do any analysis or filtering of the errors. Client-side
errors are exceptionally noisy, with many false positives due to
browser extensions or similar, so determining real real errors from a
stream of un-grouped emails or messages in a stream is quite
difficult.
Add a client-side Javascript sentry integration. To provide useful
backtraces, this requires extending the pre-deploy hooks to upload the
source-maps to Sentry. Additional keys are added to the non-public
API of `page_params` to control the DSN, realm identifier, and sample
rates.
This commit renames reset_emails_in_zulip_realm function to
reset_email_visibility_to_everyone_in_zulip_realm which makes
it more clear to understand what the function actually does.
This commit also adds a comment explaining what this function
does.
Updates frontend user-facing strings with "private message" or "PM" to
use "direct message" or "DM" respectively instead.
Note that this updates translated strings as well as a few that
are not translated like search suggestions.
Updates `tools/lib/capitalization.py` for some specific strings
that are impacted by these changes, and removes "PM" and "PMs"
from checked strings.
This lets us simplify the long-ish ‘../../static/js’ paths, and will
remove the need for the ‘zrequire’ wrapper.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
Ever since we started bundling the app with webpack, there’s been less
and less overlap between our ‘static’ directory (files belonging to
the frontend app) and Django’s interpretation of the ‘static’
directory (files served directly to the web).
Split the app out to its own ‘web’ directory outside of ‘static’, and
remove all the custom collectstatic --ignore rules. This makes it
much clearer what’s actually being served to the web, and what’s being
bundled by webpack. It also shrinks the release tarball by 3%.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
Corepack manages multiple per-project version of Yarn and PNPM, which
means we have to maintain less installation code, and could help us
switch away from Yarn 1 without making the system unusable for
development of other Yarn 1 projects.
https://nodejs.org/api/corepack.html
The Unicode spaces in the timerender test resulted from an ICU
upgrade: https://github.com/nodejs/node/pull/45068.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
This commit adds dropdown for move_messages_within_stream_limit_seconds
setting which is used to control for how long the user is allowed to
edit topic.
CZO discussion
[here](https://chat.zulip.org/#narrow/stream/6-frontend/topic/code.20coverage/near/1487773).
This allows `zjquery_element.js` to have test utils that aren't always being used.
`upload.js` is mostly UI focused and has upcoming commits that are hard
to write unit tests for (and for which unit tests wouldn't make much sense).
Previously notifications.clear_compose_notifications was used accross
the codebase. Since introducing the new
compose_banner.clear_message_sent_banners function, the two functions
are similar enough that we can just use clear_message_sent_banners
everywhere. This commit also moves scroll_to_message_banner_message_id
to compose_banner.
- Updates `.prettierignore` for the new directory.
- Updates any reference to the API documentation directory for
markdown files to be `api_docs/` instead of `zerver/api/`.
- Removes a reference link from `docs/documentation/api.md` that
hasn't referenced anything in the text since commit 0542c60.
- Update rendering of API documentation for new directory.
Black 23 enforces some slightly more specific rules about empty line
counts and redundant parenthesis removal, but the result is still
compatible with Black 22.
(This does not actually upgrade our Python environment to Black 23
yet.)
Signed-off-by: Anders Kaseorg <anders@zulip.com>
These files are not Jinja2 templates, so there's no reason that they needed
to be inside `templates/zerver`. Moving them to the top level reflects their
importance and also makes it feel nicer to work on editing the help center content,
without it being unnecessary buried deep in the codebase.
In the very infrequent case that we re-upload a file, the logic for
upload-release would have the existing SHA256SUM take precedence over
the new one -- despite uploading the new file.
Skip existing filenames (i.e. from the new file) when gathering
SHA256SUM metadata used to assemble `SHA256SUMS.txt`.
`check_version` in `install-yarn` had the rather careful check that
the yarn it installed into `/usr/bin/yarn` was the yarn which was
first in the user's `$PATH`. This caused problems when the user had a
pre-existing `/usr/local/bin/yarn`; however, those problems are
limited to the `install-yarn` script itself, since the nearly all
calls to yarn from Zulip's code already hardcode the `/srv/zulip-yarn`
location, and do not depend on what is in `$PATH`.
Remove the checks in `install-yarn` that depend on the local `$PATH`,
and stop installing our `yarn` into it. We also adjust the two
callsites which did not specify the full path to `yarn`, so use
`/srv/zulip-yarn`.
Fixes: #23993
Co-authored-by: Alex Vandiver <alexmv@zulip.com>
This greatly improves the readability of the diffs and in-codebase
translation strings over using ASCII sequences for unicode in the JSON.
We've previously noticed [^1] some JSON translation files ending up with
escaped Unicode sequences on disk, which Transifex indicates is expected
behavior [^2], though it is sometimes fixed by `manage.py
compilemessages` [^3]. Further, as noted in #23932 [^4], some JSON
translation files include HTML-escaped entities like quotation marks.
This script will ingest valid JSON files and output them as proper UTF-8
files with appropriately unescaped (unless otherwise necessary, like
double quotes being backslash-escaped) sequences, except when the key
itself contains HTML escape sequences (as it's presumed the value of
such entries must be pre-escaped before being passed to consumers).
[^1]: https://chat.zulip.org/#narrow/stream/58-translation/topic/Transifex.20client/near/1479205
[^2]: https://chat.zulip.org/#narrow/stream/58-translation/topic/an.20email.20for.20Transifex.20support/near/1481287
[^3]: https://chat.zulip.org/#narrow/stream/58-translation/topic/an.20email.20for.20Transifex.20support/near/1481908
[^4]: Which is not end-to-end fixed yet by this commit: that will
require a new release of Zulip Server.
gitlint-ignore: B1, title-trailing-punctuation, body-min-length, body-is-missing
transifex-client went EOL on November 30, 2022, replaced by
transifex/cli [^1].
Swap this in-place, since per the upstream README [^2]:
> The current version of the client maintains backwards compatibility
> for the tx push and tx pull commands. So, if you have a CI setup that
> uses them, you should not have to change anything.
As the mobile team found out, this is a partial truth if one previously
used some of the more advanced CLI flags, but all workflows referenced
in tools/ and docs/ use forwards-compatible flags to the new version.
[^1]: https://github.com/transifex/transifex-client/
[^2]: a0f28a1cf3/README.md
The module has been renamed, but also the original linter rule was
likely ineffective because of the length of compose_banner calls
anyway.
I'm not particularly worried about future internationalization bugs
here, so removing seems better than replacing it.
Fixes#23625.
The U+FE0F character was appearing in the data supplied by
emoji-datasource-google, but isn't in the CLDR data. By
removing that variation selector character, all remaining
emoji (that were previously marked as "not found in custom
emoji name maps, but also not found in CLDR data. Skipping.")
are now supported.
Since we are switching to tippyjs to display popovers, extracting
this data in a commit of its own, makes further changes easier to
review and merge.
Also, we could add tests with full coverage on the data
provided to popovers, possibly a good first issue for beginners.
Using curl to POST to the CircleCI workflow endpoint on CZO:
- Doesn't work on zulip/zulip@main (CZO runs a revert)
- Sets a bad example for other orgs
- Robs us of an opportunity to dogfood our own zulip/github-actions-zulip
Refactor the Actions workflows in this repo to report failure states
using the Zulip Action, and reimplement the related helper scripts in
Python, since they'd previously mostly shelled out to Python anyway.
Previously, CSS rules were added inline for emails page in
dev server. This commit adds a new file which contains
CSS rules for the emails page in dev server. This will
also help us in adding focus for the radio buttons in
the page, which cannot be added inline to the elements,
when we remove the use of bootstrap for this page.
The script will fail if we process with a droplet object with no
.ip_address, which can sometimes happen if we just droplet.load().
Sometimes that gives droplet.ip_address set to None. If we wait in a
loop, we eventuall get the .ip_address set.
These suffixes suppress some checks in the process, but still generate
and upload a tarball, push a tag, and make a Github prerelease.
`upload-release` already understands that anything with a suffix never
becomes the "latest" release.
Previously, emoji.json was read from
"$ZULIP_PATH/node_modules/emoji-datasource-google/emoji.json".
This path doesn't exist in production when installing from scratch from
a release tarball. And so, we ensure emoji.json exists by copying it to
`static/generated/emoji`.
With tweaks to comments by tabbott.
Fixes: #23469
In 2f547ea, the custom `has_error` logic in BaseDocumentationSpider
was removed in favor of checking whether any errors were logged.
One reference to that method was missed in UnusedImagesLinterSpider
and went unremarked as no images were added to the documentation
that were not used.
Replaces that reference with an error log.
Fixes a warning caused by using the deprecated `2.6` value
of the `REQUEST_FINGERPRINTER_IMPLEMENTATION` setting.
Upgrades `settings.py` to what would have been generated
by starting the documentation_crawler project using the
`startproject` command of scrapy v2.7.
This is still messy, in that it does the `chdir` using a different
library than the rest, but it's at least more consistent with the
codebase and it should be possible to bulk migrate the `ZULIP_PATH`
calculation, which we have in a dozen files, to pathlib later.
Setting `credential_source` is used when assuming role credentials --
that is, when running as one role, use the AssumeRole right to become
someone else.
The AWS command-line tools only do this if `role_arn`, the role to
assume, is also set -- if it is not set, it transparently falls
through to IAM role attached to the EC2 instance profile. However,
with the `aws-sdk-go` package, used by Teleport, this configuration
produces an error.
Remove the `credential_source = Ec2InstanceMetadata` line, which isn't
necessary for the AWS CLI, and interferes with Teleport operation.
This commit introduces the change of rendering private messages
section as collapsible, whose data-fetching logic came with zulip#21357.
We now have separated out `Private messages` from `top_left_corner`
section and shifted it below the `global_filters` in a different
separate section along with stream list with common scroll bar
in left-sidebar.
The new PM section will be opened by-default on loading the page
and will have a toggle-icon in its header, clicking on which makes the
section collapse/expand accordingly.
In default view, only recent 5 PM threads would be shown
and would append the active conversation as the 6th one at last
if not present in those 5, similar to how topics list work.
In PM section with unreads, a maximum of 8 conversations
would be shown and rest of them would be hidden behind
the 'more conversations' li-item, clicking on which takes
to the zoomedIn view of PM section where all the present
PM threads would be visible and rest of the sections of left-sidebar
will get collapsed.
Fixes#20870.
Co-authored-by: Aman Agrawal <amanagr@zulip.com>
Before Zulip 4.9, the Zulip install process left any already-installed
rabbitmq with whatever nodename it had previously configured. Wince
this encodes the name of the host when it was installed, this does not
function well with containers.
Leave rabbitmq-server uninstalled, which lets the Zulip installation
process set the nodename to `localhost`, which ensures that it is
usable across container restarts.
Since /communities/#all was failing the spider test due to `#all`
not being present as an `id` on the page, we exclude it from
this check saying that it is not really a documentation page.
Fixes#22984
Add an `@` icon in unread topics where user is mentioned.
We track a new set of `stream_id:topic` pairs for the unread mentions
so that recent topics instantly knows if a topic is unread and mentioned
or not.
This commit adds private messages to the Recent topics view, to make
it an all-encompassing overview of recent activity visible to the user.
We add a filter "Include PM" to toggle whether PMs should be shown in
recent topics.
Fixes#19449.
Zixuan James Li
evykassirer
Tim Abbott
Anders Kaseorg
Lalit
Greg Price
Alex Vandiver
Lauryn Menard
Daniil Fadeev
Sahil Batra
Aman Agrawal
AcKindle3
sbansal1999
Prakhar Pratyush
Hardik Dharmani
human
Adarsh Tiwari
palashb01
Lauryn Menard
Josh Klar
Yuri Sulyma
Mateusz Mandera
rht
David Rosa
jai2201
Barissa-Imran
madrix01
Ganesh Pawar