The way the flow goes now is this:
1. The user initiaties login via "Billing" in the gear menu.
2. That takes them to `/self-hosted-billing/` (possibly with a
`next_page` param if we use that for some gear menu options).
3. The server queries the bouncer to give the user a link with a signed
access token.
4. The user is redirected to that link (on `selfhosting.zulipchat.com`).
Now we have two cases, either the user is logging in for the first time
and already did in the past.
If this is the first time, we have:
5. The user is asked to fill in their email in a form that's shown,
pre-filled with the value provided inside the signed access token.
They POST this to the next endpoint.
6. The next endpoint sends a confirmation email to that address and asks
the user to go check their email.
7. The user clicks the link in their email is taken to the
from_confirmation endpoint.
8. Their initial RemoteBillingUser is created, a new signed link like in
(3) is generated and they're transparently taken back to (4),
where now that they have a RemoteBillingUser, they're handled
just like a user who already logged in before:
If the user already logged in before, they go straight here:
9. "Confirm login" page - they're shown their information (email and
full_name), can update
their full name in the form if they want. They also accept ToS here
if necessary. They POST this form back to
the endpoint and finally have a logged in session.
10. They're redirected to billing (or `next_page`) now that they have
access.
For the last form (with Full Name and ToS consent field), this pretty
shamelessly re-uses and directly renders the
corporate/remote_realm_billing_finalize_login_confirmation.html
template. That's probably good in terms of re-use, but calls for a
clean-up commit that will generalize the name of this template and the
classes/ids in the HTML.
The "User groups" panel is now removed from settings overlay
and we instead use new "#groups" UI.
This commit also makes some changes to tests to ensure coverage
for pill_typeahead.js which was previously done by
settings_user_group_legacy.test.js. We have still not got
complete coverage on user_pill.ts as we have removed
settings_user_group_legacy.test.js, but we just add the file
to EXEMPT_FILS list for now and will handle it in future.
Fixes#28012.
This commit adds a new endpoint 'users/me/onboarding_steps'
deprecating the older 'users/me/hotspots' to mark hotspot as read.
We also renamed the view `mark_hotspot_as_read` to
`mark_onboarding_step_as_read`.
Reason: Our plan is to make this endpoint flexible to support
other types of UI elements not just restricted to hotspots.
This prep commit moves the 'rename_indexes_constraints'
function to 'lib/migrate' as we're going to re-use it for
the 'UserHotspot' to 'OnboardingStep' table rename operation.
In general, this function would be helpful in migrations
involving table rename operations, subject to the caution
mentioned in the function via comments.
This commit adds code to not allow Zulip Cloud organizations that are not
on the Plus plan to change the "can_access_all_users_group" setting.
Fixes#27877.
This default setup will be more realistic, matching the ordinary
conditions for a modern server.
Especially needed as we add bouncer code that will expect to have
RemoteRealm entries for realm_uuid values for which it receives
requests.
Refactors get_page helper function so that the updates to the
query data for each row is done in the function that processes
the request.
Adds columns to the remote installation page for both the support
and analytics links.
Adds `analytics/views/remote_activity.py` to the files without
100% backend test coverage.
This makes it possible for a self-hosted realm administrator to
directly access a logged-page on the push notifications bouncer
service, enabling billing, support contacts, and other administrator
for enterprise customers to be managed without manual setup.
Earlier the `/poll` slash command was the only way to create polls.
To increase user friendliness with a GUI, a button to launch a modal
to create a poll, has been added to the compose box. This button is
enabled only when the compose box is empty, to avoid complexities with
losing / having to save as draft any message already being composed.
The modal has a form which on submission frames a message using the
`/poll` syntax and the data input in the form, and sets the content of
the compose box to that message, which the user can then send. The
question field is mandatory for form submission.
Fixes: #20304.
Pressing `r` to open compose box, or search via hotkey didn't work
for gear menu while it worked for other popovers. So, this is an
attempt to unify that behavior so that if the hotkeys are not
handles the navbar_menus popovers, then can be handled elsewhere.
Add a personal menu dropdown that opens on clicking user avatar icon
in navbar added in previous commit.
The args passed to render_personal_menu() in onShow() are returned by
get_personal_menu_content_context() in popover_menus_data.js so that
they can be unit tested.
Additionally, added CSS to get a custom arrow for dropdown menu.
Added a `?` hotkey in keyboard shortcuts option in personal_menu
dropdown in a style similar to our tooltip's hotkey by adding
? in a span with class .tooltip-hotkey-hint and adding some CSS.
Fixes part of #22802.
This commit adds code to add "(guest)" to user names of guest
users in the following places -
- right sidebar
- user pills, including the pills in search suggestion typehaead
- typeaheads for user
- sender names in message feed
- user profile popover and modals.
- user name in not subscribed warning banner.
Note that the indicator is shown only if enable_guest_user_indicator
setting is set to true.
As a result of this change, we now translate "deactivated" text
shown in user pills for deactivated users.
Fixes part of #26700.
This commit also limits `stream_bar.decorate` to only
be able to be called for stream messages, since it's
an undefined string_id is no longer a sign that
you're dealing with a DM.
These changes adds a new scheduled message indicator in conversation views
which informs the user of the number of messages that are scheduled to be
sent to the current view.
Fixes: #25584.
This is a bit hacky, but avoids incorrectly checking Handlebars
templates for Django style templates, which in particular interacts
poorly with ICU Message plural syntax.
The vast majority of deployments do not need landing page assets
generated every deploy, which takes more than 15s. This also removes
them from built tarballs, which also do not need them.
This commit adds two drop-down settings in 'SETTINGS / NOTIFICATIONS'
and 'SETTINGS / DEFAULT USER SETTINGS'.
The new settings lie in a new section named "Topic notifications",
just below the "Noification triggers" section.
Label: "Automatically follow topics"
Options: "Topics I participate in", "Topics I send a message to",
"Topics I start", and "Never".
Label: "Automatically unmute topics in muted streams"
Options: "Topics I participate in", "Topics I send a message to",
"Topics I start", and "Never".
Fixes#25914.
This commit adds two user settings, named
* `automatically_follow_topics_policy`
* `automatically_unmute_topics_in_muted_streams_policy`
The settings control the user's preference on which topics they
will automatically 'follow' or 'unmute in muted streams'.
The policies offer four options:
1. Topics I participate in
2. Topics I send a message to
3. Topics I start
4. Never (default)
There is no support for configuring the settings through the UI yet.
Adds warning banner to the organization settings overlay/tabs
for demo organizations. For owners, clicking on the link in the
banner opens a modal to convert the demo organization into a
permanent organization.
For admins that are not owners, clicking on the link will go
to the help center article on demo organizations.
Non-admin users will not see the warning banner.
The modal for converting a demo organization requires the owner
to have set their email address. Once the owner's email address
is set, then the organization can be made permanent by changing
the subdomain of the organization. The deletion date for the
demo organization will be removed as part of updating the
subdomain.
The organization owner must also have updated the organization
type to be any value other than unsepecified to convert the
demo organization to a permanent organization. The modal's
submit button will be disabled if that is the value for the
organization type in the form. The demo organizations created
in the dev environment now have their organization type set to
unspecified on creation.
This is a part of #19523.
Co-authored by: Lauryn Menard <lauryn@zulip.com>
In commit 268f858f3, we removed the "realm_filters" event from the
schemas that we test in `zerver/lib/event_schemas.py`, but the event
is still documented (as deprecated) in the api/get-events doc.
Updates `tools/check_schemas` to not print a warning for an event
schema in the OpenAPI documentation if it's include in the list of
deprecated events list.
Letting the clock tick without a reason introduces the
possibility of nondeterministic test failures depending on the execution
time. The default value when not specified is tick=True, which makes it
easy to miss.
The rule doesn't prohibit setting tick=True, as perhaps there will be
tests wanting to use that feature on purpose, but such a test should
explicitly set it to make the intent clear.
Refactor tools/lib/provision_inner to conditionally write activation commands to user's bash profile based on the host machine type. Automatic activation now skipped for native linux containers.
Fixes#15029
Semgrep 0.118.0 changed the default of --scan-unknown-extensions to
false. It also seems that it no longer respects --lang (or never
did), so rename the config file to reflect that it only includes
Python rules, to make it clear that additional languages will require
separate config files.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
This commit creates a new file integration_url_modal.js which
now contains the code for showing integration url modal.
Since show_generate_integration_url_modal is used in multiple
places, this change helps us in avoiding import cycles.
This commit creates a new file user_deactivation_ui.js which
now contains the code for showing deactivation modal for user
and bots.
Since confirm_deactivation and confirm_bot_deactivation functions
are used in multiple places, this change helps us in avoiding
import cycles.
This commit moves initialize_custom_pronouns_type_fields,
initialize_custom_user_type_fields, initialize_custom_date_type_fields
and append_custom_profile_fields functions to the new file
custom_profile_fields_ui.js from settings_account.js since
they are used for both showing custom profile fields in
"Profile" settings panel and "Edit user" form shown in
user profile modal. This change also helps us in avoiding
import cycles.
We already allowed reruns for failing tests, and this adds
the ability to rerun tests that succeeded as well, which is
helpful for debugging flaky tests.
The intent behind this commit is to tidy up how we handle user info
popovers. The first step is to move everything related to them into
its own module. This commit should not have any functional changes.
The type annotation for functools.partial uses unchecked Any for all
the function parameters (both early and late). returns.curry.partial
uses a mypy plugin to check the parameters safely.
https://returns.readthedocs.io/en/latest/pages/curry.html
Signed-off-by: Anders Kaseorg <anders@zulip.com>
This is a preparatory commit before we migrate `user_group_popover`
from Bootstrap to Tippy library.
The previous implementation was weirdly sharing the logic around
`current_message_info_popover_elem` with the user info popovers based
on a message; very likely an unfortunate latent bug caused by
copy/paste.
To address that, we need to add dedicated functions like
get_user_group_popover_items to avoid breaking keyboard navigation
with this extraction.
We want to reject ambiguous type annotations that set ApiParamConfig
inside a Union. If a parameter is Optional and has a default of None, we
prefer Annotated[Optional[T], ...] over Optional[Annotated[T, ...]].
This implements a check that detects Optional[Annotated[T, ...]] and
raise an assertion error if ApiParamConfig is in the annotation. It also
checks if the type annotation contains any ApiParamConfig objects that
are ignored, which can happen if the Annotated type is nested inside
another type like List, Union, etc.
Note that because
param: Annotated[Optional[T], ...] = None
and
param: Optional[Annotated[Optional[T], ...]] = None
are equivalent in runtime prior to Python 3.11, there is no way for us
to distinguish the two. So we cannot detect that in runtime.
See also: https://github.com/python/cpython/issues/90353
The goal of typed_endpoint is to replicate most features supported by
has_request_variables, and to improve on top of it. There are some
unresolved issues that we don't plan to work on currently. For example,
typed_endpoint does not support ignored_parameters_supported for 400
responses, and it does not run validators on path-only arguments.
Unlike has_request_variables, typed_endpoint supports error handling by
processing validation errors from Pydantic.
Most features supported by has_request_variables are supported by
typed_endpoint in various ways.
To define a function, use a syntax like this with Annotated if there is
any metadata you want to associate with a parameter, do note that
parameters that are not keyword-only are ignored from the request:
```
@typed_endpoint
def view(
request: HttpRequest,
user_profile: UserProfile,
*,
foo: Annotated[int, ApiParamConfig(path_only=True)],
bar: Json[int],
other: Annotated[
Json[int],
ApiParamConfig(
whence="lorem",
documentation_status=NTENTIONALLY_UNDOCUMENTED
)
] = 10,
) -> HttpResponse:
....
```
There are also some shorthands for the commonly used annotated types,
which are encouraged when applicable for better readability and less
typing:
```
WebhookPayload = Annotated[Json[T], ApiParamConfig(argument_type_is_body=True)]
PathOnly = Annotated[T, ApiParamConfig(path_only=True)]
```
Then the view function above can be rewritten as:
```
@typed_endpoint
def view(
request: HttpRequest,
user_profile: UserProfile,
*,
foo: PathOnly[int],
bar: Json[int],
other: Annotated[
Json[int],
ApiParamConfig(
whence="lorem",
documentation_status=INTENTIONALLY_UNDOCUMENTED
)
] = 10,
) -> HttpResponse:
....
```
There are some intentional restrictions:
- A single parameter cannot have more than one ApiParamConfig
- Path-only parameters cannot have default values
- argument_type_is_body is incompatible with whence
- Arguments of name "request", "user_profile", "args", and "kwargs" and
etc. are ignored by typed_endpoint.
- positional-only arguments are not supported by typed_endpoint. Only
keyword-only parameters are expected to be parsed from the request.
- Pydantic's strict mode is always enabled, because we don't want to
coerce input parsed from JSON into other types unnecessarily.
- Using strict mode all the time also means that we should always use
Json[int] instead of int, because it is only possible for the request
to have data of type str, and a type annotation of int will always
reject such data.
typed_endpoint's handling of ignored_parameters_unsupported is mostly
identical to that of has_request_variables.
**Background**
User groups are expected to comply with the DAG constraint for the
many-to-many inter-group membership. The check for this constraint has
to be performed recursively so that we can find all direct and indirect
subgroups of the user group to be added.
This kind of check is vulnerable to phantom reads which is possible at
the default read committed isolation level because we cannot guarantee
that the check is still valid when we are adding the subgroups to the
user group.
**Solution**
To avoid having another transaction concurrently update one of the
to-be-subgroup after the recursive check is done, and before the subgroup
is added, we use SELECT FOR UPDATE to lock the user group rows.
The lock needs to be acquired before a group membership change is about
to occur before any check has been conducted.
Suppose that we are adding subgroup B to supergroup A, the locking protocol
is specified as follows:
1. Acquire a lock for B and all its direct and indirect subgroups.
2. Acquire a lock for A.
For the removal of user groups, we acquire a lock for the user group to
be removed with all its direct and indirect subgroups. This is the special
case A=B, which is still complaint with the protocol.
**Error handling**
We currently rely on Postgres' deadlock detection to abort transactions
and show an error for the users. In the future, we might need some
recovery mechanism or at least better error handling.
**Notes**
An important note is that we need to reuse the recursive CTE query that
finds the direct and indirect subgroups when applying the lock on the
rows. And the lock needs to be acquired the same way for the addition and
removal of direct subgroups.
User membership change (as opposed to user group membership) is not
affected. Read-only queries aren't either. The locks only protect
critical regions where the user group dependency graph might violate
the DAG constraint, where users are not participating.
**Testing**
We implement a transaction test case targeting some typical scenarios
when an internal server error is expected to happen (this means that the
user group view makes the correct decision to abort the transaction when
something goes wrong with locks).
To achieve this, we add a development view intended only for unit tests.
It has a global BARRIER that can be shared across threads, so that we
can synchronize them to consistently reproduce certain potential race
conditions prevented by the database locks.
The transaction test case lanuches pairs of threads initiating possibly
conflicting requests at the same time. The tests are set up such that exactly N
of them are expected to succeed with a certain error message (while we don't
know each one).
**Security notes**
get_recursive_subgroups_for_groups will no longer fetch user groups from
other realms. As a result, trying to add/remove a subgroup from another
realm results in a UserGroup not found error response.
We also implement subgroup-specific checks in has_user_group_access to
keep permission managing in a single place. Do note that the API
currently don't have a way to violate that check because we are only
checking the realm ID now.
Fixes#11767.
Previously multi-character emoji sequences weren't matched in the
emoji regex, so we'd convert the characters to separate images,
breaking the intended display.
This change allows us to match the full emoji sequence, and
therefore show the correct image.
This saves the blue box position as state on the location
in brower history, with `history.replaceState`.
The position is restored when a narrow is activated.
Fixes#20066.
Tweaked provision script to run successfully in Fedora 38 and
included a script to build the groonga libs from source because
the packages in Fedora repos are outdated.
There is a major version jump from the last supported version (F34)
which is EOL so references and support for older versions were
removed.
Fixes: #20635
It's relatively rare that you want to read the coverage report
if the coverage tests pass. This allows devs to get quicker
feedback. Particularly on tricky rebases this can be a real
time saver. It takes about a minute to write the coverage
report on my box.
Dropping support for url_prefix for RealmPlayground, the server now uses
url_template instead only for playground creation, retrieval and audit
logging upon removal.
This does the necessary handling so that url_template is expanded with
the extracted code.
Fixes#25723.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
Restore the default django.utils.log.AdminEmailHandler when
ERROR_REPORTING is enabled. Those with more sophisticated needs can
turn it off and use Sentry or a Sentry-compatible system.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
Translators benefit from the extra information in the field names, and
need the reordering freedom that isn’t available with multiple
positional fields.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
This fixes some deprecation use of the APIs, including using "new"
instead of "true" for the "headless" flag when launching puppeteer and
using $$ instead of $x when using XPath selectors.
We also use {waitUntil: "networkidle2"} to fix the issue of webhook bot
avatar not being loaded consistently when generating the screenshots
(this happened to webhooks like Slack and Harbor).
Positioning of the clip area is adjusted to take the new grid layout
into account.
This allows the user to skip generating screenshots up to certain
integration when working through generating screenshots for all
integrations.
This also allow the user to select a number of integrations to
selectively generate screenshots for.
Release notes in GitHub have significant-newlines, which makes our
canonical `changelog.md` content look not-great -- and the relative
links are also broken, as they have the wrong relative path.
Switch to linking to the canonical release notes on ReadTheDocs, now
that those have proper anchors.
By relocating helper methods into a mixin class, we can be more flexible
with managing transactions in test cases, without always forcing the
django.test.TestCase behavior of always putting the test case into an
atomic transaction.
We include a check for side effects in ZulipTransactionTestCase. It only
checks for the set of row ids in all tables before and after each test.
It is not a comprehensive check for side effects, but should be
sufficient for the basics without much performance overhead.
Previously, the "assets" subdirectory of various bots was not
being copied to the "static/generated/bots/" folder, which
resulted in the documentation on some pages not loading at all.
If the number of threads is not specified
while profiling then use a single thread. This is
because profiling across multiple threads (earlier
default behaviour) may obscure the accurate
measurement of which functions are the most costly
due to thread blocking.
Signed-off-by: Akshat <akshat25iiit@gmail.com>
This was only necessary for PGroonga 1.x, and the `pgroonga` schema
will most likely be removed at some point inthe future, which will
make this statement error out.
Drop the unnecessary statement.
This raises an error when we forget to make migration non-atomic when
doing it in batches. BATCH_SIZE seems to be a good indicator for when
we do batching with few early exceptions.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
Alex Vandiver
Varun Singh
Varun Singh
evykassirer
sayyedarib
Karl Stolley
Anders Kaseorg
Lalit
Mateusz Mandera
Tim Abbott
Sahil Batra
Prakhar Pratyush
Aman Agrawal
Lauryn Menard
N-Shar-ma
Aryan Bhokare
Junyao Chen
Tam Le
Hardik Dharmani
Joelute
Prakhar Pratyush
Eeshan Garg
Sudip Bose
YashRE42
Daniil Fadeev
Zixuan James Li
shu.chen
Steve Howell
AcKindle3
xoldyckk
Satyam Bansal
Akshat