Commit Graph

47634 Commits

Author SHA1 Message Date
Mateusz Mandera 09dc166b45 do_delete_old_unclaimed_attachments: Consider ArchivedAttachment rows.
This function is oblivious to the existence of ArchivedAttachment, which
is incorrect. A file can be removed if and only if it is not referenced
by any Messages or ArchivedMessages.
2022-06-02 17:32:23 -07:00
Mateusz Mandera 5ff4754090 test_upload: Fix some URLs to uploaded files.
Using http://localhost:9991 is incorrect - e.g. messages sent with file
urls constructed trigger do_claim_attachments to be called with empty
list in potential_path_ids.

realm.host should be used in all these places, like in the other tests
in the file.
2022-06-02 17:32:23 -07:00
evykassirer 72166f1b03 puppeteer: Fix get_element_text including hidden text.
Previously, this function used the `textContent` property, which
includes hidden elements. Fix this by using innerText instead.
2022-06-02 16:44:24 -07:00
evykassirer 67771f80d0 drafts: Remove 'pro tip' from draft modal.
This keyboard shortcut advertisement dates from the original version
of the overlay, before we advertised the shortcut in other ways.

Discussion here:
https://github.com/zulip/zulip/pull/22127#discussion_r882270506
2022-06-02 16:42:37 -07:00
evykassirer e039bdd1e1 narrow: Replace pm_string() with _id and _email variants.
This makes the format returned by these functions more clear.
2022-06-02 16:41:06 -07:00
Sahil Batra 6e8523015c populate_db: Add users to initial private stream in mit and lear realm.
We do not allow keeping vacant private streams as we deactivate them
when all users are unsubscribed, so it is better to add at least one
user in the initial private stream created while creating realm.
2022-06-02 13:42:40 -07:00
Alex Vandiver aa46d8d2a8 puppet: Enable strict typo checking in uwsgi. 2022-06-02 13:20:48 -07:00
Alex Vandiver 18ec3b6215 puppet: Enable background worker threads in uwsgi.
Without this, uwsgi does not release the GIL before going back into
`epoll_wait` to wait for the next request.  This results in any
background threads languishing, unserviced.[1]

Practically, this results in Sentry background reporter threads timing
out when attempting to post results -- but only in situations with low
traffic, as in those significant time is spent in `epoll_wait`.  This
is seen in logs as:

    WARN [urllib3.connectionpool] Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLEOFError(8, 'EOF occurred in violation of protocol (_ssl.c:1131)'))': /api/123456789/envelope/

Or:

    WARN [urllib3.connectionpool] Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProxyError('Cannot connect to proxy.', RemoteDisconnected('Remote end closed connection without response'))': /api/123456789/envelope/

Sentry attempts to detect this and warn, but due to startup ordering,
the warning is not printed without lazy-loading.

Enable threads, at a miniscule performance cost, in order to support
background workers like Sentry[2].

[1] https://github.com/unbit/uwsgi/issues/1141#issuecomment-169042767
[2] https://docs.sentry.io/clients/python/advanced/#a-note-on-uwsgi
2022-06-02 13:20:48 -07:00
Alex Vandiver 919c904091 puppet: Give the uwsgi processes a shorter process name.
Previously, the complete command line, which is quite long, is shown:

    3963143 ?        SN     0:00 /home/zulip/deployments/current/zulip-current-venv/bin/uwsgi --ini /etc/zulip/uwsgi.ini
    3963144 ?        SN     0:03  \_ /home/zulip/deployments/current/zulip-current-venv/bin/uwsgi --ini /etc/zulip/uwsgi.ini
    3963145 ?        SN     0:03  \_ /home/zulip/deployments/current/zulip-current-venv/bin/uwsgi --ini /etc/zulip/uwsgi.ini
    3963146 ?        SN     0:03  \_ /home/zulip/deployments/current/zulip-current-venv/bin/uwsgi --ini /etc/zulip/uwsgi.ini
    3963147 ?        SN     0:03  \_ /home/zulip/deployments/current/zulip-current-venv/bin/uwsgi --ini /etc/zulip/uwsgi.ini
    3963148 ?        SN     0:03  \_ /home/zulip/deployments/current/zulip-current-venv/bin/uwsgi --ini /etc/zulip/uwsgi.ini
    3963149 ?        SN     0:03  \_ /home/zulip/deployments/current/zulip-current-venv/bin/uwsgi --ini /etc/zulip/uwsgi.ini

Configure uwsgi to rename and number the processes.  This results in:

    3907613 ?        SN     0:00 zulip-django uWSGI master
    3907614 ?        SN     0:05  \_ zulip-django uWSGI worker 1
    3907615 ?        SN     0:03  \_ zulip-django uWSGI worker 2
    3907616 ?        SN     0:05  \_ zulip-django uWSGI worker 3
    3907617 ?        SN     0:05  \_ zulip-django uWSGI worker 4
    3907618 ?        SN     0:05  \_ zulip-django uWSGI worker 5
    3907619 ?        SN     0:05  \_ zulip-django uWSGI worker 6
2022-06-02 13:20:48 -07:00
Alex Vandiver a522ad1d9a puppet: Always create a uwsgi master control socket.
This is potentially useful even with rolling restarts disabled.
2022-06-02 13:20:48 -07:00
Alex Vandiver 721a101f12 puppet: Reorganize and comment uwsgi.ini file.
As the uwsgi documentation is somewhat obtuse, more comments are added
here than might usually be.
2022-06-02 13:20:48 -07:00
madrix01 b5e21ceeaf recent_topics: Prevent topic name from extending into unread column.
Previously, when there were no unread messages in a topic, the topic
name was allowed to expand into the space allocated to the unread
count. This matched the behavior of the left sidebar. However, the
left sidebar has extremely limited horizontal space, and being able to
display a few extra characters is useful; recent topics does not have
this constraint. Further, recent topics wraps long topic names on
overflow, which looks ugly when using use the unread count's space.

So we switch to having the unread count element consume space even
when there is no count, using `visiblity:hidden`.

Fixes a part of #19449
2022-06-02 12:49:02 -07:00
Mateusz Mandera 3ce61958a8 docs: Fix typo in gitlab callback url in dev setup instructions.
The endpoint is /complete/gitlab/ - with the slash.
2022-06-02 12:04:18 -07:00
Anders Kaseorg 2e9d5193b4 dependencies: Upgrade JavaScript dependencies.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-06-02 12:03:49 -07:00
Anders Kaseorg 79842fd228 install-shfmt: Upgrade shfmt from 3.4.3 to 3.5.0.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-06-02 12:03:49 -07:00
Anders Kaseorg 61c9740bbd install-yarn: Upgrade Yarn from 1.22.17 to 1.22.18.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-06-02 12:03:49 -07:00
Anders Kaseorg 2007c75061 install-node: Upgrade Node.js from 16.14.1 to 16.15.1.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-06-02 12:03:49 -07:00
Aman Agrawal 422b096b9a language_selection_modal: Add more padding between languages. 2022-06-01 17:08:00 -07:00
Aman Agrawal 66b80c8ae8 admin: Use language_selection_widget to set notification language.
Fixes #21948
2022-06-01 17:08:00 -07:00
Aman Agrawal 59e676b1f5 gear_menu: Allow spectators to select their default language.
Set the default_language as cookie and reload the page so that
the spectator can immediately see the language change in effect.

We can reload the page forcefully for spectators since there is
no chance of any work being lost. It is possible that the spectator
may lose the selected message on doing so.

This requires a new dependency, to be able to set cookies from
frontend JavaScript.

Fixes #21961
2022-06-01 17:08:00 -07:00
Aman Agrawal 085443c722 home: Get default_language info from cookie for spectator.
This allows us to set default_language as cookie in frontend which
can be used on server load to send the relevant
translation data to the client.
2022-06-01 17:08:00 -07:00
Aman Agrawal 2c177f3c53 settings_display: Extract function to launch language selection modal.
It will be used to launch it for spectators using the same shared
code.
2022-06-01 17:08:00 -07:00
Lauryn Menard 214b1a5eba help-docs: Update accessing private messages information.
Removes unnecessary mention of mac keyboard shortcut because the
documentation updates when a mac keyboard is detected.

Also, removes reference to accessing group private messages from
the right sidbar.
2022-06-01 16:38:38 -07:00
Alex Vandiver 18230fcd99 docs: Correct and clarify wal-g backup documentation.
Backups are written every 16k of WAL archive, and by default do not
have an upper limit on how out of date they are, as `archive_timeout`
defaults to 0.

Also emphasize that these are streaming backups, not just one
point-in-time backup daily.

Fixes #21976.
2022-06-01 16:11:32 -07:00
Adam Sah ba5cf331a2 testing: 100% coverage for zerver/tests/test_digest.py. 2022-06-01 16:09:13 -07:00
Alex Vandiver 30457ecd02 upgrade-zulip-from-git: Stop mirroring the remote.
The local `/srv/zulip.git` directory has been cloned with `--mirror`
since it was first created as a local cache in dc4b89fb08.  This
made some sense at the time, since it was purely a cache of the
remote, and not a home to local branches of its own.

That changed in 3f83b843c2, when we began using `git worktree`,
which caused the `deployment-...` branches to begin being stored in
`/src/zulip.git`.  This caused intermixing of local and remote
branches.

When 02582c6956 landed, the addition of `--prune` caused all but the
most recent deployment branch to be deleted upon every fetch --
leaving previous deployments with non-existent branches checked out:

```
zulip@example-prod-host:~/deployments/last$ git status
On branch deployment-2022-04-15-23-07-55

No commits yet

Changes to be committed:
  (use "git rm --cached <file>..." to unstage)
	new file:   .browserslistrc
	new file:   .codecov.yml
	new file:   .codespellignore
	new file:   .editorconfig
[...snip list of every file in repo...]
```

Switch `/srv/zulip.git` to no longer be a `--mirror` cache of the
origin.  We reconfigure the remote to drop `remote.origin.mirror`, and
delete all refs under `refs/pulls/` and `refs/heads/`, while
preserving any checked-out branches.  `refs/pulls/`, if the remote is
the canonical upstream, contains _tens of thousands_ of refs, so
pruning those refs trims off 20% of the repository size.

Those savings require a `git gc --prune=now`, otherwise the dangling
objects are ejected from the packfiles, which would balloon the
repository up to more than three times its previous size.  Repacking
the repository is reasonable, in general, after removing such a large
number of refs -- and the `--prune=now` is safe and will not lose
data, as the `--mirror` was good at ensuring that the repository could
not be used for any local state.

The refname in the upgrade process was previously resolved from the
union of local and remote refs, since they were in the same namespace.
We instead now only resolve arguments as tags, then origin branches;
this means that stale local branches will be skipped.  Users who want
to deploy from local branches can use `--remote-url=.`.

Because the `scripts/lib/upgrade-zulip-from-git` file is "stage 1" and
run from the old version's code, this will take two invocations of
`upgrade-zulip-from-git` to take effect.

Fixes #21901.
2022-06-01 16:06:15 -07:00
Anders Kaseorg 9ee636e920 requirements: Upgrade Python requirements.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-06-01 16:01:54 -07:00
Tim Abbott 0255cbda03 settings: Allow customizing HOME_NOT_LOGGED_IN.
This can be useful if one wants a third-party authentication system to
apply first.
2022-06-01 15:37:31 -07:00
Lauryn Menard 7e9b1a13b0 help_docs: Document subscribing a user to stream via mention.
Updates the help center documentation on subscribing users to
streams to include description for doing so via mentioning a
user while composing a message.

Fixes #21796.
2022-06-01 15:15:34 -07:00
Lauryn Menard 8d8bb88bb4 help-docs: Extend notification bot documentation.
Updates the documentation for configuring the notification bot to
include information about the non-configurable messages sent for
stream settings / permissions changes as well as topic resolve
events.

Adds more detailed sections / information about the configurable
aspects of the notification bot, and information about the topics
for the various messages sent by the notification bot.

Fixes #21947.
2022-06-01 15:11:41 -07:00
Lauryn Menard 2047ca8f16 api-docs: Add changelog entry for user/stream ID narrow options.
Adds a 2.1 release changelog entry for adding support for user
and stream IDs in search/narrow options. Also, adds a Changes
note in the narrow parameter in the OpenAPI `get-messages`
endpoint definition.

Both link to the api documentation for constructing a narrow,
where the 2.1 release update is already mentioned.

Fixes #9474.
2022-06-01 15:11:26 -07:00
Lauryn Menard 82c9c6f599 api-docs: Expand use of `SimpleSuccess` response schema.
Use `SimpleSuccess` response schema for all endpoints that were
already returning a success (200) response without any data beyond
the `response` and `msg` fields, which are standard for all
endpoint responses.

Prep commit for adding `ignored_parameters_unsupported` to
`json_success` responses.
2022-06-01 15:09:39 -07:00
Lauryn Menard 861da7ced8 help-docs: Add manage a user's stream subscriptions article.
Adds a new help center article focused on managing a single
user's stream subscriptions.

Creates a shared file for instructions to navigate to a user's
full profile via the right sidebar, which is used in three help
center articles.

Fixes #21795.
2022-06-01 15:05:55 -07:00
Lauryn Menard 57c8b56902 help-docs: Add mastering the compose box article.
Adds an article for more advanced techniques and workflows for
the compose box and documents the new 'Go to conversation' button.

Also adds cross-links between compose box articles.

Fixes #21959.

Co-authored-by: Alya Abbott <alya@zulip.com>
2022-06-01 14:57:50 -07:00
Anders Kaseorg ecb900bd03 requirements: Update zulint with multiline pattern support.
https://github.com/zulip/zulint/pull/28

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-06-01 14:34:58 -07:00
Alex Vandiver 3741c1c034 puppet: Switch to checking time against the AWS timeserver.
Since this is what chrony is sync'ing to, it lessens the chance of
spurious firings of this alert.

See https://aws.amazon.com/blogs/aws/keeping-time-with-amazon-time-sync-service/
2022-05-31 22:57:32 -07:00
Aman Agrawal a94a06da2b docs: Correct social authentication URL for development setups.
Since `SOCIAL_AUTH_SUBDOMAIN` is set to `auth` in development
environment, this is correct authentication URL.
2022-05-31 22:55:12 -07:00
Anders Kaseorg f2d0ae3255 stream_create: Fix crash on stream creation error.
Commit a9ca5f603b (#15863) incorrectly
translated this; stream_name_error is not a jQuery object.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-05-31 22:54:57 -07:00
Anders Kaseorg ddcd69cb41 gear_menu: Link to /devlogin/ in development environment.
It’d be nicer to use hash_util.build_login_link which also remembers
the current hash, but that doesn’t help when the gear menu is only
rendered once at page load time.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-05-31 22:54:11 -07:00
Lani 5a9cba1bba emoji_picker: Allow filtering emojis by unicode characters in emoji picker.
This PR implements literal emoji match in the emoji picker (for reactions)
and in emoji typeaheads (in compose box)

Tested on mobile browser by opening the emoji picker with the
reaction button, selecting an emoji via the native keyboard, and
ensuring the selected emoji appears in the emoji picker’s search
result.

Fixes #21714.
2022-05-31 10:02:57 -07:00
Lani 986dbc44b7 composebox_typeahead: Make reaction_type available in composebox emojis.
In order to make it possible to search emojis with emoji literals,
we need to make the reaction_type of every emoji available in the
emoji objects used by the composebox.
2022-05-31 10:02:57 -07:00
Zixuan James Li 4c6f2ae7be typing: Add assertions for authentication.
Signed-off-by: Zixuan James Li <359101898@qq.com>
2022-05-31 09:43:55 -07:00
Zixuan James Li c572d9be5a typing: Add none-checks for db queries.
Signed-off-by: Zixuan James Li <359101898@qq.com>
2022-05-31 09:43:55 -07:00
Zixuan James Li 4a5043dd6e typing: Add none-checks for miscellaneous cases.
Signed-off-by: Zixuan James Li <359101898@qq.com>
2022-05-31 09:43:55 -07:00
Zixuan James Li e338ada66c typing: Add none-checks for Recipient objects.
Signed-off-by: Zixuan James Li <359101898@qq.com>
2022-05-31 09:43:55 -07:00
Zixuan James Li bb6a934c8d typing: Add appropriate none-checks for LOCAL_UPLOADS_DIR.
This is a part of django-stubs refactorings.

Signed-off-by: Zixuan James Li <359101898@qq.com>
2022-05-31 09:43:55 -07:00
Zixuan James Li d5517932cd typing: Use BaseDatabaseSchemaEditor in place of DatabaseSchemaEditor.
This is a part of #18777.

Signed-off-by: Zixuan James Li <359101898@qq.com>
2022-05-30 14:18:53 -07:00
Zixuan James Li 63e9ae8389 typing: Apply trivial fixes to adjust edge cases in typing.
Add none-checks, rename variables (to avoid redefinition of
the same variable with different types error), add necessary
type annotations.

This is a part of #18777.

Signed-off-by: Zixuan James Li <359101898@qq.com>
2022-05-30 12:03:51 -07:00
Zixuan James Li c34ac1fcd4 typing: Access url via key "Location" instead of attribute "url".
This is a part of #18777.

Signed-off-by: Zixuan James Li <359101898@qq.com>
2022-05-30 11:59:47 -07:00
Lauryn Menard bb45c04136 frontend: Update when communities directory checkbox is disabled.
Changes the admin UI for the communities directory checkbox to use
the `realm_push_notifications_enabled` page param instead of the
`server_web_public_streams_enabled` page param.

Updates help center documentation about the communities directory
to have clearer information about how the setting works for
self-hosted communitites.
2022-05-27 17:08:15 -07:00