Commit Graph

37119 Commits

Author SHA1 Message Date
Aman Agrawal 6950d8d769 provision: Don't run migrations on `zulip` db in CircleCI.
The automated tests running in CircleCI don't actually use the `zulip`
db, so we can skip running migrations on it in some CircleCI shards to
save time.

NOTE: This only effects build jobs that run provision, except the
`production-build` job where we skip building the dbs altogether.
Migrations still run on `focal-backend` build job to ensure
we are testing all our development setup code.
2020-06-16 15:37:32 +05:30
Ryan Rehman c7e39ef090 narrow: Move the top of narrow notices to `message_scroll.js`.
We refactor these 2 notices to match with the loading indicators,
thus they have been moved to `message_scroll.js`.

After a successful message fetch, we have logic to decide whether
we want to display the notices and also whether we want to hide
the loading indicators (which are already displayed).

We also conservatively hide the notices similar to the indicators
every time we narrow.
The only exception is that we show the history limit notice on
deactivating the narrow (visiting `home_msg_list`).
2020-06-16 00:21:21 -07:00
Hashir Sarwar 31b6867360 push_notifications: Remove redundant `user_profile` assignment. 2020-06-15 22:32:12 -07:00
Ryan Rehman 577055fb54 message view: Fix double loading indicator situation.
Since on narrowing we call `load_messages_for_narrow`,
which fetches both top and bottom messages, two loading
indicators were temporarily displayed.
This was also the case for the `home_msg_list` when we
call `mesage_fetch.initialize` on startup.

To resolve this we do not display the bottom loading
indicator (for new messages), if the older messages
are being fetched too. This is only for the initial
narrow change, and the bottom loading indicator will
be displayed correctly when the user is at the bottom.

This fixes a regression introduced when we added bottom loading
indicators at all, which was temporarily reverted in
67053ff479 before being restored in the
last couple commits.
2020-06-15 22:28:39 -07:00
Ryan Rehman a630f291b9 message view: Refactor top loading indicators.
This commit makes the `loading_older_messages_indicator` similar
to the `loading_newer_messages_indicator`.
Now all the decisions about whether to show a loading indicator
will be made from the `fetch_status` API. We still hide the
indicators everytime the view is changed, as explained in the
previous commit.
2020-06-15 22:26:35 -07:00
Ryan Rehman e0b1fdb81c message view: Show home view bottom loading indicators.
As explained in 67053ff479,
multiple message fetches may be taking place at the same time.
So some other narrows / the home message list's indicator might
get shown for the current narrow.

This commit moves the updation of the indicators display logic
to the `fetch_status` API.
Now the `loading_newer_messages_indicator` gets displayed along
with the `loading_newer` = true updation for that narrow's message
list, i.e. just before we send the API request. But only if the
message list we are fetching matches with our current message list.

The same indicator is hidden similarly, along with the
`loading_older` = false updation for that narrow's message list,
i.e. just after the success response is recieved. But only if
the message list whose data we recieved  matches with our current
message list.

Also the indicators are hidden everytime we activate narrow
or deactivate narrow (`home_msg_list`). And on entering
`narrow.activate` we fetch for it's messages so they get
displayed again, if need be.
This is the reason `message_scroll.hide_indicators();` was
moved to a location above `fetch_messages`.

Fixes #15374.
2020-06-15 22:25:41 -07:00
Aman Agrawal da84b19aea upgrade-zulip: Shutdown servers with <3GB RAM when buiding static.
Fixes #14643.

This is to avoid running out of memory when building static assets
with webpack while server is running in low ram systems.
2020-06-15 22:17:02 -07:00
Aman Agrawal 81195abdbd upgrade-zulip: Extract shutdown call into a function.
This will help us call it as needed.
2020-06-15 22:17:02 -07:00
Vishnu KS 18ecf9bcfa backup: Make restore-backup work in docker.
Co-authored-by: Anders Kaseorg <anders@zulip.com>
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-15 21:37:14 -07:00
Anders Kaseorg fa2496c229 terminate-psql-sessions: Rely on the caller to set PGHOST, PGUSER.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-15 21:37:14 -07:00
Vishnu KS f2ce856b8f scripts: Don't terminate current session in terminate-psql-sessions.
This is a prep commit. Running terminate-psql-sessions command on
docker-zulip results in the script exiting with non-zero exit status
2. This is because the current session also gets terminated while
running terminate-psql-sessions command. To prevent that from happening
we don't terminate the session created by terminate-psql-sessions.
2020-06-15 21:37:14 -07:00
Ryan Rehman b40c8bd9cd search suggestions: Refactor `get_default_suggestion`.
This commit actually just deletes the `get_default_suggestion`
function while the `get_default_suggestion_legacy` function's
logic remains the exact same, it is just renamed.

Since the operator can never be undefined as mentioned in the
previous commit, we do not require the check with undefined
and as a result the, `if (suggestion)` condition can be removed.
2020-06-15 20:16:40 -07:00
Ryan Rehman 72816b9408 minor: Refactor get_search_result of search suggestions.
The 3 changes made here are as follows:
* The if block for both the functions is simplified as
  `Filter.parse` will always return an array and also
  [].slice(0, -1) === [] is true.
* The code where `base_operators` is declared is moved
  to just before where it is actually used.
* The `base` variable declaration is changed to match
  the pattern of that present in the non-legacy function.
  Its value remains the same.

This is a prep commit for when we want to merge the
`get_search_result_legacy` and `get_search_result` functions.
2020-06-15 20:16:40 -07:00
Ryan Rehman 9c5d9c961c search: Parse search string correctly.
This is the exact same bug as observed in
02ab48a61e.

The bug is in the way we invoke `Filter.parse`.
`Filter.parse` returns a list of operators which
can contain only one 'search' term at max.
All strings with the 'search' operator present
in the query are combined to form this 'search'
term.

However on concatenating two filters we may get
two terms containing the 'search' operator. This
will lead to the search suggestions getting
generated based on only the last 'search' operator
term instead of all the terms having the 'search'
operator.

This is evident from the test change as suggestions
should be based on "s stream:of" but instead they
were based on just the latest query.
2020-06-15 20:16:40 -07:00
Tim Abbott 26396c5e25 puppet: Fix exceptions with multiple certbot declarations.
Since 9e8f1aacb3, zulip_ops machines
might have two Package declarations for `certbot`, which doesn't work
in puppet.

The fix is, as usual, to use our `zulip::safepackage` wrapper instead.
2020-06-15 18:21:33 -07:00
Dinesh 8fea2d448a docs: Mention apple in various places listing authentication methods. 2020-06-15 18:06:24 -07:00
Alex Vandiver 7fe1780088 provision: Install `crudini` for zulip.conf shell scripting.
A few tools already use this, and it is part of server installation;
install it in development environments as well.
2020-06-15 17:58:39 -07:00
Tim Abbott 5b49839b08 bankruptcy: Clear mobile push notifications first.
This likely fix a bug that can leak thousands of messages into the
invalid state where:
* user_message.flags.active_mobile_push_notification is True
* user_message.flags.read is True

which is intended to be impossible except during the transient process
between marking messages as read sending the "remove push
notifications" event.

The bug is that if a user who is declaring bankruptcy with 10,000s of
unreads ends up having the database query to mark all of those as read
take 60s, the Django/uwsgi request will time out and kill the process.
If the postgres transaction still completes, we'll end up with the
second half of this function never being run.

A safer ordering is to do the smaller queries first.

We do this in a loop for correctness in the unlikely event there are
more than 10,000 of these.
2020-06-15 17:43:43 -07:00
YashRE42 46247623fc navbar: Fix navbar for unknown streams.
In commit 35c8dcb599 we introduced the
`_stream_params` object within filter.js but we didn't correctly
handle cases where `_stream param`s is undefined within `get_title()`,
`generate_url()` and `get_icon()`, which cause the navbar to if eg a
guest user tries to access a stream they weren't subscribed to.

This commit fixes this by:
  * Adding the relevant checks
  * Adding node tests that include non-existent streams.
  * Adds the 'question-circle-o' icon for non-existent stream narrows.

A side note here is that "non-existent streams" fall under
"common narrows" as per our current definitions, which doesn't really
make sense but shouldn't bother us.

Fixes: #15387.
2020-06-15 16:48:24 -07:00
YashRE42 1e8b922548 navbar: Add missing internationalization to navbar.
This commit adds translation tags to a few user facing strings which
weren't translated prior:
 - "Unknown streams" text and description.
 - "All messages" heading.
 - Tooltip text for precise count of subscribed users.

The numeric count itself is not translated, because we do not do
similar anywhere else in the UI.
2020-06-15 16:48:24 -07:00
Anders Kaseorg cf6981eef0 lint: Remove other rules about percent formatting.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-15 16:24:46 -07:00
Anders Kaseorg f364d06fb5 python: Convert percent formatting to .format for translated strings.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-15 16:24:46 -07:00
Mateusz Mandera eda396376c actions: Remove redundant "assert is None" in do_update_message_flags.
The type of messages arg has changed since this was added.
2020-06-15 16:20:49 -07:00
Mateusz Mandera f467ab046e messages: Remove redundant "assert is not None" for mypy.
Apparently new mypy is able to infer correctly and doesn't need this
assert anymore.
2020-06-15 16:20:43 -07:00
Alex Vandiver bff3b540b1 puppet: Postgres replication should always switch to latest timeline.
Omission of this setting makes resuming after a primary switchover
difficult-to-impossible.  It is the default in PostgreSQL 12.
2020-06-15 16:18:07 -07:00
Alex Vandiver f8fc3a16eb puppet: Use "primary" / "replica" consistently in comments.
The style guide for Zulip is to always use "primary" and "replica"
when describing database replication.  Adjust a few comments under
`puppet/` that do not adhere to this.

Unfortunately, some references still remain to the insensitive and
inaccurate "master" / "slave" terminology.  However, these are only in
files which we are attempting to preserve as close to the upstream
versions they are derived from (e.g. postgresql.conf,
postfix/master.cf).
2020-06-15 16:18:07 -07:00
Alex Vandiver 5f433d6eeb puppet: Remove vestigial check_postgres.pl.
65774e1c4f switched from using the bundled check_postgres.pl to using
the version from packages; the file itself remained, however.

Remove it, and clean up references to it.

Fixes #15389.
2020-06-15 16:18:07 -07:00
Alex Vandiver 7d4a370a57 puppet: Move monitoring of pg replication to the pg hosts.
Instead of SSH'ing around to them, run directly on the database hosts.
This means that the replicas do not know how many bytes behind they
are in _receiving_ the wall logs; thus, the monitoring also extends to
the primary database, which knows that information for each replica.
This also allows for detecting when there are too few active replicas.
2020-06-15 16:18:07 -07:00
Tim Abbott 05455f432e sentry: Enable the javascript name for that platform.
Hopefully this will mean our Sentry integration properly handles
javascript events now.
2020-06-15 13:01:39 -07:00
Tim Abbott 3087fbe854 stream_topic_history: Pluralize remove_message.
Since this function now does a bulk operation with several messages,
we should make sure it's named appropriately.
2020-06-15 10:52:04 -07:00
Tim Abbott ba2de34414 message edit: Move and clarify an important comment. 2020-06-15 10:47:10 -07:00
Aman Agrawal 0cffaa7b5d ui: Upgrade remove_message to allow removing messages in bulk.
We simply pass the visible message ids to remove_and_rerender
which supports bulk delete operation.

This helps us avoid deleting messages in a loop which freezes the
UI for the duration of the loop.
2020-06-15 10:41:12 -07:00
Aman Agrawal ccc0c8706c delete_message: Support `bulk_message_deletion`.
Fixes #15285
This event will be used more now for guest users when moving
topic between streams (See #15277). So, instead of deleting
messages in the topic as part of different events which is
very slow and a bad UX, we now handle the messages to delete in
bulk which is a much better UX.
2020-06-15 10:41:12 -07:00
Gittenburg 52b7dac6af emoji_picker: Fix DOM race condition with offsets.
We need to wait for the DOM to be ready before we save the
section_head_offsets ... otherwise they can be off.

Fixes #15380.
2020-06-15 10:34:07 -07:00
Tim Abbott f448ee404a lint: Fix a new % format string that should be fstring. 2020-06-15 00:12:08 -07:00
Anders Kaseorg 769e198ab0 lint: Prohibit percent formatting on literal format strings.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-14 23:27:22 -07:00
Anders Kaseorg 5dc9b55c43 python: Manually convert more percent-formatting to f-strings.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-14 23:27:22 -07:00
Anders Kaseorg 3461db7ef5 python: Convert percent formatting to "".format in certain files.
These files can’t use f-strings yet because they need to run in Python
2 or Python 3.5.

Generated by pyupgrade.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-14 23:27:22 -07:00
Anders Kaseorg 74c17bf94a python: Convert more percent formatting to Python 3.6 f-strings.
Generated by pyupgrade --py36-plus.

Now including %d, %i, %u, and multi-line strings.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-14 23:27:22 -07:00
Anders Kaseorg 1ed2d9b4a0 logging: Use logging.exception and exc_info for unexpected exceptions.
logging.exception() and logging.debug(exc_info=True),
etc. automatically include a traceback.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-14 23:27:22 -07:00
Anders Kaseorg 4b6d2cf25f logging: Pass more format arguments to logging.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-14 23:27:22 -07:00
Anders Kaseorg 1a3441dbf5 confirmation: Pass realm rather than host to confirmation_url.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-14 23:27:22 -07:00
Hemanth V. Alluri 545f40b2c2 webhooks/circleci: Add support for super minimal payloads.
After merging PR #15355 we found that CircleCI may send
super minimal payloads. This does not seem to depend on
.circleci/config.yml since we received two different
types of payloads off of the same configuration.

Signed-off-by: Hemanth V. Alluri <hdrive1999@gmail.com>
2020-06-14 22:35:41 -07:00
Aman Agrawal cda7b2f539 deletion: Add support for bulk message deletion events.
This is designed to have no user-facing change unless the client
declares bulk_message_deletion in its client_capabilities.

Clients that do so will receive a single bulk event for bulk deletions
of messages within a single conversation (topic or PM thread).

Backend implementation of #15285.
2020-06-14 22:34:00 -07:00
Aman Agrawal 21ac1fb32c register: Refactor to pass client_capabilities directly.
We extract values from client_capabilities directly in
do_events_register where we decide how to process the
extracted variables.
2020-06-14 21:42:50 -07:00
Tim Abbott d97c891afe realm owners: Remove unnecessary duplicate strings. 2020-06-14 21:32:10 -07:00
sahil839 18429cfd29 realm: Allow only owners to set message retention policy for a realm.
This commits adds restriction on admins to set message retention policy.
We now only allow only organization owners to set message retention
policy.

Dropdown for changing retention policy is disabled in UI for admins also.
2020-06-14 21:23:53 -07:00
sahil839 87e72ac8e2 realm: Allow only owners to configure auth methods for a realm.
This commit adds the restriction on configuring auth methods for
admins. We now allow only owners to configure the auth methods
for realm.
2020-06-14 21:23:51 -07:00
sahil839 a6f31c3668 settings: Disable organization deactivation button for admins.
This commit adds the code to disable deactivate organization button
for admins. We now allow only owners to deactivate the organization.

The backend implementation for allowing only owners to deactivate
is already added in 81c28c1.
2020-06-14 21:20:30 -07:00
sahil839 dbd64c4362 settings_users: Only owner can deactivate other owners.
This commit adds the restriction of deactivating owners for admins
by disabling the deactivating button in the UI. Only owners are
allowed to deactivate other owners. The backend part of this is
already implemented in 86b52ef.
2020-06-14 21:20:30 -07:00