Commit Graph

37270 Commits

Author SHA1 Message Date
Aman Agrawal e769323d0c recent_topics: Refactor is_topic_hidden.
* We now only get info from other libraries when required.
* muted renamed to include_muted.
* function renamed to filters_should_hide_topic.
2020-06-13 16:29:19 -07:00
Aman Agrawal 6490f666af recent_topics: Fix boundary typo. 2020-06-13 16:29:19 -07:00
Aman Agrawal d7d5bc208c recent_topics: Rename count_senders for clarity. 2020-06-13 16:29:19 -07:00
Aman Agrawal 0eda671299 recent_topics: Remove depricated arg in process_message.
This arg was used to determine rerender mechanism after the
message is processed.
2020-06-13 16:29:19 -07:00
Aman Agrawal 5931b69c57 recent_topics: Add note about reify_message_id_if_available. 2020-06-13 16:29:19 -07:00
Anders Kaseorg 57a80856a5 python: Convert more "".format to Python 3.6 f-strings.
Generated by pyupgrade --py36-plus --keep-percent-format.

Now including %d, %i, %u, and multi-line strings.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-13 15:39:00 -07:00
Aman Agrawal 4aeb02f73d stream_header_colorblock: Create resuable css class.
stream-selection-header-colorblock css class can now be used
to reproduce stream header color block everywhere.
2020-06-13 15:38:40 -07:00
Anders Kaseorg 6358339581 backends: Convert ExternalAuthDataDict to Python 3.6 style.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-13 15:37:25 -07:00
Anders Kaseorg 46d0018fbb types: Convert ProfileDataElement to Python 3.6 style and total.
The Python 3.6 style does support non-total and even partially-total
TypedDict, but total gives us better guarantees.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-13 15:37:25 -07:00
Steve Howell 686ae3eef4 node tests: Use less mocking for stream events.
We now use the real implementation of
`stream_data` in our tests (as well as `people`),
while still just checking stubs for "heavier"
functions that we dispatch to.

Also, we use our regular blueslip helpers.
2020-06-13 15:35:39 -07:00
Steve Howell 78836378e5 node tests: Extract dispatch_subs.js.
This is mostly a code move.  There is a bit of
boilerplate at the top, and I just use
`assert.deepEqual` instead of `assert_same`.

I also use a little wrapper to provide
output like this:

    Starting node tests...
    running tests for dispatch_subs
            test: add
            test: peer add/remove
            test: remove
            test: update
            test: add error handling
            test: peer event error handling

One little piece of code that was obsolete
simply got deleted, not moved.

One of the goals here is to un-stub the
stream_data layer.
2020-06-13 15:35:39 -07:00
Steve Howell 22bf4696f5 refactor: Avoid triggers in stream_edit.
We extract stream_edit.rerender to make
the live-update code easier to follow.

The function should eventually be inlined,
but I want to clean up some other stuff first.
2020-06-13 15:35:39 -07:00
Steve Howell 186973cde2 refactor: Extract add_peer and remove_peer helpers.
These are basically shims for some deeper refactorings.

I basically just try to make the code express the
problems more clearly:

    - use stream_name instead of sub
    - make early-exit more explicit
    - make it clear that add_subscriber needlessly
      requires a name
    - make it clear we have an unnecessary loop

I also fixed some phony data in the test.
2020-06-13 15:35:39 -07:00
Steve Howell b965766f87 refactor: Call compose_fade without triggers.
We are trying to phase out the trigger-event way
of telling modules to do something.

In this case we not only remove the indirection
of the event handler, but we also get to remove
`compose_fade` from the `ui_init` startup sequence.

This also has us update `compose_fade` outside
the loop, although that's only a theoretical
improvement, since I don't think `peer_add` events
every actually include multiple streams.

To make the dispatch tests a little flatter, I
added a one-line change to zjsunit to add
`make_stub` to `global`.

To manually test:
    * have Aaron reply to Denmark (keep compose box open)
    * have Iago add Hamlet to Denmark
    * have Hamlet unsubscribe
2020-06-13 15:35:39 -07:00
Steve Howell 9d9342c2a5 dead code: Remove integration_bot_widget code.
This was work in progress from summer 2017
that never got completed.

See 534aa0b105
2020-06-13 15:35:39 -07:00
Anders Kaseorg 0d6c771baf python: Guard against default value mutation with read-only types.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-13 15:31:27 -07:00
Anders Kaseorg edf411718c openapi: Fix documentation of narrow parameter.
The narrow parameter was incorrectly documented as a one-level array
rather than an array of arrays, and the tests incorrectly expected
this due to a combination of design and implementation bugs.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-13 15:31:27 -07:00
Anders Kaseorg 91a86c24f5 python: Replace None defaults with empty collections where appropriate.
Use read-only types (List ↦ Sequence, Dict ↦ Mapping, Set ↦
AbstractSet) to guard against accidental mutation of the default
value.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-13 15:31:27 -07:00
Anders Kaseorg cfcbf58cd1 do_render_markdown: Remove unused message_user_ids parameter.
It’s unused since commit 7c5f316cb8
(#11586).

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-13 15:31:27 -07:00
Anders Kaseorg 817dbde29a mypy.ini: Remove some unnecessary strict_optional = False lines.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-13 15:31:27 -07:00
Anders Kaseorg 69c0959f34 python: Fix misuse of Optional types for optional parameters.
There seems to have been a confusion between two different uses of the
word “optional”:

• An optional parameter may be omitted and replaced with a default
  value.
• An Optional type has None as a possible value.

Sometimes an optional parameter has a default value of None, or None
is otherwise a meaningful value to provide, in which case it makes
sense for the optional parameter to have an Optional type.  But in
other cases, optional parameters should not have Optional type.  Fix
them.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-13 15:31:27 -07:00
Dinesh 7229088b6c tools: Fix message-screenshot.js failing to take screenshot.
To take screenshots, we log in as Desdemonia who doesn't seem
to have permissions to the streams where we capture screenshots.
This commit fixes it by logging in as Iago.

Also added a waitForNavigation until DOMContentLoaded to
avoid flakes that could occur because of us going to another URL
before Iago gets logged in.
2020-06-13 07:20:32 -04:00
Dinesh a92b31db05 node_tests: Fix incorrect stub of discard button.
The discard button stub in createSaveButtons is set same
as that of save button. This isn't desired because it hides
the save button on running `change_save_button_sate` instead
of hiding discard button. It was previously working as we weren't
testing visibility of save button anywhere.
2020-06-12 19:20:46 -04:00
Alex Vandiver 97b9308781 puppet: Merge multiple postgres roles in `zulip_ops`.
All differences between the primary and replica roles having been
merged, fold the `postgres_common`, `postgres_master`, and
`postgres_slave` roles into just `postgres_appdb`.
2020-06-12 14:57:46 -07:00
Alex Vandiver 55bd31721d puppet: Remove custom `vm.dirty_ratio` and `vm.dirty_background_ratio`.
These values differed between the primary and secondary database
hosts, for unclear reasons.  The differences date back to their
introduction in 387f63deaa.  As the comment in the replica
confguration notes, settings of `vm.dirty_ratio = 10` and
`vm.dirty_background_ratio = 5` matched the kernel defaults for
"newer" kernels; however, kernel 2.6.30 bumped those to 20 and 10,
respectively[1], as a fix for underlying logic now being more correct.

Remove these overrides; they should at very least be consistent across
roles, and the previous values look to be an attempt to tune for a
very much older version of the Linux kernel, which was using an
different, buggier, algorithm under the hood.

[1] 1b5e62b42b
2020-06-12 14:57:46 -07:00
Alex Vandiver f39816e768 puppet: Stop distributing recovery.conf file.
This file controls streaming replication, and recovery using wal-g on
the secondary.  The `primary_conninfo` data needs to change on short
notice when database failover happens, in a way that is not suitable
for being controlled by puppet.

PostgreSQL 12, in fact, removes the use of the `recovery.conf` file[1];
the `primary_conninfo` and `restore_command` information goes into the
main `postgresql.conf` file, and the standby status is controlled by
the presence of absence of an empty `standby.signal` file.

Remove the puppet control of the `recovery.conf` file.

[1] https://pgstef.github.io/2018/11/26/postgresql12_preview_recovery_conf_disappears.html
2020-06-12 14:57:46 -07:00
Steve Howell 09f5090d1f test: Simplify test_multi_user_subscription.
We triage on the event's "op" field instead
of using the type check.

And then we still get the implicit assertion
that we any "add" event produces a dict
for "subscriptions", since we dereference it
in the assertion for "subscribers".
2020-06-12 14:00:53 -07:00
Steve Howell cc7e11f868 node tests: Extract lib/events.js.
The immediate goal here is to make it a little
easier to share code among the node tests.
2020-06-12 14:00:53 -07:00
Alex Vandiver 316498a169 puppet: Remove unnecessary nagios authentication setup.
Since the nagios authentication is stored _in the database_, it is
unnecessary to run if the database is simply a replica of the
production database.  The only case in which this statement would have
an effect is if the postgres node contains a _different_ (or empty)
database, which `setup_disks` now effectively prevents.

Remove the unnecessary step.
2020-06-11 21:01:49 -07:00
Alex Vandiver 0774f54c1b puppet: Move to `setup_disks` to postgres_common.
The tooling should now be run no matter if the node is a primary or
replica.
2020-06-11 21:01:49 -07:00
Alex Vandiver 6f6a0e890a puppet: Run setup_disks based on symlink; remove mdadm dependency.
481613a344 updated the `setup_disks` script to no longer reference
`mdadm`, since we no longer set up RAID on servers.

Update the puppet that would call it to remove the `mdadm` dependency,
and run only if the state is not what it produces -- namely, a symlink
for `/var/lib/postgresql`, which must point to an existent
`/srv/postgresql` directory.
2020-06-11 21:01:49 -07:00
Alex Vandiver 1dc2de5026 puppet: Update setup-disks to be idempotent.
The end state it produces is _either_:

 - `/srv/postgresql` already existed, which was symlinked into
   `/var/lib/postgresql`; postgres is left untouched.  This is the
   situation if `setup_disks` is run on the database primary, or a
   replica which was correctly configured.

 - An empty `/srv/postgresql` now exists, symlinked into
   `/var/lib/postgresql`, and postgres is stopped.  This is the
   situation if `puppet` was just run on a new host, or a
   previously-configured host was rebooted (clearing the temporary
   disk in `/dev/nvme0`)

In the latter case, where `/srv/postgresql` is now empty, any previous
contents of `/var/lib/postgresql` are placed under `/root`,
timestamped for uniqueness.

In either case, the tool should now be idempotent.
2020-06-11 21:01:49 -07:00
Alex Vandiver 8373f5f4b9 puppet: Make parent directories of postgresql.conf
This fixes errors when provisioning a new system (or version of
postgres) when the configuration file cannot be written because its
parent directories do not exist.

Files inherently depend on their containing directories, so no
explicit dependencies are necessary.
2020-06-11 20:56:55 -07:00
Alex Vandiver 9fd7a026ad puppet: Pull postgres data directory into postgres_appdb_base.
The `pg_datadir` variable was only used, and accurate, for CentOS.
Pull it out into `postgres_app_base`, broaden it to being accurate on
Debian-based systems as well, and use it consistently in the
templates.
2020-06-11 20:56:55 -07:00
Alex Vandiver 16c4cea951 puppet: Pull postgres config directory into postgres_appdb_base.
As the previous commit, this is currently only used in tuning, but is
a property of the whole postgres configuration; move it there, as just
the directory, not the file.

Use this directory consistently in the erb templates.  Since we
produce a `pg_hba.conf`, it makes sense that we point to the path that we
know that we explicitly wrote to, for instance.
2020-06-11 20:56:55 -07:00
Alex Vandiver 2a7373b602 puppet: Pull postgres restart config into postgres_appdb_base.
While it is only currently used in the tuning configuration, it is a
property of the base configuration, and fits more clearly into the
case block there.
2020-06-11 20:56:55 -07:00
Anders Kaseorg 59e3c1f5e4 install-semgrep: Upgrade semgrep to 0.10.1.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-11 18:23:17 -07:00
SiddharthVarshney 60010449d2 icon: Import a font-awesome 5 third party icon.
This is a prep commit for replacement of chevron
from sidebars.

This commit will add ellipsis-v icon in svg format downloaded
from font-awesome 5. This has to be done because font-awesome 4.7
(the version we are using) does not have this icon with
circular dots.

And font-awesome 5 as a whole doesn't make sense to upgrade to because
it's intentionally semi-crippled as part of their business plan.

Also include entry in THIRDYPARTY and Licence details.
2020-06-11 17:15:55 -07:00
SiddharthVarshney b82ca432bb right-sidebar: Fix the height of user presence list elements.
This will adjust the height of user presence list elements to
match with the left-sidebar list elements i.e. 23px.

Extra margin is removed to avoid increase in too much spacing
between elements.
2020-06-11 17:14:09 -07:00
SiddharthVarshney 46a98d2eec right-sidebar: Fix menu icon hover color.
This will fix the menu icon hover effect for the day mode.
2020-06-11 17:14:09 -07:00
Anders Kaseorg 48e55875fe lint: Run isort as a linter.
With a change to increase the line length to 100 characters by
tabbott.

Fixes #2665.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-11 16:47:07 -07:00
Anders Kaseorg 365fe0b3d5 python: Sort imports with isort.
Fixes #2665.

Regenerated by tabbott with `lint --fix` after a rebase and change in
parameters.

Note from tabbott: In a few cases, this converts technical debt in the
form of unsorted imports into different technical debt in the form of
our largest files having very long, ugly import sequences at the
start.  I expect this change will increase pressure for us to split
those files, which isn't a bad thing.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-11 16:45:32 -07:00
Anders Kaseorg b666aef2d3 requirements: Upgrade isort to 5.0.0 prerelease from Git.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-11 16:05:58 -07:00
Anders Kaseorg 69730a78cc python: Use trailing commas consistently.
Automatically generated by the following script, based on the output
of lint with flake8-comma:

import re
import sys

last_filename = None
last_row = None
lines = []

for msg in sys.stdin:
    m = re.match(
        r"\x1b\[35mflake8    \|\x1b\[0m \x1b\[1;31m(.+):(\d+):(\d+): (\w+)", msg
    )
    if m:
        filename, row_str, col_str, err = m.groups()
        row, col = int(row_str), int(col_str)

        if filename == last_filename:
            assert last_row != row
        else:
            if last_filename is not None:
                with open(last_filename, "w") as f:
                    f.writelines(lines)

            with open(filename) as f:
                lines = f.readlines()
            last_filename = filename
        last_row = row

        line = lines[row - 1]
        if err in ["C812", "C815"]:
            lines[row - 1] = line[: col - 1] + "," + line[col - 1 :]
        elif err in ["C819"]:
            assert line[col - 2] == ","
            lines[row - 1] = line[: col - 2] + line[col - 1 :].lstrip(" ")

if last_filename is not None:
    with open(last_filename, "w") as f:
        f.writelines(lines)

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-06-11 16:04:12 -07:00
Alex Vandiver b114eb2f10 puppet: Rename env-wal-e to env-wal-g.
It runs wal-g now, not wal-e; make its name respect that.
2020-06-11 15:52:43 -07:00
Alex Vandiver 4fe0444108 puppet: Install wal-g, not wal-e. 2020-06-11 15:52:43 -07:00
YashRE42 6506447bf1 navbar: Only set searchbox text when displaying the searchbox.
Previously, the navbar failed at managing the searchbox text state in
cases where, eg, the user performs navigation by browser history.

This commit resolves the issue by ensuring that the searchbox text is
only (and always) set when the searchbox is made visible, and as such
there is no "state" to manage and we will always display the correct
text.

It also adds a test in `search_legacy.js` to make sure that the search
text is placed as intended.

Fixes: #14771.
2020-06-11 15:49:12 -07:00
YashRE42 939f040674 navbar: Extract `reset_searchbox_text` in tab_bar.
The navbar currently fails at managing the searchbox text state in
cases where, eg, the user performs navigation by browser history.
This is a prep commit that will help resolve the bug.
2020-06-11 15:49:12 -07:00
Anders Kaseorg 0e5946ee5a python: Add noqa comments for the specific star imports we allow.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-11 15:36:43 -07:00
Anders Kaseorg 491fb3c3cf urls: Avoid bare tuples.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-06-11 15:36:43 -07:00