Commit Graph

557 Commits

Author SHA1 Message Date
Steve Howell d9740045a5 refactor: Eliminate checks in build_stream_dict_for_sub.
We eliminate some redundant checks.

We also consistently provide a `subscribers` field
in our stream data with `[]`, even if our users
can't access subscribers.  We therefore bump
the API version and tweak the docs.  (See further
down for a detailed justification of the change.)

Even though it is sometimes fine to have redundant code
that is defensive in nature, some upcoming changes are gonna
move subscriber-related logic out of build_stream_dict_for_sub
for certain codepaths as part of our effort to streamline
the payload for subscribers within page_params.

So we can't rely on the code that I removed here
inside of build_stream_dict_for_sub.

Anyway, it makes more sense to do these checks explicitly
in the validate function.

The code in build_stream_dict_for_sub was almost effectively
a noop, since the validation function was already preventing
us from getting subscriber info.  The only difference it
made was sometimes converting `[]` to `None`, and then
subsequently omitting the subscribers field.

Neither ZT nor the webapp make any distinction between
`[]` or <missing key> for the `subscribers` data in
`page_params`.

The webapp has had this code for a long time (and now
equivalent code elsewhere in this PR):

    if (!Object.prototype.hasOwnProperty.call(sub, "subscribers")) {
        sub.subscribers = new LazySet([]);
    }

The webapp calculates access based on booleans, anyway:

    sub.can_access_subscribers =
        page_params.is_admin || sub.subscribed ||
        (!page_params.is_guest && !sub.invite_only);

And ZT would choke if `subscribers` were missing, except that
it never gets to the relevant code due to other checks:

    def get_other_subscribers_in_stream(<snip>):
        assert stream_id is not None or stream_name is not None

        if stream_id:
            assert self.is_user_subscribed_to_stream(stream_id)

            return [sub
                    for sub in self.stream_dict[stream_id]['subscribers']
                    if sub != self.user_id]
        else:
            return [sub
                    for _, stream in self.stream_dict.items()
                    for sub in stream['subscribers']
                    if stream['name'] == stream_name
                    if sub != self.user_id]

You could make a semantic argument that we should prefer
<missing key> to `[]` when subscribers aren't even available, but
we have precedent from the way that `bulk_get_subscriber_user_ids`
has traditionally populated its result:

    result: Dict[int, List[int]] =
        {stream["id"]: [] for stream in stream_dicts}

If we changed `stream_dicts` to `target_stream_dicts` we
would faciliate a move toward `None`, but it would just cause
headaches for other server code as well as the frontends
(which, to reiterate, already prefer the empty array
for convenience).
2021-01-21 15:04:07 -08:00
Aman Agrawal c685d36821 hipchat_import: Remove tool from codebase.
Remove functions and scripts used by HipChat import tool and
those which will no longer be required in future.
2020-12-23 08:28:49 -08:00
Anders Kaseorg 6c989550ba dependencies: Upgrade JavaScript dependencies.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-12-22 16:14:28 -08:00
Mateusz Mandera 160cc5120a api: Require can_create_users permission to create users via API.
Allowing any admins to create arbitrary users is not ideal because it
can lead to abuse issues.  We should require something stronger that
requires the server operator's approval and thus we add a new
can_create_users permission.
2020-12-21 13:20:21 -08:00
Sutou Kouhei 0d3f9fc855 install: Use PGroonga packages built for PostgreSQL packages by PGDG
Because we always use PostgreSQL packages by PGDG since Zulip 3.0.

Fixes #16058.
2020-12-18 15:38:21 -08:00
ryanreh99 10559c05ee build_pygments_data: Include zulip specific data in langs.
This fixes a bug where the typeahead did not include the
zulip specific langs (such a `quote`, `spoiler` and `math`)
as these weren't passed to the typeahead's source.

Introduced in af64c52166.

Fixes #16862.
2020-12-15 16:41:19 -08:00
Anders Kaseorg 2ab0b3d4fc validator: Reject ISO 8601 dates missing leading zeros.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-12-15 16:36:50 -08:00
Anders Kaseorg b3aa44c914 install-shfmt: Upgrade shfmt from 3.1.2 to 3.2.1.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-12-09 23:07:40 -08:00
Anders Kaseorg 77fdac3579 install-node: Upgrade Node.js to 14.15.1 and nvm to 0.37.2.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-12-09 23:07:40 -08:00
Anders Kaseorg 984a5bb737 dependencies: Upgrade JavaScript dependencies.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-12-09 23:07:38 -08:00
Anders Kaseorg 362d4a6a8d requirements: Upgrade Python requirements.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-12-09 14:24:54 -08:00
Anders Kaseorg af64c52166 build_pygments_data: Build langs without incorrect mutation.
The previous loop generated broken entries for duplicated language
aliases:

"juttle": {"priority": {"priority": 0, "pretty_name": "Juttle"}, "pretty_name": "Juttle"}

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-11-16 18:51:37 -08:00
Anders Kaseorg 13e35bfa94 mypy: Use sqlalchemy-stubs.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-11-16 18:17:41 -08:00
Anders Kaseorg 1275613812 requirements: Upgrade mypy to 0.790.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-11-12 15:44:30 -08:00
Aman Agrawal b2841c7d02 dev_requirements: Upgrade zulint.
This tells users how autofix errors for linters which support it.
This is important since only way to fix prettier errors is
running lint with `--fix` which now the linter will gladly print
with the error.
2020-11-03 14:49:02 -08:00
Vishnu KS 95cd031274 version: Bump provision version for updating email templates. 2020-10-30 11:50:30 -07:00
Anders Kaseorg 7fbd75630f requirements: Upgrade Python requirements.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-10-28 23:39:15 -07:00
Vishnu KS fdea49742c apps: Use GitHub API for generating the web app download link. 2020-10-28 23:04:14 -07:00
Anders Kaseorg 23a289ecd5 install-node: Upgrade Node.js to 12.19.0 and Yarn to 1.22.10.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-10-28 11:45:02 -07:00
Steve Howell 0e7e4f1ad3 api feature level: Bump level for peer_add/peer_remove. 2020-10-26 07:18:08 -04:00
Anders Kaseorg 0f11e2a361 dependencies: Upgrade JavaScript dependencies.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-10-24 02:09:14 -07:00
Anders Kaseorg efa8dd3a47 compilemessages: Sort language list with Unicode Collation Algorithm.
Right now the list of languages in Display settings → Default language
is sorted in an unintuitive order due to the varying case conventions:

British English
Chinese (Taiwan)
Deutsch
English
Hindi
Indonesian (Indonesia)
Lietuviškai
Magyar
Malayalam
Nederlands
Português
Română
Tiếng Việt
Türkçe
català
español
français
galego
italiano
polski
suomi
svenska
česky
Русский
Українська
български
српски
فارسی
தமிழ்
日本語
简体中文
繁體中文
한국어

Fix the sort to use the locale-independent Unicode Collation
Algorithm:

British English
català
česky
Chinese (Taiwan)
Deutsch
English
español
français
galego
Hindi
Indonesian (Indonesia)
italiano
Lietuviškai
Magyar
Malayalam
Nederlands
polski
Português
Română
suomi
svenska
Tiếng Việt
Türkçe
български
Русский
српски
Українська
فارسی
தமிழ்
한국어
日本語
简体中文
繁體中文

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-10-21 15:22:18 -07:00
Anders Kaseorg 5c6202ca97 notifications: Use a dynamic SVG favicon for unread counts.
Closes #2304.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-10-21 07:43:00 -07:00
Anders Kaseorg 8152cd8566 requirements: Upgrade Python requirements.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-10-19 11:54:14 -07:00
Anders Kaseorg d81a93cdf3 requirements: Upgrade markdown to 3.3.1.
Upstream has slightly changed the whitespace around stashes.  Take
this opportunity to clean up the extra blank lines we were outputting.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-10-19 11:54:14 -07:00
Anders Kaseorg 449b16be59 dependencies: Use patched source-map with a performance fix.
https://github.com/mozilla/source-map/issues/370

This makes Webpack about 9% faster.  Before:

$ multitime -n9 -s0 node_modules/.bin/webpack --config-name=frontend --display=errors-only -p
===> multitime results
1: node_modules/.bin/webpack --config-name=frontend --display=errors-only -p
            Mean        Std.Dev.    Min         Median      Max
real        18.243      0.107       18.090      18.236      18.443
user        27.913      0.188       27.714      27.843      28.251
sys         2.028       0.043       1.933       2.039       2.074

After:

$ multitime -n9 -s0 node_modules/.bin/webpack --config-name=frontend --display=errors-only -p
===> multitime results
1: node_modules/.bin/webpack --config-name=frontend --display=errors-only -p
            Mean        Std.Dev.    Min         Median      Max
real        16.686      0.085       16.542      16.684      16.885
user        25.965      0.167       25.559      26.022      26.163
sys         1.965       0.064       1.807       1.998       2.010

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-10-15 14:38:41 -07:00
Anders Kaseorg 2a4a775e2b requirements: Install PycURL for Thumbor.
Fixes “thumbor:WARNING pycurl usage is advised. It could not be loaded
properly. Verify install...”.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-10-14 10:52:32 -07:00
Anders Kaseorg 4fe06a141e eslint: Enable eslint-plugin-unicorn.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-10-07 16:00:33 -07:00
sahil839 78b98d8067 realm: Add wildcard_mention_policy setting.
We add a new wildcard_mention_policy setting to handle wildcard
mentions in large streams, with a wide range of policies available to
organizations.

We set the default to the safe option for preventing accidental spam:
only stream administrators being able to use wildcard mentions in
large streams.
2020-10-01 12:18:03 -07:00
Anders Kaseorg cfd93096b5 openapi: Remove yamole.
As explained in the previous commit, yamole preprocessed allOf with an
algorithm that is not standards compliant.  We replicate that
algorithm, but importantly, we only use it for our own code and not
for building the openapi_core RequestValidator.

This improves the time taken by OpenAPISpec().check_reload() from
1.69s to 0.53s, nearly all of which is inside
openapi_core.create_spec.

Closes #10484.  Significantly improves #16068.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-09-29 16:47:10 -07:00
Anders Kaseorg e4931c5bc4 people: Remove unorm polyfill.
We no longer support IE11.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-09-25 16:54:25 -07:00
Tim Abbott 089225ea36 version: Update latest desktop release version. 2020-09-25 13:00:46 -07:00
Anders Kaseorg 58808c2362 dependencies: Upgrade JavaScript dependencies.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-09-24 16:33:45 -07:00
Anders Kaseorg 3a39c6f67f dependencies: Downgrade simplebar to 5.2.1.
SimpleBar 6.0.0-beta.2 through -beta.6 are built with ES6 syntax (I
assume inadvertently: https://github.com/Grsmto/simplebar/issues/523),
and its latest tag has moved back to 5.2.1 anyway.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-09-24 16:13:47 -07:00
Alex Vandiver 4361ce1246 markdown: Use tlds package to keep updated list of TLDs.
Also remove a useage of "blacklist."
2020-09-21 21:03:29 -07:00
Sumanth V Rao 2b9f7916c5 api: Document markdown changes for code blocks and bump API_FEATURE_LEVEL. 2020-09-18 17:15:47 -07:00
Sumanth V Rao 4a2791e2a3 tools/build_pygments_data: Map pygments language aliases to lexer name.
We need this information in the frontend to:
    * Display the 'view in playground' option for locally echoed messages.
    * When we add a UI settings for realm admins to configure their
      playground choices, we'll need to use these canonicalized aliases
      for displaying the option.

Hence, this tweaks the tool which generates pygments_data.json to contain
the data we need.

Bumping major PROVISION_VERSION since folks need to provision in both
directions.

Tests amended.
2020-09-18 17:12:26 -07:00
Anders Kaseorg b7874ac82e install-node: Upgrade Node.js to 12.18.4 and Yarn to 1.22.5.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-09-15 16:33:28 -07:00
Anders Kaseorg 7c755f1394 dependencies: Upgrade JavaScript dependencies.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-09-15 16:33:28 -07:00
Anders Kaseorg e84c7fb09f requirements: Remove django-cookies-samesite.
Its functionality was added to Django upstream in 2.1.  Also remove
the SESSION_COOKIE_SAMESITE = 'Lax' setting since it’s the default.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-09-14 17:23:56 -07:00
Anders Kaseorg fd59db292f requirements: Upgrade Python requirements.
Fixes #16335.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-09-13 20:59:09 -07:00
Anders Kaseorg dfab09b17d markdown: Replace hyperlink requirement with urllib.parse.
The previous code only worked by accident and hyperlink 20.0.0 breaks
it.

>>> hyperlink.parse("example.com").replace(scheme="https")
DecodedURL(url=URL.from_text('https:example.com'))

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-09-13 15:37:28 -07:00
Anders Kaseorg d9860d40a6 dependencies: Upgrade JavaScript dependencies.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-09-02 19:34:37 -07:00
Priyank Patel 5c53c44ee5 dependencies: Upgrade babel-plugin-rewire-ts to v1.4.0.
This version keeps the internal and external state of the ES6 module
in sync so its state is similar to a CommonJS module.  Since we don't
need to ask rewire for the internal state of the module, the tests
will be same as they were before the TS/ES6/remove from window
migrations.
2020-09-01 19:55:58 -07:00
Anders Kaseorg b9184eee65 eslint: Use eslint-import-resolver-webpack.
This uses our Webpack configuration to more accurately predict how our
imports will actually behave; for example, it automatically resolves
the .ts extension.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-09-01 16:43:02 -07:00
Priyank Patel 05f3d552d3 yarn.lock: Update lockfile after phantomjs-prebuild removal.
I think this needs an update because 2f5f5d7749,
which removes phantomjs-prebuilt didn't update the yarn.lock.
2020-08-31 11:21:15 -07:00
Priyank Patel c1ba648b4f dependencies: Add xvfb dependency.
The node package allow use to control xvfb apt package in puppeteer
tests. This help us create a fake display so we can run puppeteer in
headful (headless: false) mode, which is required to use the chrome
extension desktop capture API.
2020-08-30 23:38:52 -07:00
Tim Abbott 2f5f5d7749 dependencies: Remove PhantomJS from development environment. 2020-08-30 17:16:02 -07:00
Dinesh 2d22d88700 casper: Remove few traces of casper.
Now that all casper tests have been migrated to
puppeteer, there's no need for having casper
related things.

Removed the casperjs package and removed/replaced
casper in few places with puppeteer.

Only removed few of them which I'm confident
about. Also didn't make any changes in docs
as it would be easier to remove them while
adding puppeteer docs.
2020-08-30 17:16:02 -07:00
Aman 46a86e218e requirements: Upgrade zulint.
This upgrade allows us to exclude a line multiple times for a
pattern.
2020-08-24 16:10:10 -07:00