Commit Graph

39279 Commits

Author SHA1 Message Date
Tim Abbott 089225ea36 version: Update latest desktop release version. 2020-09-25 13:00:46 -07:00
Tim Abbott 94a9fa1891 event_schema: Add documentation and rename a few functions.
This should help make this revised subsystem readable for more new
contributors.  We still need to make updates to the high-level
documentation.
2020-09-25 12:53:00 -07:00
Steve Howell 5b7c9c4714 test_events: Add check_realm_user_remove. 2020-09-25 11:43:20 -07:00
Steve Howell 7bb7f2943f event_schema: Finish extraction with realm_emoji/update.
We now no longer define any schemas in test_events--all
of them are in event_schema, which helps our tooling
cross-check schemas for openapi and node tests.
2020-09-25 11:43:20 -07:00
Steve Howell ae4d083a5a event_schema: Extract check_realm_domains_*. 2020-09-25 11:43:20 -07:00
Steve Howell 298bed9fa1 event_schema: Split check_update_message_flags. 2020-09-25 11:43:20 -07:00
Steve Howell f6e0171d02 event_schema: Split check_reaction into add/remove.
It happens that whether you add a reaction or remove
a reaction, we send the exact same fields, just using
a different op code.

This sort of symmetry is actually kind of rare, as
usually "add" events have more fields, and "remove" events
might just send an id of something to remove.

Our openapi schema treats these as two seperate events,
so we are more consistent with it, and it helps our
schema-checking tooling for node fixtures, too.

Note that we now have to exempt the two events from
our openapi checks, due to the is_mirror_dummy field
in the deprecated user block.  We can decide how to
handle this later--one possibility is to just add it
as an optional field on the event_schema side.
2020-09-25 11:43:20 -07:00
Steve Howell b7b2546f44 event_schema: Extract check_subscription_update.
Note that we use value_type for value instead of
bool, since properties can be non-bool things
like color, which we just don't test now.  We
should test them.

We more than compensate for this by checking
the actual value of the value in
check_subscription_update.
2020-09-25 11:43:20 -07:00
Steve Howell b920ebce81 event_schema: Extract check_has_zoom_token. 2020-09-25 11:43:20 -07:00
Steve Howell 0c4286222f event_schema: Extract check_realm_update_dict. 2020-09-25 11:43:20 -07:00
Steve Howell 6ec6525624 event_schema: Extract check_delete_message.
There is a legacy format where we send
singular "message_id" instead of plural
"message_ids".

Then there are different fields for "private"
and "stream" message types.
2020-09-25 11:43:20 -07:00
Steve Howell 88165aee6b event_schema: Extract check_user_group_update. 2020-09-25 11:43:20 -07:00
Steve Howell aaaac11661 event_schema: Extract check_user_group_remove. 2020-09-25 11:43:20 -07:00
Steve Howell 1b7af13f37 event_schema: Extract check_user_group_remove_members. 2020-09-25 11:43:20 -07:00
Steve Howell 19b7739065 event_schema: Extract check_user_group_add_members. 2020-09-25 11:43:20 -07:00
Steve Howell 4084f0b949 event_schema: Extract check_realm_user_add.
Note that we make the schema for profile_data
slightly more realistic, but it doesn't actually get
exercised by our current tests (apart from
making sure it's a dict), since we don't have
profile data for our test realm.

We also don't have the optional fields for bots,
since our tests don't exercise that, nor
delivery_email.

So we exempt realm_user_add_event from openapi
checks for now.

When we try to match the openapi specs better, we
will probably want to add a few tests to test_events.

Obviously getting good coverage for adding users
would be nice for all these scenarios:

    * delivery_email matters
    * bots
    * realm has profile fields
2020-09-25 11:43:19 -07:00
Steve Howell dc2176a965 event_schema: Extract check_presence. 2020-09-25 11:43:19 -07:00
Steve Howell 6c74a44697 data_types: Generalize StringDictType.
This is a prep commit for supporting "presence"
events, where the key of the dictionary is some
arbitrary string like "website" but the value
of the dictionary is another dictionary itself
with keys that are more like variable names.
2020-09-25 11:43:19 -07:00
Steve Howell 4f3d5f2d87 event_schema: Extract check_realm_filters.
We have some known issues with representing
tuples in openapi, so we exempt realm_filters
from the relevant check.
2020-09-25 11:43:19 -07:00
Steve Howell e40a5400e5 event_schema: Extract check_muted_topics.
This also forces us to create TupleType.

We exempt this from the openapi check,
since we haven't figured out how to model
tuples in openapi with the same precision
as event_schema (and it may be impossible).

Long term we just want to stop dealing in
tuples, of course.
2020-09-25 11:43:19 -07:00
orientor 91ca1afe98 data_type: Add StringDict data type.
StringDict is a data type for representing dictionaries where
all keys and values are strings. Add this data type to data_types.py
and edit other files so that this data type is put to use and tested.

(slightly tweaked by @showell to remove a comment and shorten
a var name now that we have a proper data type)
2020-09-25 11:43:19 -07:00
Steve Howell 78a2059b8d event schema: Extract attachment checkers. 2020-09-25 11:43:19 -07:00
Steve Howell 4a947c971d event_schema: Extract check_realm_export.
These are all trivial transformations.

Note that we don't insist timestamps are
floats; the NumberType class allows ints
too.
2020-09-25 11:43:19 -07:00
Steve Howell d28c01284c event_schema: Extract check_hotspots.
This forces us to introduce a NumberType.
2020-09-25 11:43:19 -07:00
Steve Howell cf26151cea event_schema: Use realm_user_person_types.
For realm_user events, we now structure the
person type as a union of dicts, which is
more consistent with how we model this in
our openapi spec.
2020-09-25 11:43:19 -07:00
Steve Howell 10952394b0 test_events: Use int value of message_retention_days.
We also make our schema in event_schema reflect this,
which in turn makes us match the already accurate
openapi spec, so we no longer need to exempt four
types of events from our sanity checks.
2020-09-25 11:43:19 -07:00
Steve Howell 73e7f7edec check-node-fixtures: Compare python/openapi schemas.
We might want to rename the tool to something more
general now, since we are really reconciling three
things:

    - node fixtures
    - event_schema checkers for test_events
    - openapi specs

The way we compare python and openapi schemas is
as follows:

    - first convert openapi schemas to be build
      from DictType, ListType, etc. with from_opeapi

    - do a diff on the schemas

Most of the new code is just having the FooType
family of classes serialize themselves with schema().
2020-09-25 11:43:19 -07:00
Steve Howell 2b76eb767f event_schema: Change propagate_mode to an enum. 2020-09-25 11:43:19 -07:00
Steve Howell aca641a4d1 refactor: Extract data_types module.
Defining types with an object hierarchy
of type classes will allow us to build
functionality that was impossible (or
really janky) with the validators.py
approach of composing functions.

Most of the changes to event_schema.py
were automated search/replaces.

This patch doesn't really yet take
advantage of the new FooType classes,
but we will use it soon to audit our
openapi specs.
2020-09-25 11:43:19 -07:00
Alex Vandiver 36ea307fbf puppet: Depend other changes on sharding.py validation.
Use the validation of the tornado sharding config that
`stage_updated_sharding` does, by depending on it.  This ensures that
we don't write out a supervisor or nginx config based on a
bad (e.g. non-sequential) list of tornado ports.
2020-09-25 10:52:40 -07:00
Alex Vandiver c0e240277b tornado: Remove fingerprinting, write out .tmp files always.
Fingerprinting the config is somewhat brittle -- it requires either
custom bootstrapping for old (fingerprint-less) configs, and may have
false-positives.

Since generating the config is lightweight, do so into the .tmp files,
and compare the output to the originals to determine if there are
changes to apply.

In order to both surface errors, as well as notify the user in case a
restart is necessary, we must run it twice.  The `onlyif`
functionality cannot show configuration errors to the user, only
determine if the command runs or not.  We thus run the command once,
judging errors as "interesting" enough to run the actual command,
whose failure will be verbose in Puppet and halt any steps that depend
on it.

Removing the `onlyif` would result in `stage_updated_sharding` showing
up in the output of every Puppet run, which obscures the important
messages it displays when an update to sharding is necessary.
Removing the `command` (e.g. making it an `echo`) would result in
removing the ability to report configuration errors.  We thus have no
choice but to run it twice; this is thankfully low-overhead.
2020-09-25 10:52:40 -07:00
Anders Kaseorg 58808c2362 dependencies: Upgrade JavaScript dependencies.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-09-24 16:33:45 -07:00
Anders Kaseorg 3a39c6f67f dependencies: Downgrade simplebar to 5.2.1.
SimpleBar 6.0.0-beta.2 through -beta.6 are built with ES6 syntax (I
assume inadvertently: https://github.com/Grsmto/simplebar/issues/523),
and its latest tag has moved back to 5.2.1 anyway.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-09-24 16:13:47 -07:00
Wes Galbraith 9645959ac4 populate_db: Add emoji reactions to development environment database.
This change adds automated generated emoji reactions to the data in
the development environment's database.

Fixes part of #14991.
2020-09-23 16:10:37 -07:00
Steve Howell f29b2884ca bitbucket2: Format user info consistently.
We now use get_user_info() to format all of our
users in messages.
2020-09-23 15:31:38 -07:00
Steve Howell e0b6619dac bitbucket2: Simplify how we display user for fork events.
Even before GDPR changes, it was strange that we displayed
users differently for fork events vs. all other events.

After GDPR, we don't even get the `username` field any
more.

So now we simply use `display_name` if available, and then
we try `nickname`.

See https://developer.atlassian.com/cloud/bitbucket/bitbucket-api-changes-gdpr/
for more context.
2020-09-23 15:31:38 -07:00
Steve Howell 1ef8d79352 bitbucket: Decouple BITBUCKET_FORK_BODY between versions.
We were trying to share the same format string between
the two different versions of bitbucket, but this only
creates confusion, as the two versions are only close
enough to be confusing.

The format string might be the same, but the semantics
are different, as well as the eventual outputs.

For example, the {username} piece here is simple in version
2, but in version 3 we append a url to the user's name.
2020-09-23 15:31:38 -07:00
Alex Vandiver b06253049a stripe: Change `assert_called` to `assert_called_once`.
Prefer using `assert_called_once` to protect against places where a
mock might be reused, and in so doing have been previously called,
thus making the second usage of `assert_called` not assert anything of
note.
2020-09-23 15:29:47 -07:00
Alex Vandiver ab4f6b54ea stripe: Do not log credit card issues as errors.
Problems with the card itself should not be logged as errors -- while
perhaps notable in aggregate, they are not worthy of being logged to
Sentry, for instance.

Downgrade these to `info`; continue to log other problems at the
`error` level.  This updates tests for this change, and in so doing
corrects a test that does not do its job, due to a missing
`reset_mock`.
2020-09-23 15:29:47 -07:00
sahil839 fe370debe5 tests: Rename stream messages tests in test_message_send.py.
This commit renames 'test_message_to_self' and
'test_api_message_to_self' tests to
'test_message_to_stream_by_name' and
'test_api_message_to_stream_by_name' to depict
the actual purpose of these tests.
2020-09-23 15:28:31 -07:00
Aman Agrawal 5ae2325979 events: Don't send presence data for web public guests.
We disable presence and hide list of users in right sidebar
via setting realm_presence_disabled to false here for
web public guests.
2020-09-23 12:11:22 -07:00
Aman Agrawal 5153a036f2 events: Disable topic edit & msg edit/delete for web public guest.
To ensure web public guests cannot change any data, it's natural
that we disable topic & msg edit / delete ops.
2020-09-23 12:11:22 -07:00
Aman Agrawal 3ec23e1a9d fetch_initial_state_data: Handle case of web public guests.
user_profile will be None for web_public_guests here.  Hence, for
settings (of which most be inaccessible by web public guest),
which require a user_profile, we either set an empty value for
them or set them to a default value. This will help render
the frontend or extend support to our clients without breaking
a lot of code.

Tweaked by tabbott to add many comments.
2020-09-23 12:11:15 -07:00
Aman Agrawal 9cabd8f9cb process_client: Don't update activity of unauthenticated users.
This allows wrapper `add_logging_data` to be used to add
client information for unauthenticated users (or web public guests).
2020-09-23 12:07:06 -07:00
Aman Agrawal 6012d3cff2 get_raw_user_data: Support acting_user=None.
This already had support for web_public_guest, fixed mypy
annotation.
2020-09-23 12:07:06 -07:00
Aman Agrawal c88b4cba60 format_user_row: Support acting_user=None. 2020-09-23 12:07:06 -07:00
Aman Agrawal 48492a0633 fetch_initial_state_data: Pass realm as independent parameter.
This removes dependency of the function on user_profile to get the
realm, which will be useful when user_profile is None in case of web
public guests.
2020-09-23 12:06:54 -07:00
Gittenburg 6e95809cc4 emoji_picker: Fix search input length.
This was probably a regression from our upgrade to bootstrap 2.3.2.
2020-09-23 11:33:27 -07:00
Anders Kaseorg 9238813135 js: Use destructuring for require statements.
This allows import/order to auto-fix blocks including these
statements.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-09-23 09:06:07 -04:00
Anders Kaseorg 5216b70691 lint: Remove custom rule already enforced by Prettier.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-09-22 18:11:21 -07:00