Commit Graph

38043 Commits

Author SHA1 Message Date
Steve Howell 92a552d2ad REQ: Check value in update_user_custom_profile_data.
This tightens our checking of user-supplied data
for this endpoint:

    path('users/me/profile_data', rest_dispatch,
         {'PATCH': 'zerver.views.custom_profile_fields.update_user_custom_profile_data',
         ...

We now explicitly require the `value` field
to be present in the dicts being passed in
here, as part of `REQ`.  There is no reason
that our current clients would be sending
extra fields here, and we would just ignore
them anyway, so we also move to using
check_dict_only.

Here is some relevant webapp code (see settings_account.js):

    fields.push({id: field.id, value: user_ids});
    update_user_custom_profile_fields(fields, channel.patch);

    settings_ui.do_settings_change(method, "/json/users/me/profile_data",
                                   {data: JSON.stringify([field])}, spinner_element);

The webapp code sends fields one at a time
as one-element arrays, which is strange, but
that is out of the scope of this change.
2020-06-25 10:54:15 -07:00
Priyank Patel 8391250e60 puppeteer: Fix a flake in 02-message-basics.js.
The flake was caused due to the fact that current_msg_list was not
populated with any messages in rare cases. We were missing a check
to guard against current message list having no messages. The
failure screenshot show no messages to prove this.

Relevant error:
Evaluation failed: TypeError: Cannot read property 'raw_content' of undefined
2020-06-25 10:33:23 -07:00
Mateusz Mandera 85d4536486 docs: Update some comments for the new release versioning scheme.
With the new scheme, the equivalent of 2.3 is 4.0.
2020-06-25 10:33:03 -07:00
Anders Kaseorg 58bc175526 models: Avoid undocumented sre_constants module.
>>> sre_constants.error is re.error
True

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-24 23:18:17 -07:00
Anders Kaseorg 78ed602160 isort: Treat tc_aws and thumbor as third-party.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-24 21:02:45 -07:00
Eeshan Garg 267cb8999b requirements: Upgrade fakeldap dependency to the latest version. 2020-06-24 20:14:32 -07:00
Anders Kaseorg 123d527c9e check-issue-labels: Fix configparser import.
Commit ccf2792c1c (#7296) migrated this
incorrectly.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-24 19:26:02 -07:00
Tim Abbott 3bdf4ef21f Revert "search: Enable pills-based UI for testing."
This reverts commit 90e180f974.

This was meant to stay on the chat.zulip.org branch.
2020-06-24 16:58:58 -07:00
Tim Abbott e46bbf18eb docs: Change next planned major release to 3.0.
After some discussion, everyone seems to agree that 3.0 is the more
appropriate version number for our next major release.  This updates
our documentation to reflect that we'll be using 3.0 as our next major
release.
2020-06-24 16:27:27 -07:00
Alex Vandiver e0b725be63 test-install: Wait for network in the lxc container.
Ubuntu 20.04 "focal" comes up to runlevel 5 several seconds before it
is able to successfully resolve hosts, causing `prepare-base` to fail
while fetching from the apt repositories.

Add an additional check to verify that outbound networking is running
before returning from `lxc-wait`.
2020-06-24 16:07:20 -07:00
Aman Agrawal fc31eaa356 recent_topics: Show a line below the focused element. 2020-06-24 15:37:46 -07:00
Aman Agrawal 4cd7f2a329 recent_topics: Wrap focusable elements around a div.
This avoids hard coding the focusable elements.
2020-06-24 15:33:25 -07:00
orientor 8ab6182683 fetch_api_key: Return `email` in json_response.
`/api/v1/fetch_api_key`'s response had a key `email` with the user's
delivery email. But its JSON counterpart `/json/fetch_api_key`, which
has a completely different implementation, did not return `email` in
its success response.

So to avoid confusion, the non-API endpoint, `/json/fetch_api_key`
response has been made identical with it's `/api` counterpart by
adding the `email` key. Also it is safe to send as the calling user
will only see their own email.
2020-06-24 15:13:31 -07:00
Steve Howell 2dd1ccef15 check_list: Make sub_validator a required argument.
This allows us to avoid the overload decorators.
2020-06-24 15:04:59 -07:00
Steve Howell de2285222d tests: Use check_tuple for realm filters.
This also sets the stage to simplify
our check_list validator.
2020-06-24 15:01:57 -07:00
Steve Howell c7b82d3ece mypy: Use tuples for muted_topics.
We now have our muted topics use tuples internally,
which allows us to tighten up the annotation
for get_topic_mutes, as well as our schema
checking.

We want to deprecate sub_validator=None
for check_list, so we also introduce
check_tuple here.  Now we also want to deprecate
check_tuple, but it's at least isolated now.

We will use this for data structures that are tuples,
but which are sent as lists over the wire.  Fortunately,
we don't have too many of those.

The plan is to convert tuples to dictionaries,
but backward compatibility may be tricky in some
places.
2020-06-24 15:01:57 -07:00
Steve Howell 8e8228535e tests: Use check_dict for external_authentication_methods
This is still imperfect, but the only goal for now is
to make sure that `check_list` always get a sub_validator.
2020-06-24 15:01:57 -07:00
sahil839 fe35f1e9bd invites: Do not return multi-use invites to non-admin users.
This commit changes do_get_user_invites function to not return
multiuse invites to non-admin users. We should only return multiuse
invites to admins, as we only allow admins to create them.
2020-06-24 15:01:41 -07:00
Alex Vandiver 8499b2c0dc test-install: Add support for focal. 2020-06-24 13:03:13 -07:00
Alex Vandiver 8719fba3c4 test-install: Stop installing postgres-10.
As in the previous commit, we can no longer pre-install the wrong
version of postgres.  Unfortunately, this leaves it out of the base
image and thus makes testing installs longer.
2020-06-24 13:03:13 -07:00
Alex Vandiver 31f1f10501 installer: Halt if wrong version of PostgreSQL is already installed.
49a7a66004 and immediately previous commits began installing
PostgreSQL 12 from their apt repository.  On machines which already
have the distribution-provided version of PostgreSQL installed,
however, this leads to failure to apply puppet when restarting
PostgreSQL 12, as both attempt to claim the same port.

During installation, if we will be installing PostgreSQL, look for
other versions than what we will install, and abort if they are
found.  This is safer than attempting to automatically uninstall or
reconfigure existing databases.
2020-06-24 12:57:38 -07:00
Alex Vandiver 814198d649 installer: Abstract out version of postgres installed.
This allows for installing from-scratch with a different pinned
version of PostgreSQL, and provides a single place to change when the
default should increase.
2020-06-24 12:57:38 -07:00
Alex Vandiver ca9d27175b installer: Write PostgreSQL version based on puppet classes.
Using `/etc/init.d/postgresql` as the detection of if Postgres is on
the server is incorrect, because this line runs _before_ puppet and
any packages are installed.  Thus, it cannot tell the difference
between a new Ubuntu one-host first-time-install without PostgreSQL
yet, and one which is merely a front-end and will never have
PostgreSQL.  This leads to failures in first-time installs:

```
Error: Evaluation Error: Error while evaluating a Function Call,
  Could not find template 'zulip/postgresql//postgresql.conf.template.erb'
```

The only way to detect if PostgreSQL will be present in the _end_
state of the install is to examine the puppet classes that are
applied.

To do this, we must inspect `PUPPET_CLASSES`.  Unfortunately, this can
be fragile to subclassing (e.g. `zulip_ops::postgres_appdb`).  We
might desire to use `puppet apply --write-catalog-summary` to deduce
the _applied_ classes, which would unroll the inheritance; however,
this causes a chicken-and-egg problem, because `zulip.conf` must be
already written out (including a value for `postgresql.version`, if
necessary!) before such a puppet run could successfully complete.

Switch to predicating the `postgresql.version` key on the puppet
classes that are known to install postgres.
2020-06-24 12:57:38 -07:00
Alex Vandiver 253246185f installer: Update documentation.
Where appropriate, documentation wording is shared with
docs/production/install.md
2020-06-24 12:57:38 -07:00
Alex Vandiver 85dbb13c56 installer: Abstract out apt/yum divide into a variable.
This check is done in several places, using a somewhat fragile `case`
statement; move it into an explicit variable.
2020-06-24 12:57:38 -07:00
Alex Vandiver 876ee4a8ed installer: Remove code specific to stretch or xenial.
Support for Xenial and Stretch was removed (5154ddafca, 0f4b1076ad,
8944e0ad53, 79acd5ae40, 1219a2e854), but not all codepaths were
updated to remove their conditionals on it.

Remove all code predicated on Xenial or Stretch.  debathena support
was migrated to Bionic, since that appears to be the current state of
existing debathena servers.
2020-06-24 12:57:38 -07:00
Alex Vandiver e4899eae8b installer: Sync the claimed supported distros with the check.
0f4b1076ad removed Ubuntu 16.04 "xenial" and Debian 9 "stretch" from
the printed list of supported operating systems, but left them in the
verification check that controls if that message is printed,
effectively continuing to support them.

Conversely, 439f0d3004 added Ubuntu 20.04 "focal" to the check, but
not to the printed list.

Synchronize to check and print the right supported distributions:
Ubuntu 18.04 "bionic", Ubuntu 20.04 "focal", and Debian 10 "buster".
2020-06-24 12:57:38 -07:00
Alex Vandiver 58cb7cecd8 installer: Remove `--remote-postgres`, redundant with `--no-init-db`.
The previous commit removed the only behavior difference between the
two flags; both of them skip user/database creation, and the tables
therein.

Of the two options `--no-init-db` is more explicit as to what it does,
as opposed to just one facet of when it might be used; remove
`--remote-postgres`.
2020-06-24 12:57:38 -07:00
Alex Vandiver 7c6a25a43d installer: Group and unify ordering of installer options.
This also adds the missing `--no-overwrite-settings` option to
`--help`.
2020-06-24 12:57:38 -07:00
Alexandre Gigleux fa1071335b
tools: Configure Zulip to be scannable by SonarCloud.
Apparently, SonarCloud fails on our repository for unknown reasons.

But it runs if one restricts to just Python and HTML files.
2020-06-24 12:41:17 -07:00
Tim Abbott 148f74d3a7 css: Move night theme pygments CSS to night_mode.scss.
The previous architecture did not work properly with the automatically
detected night theme, resulting in a weird mix of the night and day
themes on code blocks.

I'm not thrilled with the requirement this imposes that all of our
night theme CSS needs to be in one file, but we do need to get a quick
fix out here.

Fixes #15554.
2020-06-24 12:35:02 -07:00
Tim Abbott 2a6ef2b9eb models: Add missing migrations from invite_as change.
I neglected to `git add` after doing `renumber-migrations` for
1f8f227444.
2020-06-24 11:39:01 -07:00
Tim Abbott ac9ecbe8de docs: Clarify variable variants in translating docs. 2020-06-24 11:34:00 -07:00
arpit551 e19aef569a ci: Update Circle CI images.
Removed postgresql and python 2 packages when creating docker images
for Circle CI. Updated the images.
2020-06-24 11:19:47 -07:00
sahil839 136c005f3f settings_invites: Remove invited_as_values map.
This commit removes invited_as_values map in settings_invites.js.
This object has been removed to avoid duplication as we already
have role values in settings_config.js.

A similar map is created from settings_config.user_role_values
in settings_config.js and is used to populate invited_as_text
for invites.
2020-06-24 11:09:07 -07:00
sahil839 1f8f227444 models: Update values of PreregistrationUser.invite_as dict.
This commit changes the PreregistrationUser.invite_as dict to have
same set of values as we have for UserProfile.role.

This also adds a data migration to update the already exisiting
PreregistrationUser and MultiuseInvite objects.
2020-06-24 11:09:07 -07:00
Tim Abbott b7b173d2ae realm: Fix type-checking for message_retention_days.
The best way to handle this is to have the potentially string-typed
value be a separate variable.
2020-06-24 11:01:34 -07:00
Tim Abbott 90e180f974 search: Enable pills-based UI for testing. 2020-06-24 10:44:37 -07:00
Rohitt Vashishtha 6ba8fb518b markdown-timestamp: Show flatpickr using typeaheads.
We leverage the composebox typeaheads to show flatpickr to pick dates
and times for the !time syntax.

We use moment.js to try and parse the time from current token. If we
are successful, we initialize flatpickr with the parsed time, else we
default to using the current time.
2020-06-24 10:44:37 -07:00
Mateusz Mandera 890cafac11 retention: Use batch size of 100 for stream messages.
Streams can have lots of subscribers, meaning that the archiving process
will be moving tons of UserMessages per message. For that reason, using
a smaller batch size for stream messages is justified.

Some personal messages need to be added in test_scrub_realm to have
coverage of do_delete_messages_by_sender after these changes.
2020-06-24 10:41:00 -07:00
Mateusz Mandera 0c6497d43a retention: Add restore_retention_policy_deletions_for_stream function. 2020-06-24 10:40:38 -07:00
Mateusz Mandera 468f8cf488 retention: Improve logging of transactions. 2020-06-24 10:40:38 -07:00
Pragati Agrawal 33fa307c95 org settings: Display empty input when message retention days is forever.
Currently we display -1 in input box of id_realm_message_retention_days
when realm_message_retention_days is -1, which isn't user friendly.
Displaying the input box as empty is more intuitive.

And if the user tries to submit an empty input box we throw invalid JSON
error that isn't user friendly either, so fixed that too. In the ideal
case, we shouldn't send the request at first place to the backend when we
don't have any input.
2020-06-24 10:38:58 -07:00
Pragati Agrawal 1562ec758e org settings: Use 'forever' value instead of -1 for message_retention_days.
Currently, we use -1 as the Realm.message_retention_days value to retain
message forever unless specified at stream level for a particular stream,
that is, no policy set at the realm level. But this is incoherent with what
we use for Stream.message_retention_days where -1 means

> disable retention policy for this stream unconditionally

that can be confusing from an API standpoint.

So instead of trying some hack to reset the value to NULL or using some
other value like -2 for RETAIN_MESSAGE_FOREVER and use that for API. It is
much more intuitive to use a string like 'forever' that can be mapped to
RETAIN_MESSAGE_FOREVER at the backend. And this is similar to what we use
for streams settings as well.
2020-06-24 10:38:58 -07:00
Pragati Agrawal 7b3f4755ff streams: Fix bug that allows 0 value for message_retention_days. 2020-06-24 10:38:58 -07:00
Pragati Agrawal 72dd5bfa77 org settings: Use get_input_element_value style function to set value.
`get_input_element_value()` function is more reliable to detect the input
element type and extract it's value. But the current way of setting the
value of input elements relies on first checking the `property_value` type.
Which is fine, but for the cases when the property value is null, and we
want to set element value as empty, this method will throw an error as it's
unable to detect the appropriate element type. This new function
`set_input_element_value` first rely on property value and then use
`setting-widget-type` as a fallback.
2020-06-24 10:38:58 -07:00
Pragati Agrawal 1737cfdc7d org settings: Fix change detection of setting input elements.
This fixes the change detection of org settings input elements. Luckily,
this regression didn't break our populate_data_for_request and that's also
why this didn't come to notice.
2020-06-24 10:38:58 -07:00
Mateusz Mandera 7a03e2a7fe retention: Replace Realm.message_retention_days None value with -1.
To be more consistent with the meaning in the Stream model, and to make
it easier to have a reasonable settings API, we get rid of the None
value for Realm.message_retention_days in favor of the value -1 to
represent the "don't delete messages" default policy.
2020-06-24 10:33:21 -07:00
Hashir Sarwar 9b8521faee people: Enable `user_avatar_url_field_optional` in webapp.
In 5200598a31, we introduced a new
client capability that can be used to avoid unreasonable network
bandwidth consumed sending avatar URLs of long term idle users in
organizations with 10,000s members.

This commit enables this feature and adds support for it to the web
client.
2020-06-24 10:06:35 -07:00
Vishnu KS 138a0cb073 docs: Use modern django URL patterns in examples. 2020-06-24 09:57:43 -07:00