Commit Graph

37880 Commits

Author SHA1 Message Date
Steve Howell 2dd1ccef15 check_list: Make sub_validator a required argument.
This allows us to avoid the overload decorators.
2020-06-24 15:04:59 -07:00
Steve Howell de2285222d tests: Use check_tuple for realm filters.
This also sets the stage to simplify
our check_list validator.
2020-06-24 15:01:57 -07:00
Steve Howell c7b82d3ece mypy: Use tuples for muted_topics.
We now have our muted topics use tuples internally,
which allows us to tighten up the annotation
for get_topic_mutes, as well as our schema
checking.

We want to deprecate sub_validator=None
for check_list, so we also introduce
check_tuple here.  Now we also want to deprecate
check_tuple, but it's at least isolated now.

We will use this for data structures that are tuples,
but which are sent as lists over the wire.  Fortunately,
we don't have too many of those.

The plan is to convert tuples to dictionaries,
but backward compatibility may be tricky in some
places.
2020-06-24 15:01:57 -07:00
Steve Howell 8e8228535e tests: Use check_dict for external_authentication_methods
This is still imperfect, but the only goal for now is
to make sure that `check_list` always get a sub_validator.
2020-06-24 15:01:57 -07:00
sahil839 fe35f1e9bd invites: Do not return multi-use invites to non-admin users.
This commit changes do_get_user_invites function to not return
multiuse invites to non-admin users. We should only return multiuse
invites to admins, as we only allow admins to create them.
2020-06-24 15:01:41 -07:00
Alex Vandiver 8499b2c0dc test-install: Add support for focal. 2020-06-24 13:03:13 -07:00
Alex Vandiver 8719fba3c4 test-install: Stop installing postgres-10.
As in the previous commit, we can no longer pre-install the wrong
version of postgres.  Unfortunately, this leaves it out of the base
image and thus makes testing installs longer.
2020-06-24 13:03:13 -07:00
Alex Vandiver 31f1f10501 installer: Halt if wrong version of PostgreSQL is already installed.
49a7a66004 and immediately previous commits began installing
PostgreSQL 12 from their apt repository.  On machines which already
have the distribution-provided version of PostgreSQL installed,
however, this leads to failure to apply puppet when restarting
PostgreSQL 12, as both attempt to claim the same port.

During installation, if we will be installing PostgreSQL, look for
other versions than what we will install, and abort if they are
found.  This is safer than attempting to automatically uninstall or
reconfigure existing databases.
2020-06-24 12:57:38 -07:00
Alex Vandiver 814198d649 installer: Abstract out version of postgres installed.
This allows for installing from-scratch with a different pinned
version of PostgreSQL, and provides a single place to change when the
default should increase.
2020-06-24 12:57:38 -07:00
Alex Vandiver ca9d27175b installer: Write PostgreSQL version based on puppet classes.
Using `/etc/init.d/postgresql` as the detection of if Postgres is on
the server is incorrect, because this line runs _before_ puppet and
any packages are installed.  Thus, it cannot tell the difference
between a new Ubuntu one-host first-time-install without PostgreSQL
yet, and one which is merely a front-end and will never have
PostgreSQL.  This leads to failures in first-time installs:

```
Error: Evaluation Error: Error while evaluating a Function Call,
  Could not find template 'zulip/postgresql//postgresql.conf.template.erb'
```

The only way to detect if PostgreSQL will be present in the _end_
state of the install is to examine the puppet classes that are
applied.

To do this, we must inspect `PUPPET_CLASSES`.  Unfortunately, this can
be fragile to subclassing (e.g. `zulip_ops::postgres_appdb`).  We
might desire to use `puppet apply --write-catalog-summary` to deduce
the _applied_ classes, which would unroll the inheritance; however,
this causes a chicken-and-egg problem, because `zulip.conf` must be
already written out (including a value for `postgresql.version`, if
necessary!) before such a puppet run could successfully complete.

Switch to predicating the `postgresql.version` key on the puppet
classes that are known to install postgres.
2020-06-24 12:57:38 -07:00
Alex Vandiver 253246185f installer: Update documentation.
Where appropriate, documentation wording is shared with
docs/production/install.md
2020-06-24 12:57:38 -07:00
Alex Vandiver 85dbb13c56 installer: Abstract out apt/yum divide into a variable.
This check is done in several places, using a somewhat fragile `case`
statement; move it into an explicit variable.
2020-06-24 12:57:38 -07:00
Alex Vandiver 876ee4a8ed installer: Remove code specific to stretch or xenial.
Support for Xenial and Stretch was removed (5154ddafca, 0f4b1076ad,
8944e0ad53, 79acd5ae40, 1219a2e854), but not all codepaths were
updated to remove their conditionals on it.

Remove all code predicated on Xenial or Stretch.  debathena support
was migrated to Bionic, since that appears to be the current state of
existing debathena servers.
2020-06-24 12:57:38 -07:00
Alex Vandiver e4899eae8b installer: Sync the claimed supported distros with the check.
0f4b1076ad removed Ubuntu 16.04 "xenial" and Debian 9 "stretch" from
the printed list of supported operating systems, but left them in the
verification check that controls if that message is printed,
effectively continuing to support them.

Conversely, 439f0d3004 added Ubuntu 20.04 "focal" to the check, but
not to the printed list.

Synchronize to check and print the right supported distributions:
Ubuntu 18.04 "bionic", Ubuntu 20.04 "focal", and Debian 10 "buster".
2020-06-24 12:57:38 -07:00
Alex Vandiver 58cb7cecd8 installer: Remove `--remote-postgres`, redundant with `--no-init-db`.
The previous commit removed the only behavior difference between the
two flags; both of them skip user/database creation, and the tables
therein.

Of the two options `--no-init-db` is more explicit as to what it does,
as opposed to just one facet of when it might be used; remove
`--remote-postgres`.
2020-06-24 12:57:38 -07:00
Alex Vandiver 7c6a25a43d installer: Group and unify ordering of installer options.
This also adds the missing `--no-overwrite-settings` option to
`--help`.
2020-06-24 12:57:38 -07:00
Alexandre Gigleux fa1071335b
tools: Configure Zulip to be scannable by SonarCloud.
Apparently, SonarCloud fails on our repository for unknown reasons.

But it runs if one restricts to just Python and HTML files.
2020-06-24 12:41:17 -07:00
Tim Abbott 148f74d3a7 css: Move night theme pygments CSS to night_mode.scss.
The previous architecture did not work properly with the automatically
detected night theme, resulting in a weird mix of the night and day
themes on code blocks.

I'm not thrilled with the requirement this imposes that all of our
night theme CSS needs to be in one file, but we do need to get a quick
fix out here.

Fixes #15554.
2020-06-24 12:35:02 -07:00
Tim Abbott 2a6ef2b9eb models: Add missing migrations from invite_as change.
I neglected to `git add` after doing `renumber-migrations` for
1f8f227444.
2020-06-24 11:39:01 -07:00
Tim Abbott ac9ecbe8de docs: Clarify variable variants in translating docs. 2020-06-24 11:34:00 -07:00
arpit551 e19aef569a ci: Update Circle CI images.
Removed postgresql and python 2 packages when creating docker images
for Circle CI. Updated the images.
2020-06-24 11:19:47 -07:00
sahil839 136c005f3f settings_invites: Remove invited_as_values map.
This commit removes invited_as_values map in settings_invites.js.
This object has been removed to avoid duplication as we already
have role values in settings_config.js.

A similar map is created from settings_config.user_role_values
in settings_config.js and is used to populate invited_as_text
for invites.
2020-06-24 11:09:07 -07:00
sahil839 1f8f227444 models: Update values of PreregistrationUser.invite_as dict.
This commit changes the PreregistrationUser.invite_as dict to have
same set of values as we have for UserProfile.role.

This also adds a data migration to update the already exisiting
PreregistrationUser and MultiuseInvite objects.
2020-06-24 11:09:07 -07:00
Tim Abbott b7b173d2ae realm: Fix type-checking for message_retention_days.
The best way to handle this is to have the potentially string-typed
value be a separate variable.
2020-06-24 11:01:34 -07:00
Tim Abbott 90e180f974 search: Enable pills-based UI for testing. 2020-06-24 10:44:37 -07:00
Rohitt Vashishtha 6ba8fb518b markdown-timestamp: Show flatpickr using typeaheads.
We leverage the composebox typeaheads to show flatpickr to pick dates
and times for the !time syntax.

We use moment.js to try and parse the time from current token. If we
are successful, we initialize flatpickr with the parsed time, else we
default to using the current time.
2020-06-24 10:44:37 -07:00
Mateusz Mandera 890cafac11 retention: Use batch size of 100 for stream messages.
Streams can have lots of subscribers, meaning that the archiving process
will be moving tons of UserMessages per message. For that reason, using
a smaller batch size for stream messages is justified.

Some personal messages need to be added in test_scrub_realm to have
coverage of do_delete_messages_by_sender after these changes.
2020-06-24 10:41:00 -07:00
Mateusz Mandera 0c6497d43a retention: Add restore_retention_policy_deletions_for_stream function. 2020-06-24 10:40:38 -07:00
Mateusz Mandera 468f8cf488 retention: Improve logging of transactions. 2020-06-24 10:40:38 -07:00
Pragati Agrawal 33fa307c95 org settings: Display empty input when message retention days is forever.
Currently we display -1 in input box of id_realm_message_retention_days
when realm_message_retention_days is -1, which isn't user friendly.
Displaying the input box as empty is more intuitive.

And if the user tries to submit an empty input box we throw invalid JSON
error that isn't user friendly either, so fixed that too. In the ideal
case, we shouldn't send the request at first place to the backend when we
don't have any input.
2020-06-24 10:38:58 -07:00
Pragati Agrawal 1562ec758e org settings: Use 'forever' value instead of -1 for message_retention_days.
Currently, we use -1 as the Realm.message_retention_days value to retain
message forever unless specified at stream level for a particular stream,
that is, no policy set at the realm level. But this is incoherent with what
we use for Stream.message_retention_days where -1 means

> disable retention policy for this stream unconditionally

that can be confusing from an API standpoint.

So instead of trying some hack to reset the value to NULL or using some
other value like -2 for RETAIN_MESSAGE_FOREVER and use that for API. It is
much more intuitive to use a string like 'forever' that can be mapped to
RETAIN_MESSAGE_FOREVER at the backend. And this is similar to what we use
for streams settings as well.
2020-06-24 10:38:58 -07:00
Pragati Agrawal 7b3f4755ff streams: Fix bug that allows 0 value for message_retention_days. 2020-06-24 10:38:58 -07:00
Pragati Agrawal 72dd5bfa77 org settings: Use get_input_element_value style function to set value.
`get_input_element_value()` function is more reliable to detect the input
element type and extract it's value. But the current way of setting the
value of input elements relies on first checking the `property_value` type.
Which is fine, but for the cases when the property value is null, and we
want to set element value as empty, this method will throw an error as it's
unable to detect the appropriate element type. This new function
`set_input_element_value` first rely on property value and then use
`setting-widget-type` as a fallback.
2020-06-24 10:38:58 -07:00
Pragati Agrawal 1737cfdc7d org settings: Fix change detection of setting input elements.
This fixes the change detection of org settings input elements. Luckily,
this regression didn't break our populate_data_for_request and that's also
why this didn't come to notice.
2020-06-24 10:38:58 -07:00
Mateusz Mandera 7a03e2a7fe retention: Replace Realm.message_retention_days None value with -1.
To be more consistent with the meaning in the Stream model, and to make
it easier to have a reasonable settings API, we get rid of the None
value for Realm.message_retention_days in favor of the value -1 to
represent the "don't delete messages" default policy.
2020-06-24 10:33:21 -07:00
Hashir Sarwar 9b8521faee people: Enable `user_avatar_url_field_optional` in webapp.
In 5200598a31, we introduced a new
client capability that can be used to avoid unreasonable network
bandwidth consumed sending avatar URLs of long term idle users in
organizations with 10,000s members.

This commit enables this feature and adds support for it to the web
client.
2020-06-24 10:06:35 -07:00
Vishnu KS 138a0cb073 docs: Use modern django URL patterns in examples. 2020-06-24 09:57:43 -07:00
Steve Howell bc53fe1606 mypy: Use equals([]) to check services.
We are trying to deprecate sub_validator=None
for check_list.
2020-06-24 10:35:36 -04:00
Greg Price baa09f0518 docs: Clarify instructions on setting up email for Apple auth.
This comes after I read through all the linked docs, and went through
the UI to do this registration.
2020-06-23 19:04:26 -07:00
Greg Price 71c995d50d docs: Explain a bit more explicitly how to configure Apple auth.
In particular the Services ID and Bundle ID each have one of Apple's
random-looking 10-character identifiers, in addition to the Java-style
names the admin chooses.  Best to be clear about what names are
supposed to be the chosen names and which are supposed to be the
random-looking assigned names.

(I don't know of any docs elsewhere making this clear -- but I guessed
it'd be this way, and empirically it works.)

Also mention you need to enable the backend. :-)
2020-06-23 19:04:26 -07:00
Greg Price 0021dfe8e7 auth settings: Put Apple "Team ID" before the things it namespaces.
I believe the Bundle ID (aka App ID) and Services ID have meaning only
relative to a specific Team ID.  In particular, in some places in the
developer.apple.com UI, they're displayed in a fully-qualified form
like "ABCDE12345.com.example.app", where "com.example.app" is the
App ID or Services ID and ABCDE12345 is the Team ID.
2020-06-23 19:04:26 -07:00
Greg Price 9b620dfe6c docs: Break up a long sentence in Apple auth instructions.
This also helps keep the actionable part short, by separating
the actionable instruction from the longer parenthetical
explaining what it's for.
2020-06-23 19:04:26 -07:00
Ryan Rehman b57ebe717e input pill: Backspace deletes pills if caret is at start of input.
This allows deleting the last pill by the backspace key,
even if there is text input present.
2020-06-23 17:26:46 -07:00
Ryan Rehman 1e0340e0a3 css: Allow cross icon to stay at end of searchbox.
Fixes #10026.
2020-06-23 17:25:45 -07:00
Tim Abbott 4f7848d9b2 css: Fix starred messages not being displayed.
This change was missed in a4f5b0c635.

Ideally, we'd figure out a refactoring that made these definitions
only appear in one place.
2020-06-23 17:22:16 -07:00
Mateusz Mandera 7fe52bbb9e tests: Clean up the subdomain argument to social_auth_test.
subdomain=None didn't make much sense as a value, and wasn't actually in
use anywhere, except one test where it was accidental. All tests specify
the subdomain explicitly, so we should change the type to str, and make
it an obligatory kwarg.
2020-06-23 17:14:31 -07:00
Brainrecursion 30eaed0378 saml: Add option to restrict subdomain access based on SAML attributes.
Adds the ability to set a SAML attribute which contains a
list of subdomains the user is allowed to access. This allows a Zulip
server with multiple organizations to filter using SAML attributes
which organization each user can access.

Cleaned up and adapted by Mateusz Mandera to fit our conventions and
needs more.

Co-authored-by: Mateusz Mandera <mateusz.mandera@zulip.com>
2020-06-23 17:14:31 -07:00
Alex Vandiver 2283a16476
ci: Only attempt to post the failure if we have a key.
Only some branches define the `ZULIP_BOT_KEY`; others, including PRs,
thus currently fail to POST:

    {"result":"error","msg":"Malformed API key","code":"INVALID_API_KEY"}

Simply abort early, without attempting to report the result, if the
key is not present.
2020-06-23 16:43:18 -07:00
Anders Kaseorg d6655689f5 validator: Remove unused type_structure system.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-23 16:30:17 -07:00
Anders Kaseorg 07fa63e0c8 validator: Fix type errors hidden by bad Any use in set_type_structure.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-23 16:30:17 -07:00