Commit Graph

205 Commits

Author SHA1 Message Date
Steve Howell 2fb67b3f32 refactor: Extract add/remove_subscriptions_schema.
Now we are consistent about validating color/description.

Ideally we wouldn't need to validate the
`streams_raw` parameters multiple times per
request, but the outer function here changes
the error messages to explicitly reference
the "delete" and "add" request variables.

And for the situation where the user-supplied
parameters are correct, the performance penalty
for checking them twice is extremely negligible.

So it's probably fine for now to just make sure
we use the same validators in all the relevant
places.

There's probably some deeper refactor that we
can do to eliminate the whole `compose_views`
scheme.  And it's also not entirely clear to
me that we really need to support the update
endpoint.  But that's all out of the scope of
this commit.
2020-06-25 13:52:59 -07:00
Pragati Agrawal 1562ec758e org settings: Use 'forever' value instead of -1 for message_retention_days.
Currently, we use -1 as the Realm.message_retention_days value to retain
message forever unless specified at stream level for a particular stream,
that is, no policy set at the realm level. But this is incoherent with what
we use for Stream.message_retention_days where -1 means

> disable retention policy for this stream unconditionally

that can be confusing from an API standpoint.

So instead of trying some hack to reset the value to NULL or using some
other value like -2 for RETAIN_MESSAGE_FOREVER and use that for API. It is
much more intuitive to use a string like 'forever' that can be mapped to
RETAIN_MESSAGE_FOREVER at the backend. And this is similar to what we use
for streams settings as well.
2020-06-24 10:38:58 -07:00
Pragati Agrawal 7b3f4755ff streams: Fix bug that allows 0 value for message_retention_days. 2020-06-24 10:38:58 -07:00
Anders Kaseorg 07fa63e0c8 validator: Fix type errors hidden by bad Any use in set_type_structure.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-23 16:30:17 -07:00
Anders Kaseorg 7e9db327b3 request: Improve validator type so mypy can check it against REQ.
Old: a validator returns None on success and returns an error string
on error.

New: a validator returns the validated value on success and raises
ValidationError on error.

This allows mypy to catch mismatches between the annotated type of a
REQ parameter and the type that the validator actually validates.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-20 22:29:15 -07:00
Steve Howell cc3d87b078 refactor: Rename check_variable_type.
Rename the validator to check_union, to conform
more to Python typing nomenclature.

And we rename one of the test helpers to the
simpler `check_types`.  (The test helper
was using "variable" in the "var" sense.)
2020-06-20 05:22:35 -04:00
Steve Howell 2e9f689add mypy: Fix annotation for principals. 2020-06-19 21:23:23 -07:00
sahil839 c488a35f10 streams: Add API for changing stream-level message_retention_days.
This commit adds backend support for setting message_retention_days
while creating streams and updating it for an existing stream. We only
allow organization owners to set/update it for a stream.

'message_retention_days' field for a stream existed previously also, but
there was no way to set it while creating streams or update it for an
exisiting streams using any endpoint.
2020-06-18 17:00:15 -07:00
Anders Kaseorg f364d06fb5 python: Convert percent formatting to .format for translated strings.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-15 16:24:46 -07:00
Anders Kaseorg 0d6c771baf python: Guard against default value mutation with read-only types.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-13 15:31:27 -07:00
Anders Kaseorg 365fe0b3d5 python: Sort imports with isort.
Fixes #2665.

Regenerated by tabbott with `lint --fix` after a rebase and change in
parameters.

Note from tabbott: In a few cases, this converts technical debt in the
form of unsorted imports into different technical debt in the form of
our largest files having very long, ugly import sequences at the
start.  I expect this change will increase pressure for us to split
those files, which isn't a bad thing.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-11 16:45:32 -07:00
Anders Kaseorg 69730a78cc python: Use trailing commas consistently.
Automatically generated by the following script, based on the output
of lint with flake8-comma:

import re
import sys

last_filename = None
last_row = None
lines = []

for msg in sys.stdin:
    m = re.match(
        r"\x1b\[35mflake8    \|\x1b\[0m \x1b\[1;31m(.+):(\d+):(\d+): (\w+)", msg
    )
    if m:
        filename, row_str, col_str, err = m.groups()
        row, col = int(row_str), int(col_str)

        if filename == last_filename:
            assert last_row != row
        else:
            if last_filename is not None:
                with open(last_filename, "w") as f:
                    f.writelines(lines)

            with open(filename) as f:
                lines = f.readlines()
            last_filename = filename
        last_row = row

        line = lines[row - 1]
        if err in ["C812", "C815"]:
            lines[row - 1] = line[: col - 1] + "," + line[col - 1 :]
        elif err in ["C819"]:
            assert line[col - 2] == ","
            lines[row - 1] = line[: col - 2] + line[col - 1 :].lstrip(" ")

if last_filename is not None:
    with open(last_filename, "w") as f:
        f.writelines(lines)

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-06-11 16:04:12 -07:00
Anders Kaseorg 67e7a3631d python: Convert percent formatting to Python 3.6 f-strings.
Generated by pyupgrade --py36-plus.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-10 15:02:09 -07:00
sahil839 43bff1f67a streams: Extract removing_someone_else logic to a function.
This commit extracts removing_someone_else logic to the function
'check_if_removing_someone_else' for better readability.

This is a follow up to 2187c84.
2020-06-05 16:04:28 -07:00
sahil839 0b432e26b6 test_subs: Add tests for unsubscribing multiple users from stream.
This commit adds tests for unsusbcribing multiple users from a
stream and fixes the missing coverage issue introduced in 2187c84.
2020-06-02 16:27:22 -07:00
Tim Abbott fbcf6a7d72 streams: Add nocoverage to fix CI. 2020-06-01 16:12:13 -07:00
sahil839 2187c84ed9 stream: Add support for sending user_id to endpoint when subscribing users.
This commit modifies the backend to accept user ids when subscribing
users to streams.

It also migrates all existing tests to use this API, aside from a
small set of tests for the legacy API.
2020-06-01 15:59:04 -07:00
Pragati Agrawal bd9b74436c org settings: Enable message_retention_days in org settings UI.
Since production testing of `message_retention_days` is finished, we can
enable this feature in the organization settings page. We already had this
setting in frontend but it was bit rotten and not rendered in templates.

Here we replaced our past text-input based setting with a
dropdown-with-text-input setting approach which is more consistent with our
existing UI.

Along with frontend changes, we also incorporated a backend change to
handle making retention period forever. This change introduces a new
convertor `to_positive_or_allowed_int` which only allows positive integers
and an allowed value for settings like `message_retention_days` which can
be a positive integer or has the value `Realm.RETAIN_MESSAGE_FOREVER` when
we change the setting to retain message forever.

This change made `to_not_negative_int_or_none` redundant so removed it as
well.

Fixes: #14854
2020-05-08 14:09:31 -07:00
Anders Kaseorg fead14951c python: Convert assignment type annotations to Python 3.6 style.
This commit was split by tabbott; this piece covers the vast majority
of files in Zulip, but excludes scripts/, tools/, and puppet/ to help
ensure we at least show the right error messages for Xenial systems.

We can likely further refine the remaining pieces with some testing.

Generated by com2ann, with whitespace fixes and various manual fixes
for runtime issues:

-    invoiced_through: Optional[LicenseLedger] = models.ForeignKey(
+    invoiced_through: Optional["LicenseLedger"] = models.ForeignKey(

-_apns_client: Optional[APNsClient] = None
+_apns_client: Optional["APNsClient"] = None

-    notifications_stream: Optional[Stream] = models.ForeignKey('Stream', related_name='+', null=True, blank=True, on_delete=CASCADE)
-    signup_notifications_stream: Optional[Stream] = models.ForeignKey('Stream', related_name='+', null=True, blank=True, on_delete=CASCADE)
+    notifications_stream: Optional["Stream"] = models.ForeignKey('Stream', related_name='+', null=True, blank=True, on_delete=CASCADE)
+    signup_notifications_stream: Optional["Stream"] = models.ForeignKey('Stream', related_name='+', null=True, blank=True, on_delete=CASCADE)

-    author: Optional[UserProfile] = models.ForeignKey('UserProfile', blank=True, null=True, on_delete=CASCADE)
+    author: Optional["UserProfile"] = models.ForeignKey('UserProfile', blank=True, null=True, on_delete=CASCADE)

-    bot_owner: Optional[UserProfile] = models.ForeignKey('self', null=True, on_delete=models.SET_NULL)
+    bot_owner: Optional["UserProfile"] = models.ForeignKey('self', null=True, on_delete=models.SET_NULL)

-    default_sending_stream: Optional[Stream] = models.ForeignKey('zerver.Stream', null=True, related_name='+', on_delete=CASCADE)
-    default_events_register_stream: Optional[Stream] = models.ForeignKey('zerver.Stream', null=True, related_name='+', on_delete=CASCADE)
+    default_sending_stream: Optional["Stream"] = models.ForeignKey('zerver.Stream', null=True, related_name='+', on_delete=CASCADE)
+    default_events_register_stream: Optional["Stream"] = models.ForeignKey('zerver.Stream', null=True, related_name='+', on_delete=CASCADE)

-descriptors_by_handler_id: Dict[int, ClientDescriptor] = {}
+descriptors_by_handler_id: Dict[int, "ClientDescriptor"] = {}

-worker_classes: Dict[str, Type[QueueProcessingWorker]] = {}
-queues: Dict[str, Dict[str, Type[QueueProcessingWorker]]] = {}
+worker_classes: Dict[str, Type["QueueProcessingWorker"]] = {}
+queues: Dict[str, Dict[str, Type["QueueProcessingWorker"]]] = {}

-AUTH_LDAP_REVERSE_EMAIL_SEARCH: Optional[LDAPSearch] = None
+AUTH_LDAP_REVERSE_EMAIL_SEARCH: Optional["LDAPSearch"] = None

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-04-22 11:02:32 -07:00
Anders Kaseorg 5901e7ba7e python: Convert function type annotations to Python 3 style.
Generated by com2ann (slightly patched to avoid also converting
assignment type annotations, which require Python 3.6), followed by
some manual whitespace adjustment, and six fixes for runtime issues:

-    def __init__(self, token: Token, parent: Optional[Node]) -> None:
+    def __init__(self, token: Token, parent: "Optional[Node]") -> None:

-def main(options: argparse.Namespace) -> NoReturn:
+def main(options: argparse.Namespace) -> "NoReturn":

-def fetch_request(url: str, callback: Any, **kwargs: Any) -> Generator[Callable[..., Any], Any, None]:
+def fetch_request(url: str, callback: Any, **kwargs: Any) -> "Generator[Callable[..., Any], Any, None]":

-def assert_server_running(server: subprocess.Popen[bytes], log_file: Optional[str]) -> None:
+def assert_server_running(server: "subprocess.Popen[bytes]", log_file: Optional[str]) -> None:

-def server_is_up(server: subprocess.Popen[bytes], log_file: Optional[str]) -> bool:
+def server_is_up(server: "subprocess.Popen[bytes]", log_file: Optional[str]) -> bool:

-    method_kwarg_pairs: List[FuncKwargPair],
+    method_kwarg_pairs: "List[FuncKwargPair]",

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-04-18 20:42:48 -07:00
Kartik Srivastava a060691389 default streams: Change add api to use stream_id.
This refactors add_default_stream in zerver/views/streams.py to
take in stream_id as parameter instead of stream_name.

Minor changes have been made to test_subs.py and settings_streams.js
accordingly.
2020-04-15 15:47:04 -07:00
Anders Kaseorg c734bbd95d python: Modernize legacy Python 2 syntax with pyupgrade.
Generated by `pyupgrade --py3-plus --keep-percent-format` on all our
Python code except `zthumbor` and `zulip-ec2-configure-interfaces`,
followed by manual indentation fixes.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-04-09 16:43:22 -07:00
sahil839 b66a67272f models: Reuse constants for common realm permissions policies.
This is a prep commit for making use of same choices for
create_stream_policy and invite_to_stream_policy as both fields
have same set of choices.

This will be useful as we add other fields using these same types.
2020-04-03 15:17:51 -07:00
sahil839 386215614c models: Replace WAITING_PERIOD with FULL_MEMBERS in choice variables.
This commit replaces the WAITING _PERIOD with FULL_MEMBERS from
create_stream_policy and invite_to_stream_policy choices to
achieve consistency and making the variables more descriptive.
2020-04-03 15:14:13 -07:00
Steve Howell 697b00dd6e default streams: Change remove api to use stream_id.
This is a full-stack change:

    - server
    - JS code
    - templates

It's all pretty simple--just use stream_id instead
of stream_name.

I am 99% sure we don't document this API nor use it
in mobile, so it should be a safe change.
2020-03-25 17:11:25 -07:00
Mateusz Mandera 2475adbf8a messages_for_topic: Use stream.recipient_id for more efficient query. 2020-02-11 17:39:43 -08:00
Ryan Rehman 174b2abcfd settings: Migrate to stream_post_policy structure.
This commit includes a new `stream_post_policy` setting,
by replacing the `is_announcement_only` field from the Stream model,
which is done by mirroring the structure of the existing
`create_stream_policy`.

It includes the necessary schema and database migrations to migrate
the is_announcement_only boolean field to stream_post_policy,
a smallPositiveInteger field similar to many other settings.

This change is done to allow organization administrators to restrict
new members from creating and posting to a stream. However, this does
not affect admins who are new members.

With many tweaks by tabbott to documentation under /help, etc.

Fixes #13616.
2020-02-04 17:08:08 -08:00
Nat1405 d5f005fd61 wildcard_mentions_notify: Add per-stream override of global setting.
Adds required API and front-end changes to modify and read the
wildcard_mentions_notify field in the Subscription model.

It includes front-end code to add the setting to the user's "manage
streams" page. This setting will be greyed out when a stream is muted.
The PR also includes back-end code to add the setting the initial state of
a subscription.

New automated tests were added for the API, events system and front-end.
In manual testing, we checked that modifying the setting in the front end
persisted the change in the Subscription model. We noticed the notifications
were not behaving exactly as expected in manual testing; see
https://github.com/zulip/zulip/issues/13073#issuecomment-560263081 .

Tweaked by tabbott to fix real-time synchronization issues.

Fixes: #13429.
2019-12-09 16:09:38 -08:00
Anders Kaseorg cafac83676 request: Tighten type checking on REQ.
Then, find and fix a predictable number of previous misuses.

With a small change by tabbott to preserve backwards compatibility for
sending `yes` for the `forged` field.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-11-13 12:35:55 -08:00
Tim Abbott ddd1a0eb00 actions: Convert do_delete_messages to take a Realm.
The function only used the user's realm anyway, so this is a cleaner
API.

This should also make it more convenient to permanently delete
messages manually, since one doesn't have to fetch a random user in
the realm in order to delete a message using the management shell.

No functional change.
2019-11-12 12:20:31 -08:00
Rafid Aslam 718b70ec8b Rename `not_subscribed key` to `not_removed` in users/me/subscriptions.
Rename `not_subscibed_key` to `not_removed` in
`users/me/subscriptions` DELETE response.

Fixes #13277.
2019-10-13 10:30:34 +07:00
Tim Abbott 44db59bbe0 openapi: Fix handling of parameters passed via the URL/path.
Previously, our OpenAPI documentation validation was failing for some
endpoints because it didn't account for the `in: path` type of
parameter, resulting in a mismatch between what was declared via REQ
and what was declared in the OpenAPI docs.

We fix this by excluding the path type parameters in both places from
what's considered by documentation using the `path_only` flag.

I doubt this is the correct long-term fix; in particular, I don't
think we're actually running the validators for these path-only
parameters.  The examples that exist today are all IDs with validators
for being non-negative numbers, but longer-term I think we'll want to
do something different (possibly at the REQ layer, see the TODO).
2019-08-19 15:06:08 -07:00
meiomorphism 1af72a2745 subscriptions: Avoid sending unneeded subscriber information.
The `users/me/subscriptions` endpoint accidentally started returning
subscriber information for each stream.  This is convenient, but
unnecessarily costly for those clients which either don't need it
(most API apps) or already acquire this information via /register
(including Zulip's apps).

This change removes that data set from the default response.  Clients
which had come to rely on it, or would like to rely on it in future,
may still access it via an additional documented API parameter.

Fixes #12917.
2019-08-09 16:03:21 -07:00
Hemanth V. Alluri ede73ee4cd streams: New stream messages should be auto marked as read by creator.
When a person creates a new realm, they'll likely want to create a
bunch of initial streams at once. When doing so, it could be annoying
to have to mark all of the new stream notification messages as read.

Thus to make this process smoother, we should automatically mark
the messages generated by the Notification Bot in the notifications
(announcements) stream, as well as in the newly created stream itself
as read by the stream creator.

Fixes #12765.
2019-07-26 15:10:09 -07:00
Rishi Gupta 38eeb3e183 notification bot: Add constant for STREAM_EVENTS_NOTIFICATION_TOPIC. 2019-07-20 14:39:15 -07:00
Rishi Gupta c9f194be99 notification bot: Update message for stream name changes.
Also properly internationalizes string formatting arguments for the stream
created message.
2019-07-20 14:39:15 -07:00
Rishi Gupta 5cf304f8f9 notification bot: Tighten message for being added to streams. 2019-07-13 15:53:36 -07:00
Rishi Gupta 0f3c2748dd notification bot: Update stream announcement message. 2019-07-11 15:22:36 -07:00
Rishi Gupta c3adafbee9 notification bot: Remove check for existence in add_subscriptions_backend.
We implicitly assume settings.NOTIFICATION_BOT is not None just a few lines
above, in
sender = get_system_bot(settings.NOTIFICATION_BOT)
notifications.append(
    internal_prep_private_message(
        realm=user_profile.realm,
        sender=sender,
        ...
2019-07-11 14:18:21 -07:00
Rishi Gupta 6c8569d99f notification bot: Add stream creator to initial stream message.
Also removes the "Welcome", since it's cleaner to have this be strictly a
notification.
2019-07-11 14:16:56 -07:00
Rishi Gupta 5400eb62be notification bot: Inline prep_stream_welcome_message.
This will make it easier to have access to the stream creator.

The indirection also isn't really adding anything, especially given that the
announce message is inlined just above.
2019-07-11 14:16:56 -07:00
Roman Godov a50824e031 models: Rename Subscription.in_home_view field to is_muted.
This renames Subscription.in_home_view field to is_muted, for greater
clarity as to what it does just from seeing the setting name, without
having to look it up.

Also disabled an obsolete test_migrations test.

Fixes #10042.
2019-05-12 22:08:10 -07:00
David Wood 272ed90685 settings: Create an explicit invite_to_stream_policy setting.
This commit creates a new organization setting that determines whether
a user can invite other users to streams. Previously this was linked
to the waiting period threshold, but this was both not documented and
overly limiting.

With significant tweaks by tabbott to change the database model to not
involve two threshhold fields, edit the tests, etc.

This requires follow-up work to make the create stream policy setting
work how this code implies it should.

Fixes #12042.
2019-04-29 17:11:28 -07:00
Anders Kaseorg 9a9de156c3 lint: Fix calls to _() on computed strings.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-04-23 15:23:03 -07:00
Anders Kaseorg 643bd18b9f lint: Fix code that evaded our lint checks for string % non-tuple.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-04-23 15:21:37 -07:00
Eeshan Garg 9252c43225 api/streams: Support including bot owner's subscriptions.
This is important for situations such as with our Zapier app,
where the requesting user may be a bot that would like to access
its owner's subscriptions.

Tweaked by tabbott to eliminate the 2^N growth of cases in
do_get_streams.
2019-02-28 22:32:05 -08:00
Tim Abbott cbc62b8e07 streams: Prevent creation of multi-line stream descriptions.
We do not anticipate our UI for showing stream descriptions looking
reasonable for multi-line descriptions, so we should just ban creating
them.

Given the frontend changes, multi-line descriptions are only likely to
show up from importing content from other tools, in which case
replacing newlines with spaces is cleaner than the alternative.
2019-02-20 12:28:00 -08:00
Rohitt Vashishtha 57b9991396 markdown: Change syntax of silent mentions ( _@person -> @_person). 2019-02-20 10:41:42 -08:00
Eeshan Garg 179b747769 streams: Refactor multi-option helpers into separate functions.
For internal stream messages, most of the time, we have access to
a Stream object. For the few corner cases where we don't, it is a
much cleaner approach to have a separate function that accepts a
stream name than having one multi-option helper that accepts both
names and objects.
2019-02-12 11:10:26 -08:00
Eeshan Garg c240008edb internal_prep_stream_message: Support accepting a Stream object.
If the caller has access to a Stream object, it is wasteful to
query a database for a stream by ID or name. In addition, not
having to go through stream names eliminates various classes of
possible bugs involved with getting a Stream object back.
2019-02-08 08:53:16 -08:00