Commit Graph

79 Commits

Author SHA1 Message Date
Anders Kaseorg 61d0417e75 python: Replace ujson with orjson.
Fixes #6507.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-08-11 10:55:12 -07:00
Anders Kaseorg 60a25b2721 docs: Fix spelling errors caught by codespell.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-08-11 10:23:06 -07:00
Anders Kaseorg 8667e632c8 python: Use isinstance with a tuple for testing multiple types.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-30 18:58:23 -07:00
Steve Howell 6b910ff3b4 widgets: Make type checks more explicit.
Note that I don't actually convert the
checker from check_dict to check_dict_only,
because that would be a user-facing change,
but I think we can sweep a lot of things
like this after the next release.
2020-06-25 13:52:59 -07:00
Steve Howell 2dd1ccef15 check_list: Make sub_validator a required argument.
This allows us to avoid the overload decorators.
2020-06-24 15:04:59 -07:00
Steve Howell c7b82d3ece mypy: Use tuples for muted_topics.
We now have our muted topics use tuples internally,
which allows us to tighten up the annotation
for get_topic_mutes, as well as our schema
checking.

We want to deprecate sub_validator=None
for check_list, so we also introduce
check_tuple here.  Now we also want to deprecate
check_tuple, but it's at least isolated now.

We will use this for data structures that are tuples,
but which are sent as lists over the wire.  Fortunately,
we don't have too many of those.

The plan is to convert tuples to dictionaries,
but backward compatibility may be tricky in some
places.
2020-06-24 15:01:57 -07:00
Anders Kaseorg d6655689f5 validator: Remove unused type_structure system.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-23 16:30:17 -07:00
Anders Kaseorg 07fa63e0c8 validator: Fix type errors hidden by bad Any use in set_type_structure.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-23 16:30:17 -07:00
Tim Abbott dc2a045d3b docs: Add documentation on mypy @overload.
We only use this in a few places, but they're really important places
for understanding the types in the codebase, and so it's worth having
a bit of expository documentation explaining how we use it.

(And I expect we'll add more with time).
2020-06-21 10:51:47 -07:00
Anders Kaseorg 7e9db327b3 request: Improve validator type so mypy can check it against REQ.
Old: a validator returns None on success and returns an error string
on error.

New: a validator returns the validated value on success and raises
ValidationError on error.

This allows mypy to catch mismatches between the annotated type of a
REQ parameter and the type that the validator actually validates.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-20 22:29:15 -07:00
Steve Howell 22ead64d54 tests: Extract USING_TYPE_STRUCTURE. 2020-06-20 05:22:35 -04:00
Steve Howell cc3d87b078 refactor: Rename check_variable_type.
Rename the validator to check_union, to conform
more to Python typing nomenclature.

And we rename one of the test helpers to the
simpler `check_types`.  (The test helper
was using "variable" in the "var" sense.)
2020-06-20 05:22:35 -04:00
Anders Kaseorg f364d06fb5 python: Convert percent formatting to .format for translated strings.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-15 16:24:46 -07:00
Anders Kaseorg 74c17bf94a python: Convert more percent formatting to Python 3.6 f-strings.
Generated by pyupgrade --py36-plus.

Now including %d, %i, %u, and multi-line strings.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-14 23:27:22 -07:00
Anders Kaseorg 365fe0b3d5 python: Sort imports with isort.
Fixes #2665.

Regenerated by tabbott with `lint --fix` after a rebase and change in
parameters.

Note from tabbott: In a few cases, this converts technical debt in the
form of unsorted imports into different technical debt in the form of
our largest files having very long, ugly import sequences at the
start.  I expect this change will increase pressure for us to split
those files, which isn't a bad thing.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-11 16:45:32 -07:00
Anders Kaseorg 67e7a3631d python: Convert percent formatting to Python 3.6 f-strings.
Generated by pyupgrade --py36-plus.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-10 15:02:09 -07:00
Pragati Agrawal bd9b74436c org settings: Enable message_retention_days in org settings UI.
Since production testing of `message_retention_days` is finished, we can
enable this feature in the organization settings page. We already had this
setting in frontend but it was bit rotten and not rendered in templates.

Here we replaced our past text-input based setting with a
dropdown-with-text-input setting approach which is more consistent with our
existing UI.

Along with frontend changes, we also incorporated a backend change to
handle making retention period forever. This change introduces a new
convertor `to_positive_or_allowed_int` which only allows positive integers
and an allowed value for settings like `message_retention_days` which can
be a positive integer or has the value `Realm.RETAIN_MESSAGE_FOREVER` when
we change the setting to retain message forever.

This change made `to_not_negative_int_or_none` redundant so removed it as
well.

Fixes: #14854
2020-05-08 14:09:31 -07:00
Anders Kaseorg fead14951c python: Convert assignment type annotations to Python 3.6 style.
This commit was split by tabbott; this piece covers the vast majority
of files in Zulip, but excludes scripts/, tools/, and puppet/ to help
ensure we at least show the right error messages for Xenial systems.

We can likely further refine the remaining pieces with some testing.

Generated by com2ann, with whitespace fixes and various manual fixes
for runtime issues:

-    invoiced_through: Optional[LicenseLedger] = models.ForeignKey(
+    invoiced_through: Optional["LicenseLedger"] = models.ForeignKey(

-_apns_client: Optional[APNsClient] = None
+_apns_client: Optional["APNsClient"] = None

-    notifications_stream: Optional[Stream] = models.ForeignKey('Stream', related_name='+', null=True, blank=True, on_delete=CASCADE)
-    signup_notifications_stream: Optional[Stream] = models.ForeignKey('Stream', related_name='+', null=True, blank=True, on_delete=CASCADE)
+    notifications_stream: Optional["Stream"] = models.ForeignKey('Stream', related_name='+', null=True, blank=True, on_delete=CASCADE)
+    signup_notifications_stream: Optional["Stream"] = models.ForeignKey('Stream', related_name='+', null=True, blank=True, on_delete=CASCADE)

-    author: Optional[UserProfile] = models.ForeignKey('UserProfile', blank=True, null=True, on_delete=CASCADE)
+    author: Optional["UserProfile"] = models.ForeignKey('UserProfile', blank=True, null=True, on_delete=CASCADE)

-    bot_owner: Optional[UserProfile] = models.ForeignKey('self', null=True, on_delete=models.SET_NULL)
+    bot_owner: Optional["UserProfile"] = models.ForeignKey('self', null=True, on_delete=models.SET_NULL)

-    default_sending_stream: Optional[Stream] = models.ForeignKey('zerver.Stream', null=True, related_name='+', on_delete=CASCADE)
-    default_events_register_stream: Optional[Stream] = models.ForeignKey('zerver.Stream', null=True, related_name='+', on_delete=CASCADE)
+    default_sending_stream: Optional["Stream"] = models.ForeignKey('zerver.Stream', null=True, related_name='+', on_delete=CASCADE)
+    default_events_register_stream: Optional["Stream"] = models.ForeignKey('zerver.Stream', null=True, related_name='+', on_delete=CASCADE)

-descriptors_by_handler_id: Dict[int, ClientDescriptor] = {}
+descriptors_by_handler_id: Dict[int, "ClientDescriptor"] = {}

-worker_classes: Dict[str, Type[QueueProcessingWorker]] = {}
-queues: Dict[str, Dict[str, Type[QueueProcessingWorker]]] = {}
+worker_classes: Dict[str, Type["QueueProcessingWorker"]] = {}
+queues: Dict[str, Dict[str, Type["QueueProcessingWorker"]]] = {}

-AUTH_LDAP_REVERSE_EMAIL_SEARCH: Optional[LDAPSearch] = None
+AUTH_LDAP_REVERSE_EMAIL_SEARCH: Optional["LDAPSearch"] = None

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-04-22 11:02:32 -07:00
Anders Kaseorg f8c95cda51 mypy: Add specific codes to type: ignore annotations.
https://mypy.readthedocs.io/en/stable/error_codes.html

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-04-22 10:46:33 -07:00
Anders Kaseorg c734bbd95d python: Modernize legacy Python 2 syntax with pyupgrade.
Generated by `pyupgrade --py3-plus --keep-percent-format` on all our
Python code except `zthumbor` and `zulip-ec2-configure-interfaces`,
followed by manual indentation fixes.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-04-09 16:43:22 -07:00
Tim Abbott 5aa6aeb303 typing: Stop using Collection[str].
Apparently, Collection is unavailable in Python 3.5, so we can't use
it.  Reverting to a Union for now.
2020-04-03 15:53:37 -07:00
Tim Abbott a745e533fe settings: Use cleaner validators for display settings.
This simplifies the update_display_settings endpoint to use REQ for
validation, rather than custom if/else statements.

The test changes just take advantage of the now more consistent
syntax.
2020-04-03 15:09:14 -07:00
Tim Abbott 85c9ffd91c message: Validate propagate_mode parameters.
This improves the error handling for invalid values of the
propagate_mode parameter to our message editing endpoints.
Previously, invalid values would just work like change_one rather than
doing nothing.
2020-03-24 12:36:45 -07:00
Vishnu KS 1c6435d4cc validator: Optionally record a type_structure attribute.
We plan to use these records to check and record the schema of Zulip's
events for the purposes of API documentation.

Based on an original messier commit by tabbott.

In theory, a nicer version of this would be able to work directly off
the mypy type system, but this will be good enough for our use case.
2020-03-06 17:07:14 -08:00
Matheus Melo 678c3a89d0 validators: Add check_int_in validator.
This is a useful helper for our enum format fields where we want to
only allow a fixed list of integer values.
2019-11-18 15:14:56 -08:00
Rishi Gupta 2c80224e2d custom profile: Rename check_url_pattern.
validator.py has validators for the whole project, so we need a more
specific name.
2019-08-05 20:59:47 -07:00
Rishi Gupta b37c6a8644 custom profile: Update a few error messages. 2019-08-05 20:59:47 -07:00
Priyank Patel c067c155aa messages: Support passing user ID to sender operator. 2019-07-17 16:09:12 -07:00
Yashashvi Dave d7ee2aced1 models: Add `external_account` in custom profile field types.
Add new custom profile field type, External account.
External account field links user's social media
profile with account. e.g. GitHub, Twitter, etc.

Fixes part of #12302
2019-07-09 17:21:54 -07:00
Priyank Patel e3431976ce validator: Add validator for checking string or integer list.
This validator will be used to check if string or user IDs list is
passed in.
2019-06-17 17:33:58 -07:00
Yashashvi Dave f25e00bcaa zever/lib/validator: Rename function to `validate_choice_field_data`. 2019-06-06 22:36:16 -07:00
Anders Kaseorg 9a9de156c3 lint: Fix calls to _() on computed strings.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-04-23 15:23:03 -07:00
Anders Kaseorg 643bd18b9f lint: Fix code that evaded our lint checks for string % non-tuple.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-04-23 15:21:37 -07:00
Tim Abbott c05fb01cbf stream: Fix validator for stream colors.
Apparently, our new validator for stream color having a valid format
incorrectly handled colors that had duplicate characters in them.

(This is caused in part by the spectrum.js logic automatically
converting #ffff00 to #ff0, which our validator rejected).  Given that
we had old stream colors in the #ff0 format in our database anyway for
legacy, there's no benefit to banning these colors.

In the future, we could imagine standardizing the format, but doing so
will require also changing the frontend to submit colors only in the
6-character format.

Fixes an issue reported in
https://github.com/zulip/zulip/issues/11845#issuecomment-471417073
2019-03-11 11:43:39 -07:00
Rafid Aslam 9038230de8 decorator.py: Add `max_int_size` param to `to_non_negative_int()`.
Add `max_int_size` parameter to `to_non_negative_int()` in
decorator.py so it will be able to validate that the integer doesn't
exceed the integer maximum limit.

Fixes #11451
2019-03-01 11:16:46 -08:00
Anders Kaseorg f0ecb93515 zerver core: Remove unused imports.
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2019-02-02 17:41:24 -08:00
Harshit Bansal dccfadc1a5 minor: Fix some typing errors. 2019-01-29 16:01:30 -08:00
Hemanth V. Alluri 6c18ca9cac validators: Add optional keys feature to check_dict and check_dict_only.
along with the `required_keys` argument an additional `optional_keys`
argument can be passed to both `check_dict` and `check_dict_only`.
Fixes #10892.
2019-01-22 10:03:21 -08:00
Hemanth V. Alluri 1f0d319f9f validators: Add a validator called check_color for hex color codes. 2019-01-14 10:16:42 -08:00
Vishnu Ks 54a002c2e2 requirements: Upgrade pyflakes to 2.0.0.
We fix a few errors that only the new version finds.
2018-05-24 11:31:36 -07:00
Steve Howell 69517f5ac5 Support zform-based widget content in the server.
API users, particularly bots, can now send a field
called "widget_content" that will be turned into
a submessage for the web app to look at.  (Other
clients can still rely on "content" to be there,
although it's up to the bot author to make the
experience good for those clients as well.)

Right now widget_content will be a JSON string that
encodes a "zform" widget with "choices."  Our first
example will be a trivia bot, where users will see
something like this:

    Which fruit is orange in color?

        [A] orange
        [B] blackberry
        [C] strawberry

The letters will be turned into buttons on the webapp
and have canned replies.

This commit has a few parts:
    - receive widget_content in the request (simply
        validating that it's a string)
    - parse the JSON in check_message and deeply
        validate its structure
    - turn it into a submessage in widget.py
2018-05-24 09:30:22 -07:00
Aditya Bansal a68376e2ba zerver/lib: Change use of typing.Text to str. 2018-05-12 15:22:39 -07:00
Tim Abbott b1ad7593ba validators: Improve mypy type annotations.
The main thing here is writing check_string_fixed_length and
check_capped_string as returning a Validator, but we also fix issues
around passing default=None.
2018-05-03 21:27:43 -07:00
Tim Abbott ad0cfb3512 validators: Add check_string_fixed_length.
This will be useful in some upcoming work on zilencer.
2018-05-03 14:30:03 -07:00
Tim Abbott 594451707d check_capped_string: Clean up corner case.
We were rejecting strings of length equal to the max.

While we're at it, fix the unnecessary period in the error message,
which doesn't align with similar validators.
2018-05-03 14:30:03 -07:00
Yashashvi Dave 4033f210af custom fields: Add support for custom date field type. 2018-04-30 23:04:25 +05:30
Tim Abbott 976e61d687 validators: Improve error messages for check_capped_string. 2018-04-30 10:07:06 -07:00
Umair Khan cf2f6b38dd profile: Add choice field.
Fixes part of #8878
2018-04-26 00:35:51 -07:00
Umair Khan 00ffa808da check_capped_string: Return a validator.
This allows us to create custom validators.
2018-04-19 11:21:50 -07:00
Umair Khan e48f8bdafd check_short_string: Use check_capped_string for impl. 2018-04-02 09:51:09 -07:00