Commit Graph

4420 Commits

Author SHA1 Message Date
Anders Kaseorg fead14951c python: Convert assignment type annotations to Python 3.6 style.
This commit was split by tabbott; this piece covers the vast majority
of files in Zulip, but excludes scripts/, tools/, and puppet/ to help
ensure we at least show the right error messages for Xenial systems.

We can likely further refine the remaining pieces with some testing.

Generated by com2ann, with whitespace fixes and various manual fixes
for runtime issues:

-    invoiced_through: Optional[LicenseLedger] = models.ForeignKey(
+    invoiced_through: Optional["LicenseLedger"] = models.ForeignKey(

-_apns_client: Optional[APNsClient] = None
+_apns_client: Optional["APNsClient"] = None

-    notifications_stream: Optional[Stream] = models.ForeignKey('Stream', related_name='+', null=True, blank=True, on_delete=CASCADE)
-    signup_notifications_stream: Optional[Stream] = models.ForeignKey('Stream', related_name='+', null=True, blank=True, on_delete=CASCADE)
+    notifications_stream: Optional["Stream"] = models.ForeignKey('Stream', related_name='+', null=True, blank=True, on_delete=CASCADE)
+    signup_notifications_stream: Optional["Stream"] = models.ForeignKey('Stream', related_name='+', null=True, blank=True, on_delete=CASCADE)

-    author: Optional[UserProfile] = models.ForeignKey('UserProfile', blank=True, null=True, on_delete=CASCADE)
+    author: Optional["UserProfile"] = models.ForeignKey('UserProfile', blank=True, null=True, on_delete=CASCADE)

-    bot_owner: Optional[UserProfile] = models.ForeignKey('self', null=True, on_delete=models.SET_NULL)
+    bot_owner: Optional["UserProfile"] = models.ForeignKey('self', null=True, on_delete=models.SET_NULL)

-    default_sending_stream: Optional[Stream] = models.ForeignKey('zerver.Stream', null=True, related_name='+', on_delete=CASCADE)
-    default_events_register_stream: Optional[Stream] = models.ForeignKey('zerver.Stream', null=True, related_name='+', on_delete=CASCADE)
+    default_sending_stream: Optional["Stream"] = models.ForeignKey('zerver.Stream', null=True, related_name='+', on_delete=CASCADE)
+    default_events_register_stream: Optional["Stream"] = models.ForeignKey('zerver.Stream', null=True, related_name='+', on_delete=CASCADE)

-descriptors_by_handler_id: Dict[int, ClientDescriptor] = {}
+descriptors_by_handler_id: Dict[int, "ClientDescriptor"] = {}

-worker_classes: Dict[str, Type[QueueProcessingWorker]] = {}
-queues: Dict[str, Dict[str, Type[QueueProcessingWorker]]] = {}
+worker_classes: Dict[str, Type["QueueProcessingWorker"]] = {}
+queues: Dict[str, Dict[str, Type["QueueProcessingWorker"]]] = {}

-AUTH_LDAP_REVERSE_EMAIL_SEARCH: Optional[LDAPSearch] = None
+AUTH_LDAP_REVERSE_EMAIL_SEARCH: Optional["LDAPSearch"] = None

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-04-22 11:02:32 -07:00
Anders Kaseorg f8c95cda51 mypy: Add specific codes to type: ignore annotations.
https://mypy.readthedocs.io/en/stable/error_codes.html

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-04-22 10:46:33 -07:00
Tim Abbott 859b48c491 test_upload: Remove an erroneous nondeterministically correct check.
See the comment added, but basically this test would fail if the two
URLs were computed in different seconds.
2020-04-21 17:58:19 -07:00
Anders Kaseorg 088f7ee5d6 python: Convert type checks to isinstance checks.
Generated by autopep8 --aggressive, with the setup.cfg configuration
from #14532.  In general, an isinstance check may not be equivalent to
a type check because it includes subtypes; however, that’s usually
what you want.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-04-21 17:58:09 -07:00
Anders Kaseorg 1cf63eb5bf python: Whitespace fixes from autopep8.
Generated by autopep8, with the setup.cfg configuration from #14532.
I’m not sure why pycodestyle didn’t already flag these.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-04-21 17:58:09 -07:00
Mateusz Mandera 62c0ab3f9d saml: Change which IdPs are returned to get_external_method_dicts.
If queried without a realm, get_external_method_dicts should only
have IdPs that can be used on all realms.
2020-04-21 13:49:34 -07:00
Kartik Srivastava b29ccdf51c
api: Refactor get_members_backend to return a single bot's data.
This makes `get_members_backend` in zerver/views/users.py to
return a single bot's data too.
2020-04-21 13:45:58 -07:00
Hashir Sarwar e3b90a5ec8 api: Add a monotonic integer "feature level" for non-webapp clients.
The purpose is to provide a way for (non-webapp) clients,
like the mobile and terminal apps, to tell whether the
server it's talking to is new enough to support a given
API feature -- in particular a way that

* is finer-grained than release numbers, so that for
features developed after e.g. 2.1.0 we can use them
immediately on servers deployed from master (like
chat.zulip.org and zulipchat.com) without waiting the
months until a 2.2 release;

* is reliable, unlike e.g. looking at the number of
commits since a release;

* doesn't lead to a growing bag of named feature flags
which the server has to go on sending forever.

Tweaked by tabbott to extend the documentation.

Closes #14618.
2020-04-21 13:37:57 -07:00
shubhamgupta2956 7c3169a1d9 api_docs: Add "MessageId" common component.
To facilitate re-use of the same parameters in other paths, this commit
store the content of the parameter "message_id" in components.
2020-04-20 17:52:49 -07:00
shubhamgupta2956 d5f96c3c12 api_docs: Add "Stream" common component.
To facilitate re-use of the same parameters in other paths, this commit
store the content of the parameter "stream" in components.
2020-04-20 17:49:53 -07:00
Mateusz Mandera 4018dcb8e7 upload: Include filename at the end of temporary access URLs. 2020-04-20 10:25:48 -07:00
Anders Kaseorg 5901e7ba7e python: Convert function type annotations to Python 3 style.
Generated by com2ann (slightly patched to avoid also converting
assignment type annotations, which require Python 3.6), followed by
some manual whitespace adjustment, and six fixes for runtime issues:

-    def __init__(self, token: Token, parent: Optional[Node]) -> None:
+    def __init__(self, token: Token, parent: "Optional[Node]") -> None:

-def main(options: argparse.Namespace) -> NoReturn:
+def main(options: argparse.Namespace) -> "NoReturn":

-def fetch_request(url: str, callback: Any, **kwargs: Any) -> Generator[Callable[..., Any], Any, None]:
+def fetch_request(url: str, callback: Any, **kwargs: Any) -> "Generator[Callable[..., Any], Any, None]":

-def assert_server_running(server: subprocess.Popen[bytes], log_file: Optional[str]) -> None:
+def assert_server_running(server: "subprocess.Popen[bytes]", log_file: Optional[str]) -> None:

-def server_is_up(server: subprocess.Popen[bytes], log_file: Optional[str]) -> bool:
+def server_is_up(server: "subprocess.Popen[bytes]", log_file: Optional[str]) -> bool:

-    method_kwarg_pairs: List[FuncKwargPair],
+    method_kwarg_pairs: "List[FuncKwargPair]",

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-04-18 20:42:48 -07:00
Udit107710 16218d6de3 streams: Remove dependency of streams on actions.
Refactored code in actions.py and streams.py to move stream related
functions into streams.py and remove the dependency on actions.py.

validate_sender_can_write_to_stream function in actions.py was renamed
to access_stream_for_send_message in streams.py.
2020-04-18 16:56:59 -07:00
Tim Abbott 6dc12295db test_alert_words: Fix broken accidentally working test.
This test was passing a string, not an Iterable[str], and effectively
a quirk in the remove_alert_words implementation happened to result in
processing each character in the string working.
2020-04-18 16:19:15 -07:00
Tim Abbott 43241250b6 test_alert_words: Use better variable names. 2020-04-18 16:19:15 -07:00
Tim Abbott da6a295f17 test_alert_words: Clean up line-wrapping. 2020-04-18 16:19:15 -07:00
Tim Abbott 178ce4ae6a test_alert_words: Use set comparisons for alert word sets.
This is important, because we're going to relax how fixed ordering is
with the migration away from a JSON list inside UserProfile.
2020-04-18 16:19:15 -07:00
Abhishek-Balaji c83f147a9a alert_words: Remove unnecessary do_set_alert_words.
This function was only in test_bugdown.py and did the same thing as
add_alert_words in that context.
2020-04-18 16:18:59 -07:00
wowol 507f889901 send_custom_email: Add support for emailing all admins.
This provides a convenient way to send a custom email to just the
administrators of an organization.

Fixes part of #13413.
2020-04-18 13:27:30 -07:00
Anders Kaseorg 8e93175822 requirements: Upgrade Python-Markdown from 3.1.1 to 3.2.1.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-04-18 13:09:51 -07:00
Anders Kaseorg ddcb828349 markdown: Match Python-Markdown code whitespace more closely in JS.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-04-18 13:09:51 -07:00
Tim Abbott 0ccc0f02ce upload: Support requesting a temporary unauthenticated URL.
This is be useful for the mobile and desktop apps to hand an uploaded
file off to the system browser so that it can render PDFs (Etc.).

The S3 backend implementation is simple; for the local upload backend,
we use Django's signing feature to simulate the same sort of 60-second
lifetime token.

Co-Author-By: Mateusz Mandera <mateusz.mandera@protonmail.com>
2020-04-17 09:08:10 -07:00
orientor cfa7724bcc openapi: Use "description: |" for multiline paragraphs.
"description: |" supports markdown and is overall better for
writing multiline paragraphs. So use it in multiline paragraphs
and line-wrap the newly formed paragraphs accordingly.

Edited by tabbott to change most single-line descriptions to use this
format as well.
2020-04-16 20:02:02 -07:00
Puneeth Chaganti 4d2ce607c9 tools: Add script to trigger webhook notification using fixtures.
When creating a webhook integration or creating a new one, it is a pain to
create or update the screenshots in the documentation. This commit adds a
tool that can trigger a sample notification for the webhook using a fixture,
that is likely already written for the tests.

Currently, the developer needs to take a screenshot manually, but this could
be automated using puppeteer or something like that.

Also, the tool does not support webhooks with basic auth, and only supports
webhooks that use json fixtures. These can be fixed in subsequent commits.
2020-04-16 19:25:13 -07:00
Mateusz Mandera 7ed3c3f9f0 saml: Add setting to require limit_to_subdomains on configured IdPs.
If SAML_REQUIRE_LIMIT_TO_SUBDOMAINS is enabled, the configured IdPs will
be validated and cleaned up when the saml backend is initialized.
settings.py would be a tempting and more natural place to do this
perhaps, but in settings.py we don't do logging and we wouldn't be able
to write a test for it.
2020-04-16 17:04:12 -07:00
Mateusz Mandera 143db68422 saml: Implement limiting of IdP to specified realms.
Through the limit_to_subdomains setting on IdP dicts it's now possible
to limit the IdP to only allow authenticating to the specified realms.

Fixes #13340.
2020-04-16 17:04:08 -07:00
Mateusz Mandera 7a9d592dbe tests: Remove out-of-date comment on a saml test for multiple idps. 2020-04-16 17:02:36 -07:00
Ryan Rehman 9340cd1a0b muting: Send muted_topic's date_muted field to frontend. 2020-04-15 15:48:25 -07:00
Kartik Srivastava a060691389 default streams: Change add api to use stream_id.
This refactors add_default_stream in zerver/views/streams.py to
take in stream_id as parameter instead of stream_name.

Minor changes have been made to test_subs.py and settings_streams.js
accordingly.
2020-04-15 15:47:04 -07:00
Mateusz Mandera fbc8325d0e test-backend: Remove rate_limiter from not_yet_fully_covered.
rate_limiter.py now has sufficient test coverage to remove from the
list of exclusions.

Tweaked by tabbott to handle @abstractmethod in a better way.
2020-04-15 11:20:37 -07:00
Mateusz Mandera 5f9da3053d rate_limiter: Handle edge case where rules list may be empty. 2020-04-15 11:20:37 -07:00
Hashir Sarwar b577366a05 rate_limiter: Add an in-process implementation for Tornado.
The Redis-based rate limiting approach takes a lot of time talking to
Redis with 3-4 network requests to Redis on each request.  It had a
negative impact on the performance of `get_events()` since this is our
single highest-traffic endpoint.

This commit introduces an in-process rate limiting alternate for
`/json/events` endpoint. The implementation uses Leaky Bucket
algorithm and Python dictionaries instead of Redis. This drops the
rate limiting time for `get_events()` from about 3000us to less than
100us (on my system).

Fixes #13913.

Co-Author-by: Mateusz Mandera <mateusz.mandera@protonmail.com>
Co-Author-by: Anders Kaseorg <anders@zulipchat.com>
2020-04-15 11:20:37 -07:00
wowol fcf4c9639e emails: Add tests for sending custom emails.
This requires configuring the linter to allow using "subject" in these
files.
2020-04-14 10:50:43 -07:00
Abhishek-Balaji 68257e28ce emoji_name: Raise correct exception if emoji_name is missing.
Right now, the message is "Invalid characters in emoji name" when
the emoji_name is empty. Changing check_valid_emoji_name() in
zerver/lib/emoji.py which validates the name to accomodate the case
of missing name. The new message is "Emoji name is missing".
2020-04-12 11:52:45 -07:00
Puneeth Chaganti 67822a0544 messages: Fix 500 on queries with empty string as search operand.
The error is PGroonga specific since `pgroonga_query_extract_keywords` does
not handle empty string inputs correctly. This commit prevents search
narrows from having empty operands.

Closes #14405
2020-04-12 11:04:09 -07:00
Mateusz Mandera 770086f983 url_preview: Discard url in oembed if server returns invalid json.
This fixes the scenario where we'd get errors in the
FetchLinksEmbedData queue processor if oembed got invalid json from the
URL.
2020-04-11 11:54:54 -07:00
Anders Kaseorg 99242138a7 static: Serve webpack bundles from the root domain.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-04-10 00:48:02 -07:00
Anders Kaseorg 145910bdd5 test_events: Fix incorrect merge conflict resolution.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-04-09 16:59:47 -07:00
Anders Kaseorg c734bbd95d python: Modernize legacy Python 2 syntax with pyupgrade.
Generated by `pyupgrade --py3-plus --keep-percent-format` on all our
Python code except `zthumbor` and `zulip-ec2-configure-interfaces`,
followed by manual indentation fixes.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-04-09 16:43:22 -07:00
sahil839 65d953b2af settings: Add option to disable video call in org settings.
Option is added to video_chat_provider settings for disabling
video calls.

Video call icon is hidden in two cases-
1. video_chat_provider is set to disabled.
2. video_chat_provider is set to Jitsi and settings.JITSI_SERVER_URL
   is none.

Relevant tests are added and modified.

Fixes #14483
2020-04-09 16:03:30 -07:00
Rohitt Vashishtha f9caf522f0 markdown: Allow setting a default language for code blocks.
This adds a new realm setting: default_code_block_language.

This PR also adds a new widget to specify a language, which
behaves somewhat differently from other widgets of the same
kind; instead of exposing methods to the whole module, we
just create a single IIFE that handles all the interactions
with the DOM for the widget.

We also move the code for remapping languages to format_code
function since we want to preserve the original language to
decide if we override it using default_code_clock_language.

Fixes #14404.
2020-04-09 16:02:02 -07:00
Rohitt Vashishtha 3f6541b306 bugdown: Add 'none' as alias for no syntax highlighting in codeblocks.
This is a precursor to #14404.
2020-04-09 16:02:02 -07:00
Tim Abbott 5988d021f9 test_docs: Fix use of fixture when testing /team data.
This fixes a bug in how 449f7e2d4b
managed its fixture file.
2020-04-08 13:09:09 -07:00
Vishnu KS 449f7e2d4b team: Generate team page data using cron job.
This eliminates the contributors data as a possible source of
flakiness when installing Zulip from Git.

Fixes #14351.
2020-04-08 12:52:31 -07:00
Vishnu KS 8415a1472a tools: Rename update-authors-json to fetch-contributor-data. 2020-04-08 12:40:00 -07:00
Mateusz Mandera 218be002f1 rate_limiter: Add more detailed automated tests.
Extracted by tabbott from the original commit to support testing
without the Tornado version merged yet.
2020-04-08 10:40:26 -07:00
Mateusz Mandera 46a02e70b0 rate_limiter: Fix inconsistency in an edge case in redis limiter.
If we had a rule like "max 3 requests in 2 seconds", there was an
inconsistency between is_ratelimited() and get_api_calls_left().
If you had:
request #1 at time 0
request #2 and #3 at some times < 2

Next request, if exactly at time 2, would not get ratelimited, but if
get_api_calls_left was called, it would return 0. This was due to
inconsistency on the boundary - the check in is_ratelimited was
exclusive, while get_api_calls_left uses zcount, which is inclusive.
2020-04-08 10:29:18 -07:00
Wyatt Hoodes 13f86f35d9 zcommand: Add `/fluid-width` and `/fixed-width` slash commands. 2020-04-07 20:54:34 -07:00
Vishnu KS abad5365fa tools: Move duplicate_commits.json file to tools directory.
This is a prep commit for generating /team page data
using cron job. zerver/tests directory is not present in
production installation. So moving the file from the directory
tests to tools.
2020-04-07 16:06:29 -07:00
Tim Abbott 843345dfee message_edit: Add backend for moving a topic to another stream.
This commit reuses the existing infrastructure for moving a topic
within a stream to add support for moving topics from one stream to
another.

Split from the original full-feature commit so that we can merge just
the backend, which is finished, at this time.

This is a large part of #6427.

The feature is incomplete, in that we don't have real-time update of
the frontend to handle the event, documentation, etc., but this commit
is a good mergable checkpoint that we can do further work on top of.
We also still ideally would have a test_events test for the backend,
but I'm willing to leave that for follow-up work.

This appears to have switched to tabbott as the author during commit
squashing sometime ago, but this commit is certainly:

Co-Authored-By: Wbert Adrián Castro Vera <wbertc@gmail.com>
2020-04-07 14:19:19 -07:00