Commit Graph

4427 Commits

Author SHA1 Message Date
Abhishek-Balaji 052368bd3e alert_words: Move alert_words from UserProfile to separate model.
Previously, alert words were a JSON list of strings stored in a
TextField on user_profile.  That hacky model reflected the fact that
they were an early prototype feature.

This commit migrates from that to a separate table, 'AlertWord'.  The
new AlertWord has user_profile, word, id and realm(denormalization so
we can provide a nice index for fetching all the alert words in a
realm).

This transition requires moving the logic for flushing the Alert Words
caches to their own independent feature.

Note that this commit should not be cherry-picked without the
following commit, which fixes case-sensitivity issues with Alert Words.
2020-04-27 11:29:50 -07:00
Tim Abbott 2b59590e1f test_unread: Remove test for removed mobile hack.
This should have been included in
731cd79436.
2020-04-26 23:41:53 -07:00
Puneeth Chaganti 8f32db81a1 home: Don't use pointer to compute furthest_read_time.
When a user is reading messages only in stream or topic narrows, the pointer
can be left far behind. Using this to compute the furthest_read_time causes
the banckruptcy banner to be shown even when a user has been actively
reading messages. This commit switches to using the sent time on the last
message that the user has read to compute the furthest read time.
2020-04-26 22:41:52 -07:00
rebtung f7fbe3419f api: Improve consistency of reactions API.
Previously, the message and event APIs represented the user differently
for the same reaction data. To make this more consistent, I added a
user_id field to the reaction dict for both messages and events. I
updated the front end to use the user_id field rather than the user
dict. Lastly, I updated front end and back end tests that used user
info.

I primarily tested this by running my local Zulip build and
adding/removing reactions from messages.

Fixes #12049.
2020-04-26 10:35:55 -07:00
Rohitt Vashishtha ce6c57b714 markdown: Set default code block language for quotes and latex as well.
In the original implementation, we were checking for the default language
inside format_code, which resulted in the setting being ignored when set to
quote, math, tex or latex. We shift the validation to `check_for_new_fence`

We also update the tests to use a saner naming scheme for the variables.
2020-04-23 17:51:01 -07:00
Dinesh 2735860f01 browser_support: Drop support for Internet Explorer.
Internet Explorer does not support `position: sticky` which improves
floating recipient bar behavior during scrolling which is one of the
issues blocking PR #9910.
IE also does not support some features that modern browsers support
hence may not super well.
This commit adds an error page that'll be displayed when a user logs
in from Internet Explorer. Also, a test is added.
2020-04-23 17:50:28 -07:00
Rohitt Vashishtha 2825e6ad48 i18n: Support subexpressions of type (t "text") in Handlebars. 2020-04-22 17:57:16 -07:00
Anders Kaseorg fead14951c python: Convert assignment type annotations to Python 3.6 style.
This commit was split by tabbott; this piece covers the vast majority
of files in Zulip, but excludes scripts/, tools/, and puppet/ to help
ensure we at least show the right error messages for Xenial systems.

We can likely further refine the remaining pieces with some testing.

Generated by com2ann, with whitespace fixes and various manual fixes
for runtime issues:

-    invoiced_through: Optional[LicenseLedger] = models.ForeignKey(
+    invoiced_through: Optional["LicenseLedger"] = models.ForeignKey(

-_apns_client: Optional[APNsClient] = None
+_apns_client: Optional["APNsClient"] = None

-    notifications_stream: Optional[Stream] = models.ForeignKey('Stream', related_name='+', null=True, blank=True, on_delete=CASCADE)
-    signup_notifications_stream: Optional[Stream] = models.ForeignKey('Stream', related_name='+', null=True, blank=True, on_delete=CASCADE)
+    notifications_stream: Optional["Stream"] = models.ForeignKey('Stream', related_name='+', null=True, blank=True, on_delete=CASCADE)
+    signup_notifications_stream: Optional["Stream"] = models.ForeignKey('Stream', related_name='+', null=True, blank=True, on_delete=CASCADE)

-    author: Optional[UserProfile] = models.ForeignKey('UserProfile', blank=True, null=True, on_delete=CASCADE)
+    author: Optional["UserProfile"] = models.ForeignKey('UserProfile', blank=True, null=True, on_delete=CASCADE)

-    bot_owner: Optional[UserProfile] = models.ForeignKey('self', null=True, on_delete=models.SET_NULL)
+    bot_owner: Optional["UserProfile"] = models.ForeignKey('self', null=True, on_delete=models.SET_NULL)

-    default_sending_stream: Optional[Stream] = models.ForeignKey('zerver.Stream', null=True, related_name='+', on_delete=CASCADE)
-    default_events_register_stream: Optional[Stream] = models.ForeignKey('zerver.Stream', null=True, related_name='+', on_delete=CASCADE)
+    default_sending_stream: Optional["Stream"] = models.ForeignKey('zerver.Stream', null=True, related_name='+', on_delete=CASCADE)
+    default_events_register_stream: Optional["Stream"] = models.ForeignKey('zerver.Stream', null=True, related_name='+', on_delete=CASCADE)

-descriptors_by_handler_id: Dict[int, ClientDescriptor] = {}
+descriptors_by_handler_id: Dict[int, "ClientDescriptor"] = {}

-worker_classes: Dict[str, Type[QueueProcessingWorker]] = {}
-queues: Dict[str, Dict[str, Type[QueueProcessingWorker]]] = {}
+worker_classes: Dict[str, Type["QueueProcessingWorker"]] = {}
+queues: Dict[str, Dict[str, Type["QueueProcessingWorker"]]] = {}

-AUTH_LDAP_REVERSE_EMAIL_SEARCH: Optional[LDAPSearch] = None
+AUTH_LDAP_REVERSE_EMAIL_SEARCH: Optional["LDAPSearch"] = None

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-04-22 11:02:32 -07:00
Anders Kaseorg f8c95cda51 mypy: Add specific codes to type: ignore annotations.
https://mypy.readthedocs.io/en/stable/error_codes.html

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-04-22 10:46:33 -07:00
Tim Abbott 859b48c491 test_upload: Remove an erroneous nondeterministically correct check.
See the comment added, but basically this test would fail if the two
URLs were computed in different seconds.
2020-04-21 17:58:19 -07:00
Anders Kaseorg 088f7ee5d6 python: Convert type checks to isinstance checks.
Generated by autopep8 --aggressive, with the setup.cfg configuration
from #14532.  In general, an isinstance check may not be equivalent to
a type check because it includes subtypes; however, that’s usually
what you want.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-04-21 17:58:09 -07:00
Anders Kaseorg 1cf63eb5bf python: Whitespace fixes from autopep8.
Generated by autopep8, with the setup.cfg configuration from #14532.
I’m not sure why pycodestyle didn’t already flag these.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-04-21 17:58:09 -07:00
Mateusz Mandera 62c0ab3f9d saml: Change which IdPs are returned to get_external_method_dicts.
If queried without a realm, get_external_method_dicts should only
have IdPs that can be used on all realms.
2020-04-21 13:49:34 -07:00
Kartik Srivastava b29ccdf51c
api: Refactor get_members_backend to return a single bot's data.
This makes `get_members_backend` in zerver/views/users.py to
return a single bot's data too.
2020-04-21 13:45:58 -07:00
Hashir Sarwar e3b90a5ec8 api: Add a monotonic integer "feature level" for non-webapp clients.
The purpose is to provide a way for (non-webapp) clients,
like the mobile and terminal apps, to tell whether the
server it's talking to is new enough to support a given
API feature -- in particular a way that

* is finer-grained than release numbers, so that for
features developed after e.g. 2.1.0 we can use them
immediately on servers deployed from master (like
chat.zulip.org and zulipchat.com) without waiting the
months until a 2.2 release;

* is reliable, unlike e.g. looking at the number of
commits since a release;

* doesn't lead to a growing bag of named feature flags
which the server has to go on sending forever.

Tweaked by tabbott to extend the documentation.

Closes #14618.
2020-04-21 13:37:57 -07:00
shubhamgupta2956 7c3169a1d9 api_docs: Add "MessageId" common component.
To facilitate re-use of the same parameters in other paths, this commit
store the content of the parameter "message_id" in components.
2020-04-20 17:52:49 -07:00
shubhamgupta2956 d5f96c3c12 api_docs: Add "Stream" common component.
To facilitate re-use of the same parameters in other paths, this commit
store the content of the parameter "stream" in components.
2020-04-20 17:49:53 -07:00
Mateusz Mandera 4018dcb8e7 upload: Include filename at the end of temporary access URLs. 2020-04-20 10:25:48 -07:00
Anders Kaseorg 5901e7ba7e python: Convert function type annotations to Python 3 style.
Generated by com2ann (slightly patched to avoid also converting
assignment type annotations, which require Python 3.6), followed by
some manual whitespace adjustment, and six fixes for runtime issues:

-    def __init__(self, token: Token, parent: Optional[Node]) -> None:
+    def __init__(self, token: Token, parent: "Optional[Node]") -> None:

-def main(options: argparse.Namespace) -> NoReturn:
+def main(options: argparse.Namespace) -> "NoReturn":

-def fetch_request(url: str, callback: Any, **kwargs: Any) -> Generator[Callable[..., Any], Any, None]:
+def fetch_request(url: str, callback: Any, **kwargs: Any) -> "Generator[Callable[..., Any], Any, None]":

-def assert_server_running(server: subprocess.Popen[bytes], log_file: Optional[str]) -> None:
+def assert_server_running(server: "subprocess.Popen[bytes]", log_file: Optional[str]) -> None:

-def server_is_up(server: subprocess.Popen[bytes], log_file: Optional[str]) -> bool:
+def server_is_up(server: "subprocess.Popen[bytes]", log_file: Optional[str]) -> bool:

-    method_kwarg_pairs: List[FuncKwargPair],
+    method_kwarg_pairs: "List[FuncKwargPair]",

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-04-18 20:42:48 -07:00
Udit107710 16218d6de3 streams: Remove dependency of streams on actions.
Refactored code in actions.py and streams.py to move stream related
functions into streams.py and remove the dependency on actions.py.

validate_sender_can_write_to_stream function in actions.py was renamed
to access_stream_for_send_message in streams.py.
2020-04-18 16:56:59 -07:00
Tim Abbott 6dc12295db test_alert_words: Fix broken accidentally working test.
This test was passing a string, not an Iterable[str], and effectively
a quirk in the remove_alert_words implementation happened to result in
processing each character in the string working.
2020-04-18 16:19:15 -07:00
Tim Abbott 43241250b6 test_alert_words: Use better variable names. 2020-04-18 16:19:15 -07:00
Tim Abbott da6a295f17 test_alert_words: Clean up line-wrapping. 2020-04-18 16:19:15 -07:00
Tim Abbott 178ce4ae6a test_alert_words: Use set comparisons for alert word sets.
This is important, because we're going to relax how fixed ordering is
with the migration away from a JSON list inside UserProfile.
2020-04-18 16:19:15 -07:00
Abhishek-Balaji c83f147a9a alert_words: Remove unnecessary do_set_alert_words.
This function was only in test_bugdown.py and did the same thing as
add_alert_words in that context.
2020-04-18 16:18:59 -07:00
wowol 507f889901 send_custom_email: Add support for emailing all admins.
This provides a convenient way to send a custom email to just the
administrators of an organization.

Fixes part of #13413.
2020-04-18 13:27:30 -07:00
Anders Kaseorg 8e93175822 requirements: Upgrade Python-Markdown from 3.1.1 to 3.2.1.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-04-18 13:09:51 -07:00
Anders Kaseorg ddcb828349 markdown: Match Python-Markdown code whitespace more closely in JS.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-04-18 13:09:51 -07:00
Tim Abbott 0ccc0f02ce upload: Support requesting a temporary unauthenticated URL.
This is be useful for the mobile and desktop apps to hand an uploaded
file off to the system browser so that it can render PDFs (Etc.).

The S3 backend implementation is simple; for the local upload backend,
we use Django's signing feature to simulate the same sort of 60-second
lifetime token.

Co-Author-By: Mateusz Mandera <mateusz.mandera@protonmail.com>
2020-04-17 09:08:10 -07:00
orientor cfa7724bcc openapi: Use "description: |" for multiline paragraphs.
"description: |" supports markdown and is overall better for
writing multiline paragraphs. So use it in multiline paragraphs
and line-wrap the newly formed paragraphs accordingly.

Edited by tabbott to change most single-line descriptions to use this
format as well.
2020-04-16 20:02:02 -07:00
Puneeth Chaganti 4d2ce607c9 tools: Add script to trigger webhook notification using fixtures.
When creating a webhook integration or creating a new one, it is a pain to
create or update the screenshots in the documentation. This commit adds a
tool that can trigger a sample notification for the webhook using a fixture,
that is likely already written for the tests.

Currently, the developer needs to take a screenshot manually, but this could
be automated using puppeteer or something like that.

Also, the tool does not support webhooks with basic auth, and only supports
webhooks that use json fixtures. These can be fixed in subsequent commits.
2020-04-16 19:25:13 -07:00
Mateusz Mandera 7ed3c3f9f0 saml: Add setting to require limit_to_subdomains on configured IdPs.
If SAML_REQUIRE_LIMIT_TO_SUBDOMAINS is enabled, the configured IdPs will
be validated and cleaned up when the saml backend is initialized.
settings.py would be a tempting and more natural place to do this
perhaps, but in settings.py we don't do logging and we wouldn't be able
to write a test for it.
2020-04-16 17:04:12 -07:00
Mateusz Mandera 143db68422 saml: Implement limiting of IdP to specified realms.
Through the limit_to_subdomains setting on IdP dicts it's now possible
to limit the IdP to only allow authenticating to the specified realms.

Fixes #13340.
2020-04-16 17:04:08 -07:00
Mateusz Mandera 7a9d592dbe tests: Remove out-of-date comment on a saml test for multiple idps. 2020-04-16 17:02:36 -07:00
Ryan Rehman 9340cd1a0b muting: Send muted_topic's date_muted field to frontend. 2020-04-15 15:48:25 -07:00
Kartik Srivastava a060691389 default streams: Change add api to use stream_id.
This refactors add_default_stream in zerver/views/streams.py to
take in stream_id as parameter instead of stream_name.

Minor changes have been made to test_subs.py and settings_streams.js
accordingly.
2020-04-15 15:47:04 -07:00
Mateusz Mandera fbc8325d0e test-backend: Remove rate_limiter from not_yet_fully_covered.
rate_limiter.py now has sufficient test coverage to remove from the
list of exclusions.

Tweaked by tabbott to handle @abstractmethod in a better way.
2020-04-15 11:20:37 -07:00
Mateusz Mandera 5f9da3053d rate_limiter: Handle edge case where rules list may be empty. 2020-04-15 11:20:37 -07:00
Hashir Sarwar b577366a05 rate_limiter: Add an in-process implementation for Tornado.
The Redis-based rate limiting approach takes a lot of time talking to
Redis with 3-4 network requests to Redis on each request.  It had a
negative impact on the performance of `get_events()` since this is our
single highest-traffic endpoint.

This commit introduces an in-process rate limiting alternate for
`/json/events` endpoint. The implementation uses Leaky Bucket
algorithm and Python dictionaries instead of Redis. This drops the
rate limiting time for `get_events()` from about 3000us to less than
100us (on my system).

Fixes #13913.

Co-Author-by: Mateusz Mandera <mateusz.mandera@protonmail.com>
Co-Author-by: Anders Kaseorg <anders@zulipchat.com>
2020-04-15 11:20:37 -07:00
wowol fcf4c9639e emails: Add tests for sending custom emails.
This requires configuring the linter to allow using "subject" in these
files.
2020-04-14 10:50:43 -07:00
Abhishek-Balaji 68257e28ce emoji_name: Raise correct exception if emoji_name is missing.
Right now, the message is "Invalid characters in emoji name" when
the emoji_name is empty. Changing check_valid_emoji_name() in
zerver/lib/emoji.py which validates the name to accomodate the case
of missing name. The new message is "Emoji name is missing".
2020-04-12 11:52:45 -07:00
Puneeth Chaganti 67822a0544 messages: Fix 500 on queries with empty string as search operand.
The error is PGroonga specific since `pgroonga_query_extract_keywords` does
not handle empty string inputs correctly. This commit prevents search
narrows from having empty operands.

Closes #14405
2020-04-12 11:04:09 -07:00
Mateusz Mandera 770086f983 url_preview: Discard url in oembed if server returns invalid json.
This fixes the scenario where we'd get errors in the
FetchLinksEmbedData queue processor if oembed got invalid json from the
URL.
2020-04-11 11:54:54 -07:00
Anders Kaseorg 99242138a7 static: Serve webpack bundles from the root domain.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-04-10 00:48:02 -07:00
Anders Kaseorg 145910bdd5 test_events: Fix incorrect merge conflict resolution.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-04-09 16:59:47 -07:00
Anders Kaseorg c734bbd95d python: Modernize legacy Python 2 syntax with pyupgrade.
Generated by `pyupgrade --py3-plus --keep-percent-format` on all our
Python code except `zthumbor` and `zulip-ec2-configure-interfaces`,
followed by manual indentation fixes.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-04-09 16:43:22 -07:00
sahil839 65d953b2af settings: Add option to disable video call in org settings.
Option is added to video_chat_provider settings for disabling
video calls.

Video call icon is hidden in two cases-
1. video_chat_provider is set to disabled.
2. video_chat_provider is set to Jitsi and settings.JITSI_SERVER_URL
   is none.

Relevant tests are added and modified.

Fixes #14483
2020-04-09 16:03:30 -07:00
Rohitt Vashishtha f9caf522f0 markdown: Allow setting a default language for code blocks.
This adds a new realm setting: default_code_block_language.

This PR also adds a new widget to specify a language, which
behaves somewhat differently from other widgets of the same
kind; instead of exposing methods to the whole module, we
just create a single IIFE that handles all the interactions
with the DOM for the widget.

We also move the code for remapping languages to format_code
function since we want to preserve the original language to
decide if we override it using default_code_clock_language.

Fixes #14404.
2020-04-09 16:02:02 -07:00
Rohitt Vashishtha 3f6541b306 bugdown: Add 'none' as alias for no syntax highlighting in codeblocks.
This is a precursor to #14404.
2020-04-09 16:02:02 -07:00
Tim Abbott 5988d021f9 test_docs: Fix use of fixture when testing /team data.
This fixes a bug in how 449f7e2d4b
managed its fixture file.
2020-04-08 13:09:09 -07:00