Commit Graph

11214 Commits

Author SHA1 Message Date
Rohitt Vashishtha ce6c57b714 markdown: Set default code block language for quotes and latex as well.
In the original implementation, we were checking for the default language
inside format_code, which resulted in the setting being ignored when set to
quote, math, tex or latex. We shift the validation to `check_for_new_fence`

We also update the tests to use a saner naming scheme for the variables.
2020-04-23 17:51:01 -07:00
Dinesh 2735860f01 browser_support: Drop support for Internet Explorer.
Internet Explorer does not support `position: sticky` which improves
floating recipient bar behavior during scrolling which is one of the
issues blocking PR #9910.
IE also does not support some features that modern browsers support
hence may not super well.
This commit adds an error page that'll be displayed when a user logs
in from Internet Explorer. Also, a test is added.
2020-04-23 17:50:28 -07:00
Eeshan Garg 75b2264a3f pypi: Upgrade zulip/zulip-bots dependencies to version 0.7.0.
Includes this change:
* openapi/python_examples: Update get_single_user.

This updates get_single_user to pass keyword arguments to
get_user_by_id instead of passing a dictionary.

Which is required for CI to pass, as we indeed fixed the API of that
function (which had only been present with the wrong API for one release).
2020-04-23 17:41:47 -07:00
Vishnu KS 22b5297e32 actions: Mark strings in send_welcome_bot_response for translation. 2020-04-23 17:19:27 -07:00
Vishnu KS 03286012d2 onboarding: Mark strings in send_initial_realm_messages for translation. 2020-04-23 17:18:39 -07:00
Vishnu KS 9a2ee29ac7 onboarding: Mark strings in send_initial_pms for translation. 2020-04-23 17:15:31 -07:00
Rohitt Vashishtha 2825e6ad48 i18n: Support subexpressions of type (t "text") in Handlebars. 2020-04-22 17:57:16 -07:00
sahil839 3a7de8ad3b models: Add has_permission as a generic function for different policies.
This commit removes can_create_streams and can_subscribe_other_users
to use has_permission as a generic function in UserProfile model for
these settings policies.

Relevant changes are made to events.py to avoid duplication at some
places.
2020-04-22 17:48:52 -07:00
Puneeth Chaganti 87648596ee integrations: Add script to create bot avatars for integrations.
These avatars can be directly used by bots used for documenting these
integrations. In future, these could be used by bots created from a web UI.
2020-04-22 17:45:30 -07:00
Puneeth Chaganti 679f092a75 integrations: Extract code to get logo path for integrations. 2020-04-22 17:45:30 -07:00
Steve Howell 02252c255a db tools: Use common scheme for digests.
We have two different digest schemes to make
sure we keep the database up to date.  There
is the migration digest, which is NOT in the
scope of this commit, and which already
used the mechanism we use for other tools.

Here we are talking about the digest for
important files like `populate_db.py`.

Now our scheme is more consistent with how we
check file changes for other tools (as
well as the aformentioned migration files).

And we only write one hash file, instead of
seven.

And we only write the file when things have
actually changed.

And we are explicit about side effects.

Finally, we include a couple new bot settings
in the digest:

    INTERNAL_BOTS
    DISABLED_REALM_INTERNAL_BOTS

NOTE: This will require a one-time transition,
where we rebuild both databases (dev/test).
It takes a little over two minutes for me,
so it's not super painful.

I bump the provision version here, even
though you don't technically need it (since
the relevant tools are actually using the
digest files to determine if they need to
rebuild the database).  I figure it's just
good to explicitly make this commit trigger
a provision, and the user will then see
the one-time migration of the hash files
with a little bit less of a surprise.

And I do a major bump, not a minor bump,
because when we go in the reverse direction,
the old code will have to rebuild the
database due to the legacy hash files not
being around, so, again, I just prefer it
to be explicit.
2020-04-22 14:41:42 -07:00
Mateusz Mandera ec6022a186 logging_util: Handle record.status_code missing in skip_200_and_304.
Fixes #14595.
Invalid HTTP requests could end up in an unhandled exception in
skip_200_and_304 due the record not having the status_code attribute
set. With this change we'll avoid the exception

Example:
curl  -X POST -H 'Transfer-Encoding : chunked' --data-binary 'a' 'http://zulipdev.com:9991/json/messages/57'

2020-04-21 10:56:22.007 WARN [django.server] "POST /json/messages/57 HTTP/1.1" 405 95
2020-04-21 10:56:22.007 INFO [django.server] code 400, message Bad request syntax ('a')
2020-04-21 10:56:22.008 WARN [django.server] "a" 400 -
2020-04-22 11:26:13 -07:00
Anders Kaseorg fead14951c python: Convert assignment type annotations to Python 3.6 style.
This commit was split by tabbott; this piece covers the vast majority
of files in Zulip, but excludes scripts/, tools/, and puppet/ to help
ensure we at least show the right error messages for Xenial systems.

We can likely further refine the remaining pieces with some testing.

Generated by com2ann, with whitespace fixes and various manual fixes
for runtime issues:

-    invoiced_through: Optional[LicenseLedger] = models.ForeignKey(
+    invoiced_through: Optional["LicenseLedger"] = models.ForeignKey(

-_apns_client: Optional[APNsClient] = None
+_apns_client: Optional["APNsClient"] = None

-    notifications_stream: Optional[Stream] = models.ForeignKey('Stream', related_name='+', null=True, blank=True, on_delete=CASCADE)
-    signup_notifications_stream: Optional[Stream] = models.ForeignKey('Stream', related_name='+', null=True, blank=True, on_delete=CASCADE)
+    notifications_stream: Optional["Stream"] = models.ForeignKey('Stream', related_name='+', null=True, blank=True, on_delete=CASCADE)
+    signup_notifications_stream: Optional["Stream"] = models.ForeignKey('Stream', related_name='+', null=True, blank=True, on_delete=CASCADE)

-    author: Optional[UserProfile] = models.ForeignKey('UserProfile', blank=True, null=True, on_delete=CASCADE)
+    author: Optional["UserProfile"] = models.ForeignKey('UserProfile', blank=True, null=True, on_delete=CASCADE)

-    bot_owner: Optional[UserProfile] = models.ForeignKey('self', null=True, on_delete=models.SET_NULL)
+    bot_owner: Optional["UserProfile"] = models.ForeignKey('self', null=True, on_delete=models.SET_NULL)

-    default_sending_stream: Optional[Stream] = models.ForeignKey('zerver.Stream', null=True, related_name='+', on_delete=CASCADE)
-    default_events_register_stream: Optional[Stream] = models.ForeignKey('zerver.Stream', null=True, related_name='+', on_delete=CASCADE)
+    default_sending_stream: Optional["Stream"] = models.ForeignKey('zerver.Stream', null=True, related_name='+', on_delete=CASCADE)
+    default_events_register_stream: Optional["Stream"] = models.ForeignKey('zerver.Stream', null=True, related_name='+', on_delete=CASCADE)

-descriptors_by_handler_id: Dict[int, ClientDescriptor] = {}
+descriptors_by_handler_id: Dict[int, "ClientDescriptor"] = {}

-worker_classes: Dict[str, Type[QueueProcessingWorker]] = {}
-queues: Dict[str, Dict[str, Type[QueueProcessingWorker]]] = {}
+worker_classes: Dict[str, Type["QueueProcessingWorker"]] = {}
+queues: Dict[str, Dict[str, Type["QueueProcessingWorker"]]] = {}

-AUTH_LDAP_REVERSE_EMAIL_SEARCH: Optional[LDAPSearch] = None
+AUTH_LDAP_REVERSE_EMAIL_SEARCH: Optional["LDAPSearch"] = None

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-04-22 11:02:32 -07:00
Anders Kaseorg af1aef9b39 models: Move flush_per_request_caches after cache definitions.
This will work around https://bugs.python.org/issue34939 when we
convert the type comment to a Python 3.6 style annotation.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-04-22 10:49:20 -07:00
Anders Kaseorg f8c95cda51 mypy: Add specific codes to type: ignore annotations.
https://mypy.readthedocs.io/en/stable/error_codes.html

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-04-22 10:46:33 -07:00
Anders Kaseorg 029bfb9fee mypy: Remove unnecessary type: ignore annotations.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-04-22 10:46:33 -07:00
Steve Howell 239474124e test-backend: Remove generate_fixtures option.
We remove the `generate_fixtures` option here mostly
for simplicity, but in particular to facilitate
an upcoming commit to simplify the job of
`generate-fixtures` (and remove its `--force` option).

The command line option here for `test-backend`
was really calling `generate_fixtures --force`,
which we're about to rename `tools/rebuild-test-database`.

The `test-backend` tools is already smart about catching
up on migrations, so we generally don't need to tell it
to repair the database.

And if the database does get corrupt, you can just do
it directly with `tools/rebuild-test-database`.

This eliminates the `use_force` flag in
`update_test_databases_if_required`, which was easy
to confuse with `rebuild_test_database`.

The other caller wasn't using `use_force`.
2020-04-22 09:24:42 -07:00
Steve Howell 23f09fadfa refactor: Use run in update_test_databases_if_required.
Use `run` to run the tools, and take advantage
that `rebuild-dev-database` is really the same
as `generate-fixtures --force`.
2020-04-22 09:24:42 -07:00
Steve Howell b10be1f8b7 refactor: Early-exit in update_test_databases_if_required.
Just make each conditional run what it needs to run.  The
simplicity that this provides will be more apparent
soon.
2020-04-22 09:24:42 -07:00
Steve Howell 67d0349239 refactor: Use precise names for migration helpers.
Somewhat confusingly, we have two types of different
digests related to databases.  The migration digests
are pragmatic, since changes to migrations are a bit
more frequent for certain use cases and don't
necessitate a complete rebuild of the database.

Anyway, these are just more specific names.
2020-04-22 09:24:42 -07:00
Tim Abbott 859b48c491 test_upload: Remove an erroneous nondeterministically correct check.
See the comment added, but basically this test would fail if the two
URLs were computed in different seconds.
2020-04-21 17:58:19 -07:00
Anders Kaseorg 088f7ee5d6 python: Convert type checks to isinstance checks.
Generated by autopep8 --aggressive, with the setup.cfg configuration
from #14532.  In general, an isinstance check may not be equivalent to
a type check because it includes subtypes; however, that’s usually
what you want.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-04-21 17:58:09 -07:00
Anders Kaseorg 1cf63eb5bf python: Whitespace fixes from autopep8.
Generated by autopep8, with the setup.cfg configuration from #14532.
I’m not sure why pycodestyle didn’t already flag these.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-04-21 17:58:09 -07:00
Kartik Srivastava 18b577f600 lib/bugdown/api_code_examples: Refactor extract_python_code_example.
This refactors `extract_python_code_example` to accept an
`example_regex` parameter. It can now be used to extract code examples
from javascript_examples.py.
2020-04-21 17:31:07 -07:00
Tim Abbott 703fae8980 send_custom_email: Use a special .gitignored directory.
Previously, the send_custom_email code path leaked files in paths that
were not `.gitignored`, under templates/zerver/emails.

This became problematic when we added automated tests for this code
path, as it meant we leaked these files every time `test-backend` ran.

Fix this by ensuring all the files we generate are in this special
subdirectory.
2020-04-21 16:50:11 -07:00
Mateusz Mandera 62c0ab3f9d saml: Change which IdPs are returned to get_external_method_dicts.
If queried without a realm, get_external_method_dicts should only
have IdPs that can be used on all realms.
2020-04-21 13:49:34 -07:00
Kartik Srivastava b29ccdf51c
api: Refactor get_members_backend to return a single bot's data.
This makes `get_members_backend` in zerver/views/users.py to
return a single bot's data too.
2020-04-21 13:45:58 -07:00
Hashir Sarwar e3b90a5ec8 api: Add a monotonic integer "feature level" for non-webapp clients.
The purpose is to provide a way for (non-webapp) clients,
like the mobile and terminal apps, to tell whether the
server it's talking to is new enough to support a given
API feature -- in particular a way that

* is finer-grained than release numbers, so that for
features developed after e.g. 2.1.0 we can use them
immediately on servers deployed from master (like
chat.zulip.org and zulipchat.com) without waiting the
months until a 2.2 release;

* is reliable, unlike e.g. looking at the number of
commits since a release;

* doesn't lead to a growing bag of named feature flags
which the server has to go on sending forever.

Tweaked by tabbott to extend the documentation.

Closes #14618.
2020-04-21 13:37:57 -07:00
Steve Howell 105c168b19 test-api: Avoid flakes with curl tests. 2020-04-21 11:30:29 -04:00
shubhamgupta2956 7c3169a1d9 api_docs: Add "MessageId" common component.
To facilitate re-use of the same parameters in other paths, this commit
store the content of the parameter "message_id" in components.
2020-04-20 17:52:49 -07:00
shubhamgupta2956 05bb1e7075 api_docs: Add "Content" common component.
To facilitate re-use of the same parameters in other paths, this commit
store the content of the parameter "content" in components.
2020-04-20 17:52:49 -07:00
shubhamgupta2956 0425f53e88 api_docs: Add "ClientGravatar" common component.
To facilitate re-use of the same parameters in other paths, this commit
store the content of the parameter "client_gravatar" in components.
2020-04-20 17:52:49 -07:00
shubhamgupta2956 c814afd4a7 api_docs: Add "StreamIdInPath" common component.
To facilitate re-use of the same parameters in other paths, this commit
store the content of the parameter "stream_id" (in path) in components.
2020-04-20 17:52:48 -07:00
shubhamgupta2956 942fd39065 api_docs: Add "StreamIdInQuery" common component.
To facilitate re-use of the same parameters in other paths, this commit
store the content of the parameter "stream_id" (in query) in components.
2020-04-20 17:52:30 -07:00
shubhamgupta2956 d5f96c3c12 api_docs: Add "Stream" common component.
To facilitate re-use of the same parameters in other paths, this commit
store the content of the parameter "stream" in components.
2020-04-20 17:49:53 -07:00
shubhamgupta2956 22c273eacd api_docs: Add "QueueId" common component.
To facilitate re-use of the same parameters in other paths, this commit
store the content of the parameter "queue_id" in components.
2020-04-20 17:49:53 -07:00
shubhamgupta2956 86eed5d9bb api_docs: Add "Topic" common component.
To facilitate re-use of the same parameters in other paths, this commit
store the content of the parameter "topic" in components.
2020-04-20 17:49:52 -07:00
shubhamgupta2956 05b1ddeb84 api_docs: Add "GroupId" common component.
To facilitate re-use of the same parameters in other paths, this commit
store the content of the parameter "group_id" in components.
2020-04-20 17:47:13 -07:00
shubhamgupta2956 8e617d7f26 api_docs: Add "Narrow" common component.
To facilitate re-use of the same parameters in other paths, this commit
store the content of the parameter "narrow" in components.
2020-04-20 17:47:13 -07:00
Steve Howell 51f74a7fd8 provision: Manage digests more rigorously.
We now have two functions related to digests
for processes:

    is_digest_obsolete
    write_digest_file

In most cases we now **wait** to write the
digest file until after we've successfully
run a process with its new inputs.

In one place, for database migrations, we
continue to write the digest optimistically.
We'll want to fix this, but it requires a
little more code cleanup.

Here is the typical sequence of events:

    NEVER RUN -
        is_digest_obsolete returns True
        quickly (we don't compute a hash)

        write_digest_file does a write (duh)

    AFTER NO CHANGES -
        is_digest_obsolete returns False
        after reading one file for old
        hash and multiple files to compute
        hash

        most callers skip write_digest_file

        (no files are changed)

    AFTER SOME CHANGES -
        is_digest_obsolete returns False
        after doing full checks

        most callers call write_digest_file
        *after* running a process
2020-04-20 15:06:47 -07:00
Steve Howell e66bd6a7a4 provision: Put hash_name argument first (minor). 2020-04-20 15:06:47 -07:00
Steve Howell ca49f38619 provision: Extract helpers for paths to hash.
I make these all functions for consistency,
and in particular I want to continue to avoid
`glob.glob` calls until we are actually
computing hashes.

This is mostly a prep to allow us to do
hashing in two separate places:

    - check hashes
    - update hashes

We would only update hashes **after** running
processes anew.

For `provision_inner` I considered using a
class to put the three path-related helpers
into a mini namespace, but it felt too heavy.

It wouldn't be completely implausible here
to extract something like a JSON config
file that has a list of globs for each
process that we do path-hashing for, but I
want to clean up other stuff first.
2020-04-20 15:06:47 -07:00
Steve Howell 2dd6e6f568 refactor: Add Database.database_exists(). 2020-04-20 15:06:47 -07:00
Steve Howell 4822f8d7d6 refactor: Add Database.template_status.
This is mostly a pure code move from
template_database_status().
2020-04-20 15:06:47 -07:00
Steve Howell 0ea4f727d4 refactor: Change params to template_database_status(). 2020-04-20 15:06:47 -07:00
Steve Howell 108b43c873 refactor: Add Database.what_to_do_with_migrations().
This is purely a code move and s/database/self/.
2020-04-20 15:06:47 -07:00
Steve Howell cce223965b refactor: Tweak args to what_to_do_with_migrations.
This is a minor prep commit--we'll move it into
the class next.
2020-04-20 15:06:47 -07:00
Steve Howell 5c5d85cf19 test databases: Add Database.run_db_migrations().
We can reduce some code duplication by having this
on the class.
2020-04-20 15:06:47 -07:00
Steve Howell 1795c06a53 tests databases: Clean up Database class.
We now remove the `Type` and `_TYPE` suffixes,
as we will start treating this like a real
class with behavior, instead of a glorified
struct.

We pass in `platform_type`, so that we can
just derive some of our data from that,
where naming conventions apply.

And we use the name `migrations_status_path`,
instead of the name `migration_status`, which
had two different meanings before this change.
2020-04-20 15:06:47 -07:00
Steve Howell 33cbb4f688 provision: Early-exit in template_database_status.
This is a pure refactor, and we just early-exit
in case the datbase doesn't exist (knowing that
that can be a bit of a lie now--see the comment
I added.)
2020-04-20 15:06:47 -07:00