Commit Graph

11248 Commits

Author SHA1 Message Date
Tim Abbott cbe0723199 api docs: Add an API changelog page.
I imagine this can be improved in various ways, but I've initialized
this with all the **Changes** entries recorded in either zulip.yaml or
the rest of the API documentation, and I expect we'll be able to
iterate on this effectively.

It'll also be useful as a record of changes that we should remember to
document the API documentation as we document more endpoints that
currently don't discuss these issues.

While working on this, I fixed various issues where feature levels
could be mentioned or endpoints didn't properly document changes.
2020-04-28 22:23:25 -07:00
Mateusz Mandera f1ec02b40a auth: Add ExternalAuthResult to manage data in authentication flows.
This new type eliminates a bunch of messy code that previously
involved passing around long lists of mixed positional keyword and
arguments, instead using a consistent data object for communicating
about the state of an external authentication (constructed in
backends.py).

The result is a significantly more readable interface between
zproject/backends.py and zerver/views/auth.py, though likely more
could be done.

This has the side effect of renaming fields for internally passed
structures from name->full_name, next->redirect_to; this results in
most of the test codebase changes.

Modified by tabbott to add comments and collaboratively rewrite the
initialization logic.
2020-04-28 22:19:02 -07:00
Vishnu KS 3f69500765 billing: Rename downgrade_realm_for_deactivation to downgrade_now. 2020-04-28 22:05:49 -07:00
akashaviator 28d58c848d api: Guess emoji_type when adding a reaction.
This changes add_reaction in zerver.views.reactions to allow
calling POST ../messages/{message_id}/reactions api endpoint with
emoji_name only, even in the case of a custom emoji.
2020-04-28 20:49:39 -07:00
Steve Howell cf78cb0d6e templates: Use `<hr>` and `<br>` consistently.
We now prevent these variations:

    * <hr/>
    * <hr />
    * <br/>
    * <br />

We could enforce similar consistency for other void
tags, if we wished, but these two are particularly
prevalent.
2020-04-28 17:05:48 -07:00
Tim Abbott 1be4cc5f17 openapi: Fix servers section to work in online editor. 2020-04-28 14:27:43 -07:00
Puneeth Chaganti 88c297dffe tools: Allow specifying more options when capturing screenshot.
The name and directory into which the screenshot should be captured can be
specified now, apart from specifying the fixture file to use.
2020-04-28 13:00:09 -07:00
Puneeth Chaganti 2c325e34ac integrations: Remove mutable default argument for config_options. 2020-04-28 13:00:09 -07:00
Puneeth Chaganti 39fe206289 integrations: Change legacy arg to a bool from Optional[bool]. 2020-04-28 13:00:09 -07:00
Puneeth Chaganti f280b64caa integrations: Remove hasattr check for stream_name on Integrations.
`stream_name` is always set on `Integration`. If it is not explicitly
specified, it is set to the name of the integration.
2020-04-28 13:00:09 -07:00
orientor 4d8c988ef2 openapi: Use description markdown for rendering endpoint descriptions.
Firstly, change endpoint descriptions in zulip.yaml so that they
match their counterpart in the api docs. Then edit the api docs
so that they use api description markdown extension for displaying
endpoint description.
2020-04-28 12:57:19 -07:00
orientor 64c6bab276 openapi: Create markdown extension for rendering endpoint descriptions.
Add function in openapi.py to access endpoint descriptions written
in zulip.yaml. Use this function for creating a markdown extension
for rendering endpoint descriptions written in zulip.yaml.

We use this extension for a single endpoint to get test coverage.
2020-04-28 12:57:19 -07:00
Steve Howell 1aaef75bc5 test databases: Fix dev/test typo.
This line was changed to the wrong tool by
accident in 23f09fadfa.
2020-04-28 07:13:22 -04:00
Abhishek-Balaji d3db174be9 alert_words: Change test_alert_words to use actions.do_*().
Changing test_alert_words to use do_add_alert_words() and
do_remove_alert_words() from lib/actions.py instead of the
existing add_user_alert_words() and remove_user_alert_words()
as is the general policy of calling these functions when we
are updating the database.
2020-04-27 22:53:25 -07:00
Tim Abbott 976e554799 Revert "home: Don't use pointer to compute furthest_read_time."
This reverts commit 8f32db81a1.

This change unfortunately requires an index that we don't have, and
thus is incredibly expensive.  We'll need to do a thoughtful reworking
before we can integrate it again.
2020-04-27 22:29:39 -07:00
Tim Abbott 03fecba917 alert_words: Fix cache flushing behavior and add tests.
The post_init cache-flushing behavior in the original alert words
migration was subtly wrong; while it may have passed tests, it didn't
have the right ordering for unlikely races.

We use post_save rather than post_init hooks precisely because they
ensure that we flush the cache after we know the database has been
updated and any future reads from the database will have the latest
state.
2020-04-27 11:48:32 -07:00
Tim Abbott 8e5b0351b3 alert_words: Fix case-sensitivity of alert words.
Previously, alert words were case-insensitive in practice, by which I
mean the Markdown logic had always been case-insensitive; but the data
model was not, so you could create "duplicate" alert words with the
same words in different cases.  We fix this inconsistency by making
the database model case-insensitive.

I'd prefer to be using the Postgres `citext` extension to have
postgres take care of case-insensitive logic for us, but that requires
installing a postgres extension as root on the postgres server, which
is a pain and perhaps not worth the effort to arrange given that we
can achieve our goals with transaction when adding alert words.

We take advantage of the migrate_alert_words migration we're already
doing for all users to effect this transition.

Fixes #12563.
2020-04-27 11:31:51 -07:00
Abhishek-Balaji 052368bd3e alert_words: Move alert_words from UserProfile to separate model.
Previously, alert words were a JSON list of strings stored in a
TextField on user_profile.  That hacky model reflected the fact that
they were an early prototype feature.

This commit migrates from that to a separate table, 'AlertWord'.  The
new AlertWord has user_profile, word, id and realm(denormalization so
we can provide a nice index for fetching all the alert words in a
realm).

This transition requires moving the logic for flushing the Alert Words
caches to their own independent feature.

Note that this commit should not be cherry-picked without the
following commit, which fixes case-sensitivity issues with Alert Words.
2020-04-27 11:29:50 -07:00
Abhishek-Balaji 818776faae alert bugdown: Change name of AlertWordsNotificationProcessor.
This is a precursor commit to change the name of
AlertWordNotificationProcessor to AlertWordsNotificationProcessor
to match the change from UserProfile.alert_words to Alertword.
2020-04-27 10:45:18 -07:00
Rohitt Vashishtha 087b54aa45 markdown: Keep only 'text' for a no-op pygments rendering.
Previously, we added support for 'none', 'plain' and 'noop' and a
function `lang = remap_language(lang)`. This also had the potential
to encourage adding more remappings- something that we deliberatly
want to keep to a minimum.

For context, Anders K doesn't want us to keep any remapping (only
keeping 'text' which is the default no-op lexer that pygments has)
and Tim wants to keep 'plain' and 'text'. We should only document
and advertise 'text'.
2020-04-27 10:40:29 -07:00
Tim Abbott 2b59590e1f test_unread: Remove test for removed mobile hack.
This should have been included in
731cd79436.
2020-04-26 23:41:53 -07:00
Puneeth Chaganti 8f32db81a1 home: Don't use pointer to compute furthest_read_time.
When a user is reading messages only in stream or topic narrows, the pointer
can be left far behind. Using this to compute the furthest_read_time causes
the banckruptcy banner to be shown even when a user has been actively
reading messages. This commit switches to using the sent time on the last
message that the user has read to compute the furthest read time.
2020-04-26 22:41:52 -07:00
Tim Abbott d8def0de55 pointer: Remove useless last_pointer_updater field.
This hasn't done anything in at least 5 years.
2020-04-26 22:39:24 -07:00
Tim Abbott 731cd79436 messages: Remove first_unread pointer hack.
This hack was important when only the mobile apps (and not the webapp)
were using the unread_msgs data structure and the first_unread
infrastructure.  Now that the webapp is using those things, there
aren't leaked ancient unread messages that aren't accessible on the
webapp, so any few users still in this situation can get out of it by
just reading the problematic messages.
2020-04-26 22:37:41 -07:00
Tim Abbott 06f3664cde pointer: Remove unread debugging management commands.
I don't think we've had a use for these tools since our unread systems
stabilized shortly after they were written, so it makes sense to just
remove them rather than updating them for the pointer migration.
2020-04-26 22:37:41 -07:00
Tim Abbott 69ae4931c3 migrations: Use django.db.backends.postgresql.schema.
This replaces django.db.backends.postgresql_psycopg2, which has been
an alias to django.db.backends.postgresql since Django 1.9.
2020-04-26 22:20:24 -07:00
Abhishek-Balaji 2ff1527be8 models: Switch from NullBooleanField to BooleanField.
In Django 2.1, the preferred way to express a nullable BooleanField
changed from NullBooleanField to passing null=True to BooleanField.

This updates our codebase to use the preferred API.  Tweaked by
tabbott to update the linter rules.

The migration is a noop for Django accounting only.

Part of #11341.
2020-04-26 22:13:28 -07:00
Mateusz Mandera 3b582931af migration: Migrate messages tied to removed bots.
This cleans up any messages that might have been exchanged with
`NEW_USER_BOT` or `FEEDBACK_BOT` (cross-realm bots that were last
used, as far as we know, years ago) that have been completely removed
from the codebase.

Details on the algorithm are in the migration code itself.

Fixes #13583.
2020-04-26 11:08:39 -07:00
rebtung f7fbe3419f api: Improve consistency of reactions API.
Previously, the message and event APIs represented the user differently
for the same reaction data. To make this more consistent, I added a
user_id field to the reaction dict for both messages and events. I
updated the front end to use the user_id field rather than the user
dict. Lastly, I updated front end and back end tests that used user
info.

I primarily tested this by running my local Zulip build and
adding/removing reactions from messages.

Fixes #12049.
2020-04-26 10:35:55 -07:00
Puneeth Chaganti 2a65be2bf5 url preview: Use Chrome's user agent instead of a Zulip one.
Some sites don't render correctly unless you are one of the latest browsers.
YouTube Music, for instance, changes the page title to "Your browser is
deprecated, please upgrade.", which makes our URL previews look bad.
2020-04-26 10:16:43 -07:00
pemontto 01a57ddd9d webhooks: Use Gitlab project web_url over repo homepage.
This is preferred because the `repository` field isn't available for
project-level events.
2020-04-24 17:50:47 -07:00
Wyatt Hoodes 7f637f7cb1 migrations: Deprecate migration 0077 into a noop.
This ancient migration imports boto, which interferes
with our upgrade to boto3.

> git name-rev f13d6a18eb
f13d6a18eb tags/1.6.0~1082

We can safely assume nobody is upgrading from a server on <1.6.0,
since we have no supported platforms in common with those releases.
2020-04-24 17:03:14 -07:00
Wyatt Hoodes 6d686d0e4b migrations: Deprecate migration 0064 into a noop.
This ancient migration imports boto, which interferes
with our upgrade to boto3.

> git name-rev 8ae35211b5
8ae35211b5 tags/1.6.0~1924

We can safely assume nobody is upgrading from a server on <1.6.0,
since we have no supported platforms in common with those releases.
2020-04-24 17:03:14 -07:00
Wyatt Hoodes 3e8d877564 migrations: Deprecate migration 0060 into a noop.
This ancient migration imports boto, which interferes
with our upgrade to boto3.

> git name-rev a32f666f5c
a32f666f5c tags/1.6.0~2384

We can safely assume nobody is running servers on <1.6.0; there are no
supported platforms in common with 1.6.0 anyway.
2020-04-24 17:03:14 -07:00
Rohitt Vashishtha ce6c57b714 markdown: Set default code block language for quotes and latex as well.
In the original implementation, we were checking for the default language
inside format_code, which resulted in the setting being ignored when set to
quote, math, tex or latex. We shift the validation to `check_for_new_fence`

We also update the tests to use a saner naming scheme for the variables.
2020-04-23 17:51:01 -07:00
Dinesh 2735860f01 browser_support: Drop support for Internet Explorer.
Internet Explorer does not support `position: sticky` which improves
floating recipient bar behavior during scrolling which is one of the
issues blocking PR #9910.
IE also does not support some features that modern browsers support
hence may not super well.
This commit adds an error page that'll be displayed when a user logs
in from Internet Explorer. Also, a test is added.
2020-04-23 17:50:28 -07:00
Eeshan Garg 75b2264a3f pypi: Upgrade zulip/zulip-bots dependencies to version 0.7.0.
Includes this change:
* openapi/python_examples: Update get_single_user.

This updates get_single_user to pass keyword arguments to
get_user_by_id instead of passing a dictionary.

Which is required for CI to pass, as we indeed fixed the API of that
function (which had only been present with the wrong API for one release).
2020-04-23 17:41:47 -07:00
Vishnu KS 22b5297e32 actions: Mark strings in send_welcome_bot_response for translation. 2020-04-23 17:19:27 -07:00
Vishnu KS 03286012d2 onboarding: Mark strings in send_initial_realm_messages for translation. 2020-04-23 17:18:39 -07:00
Vishnu KS 9a2ee29ac7 onboarding: Mark strings in send_initial_pms for translation. 2020-04-23 17:15:31 -07:00
Rohitt Vashishtha 2825e6ad48 i18n: Support subexpressions of type (t "text") in Handlebars. 2020-04-22 17:57:16 -07:00
sahil839 3a7de8ad3b models: Add has_permission as a generic function for different policies.
This commit removes can_create_streams and can_subscribe_other_users
to use has_permission as a generic function in UserProfile model for
these settings policies.

Relevant changes are made to events.py to avoid duplication at some
places.
2020-04-22 17:48:52 -07:00
Puneeth Chaganti 87648596ee integrations: Add script to create bot avatars for integrations.
These avatars can be directly used by bots used for documenting these
integrations. In future, these could be used by bots created from a web UI.
2020-04-22 17:45:30 -07:00
Puneeth Chaganti 679f092a75 integrations: Extract code to get logo path for integrations. 2020-04-22 17:45:30 -07:00
Steve Howell 02252c255a db tools: Use common scheme for digests.
We have two different digest schemes to make
sure we keep the database up to date.  There
is the migration digest, which is NOT in the
scope of this commit, and which already
used the mechanism we use for other tools.

Here we are talking about the digest for
important files like `populate_db.py`.

Now our scheme is more consistent with how we
check file changes for other tools (as
well as the aformentioned migration files).

And we only write one hash file, instead of
seven.

And we only write the file when things have
actually changed.

And we are explicit about side effects.

Finally, we include a couple new bot settings
in the digest:

    INTERNAL_BOTS
    DISABLED_REALM_INTERNAL_BOTS

NOTE: This will require a one-time transition,
where we rebuild both databases (dev/test).
It takes a little over two minutes for me,
so it's not super painful.

I bump the provision version here, even
though you don't technically need it (since
the relevant tools are actually using the
digest files to determine if they need to
rebuild the database).  I figure it's just
good to explicitly make this commit trigger
a provision, and the user will then see
the one-time migration of the hash files
with a little bit less of a surprise.

And I do a major bump, not a minor bump,
because when we go in the reverse direction,
the old code will have to rebuild the
database due to the legacy hash files not
being around, so, again, I just prefer it
to be explicit.
2020-04-22 14:41:42 -07:00
Mateusz Mandera ec6022a186 logging_util: Handle record.status_code missing in skip_200_and_304.
Fixes #14595.
Invalid HTTP requests could end up in an unhandled exception in
skip_200_and_304 due the record not having the status_code attribute
set. With this change we'll avoid the exception

Example:
curl  -X POST -H 'Transfer-Encoding : chunked' --data-binary 'a' 'http://zulipdev.com:9991/json/messages/57'

2020-04-21 10:56:22.007 WARN [django.server] "POST /json/messages/57 HTTP/1.1" 405 95
2020-04-21 10:56:22.007 INFO [django.server] code 400, message Bad request syntax ('a')
2020-04-21 10:56:22.008 WARN [django.server] "a" 400 -
2020-04-22 11:26:13 -07:00
Anders Kaseorg fead14951c python: Convert assignment type annotations to Python 3.6 style.
This commit was split by tabbott; this piece covers the vast majority
of files in Zulip, but excludes scripts/, tools/, and puppet/ to help
ensure we at least show the right error messages for Xenial systems.

We can likely further refine the remaining pieces with some testing.

Generated by com2ann, with whitespace fixes and various manual fixes
for runtime issues:

-    invoiced_through: Optional[LicenseLedger] = models.ForeignKey(
+    invoiced_through: Optional["LicenseLedger"] = models.ForeignKey(

-_apns_client: Optional[APNsClient] = None
+_apns_client: Optional["APNsClient"] = None

-    notifications_stream: Optional[Stream] = models.ForeignKey('Stream', related_name='+', null=True, blank=True, on_delete=CASCADE)
-    signup_notifications_stream: Optional[Stream] = models.ForeignKey('Stream', related_name='+', null=True, blank=True, on_delete=CASCADE)
+    notifications_stream: Optional["Stream"] = models.ForeignKey('Stream', related_name='+', null=True, blank=True, on_delete=CASCADE)
+    signup_notifications_stream: Optional["Stream"] = models.ForeignKey('Stream', related_name='+', null=True, blank=True, on_delete=CASCADE)

-    author: Optional[UserProfile] = models.ForeignKey('UserProfile', blank=True, null=True, on_delete=CASCADE)
+    author: Optional["UserProfile"] = models.ForeignKey('UserProfile', blank=True, null=True, on_delete=CASCADE)

-    bot_owner: Optional[UserProfile] = models.ForeignKey('self', null=True, on_delete=models.SET_NULL)
+    bot_owner: Optional["UserProfile"] = models.ForeignKey('self', null=True, on_delete=models.SET_NULL)

-    default_sending_stream: Optional[Stream] = models.ForeignKey('zerver.Stream', null=True, related_name='+', on_delete=CASCADE)
-    default_events_register_stream: Optional[Stream] = models.ForeignKey('zerver.Stream', null=True, related_name='+', on_delete=CASCADE)
+    default_sending_stream: Optional["Stream"] = models.ForeignKey('zerver.Stream', null=True, related_name='+', on_delete=CASCADE)
+    default_events_register_stream: Optional["Stream"] = models.ForeignKey('zerver.Stream', null=True, related_name='+', on_delete=CASCADE)

-descriptors_by_handler_id: Dict[int, ClientDescriptor] = {}
+descriptors_by_handler_id: Dict[int, "ClientDescriptor"] = {}

-worker_classes: Dict[str, Type[QueueProcessingWorker]] = {}
-queues: Dict[str, Dict[str, Type[QueueProcessingWorker]]] = {}
+worker_classes: Dict[str, Type["QueueProcessingWorker"]] = {}
+queues: Dict[str, Dict[str, Type["QueueProcessingWorker"]]] = {}

-AUTH_LDAP_REVERSE_EMAIL_SEARCH: Optional[LDAPSearch] = None
+AUTH_LDAP_REVERSE_EMAIL_SEARCH: Optional["LDAPSearch"] = None

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-04-22 11:02:32 -07:00
Anders Kaseorg af1aef9b39 models: Move flush_per_request_caches after cache definitions.
This will work around https://bugs.python.org/issue34939 when we
convert the type comment to a Python 3.6 style annotation.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-04-22 10:49:20 -07:00
Anders Kaseorg f8c95cda51 mypy: Add specific codes to type: ignore annotations.
https://mypy.readthedocs.io/en/stable/error_codes.html

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-04-22 10:46:33 -07:00
Anders Kaseorg 029bfb9fee mypy: Remove unnecessary type: ignore annotations.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-04-22 10:46:33 -07:00