Commit Graph

924 Commits

Author SHA1 Message Date
Tim Abbott 10c47b5d6c version: Update version and changelog after 4.6 release. 2021-09-23 16:19:57 -07:00
Sahil Batra c233ee9935 settings: Migrate twenty_four_hour_time setting to RealmUserDefault.
This commit removes the existing default_twenty_four_hour_time field in
Realm table which was used to set the twenty_four_hour_time setting of
new user on joining and instead we now use the twenty_four_hour_time
field of RealmUserDefault table for the same.

With some tweaks by tabbott to clarify the documentation.
2021-09-23 10:44:42 -07:00
Eeshan Garg 4118c4a56b find_team: Send one email per email address, not per organization.
With changes to the copy for these emails by tabbott.

Fixes #19659.
2021-09-21 15:56:50 -07:00
Aman Agrawal 6a78112940 subscribe: Allow web public stream creation via the API.
User can now create web public stream via the /subscribe API.
So, when a web public stream present in the API request does not
exist, it will be created now by specifying the is_web_public
parameter. The parameter would have been ignored without this
commit.
2021-09-21 11:20:36 -07:00
Anders Kaseorg 6a1a665f11 requirements: Upgrade Python requirements.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-09-17 17:05:18 -07:00
Anders Kaseorg 5ab90c541c requirements: Install gitlint from its Debian package.
gitlint has a bunch of pinned requirements that hold back important
upgrades and conflict with other packages’ requirements.  The gitlint
author has rejected proposals to unpin them because it might increase
the amount of maintenance he needs to do
(https://github.com/jorisroovers/gitlint/pull/133).  That decision is
his to make, but _somebody_ needs to do the maintenance, so we
delegate it to Debian and Ubuntu.  If that means using a significantly
older version of gitlint, that’s a tradeoff we need to make to keep
the rest of our requirements current.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-09-17 17:05:18 -07:00
Anders Kaseorg 79b88b79bb webpack: Upgrade to Webpack 5.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-09-14 17:02:03 -07:00
Rohitt Vashishtha ddcea9c64b postcss: Cleanly import flatpickr dark theme.
We add postcss-import for night_mode.css only. This plugin inlines
the imports of external files, instead of letting the file go via
our usual webpack toolchain.

We do this so that we can use the postcss-prefixwrap plugin to scope
the third-party CSS properly and use it inside our night-mode class.

Fixes #10607.

[anders@zulip.com: Replace postcss-wrap with postcss-prefixwrap.]

Co-authored-by: Anders Kaseorg <anders@zulip.com>
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-09-13 17:12:49 -07:00
Anders Kaseorg 9bed17e0ab install-node: Upgrade Node.js from 14.17.5 to 14.17.6.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-09-13 10:12:43 -07:00
Anders Kaseorg dadacc38a8 dependencies: Upgrade JavaScript dependencies.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-09-13 10:12:43 -07:00
Anders Kaseorg 04075201a0 dependencies: Upgrade webpack-dev-server to 4.2.0.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-09-13 10:12:43 -07:00
Riken Shah 8c31e6f96e emoji: Add backend changes to support still image for animated emojis.
Now, when we add a custom animated emoji to the realm
we also save a still image of it (1st frame of the gif). So
we can avoid showing an animated emoji every time.
2021-09-12 07:13:04 +00:00
Anders Kaseorg b7b4c033a5 check-openapi: Use yaml library for better error messages.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-09-10 10:39:25 -07:00
Sahil Batra 693d58265e realm: Add 'PATCH /realm/user_settings_defaults' endpoint.
The realm-level default value of settngs for new users will
be updated using this endpoint.
2021-09-09 10:55:18 -07:00
Sahil Batra 17087cf06f actions: Add do_set_realm_user_default_setting.
This commit adds do_set_realm_user_default_setting which
will be used to change the realm-level defaults of settings
for new users.

We also add a new event type "realm_user_settings_defaults"
for these settings and a "realm_user_settings_default" object
in '/register' response containing all the realm-level default
settings.
2021-09-09 10:38:38 -07:00
Eeshan Garg f0ea002d94 events: Add demo_organization_scheduled_deletion_date to /register.
This commit adds `demo_organization_scheduled_deletion_date` to
the `realm` section of the `/register` response so that it is
available to clients when enabled.

This is a part of #19523.
2021-09-07 17:40:33 -07:00
Aman Agrawal 1347ee98ee compose: Extract function to format text in compose_ui.
Along with the extraction, we do some simplifications of inserting
text in compose too. This same function can now be used in
compose formatting popover too.

We use "text-field-edit", which has good cross-browser `undo` /
`redo` support, to do the text replace for us instead of writing
that logic ourselves.
2021-09-07 14:57:17 -07:00
Anders Kaseorg 817146c28b python: Upgrade SQLAlchemy from 1.3.24 to 1.4.23.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-08-31 06:47:39 -07:00
Mateusz Mandera 78297efefd ldap: Use a workaround instead of forking django-auth-ldap.
Till now, we've been forking django-auth-ldap at
https://github.com/zulip/django-auth-ldap to put the
LDAPReverseEmailSearch feature in it, hoping to get it merged
upstream in https://github.com/django-auth-ldap/django-auth-ldap/pull/150

The efforts to get it merged have stalled for now however and we don't
want to be on the fork forever, so this commit puts the email search
feature as a clumsy workaround inside our codebase and switches to using
the latest upstream release instead of the fork.
2021-08-28 23:11:09 -07:00
Alex Vandiver 710a92f48c tools: Add a tool to upload release tarballs to S3. 2021-08-26 17:21:31 -07:00
Anders Kaseorg 019bd848ca requirements: Upgrade Python requirements.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-08-24 07:51:48 -07:00
Anders Kaseorg cea9fe5f13 dependencies: Upgrade JavaScript dependencies.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-08-23 14:30:02 -07:00
Anders Kaseorg ad480a8b54 install-shfmt: Upgrade shfmt from 3.3.0 to 3.3.1.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-08-23 12:34:05 -07:00
Anders Kaseorg 7b2e585213 install-yarn: Upgrade Yarn from 1.22.10 to 1.22.11.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-08-23 12:33:27 -07:00
Anders Kaseorg ebb8e9109c install-node: Upgrade Node.js from 14.17.3 to 14.17.5.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-08-23 12:29:04 -07:00
Sahil Batra 5f5c88a4e2 user_groups: Add 'is_system_group' field to objects passed to clients. 2021-08-19 02:56:43 -07:00
Anders Kaseorg 672358c5da docs: Migrate from recommonmark to MyST-Parser.
Recommonmark is no longer maintained, and MyST-Parser is much more
complete.

https://myst-parser.readthedocs.io/

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-08-19 00:53:32 -07:00
Anders Kaseorg 1bdb7b1141 mypy: Add boto3-stubs.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-08-09 20:32:19 -07:00
Anders Kaseorg 05124737f7 mypy: Add backoff-stubs.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-08-09 20:32:19 -07:00
Anders Kaseorg 09564e95ac mypy: Add types-psycopg2.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-08-09 20:32:19 -07:00
Anders Kaseorg 5c90522e69 mypy: Add types-Pillow.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-08-09 20:32:19 -07:00
Anders Kaseorg 6289803368 mypy: Add types-oauthlib.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-08-09 20:32:19 -07:00
Anders Kaseorg 4c73926ad9 mypy: Add types-jsonschema.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-08-09 20:32:19 -07:00
Anders Kaseorg 58cfc152a0 mypy: Add types-commonmark.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-08-09 20:32:19 -07:00
Gaurav Pandey 067a32925d api: Default `client_gravatar` to `true`.
Default `client_gravatar` to `true` for all endpoints passing the
parameter to avoid extra payload.

Fixes #11214.
2021-08-08 16:53:05 -07:00
Sahil Batra 5459a92e4a setting: Use "unlimited" instead of "forever" for retention setting.
This commit updates both the stream-level and realm-level message
retention setting to use 'unlimited' instead of 'forever' to set
message retention setting to "retain messages forever".
2021-08-08 15:56:57 -07:00
Anders Kaseorg 1a3e3f9072 mypy: Add lxml-stubs.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-08-04 11:29:20 -07:00
Tim Abbott 1685d207c8 api docs: Fix API documentation bugs issues with last commit.
* Actually bump API_FEATURE_LEVEL.
* Document changes in the endpoint documentation as well.
* Use standard linking/wording approach in API changelog.
2021-08-04 10:55:03 -07:00
Anders Kaseorg 49907173b2 mypy: Add types-Pygments.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-08-02 22:31:46 -07:00
Anders Kaseorg 04feadd917 mypy: Add pika-stubs.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-08-02 22:31:46 -07:00
Sahil Batra 7959ae3fab events: Add new event type 'user_settings' for updating user settings.
We send a event with type 'user_settings' on updating user's display
and notification settings.

The old event types - 'update_global_notifications' and
'update_display_settings', are still supported for backwards
compatibility.
2021-08-01 15:30:17 -07:00
Gaurav Pandey d89b4dcd0b api: Return zulip_merge_base alongside zulip_version.
Return zulip_merge_base alongside zulip_version
in `/register`, `/event` and `/server_settings`
endpoint so that the value can be used by other
clients.
2021-07-30 12:03:41 -07:00
Hemanth V. Alluri c00089ac28 drafts: Add support for toggling drafts synchronization.
With changes mostly to the API documentation by tabbott.
2021-07-30 10:00:27 -07:00
Riken Shah 9fadd43830 user_status: Add backend changes to support status emoji.
In this commit:

* We update the `UserStatus` model to accept
`AbstractReaction` as a base class so, we can get all the
fields related to store status emoji.

* We update the user status endpoint
(`users/me/status`) to accept status emoji fields.

* We update the user status event to add status emoji
fields.

Co-authored-by: Yash Rathore <33805964+YashRE42@users.noreply.github.com>
2021-07-27 17:57:15 -07:00
sahil839 4b1313a92b models: Replace add_emoji_by_admins_only with add_custom_emoji_policy.
This commit replaces boolean field add_emoji_by_admins_only with an
integer field add_custom_emoji_policy as we would also add full members
and moderators option for this setting in further commits.
2021-07-27 16:41:22 -07:00
Sahil Batra 79b0ff5b23 events: Send enter_sends if update_display_settings in fetch_event_types.
We now send 'enter_sends' in response of '/register' endpoint if
'update_display_settings' is present in 'fetch_event_types' instead
of 'realm_user'.
2021-07-27 15:54:16 -07:00
Tim Abbott 361112a08f version: Update version and changelog after 4.5 release. 2021-07-25 16:05:22 -07:00
Mateusz Mandera 119f1da04a bots: Rename is_cross_realm_bot API field to is_system_bot.
Cross realm bots will soon stop being a thing. This param is responsible
for displaying "System Bot" in the user info popover - so this rename is the
right way to handle the situation.

We will likely want to rename the `cross_realm_bots` section as well,
but that is a more involved API migration.
2021-07-24 15:46:40 -07:00
Gaurav Pandey 7f0dd81ada
api: Bump feature level to 82.
Bump api feature level to 82 after changes in 5db4fe8652.
2021-07-24 09:48:47 -07:00
Alex Vandiver 34833aa795 version: Update version and changelog after 4.4 release. 2021-07-22 17:12:36 -07:00
Sahil Batra 318d71469b api: Remove '/users/me/enter-sends' endpoint.
We remove the '/users/me/enter-sends' endpoint
and 'enter_sends' setting will now be edited
using the '/settings' endpoint.
2021-07-22 11:01:43 -07:00
Tim Abbott 256091dc15 settings: Merge settings API endpoints.
This API change removes unnecessary complexity from a client that
wants to change a user's personal settings, and also saves developers
from needing to make decisions about what sort of setting something is
at the API level.

We preserve the old settings endpoints as mapping to the same function
as the new one for backwards-compatibility. We delete the
documentation for the old endpoints, though the documentation for the
merged /settings endpoint mentions how to use the old endpoints when
needed.

We migrate all backend tests to the new endpoints, except for
individual tests for each legacy endpoint to verify they still work.

Co-authored-by: sahil839 <sahilbatra839@gmail.com>
2021-07-19 21:14:32 -07:00
Tim Abbott 95606a7347 api: Return user IDs, not display emails, in subscribers endpoints.
Sometime in the deep past, Zulip the GET /users/me/subscriptions
endpoint started returning subscribers.  We noticed this and made it
optional via the include_subscribers parameter in
1af72a2745, however, we didn't notice
that they were being returned as emails rather than user IDs.

We migrated the core /register code paths to use subscriber IDs years
ago; this change completes that for the endpoints we forgot about.

The documentation allowed this error because we apparently had no
tests for this code path that used the actual API.
2021-07-18 11:32:28 -07:00
sahil839 d5a0c1ede5 api: Remove "full_name" and "account_email" from response of '/settings'.
We remove the "full_name" and "account_email" fields from the response
of 'PATCH /settings' endpoint. These fields were part of the response
to make sure that we tell that the parameters not present in response
were ignored.

We can remove these fields as 'ignored_parameters_unsupported' now
specifies which parameters were ignored and not supported by the
endpoint.
2021-07-16 16:10:15 -07:00
Priyansh Garg c1885c1b98 requirements: Add pymongo to support reading bson files.
Required for reading bson files from mongodb dump in rocketchat
import tool.
2021-07-15 14:28:23 -07:00
Anders Kaseorg de9ddc66eb dependencies: Upgrade JavaScript dependencies.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-07-14 16:47:00 -07:00
Priyank Patel fdedced521 ts: Convert password_quality module to TypeScript. 2021-07-09 12:42:31 -07:00
Gaurav Pandey d2074fc10c events: Remove `sender_id` from `delete_message` event.
Remove `sender_id` as part of responses from `/events`
api for `delete_message` event when `message_type` is
`private`.
2021-07-08 13:40:32 -07:00
Tim Abbott af47fa705e exceptions: Use HTTP 401 code for authentication errors. 2021-07-08 10:33:08 -07:00
Anders Kaseorg 604f6c491c webpack: Fix webpack-bundle-tracker hot update bug.
https://github.com/django-webpack/webpack-bundle-tracker/pull/101

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-07-06 15:53:48 -07:00
Anders Kaseorg 08592c9731 requirements: Upgrade Python requirements.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-07-06 08:40:32 -07:00
Anders Kaseorg 2aec7b1644 dependencies: Upgrade JavaScript dependencies.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-07-05 14:51:24 -07:00
Anders Kaseorg d83c91526b install-node: Upgrade Node.js from 14.17.0 to 14.17.3.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-07-05 14:51:24 -07:00
Anders Kaseorg 99533b4aa2 js: Fix, and check for, non-ES5 syntax in webpack bundles.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-07-05 13:32:37 -07:00
Anders Kaseorg c56440ded0 requirements: Upgrade Python requirements.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-07-05 12:23:06 -07:00
Vishnu KS e0f5fadb79 billing: Downgrade small realms that are behind on payments.
An organization with at most 5 users that is behind on payments isn't
worth spending time on investigating the situation.

For larger organizations, we likely want somewhat different logic that
at least does not void invoices.
2021-07-02 13:19:12 -07:00
sahil839 828759d2ba models: Replace allow_community_topic_editing with edit_topic_policy.
This commit replaces the allow_community_topic_editing boolean with
integer field edit_topic_policy and includes both frontend and
backend changes.

We also update settings_ui.disable_sub_settings_onchange to not
change the color of label as we did previously when the setting
was a checkbox. But now as the setting is dropdown we keep the
label as it is and we don't do anything with label when disabling
dropdowns. Also, this function was used only here so we can safely
change this.
2021-06-16 14:59:36 -07:00
Anders Kaseorg a9f6764d8c eslint: Add eslint-plugin-formatjs.
This guards against various mistakes, such as setting defaultMessage
to a computed expression that can’t be extracted.

https://formatjs.io/docs/tooling/linter/

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-06-13 22:32:40 -07:00
Anders Kaseorg 321016668e dependencies: Downgrade @types/css-minimizer-webpack-plugin.
This avoids pulling in an extra unused copy of webpack 5.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-06-09 16:17:32 -07:00
Anders Kaseorg bc169d63a7 install-node: Upgrade Node.js from 14.16.1 to 14.17.0.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-06-08 16:02:12 -07:00
Anders Kaseorg 3d8fc7d85f install-shfmt: Upgrade shfmt from 3.2.4 to 3.3.0.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-06-08 16:01:18 -07:00
Anders Kaseorg ee48dc6d33 install-shellcheck: Upgrade ShellCheck from 0.7.1 to 0.7.2.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-06-08 16:01:14 -07:00
Anders Kaseorg ad7466a2bf dependencies: Upgrade JavaScript dependencies.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-06-08 10:15:52 -07:00
Anders Kaseorg 61e1e38a00 requirements: Upgrade Python requirements.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-06-07 17:57:51 -07:00
Gaurav Pandey 9b696cf212 api: Expose event_queue_longpoll_timeout_seconds in /register.
Rename poll_timeout to event_queue_longpoll_timeout_seconds
and change its value from 90000 ms to 90 sec. Expose its
value in register api response when realm data is fetched.
Bump API_FEATURE_LEVEL to 74.
2021-06-05 07:37:19 -07:00
Vishnu KS 5db53029a5 api: Include is_billing_admin as an attribute in user response.
This is sufficiently useful that it should be made available to clients.
2021-06-03 10:27:07 -07:00
Tim Abbott 84c7c8cc0d version: Update version and changelog after 4.3 release. 2021-06-02 12:55:50 -07:00
Gaurav Pandey 64bd461bad register api: Rename realm_upload_quota.
Rename `realm_upload_quota` to `realm_upload_quota_mib`
reflecting file size in mebibytes. Update frontend and bump
API_FEATURE_LEVEL.
2021-05-26 11:38:24 -07:00
Sumanth V Rao 83a2479a5a build_pygments_data: Reprioritize and categorize pygment language codes.
We now organize the pygment language codes into meaningful categories
- default, custom and aliases.

Further the `lang.json` list now contains a dataset extracted from the
"language" section of https://insights.stackoverflow.com/survey/2020
and is prioritized based on current language trends.
2021-05-24 23:15:39 -07:00
Tim Abbott c9326d1060 version: Add reminder to document Changes details for API changes. 2021-05-24 13:37:22 -07:00
Mateusz Mandera e7033fcd41 dependecies: Install python-social-auth with OIDC support. 2021-05-23 13:30:17 -07:00
Aman Agrawal 70c0abc2e5 do_change_stream_invite_only: Ensure stream is not web public.
When changing stream permissions to invite_only or public, ensure
that stream doesn't have is_web_public set to True.
2021-05-20 15:23:19 -07:00
Tim Abbott 2e67b879ed api: Add server_timestamp to register response.
Since this is currently only useful to interpret presence data, we
send this only if presence is requested.

I'm not sure that server_timestamp is the right name for this field,
but ultimately it should match the main presence API format.
2021-05-20 14:57:34 -07:00
Anders Kaseorg e6651a1bbd requirements: Remove ijson.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-05-20 13:12:16 -07:00
Eeshan Garg 69309540c0 pypi: Upgrade zulip/zulip-bots dependencies to version 0.8.0. 2021-05-20 11:06:48 -07:00
Tim Abbott ed04966f79 version: Update version and changelog following 4.2 release. 2021-05-13 22:09:48 -07:00
Tim Abbott 6d4e7aa307 version: Update version and changelog following 4.1 release. 2021-05-13 18:58:11 -07:00
Tim Abbott c6de9736b8 version: Link blog post from 4.0 release. 2021-05-13 17:53:44 -07:00
Tim Abbott bf4f6544e7 version: Update version following 4.0 release. 2021-05-13 15:48:15 -07:00
Tim Abbott 825192a051 Release Zulip Server 4.0. 2021-05-13 15:37:37 -07:00
Tim Abbott 454eced466 version: Bump required desktop app versions.
Older desktop releases work fine against modern Zulip, but have
security issues that mean users should upgrade.
2021-05-13 12:47:53 -07:00
Anders Kaseorg 668b5137b0 version: Display Zulip version in About Zulip dialog.
We record Git details about the merge-base with upstream branches in
the zulip-git-version file, if the upstream repository is available.

Note that the first Git upgrade after merging the parent commit will
not include the merge-base details, since the upstream repository will
not have been available.

Co-authored-by: Tim Abbott <tabbott@zulip.com>
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-05-13 11:36:12 -07:00
Anders Kaseorg 42abc1b635 dependencies: Upgrade JavaScript dependencies.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-05-10 18:07:13 -07:00
Gaurav Pandey 96e035a2f0 api: Fix encoding of strings in streams endpoint.
* Remove unnecessary json_validator for string parameters.
* Update frontend to pass right parameter.

Bump api feature level and highlight the fix for `emojiset`
parameter of `settings/display` endpoint in zulip.yaml file.

Fixes part of #18035.
2021-05-10 10:29:22 -07:00
Anders Kaseorg 9ba48c4ed3 requirements: Upgrade Python requirements.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-05-07 22:42:39 -07:00
Gaurav Pandey 00647b6fa7 api: Fix encoding of strings in video calls endpoint.
This removes unnecessary json_validator for string parameters in the
BigBlueButton video calls endpoints.  Note that this breaks links to
video meetings sent before the upgrade; there's not much we can do
about that.

Since this is the last commit in this series, we update the
ZULIP_FEATURE_LEVEL for this batch of changes.

Fixes part of #18035.
2021-05-07 14:11:48 -07:00
Anders Kaseorg 405bc8dabf requirements: Remove Thumbor.
Thumbor and tc-aws have been dragging their feet on Python 3 support
for years, and even the alphas and unofficial forks we’ve been running
don’t seem to be maintained anymore.  Depending on these projects is
no longer viable for us.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-05-06 20:07:32 -07:00
Tim Abbott 4fd0725678 version: Update for 4.0-rc1 release. 2021-05-03 17:37:01 -07:00
sahil839 6e672ebbee settings: Add moderators option for wildcard_mention_policy. 2021-05-03 12:12:01 -07:00
Mateusz Mandera 1d9fb4f988 django: Upgrade Zulip to Django 3.2 LTS.
This is a straightforward upgrade in terms of changes needed.

Necessary changes were:
- Set `DEFAULT_AUTO_FIELD`
  https://docs.djangoproject.com/en/3.2/releases/3.2/#customizing-type-of-auto-created-primary-keys

- `The default_app_config application configuration variable is deprecated, due
  to the now automatic AppConfig discovery.`
  https://docs.djangoproject.com/en/3.2/releases/3.2/#automatic-appconfig-discovery

To handle this one, we can remove default_app_config from
zerver/__init__.py because it satisfies what release notes describe in
https://docs.djangoproject.com/en/3.2/releases/3.2/#automatic-appconfig-discovery:
"Most pluggable applications define an AppConfig subclass in an apps.py
submodule. Many define a default_app_config variable pointing to this
class in their __init__.py.  When the apps.py submodule exists and
defines a single AppConfig subclass, Django now uses that configuration
automatically, so you can remove default_app_config."

An important note is that rebuild-test-database needs to be run after
this upgrade in dev environment - if tests are run with test db that was
built on the previous version, they will fail due to a mysterious bug
(?), where changing attributes of a user and .save()ing after logging in
in the test via self.login_user, causes getting logged out - the next
requests via self.client_get etc. are unauthed for some reason,
unless self.login_user is called again. This behavior is no longer
exhibited upon rebuilding the test db - and I can't reproduce it in
production or dev db. So this can likely be reasonably dismissed as some
quirk of the test client system that won't be relevant in the future and
doesn't impact production.
2021-05-03 08:36:22 -07:00
sahil839 e669818544 invites: Allow user to be invited as a moderator.
This commit adds both frontend and backend code to invite a user as
moderator. We allow only existing owners and admins to invite a user
as a moderator.
2021-04-30 15:57:09 -07:00
sahil839 bd78b1ff90 events: Add 'is_moderator' field to the page_params object sent to clients. 2021-04-29 15:18:49 -07:00
sahil839 34f134d58d users: Add role field to user objects returned by format_user_row.
This commit modifies the user objects returned by 'GET /users',
'GET /users/me', 'GET /users/{user_id}' and 'GET /users/{email}'
endpoints to include role field.

We also include role field in the page_params['realm_users'] dict
and in the person object sent in (type="realm_user", op="add")
event.
2021-04-29 15:13:50 -07:00
Dinesh 27e4f5da92 typing: Support sending stream/topic typing status.
This extends the /json/typing endpoint to also accept
stream_id and topic. With this change, the requests
sent to /json/typing should have these:
* `to`: a list set to
    - recipients for a PM
    - stream_id for a stream message
* `topic`, in case of stream message
along with `op`(start or stop).

On receiving a request with stream_id and topic, we send
typing events to clients with stream_typing_notifications set
to True for all users subscribed to that stream.
2021-04-27 20:52:21 -07:00
Cyril Pletinckx 9afde790c6 email: Open a single SMTP connection to send email batches.
Previously the outgoing emails were sent over several SMTP
connections through the EmailSendingWorker; establishing a new
connection each time adds notable overhead.

Redefine EmailSendingWorker worker to be a LoopQueueProcessingWorker,
which allows it to handle batches of events. At the same time, persist
the connection across email sending, if possible.

The connection is initialized in the constructor of the worker
in order to keep the same connection throughout the whole process.
The concrete implementation of the consume_batch function is simply
processing each email one at a time until they have all been sent.

In order to reuse the previously implemented decorator to retry
sending failures a new method that meets the decorator's required
arguments is declared inside the EmailSendingWorker class. This
allows to retry the sending process of a particular email inside
the batch if the caught exception leaves this process retriable.

A second retry mechanism is used inside the initialize_connection
function to redo the opening of the connection until it works or
until three attempts failed. For this purpose the backoff module
has been added to the dependencies and a test has been added to
ensure that this retry mechanism works well.

The connection is closed when the stop method is called.

Fixes: #17672.
2021-04-26 17:27:22 -07:00
Anders Kaseorg dd3fa4ac52 templates: Mark all void tags as self-closing.
This reverses the policy that was set, but incompletely enforced, by
commit 951514dd7d.  The self-closing tag
syntax is clearer, more consistent, simpler to parse, compatible with
XML, preferred by Prettier, and (most importantly now) required by
FormatJS.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-04-21 09:49:34 -07:00
Anders Kaseorg 7177529107 templates: Use HTML5 doctype for emails.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-04-21 09:49:34 -07:00
akshatdalton 6509c4f8f4 linkifiers: Add an API to support the editing of linkifier.
This commit adds an API to `zproject/urls.py` to edit/update
the realm linkifier. Its helper function to update the
database is added in `zerver/lib/actions.py`.

`zulip.yaml` is documented accordingly as well, clearly
stating that this API updates one linkifier at a time.

The tests are added for the API and helper function which
updates the realm linkifier.

Fixes #10830.
2021-04-19 18:01:45 -07:00
sahil839 2dc99aa90f settings: Add new setting for controlling who can move msgs to stream.
This commit adds a new setting 'move_messages_between_streams_policy`
for controlling who can move messages from one stream to other.
2021-04-16 15:10:39 -07:00
Anders Kaseorg 36e938bf9f requirements: Upgrade Python requirements.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-04-15 21:47:33 -07:00
Aman Agrawal 802c450b3f realm: Add setting to configure GIPHY rating.
Organization admins can use this setting to restrict the maximum
rating of GIFs that will be retrieved from GIPHY. Also, there
is option to disable GIPHY too.
2021-04-14 10:29:39 -07:00
Anders Kaseorg d8a36d0702 i18n: Remove i18next.
Fixes #17890.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-04-13 17:41:10 -07:00
Anders Kaseorg 4b4cea90a3 i18n: Initialize FormatJS.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-04-13 17:41:10 -07:00
Abhijeet Prasad Bodas 3947b0c80a linkifiers: Update API to send data using dictionaries.
* This introduces a new event type `realm_linkifiers` and
a new key for the initial data fetch of the same name.
Newer clients will be expected to use these.

* Backwards compatibility is ensured by changing neither
the current event nor the /register key. The data which
these hold is the same as before, but internally, it is
generated by processing the `realm_linkifiers` data.
We send both the old and the new event types to clients
whenever the linkifiers are changed.
Older clients will simply ignore the new event type, and
vice versa.

* The `realm/filters:GET` endpoint (which returns tuples)
is currently used by none of the official Zulip clients.
This commit replaces it with `realm/linkifiers:GET` which
returns data in the new dictionary format.
TODO: Update the `get_realm_filters` method in the API
bindings, to hit this new URL instead of the old one.

* This also updates the webapp frontend to use the newer
events and keys.
2021-04-13 12:16:07 -07:00
Tim Abbott 4a3ad0da06 api: Improve encoding of stream/topic max field lengths.
Previously, you had to request the `stream` event type in order to get
the stream-level parameters; this was a bad design in part because the
`subscription` event type has similar data and is preferred by most
clients.

So we move these to the `realm` object.  We also add the maximum topic
length, as an adjacent parameter.

While changing this, we also fix these to better match the names of
similar API parameters.
2021-04-10 10:07:57 -07:00
Tim Abbott cdbcb43706 api: Fix encoding of strings in realm endpoint.
* Don't require strings to be unnecessarily JSON-encoded.
* Use check_capped_string rather than custom code for length checks.
* Update frontend to pass the right parameters.

With a much simplified populate_data_for_request design suggested by
Anders; we only support a handful of data types, all of which are
correctly encoded automatically by jQuery.

Fixes part of #18035.
2021-04-09 16:27:30 -07:00
Anders Kaseorg b060feba4c dependencies: Upgrade Source Code Pro font from Git.
This works around a bug where code text appears dark on dark in night
mode with Firefox on macOS:
https://bugzilla.mozilla.org/show_bug.cgi?id=1520157

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-04-08 22:59:24 -07:00
Anders Kaseorg de014a30e0 dependencies: Upgrade JavaScript dependencies.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-04-07 21:05:01 -07:00
Anders Kaseorg b6b117274c install-node: Upgrade Node.js to 14.16.1 and nvm to 0.38.0.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-04-07 21:05:01 -07:00
Anders Kaseorg 6bc3b267e1 webpack: Switch to css-minimizer-webpack-plugin.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-04-07 18:43:36 -07:00
Anders Kaseorg 2ce24bcd2b dependencies: Upgrade webpack-cli from 3.3.12 to 4.6.0.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-04-07 18:43:36 -07:00
sahil839 86f7b6af60 events: Pass can_invite_others_to_realm to client using page_params.
This commit adds backend code for passing can_invite_others_to_realm
field to clients using the fetch_initial_state_data in the page_params
object.

Though this field is not used by webapp as of now, but will be used
to fix a bug of incorreclty showing the invite users option in
settings overlay in the next commit.
2021-04-07 09:05:16 -07:00
sahil839 4c8339fa8c settings: Replace invite_by_admins_policy with invite_to_realm_policy.
This commit replaces invite_by_admins_policy, which was a bool field,
with a new enum field invite_by_realm_policy.

Though the final goal is to add moderators and full members option
using COMMON_POLICY_TYPES, but this will be done in a separate
commit to make this easy for review.
2021-04-07 09:02:33 -07:00
Aman Agrawal 9c6f5d5946 tippyjs: Add library to be used for showing tooltips.
Our aim is to use this library to remove use of bootstrap-tooltip
for showing popovers and tooltips. This will remove our
dependency on bootstrap for showing tooltips. Thus, bootstrap
can be upgrade more independently.
2021-04-07 01:16:28 -07:00
Sumanth V Rao 1ac8fe7538 events/tests/api: Send realm_playground events to clients.
We send the whole data set as a part of the event rather than
doing an add/remove operation for couple of reasons:
    * This would make the client logic simpler.
    * The playground data is small enough for us to not worry
      about performance.

Tweaked both `fetch_initial_state_data` and `apply_events` to
handle the new playground event.

Tests added to validate the event matches the expected schema.

Documented realm_playgrounds sections inside /events and
/register to support our openapi validation system in test_events.

Tweaked other tests like test_event_system.py and test_home.py
to account for the new event being generated.

Lastly, documented the changes to the API endpoints in
api/changelog.md and bumped API_FEATURE_LEVEL.

Tweaked by tabbott to add an `id` field in RealmPlayground objects
sent to clients, which is essential to sending the API request to
remove one.
2021-04-06 20:56:58 -07:00
Abhijeet Prasad Bodas 3bfcaa3968 mute user: Add backend infrastructure code.
Adds backend code for the mute users feature.
This is just infrastructure work (database
interactions, helpers, tests, events, API docs
etc) and does not involve any behavioral/semantic
aspects of muted users.

Adds POST and DELETE endpoints, to keep the
URL scheme mostly consistent in terms of `users/me`.

TODOs:
1. Add tests for exporting `zulip_muteduser` database table.
2. Add dedicated methods to python-zulip-api to be used
   in place of the current `client.call_endpoint` implementation.
2021-04-06 18:44:08 -07:00
Anders Kaseorg 3c41db7f1a dependencies: Upgrade to webpack-bundle-tracker 1.0.0-alpha.1.
This also seems unmaintained, but is, at least, released.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-04-06 09:31:35 -07:00
Aman Agrawal 5e83965e80 giphy: Use GIPHY web SDK to allow inserting GIFs in compose box.
We use GIPHY web SDK to create popover containing GIFs in a
grid format. Simply clicking on the GIFs will insert the GIF in the compose
box.

We add GIPHY logo to compose box action icons which opens the GIPHY
picker popover containing GIFs with "Powered by GIPHY"
attribution.
2021-04-05 15:04:49 -07:00
Sumanth V Rao e12f682e2e markdown: Include text & url in `topic_links` parameter of our API.
The linkifier code now includes both the shortened text and the expanded
URL, sorted by the order of the occurrence in a topic. This list is passed
back in the `topic_links` parameter of the /messages and the /events APIs.

topic_links earlier vs now:

earlier: ['https://www.google.com', 'https://github.com/zulip/zulip/32']

now: [{'url': 'https://www.google.com', 'text': 'https://www.google/com},
      {'url': 'https://github.com/zulip/zulip/32', 'text': '#32'}]

Similarly, the topic_links local echo logic in the frontend now returns
back an object.

Fixes: #17109.
2021-03-30 15:53:07 -07:00
Anders Kaseorg d55dc6f8f1 requirements: Upgrade python-zulip-api from Git.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-03-26 16:31:03 -07:00
shanukun bc2d58ad4a custom_profile_fields: Remove op field for the event.
* `op` (operation) field, added in f6fb88549f, was never intended for
`custom_profile_fields` event. This commit removes the `op` as it doesn't
have any use in the code.

* As a part of cleanup, this also eliminates the schema check warnings
for `custom_profile_fields` event, mentioned in #17568.
2021-03-26 16:28:33 -07:00
Anders Kaseorg 4703256c91 requirements: Upgrade Python requirements.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-03-26 08:51:16 -07:00
Anders Kaseorg 2831f9b60e install-shfmt: Upgrade shfmt from 3.2.2 to 3.2.4.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-03-26 08:51:16 -07:00
Anders Kaseorg e62b0d6b5f puppeteer_tests: Translate logged traces and errors with source maps.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-03-25 02:05:19 -07:00
Tim Abbott f121e40848 message: Record whether unread_msgs data is truncated.
This is preparatory work for investigating reports of missing unread
messages.

It's a little surprising that not test failed after adding the code
without API documentation.

Co-Author-By: Tushar Upadhyay (tushar912).
2021-03-21 19:48:13 -07:00
Anders Kaseorg 6364e1b5f3 requirements: Upgrade talon fork to 1.4.8.
https://github.com/mailgun/talon/pull/200

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-03-18 17:10:18 -07:00
Tushar912 55de66f944 api: Support user_id in get_user_presence_endpoint.
This is part of our general process of replacing emails, which are not
static with time, with user_ids when referring to users in the API.

We still keep the `email` reference option, since it can be useful for
linking third-party applications to Zulip on an intranet that might
have a user's corporate email handy and not want to do the extra round
trip to lookup the user.

The name of the parameter, user_id_or_email, was chosen to to make it
clear that the default/preferred option is user_id.

Fixes #14304.
2021-03-11 20:14:04 -08:00
Aman Agrawal e587c029f6 display_settings: Allow user to set default_view.
TextField is used to allow users to set long stream + topic narrow
names in the urls.

We currently restrict users to only set "all_messages" and
"recent_topics" as narrows.

This commit achieves 3 things:
* Removes recent topics as the default view which loads when
  hash is empty.
* Loads default_view when hash is empty.
* Loads default_view on pressing escape key when it is unhandled by
  other present UI elements.

NOTE: After this commit loading zulip with an empty hash will
automatically set hash to default_view.  Ideally, we'd just display
the default view without a hash, but that involves extra complexity.

One exception is when user is trying to load an overlay directly,
i.e. zulip is loaded with an overlay hash. In this case,
we render recent topics is background irrespective of default_view.

We consider this last detail to be a bug not important enough to block
adding this setting.
2021-03-11 18:09:08 -08:00
sahil839 9a432b0c3b events: Remove name field from update subscription events.
This commit removes name field from update subscription
events, as it is not used by any of the clients, and use
stream_id in the events code instead.
2021-03-07 22:03:24 -08:00
Steve Howell 30c7108955 zjsunit: Remove rewiremock dependency.
We now just use a module._load hook to inject
stubs into our code.

For conversion purposes I temporarily maintain
the API of rewiremock, apart from the enable/disable
pieces, but I will make a better wrapper in an
upcoming commit.

We can detect when rewiremock is called after
zrequire now, and I fix all the violations in
this commit, mostly by using override.

We can also detect when a mock is needlessly
created, and I fix all the violations in this
commit.

The one minor nuisance that this commit introduces
is that you can only stub out modules in the Zulip
source tree, which is now static/js.  This should
not really be a problem--there are usually better
techniques to deal with third party depenencies.
In the prior commit I show a typical workaround,
which is to create a one-line wrapper in your
test code.  It's often the case that you can simply
use override(), as well.

In passing I kill off `reset_modules`, and I
eliminated the second argument to zrequire,
which dates back to pre-es6 days.
2021-03-06 11:10:57 -05:00
Alex Vandiver ea2e5de54f requirements: Downgrade jedi dependency for ipython compatibility.
ipython < 7.20.0 is incompatible with jedi >= 0.18.0; it fails to
tab-complete in `./manage.py shell`, as described in
ipython/ipython#12740.

We cannot bump the ipython dependency because ipython 7.20.0 requires
Python 3.7, and we must support Python 3.6 due to Ubuntu 18.04
support.  Our only solution is thus to cap the version of `jedi` to
the last one before its API changed.
2021-03-04 18:08:45 -08:00
Anders Kaseorg 4bbfac8aa9 requirements: Upgrade Python requirements.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-03-03 21:46:42 -08:00
Anders Kaseorg b8e5e34c01 install-shfmt: Upgrade shfmt from 3.2.1 to 3.2.2.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-03-03 21:46:42 -08:00
Anders Kaseorg 25bb98dcf5 install-node: Upgrade Node.js from 14.15.1 to 14.16.0.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-03-03 21:46:42 -08:00
Anders Kaseorg 9613c66560 dependencies: Upgrade JavaScript dependencies.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-03-03 21:46:42 -08:00
Anders Kaseorg cce3257f2c zjsunit: Remove babel-plugin-lazy-require.
There are no more require() calls in tested modules.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-03-02 17:06:35 -08:00
sahil839 b53c773987 events: Remove email field from update subscription events.
This commit removes email field from update subscription
events, as email field is of no use in this case.
2021-03-01 14:52:06 -08:00
Anders Kaseorg 4df3fef88f zjsunit: Add babel-plugin-lazy-require.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-02-28 14:23:00 -08:00
Anders Kaseorg 3af6724438 dependencies: Upgrade clean-css 5.0.1 to 5.1.0.
This fixes a minification issue affecting our orange user circles.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-02-25 20:08:36 -08:00
Anders Kaseorg ba0ee6ddfa dependencies: Upgrade Source Sans Pro font to Source Sans 3.
https://blog.adobe.com/en/2020/11/30/whats-new-in-source-sans-3.html

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-02-23 14:55:33 -08:00
Anders Kaseorg cdff4cfe53 dependencies: Add ts-node.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-02-22 16:03:10 -08:00
Anders Kaseorg 3869727655 dependencies: Upgrade puppeteer from 7.0.4 to 7.1.0.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-02-22 16:03:10 -08:00
Tushar912 dfafdda9b3 api: Add REST API endpoint for looking up a user by email address.
Add new rest api endpoint GET users/{email} for looking up a user by
email, which is useful especially for corporate API applications that
might already have a user's email address.

Fixes #14302.
2021-02-15 17:38:33 -08:00
Anders Kaseorg f21fbea7c7 lint: Replace (most of) pycodestyle with Black.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-02-12 13:11:19 -08:00
Anders Kaseorg 6e4c3e41dc python: Normalize quotes with Black.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-02-12 13:11:19 -08:00
Anders Kaseorg 11741543da python: Reformat with Black, except quotes.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-02-12 13:11:19 -08:00
Anders Kaseorg 2024859d74 dependencies: Upgrade JavaScript dependencies.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-02-10 15:55:49 -08:00
Anders Kaseorg f38d68e110 requirements: Upgrade Python requirements.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-02-10 10:15:50 -08:00
Anders Kaseorg 9896782fd1 dependencies: Remove XDate.
It’s even more unmaintained than Moment and doesn’t add any
functionality we don’t already have.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-02-07 06:41:33 -05:00
aryanshridhar f92f99d92d dependencies: Replace moment.js with date-fns.
Replaced methods/functions of moment.js with date-fns library.
The motive was to replace it with a smaller frontend timezone library.

Date-fns ~ 11.51 kb
moment.js ~ 217.87 kb

Some of the format strings change because date-fns encodes them
differently from how moment did.

Fixes #16373.
2021-02-05 11:04:32 -08:00
Anders Kaseorg aa650a4c88 js: Escape strings interpolated into CSS selectors with CSS.escape.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-02-04 11:00:06 -08:00
Aman Agrawal 58a4469896 version: Correct API feature level.
API feature level should have been increased along with #17010
which increased the community topic editing time.
2021-02-03 10:20:59 -05:00
Anders Kaseorg 049d140557 requirements: Upgrade moto to 1.3.17.dev230.
This unblocks the upgrade to responses 0.12.1.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-01-29 12:02:43 -08:00
Anders Kaseorg d33217f2ef requirements: Remove unused httplib2 requirement.
It’s unused since commit 8dd95bd057
(#13718).

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-01-26 16:33:32 -08:00
Anders Kaseorg a7bd1f8049 requirements: Upgrade Python requirements. 2021-01-26 13:27:50 -08:00
Mateusz Mandera 1432067959 dependencies: Upgrade to Django 3.1.
https://docs.djangoproject.com/en/3.1/releases/3.1/

- django.contrib.postgres.fields.JSONField is deprecated and should be
  replaced with models.JSONField
-  The internals of the implementation in the postgresql backend have
   changed a bit in
   f48f671223
   and thus we need to make an ugly tweak in test_runner.
- app_directories.Loader.get_dirs() now returns a list of PosixPath so
  we need to make a small tweak in TwoFactorLoader for that (PosixPath
  is not iterable)

Fixes #16010.
2021-01-26 10:20:00 -08:00
Anders Kaseorg 2db2f2d6cd dependencies: Upgrade JavaScript dependencies.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-01-25 14:53:19 -08:00
Steve Howell d9740045a5 refactor: Eliminate checks in build_stream_dict_for_sub.
We eliminate some redundant checks.

We also consistently provide a `subscribers` field
in our stream data with `[]`, even if our users
can't access subscribers.  We therefore bump
the API version and tweak the docs.  (See further
down for a detailed justification of the change.)

Even though it is sometimes fine to have redundant code
that is defensive in nature, some upcoming changes are gonna
move subscriber-related logic out of build_stream_dict_for_sub
for certain codepaths as part of our effort to streamline
the payload for subscribers within page_params.

So we can't rely on the code that I removed here
inside of build_stream_dict_for_sub.

Anyway, it makes more sense to do these checks explicitly
in the validate function.

The code in build_stream_dict_for_sub was almost effectively
a noop, since the validation function was already preventing
us from getting subscriber info.  The only difference it
made was sometimes converting `[]` to `None`, and then
subsequently omitting the subscribers field.

Neither ZT nor the webapp make any distinction between
`[]` or <missing key> for the `subscribers` data in
`page_params`.

The webapp has had this code for a long time (and now
equivalent code elsewhere in this PR):

    if (!Object.prototype.hasOwnProperty.call(sub, "subscribers")) {
        sub.subscribers = new LazySet([]);
    }

The webapp calculates access based on booleans, anyway:

    sub.can_access_subscribers =
        page_params.is_admin || sub.subscribed ||
        (!page_params.is_guest && !sub.invite_only);

And ZT would choke if `subscribers` were missing, except that
it never gets to the relevant code due to other checks:

    def get_other_subscribers_in_stream(<snip>):
        assert stream_id is not None or stream_name is not None

        if stream_id:
            assert self.is_user_subscribed_to_stream(stream_id)

            return [sub
                    for sub in self.stream_dict[stream_id]['subscribers']
                    if sub != self.user_id]
        else:
            return [sub
                    for _, stream in self.stream_dict.items()
                    for sub in stream['subscribers']
                    if stream['name'] == stream_name
                    if sub != self.user_id]

You could make a semantic argument that we should prefer
<missing key> to `[]` when subscribers aren't even available, but
we have precedent from the way that `bulk_get_subscriber_user_ids`
has traditionally populated its result:

    result: Dict[int, List[int]] =
        {stream["id"]: [] for stream in stream_dicts}

If we changed `stream_dicts` to `target_stream_dicts` we
would faciliate a move toward `None`, but it would just cause
headaches for other server code as well as the frontends
(which, to reiterate, already prefer the empty array
for convenience).
2021-01-21 15:04:07 -08:00
Aman Agrawal c685d36821 hipchat_import: Remove tool from codebase.
Remove functions and scripts used by HipChat import tool and
those which will no longer be required in future.
2020-12-23 08:28:49 -08:00
Anders Kaseorg 6c989550ba dependencies: Upgrade JavaScript dependencies.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-12-22 16:14:28 -08:00
Mateusz Mandera 160cc5120a api: Require can_create_users permission to create users via API.
Allowing any admins to create arbitrary users is not ideal because it
can lead to abuse issues.  We should require something stronger that
requires the server operator's approval and thus we add a new
can_create_users permission.
2020-12-21 13:20:21 -08:00
Sutou Kouhei 0d3f9fc855 install: Use PGroonga packages built for PostgreSQL packages by PGDG
Because we always use PostgreSQL packages by PGDG since Zulip 3.0.

Fixes #16058.
2020-12-18 15:38:21 -08:00
ryanreh99 10559c05ee build_pygments_data: Include zulip specific data in langs.
This fixes a bug where the typeahead did not include the
zulip specific langs (such a `quote`, `spoiler` and `math`)
as these weren't passed to the typeahead's source.

Introduced in af64c52166.

Fixes #16862.
2020-12-15 16:41:19 -08:00
Anders Kaseorg 2ab0b3d4fc validator: Reject ISO 8601 dates missing leading zeros.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-12-15 16:36:50 -08:00
Anders Kaseorg b3aa44c914 install-shfmt: Upgrade shfmt from 3.1.2 to 3.2.1.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-12-09 23:07:40 -08:00
Anders Kaseorg 77fdac3579 install-node: Upgrade Node.js to 14.15.1 and nvm to 0.37.2.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-12-09 23:07:40 -08:00
Anders Kaseorg 984a5bb737 dependencies: Upgrade JavaScript dependencies.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-12-09 23:07:38 -08:00
Anders Kaseorg 362d4a6a8d requirements: Upgrade Python requirements.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-12-09 14:24:54 -08:00
Anders Kaseorg af64c52166 build_pygments_data: Build langs without incorrect mutation.
The previous loop generated broken entries for duplicated language
aliases:

"juttle": {"priority": {"priority": 0, "pretty_name": "Juttle"}, "pretty_name": "Juttle"}

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-11-16 18:51:37 -08:00
Anders Kaseorg 13e35bfa94 mypy: Use sqlalchemy-stubs.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-11-16 18:17:41 -08:00
Anders Kaseorg 1275613812 requirements: Upgrade mypy to 0.790.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-11-12 15:44:30 -08:00
Aman Agrawal b2841c7d02 dev_requirements: Upgrade zulint.
This tells users how autofix errors for linters which support it.
This is important since only way to fix prettier errors is
running lint with `--fix` which now the linter will gladly print
with the error.
2020-11-03 14:49:02 -08:00
Vishnu KS 95cd031274 version: Bump provision version for updating email templates. 2020-10-30 11:50:30 -07:00
Anders Kaseorg 7fbd75630f requirements: Upgrade Python requirements.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-10-28 23:39:15 -07:00
Vishnu KS fdea49742c apps: Use GitHub API for generating the web app download link. 2020-10-28 23:04:14 -07:00
Anders Kaseorg 23a289ecd5 install-node: Upgrade Node.js to 12.19.0 and Yarn to 1.22.10.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-10-28 11:45:02 -07:00
Steve Howell 0e7e4f1ad3 api feature level: Bump level for peer_add/peer_remove. 2020-10-26 07:18:08 -04:00
Anders Kaseorg 0f11e2a361 dependencies: Upgrade JavaScript dependencies.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-10-24 02:09:14 -07:00
Anders Kaseorg efa8dd3a47 compilemessages: Sort language list with Unicode Collation Algorithm.
Right now the list of languages in Display settings → Default language
is sorted in an unintuitive order due to the varying case conventions:

British English
Chinese (Taiwan)
Deutsch
English
Hindi
Indonesian (Indonesia)
Lietuviškai
Magyar
Malayalam
Nederlands
Português
Română
Tiếng Việt
Türkçe
català
español
français
galego
italiano
polski
suomi
svenska
česky
Русский
Українська
български
српски
فارسی
தமிழ்
日本語
简体中文
繁體中文
한국어

Fix the sort to use the locale-independent Unicode Collation
Algorithm:

British English
català
česky
Chinese (Taiwan)
Deutsch
English
español
français
galego
Hindi
Indonesian (Indonesia)
italiano
Lietuviškai
Magyar
Malayalam
Nederlands
polski
Português
Română
suomi
svenska
Tiếng Việt
Türkçe
български
Русский
српски
Українська
فارسی
தமிழ்
한국어
日本語
简体中文
繁體中文

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-10-21 15:22:18 -07:00
Anders Kaseorg 5c6202ca97 notifications: Use a dynamic SVG favicon for unread counts.
Closes #2304.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-10-21 07:43:00 -07:00
Anders Kaseorg 8152cd8566 requirements: Upgrade Python requirements.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-10-19 11:54:14 -07:00
Anders Kaseorg d81a93cdf3 requirements: Upgrade markdown to 3.3.1.
Upstream has slightly changed the whitespace around stashes.  Take
this opportunity to clean up the extra blank lines we were outputting.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-10-19 11:54:14 -07:00
Anders Kaseorg 449b16be59 dependencies: Use patched source-map with a performance fix.
https://github.com/mozilla/source-map/issues/370

This makes Webpack about 9% faster.  Before:

$ multitime -n9 -s0 node_modules/.bin/webpack --config-name=frontend --display=errors-only -p
===> multitime results
1: node_modules/.bin/webpack --config-name=frontend --display=errors-only -p
            Mean        Std.Dev.    Min         Median      Max
real        18.243      0.107       18.090      18.236      18.443
user        27.913      0.188       27.714      27.843      28.251
sys         2.028       0.043       1.933       2.039       2.074

After:

$ multitime -n9 -s0 node_modules/.bin/webpack --config-name=frontend --display=errors-only -p
===> multitime results
1: node_modules/.bin/webpack --config-name=frontend --display=errors-only -p
            Mean        Std.Dev.    Min         Median      Max
real        16.686      0.085       16.542      16.684      16.885
user        25.965      0.167       25.559      26.022      26.163
sys         1.965       0.064       1.807       1.998       2.010

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-10-15 14:38:41 -07:00
Anders Kaseorg 2a4a775e2b requirements: Install PycURL for Thumbor.
Fixes “thumbor:WARNING pycurl usage is advised. It could not be loaded
properly. Verify install...”.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-10-14 10:52:32 -07:00
Anders Kaseorg 4fe06a141e eslint: Enable eslint-plugin-unicorn.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-10-07 16:00:33 -07:00
sahil839 78b98d8067 realm: Add wildcard_mention_policy setting.
We add a new wildcard_mention_policy setting to handle wildcard
mentions in large streams, with a wide range of policies available to
organizations.

We set the default to the safe option for preventing accidental spam:
only stream administrators being able to use wildcard mentions in
large streams.
2020-10-01 12:18:03 -07:00
Anders Kaseorg cfd93096b5 openapi: Remove yamole.
As explained in the previous commit, yamole preprocessed allOf with an
algorithm that is not standards compliant.  We replicate that
algorithm, but importantly, we only use it for our own code and not
for building the openapi_core RequestValidator.

This improves the time taken by OpenAPISpec().check_reload() from
1.69s to 0.53s, nearly all of which is inside
openapi_core.create_spec.

Closes #10484.  Significantly improves #16068.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-09-29 16:47:10 -07:00
Anders Kaseorg e4931c5bc4 people: Remove unorm polyfill.
We no longer support IE11.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-09-25 16:54:25 -07:00
Tim Abbott 089225ea36 version: Update latest desktop release version. 2020-09-25 13:00:46 -07:00
Anders Kaseorg 58808c2362 dependencies: Upgrade JavaScript dependencies.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-09-24 16:33:45 -07:00
Anders Kaseorg 3a39c6f67f dependencies: Downgrade simplebar to 5.2.1.
SimpleBar 6.0.0-beta.2 through -beta.6 are built with ES6 syntax (I
assume inadvertently: https://github.com/Grsmto/simplebar/issues/523),
and its latest tag has moved back to 5.2.1 anyway.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-09-24 16:13:47 -07:00
Alex Vandiver 4361ce1246 markdown: Use tlds package to keep updated list of TLDs.
Also remove a useage of "blacklist."
2020-09-21 21:03:29 -07:00
Sumanth V Rao 2b9f7916c5 api: Document markdown changes for code blocks and bump API_FEATURE_LEVEL. 2020-09-18 17:15:47 -07:00
Sumanth V Rao 4a2791e2a3 tools/build_pygments_data: Map pygments language aliases to lexer name.
We need this information in the frontend to:
    * Display the 'view in playground' option for locally echoed messages.
    * When we add a UI settings for realm admins to configure their
      playground choices, we'll need to use these canonicalized aliases
      for displaying the option.

Hence, this tweaks the tool which generates pygments_data.json to contain
the data we need.

Bumping major PROVISION_VERSION since folks need to provision in both
directions.

Tests amended.
2020-09-18 17:12:26 -07:00
Anders Kaseorg b7874ac82e install-node: Upgrade Node.js to 12.18.4 and Yarn to 1.22.5.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-09-15 16:33:28 -07:00
Anders Kaseorg 7c755f1394 dependencies: Upgrade JavaScript dependencies.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-09-15 16:33:28 -07:00
Anders Kaseorg e84c7fb09f requirements: Remove django-cookies-samesite.
Its functionality was added to Django upstream in 2.1.  Also remove
the SESSION_COOKIE_SAMESITE = 'Lax' setting since it’s the default.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-09-14 17:23:56 -07:00
Anders Kaseorg fd59db292f requirements: Upgrade Python requirements.
Fixes #16335.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-09-13 20:59:09 -07:00
Anders Kaseorg dfab09b17d markdown: Replace hyperlink requirement with urllib.parse.
The previous code only worked by accident and hyperlink 20.0.0 breaks
it.

>>> hyperlink.parse("example.com").replace(scheme="https")
DecodedURL(url=URL.from_text('https:example.com'))

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-09-13 15:37:28 -07:00
Anders Kaseorg d9860d40a6 dependencies: Upgrade JavaScript dependencies.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-09-02 19:34:37 -07:00
Priyank Patel 5c53c44ee5 dependencies: Upgrade babel-plugin-rewire-ts to v1.4.0.
This version keeps the internal and external state of the ES6 module
in sync so its state is similar to a CommonJS module.  Since we don't
need to ask rewire for the internal state of the module, the tests
will be same as they were before the TS/ES6/remove from window
migrations.
2020-09-01 19:55:58 -07:00
Anders Kaseorg b9184eee65 eslint: Use eslint-import-resolver-webpack.
This uses our Webpack configuration to more accurately predict how our
imports will actually behave; for example, it automatically resolves
the .ts extension.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-09-01 16:43:02 -07:00
Priyank Patel 05f3d552d3 yarn.lock: Update lockfile after phantomjs-prebuild removal.
I think this needs an update because 2f5f5d7749,
which removes phantomjs-prebuilt didn't update the yarn.lock.
2020-08-31 11:21:15 -07:00
Priyank Patel c1ba648b4f dependencies: Add xvfb dependency.
The node package allow use to control xvfb apt package in puppeteer
tests. This help us create a fake display so we can run puppeteer in
headful (headless: false) mode, which is required to use the chrome
extension desktop capture API.
2020-08-30 23:38:52 -07:00
Tim Abbott 2f5f5d7749 dependencies: Remove PhantomJS from development environment. 2020-08-30 17:16:02 -07:00
Dinesh 2d22d88700 casper: Remove few traces of casper.
Now that all casper tests have been migrated to
puppeteer, there's no need for having casper
related things.

Removed the casperjs package and removed/replaced
casper in few places with puppeteer.

Only removed few of them which I'm confident
about. Also didn't make any changes in docs
as it would be easier to remove them while
adding puppeteer docs.
2020-08-30 17:16:02 -07:00
Aman 46a86e218e requirements: Upgrade zulint.
This upgrade allows us to exclude a line multiple times for a
pattern.
2020-08-24 16:10:10 -07:00
orientor 372e010dbb events: Add `op` field to `update_message_flags` events.
`update_message_flags` events used `operation` instead of `op`, the
latter being the standard field used in other events. So add `op`
field to `update_message_flags` and mark `operation` as deprecated,
so that it can be removed later.
2020-08-24 12:42:03 -07:00
sahil839 f046c9c58a streams: Add role field to Subscription objects passed to clients.
This commit adds "role" field to the Subscription objects passed to
clients.  This is important preparation for being able to work on the
frontend for this feature.
2020-08-14 16:33:11 -07:00
Anders Kaseorg 4093514b93 dependencies: Upgrade babel-eslint to @babel/eslint-parser.
https://babeljs.io/blog/2020/07/13/the-state-of-babel-eslint

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-08-12 18:37:25 -07:00
Anders Kaseorg bc1dfb4b10 dependencies: Upgrade JavaScript dependencies.
css-loader@^4 is also available, but we can’t use it yet because of
https://github.com/webpack-contrib/css-loader/issues/1164.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-08-12 18:37:25 -07:00
Anders Kaseorg 0f608176ad install-node: Upgrade Node.js from 12.18.2 to 12.18.3.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-08-12 18:37:25 -07:00
Kartik Srivastava 63173d5554 api: Return 'user_id' in 'POST /users' response.
This adds 'user_id' to the simple success response for 'POST /users'
api endpoint, to make it convenient for API clients to get details
about users they just created.  Appropriate changes have been made in
the docs and test_users.py.

Fixes #16072.
2020-08-11 16:40:12 -07:00
Anders Kaseorg 61d0417e75 python: Replace ujson with orjson.
Fixes #6507.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-08-11 10:55:12 -07:00
Anders Kaseorg dbdf67301b memcached: Switch from pylibmc to python-binary-memcached.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-08-06 12:51:14 -07:00
Anders Kaseorg ca2dc639c4 styles: Enable most stylelint-config-standard rules.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-08-04 16:34:19 -07:00
Anders Kaseorg 7c7f030d54 styles: Enable most stylelint-config-recommended rules.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-08-04 16:34:19 -07:00
Anders Kaseorg c6ed5c81de styles: Format CSS with Prettier.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-08-04 16:34:19 -07:00
Manav Mehta 69e0625a09 version: Update version for Zulip Desktop v5.4.0 release.
Update the zulip-desktop download link for the latest v5.4.0 release
This commit only updates zulipchat.com/apps/ links.
2020-07-31 10:36:12 -07:00
Anders Kaseorg b06794e9f7 requirements: Downgrade pip to 20.1.1 due to pip-tools incompatibility.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-07-30 19:16:04 -07:00
Anders Kaseorg d35b5027bb semgrep: Upgrade semgrep to 0.17.0.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-07-30 15:12:22 -07:00
Anders Kaseorg d22ec342e2 requirements: Upgrade Python requirements.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-07-30 15:12:22 -07:00
Gittenburg 6081e0228b emoji: Make 😛 have open eyes.
😛 should be the most general version, which is the one
with open eyes. Other apps do the same and it also means that :P, which
is converted to 😛 is rendered like the emoticon.

Fixes #15970.
2020-07-29 11:18:43 -07:00
Dinesh c15d7e3202 requirements: Update social-auth-core to latest version.
Uses git release as this version 3.4.0 is not released to pypi.
This is required for removing some overriden functions of
apple auth backend class AppleAuthBackend.

With the update we also make following changes:

* Fix full name being populated as "None None".
c5c74f27dd that's included in update assigns first_name and last_name
to None when no name is provided by apple. Due to this our
code is filling return_data['full_name'] to 'None None'.
This commit fixes it by making first and last name strings empty.

* Remove decode_id_token override.
Python social auth merged the PR we sent including the changes
we made to decode_id_token function. So, now there is no
necessity for the override.

* Add _AUDIENCE setting in computed_settings.py.
`decode_id_token` is dependent on this setting.
2020-07-28 17:12:49 -07:00
Anders Kaseorg d2520cd7e0 js: Replace underscore with lodash and remove it from globals.
Tweaked by tabbott to bump PROVISION_VERSION.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-07-26 16:12:06 -07:00
Alex Vandiver 473be34d90 sentry: Add sentry_sdk dependency. 2020-07-24 12:52:15 -07:00
Anders Kaseorg 0771e7fc59 eslint: Add import plugin.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-07-24 09:42:56 -07:00
Anders Kaseorg c9ce77c28b eslint: Use babel-eslint parser.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-07-24 09:42:56 -07:00
Anders Kaseorg 79d20c7c21 version: Bump PROVISION_VERSION for :slight_smile: rename.
Commit 45e19dd6b9 (#15627) required an
emoji rebuild.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-07-21 18:25:18 -07:00
Anders Kaseorg b3da022bdf install-node: Upgrade Node.js to 12.18.2.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-07-20 10:56:31 -07:00
Anders Kaseorg aa16208fd8 dependencies: Upgrade JavaScript dependencies.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-07-20 10:56:31 -07:00
Anders Kaseorg bd45216c21 eslint: Disable syntax rules handled by Prettier.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-07-17 14:31:25 -07:00
Steve Howell c60f4236a9 api: Do not require short_name to create user.
When you post to /json/users, we no longer
require or look at the short_name parameter,
since we don't use it in any meaningful way.

An upcoming commit will eliminate it from the
database.
2020-07-17 11:15:15 -07:00
Steve Howell b375581f58 api: Remove (sender_)short_name from message APIs. 2020-07-17 11:15:14 -07:00
Tim Abbott 23e0ea5e32 version: Update version following 3.0 release. 2020-07-16 02:25:45 -07:00
Tim Abbott d74664a603 Release Zulip Server 3.0. 2020-07-16 02:13:43 -07:00
Anders Kaseorg 56e8cca025 lint: Add Prettier as a linter for YAML files.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-07-14 16:25:31 -07:00
Anders Kaseorg f364414cb9 install-semgrep: Upgrade semgrep to 0.14.0.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-07-14 00:42:07 -07:00
Anders Kaseorg 79b1bf56ab requirements: Upgrade zulint.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-07-14 00:41:20 -07:00
Tim Abbott 6b1f1e74a3 version: Update version string following 3.0-rc2. 2020-07-14 00:29:59 -07:00