Commit Graph

13383 Commits

Author SHA1 Message Date
Alex Vandiver 8df82f50e4 outgoing_http: Provide a convenient way to set default headers. 2021-05-07 08:39:36 -07:00
Alex Vandiver 6339e7fd47 outgoing_http: Put the X-Smokescreen-Role in the proxy headers. 2021-05-07 08:39:36 -07:00
Alex Vandiver b88d7a741e outgoing_http: Factor out outgoing HTTP session with timeout. 2021-05-07 08:39:36 -07:00
Alex Vandiver 2dc6df33ae tests: Switch outgoing_webhook_system to use responses. 2021-05-07 08:39:36 -07:00
Alex Vandiver af26849554 outgoing_webhook: Set a default timeout of 10s.
Support for the timeouts, and tests for them, was added in
53a8b2ac87 -- though no code could have set them after 31597cf33e.

Add a 10-second default timeout.  Observationally, p99 is just about
5s, with everything else being previously being destined to meet the
30s worker timeout; 10s provides a sizable buffer between them.

Fixes #17742.
2021-05-07 08:39:36 -07:00
Tim Abbott 18595fac58 models: Fix incorrect on_delete=CASCADE values for deleted streams.
In theory, we never delete streams, but if we did delete the
notifications stream for a realm, we certainly shouldn't delete the
realm.
2021-05-06 20:41:49 -07:00
Anders Kaseorg 405bc8dabf requirements: Remove Thumbor.
Thumbor and tc-aws have been dragging their feet on Python 3 support
for years, and even the alphas and unofficial forks we’ve been running
don’t seem to be maintained anymore.  Depending on these projects is
no longer viable for us.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-05-06 20:07:32 -07:00
Anders Kaseorg 779353b44e apps: Link to macOS Apple silicon native build.
Leave the Intel build as the prominent default, since it will run on
both platforms.  (I would have liked to detect the appropriate
platform, but Apple seems to have put significant effort into making
that impossible for anti-fingerprinting reasons, which is probably an
overall good.)

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-05-06 17:52:00 -07:00
Adam Birds a3ba8b9efd integrations: Add Freshping webhook integration.
I have added a webhook integration for Freshping.
2021-05-06 11:18:35 -07:00
Adam Birds c72ef7be12 integrations: Move `get_setup_webhook_message` to `common.py`.
Move `get_setup_webhook_message` to
`zerver/lib/webhooks/common.py` so multiple integrations can use this
rather than just those which import `zerver/lib/webhooks/git.py`. Also
added the documentation for this.
2021-05-06 11:18:35 -07:00
Tim Abbott c598a84dd6 ldap: Check for deactivating owner role precisely.
Since the invariant we're trying to protect is that every realm has an
active owner, we should check precisely that.

The root bug here, which the parent commit failed to fix properly, is
that we were doing a "greater than" check when we clearly originally
meant a "less than" check -- lower role numbers have more permissions.
2021-05-06 10:13:29 -07:00
Tim Abbott 5db8e43d5c ldap: Fix incorrect check for deactivating all owners.
We apparently failed to update this when we did the migration from the
administrator role to the owner role as the top role in Zulip.
2021-05-06 09:45:52 -07:00
pletinckxc cc4cc9d14f test_send_email: Shorten tests about build_email.
Commit b7fa41601d introduced some tests
to verify the implemented changes. These tests were longer than necessary.
See https://github.com/zulip/zulip/pull/17799 for the whole discussion.
2021-05-05 20:16:11 -07:00
Cyril Pletinckx e4ff372fc3 emails: Transform SMTPException into EmailNotDeliveredException.
Django's default SMTP implementation can raise various exceptions
when trying to send an email. In order to allow Zulip calling code
to catch fewer exceptions to handle any cause of "email not
sent", we translate most of them into EmailNotDeliveredException.
The non-translated exceptions concern the connection with the
SMTP server. They were not merged with the rest to keep some
details about the nature of these.

Tests are implemented in the test_send_email.py module.
2021-05-05 20:16:11 -07:00
Alya Abbott 279f7b1c8d user docs: Create separate "Code blocks" user documentation page.
* Move the extended documentation of code blocks to a separate page.
* Merge "code playgrounds" documentation to be a section of that page.
* Document copy widget on code blocks.
* This commit changes how we refer to "```python" type syntax for code
  blocks. Instead of being called a syntax highlighting label, this is
  now referred to as a "language tag", since it serves both syntax
  highlighting and playgrounds.
* Remap all the links.
* Advertise this new page in various places that previously did not have a link.
2021-05-05 16:11:21 -07:00
Mateusz Mandera cc96f02947 presence: Disable live presence updates in larger realms.
As discussed in the comment, this is a critical scalability
optimization for organizations with thousands of users.

With substantial comment updates by tabbott.
2021-05-05 08:50:42 -07:00
Sumanth V Rao a40fe26586 help: Document playgrounds on /help, /features and /for/open-source.
Linked the Help Center document in places like
    - zulip.yaml (/events, /register/, realm/playgrounds,
                  /realm/playgrounds/{playground_id})
    - /help/format-your-message-using-markdown (Linked to make
      users reading the markdown code block style, aware of this
      feature)
    - /templates/settings/playground_settings_admin.hbs (Linked
      as a reference to read more about playgrounds before
      configuring one)

Also showcase the feature on /features and /for/open-source.
2021-05-04 12:19:01 -07:00
Dinesh e481f0c869 set-typing-status_api: Document `to` as required argument.
This was accidentally removed in 27e4f5da92
as the initial idea was to have another parameter `stream_id`
for stream type messages instead of sending it in this `to` parameter.

Also fixed a description of `to` for `stream` type.
2021-05-04 11:35:36 -07:00
shanukun 5cee9dee12 test_email_change: User create_confirmation_link for url.
This is a prep refactor, instead of creating Confirmation
object and using `confirmation_url` for generating confirmation
link/url, using `create_confirmation_link` would be a cleaner
approach, also this can help us avoid failing test in case
Confirmation model is changed.

Part of #16359.
2021-05-04 10:48:51 -07:00
Alex Vandiver a9688ceb75 worker: Allow long MissedMessageWorker consumes.
This will stop dropping events in the case that the background
`maybe_send_batched_email` thread takes longer than 30s.  However, see
also #15280 and the TODO comment about how we lose events upon
restart; this worker is still lossy.
2021-05-04 08:45:48 -07:00
sahil839 6e672ebbee settings: Add moderators option for wildcard_mention_policy. 2021-05-03 12:12:01 -07:00
sahil839 141232749b test_message_send: Modify test_wildcard_mention_restrictions.
This commit modifies the test_wildcard_mention_restrictions test
for checking that moderators are not allowed to send messages
with wildcard mention if wildcard_mention_policy is set to
WILDCARD_MENTION_POLICY_ADMINS. Previously, we were checking
for members, but it is better to check for moderators.
2021-05-03 12:12:01 -07:00
Tim Abbott 1489c24860 test_decorators: Don't try to mutate is_staff.
Apparently, after upgrading to Django 3.2, mutating is_staff and then
saving can result in a user's session being destroyed.

In any case, this test is probably better written using two different
users with the different roles, which we have in our initial database
anyway.
2021-05-03 09:26:00 -07:00
Mateusz Mandera 6a8586e989 upload: Mention new difference between sanitize_name and slugify.
In Django 3.2 slugify strips trailing dashes and underscores:
0382ecfe02

sanitize_name doesn't so this difference should be documented like the
others.
2021-05-03 08:36:22 -07:00
Mateusz Mandera 389c7bdb5a upload: Fix docstring and regex in sanitize_name regarding underscore.
Underscore character is already covered by \w, so _ in the regex is
redundant. Also the docstring is mildly incorrect - underscore already
is an allowed character by django's slugify (and always was) for the
aforementioned reason.
2021-05-03 08:36:22 -07:00
Mateusz Mandera 1d9fb4f988 django: Upgrade Zulip to Django 3.2 LTS.
This is a straightforward upgrade in terms of changes needed.

Necessary changes were:
- Set `DEFAULT_AUTO_FIELD`
  https://docs.djangoproject.com/en/3.2/releases/3.2/#customizing-type-of-auto-created-primary-keys

- `The default_app_config application configuration variable is deprecated, due
  to the now automatic AppConfig discovery.`
  https://docs.djangoproject.com/en/3.2/releases/3.2/#automatic-appconfig-discovery

To handle this one, we can remove default_app_config from
zerver/__init__.py because it satisfies what release notes describe in
https://docs.djangoproject.com/en/3.2/releases/3.2/#automatic-appconfig-discovery:
"Most pluggable applications define an AppConfig subclass in an apps.py
submodule. Many define a default_app_config variable pointing to this
class in their __init__.py.  When the apps.py submodule exists and
defines a single AppConfig subclass, Django now uses that configuration
automatically, so you can remove default_app_config."

An important note is that rebuild-test-database needs to be run after
this upgrade in dev environment - if tests are run with test db that was
built on the previous version, they will fail due to a mysterious bug
(?), where changing attributes of a user and .save()ing after logging in
in the test via self.login_user, causes getting logged out - the next
requests via self.client_get etc. are unauthed for some reason,
unless self.login_user is called again. This behavior is no longer
exhibited upon rebuilding the test db - and I can't reproduce it in
production or dev db. So this can likely be reasonably dismissed as some
quirk of the test client system that won't be relevant in the future and
doesn't impact production.
2021-05-03 08:36:22 -07:00
Tim Abbott 935a7b18cb test_signup: For test overly sensitive to whitespace.
This test started failing after a rebase from a branch 2 weeks ago
where it passed.
2021-05-02 11:35:32 -07:00
Tim Abbott 97b08b77df registration: Improve error handling for source_realm_id.
This restores the previous behavior that we store None rather than
throwing an exception in the presence of invalid input.
2021-05-02 11:12:49 -07:00
Siddharth Asthana 55da3329ef registration: Rename source_realm field to source_realm_id.
Now that we are passing source realm's id instead of string_id in
source realm selector, it makes sense to rename the "source_realm" field
to "source_realm_id".
2021-05-02 11:12:49 -07:00
Siddharth Asthana 4262c04db1 registration: Encode source realm as an integer.
In the source realm selector, when we select a realm from which we want
to import the data, we pass the source realm's string_id. The problem
with this approach is that the string_id can be an empty string. This
commit makes the source_realm pass the realm's id instead of string_id.
Now, the source_realm's value will either be an integer or "" (empty
string) when we don't want to import settings from any realm.
2021-05-02 11:12:49 -07:00
Anders Kaseorg 995389b4c1 markdown: Don’t apply further Markdown processing to KaTeX output.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-05-01 15:43:35 -07:00
Anders Kaseorg 802c8de0e8 fenced_code: Wrap math blocks in the frontend like the backend.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-05-01 15:36:15 -07:00
m-e-l-u-h-a-n 201dd7507f left-sidebar: Show starred message counts by default.
It is changed just to make this feature more discoverable.

Fixes: #18305.
2021-05-01 07:27:08 -07:00
pilgrim2308 a74b52db22 UI: Add Zulip version in gear menu.
Currently only enabled in development, since the exact details don't
seem right..

Co-Author-By: Signior-X <b19188@students.iitmandi.ac.in>
Co-Author-By: Aman Agrawal <amanagr@zulip.com>

Implements UI for #8005.
2021-04-30 17:58:08 -07:00
sahil839 3c66131fae models: Fix comment above INVITE_AS dict.
The comment mentioned the values in this dict to be consistent
wth values in settings_config.invited_as_values.
But settings_config.invited_as_values was replaced by
settings_config.user_role_values in 136c005f3f, so
updated the comment accordingly.
2021-04-30 15:57:09 -07:00
sahil839 e669818544 invites: Allow user to be invited as a moderator.
This commit adds both frontend and backend code to invite a user as
moderator. We allow only existing owners and admins to invite a user
as a moderator.
2021-04-30 15:57:09 -07:00
sahil839 a4c3224328 users: Remove redundant get_role_for_new_user in lib/create_user.py.
The function get_role_for_new_user was added to get role from the
invited_as value, as invited_as values were one of (1,2,3,4)
previously, but it was then changed to be the actual role value,
i.e. one of (100, 200, 400, 600), in 1f8f227444.

So, we can safely remove this function now and use invited_as value
directly and handle realm_creation case by using an if condition.
2021-04-30 15:57:09 -07:00
Alex Vandiver fd1774dcba push_notifications: Give full stack information on an exception.
This error has been seen in production instances, but we need more
context to be able to determine what might be causing it.
2021-04-30 14:03:52 -07:00
Wesley Aptekar-Cassels 6b7a3fb74a markdown: Rewrite all external images to use Camo.
Requesting external images is a privacy risk, so route all external
images through Camo.

Tweaked by tabbott for better test coverage, more comments, and to fix
bugs.
2021-04-30 10:36:16 -07:00
Ganesh Pawar ddf2127035 widgets: Prevent edits to widgets.
As of now, editing a widget doesn't update the rendered content.
It's important to ensure that existing votes or options added later on
don't get deleted when rendered.
This seems more complex than it's worth.

For now, we just prevent edits to widgets.
This commit makes the UI clearer that editing widgets isn't allowed.

See also:
https://github.com/zulip/zulip/issues/14229
https://github.com/zulip/zulip/issues/14799

Fixes #17156
2021-04-30 09:55:25 -07:00
Vishnu KS 7f3fc3423b audit log: Create audit log when a realm is created.
This is mainly useful in recording the user who created the realm,
when possible.
2021-04-30 09:25:11 -07:00
Ganesh Pawar 830f1fa8c5 upload: Refactor and add tests for ensure_avatar_image in upload.py.
`ensure_basic_avatar_image` and `ensure_medium_avatar_image` are
essentially the same thing, except a size parameter.
So, refactor them into a single function.

This doesn't introduce any functional changes.
2021-04-29 21:18:13 -07:00
Tim Abbott 615ad2d5d8 middleware: Simplify logic for parsing user-agent.
This avoids calling parse_user_agent twice when dealing with official
Zulip clients, and also makes the logical flow hopefully easier to read.

We move get_client_name out of decorator.py, since it no longer
belongs there, and give it a nicer name.
2021-04-29 17:47:41 -07:00
Tim Abbott 3cf0156997 context_processors: Remove unnecessary get_client_name call.
This detail is now being managed by our middleware.
2021-04-29 17:47:32 -07:00
orientor fe260fb892 middleware: Show client version in logging if available.
Fixes #14067.
2021-04-29 17:07:37 -07:00
orientor ac203cd9f1 middleware: Add client_version attribute to request. 2021-04-29 17:03:40 -07:00
orientor 6224d83dea middleware: Get client name in LogRequests instead of process_client.
This ensures it is present for all requests; while that was already
essentially true via process_client being called from every standard
decorator, this allows middleware and other code to rely on this
having been set.
2021-04-29 17:03:05 -07:00
Anders Kaseorg 871e73ab8f mypy: Don’t use Iterable for values iterated multiple times.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-04-29 16:06:17 -07:00
Anders Kaseorg 393ffcb318 actions: Replace SizedTextIterable with standard Collection[str].
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-04-29 16:06:17 -07:00
Anders Kaseorg 48a52ac56e events: Convert recursion to a loop in do_events_register.
Fixes #18310.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-04-29 16:03:05 -07:00