Commit Graph

6086 Commits

Author SHA1 Message Date
Tim Abbott 2c01354569 management: Use required kwargs in add_realm_args.
This makes management commands more readable, since one doesn't need
to know details of how the library works to read based code.
2021-05-10 12:30:58 -07:00
akshatdalton 55f4996f16 markdown: Fix silent wildcard mentions bug.
A message containing wildcard mention when quoted (which
is turned into a silent mention) or message with silent
wildcard mention notifies the users by sending desktop,
sound, and missed message email notifications. This
is clearly a bug which is fixed by this commit.

Fixes: #18354.
2021-05-10 12:19:40 -07:00
Anders Kaseorg 544bbd5398 docs: Fix capitalization mistakes.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-05-10 09:57:26 -07:00
Tim Abbott 8b75b6f14f message_edit: Use target_message as local variable name.
This name helps emphasize the fact that there's a single targeted
message, even though multiple messages may be affected by the edit.
2021-05-09 21:10:32 -07:00
Aman Agrawal c70541c905 do_update_message: Reuse extracted message IDs from changed_msgs. 2021-05-09 21:05:27 -07:00
PIG208 7150fe5dc5 backend: Extract check_update_message from update_message_backend. 2021-05-09 20:44:04 -07:00
Adam Birds 589a7799dc integrations: Add Uptime Robot integration.
Note that the documentation cannot fully use our macros, because
Uptime Robot requires an & of the end of the URL, because of how it
passes its payload.

Fixes #13854. Fixes #13939.
2021-05-09 20:34:19 -07:00
Anders Kaseorg d0c6f4f400 python: Strip leading and trailing spaces from docstrings.
This is enforced by Black ≥ 21.4b0.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-05-07 22:42:39 -07:00
Steve Howell 91599df0e2 send performance: Optimize create_user_messages.
We combine the two loops into one, so that we
can check our flags before creating the
UserMessageList object.

And we lift a few calculations out of the loop.

For 8k users, with 95% long-term-idle, this was
about a 10x speedup for me.  (~30ms -> 3ms)
2021-05-07 18:40:30 -04:00
Vishnu KS 772500d1c6 validators: Make to_positive_or_allowed_int an optional argument. 2021-05-07 09:37:41 -07:00
Alex Vandiver 8df82f50e4 outgoing_http: Provide a convenient way to set default headers. 2021-05-07 08:39:36 -07:00
Alex Vandiver 6339e7fd47 outgoing_http: Put the X-Smokescreen-Role in the proxy headers. 2021-05-07 08:39:36 -07:00
Alex Vandiver b88d7a741e outgoing_http: Factor out outgoing HTTP session with timeout. 2021-05-07 08:39:36 -07:00
Alex Vandiver 2dc6df33ae tests: Switch outgoing_webhook_system to use responses. 2021-05-07 08:39:36 -07:00
Alex Vandiver af26849554 outgoing_webhook: Set a default timeout of 10s.
Support for the timeouts, and tests for them, was added in
53a8b2ac87 -- though no code could have set them after 31597cf33e.

Add a 10-second default timeout.  Observationally, p99 is just about
5s, with everything else being previously being destined to meet the
30s worker timeout; 10s provides a sizable buffer between them.

Fixes #17742.
2021-05-07 08:39:36 -07:00
Anders Kaseorg 405bc8dabf requirements: Remove Thumbor.
Thumbor and tc-aws have been dragging their feet on Python 3 support
for years, and even the alphas and unofficial forks we’ve been running
don’t seem to be maintained anymore.  Depending on these projects is
no longer viable for us.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-05-06 20:07:32 -07:00
Anders Kaseorg 779353b44e apps: Link to macOS Apple silicon native build.
Leave the Intel build as the prominent default, since it will run on
both platforms.  (I would have liked to detect the appropriate
platform, but Apple seems to have put significant effort into making
that impossible for anti-fingerprinting reasons, which is probably an
overall good.)

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-05-06 17:52:00 -07:00
Adam Birds a3ba8b9efd integrations: Add Freshping webhook integration.
I have added a webhook integration for Freshping.
2021-05-06 11:18:35 -07:00
Adam Birds c72ef7be12 integrations: Move `get_setup_webhook_message` to `common.py`.
Move `get_setup_webhook_message` to
`zerver/lib/webhooks/common.py` so multiple integrations can use this
rather than just those which import `zerver/lib/webhooks/git.py`. Also
added the documentation for this.
2021-05-06 11:18:35 -07:00
Cyril Pletinckx e4ff372fc3 emails: Transform SMTPException into EmailNotDeliveredException.
Django's default SMTP implementation can raise various exceptions
when trying to send an email. In order to allow Zulip calling code
to catch fewer exceptions to handle any cause of "email not
sent", we translate most of them into EmailNotDeliveredException.
The non-translated exceptions concern the connection with the
SMTP server. They were not merged with the rest to keep some
details about the nature of these.

Tests are implemented in the test_send_email.py module.
2021-05-05 20:16:11 -07:00
Alya Abbott 279f7b1c8d user docs: Create separate "Code blocks" user documentation page.
* Move the extended documentation of code blocks to a separate page.
* Merge "code playgrounds" documentation to be a section of that page.
* Document copy widget on code blocks.
* This commit changes how we refer to "```python" type syntax for code
  blocks. Instead of being called a syntax highlighting label, this is
  now referred to as a "language tag", since it serves both syntax
  highlighting and playgrounds.
* Remap all the links.
* Advertise this new page in various places that previously did not have a link.
2021-05-05 16:11:21 -07:00
Mateusz Mandera cc96f02947 presence: Disable live presence updates in larger realms.
As discussed in the comment, this is a critical scalability
optimization for organizations with thousands of users.

With substantial comment updates by tabbott.
2021-05-05 08:50:42 -07:00
Sumanth V Rao a40fe26586 help: Document playgrounds on /help, /features and /for/open-source.
Linked the Help Center document in places like
    - zulip.yaml (/events, /register/, realm/playgrounds,
                  /realm/playgrounds/{playground_id})
    - /help/format-your-message-using-markdown (Linked to make
      users reading the markdown code block style, aware of this
      feature)
    - /templates/settings/playground_settings_admin.hbs (Linked
      as a reference to read more about playgrounds before
      configuring one)

Also showcase the feature on /features and /for/open-source.
2021-05-04 12:19:01 -07:00
sahil839 6e672ebbee settings: Add moderators option for wildcard_mention_policy. 2021-05-03 12:12:01 -07:00
Mateusz Mandera 6a8586e989 upload: Mention new difference between sanitize_name and slugify.
In Django 3.2 slugify strips trailing dashes and underscores:
0382ecfe02

sanitize_name doesn't so this difference should be documented like the
others.
2021-05-03 08:36:22 -07:00
Mateusz Mandera 389c7bdb5a upload: Fix docstring and regex in sanitize_name regarding underscore.
Underscore character is already covered by \w, so _ in the regex is
redundant. Also the docstring is mildly incorrect - underscore already
is an allowed character by django's slugify (and always was) for the
aforementioned reason.
2021-05-03 08:36:22 -07:00
Siddharth Asthana 55da3329ef registration: Rename source_realm field to source_realm_id.
Now that we are passing source realm's id instead of string_id in
source realm selector, it makes sense to rename the "source_realm" field
to "source_realm_id".
2021-05-02 11:12:49 -07:00
Siddharth Asthana 4262c04db1 registration: Encode source realm as an integer.
In the source realm selector, when we select a realm from which we want
to import the data, we pass the source realm's string_id. The problem
with this approach is that the string_id can be an empty string. This
commit makes the source_realm pass the realm's id instead of string_id.
Now, the source_realm's value will either be an integer or "" (empty
string) when we don't want to import settings from any realm.
2021-05-02 11:12:49 -07:00
Anders Kaseorg 995389b4c1 markdown: Don’t apply further Markdown processing to KaTeX output.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-05-01 15:43:35 -07:00
sahil839 a4c3224328 users: Remove redundant get_role_for_new_user in lib/create_user.py.
The function get_role_for_new_user was added to get role from the
invited_as value, as invited_as values were one of (1,2,3,4)
previously, but it was then changed to be the actual role value,
i.e. one of (100, 200, 400, 600), in 1f8f227444.

So, we can safely remove this function now and use invited_as value
directly and handle realm_creation case by using an if condition.
2021-04-30 15:57:09 -07:00
Alex Vandiver fd1774dcba push_notifications: Give full stack information on an exception.
This error has been seen in production instances, but we need more
context to be able to determine what might be causing it.
2021-04-30 14:03:52 -07:00
Wesley Aptekar-Cassels 6b7a3fb74a markdown: Rewrite all external images to use Camo.
Requesting external images is a privacy risk, so route all external
images through Camo.

Tweaked by tabbott for better test coverage, more comments, and to fix
bugs.
2021-04-30 10:36:16 -07:00
Ganesh Pawar ddf2127035 widgets: Prevent edits to widgets.
As of now, editing a widget doesn't update the rendered content.
It's important to ensure that existing votes or options added later on
don't get deleted when rendered.
This seems more complex than it's worth.

For now, we just prevent edits to widgets.
This commit makes the UI clearer that editing widgets isn't allowed.

See also:
https://github.com/zulip/zulip/issues/14229
https://github.com/zulip/zulip/issues/14799

Fixes #17156
2021-04-30 09:55:25 -07:00
Vishnu KS 7f3fc3423b audit log: Create audit log when a realm is created.
This is mainly useful in recording the user who created the realm,
when possible.
2021-04-30 09:25:11 -07:00
Ganesh Pawar 830f1fa8c5 upload: Refactor and add tests for ensure_avatar_image in upload.py.
`ensure_basic_avatar_image` and `ensure_medium_avatar_image` are
essentially the same thing, except a size parameter.
So, refactor them into a single function.

This doesn't introduce any functional changes.
2021-04-29 21:18:13 -07:00
orientor 6224d83dea middleware: Get client name in LogRequests instead of process_client.
This ensures it is present for all requests; while that was already
essentially true via process_client being called from every standard
decorator, this allows middleware and other code to rely on this
having been set.
2021-04-29 17:03:05 -07:00
Anders Kaseorg 871e73ab8f mypy: Don’t use Iterable for values iterated multiple times.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-04-29 16:06:17 -07:00
Anders Kaseorg 393ffcb318 actions: Replace SizedTextIterable with standard Collection[str].
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-04-29 16:06:17 -07:00
Anders Kaseorg 48a52ac56e events: Convert recursion to a loop in do_events_register.
Fixes #18310.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-04-29 16:03:05 -07:00
sahil839 bd78b1ff90 events: Add 'is_moderator' field to the page_params object sent to clients. 2021-04-29 15:18:49 -07:00
sahil839 34f134d58d users: Add role field to user objects returned by format_user_row.
This commit modifies the user objects returned by 'GET /users',
'GET /users/me', 'GET /users/{user_id}' and 'GET /users/{email}'
endpoints to include role field.

We also include role field in the page_params['realm_users'] dict
and in the person object sent in (type="realm_user", op="add")
event.
2021-04-29 15:13:50 -07:00
Alex Vandiver 8711ab7676 outgoing_webhook: Add a logging statement for each outgoing webhook.
This will help determine potentail timeout lengths, as well as serve
as a generally-useful log for locations which do not have Smokescreen
enabled.

In service of #17742.
2021-04-29 12:37:14 -07:00
Adam Birds db16dd9d26
integrations: Change display name for Json integration to use a capital F.
Change display name for Json integration to use a cpital F instead of
lowercase which is the standard across the rest of our integrations.
2021-04-29 12:35:12 -07:00
m-e-l-u-h-a-n 65c400e06d api: Add zulip_version and zulip_feature_level in restart event.
This help mobile and terminal clients understand whether a server
restart changed API feature levels or not, which in turn determines
whether they will need to resynchronize their data.

Also add tests and documentation for this previously undocumented
event type.

Fixes: #18205.
2021-04-29 12:08:15 -07:00
m-e-l-u-h-a-n d2c18e28a4 api: Handle restart events in apply_events.
Event of type restart could not be handled properly, because of
its special behavior. For handling this event in most natural way
we recursively call `do_events_register` when restart event is
recieved, based on custom error created for this event.

Testing: Second call to get_user_events due to recursive calling
of do_event_register, is expected to not contain the restart event.
So new test added in test_event_system.py are based on above behavior
of get_user_events.

Fixes: #15541.
2021-04-29 11:40:59 -07:00
pletinckxc 184df7e656 emails: Fix outgoing email handling inside the dev environment.
Commit 9afde790c6 introduced a bug
concerning outgoing emails inside the development environment. These
emails are not supposed to use a real connection with a mail
server as the send_messages function is overwritten inside the
EmailLogBackEnd class.

The bug was happening inside the initialize_connection function that
was introduced in the above-mentioned commit. This function is used
to refresh the connection with an SMTP server that would have closed
it. As the socket used to communicate with the server is not
initialized inside the development environment this function was
wrongly trying to send no-op commands.

The fix just checks that the connection argument of the function is
an EmailLogBackEnd object before trying the no-op command.
Additionally as it is sometimes useful to be able to send outgoing
emails inside the development environment the get_forward_address
function is used to check if a real connection exists between Zulip
and the server. If it is the case, as EmailLogBackEnd is a subclass
of smtp.EmailBackend, the connection will be nicely refreshed.

This commit was tested manually by checking that the console prints
correctly that an email is sent to the user when it signs in inside
the development environment. It was also tested when a mail provider
is specified and the mails were correctly received.
2021-04-28 18:00:37 -07:00
Tim Abbott f52005b638 email: Set List-Unsubscribe headers where we have unsubscribe links.
Since we have a convention of using `unsubscribe_link` as the context
variable name, this will apply to all of our emails with confirmation
links.
2021-04-28 15:33:32 -07:00
PIG208 9c0ce19104 reactions: Extract check_add_reaction from add_reaction. 2021-04-28 09:11:08 -07:00
strifel 02c5676a67 ldap: Add advanced LDAP realm access control.
This allows access to be more configurable than just setting one
attribute.  This can be configured by setting the setting
AUTH_LDAP_ADVANCED_REALM_ACCESS_CONTROL.
2021-04-28 09:08:13 -07:00
PIG208 427506b742 embedded bot: Return message id for send_message and send_reply.
This change is made to comply with the corresponding views for
the API. The incrementor implementation in zulip_bots won't work
otherwise if send_message and send_reply return None as it needs
the message id.
2021-04-28 08:32:21 -07:00