Commit Graph

4491 Commits

Author SHA1 Message Date
Rohitt Vashishtha 88367a129c markdown: Disable tex and latex for math rendering.
We now parse tex and latex as regular languages, highlighting them
with pygments. We only allow 'math' to trigger latex rendering,
which is in line with the documentation.
2020-05-21 12:30:27 -07:00
Tim Abbott e59a2a23c3 portico: Fix a few issues with /for/research. 2020-05-20 17:57:22 -07:00
Rohitt Vashishtha 52c25a9301 markdown-timestamp: Use data-timestamp attribute.
This commit shifts our timestamp syntax to be of the form:

    <span class="timestamp data-timestamp="123456"></span>

since value is not a valid attribute of span elements.
2020-05-20 14:28:08 -07:00
Rohitt Vashishtha b062e8332f markdown: Add timestamp syntax to markdown processors.
This adds support for syntax like: !time(Jun 7 2017, 6:30 PM) so that
everyone sees the time in their own local timezone. This can be used
when scheduling online meetings, etc.

This adds some hardcoded values for timezones, because of there
being no sureshot way of determining the timezone easily. However,
since the main way of using the feature should be a typeahead for
entering the time, this shouldn't be cause of much concern.

Fixes #5176.
2020-05-20 14:23:55 -07:00
orientor c7a2c48bb7 openapi: Add tests for checking presence of various attributes.
The swagger validator is a basic tool to check whether our
openapi specification file follows the basic syntax. But to ensure
that our zulip.yaml file is not only syntactically compatible but
also describes our API well, we need to add custom tests. This
commit currently checks whether each endpoint has an `operationId`
and a valid tag. It also makes it easier to check for custom rules
in the future.
2020-05-20 13:10:58 -07:00
Mateusz Mandera 2f5fd272aa auth: Gracefully handle bad http responses from IdP in social auth.
If the IdP authentication API is flaky for some reason, it can return
bad http responses, which will raise HTTPError inside
python-social-auth. We don't want to generate a traceback
in those cases, but simply log the exception and fail gracefully.
2020-05-20 09:30:56 -07:00
Kartik Srivastava 4af7ad70e7 openapi: Add OpenAPI data for /users/{user_id}/reactivate. 2020-05-19 23:10:50 -07:00
Vishnu KS 9129da88af tests: Add test for STANDARD plan_type in test_message_retention_days. 2020-05-19 14:48:59 -07:00
Vishnu KS a699f39050 models: Use UPGRADE_TEXT_STANDARD in ensure_not_on_limited_plan. 2020-05-19 14:48:59 -07:00
sahil839 9b78a73e36 populate_db: Add new admin user as 'Desdemona'.
This commit adds a second admin user named 'Desdemona' to dev and
test database.
2020-05-19 11:42:27 -07:00
Aman Agrawal 2a4c62a326 update_to_dict_cache: Use bulk queries when preparing msgs for cache.
During events such as stream / topic name edit for a topic, we were
running queries to db in loop for each message for reactions,
submessages and realm_id. This commit reduces the queries to be
done only for realm_id, which is yet to be fixed.

This is accomplished by building messages with empty reactions
and submessages and then updating them in the messages using bulk
queries.
2020-05-19 10:30:03 -07:00
Aman Agrawal 21acc65489 test_messages: Test number of queries performed by `to_dict_uncached`.
`to_dict_uncached` performs queries to db in a loop which should be
replaced with bulk db queries.
2020-05-19 10:30:03 -07:00
sahil839 46ef6816b6 stream: Allow non admins to set stream post policy when creating streams.
This commit allows non admins to set stream post policy while creating
streams.

Restriction was there to prevent user from creating a stream in which
the user cannot post himself but this will be taken care of with
stream admin feature.
2020-05-16 14:53:22 -07:00
Mateusz Mandera b234fe8ccb retention: Pass optional realm argument to move_messages_to_archive.
This allows having the realm field of ArchiveTransaction set instead of
NULL when using move_messages_to_archive.
2020-05-16 14:46:56 -07:00
Tim Abbott bd3c0aef9b presence: Optimize performance of mobile_query.
I'm not sure exactly what series of history got us here, but we were
fetching the mobile_user_ids data for all users in the organization,
regardless of whether they were recently active (and thus relevant for
the main presence data set).  And doing so in a sloppy fashion
(sending every user ID over the wire, rather than just having the
database join on Realm).

Fixing this saves a factor of 4-5 on the total runtime of a presence
request on organizations with 10Ks of users like chat.zulip.org; more
like 25% in an organization with 150.  Since large organizations are
very heavily weighted in the overall cost of presence, this is a huge
win.

Fixes part of #13734.
2020-05-14 23:23:37 -07:00
Puneeth Chaganti 734d651b45 home: Fix furthest read time when user has multiple clients.
The query to fetch the latest user activity was missing an
`.order_by('last_visit')`. This meant that the results were being
ordered by the `id`, which resulted in us getting `update_message_flags`
action performed on the client that the user installed last, instead of
being client agnostic and fetching the "global" last
`update_message_flags` action performed by the user.
2020-05-14 13:38:09 -07:00
orientor 9170931da3 openapi: Add test for validating examples.
Zulip's openapi specification in zulip.yaml has various examples
for various schemas. Validate the example with their respective
schemas to ensure that all the examples are schematically correct.

Part of #14100.
2020-05-12 23:03:06 -07:00
Tim Abbott 35139ac559 api: Remove email field from realm_user and realm_bot events.
The `email` field for identifying the user being modified in these
events was not used by either the webapp or other official Zulip
clients.  Instead, it was legacy data from before we switched years
ago to sending user_id fields as the correct way to uniquely identify
a user.
2020-05-12 17:46:13 -07:00
Tim Abbott 496c08e26c slack import: Fix DefaultStream import of deactivated #random.
If the #random channel in Slack is deactivated, we should follow
Zulip's data model of not allowing deactivated, default streams.

This had apparently happened in zulipchat.com for a few organizations,
resulting in weird exceptions trying to invite new users.
2020-05-12 17:18:57 -07:00
Rohan Prashant 44a56a66d0 tests-backend: Bring stream_recipient.py to 100 percent coverage.
Fixes part of #7089.
2020-05-12 16:43:01 -07:00
clarammdantas 7e9024a39c popovers.js: Add version to user avatar request.
When a user changes its avatar image, the user's avatar in popovers
wasn't being correctly updated, because of browser caching of the
avatar image.  We added a version on the request to get the image in
the same format we use elsewhere, so the browser knows when to use the
cached image or to make a new request to the server.

Edited by Tim to preserve/fix sort orders in some tests, and update
zulip_feature_level.

Fixes: #14290
2020-05-12 11:09:01 -07:00
Steve Howell 9137726e38 bots: Remove `owner` (email) from bot-related payloads.
We remove the `owner` field from `page_params/realm_bots`
and bot-related events.

In the recent commit 155f6da8ba
we added `owner_id`, which we now use everywhere we need
bot owners for.

We also bump the `API_FEATURE_LEVEL` to 5 here.  We
had already documented this in the prior commit to
add `owner_id`.

Note that we don't have to worry about mobile/ZT clients
here--we only deal with bot data in the webapp.
2020-05-12 09:48:37 -07:00
Steve Howell 155f6da8ba bots: Add owner_id to bot-related payloads.
For the below payloads we want `owner_id` instead
of `owner`, which we should deprecate.  (The
`owner` field is actually an email, which is
not a stable key.)

    page_params.realm_bots

    realm_bot/add

    realm_bot/update

IMPORTANT NOTE: Some of the data served in
these payloads is cached with the key
`bot_dicts_in_realm_cache_key`.

For page_params, we get the new field
via `get_owned_bot_dicts`.

For realm_bot/add, we modified
`created_bot_event`.

For realm_bot/update, we modified
`do_change_bot_owner`.

On the JS side, we no longer
look up the bot's owner directly in
`server_events_dispatch` when we get
a realm_bot/update event. Instead, we
delegate that job to `bot_data.js`.
I modified the tests accordingly.
2020-05-11 16:16:58 -07:00
Rohitt Vashishtha 26238426cd message-edit: Process user groups for mentioned user ids.
When editing a message where we mention a usergroup, we would remove
the 'mentioned' flag from messages, resulting in the message being
hidden from your mentions in the UI. This was reported by Greg Price in
https://chat.zulip.org/#narrow/stream/9-issues/topic/missing.20mention.

We add the same code that we use in do_send_messages to calculate the
updated mentions_user_ids. We add some tests alongside other user group
mention tests in test_bugdown.
2020-05-11 14:52:11 -07:00
Rohitt Vashishtha 9506be0f4f slack-import: Downgrade Slack legacy-token check failure to warning.
Slack has disabled creation of legacy tokens, which means we have to use other
tokens for importing the data. Thus, we shouldn't throw an error if the token
doesn't match the legacy token format.

Since we do not have any other validation for those tokens yet, we log a warning
but still try to continue with the import assuming that the token has the right
scopes.

See https://api.slack.com/changelog/2020-02-legacy-test-token-creation-to-retire.
2020-05-11 13:41:50 -07:00
Mateusz Mandera dd40649e04 queue_processors: Remove the slow_queries queue.
While this functionality to post slow queries to a Zulip stream was
very useful in the early days of Zulip, when there were only a few
hundred accounts, it's long since been useless since (1) the total
request volume on larger Zulip servers run by Zulip developers, and
(2) other server operators don't want real-time notifications of slow
backend queries.  The right structure for this is just a log file.

We get rid of the queue and replace it with a "zulip.slow_queries"
logger, which will still log to /var/log/zulip/slow_queries.log for
ease of access to this information and propagate to the other logging
handlers.  Reducing the amount of queues is good for lowering zulip's
memory footprint and restart performance, since we run at least one
dedicated queue worker process for each one in most configurations.
2020-05-11 00:45:13 -07:00
Pragati Agrawal 60a762704a org settings: Fix limited plan realm can change message_retention_days.
These changes should be included in bd9b74436c,
as it makes sure that Zulip limited plan realm won't be able to change the
`message_retention_days` setting.
2020-05-09 16:40:50 -07:00
Anders Kaseorg 78c70b1424 bugdown: Leave link titles alone until clean_user_content_links.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-05-09 16:32:40 -07:00
Anders Kaseorg 8cdf2801f7 python: Convert more variable type annotations to Python 3.6 style.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-05-08 16:42:43 -07:00
Tim Abbott 200209995b test_signup: Update email content for last design tweak. 2020-05-08 15:18:21 -07:00
Puneeth Chaganti ded3b00760 home: Use last update_message_flags activity instead of pointer.
The pointer doesn't get updated when a user is only reading messages in
narrowed views. But, we use the pointer position to determine the
furthest read time, which causes the bankruptcy banner to show up even
for users who have been actively reading and sending messages.

This commit switches to using the time of the last update_message_flags
activity by a user to determine the time of last activity.
2020-05-08 15:06:06 -07:00
Puneeth Chaganti a937ea531e test_home: Return message id when sending test message. 2020-05-08 15:01:08 -07:00
Aman Agrawal 0c2b25cab3 home/page_params: Add suffix `_mib` to clarify the size units.
The docs say "megabytes" or "MB", but client software needs to
know the actual meaning, which is in mebibytes.
2020-05-08 14:54:54 -07:00
Pragati Agrawal bd9b74436c org settings: Enable message_retention_days in org settings UI.
Since production testing of `message_retention_days` is finished, we can
enable this feature in the organization settings page. We already had this
setting in frontend but it was bit rotten and not rendered in templates.

Here we replaced our past text-input based setting with a
dropdown-with-text-input setting approach which is more consistent with our
existing UI.

Along with frontend changes, we also incorporated a backend change to
handle making retention period forever. This change introduces a new
convertor `to_positive_or_allowed_int` which only allows positive integers
and an allowed value for settings like `message_retention_days` which can
be a positive integer or has the value `Realm.RETAIN_MESSAGE_FOREVER` when
we change the setting to retain message forever.

This change made `to_not_negative_int_or_none` redundant so removed it as
well.

Fixes: #14854
2020-05-08 14:09:31 -07:00
Pragati Agrawal 12504075ee org settings: Rename plan_includes_wide_organization_logo to more generic.
It's a preliminary step to enable message_retention_setting in org settings
UI, which is a non-limited plan only feature. So we require a page_param
property that tells us the limited-plan state of the Zulip realm.
2020-05-08 14:04:21 -07:00
Tim Abbott a920544bc3
invite emails: Ensure user-controlled input is always in links.
Popular email clients like Gmail will automatically linkify link-like
content present in an HTML email they receive, even if it doesn't have
links in it.  This made it possible to include what in Gmail will be a
user-controlled link in invitation emails that Zulip sends, which a
spammer/phisher could try to take advantage of to send really bad spam
(the limitation of having the rest of the invitation email HTML there
makes it hard to do something compelling here).

We close this opportunity by structuring our emails to always show the
user's name inside an existing link, so that Gmail won't do new
linkification, and add a test to help ensure we don't remove this
structure in a future design change.

Co-authored-by: Anders Kaseorg <andersk@mit.edu>
2020-05-08 14:02:41 -07:00
wowol ffed6b87dd urls: Migrate urls to use modern django pattern.
New path() function changed the way a regex pattern
is created from urls - it adds escape backslashes,
so for testing purposes we need to take care of them
and remove them, to check if urls were tested.

Additionaly, regex patterns from urls can have
[^/]+ instead of [^/]*, so we need to take care
of it too.
2020-05-07 16:28:58 -07:00
Mateusz Mandera 812ac4714f retention: Optimize fetching of realms and streams with retention policy. 2020-05-07 16:28:05 -07:00
Tim Abbott f9a61aa645 i18n: Update translation data from Transifex.
With a fix for a test.
2020-05-07 16:09:16 -07:00
Anders Kaseorg 8411f7f884 version: Update for Zulip Desktop v5.2.0.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-05-07 11:25:15 -07:00
orientor 3ffc9466c9 openapi: Combine two similar openapi markdown extensions.
Previously api_description and api_code_examples were two independent
markdown extensions for displaying OpenAPI content used in the same
places.  We combine them into a single markdown extension (with two
processors) and move them to the openapi folder to make the codebase
more readable and better group the openapi code in the same place.
2020-05-05 21:42:31 -07:00
Tim Abbott 50ed031f8e test_docs: Update a string for verifying docs. 2020-05-05 17:26:22 -07:00
Hashir Sarwar 55a8e7dff2 settings: Offer hiding presence info from other users.
For privacy-minded folks who don't want to leak the
information of whether they're online, this adds an
option to disable sending presence updates to other
users.

The new settings lies in the "Other notification
settings" section of the "Notification settings"
page, under a "Presence" subheading.

Closes #14798.
2020-05-02 15:09:05 -07:00
Tim Abbott 3eaa71cef8 test_auth_backends: Add documentation for the main test interface. 2020-05-02 14:41:21 -07:00
Dinesh 5c1fe776c3 auth: Extend the template for "choose email" in GitHub auth flow.
This commit extends the template for "choose email" to mention for
users who have unverified emails that they need to verify them before
using them for Zulip authentication.

Also modified `social_auth_test_finish` to assert if all emails
are present in "choose email" screen as we need unverified emails
to be shown to user and verified emails to login/signup.

Fixes #12638 as this was the last task for that issue.
2020-05-02 14:30:31 -07:00
Mateusz Mandera 5f15af2382 tests: Clean out unnecessary ifs from GitHubAuthBackendTest helper.
After the refactor moving this logic into a helper inside of
GitHubAuthBackendTest, these checks became unnecessary and always True.
2020-05-02 13:40:29 -07:00
Dinesh 9f3872d2b4 tests: Refactor `social_auth_test`.
As "choose email" screen is only used for GitHub auth, the part
that deals with it is separated from `social_auth_test` and
dealt in a new function `social_auth_finish`. This new
`social_auth_finish` contains only the code that deals with
authentication backends that do not have "choose email" screen.
But it is overidden in GitHub test class to handle the
"choose email" screen.
It was refactored because `expect_choose_email_screen` blocks
were confusing while figuring out how tests work on non GitHub
auths.
2020-05-02 13:40:29 -07:00
Anders Kaseorg bdc365d0fe logging: Pass format arguments to logging.
https://docs.python.org/3/howto/logging.html#optimization

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-05-02 10:18:02 -07:00
sahil839 7cfbf660e6 actions: Remove deactivated stream from default stream groups.
Stream after deactivation should be removed from all the
default stream groups that it belongs to.

Fixes #14760.
2020-05-01 17:45:06 -07:00
jagansivam28 bbf5a5efed invitation: Make Member to see invitations sent by him/her.
Member of the org can able see list of invitations sent by him/her.
given permission for the member to revoke and resend the invitations
sent by him/her and added tests for test member can revoke and resend
the invitations only sent by him/her.

Fixes #14007.
2020-05-01 17:42:49 -07:00