Commit Graph

10008 Commits

Author SHA1 Message Date
Wyatt Hoodes 4dd8c133a9 export: Rename `--upload-to-s3` to be `--upload`.
The upload option will no longer be limited to strictly S3 uploads. This
commit serves as a preliminary step for supporting LOCAL_UPLOADS_DIR as
part of the public only export feature.
2019-05-20 19:59:57 -07:00
Tim Abbott 1353e94b29 test_retention: Add print-debugging.
We've been seeing nondeterministic failures in this test suite in CI
that we can't reproduce locally; these print statements should help
track them down.
2019-05-20 19:43:28 -07:00
Mateusz Mandera 4f9bbe9d66 email_mirror: Clean up and expand test_get_missed_message_token.
This is the only function in TestEmailMirrorLibrary, so we rename this
class to more appropriate TestGetMissedMessageToken, clean it up a bit
and add some extra checks to finally get email_mirror.py to 100% test
coverage.
2019-05-20 19:35:32 -07:00
Mateusz Mandera a0efd76f4e email_mirror: Rewrite log_and_report and cover it with tests.
log_and_report and its helper functions were mostly old code no longer
well adapted to how email mirror works currently, as well as having no
test coverage. We rewrite this part of the email to report errors in a
similar manner, and add tests for it. We're able to get rid of the
clunky and now useless debug_info dictionary in process message, as
log_and_report only needs the recipient email in its third argument.
2019-05-20 19:35:32 -07:00
Mateusz Mandera 2adcdd0c25 email_mirror: Don't pass debug_info to process_stream_message.
The only place in which process_stream_message used debug_info was to
set the 'stream' key, which would only be used if ZulipEmailForwardError
was raised after this line in the code - which is impossible, because after
that line only send_zulip (which doesnt raise this exception) and
logger.info get called, then process_stream_message successfully returns
and then process_message succesfully returns as well. So this debug_info
code wasn't doing anything. We remove it.
2019-05-20 19:35:32 -07:00
Aditya Bansal 43591d6c71 archives: Display globe icon for web public streams.
In this commit we start reserving the globe icon for web public
streams and replace instances of mixed usage of globe for public
streams with a '#'.
2019-05-20 18:13:31 -07:00
Joshua Pan c28c301506 org_settings: Display delivery_email to admins always is admin UI.
Mostly rewritten by Tim Abbott to ensure it correctly implements the
desired security model.

Administrators should have access to users' real email address so that
they can contact users out-of-band.
2019-05-20 18:13:31 -07:00
Tim Abbott 07856ad648 api: Override client_gravatar for EMAIL_ADDRESS_VISIBILITY_ADMINS.
Clients won't have access to user email addresses, and thus won't be
able to compute gravatars.

The tests for this are a bit messy, in large part because our tests
for get_events call subsections of it, rather than the main function.
2019-05-20 18:13:31 -07:00
Tim Abbott bcc6949461 zilencer: Add better error handling for IntegrityError.
This provides a clean warning and 40x error, rather than a 500, for
this corner case which is very likely user error.

The test here is awkward because we have to work around
https://github.com/zulip/zulip/issues/12362.
2019-05-20 17:53:43 -07:00
Rishi Gupta fbc2239f73 keyboard shortcuts: Update description for S. 2019-05-20 15:10:11 -07:00
Puneeth Chaganti bdf8183f55 open graph: Uploaded realm icon URL is absolute for some backends.
The `LocalUploadBackend` returns a relative URL, while the `S3UploadBackend`
returns an absolute URL. This commit switches to using `urljoin` to obtain the
absolute URL, instead of simply joining strings.
2019-05-20 13:14:28 -07:00
Hemanth V. Alluri 1a12e112d9 devtools: Completely overhaul the frontend for the integrations devtool.
This commit also adds a small functionality change where the results of
each webhook fixture message sent is now displayed to the user.

With a small tweak by tabbott to fix a styling bug.

Fixes #12122.
2019-05-20 12:36:15 -07:00
Hemanth V. Alluri ef98211f68 devtools: Add support for non-json fixtures for the integrations tool.
Note: If you're going to send fixtures which are not JSON or of the
text/plain content type, make sure you set the correct content type
in the custom headers.

E.g. For the wordpress fixtures the "Content-Type" should be set to
"application/x-www-form-urlencoded".
2019-05-20 12:31:52 -07:00
Hemanth V. Alluri 8214d65336 devtools: Add support for send_all in the integrations dev panel.
Using this feature a reviewer can easily send and view all fixture
messages for any given integration - with only JSON fixtures that is.
2019-05-20 12:29:10 -07:00
K.Kanakhin e930851d16 retention-period: Add more core code for retention policy.
This is a very old commit for #106, which has been on hiatus for a few
years.  It was significantly modified by tabbott to:
* Improve coding style and variable names
* Update mypy annotations style
* Clean up the testing logic
* Update for API changes elsewhere in our system

But the actual runtime code is essentially unmodified from the
original work by Kirill.

It contains basic support for archiving Messages, UserMessages, and
Attachments with a nice test suite.  It's still not usable in
production (e.g. it will probably break Reactions, SubMessages, etc.),
but upcoming commits will address that.
2019-05-19 20:22:47 -07:00
Tim Abbott ec4221abb9 actions: Return message ID from internal_send_*_message.
This is handy for code that needs to do something with the sent
message.  We need it for a retention policy code path, but it seems
likely we'll use it a lot down the line.
2019-05-19 19:18:06 -07:00
Hemanth V. Alluri 2bd9c8cb42 devtools: Add custom HTTP headers support to the integrations dev panel.
This commit introduces a simple field where the user can now specify custom
HTTP headers. This commit does not introduce an improved system for storing
HTTP headers as fixtures - such a change would modify both the existing unit
tests as well as this devtool.
2019-05-17 11:50:53 -07:00
Hemanth V. Alluri 64b4fd5923 management: Extract parse_headers in send_webhook_fixture_message.
This way other devtools such as the integrations development panel
could also use the method.
2019-05-17 11:50:53 -07:00
Puneeth Chaganti 5f50c4d799 cache: Clear realm descriptions when description is changed.
Clear both the rendered and the text descriptions for a realm, when the
description of the realm is changed.
2019-05-17 11:31:14 -07:00
Puneeth Chaganti f084075c2a cache: Clear rendered realm text description when flushing realm.
The commit 2d9c5b3575 introduced this cache key,
and didn't setup clearing of the cache correctly.
2019-05-17 11:31:14 -07:00
Anders Kaseorg cdd09d0441 meta_tags: Remove twitter:title, twitter:description, twitter:image.
Twitter falls back to og:title, og:description, og:image, which we set
identically, so these are redundant.

https://developer.twitter.com/en/docs/tweets/optimize-with-cards/overview/markup.html

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-05-16 21:19:08 -07:00
Hemanth V. Alluri bae8295c52 devtools: Add integrations dev panel.
This commit adds a new developer tool: The "integrations dev panel"
which will serve as a replacement for the send_webhook_fixture_message
management command as a way to test integrations with much greater ease.
2019-05-15 13:07:44 -07:00
Vishnu Ks 06983298ba export: Add support for exporting realm with member consent.
This lets us handle directly in our tooling the user experience that
we document for exporting a realm with member consent (before, it
required unpleasant manual work).
2019-05-15 12:35:32 -07:00
Vishnu Ks 8ebdbea4d5 tests: Remove commented out print statements in test_import_export. 2019-05-15 11:57:31 -07:00
vinitS101 611f1f8fd2 left_sidebar: Add "+Add streams" to bottom of streamlist.
Added a new button at the bottom of the stream list which redirects
users to '/#streams/all' where they can create new streams or subscribe
to new streams.
The button is not visible to guests.

Fixes #11642.
2019-05-14 16:12:51 -07:00
Tim Abbott dc2aa031f6 linkifiers: Allow semicolons in target URLs.
This fixes another user report.  The right fix is to convert this to
use a normal URL validator after substitution.
2019-05-14 11:17:55 -07:00
Puneeth Chaganti 4ac9778d69 url preview: Catch network errors during get for page content.
We may be successfully able to get the page once, to get the content type, but
the server or network may go down and cause problems when fetching the page for
parsing its meta tags.
2019-05-13 13:55:00 -07:00
Puneeth Chaganti 9fd1c40bb1 url preview: Timeout requests after 15 seconds. 2019-05-13 13:54:59 -07:00
Puneeth Chaganti 0b76b16101 url preview: Set a custom user agent for requests.
Some sites seem to block the default user agent of the requests
library. Using a custom user agent lets us show previews for some of
these sites.
2019-05-13 13:54:43 -07:00
Puneeth Chaganti 59555ee7e5 url preview: Confirm content-type before trying to show previews.
Currently, we only show previews for URLs which are HTML pages, which could
contain other media. We don't show previews for links to non-HTML pages, like
pdf documents or audio/video files. To verify that the URL posted is an HTML
page, we verify the content-type of the page, either using server headers or by
sniffing the content.

Closes #8358
2019-05-13 13:45:17 -07:00
Tim Abbott 0f2d7a354f realm_filters: Allows more use of & and friends in URLs.
We had some excessively tight rules about what characters were
allowed, which in particular prevented using `?foo=bar&baz=quux`
structures in the realm filters URLs.

Fixes #12239.
2019-05-13 13:31:09 -07:00
Yashashvi Dave 6f6e87f45d models: Rename 'Jitsi' to 'Jitsi Meet' in Realm model.
Fixes #12293
2019-05-13 12:08:04 -07:00
Yashashvi Dave 3efead6f5f /json/realm: Return error on not-avaliable video chat provider value.
This adds some missing error handling from the video chat provider
code path.
2019-05-13 12:07:44 -07:00
Yashashvi Dave cb85ca8601 models: Alter video_chat_provider field type to integer.
Migration rewritten by tabbott because it did not work.
2019-05-13 12:02:28 -07:00
Yashashvi Dave d7f9b7eea3 models: Add dict for video chat providers in Realm model. 2019-05-13 11:34:34 -07:00
Yashashvi Dave bbff797a93 zerver/tests/test_realm: Fix not-avaliable video-chat-provider value. 2019-05-13 11:34:34 -07:00
Tim Abbott 45305c93ae auth: Fix Apache SSO port number in confirmation links.
We had a report in the thread around
https://chat.zulip.org/#narrow/stream/31-production-help/topic/Apache-based.20SSO/near/741013
that confirmation links were taking the user to the /register form on
the Apache server, which of course doesn't work because the Apache
server architecture we have is intended to only serve a single
endpoint, /accounts/login/sso, and not any static assets (etc.).

This manifested as users getting a broke page with a bunch of JS
errors about missing static assets when trying to sign up for an
account.  The right fix is to ensure that we serve these confirmation
links (and maybe in the future, redirects) to the nginx server.
2019-05-13 11:14:41 -07:00
Puneeth Chaganti 5dee17dca0 bugdown: Show previews for manually created youtube playlists.
Youtube playlists can be created by manually listing video_ids, as follows:
https://youtube.com/watch_videos?video_ids=vid1,vid2,vid3. This commit adds
previews for URLs of this type, using the first video ID.
2019-05-12 22:24:42 -07:00
Puneeth Chaganti a1f0713b2c bugdown: Show previews for youtube playlist URLs, if possible.
If a youtube playlist URL has a video-id, we show a preview for the URL.

Closes #8562
2019-05-12 22:24:42 -07:00
Puneeth Chaganti 4de261c2de bugdown: Don't show previews for youtube URLs without video ids.
`youtube.com/playlist?list=<list-id>` incorrectly matches the regex since the
change in 8afda1c1bb. The regex was modified to
match URLs of the form `youtu.be/<id>` and this playlist URL incorrectly matches
with the `<id>` set to `playlist`.

This commit avoids this match by verifying that the ID is not playlist.
2019-05-12 22:24:42 -07:00
Roman Godov a50824e031 models: Rename Subscription.in_home_view field to is_muted.
This renames Subscription.in_home_view field to is_muted, for greater
clarity as to what it does just from seeing the setting name, without
having to look it up.

Also disabled an obsolete test_migrations test.

Fixes #10042.
2019-05-12 22:08:10 -07:00
Tim Abbott 5ec8f6e812 test_migrations: Disable migrations test.
Our migrations test suite only really works when no future migrations
conflict with it.
2019-05-12 22:06:17 -07:00
Mateusz Mandera f71c1e6a0c email_mirror: Add test for process_message not finding valid recipient. 2019-05-12 18:55:47 -07:00
Mateusz Mandera fb3fa7883b email_mirror: Clean up TestStreamEmailMessagesEmptyBody.
These tests have some code and comments that only used to apply when
these empty body scenarios used to raise the regular
ZulipEmailForwardError - now they raise ZulipEmailForwardUserError.
We adapt the tests to this fact and test by mocking logging.warning and
making sure it gets called with the intended warning message. This is
also needed to cover the ZulipEmailForwardUserError case with tests to
get to 100% coverage of email_mirror.py.
2019-05-12 18:55:24 -07:00
Mateusz Mandera f892ef1d3f email_mirror: Add tests for messages with attachments. 2019-05-12 18:55:24 -07:00
Mateusz Mandera ede0991ef5 email_mirror: Add tests for filter_footer. 2019-05-12 18:55:24 -07:00
Mateusz Mandera 30c588636b email_mirror: Add test for stream missed message emails. 2019-05-12 18:55:24 -07:00
Mateusz Mandera c2e5661f69 email_mirror: Add test for redis_client.hmget returning None values.
We add a test for the case "if not all(val is not None for val in result):"
on result returned by redis_client.hmget in send_to_missed_message_address.
2019-05-12 18:55:24 -07:00
Mateusz Mandera 2e9dfc1739 email_mirror: Add test to cover empty body after footer stripping. 2019-05-12 18:55:24 -07:00
Puneeth Chaganti 62d9ad534c digest: Trigger additional query to make tests more deterministic.
A couple of tests asserted that the number of queries were within a range,
because they ran one additional query when they were run individually, as
compared to running all the tests in `TestDigestEmailMessages`. We now trigger
these additional queries within the tests, to make the tests deterministic and
assert that the number of queries is a number, instead of a range.
2019-05-09 15:10:05 -07:00
Puneeth Chaganti ab2850c225 digest: Re-enable digest emails for soft deactivated users.
Digest emails were disabled for soft deactivated users, since UserMessage
objects are created for such users lazily when they return.

We now compute the message list for gathering hot conversations by looking at
all the messages sent to the streams where the user is subscribed, while they
were subscribed.

Fixes #6297
2019-05-09 15:10:05 -07:00
Puneeth Chaganti 6abed82fb9 digest: Use one hour cutoff to generate digest emails in test.
Otherwise, the test may flake on a slow/hosed machine, where simulating a
conversation takes longer than 1 sec.
2019-05-09 15:10:05 -07:00
Pragati Agrawal a838de63d8 settings: Add option for fixed/fluid width.
This adds a new option for "fluid width" under `Display settings` section
of SETTINGS/DISPLAY SETTINGS tab.

Fixes: #11844.
2019-05-09 13:57:57 -07:00
Eeshan Garg 49ac516b50 webhooks/pivotal: Use proper punctuation. 2019-05-09 13:05:49 -07:00
Eeshan Garg 550ea64b4d webhooks/trello: Use proper punctuation. 2019-05-09 13:05:49 -07:00
Eeshan Garg b434cc7054 webhooks/taiga: Use proper punctuation and new style string formatting. 2019-05-09 13:05:49 -07:00
Eeshan Garg eaf1c4d8b2 webhooks/teamcity: Improve punctuation and message formatting. 2019-05-09 13:05:49 -07:00
Eeshan Garg 6d62522ff0 webhooks/raygun: Improve punctuation and message formatting. 2019-05-09 13:05:49 -07:00
Eeshan Garg 44038fc431 webhooks/raygun: Raise exception for unsupported event types. 2019-05-09 13:05:49 -07:00
Eeshan Garg 5a1f70f6e8 webhooks/opsgenie: Improve punctuation and message formatting. 2019-05-09 13:05:49 -07:00
Eeshan Garg 120937a1f4 webhooks/mention: Use proper message formatting. 2019-05-09 13:05:49 -07:00
Eeshan Garg 1948924cdb webhooks/jira: Improve punctuation and message formatting. 2019-05-09 13:05:49 -07:00
Eeshan Garg ef070ef019 webhooks/github: Improve punctuation. 2019-05-09 13:05:49 -07:00
Eeshan Garg 9516699c73 webhooks/front: Use proper punctuation and fix spelling errors. 2019-05-09 13:05:49 -07:00
Mateusz Mandera 40f5755546 email_mirror: Handle case of unspecified charset in Content-Type header.
If the text part of an email message didn't specify the charset in the
Content-Type header, the text content wouldn't be found. We fix this, by
assuming us-ascii charset in those cases, as specified by RFC6657:
https://tools.ietf.org/html/rfc6657
2019-05-09 09:57:40 -07:00
Harshit Bansal b553507412 subscriptions: Migrate notification setting defaults model.
This commit migrates the Subscription's notification fields from a
BooleanField to a NullBooleanField where a value of None means to
inherit the value from user's profile.

Also includes a migrations to set the corresponding settings to None
if they match the user profile's values. This migration helps us in
getting rid of the weird "Apply to all" widget that we offered on
subscription settings page.

The mobile apps can't handle None appearing as the stream-level
notification settings, so for backwards-compatibility we arrange to
only send True/False to the mobile apps by applying those defaults
server-side.  We introduce a notification_settings_null value within a
client_capabilities structure that newer versions of the mobile apps
can use to request the new model.

This mobile compatibility code is pretty effectively tested by the
existing test_events tests for the subscriptions subsystem.
2019-05-08 17:45:10 -07:00
Tim Abbott 8a2057e3cf test_events: Print out the state when it hasn't changed.
This is often a part of what one will want when debugging.
2019-05-08 17:43:21 -07:00
vinitS101 18a424be79 uploads: Remove unusable UI elements if file uploading is disabled.
If MAX_FILE_UPLOAD_SIZE is set to 0, then UI elements like the upload
icon in the compose and message edit UI and "Attachments" menu in
"/#settings" are not displayed.
A different error message is also displayed if a user tries to drag and
drop or paste a file into the compose message box.

Fixes #12152.
2019-05-08 17:10:07 -07:00
Rishi Gupta 98da11c558 support: Rename deactive to deactivated. 2019-05-08 15:09:48 -07:00
Rishi Gupta d2b32749cc audit log: Log acting_user for do_deactivate_realm.
Currently there's no way to tell the difference between "a server admin
deactivated a realm due to it being spammy" vs "a realm admin deactivated
the realm".
2019-05-08 15:09:48 -07:00
Tim Abbott ddd3a36536 soft deactivation: Remove useless conditional.
Due to my misreading the code and a sloppy search, I thought in
8218bf101c that
all_stream_subscription_logs didn't filter for streams.

While changing this, we'll switch to using `.modified_stream_id` for
potentially better performance.
2019-05-08 14:40:33 -07:00
Puneeth Chaganti d474a41c03 digest: Turn off digest_emails_enabled flag for realms by default. 2019-05-08 14:39:12 -07:00
Shubham Padia dd28413c4a ldap: Do not modify self._LDAPUser.attrs in test_query_email_attr.
Fixes #12273.
When running the test_query_email_attr test in reverse, the test failed
because self._LDAPUser.attrs was being modified and it was being shared
with other tests.
2019-05-08 09:55:44 -07:00
Vishnu Ks ce88df5069 tests: Patch maybe_update_first_visible_message_id properly.
Otherwise this will result in error when running
tests with --parallel=1 --reverse.
2019-05-08 09:55:03 -07:00
Yashashvi Dave 02a1b2e26b server events: Add realm-time-sync event to update stream privacy.
Add event to update stream settings whenever stream privacy is changed
accordingly.

Fixes #9470
2019-05-07 17:33:59 -07:00
Eeshan Garg 3af4b7d750 webhooks/freshdesk: Improve message formatting and punctuation. 2019-05-07 16:45:01 -07:00
Eeshan Garg c4e4ceadb5 webhooks/git: Use proper punctuation for branch creation templates. 2019-05-07 16:45:01 -07:00
Eeshan Garg f1228b3550 webhooks/bitbucket3: Use proper punctuation. 2019-05-07 16:45:01 -07:00
Eeshan Garg ab8aae6d0c git_webhooks: Use proper punctuation for PR/issue messages. 2019-05-07 16:45:01 -07:00
Eeshan Garg 2d4b734415 webhooks/solano: Improve message formatting and punctuation.
This commit also removes the conditional for when a build status
does not have a corresponding emoji. In such a case, it is better
to have no emoji than displaying some boilerplate text about no
appropriate emoji being available.
2019-05-07 16:45:01 -07:00
Anders Kaseorg 9efda71a4b get_realm: raise DoesNotExist instead of returning None.
This makes the implementation of `get_realm` consistent with its
declared return type of `Realm` rather than `Optional[Realm]`.

Fixes #12263.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-05-06 21:58:16 -07:00
Puneeth Chaganti 735b6cb761 digest: Remove code to gather new users and unread pms. 2019-05-06 17:43:53 -07:00
Puneeth Chaganti 77e0154800 test_users: Add assertion that guests can't see all realm users.
This commit adds coverage for the `UserProfile.can_access_all_realm_members`
method outside of tests in `test_digest`.
2019-05-06 17:43:53 -07:00
Puneeth Chaganti be762f9485 digest: Strip down the digest email removing a lot of fluff. 2019-05-06 17:43:52 -07:00
David Wood 34d810aac3 settings: Migrate to create_stream_policy structure.
This commit replaces the `create_stream_by_admins_only` setting with a
new `create_stream_policy` setting, which mirroring the structure of
the existing `invite_to_stream_policy`.

This is important preparation for migrating the waiting period feature
to be its own independent setting.

Fixes #12236.
2019-05-06 16:27:55 -07:00
Tim Abbott 3ecdabdc77 soft_deactivation: Add temporary nocoverage to fix CI. 2019-05-06 16:14:31 -07:00
Tim Abbott cf0fc7c221 test_link_embed: Fix unused variable.
This should have been in bc2ebd0f09.
2019-05-06 16:04:37 -07:00
Puneeth Chaganti bc2ebd0f09 url preview: Refactor test code to create mock responses. 2019-05-06 12:37:32 -07:00
Puneeth Chaganti da33b72848 url preview: Use in-memory caching in dev environment. 2019-05-06 12:37:32 -07:00
Puneeth Chaganti 1f6306a5a7 url preview: Cleanup import ordering. 2019-05-06 12:37:32 -07:00
Puneeth Chaganti d56b16b275 url preview: Ignore open graph tags without a content attribute. 2019-05-06 12:37:32 -07:00
Puneeth Chaganti d02eb99831 url preview: Return generic parser <p> text as str (not bs4 string). 2019-05-06 12:37:32 -07:00
Wyatt Hoodes 2b3de816a8 test_classes.py: Add activation to default language.
Running the backend tests with a high number of processes can cause
unexpected errors with language changes.  When certain tests that change
the default language, (without explicitly overriding the teardown method
to reset the default language), interleave with other tests that are
expecting the language to be in English, discrepancies arise.

This fixes a common nondeterministic test failure with high levels of
parallelization.
2019-05-06 11:16:08 -07:00
Tim Abbott 1af4f8fe20 soft_deactivation: Add some explanatory comments.
This function still doesn't make sense in the way I'd like it to, but
this at least records what algorithm we're trying to implement.
2019-05-05 18:33:15 -07:00
Tim Abbott eb97e9fae0 soft_deactivation: Fix buggy handling of race condition.
If a soft deactivated user had a subscription double-toggled without
any new messages being sent in between, add_missing_messages might
incorrectly process those two subscription changes in the wrong order.

Fortunately, the failure mode was usually to throw this exception:

django.db.utils.IntegrityError: duplicate key value violates unique
constraint "zerver_usermessage_user_profile_id_message_id_4936d0df_uniq"
DETAIL:  Key (user_profile_id, message_id)=(4, 57) already exists.

Our unit tests actually had this precise setup some fraction of the
time, because a bit of the test setup code subscribed+unsubscribed the
target user without sending any messages in between, resulting in a
test failure something like 50% of the time.

The original exception was hard to reproduce reliably originally
(resulting in an extremely annoying nondetermnistic test failure), but
is easily reproducible by changing the "id" to "-id" in this change to
always mis-order the processing of those RealmAuditLog events.
2019-05-05 18:29:20 -07:00
Tim Abbott 8218bf101c soft_deactivation: Fix buggy handling of other streams.
Previously, our soft-deactivation logic incorrectly did not filter the
set of stream subscription changes to look at to only include the
target stream.

This could result in unspecified buggy behavior.
2019-05-05 18:29:20 -07:00
Tim Abbott 8e995deab5 soft_deactivation: Make the stream_messages mutation logic clearer.
This changes our code from something that's pretty nasty bad behavior
mutating objects during a loop to only somewhat bad behavior.
2019-05-05 18:29:20 -07:00
Tim Abbott a1d2b73790 soft_deactivation: Clarify loop logic around stream_messages.
Break will do the same thing as continue here, as each iteration will
have the same result, and it's also worth explaining why this isn't
one layer up in the loop setup.
2019-05-05 18:29:20 -07:00
Puneeth Chaganti dc1571426e open graph: Use the complete URL for open graph URLs.
Closes #12199
2019-05-05 17:51:53 -07:00
Vishnu Ks 176d37a513 tests: Make tests work in reverse order in mattermost_importer. 2019-05-04 08:49:34 -07:00
vinitS101 81b5a72252 admin_settings: Change maxfilesize to max_file_upload_size.
Renamed maxfilesize to max_file_upload_size for consistency.
Related to #12152.
2019-05-03 17:36:09 -07:00
David Wood 16e9892ab2 webhooks/jira: Include issue title in messages. 2019-05-03 20:03:31 -02:30
Vishnu Ks 123bcea518 management: Don't use sys.exit(1).
Using sys.exit in a management command makes it impossible
to unit test the code in question.  The correct approach to do the same
thing in Django management commands is to raise CommandError.

Followup of b570c0dafa
2019-05-03 14:20:39 -07:00
Tim Abbott 687f1bcdaf test_classes: Fix missing flushing of per-request caches.
We should definitely be starting each test case with an empty copy of
the per-request caches, since their intended duration is even shorter
than a request.

This was masked by the fact that these caches are automatically
flushed when one makes an actual request to the Zulip API; so the
problems were only manifesting in tests like test_events, where we
call lower-level functions that access a per-request cache without
using the Zulip API.
2019-05-03 13:52:56 -07:00
Tim Abbott 5cddf07361 test_mattermost_importer: Fix buggy use of wrong paths.
This worked for me when testing locally because it was using a
constant path, and so was accessing the product under `var/` of the
previous test run.
2019-05-03 13:30:54 -07:00
Tim Abbott 0adb93db32 test_classes: Fix data import path reuse parallelism errors.
The make_import_output_dir helper function used a path determined
primarily by the filename of the fixture being used, and expected to
have complete control over that path for the duration of the test.

This resulted in nondeterministic errors if our two test classes that
ran Mattermost import code ran at the same time.
2019-05-03 13:23:09 -07:00
Shubham Padia 884c19c3bc settings: NAME_CHANGES_DISABLED disables the name change button.
Fixes #12251.

Previously when disabling name changes in server settings, instead
of realm settings, the name edit button did not get disabled.
Changing name resulted in a message stating `no changes made`.
2019-05-03 12:58:51 -07:00
Shubham Padia 8c01f396db settings: Add AVATAR_CHANGES_DISABLED server setting.
Fixes #12132.
Realm setting to disable avatar changes is already present.
The `AVATAR_CHANGES_DISABLED` setting now follows the same
2-setting model as `NAME_CHANGES_DISABLED`.
2019-05-03 12:58:51 -07:00
Shubham Padia 4994a6c2c9 realm: Add setting to disable avatar changes.
This is useful when syncing avatars from an integrated LDAP/active
directory.

The upload avatar and delete avatar buttons are hidden if avatar
changes are disabled and the user is a non-admin.
If the user has a gravatar set, then the user will not be able to
upload an image as their avatar if avatar changes are disabled.

Part of #12132.
2019-05-03 12:52:43 -07:00
Tim Abbott b570c0dafa convert_mattermost_data: Don't sys.exit(1).
This behavior ends up being quite problematic for running unit tests;
the thing one is supposed to do in management commands is raise
CommandError.
2019-05-02 11:13:41 -07:00
neiljp (Neil Pilgrim) 7bbc620b28 mypy: Require topics to be non-None in Addressee. 2019-05-01 17:52:37 -07:00
Puneeth Chaganti bfc3e3c0c7 html_to_text: Add delimiters between text from different elements.
This module is used to render the HTML of pages like our user documentation 
into text for use in open graph previews of those articles.  It provided somewhat
confusing output in the case that there were paragraph breaks in the original message,
because text with multiple paragraphs and list items does't read very well. This commit
adds `|` as a delimiter between paragraphs, and prefixes list items with a `*`.

Closes #12228
2019-05-01 17:35:20 -07:00
okay 1694831029 bugdown: Fix double processed emoji tags inside inline tags.
When an emoji is nested inside another inline tag - like em or strong -
it was getting double processed because of the way the inlinePattern
TreeProcessor runs (it runs recursively). With this fix, we set the
inner text of the emoji span as an AtomicString, preventing us from
double processing the emoji's text.

Fixes #11621

Test Plan:

* Add test case for **😄**, verify it passes.
* Go into local dev server and send "**😄**" to self and verify the DOM
does not have double <span> tags for the emoji.
* Run zerver.tests.test_push_notifications and verify the markdown test case matches
the text_content field properly
2019-05-01 17:03:15 -07:00
Eeshan Garg 28109e0f9e webhooks/jira: Ignore comment_created message.
comment_created payloads may not contain the required issue data
to format a useful notification, therefore, it is better to handle
issue comments through issue_updated events (which we already do).

Fixes: #11995.
2019-05-01 16:32:58 -07:00
Mateusz Mandera c1ceba9037 rate_limiter: Move email_mirror limiter to use rate_limit_entity.
We change the rate limiting code in the email mirror to use the new,
general rate_limit_entity function.
2019-05-01 12:54:32 -07:00
Mateusz Mandera 40763070b7 rate_limiter: Generalise some rate_limit_user code.
We create rate_limit_entity as a general rate-limiting function for
RateLimitedObjects, from code that was possible to abstract away from
rate_limit_user and that will be used for other kinds of rate limiting.
We make rate_limit_user use this new general framework from now.
2019-05-01 12:54:32 -07:00
neiljp (Neil Pilgrim) 827c016b98 mypy: Enable strict-optional for messages.py.
This also fixes an issue where we were setting num_after to None,
which should have been 0, to disable fetching messages after the
anchor.
2019-05-01 11:24:47 -07:00
neiljp (Neil Pilgrim) 47de637c3d mirrors: Refactor create_mirrored_message_users to use new Exception.
This enables the function to either return a valid UserProfile or raise
InvalidMirrorInput, which is clearer and more pythonic than the previous
approach of a tuple of a bool and Optional[UserProfile].

In making the type clearer, this improves checking with mypy.

Tests updated.
2019-05-01 11:18:22 -07:00
neiljp (Neil Pilgrim) 3d43682f1b mypy: Enable strict-optional for auth.py. 2019-05-01 10:49:25 -07:00
Tim Abbott edb956091f export: Add a blank line in S3 upload output.
This should be more readable.
2019-04-30 16:37:23 -07:00
Tim Abbott 98a1e340ad filters: Don't include realm_counts/installation_counts in errors.
This avoids unnecessarily long error emails for these.
2019-04-30 14:10:59 -07:00
David Wood 272ed90685 settings: Create an explicit invite_to_stream_policy setting.
This commit creates a new organization setting that determines whether
a user can invite other users to streams. Previously this was linked
to the waiting period threshold, but this was both not documented and
overly limiting.

With significant tweaks by tabbott to change the database model to not
involve two threshhold fields, edit the tests, etc.

This requires follow-up work to make the create stream policy setting
work how this code implies it should.

Fixes #12042.
2019-04-29 17:11:28 -07:00
Vishnu Ks 89ada6c770 tests: Remove unecessary word from convert_data test name. 2019-04-29 11:23:17 -07:00
Vishnu Ks ecdd3bea43 billing: Add cron job to run invoice_plans once a day.
Fixes #11960
2019-04-29 11:23:17 -07:00
Puneeth Chaganti f3f172991b open graph: Use uploaded realm icon for open graph images. 2019-04-28 22:50:27 -07:00
Tim Abbott 9e8cfbea3b test_email_notifications: Fix a nondeterministic test.
This test made the assumption that the two emails would be ordered in
a query without a ordering.
2019-04-28 22:00:03 -07:00
Puneeth Chaganti a98341c79f digest: Add an organisation setting to choose the digest weekday.
Closes #549.
2019-04-28 18:40:07 -07:00
Puneeth Chaganti f3d0ccb9f6 models: Add a digest_weekday field to the Realm model.
Allow realms to specify the day of the week when the digest should be sent out.
When enqueue-ing digests, pick only the realms that chose the current weekday as
the day to send out digests.
2019-04-27 15:20:41 -07:00
Puneeth Chaganti b0f485e95f org settings: Allow admins to enable or disable digest emails. 2019-04-27 15:20:41 -07:00
Mayank Madan 4c81474a4f webhooks: Add Buildbot integration.
With some minor changes by Eeshan Garg.

Fixes: #5639.
2019-04-27 15:19:16 -07:00
Eeshan Garg a73e8109b7 webhooks: Remove the legacy GitHub integration.
The github-services model for how GitHub would send requests to this
legacy integration is no longer available since earlier in 2019.
Removing this integration also allows us to finally remove
authenticated_api_view, the legacy authentication model from 2013 that
had been used for this integration (and other features long since
upgraded).

A few functions that were used by the Beanstalk webhook are moved into
that webhook's implementation directly.
2019-04-27 15:13:44 -07:00
Wyatt Hoodes d4715f23d7 public_export: Add backend API endpoint for triggering export.
An endpoint was created in zerver/views.  Basic rate-limiting was
implemented using RealmAuditLog.  The idea here is to simply log each
export event as a realm_exported event.  The number of events
occurring in the time delta is checked to ensure that the weekly
limit is not exceeded.

The event is published to the 'deferred_work' queue processor to
prevent the export process from being killed after 60s.

Upon completion of the export the realm admin(s) are notified.
2019-04-26 17:24:29 -07:00
Tim Abbott 8b5d2e9631 export: Return the S3 URL we uploaded data to.
This will make it possible to access that URL from the caller for the
data export tool.
2019-04-26 17:22:02 -07:00
Puneeth Chaganti dabe267d00 html_to_text: Add arg to specify html tags for generating text.
Closes #11497
2019-04-25 15:29:16 -07:00
Puneeth Chaganti 2d9c5b3575 open graph: Display realm description as open graph description. 2019-04-25 15:29:16 -07:00
Puneeth Chaganti a653fcca93 html_to_text: Escape text when using as description. 2019-04-25 15:29:16 -07:00
Puneeth Chaganti 009beabe53 test_middleware: Use test helper in nonexistent page open graph test. 2019-04-25 15:29:16 -07:00
Puneeth Chaganti 41ff94bdba test_middleware: Parse html in tests instead of using regex.
This slows down the tests by about 5-10% -- the tests go from 0.6s to 0.630s or
so. But, this seems like a change worth making to prevent open-graph metadata
breaking HTML.
2019-04-25 15:29:16 -07:00
Puneeth Chaganti 7d7134d45d html_to_text: Extract code for html to plain text conversion. 2019-04-25 15:29:16 -07:00
Tim Abbott cc421d4415 tests: Fix bad use of mock local variable name.
This ended up masking the mock module.
2019-04-25 15:28:10 -07:00
sameerchoubey 048ec39824 tools/test-api: Test Client.update_stream. 2019-04-25 16:24:59 -02:30
sameerchoubey 6790c22d8d tools/test-api: Test Client.update_presence. 2019-04-25 16:24:59 -02:30
sameerchoubey 598727f696 tools/test-api: Test Client.update_notification_settings. 2019-04-25 16:24:59 -02:30
sameerchoubey cedce30d99 tools/test-api: Test Client.update_user_group_members. 2019-04-25 16:24:59 -02:30
sameerchoubey 3140aabbd7 tools/test-api: Test Client.remove_user_group. 2019-04-25 16:24:59 -02:30
sameerchoubey 49e843cd75 tools/test-api: Test Client.update_user_group. 2019-04-25 16:24:59 -02:30
sameerchoubey 2c675efbba tools/test-api: Test Client.create_user_group. 2019-04-25 16:24:59 -02:30
sameerchoubey 53b58d0c7f tools/test-api: Test Client.remove_alert_words. 2019-04-25 16:24:59 -02:30
sameerchoubey c5c2c13880 tools/test-api: Test Client.add_alert_words. 2019-04-25 16:24:59 -02:30
sameerchoubey b26b4ce60c tools/test-api: Test Client.get_alert_words. 2019-04-25 16:24:59 -02:30
sameerchoubey 2d9b54f0d5 tools/test-api: Test Client.delete_stream. 2019-04-25 16:24:59 -02:30
sameerchoubey 56ad6758f6 tools/test-api: Test Client.remove_reaction. 2019-04-25 16:24:59 -02:30
Anders Kaseorg 21dc34cc52 open graph: HTML-escape og:description, twitter:description.
The entire idea of doing this operation with unchecked string
replacement in a middleware class is in my opinion extremely
ill-conceived, but this fixes the most pressing problem with it
generating invalid HTML.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-04-23 15:53:59 -07:00
Rishi Gupta 29d30ceab7 settings: Decouple enable_push_notifications_offline from PM setting.
Note that this setting has always applied to both streams and PMs; the test
just clarifies that that is the case.
2019-04-23 15:24:39 -07:00
Anders Kaseorg 9a9de156c3 lint: Fix calls to _() on computed strings.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-04-23 15:23:03 -07:00
Anders Kaseorg 643bd18b9f lint: Fix code that evaded our lint checks for string % non-tuple.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-04-23 15:21:37 -07:00
Eeshan Garg c012362fde webhooks/librato: Use proper punctuation. 2019-04-23 14:32:21 -07:00
Eeshan Garg d733cc500c webhooks/groove: Rewrite the integration and improve messages.
The initial goal was to improve message formatting and punctuation
but after a closer look, I realized that a larger refactor was
worth it for clarity and redability.
2019-04-23 14:32:21 -07:00
Tim Abbott cb6c1e7a92 export: Fix log line about zerver_scheduledemail_users. 2019-04-23 13:52:02 -07:00
Anders Kaseorg 098e5ad3de templates: Fix more low-hanging HTML validation problems.
"http://localhost:9981/help/enable-emoticon-translations":323.13-324.31: error: The “align” attribute on the “th” element is obsolete. Use CSS instead.
"http://localhost:9981/help/enable-emoticon-translations":324.45-325.31: error: The “align” attribute on the “th” element is obsolete. Use CSS instead.
"http://localhost:9981/help/enable-emoticon-translations":329.13-330.23: error: The “align” attribute on the “td” element is obsolete. Use CSS instead.
"http://localhost:9981/help/enable-emoticon-translations":330.44-331.23: error: The “align” attribute on the “td” element is obsolete. Use CSS instead.
"http://localhost:9981/help/enable-emoticon-translations":337.6-337.9: error: Stray end tag “p”.
"http://localhost:9981/help/enable-emoticon-translations":337.6-337.9: error: Cannot recover after last error. Any further errors will be ignored.
"http://localhost:9981/api/incoming-webhooks-walkthrough":381.4-381.77: error: An “img” element must have an “alt” attribute, except under certain conditions. For details, consult guidance on providing text alternatives for images.
"http://localhost:9981/apps/":192.21-192.34: error: Bad value “” for attribute “src” on element “img”: Must be non-empty.
"http://localhost:9981/apps/":192.21-192.34: error: An “img” element must have an “alt” attribute, except under certain conditions. For details, consult guidance on providing text alternatives for images.
"http://localhost:9981/features/":225.9-225.82: error: An “img” element must have an “alt” attribute, except under certain conditions. For details, consult guidance on providing text alternatives for images.
"http://localhost:9981/features/":230.72-232.8: error: Text not allowed in element “svg” in this context.
"http://localhost:9981/features/":259.9-259.91: error: An “img” element must have an “alt” attribute, except under certain conditions. For details, consult guidance on providing text alternatives for images.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-04-23 13:00:59 -07:00
Tim Abbott fa37401e69 api docs: Fix typo in typing notifications docs.
notification_to should be just to here.

This resulted from the person doing the port into these docs not
understanding the full REQ API.
2019-04-23 11:24:07 -07:00
Anders Kaseorg 61982d9d47 uploads: Revert "Url encoded name of the file should be an ascii."
This reverts commit fd9dd51d16 (#1815).

The issue described does not exist in Python 3, where urllib.parse now
_only_ accepts (Unicode) str and does the right thing with it.  The
workaround was not being triggered and would have failed if it were.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-04-22 22:28:39 -07:00
Vishwesh Jainkuniya c007b9ea4a notifcations: Remove `user` from the payload.
This contains email of the user to whom notification is being
send. This has not been used in any past mobile releases, so it is
safe to remove it.

As user_id will be stable for the user, but not email. So it's better to
start consuming `user_id` instead of email on mobile.
2019-04-22 14:50:04 -07:00
Vishwesh Jainkuniya 447a517e6f notifications: Add `user_id` in the GCM & APNS payload.
This makes it easy to uniquely identify the user account associated
with a notification by, for example, the (realm_uri, user_id) pair.

This helps improve notifications in the mobile apps.
See https://github.com/zulip/zulip-mobile/pull/3407#discussion_r266196616

Fixes #11961.
2019-04-22 14:49:03 -07:00
Puneeth Chaganti 184c0b9870 render_markdown_path: Convert dicts to dict item tuples for caching.
Calls to `render_markdown_path` weren't getting cached since the context
argument is unhashable, and the `ignore_unhashable_lru_cache` decorator ignores
such calls. This commit adds a couple of more decorators - one which converts
dict arguments to the function to a dict items tuple, and another which converts
dict items tuple arguments back to dicts. These two decorators used along with
the `ignore_unhashable_lru_cache` decorator ensure that the calls to
`render_markdown_path` with the context dict argument are also cached.

The time to run zerver.tests.test_urls.PublicURLTest.test_public_urls drops by
about 50% from 8.4s to 4.1s with this commit. The time to run
zerver.tests.test_docs.DocPageTest.test_doc_endpoints drops by about 20% from
3.2s to 2.5s.
2019-04-22 13:53:18 -07:00
Eeshan Garg d8d9695087 webhooks/newrelic: Improve message formatting and punctuation. 2019-04-17 16:22:43 -07:00
Eeshan Garg f86e445fe6 webhooks/sentry: Improve message formatting and punctuation. 2019-04-17 16:22:43 -07:00
Eeshan Garg 424061a07a webhooks/papertrail: Improve message formatting and punctuation. 2019-04-17 16:22:43 -07:00
Eeshan Garg 1c1b440adf webhooks/pingdom: Improve message formatter and punctuation. 2019-04-17 16:22:43 -07:00
Eeshan Garg b44d352af6 webhooks/pagerduty: Use colons before quote blocks. 2019-04-17 16:22:43 -07:00
Eeshan Garg e682889437 webhooks/insping: Improve message formatting and punctuation. 2019-04-17 16:22:43 -07:00
Eeshan Garg 5fccf197d1 webhooks/greenhouse: Improve message formatting and punctuation. 2019-04-17 16:22:43 -07:00
Eeshan Garg 674fcfcce1 webhooks/semaphore: Improve message formatting and punctuation. 2019-04-17 16:22:43 -07:00
Eeshan Garg 9d79becc0e webhooks/splunk: Improve message formatting and punctuation. 2019-04-17 16:22:43 -07:00
Eeshan Garg 21cdb6decc webhooks/wordpress: Improve message formatting and punctuation. 2019-04-17 16:22:43 -07:00
Eeshan Garg 0a0df3bc58 webhooks/zabbix: Improve message formatting and punctuation. 2019-04-17 16:22:43 -07:00
Shubham Padia 7743fa5297 auth: Redirect deactivated user to /login when attempting social login. (#12130) 2019-04-17 12:28:57 -07:00
Shubham Padia 31cedbc9b8 backup: Make `--output` an optional argument instead of a positional one.
Fixes #12150.
Previously the `output` argument was positional, while the docs described
the arguments as optional in the form of `--output=...`.
2019-04-17 12:14:11 -07:00
Shubham Padia 32cf192b9b backup: Add option `--skip-uploads` to skip uploads backup. 2019-04-17 12:14:11 -07:00
Shubham Padia 984d3f2f0e backup: Add option `--skip-db` to skip database backup.
This may be relevant for organizations that use a corporate remote
postgres server and just want to backup their configuration.
2019-04-17 12:14:11 -07:00
neiljp (Neil Pilgrim) 1348c8fd05 requirements: Upgrade mypy to 0.670.
Requires minor adjustment for accented 'i' in variable name.
2019-04-16 17:50:12 -07:00
Rishi Gupta 1e5df34a4c help: Make minor updates to import-from-mattermost. 2019-04-16 17:49:42 -07:00
Eeshan Garg f7fd9bd17d webhooks/statuspage: Improve message formatting and punctuation. 2019-04-14 22:20:02 -07:00
Eeshan Garg f6a601f09f webhooks/heroku: Improve message formatting and punctuation. 2019-04-14 22:20:02 -07:00
Eeshan Garg 8cf18fe4d2 webhooks/delighted: Improve message formatting and punctuation. 2019-04-14 22:20:02 -07:00
Eeshan Garg 33b0bf3cb4 webhooks/beeminder: Improve message formatting and punctuation. 2019-04-14 22:20:02 -07:00
Eeshan Garg b7ea727087 webhooks/basecamp: Improve message formatting and punctuation. 2019-04-14 22:20:02 -07:00
Eeshan Garg e7fee92463 webhooks/appveyor: Improve message formatting and punctuation. 2019-04-14 22:20:02 -07:00
Eeshan Garg 92509a0cac webhooks/ansibletower: Improve message formatting and punctuation. 2019-04-14 22:20:02 -07:00
Vishnu Ks 877d2bb68f docs: Add docs for exporting from gitlab omnibus mattermost. 2019-04-13 20:27:12 -07:00
Vishnu Ks 7718c00a36 docs: Add docs for exporting from non docker mattermost. 2019-04-13 20:27:12 -07:00
Puneeth Chaganti ca8e9fb800 bugdown: Make the youtube URL regex slightly easier to read. 2019-04-13 20:25:37 -07:00
Puneeth Chaganti 8afda1c1bb bugdown: Show preview for urls copied from the Youtube share widget. 2019-04-13 20:25:37 -07:00
Shubham Padia 3c09f226a4 auth: Redirect deactivated users with error for social auth backend.
Fixes #11937.

Also extracts the error message for a deactivated account to
`DEACTIVATED_ACCOUNT_ERROR`.
2019-04-13 19:58:15 -07:00
overide b263671c9e markdown: Fix unordered list not rendering in blockquote.
This fixes an issue where the hanging unordered list was not
rendering in blockquote; the problem was that we were not
adding an empty line(to satisfy the markdown) for hanging
unordered list if it is in blockquote. Both blockquote
and code block is fenced but we want to avoid rendering
the list if it's in the code block but not in blockquote.

Fixes: #11916.
2019-04-13 19:23:59 -07:00
Anders Kaseorg d03b8308a5 backup: Use tar --transform to arrange the tarball instead of symlinks.
This allows tar to print the real paths in error messages if something
goes wrong.

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2019-04-12 16:48:34 -07:00
Wyatt Hoodes bafcf3c664 export.py: Add 'delete after upload' option for removing tarball.
This allows removal of the local tarball upon a succesful s3 upload.

A part of the public-only-realm-export webapp feature.
2019-04-12 10:50:06 -07:00
Wyatt Hoodes 0db7d6c31b export.py: Refactor './manage.py export' core logic.
This commit serves as the first step in supporting "public export" as a
webapp feature.  The refactoring was done as a means to allow calling
the export logic from elsewhere in the codebase.
2019-04-12 10:50:06 -07:00
Tim Abbott ddf26a6ced test_events: Post-process state before comparisons in do_test.
This is important because upcoming features will include slightly more
complex logic in post_process_state that we'd ideally like to be
included in what this suite tests.

This requires a few related changes:

* A small change to post_process_state to sort the realm_users objects
  by user_id to ensure those data structures are stable.

* Improvements to the logic for checking if the initial state has
  changed to use match_states for better output.
2019-04-08 19:42:34 -07:00
Vishnu Ks 02c92e55a2 import: Add tool for importing teams from mattermost. 2019-04-05 17:53:03 -07:00
clarammdantas 85ddd73d8c message_edits: Notify subscribers when a message is changed.
Extend the list of users that have to be notified when a message is
changed, so that in addition to users who have a UserMessage row, any
users who subscribed later to a stream with history public to
subscribers will also get the update.

Fixes: #8750.
2019-04-05 17:06:24 -07:00
Eeshan Garg bc430909af webhooks/intercom: Implement support for all events. 2019-04-05 15:15:52 -07:00
Tim Abbott 4c3c669b41 mobile: Add support for recent_private_conversations API.
This adds experimental support in /register for sending key
statistical data on the last 1000 private messages that the user is a
participant in.  Because it's experimental, we require developers to
request it explicitly in production (we don't use these data yet in
the webapp, and it likely carries some perf cost).

We expect this to be extremely helpful in initializing the mobile app
user experience for showing recent private message conversations.

See the code comments, but this has been heavily optimized to be very
efficient and do all the filtering work at the database layer so that
we minimize network transit with the database.

Fixes #11944.
2019-04-04 16:47:43 -07:00
Eeshan Garg 0076b71411 webhooks/stripe: Ignore account updates with insufficient data.
Certain payloads for account updates do not include the
previous_attributes that allow us to figure out what was actually
updated. So, we just ignore just payloads.
2019-04-04 15:21:47 -07:00
Vishnu Ks d64f5e9ccd test_classes: Move make_output_dir to test_classes. 2019-04-04 13:51:52 -07:00
Vishnu Ks 719abbd352 test_classes: Move rm_tree to test_classes. 2019-04-04 13:51:52 -07:00
Vishnu Ks f517f72dd2 import: Make use of is_mirror_dummy in build_user_profile. 2019-04-04 13:51:52 -07:00
Vishnu Ks ac2de3024d test_classes: Create fixture_file_name helper. 2019-04-04 13:51:52 -07:00
Vishnu Ks bd4c3b3ebb import: Move make_user_messages to import_util.py. 2019-04-04 13:51:52 -07:00
Harshit Bansal b630a6456d commands: Fix handling of keyword arguments in `query_ldap` command.
This bug seems to be introduced by me while doing the refactoring
in `94649f58f2fe0ed78d84e597ad6676522cfef9be`.

Fixes: #12006.
2019-04-04 13:42:33 -07:00
aswanthkoleri b12f494031 documentation: Update the API documentation for Get all users. 2019-04-04 13:29:16 -07:00
aswanthkoleri b1a7d5a81d users: Add timezone field to Get all users API endpoint. 2019-04-04 13:29:14 -07:00
Tim Abbott 0831156614 auth: Use delivery_email in routes to fetch an API key + email.
This enables the mobile apps working correctly when delivery_email !=
email.
2019-04-04 13:24:29 -07:00
Tim Abbott 32c50a99fd events: Pass a Realm to get_raw_user_data.
This will be helpful for checking realm settings in upcoming commits.
2019-04-04 13:05:35 -07:00
Hemanth V. Alluri 636dad7809 webhooks/bitbucket3: Use strikethrough formatting for deleted comments. 2019-04-03 19:43:04 -02:30
Hemanth V. Alluri dffb536920 webhooks/bitbucket3: Link actor name with their profile. 2019-04-03 19:43:04 -02:30
Hemanth V. Alluri eecf95d2fd webhooks/bitbucket3: Add support for all pull-request events.
closes #11992.
2019-04-03 19:43:04 -02:30
Rishi Gupta dc98359fcf help: Add switching-between-organizations.
This is in response to a support ticket where the user had a closed left
sidebar, had added an organization, and then couldn't figure out how to
switch organizations. They had googled and found "The desktop app makes it
easy to switch between different organizations" in our help docs, which was
not sufficiently helpful.
2019-04-02 20:54:07 -07:00
Rishi Gupta 4924f7ad92 help: Update Tools & customization section to conform to style guide. 2019-04-02 17:05:59 -07:00
Rishi Gupta fcd0e8c809 help: Update Notifications articles to conform to style guide. 2019-04-02 16:45:25 -07:00
Hemanth V. Alluri 8239a3514a context_processors: Extract keys from zulip_default_context.
Previously, we had some expensive-to-calculate keys in
zulip_default_context, especially around enabled authentication
backends, which in total were a significant contributor to the
performance of various logged-out pages.  Now, these keys are only
computed for the login/registration pages where they are needed.

This is a moderate performance optimization for the loading time of
many logged-out pages.

Closes #11929.
2019-03-25 14:05:36 -07:00
Hemanth V. Alluri 9ed1dcc4b6 webhooks: Add Bitbucket Server integration.
This commit adds support for all "repo" events.
2019-03-24 22:06:29 -02:30
Tim Abbott 3da06fecd5 invite: Fix validation of referred_by field.
Previously, we could 500 if an organization administrator scanned
possible PreregistrationUser IDs looking for a valid invitation they
can interact with.

They couldn't do anything, so no security issue, but this fixes that
case to just be a 400 error as it should be.
2019-03-21 16:31:18 -07:00
Tim Abbott 1599625328 mypy: Add an explicit assert to check_message. 2019-03-21 16:31:18 -07:00
neiljp (Neil Pilgrim) 584bec70c7 i18n: Log error & return "Unknown" if invalid code in get_language_name.
Previously this could return None.
2019-03-21 15:32:13 -07:00
Mateusz Mandera c7c1dbec60 email_mirror: Raise ZulipEmailForwardError if email pattern not recognised.
With the previous commit, fixes #1836.

As specified in the issue above, we make
get_email_gateway_message_string_from_address raise an exception if
it doesn't recognise the email gateway address pattern. Then, we make
appropriate adjustments in the codepaths which call this function.
2019-03-21 15:25:57 -07:00
Mateusz Mandera e32c444ecf email_mirror: Move some helper functions out of actions.py.
These functions don't really belong in actions.py, so we move them out,
into email_mirror_helpers.py. They can't go directly into
email_mirror.py or we'd get circular imports resulting in ImportError.
2019-03-21 15:25:57 -07:00
Eeshan Garg 68877dba50 webhooks/bitbucket: Support empty push payloads with no user info. 2019-03-21 17:43:35 -02:30
Rishi Gupta 6c2c351894 onboarding: Update initial messages and stream descriptions. 2019-03-21 12:34:24 -07:00
Rishi Gupta a914b86694 onboarding: Simplify get_turtle_message. 2019-03-21 12:33:32 -07:00
Rishi Gupta e8741c448d refactoring: Move set_default_streams into do_create_realm.
After the commits leading up to this, the only meaningful use of this
function was in the realm creation process.
2019-03-21 12:33:19 -07:00
Rishi Gupta e527de3937 management: Remove set_default_streams.py.
We now support this directly in the realm admin UI.
2019-03-21 12:31:03 -07:00
Rishi Gupta e71a1a2b4e onboarding: Remove initial streams other than general and core team.
The hope is that by having a shorter list of initial streams, it'll
avoid some potential confusion confusion about the value of topics.
At the very least, having 5 streams each with 1 topic was not a good
way to introduce Zulip.

This commit minimizes changes to the message content in
`send_initial_realm_messages` to keep the diff readable. Future commits will
reshape the content.
2019-03-21 12:30:14 -07:00
Rishi Gupta d46b125bf2 onboarding: Change default notifications stream to #general. 2019-03-21 12:29:51 -07:00
Raymond Akornor d8a2bb2b7c deliver_email: Refactor to extract email delivery.
This is a follow up on #11181. We extract email delivery into
`zerver/lib/send-email.py` to make it cleaner and easy to test.
2019-03-20 14:52:05 -07:00
Tim Abbott 8e92b36cf3 models: Add more documentation for PreregistrationUser. 2019-03-20 14:23:36 -07:00
Tim Abbott e7cc246d27 actions: Add docstring for do_update_message.
This should help with its readability.
2019-03-20 12:48:47 -07:00
neiljp (Neil Pilgrim) 7b3a9f34b7 mypy: Assert if we do_update_message then we have rendered_content. 2019-03-20 12:48:47 -07:00
neiljp (Neil Pilgrim) a80617bc70 mypy: Fix color_map default parameter for Optional. 2019-03-20 12:48:43 -07:00
neiljp (Neil Pilgrim) a40a0312f1 mypy: Correct default=None parameter typing in actions.py. 2019-03-20 12:41:34 -07:00
neiljp (Neil Pilgrim) 3d9c4494f5 mypy: Extract addressee.stream_name() check to satisfy mypy. 2019-03-20 12:41:34 -07:00
Vishnu Ks d921fd25e4 import: Move SubscriberHandler to import_util. 2019-03-20 11:29:51 -07:00
Vishnu Ks 6e3720e0b7 gitter: Fix minor comment typo in build_userprofile. 2019-03-20 10:12:18 -07:00
Tim Abbott 4f2c9f46bd actions: Improve format for delete events.
There were several problems with the old format:

* The sender was not necessarily the sender; it was the person who did
  the deletion (which could be an organization administrator)
* It didn't include the ID of the sender, just the email address.
* It didn't include the recipient ID, instead having a semi-malformed
  recipient_type_id under the weird name recipient_user_ids.

Since nothing was relying on the old behavior, we can just fix the
event structure.
2019-03-19 23:16:03 -07:00
Hemanth V. Alluri 261166f96d context_processors: Cache the realm description in default context. 2019-03-18 22:19:18 -07:00
Tim Abbott 58ca1ba5a4 models: Disable caching of supported_auth_backends for now.
This seems to be causing hard-to-debug test failures.
2019-03-18 11:29:05 -07:00
Mateusz Mandera 5f88406133 rate_limit tests: Cover RateLimiterLockingException case in rate_limit_user. 2019-03-18 11:16:58 -07:00
Mateusz Mandera 1a3297d122 email_mirror: Make send_to_email_mirror.py use mirror_email_message.
We change the send_to_email_mirror management command, to send messages
to the email mirror through the mirror_email_message function instead of
process_message - this makes the message follow a similar codepath as
emails sent into the mirror with the postfix configuration, which means
going through the MirrorWorker queue. The reason for this is to make
this command useful for testing the new email mirror rate limiter.
2019-03-18 11:16:58 -07:00
Mateusz Mandera 1901775383 email_mirror: Add realm-based rate limiting.
Closes #2420

We add rate limiting (max X emails withing Y seconds per realm) to the
email mirror. By creating RateLimitedRealmMirror class, inheriting from
RateLimitedObject, and rate_limit_mirror_by_realm function, following a
mechanism used by rate_limit_user, we're able to have this
implementation mostly rely on the already existing, and proven over
time, rate_limiter.py code. The rules are configurable in settings.py in
RATE_LIMITING_MIRROR_REALM_RULES, analogically to RATE_LIMITING_RULES.

Rate limit verification happens in the MirrorWorker in
queue_processors.py. We don't rate limit missed message emails, as due
to using one time addresses, they're not a spam threat.

test_mirror_worker is adapted to the altered MirrorWorker code and a new
test - test_mirror_worker_rate_limiting is added in test_queue_worker.py
to provide coverage for these changes.
2019-03-18 11:16:58 -07:00
Mateusz Mandera 386813f42b test_queue_worker: Clean up test_mirror_worker.
We clean up test_mirror_worker for more readability, as well as make it
verify that mirror_email gets called the correct amount of times and use
a correct rcpt_to address, so that the test doesn't fail when some
verification of the address is added in the following commits
implementing rate limiting in the email mirror.
2019-03-18 11:16:58 -07:00
Mateusz Mandera 5b86734178 email_mirror: Change stream name encoding in mirror addresses.
Fixes #9840.

Old addresses caused bugs in some cases with non-latin characters in
stream names (see issue number above). We switch to using django's
slugify helper function to convert stream names to full ascii, while
also getting rid of problematic non-alphanumeric characters, in a
reasonable way. See Django's documentation for slugify to see more about
how this function works.

Tests extended by tabbott to cover cases where we do end up with ascii.
2019-03-18 11:14:22 -07:00
Mateusz Mandera a64a075ff1 email_mirror: Ignore stream_name part of receiving address.
To prepare for changing how the stream name gets encoded into mirror
email addresses while making sure old addresses keep working, we ignore
the stream_name part when receiving emails into the mirror and we only
look at the email_token to identify into which stream to mirror the
email.
2019-03-18 11:06:51 -07:00
Tim Abbott 898798a352 email_log: Fix buggy use of get_realm_stream.
I'm surprised that this wasn't a mypy error; we were passing a Realm
object as an integer, and predictably, this resulted in us
constructing a cache key that looked like this:

stream_by_realm_and_name:<Realm: zulip 1>:dd5...
2019-03-18 10:59:30 -07:00
Tim Abbott 51eeb0a3ee cache: Add missing : in test-backend key prefixes.
Previously, these cache keys looked like:
:1:9c26164d3a393e316e0f8210efe270e08710d45astream_by_realm_and_name:...

Now, they look like this:
:1:9c26164d3a393e316e0f8210efe270e08710d45a:stream_by_realm_and_name:...
2019-03-18 10:56:50 -07:00
Tim Abbott 5d20138d72 context_processors: Optimize calculation of social_backends.
This avoids a bunch of duplicated calls to auth_enabled_helper for our
social auth backends, which added up because auth_enabled_helper can
take 100us to run.
2019-03-17 15:20:36 -07:00
Tim Abbott a9350caee2 context_processors: Optimize calculation of no_auth_enabled.
This saves an unnecessary call to auth_enabled_helper.
2019-03-17 15:19:06 -07:00
Tim Abbott cb78ddc491 models: Fix performance of supported_auth_backends with caching.
See the comment, but this is a significant performance optimization
for all of our pages using common_context, because this code path is
called more than a dozen times (recursively) by common_context.
2019-03-17 15:14:05 -07:00
Tim Abbott 79ae9175bb context_processors: Avoid useless duplicate queries for realm object.
We have a few code paths that call get_realm_from_request multiple
times on the same request (e.g. the login page), once inside the view
function and once inside the common context processor code.  This
change saves a useless duplicate database query in those code paths.
2019-03-17 14:08:53 -07:00
Tim Abbott f270e354b1 login: Fix unnecessary queries on /login/ in development.
This block of code with 2 database queries is solely for the /devlogin
endpoint.  Removing that block from the /login code path makes it
easier to test /login perf in development.
2019-03-17 14:03:57 -07:00
Tim Abbott 0a7b6909b9 login: Fix a useless query with DevAuthBackend.
This block was setting `realm` to a value that it already was if you
read a bit further up in the function.
2019-03-17 14:03:27 -07:00
Hemanth V. Alluri 1269b72f47 context_processors: Do not render inline previews for realm description.
We never intended to render them for this use case as the result would
not look good, and now we have a convenient bugdown option for
controlling this behavior.

Since we're not storing the markdown rendering anywhere, there's
conveniently no data migration required.

Fixes #11889.
2019-03-17 13:55:10 -07:00
Mateusz Mandera a269c4bdd4 email_mirror: Make email_token a unique column of Stream. 2019-03-17 12:55:35 -07:00
Eeshan Garg 623ee15bee webhooks/bitbucket: Account for missing user info.
According to our logs, some BitBucket enterprise payloads may
not contain the name of the user who pushed all the commits.
2019-03-16 15:09:03 -02:30
sameerchoubey a47edd4fc8 webhooks/jira: Use dictionary to map events to functions. 2019-03-16 14:27:00 -02:30
Ben Muschol d526ff00f2 settings: Rename "user avatar" to "profile picture"
This renames references to user avatars, bot avatars, or organization
icons to profile pictures. The string in the UI are updated,
in addition to the help files, comments, and documentation. Actual
variable/function names, changelog entries, routes, and s3 buckets are
left as-is in order to avoid introducing bugs.

Fixes #11824.
2019-03-15 13:29:56 -07:00
Tim Abbott 0a145f2203 test_email_notifications: Fix a potentially flaky test.
The previous code assumed the `zulip` realm had ID 1.
2019-03-15 11:02:17 -07:00
Tim Abbott c7ebfdc978 notifications: Rename test_notifications to test_email_notifications.
Similar to the last commit, this makes the scope of this test suite
clearer.
2019-03-15 11:02:17 -07:00
Tim Abbott 50dc317466 notifications: Rename notifications.py to email_notifications.py.
This library is entirely about email notifications specifically, and
this rename should help make the codebase more readable.
2019-03-15 11:02:17 -07:00
Tim Abbott 28a4c143c8 emails: Move clear_scheduled_*emails to send_email.py.
It more properly belongs as part of our outgoing email library, not
notifications.py (which is really about email notifications).
2019-03-15 11:02:17 -07:00
Raymond Akornor 89351cdd19 send_email: Add ScheduledEmail support for multiple recipients.
Follow up on 92dc363. This modifies the ScheduledEmail model
and send_future_email to properly support multiple recipients.

Tweaked by tabbott to add some useful explanatory comments and fix
issues with the migration.
2019-03-15 11:02:12 -07:00
Tim Abbott 4f02795830 home: Fix parsing bug for initial narrows.
There's more we'll want to do here, but this at least avoids error
emails when users attempt to misuse this interface.
2019-03-14 14:53:03 -07:00
Tim Abbott 983e24a7f5 auth: Use HTTP status 404 for invalid realms.
Apparently, our invalid realm error page had HTTP status 200, which
could be confusing and in particular broken our mobile app's error
handling for this case.
2019-03-14 13:50:09 -07:00
Puneeth Chaganti d75d2c9974 soft-deactivation: Run catch-up when "auto" deactivate is run.
When soft deactivation is run for in "auto" mode (no emails are
specified and all users inactive for specified number of days are
deactivated), catch-up is also run in the "auto" mode if
AUTO_CATCH_UP_SOFT_DEACTIVATED_USERS is True.

Automatically catching up soft-deactivated users periodically would
ensure a good user experience for returning users, but on some servers
we may want to turn off this option to save on some disk space.

Fixes #8858, at least for the default configuration, by eliminating
the situation where there are a very large number of messages to recover.
2019-03-14 11:53:15 -07:00
Puneeth Chaganti c61d3420e8 soft-deactivation: Extract getting users from emails to a function. 2019-03-14 20:57:00 +05:30
Puneeth Chaganti f5c61e7d31 soft-deactivation: Patch logging.info to clean up test output. 2019-03-14 09:26:25 +05:30
Puneeth Chaganti ff07ca7417 soft-deactivation: Slightly improve the test for catching up users. 2019-03-14 09:26:06 +05:30
Puneeth Chaganti cf65136002 soft-deactivation: Add code to catch up soft deactivated users. 2019-03-13 17:23:14 -07:00
Puneeth Chaganti 52afbe5e8d soft-deactivation: Rename maybe_catch_up_soft_deactivated_user.
Rename `maybe_catch_up_soft_deactivated_user` to
`reactivate_user_if_soft_deactivated`.
2019-03-13 17:16:22 -07:00
Puneeth Chaganti 7f06f6bd73 soft-deactivation: Raise exception with all missing emails at once. 2019-03-13 17:16:22 -07:00
Puneeth Chaganti 82d9789d93 soft-deactivation: Paginate bulk creation of UserMessage rows.
A user who has been soft deactivated for a long time might have 10Ks of message
history that was "soft deactivated". It might take a minute or more to add
UserMessage rows for all of these messages, causing timeouts. So, we paginate
the creation of these UserMessage rows.
2019-03-13 17:16:22 -07:00
Puneeth Chaganti 8c5a0f585b soft-deactivation: Clarify that value being fetched is recipient_id. 2019-03-13 17:16:22 -07:00
Puneeth Chaganti 4d77ffe2cb soft-deactivation: Refactor fetching list of existing UserMessages. 2019-03-13 17:16:22 -07:00
Puneeth Chaganti 8c5425e33a soft-deactivation: Remove unnecessary select_related calls.
It will become more clear in the upcoming commits, but these calls
aren't actually useful as we're only extracting a few IDs.
2019-03-13 17:15:50 -07:00
Harshit Bansal 262eb42b77 auth: Reverse the `sort_order` parameter's semantics.
This will make sure that if a backend doesn't specify a values for
`sort_order` parameter then it will sorted to the bottom not at the
top.
2019-03-13 14:44:57 -07:00
Tim Abbott 412d35900f slack import: Fix handling of tombstone files.
Apparently, the mode attribute is not always present.
2019-03-13 14:39:20 -07:00
Tim Abbott 76a06c8e54 send_test_email: Clean up output and provide advice.
Previously, while we sent emails using both noreply addresses, we
didn't make clear what was going on, leading to some potential
confusion.
2019-03-13 14:27:52 -07:00
Tim Abbott 662fa037f2 events: Don't assume that streams/subscriptions were requested.
This fixes a bug in our race condition handling in the case that
streams and/or subscriptions register content was not requested by the
client.
2019-03-13 14:12:59 -07:00
Tim Abbott 4694de31c5 tests: Fix linter error in test_management_commands. 2019-03-13 14:06:46 -07:00
Tim Abbott 49680a4503 slack import: Skip processing tombstone files.
The tombstone files undocumented feature of Slack's export format
appears sometimes and has no real data, so we just need to skip these.

Fixes #11619.
2019-03-13 12:43:11 -07:00
Hemanth V. Alluri 99c3e2ecdc management: Support sending custom headers when testing a webhook.
this commit adds an option to specify custom headers when using the
`./manage.py send_webhook_fixture_message` tool.
2019-03-13 12:40:47 -07:00
vsvipul e859ab7545 portico: Fix broken electron check condition for password reset.
This logic for passing through whether the user was logged in never
worked, because we were trying to read the client.

Fix this, and add tests to ensure it never breaks again.

Restructured by tabbott to have completely different code with the
same intent.

Fixes #11802.
2019-03-12 11:25:29 -07:00
Harshit Bansal a6e523f9e4 ldap: Ensure email is valid for realm before registering.
Previously, the LDAP authentication model ignored the realm-level
settings for who can join a realm.  This was sort of reasonable at the
time, because the original LDAP auth was an SSO solution that didn't
allow multiple realms, and so one could fully configure authentication
settings on the LDAP side.  But now that we allow multiple realms with
the LDAP backend, one could easily imagine wanting different
restrictions on them, and so it makes sense to add this enforcement.
2019-03-12 11:09:18 -07:00
Challa Venkata Raghava Reddy b69aec2dbc streams: Add first_message_id tracking first message in stream.
This field is primarily intended to support avoiding displaying the
"more topics" feature in new organizations and streams, where we might
know that all messages in the stream are already available in the
browser.

Based on original work by Roman Godov, and significantly modified by
tabbott.

The second migration involved here could be expensive on Zulip Cloud,
but is unlikely to be an issue on other servers.
2019-03-11 13:30:49 -07:00
Vishnu Ks 9671ed7bab subdomains: Extract get_subdomain_from_hostname function. 2019-03-11 12:01:11 -07:00
Rohitt Vashishtha 51986c27a5 markdown: Use common testcases for realmfilters in topics and messages.
The actual bug in #11791 was caused by code reverted in
3ed85f4cd7, so technically #11791 is
already fixed.  However, it makes sense to add tests to ensure that it
doesn't regress in the future as part of closing out the issue.

Fixes #11791.
2019-03-11 11:48:47 -07:00
Tim Abbott c05fb01cbf stream: Fix validator for stream colors.
Apparently, our new validator for stream color having a valid format
incorrectly handled colors that had duplicate characters in them.

(This is caused in part by the spectrum.js logic automatically
converting #ffff00 to #ff0, which our validator rejected).  Given that
we had old stream colors in the #ff0 format in our database anyway for
legacy, there's no benefit to banning these colors.

In the future, we could imagine standardizing the format, but doing so
will require also changing the frontend to submit colors only in the
6-character format.

Fixes an issue reported in
https://github.com/zulip/zulip/issues/11845#issuecomment-471417073
2019-03-11 11:43:39 -07:00
Eeshan Garg 222966b6ba webhooks/zapier: Support authentication for the Zapier Zulip app.
If the user sets up a Zap using an incoming webhook bot's API
key, the authentication goes through our webhook.
2019-03-11 10:54:55 -07:00
Eeshan Garg 7b6a37780a webhooks/github: Ignore organization and milestone events.
These events are not super useful and were cluttering up our
webhook logs.
2019-03-10 14:13:17 -07:00
Eeshan Garg 6afd02bef5 webhooks/github: Restrict membership event scope to teams.
According to GitHub's webhook docs, the scope of a membership
event can only be limited to 'teams', which holds true when a
new member is added to a team. However, we just found a payload
in our logs that indicates that when a user is removed from a
team, the scope of the membership is erroneously set to
'organization', not 'team'. This is most likely a bug on
GitHub's end because such behaviour is a direct violation of
their webhook API event specifications. We account for this
by restricting membership events to teams explicitly, at least
till GitHub's docs suggest otherwise.
2019-03-10 14:13:17 -07:00
Puneeth Chaganti abc9a7a4f9 digest: Use a list of Message objects to gather_hot_conversations.
Instead of a queryset of UserMessage objects, `gather_hot_conversations` now
accepts a list of Message objects.
2019-03-09 23:25:26 -08:00