Commit Graph

3384 Commits

Author SHA1 Message Date
Tom Daff f2e06128c6 email_mirror: Add email address parsing.
When trying to find the email gateway address, use the
`email.util.getaddresses` function to deal with cases
where multiple recipients are included in the email header
or the stream address appears as an angle-addr with a
name given (e.g. if someone added it to their address book).

Added some other headers where the required address may
appear: "Resent" headers are sometimes used for forwarding,
and streams may also be found in CC. There is no way to find
the address if the email was recieved as a BCC.
2019-01-03 13:34:20 -08:00
Steve Howell a2614956d5 status: Add away_user_ids to page_params.
(Also, any client that registers for 'user_status'
will see `away_user_ids`.)
2019-01-02 09:16:31 -08:00
Steve Howell 423db23c36 status: Add endpoint to toggle "away" status. 2019-01-02 09:16:30 -08:00
Steve Howell a8301ca14a status: Add UserStatus model and core library for away status. 2019-01-02 09:12:03 -08:00
Hemanth V. Alluri e3aed0f7bc custom profile fields: Markdown rendering for custom profile field values.
This makes it possible it include our standard markdown formatting in
one's custom profile fields, allowing for links, emphasis, emoji, etc.

Fixes #10131.
2019-01-01 21:06:21 -08:00
Vishnu Ks e5f7d65231 signup: Set oldest account as default option in import settings.
Fixes: #11018
2018-12-29 15:01:09 -08:00
Steve Howell c35afb942f Accept stream_id for muting endpoints. 2018-12-27 20:26:30 -08:00
Eeshan Garg 488f558d49 test_docs: Print better error messages for failed responses. 2018-12-27 20:13:08 -08:00
Eeshan Garg 8a02e177e3 bugdown: Trigger test failure for invalid Markdown include statements.
This commit adds a custom Markdown include extension which is
identical to the original except when a macro file can't
be found, it raises a custom JsonableError exception, which
we can catch and then trigger an appropriate test failure.

Fixes: #10947
2018-12-27 20:13:08 -08:00
Rishi Gupta e7220fd71f billing: Do subscription management in-house instead of with Stripe Billing.
This is a major rewrite of the billing system. It moves subscription
information off of stripe Subscriptions and into a local CustomerPlan
table.

To keep this manageable, it leaves several things unimplemented
(downgrading, etc), and a variety of other TODOs in the code. There are also
some known regressions, e.g. error-handling on /upgrade is broken.
2018-12-22 13:39:30 -08:00
Rishi Gupta bf8db6f634 emails: Update text for confirm_registration. 2018-12-20 16:26:19 -08:00
Rishi Gupta 0844c9a6ec emails: Update text for password_reset.
Note that a pretty common use case for this is a realm admin sending this to
everyone after an import from HipChat or Slack. So this adds the realm_name
to the title (so that there is something they might recognize) and kept the
wording generic enough to accommodate the user not having clicked anything
to get this email.

Also strengthens the tests a bit to better test the complicated template
logic.
2018-12-20 16:26:19 -08:00
Rishi Gupta 2da0cb7a44 emails: Update text of realm_reactivation. 2018-12-20 16:26:19 -08:00
Rishi Gupta fb56720054 emails: Update text for invitation and invitation_reminder. 2018-12-20 16:26:19 -08:00
Tim Abbott 9c3f38a564 docs: Automatically construct OpenAPI metadata for help center.
This is somewhat hacky, in that in order to do what we're doing, we
need to parse the HTML of the rendered page to extract the first
paragraph to include in the open graph description field.  But
BeautifulSoup does a good job of it.

This carries a nontrivial performance penalty for loading these pages,
but overall /help/ is a low-traffic site compared to the main app, so
it doesn't matter much.

(As a sidenote, it wouldn't be a bad idea to cache this stuff).

There's lots of things we can improve in this, largely through editing
the articles, but we can deal with that over time.

Thanks to Rishi for writing all the tests.
2018-12-19 10:18:20 -08:00
Tim Abbott 9d058f9193 test_auth_backends: Eliminate manual lists of authentication backends.
This should dramatically reduce the manual work involved with
correctly adding a new authentication backend to Zulip with this test
suite.
2018-12-18 16:36:45 -08:00
Joshua Pan ad1df0ebeb settings: Add support for customizing the top-left logo.
This adds a new realm_logo field, which is a horizontal-format logo to
be displayed in the top-left corner of the webapp, and any other
places where we might want a wide-format branding of the organization.

Tweaked significantly by tabbott to rebase, fix styling, etc.

Fixing the styling of this feature's loading indicator caused me to
notice the loading indicator for the realm_icon feature was also ugly,
so I fixed that too.

Fixes #7995.
2018-12-18 12:44:52 -08:00
Tim Abbott d469c14d11 docs: Advertise arbitrary org GitHub patterns for linkifiers.
Now that we support this, we should make it really obvious that we do.

Also, link to the Help Center article.
2018-12-17 12:58:52 -08:00
Tim Abbott c87a533b3b realm filters: Make validation error messages more useful. 2018-12-17 12:25:12 -08:00
Tim Abbott 34e39248fc linkifiers: Add validation support for multiple items.
This is a simple change to our validation, to allow multiple copies of
the main linkifier syntax, which lets us support things like generic
GitHub URLs.

Fixes #10914.
2018-12-17 12:19:59 -08:00
Tim Abbott b29d66f34b i18n: Rename email subject files so i18n detects them.
Apparently, when we renamed these files to no longer have a .txt
extension, we accidentally removed them from the set of strings for
translation, because `manage.py makemessages` by default only
processes .txt and .html files under the templates/ directory.

Fix this by adding a .txt extension.
2018-12-17 10:00:49 -08:00
Tim Abbott b2fc017671 i18n: Use the recipient's language when sending outgoing emails.
It appears that our i18n logic was only using the recipient's language
for logged-in emails, so even properly tagged for translation and
translated emails for functions like "Find my team" and "password
reset" were being always sent in English.

With great work by Vishnu Ks on the tests and the to_emails code path.
2018-12-17 09:49:36 -08:00
Tim Abbott e9f889d57e portico: Fix totally broken ToS re-accept form styling.
This form isn't actively used, which is how it ended up broken, but it
basically didn't display its content properly at all.

Convert it to use our standard white-box framework.

This still doesn't look great in various ways, but it's at least not
obviously totally busted now.
2018-12-17 07:28:29 -08:00
Tim Abbott 8a90441d2f slack import: Import long-inactive users as long-term idle.
This avoids creating UserMessage rows for long-inactive users in
organizations with many thousands of users.
2018-12-16 18:52:20 -08:00
Tim Abbott 142d9cb63f report: Allow error-reporting views from unauthed users.
This should make it possible for blueslip error reports to be sent on
our logged-out portico pages, which should in turn make it possible to
debug any such issues as they occur.
2018-12-16 15:44:48 -08:00
ishanrai05 4105fb683b notifications: Optimize push notifications code path in tests.
This checks if push_notification_enabled() is set to false in
handle_push_notification and adds an early return statement.

This is a significant performance optimization for our unit tests
because the push notifications code path does a number of database
queries, and this migration means we don't end up doing those queries
the hundreds of times we send PMs or mentions in our tests where we're
not trying to test the push notifications functionality.

This should also have a small message sending scalability improvement
for any Zulip servers without push notifications enabled.

Tweaked by tabbott to fix a few small issues.

Fixes #10895.
2018-12-15 11:12:43 -08:00
Tim Abbott a63eae48cc test_push_notifications: Fix leak that can leak to test flakes.
While reviewing #11012, I discovered a nondeterministic result for
test_signup, which I tracked down to specifically this triple of tests
failing when run in this order:

test-backend GCMSuccessTest \
  zerver.tests.test_push_notifications.TestAPNs.test_get_apns_client \
  zerver.tests.test_signup.LoginTest.test_register

with a query count mismatch like this:

expected length: 73
actual length: 79

Comparing the list of queries, it's clear that test_register was
seeing `push_notifications_enabled()` returning True in this test order.

It's not clear why GCMSuccessTest was required here (it was!), but
further debugging determined the problem was that
`test_get_apns_client` left the _apns_client initialization system in
a state where get_apns_client would return a non-None value, resulting
in push_notifications_enabled() returning True for future tests.

The immediate fix is to just reset the `_apns_client` and
`_apns_client_initializedstate` state properly after the test runs;
but arguably we should do a larger refactor to make this less
fragile.
2018-12-15 11:12:43 -08:00
Rishi Gupta 111eda604b portico: Add /atlassian for Zulip users migrating from HipChat. 2018-12-14 23:42:47 -08:00
Tim Abbott 8c30c36006 realm filters: Expand set of characters allowed in prefixes.
Our list of allowed characters in realm filter patterns has long been
too string; fix this by extending the pattern.

Also, extend the tests to have examples of actual strings one would
use with the patterns, for clarity.

Fixes #10953, fixes #6835.
2018-12-14 11:33:13 -08:00
Tim Abbott 626e191201 ldap: Add support for automatic user deactivation/reactivation.
As part of this, extend our documentation on synchronizing data from
Active Directory.
2018-12-13 16:24:15 -08:00
Tim Abbott 0a5221a819 ldap: Extract dev_ldap_directory.py.
This gets what is fundamentally unit testing code out of backends.py.
2018-12-13 16:24:15 -08:00
Tim Abbott 5dd646f33f ldap: Add support for syncing avatar images from LDAP.
This should make life a lot more convenient for organizations that use
the LDAP integration and have their avatars in LDAP already.

This hasn't been end-to-end tested against LDAP yet, so there may be
some minor revisions, but fundamentally, it works, has automated
tests, and should be easy to maintain.

Fixes #286.
2018-12-13 13:39:22 -08:00
Rishi Gupta 8a95526ced billing: Always transition to Realm.LIMITED via do_change_plan_type.
Fixes a bug in import_realm where secondary attributes like message
visibility weren't being set, and also makes bugs like this less likely in
the future.

Also, putting the plan_type change at the end of import_realm, so that
future restrictions to LIMITED realms don't affect the import process.
2018-12-13 13:26:24 -08:00
Rishi Gupta b245c661da billing: Change do_change_plan_type to take a realm instead of a user.
More often than not, changes in plan type are not directly due to user
action.
2018-12-13 13:26:24 -08:00
Vishnu Ks 8a1794caa3 message: Store the value of first_visible_message_id in Realm table.
This eliminates a bunch of potentially buggy caching code, with no
material negative side effects.
2018-12-12 15:11:17 -08:00
Tim Abbott ee901ac8b1 django: Remove the Django contrib Sites app to fix 2FA QR codes.
Apparently, Django's get_current_site function (used, e.g., in
django-two-factor to look up the domain to use in QR codes) first
tries to use the Sites framework, and if unavailable, does the right
thing (namely, using request.get_host()).

We don't use the Sites framework for anything in Zulip, so the correct
fix is to just remove it.

Fixes #11014.
2018-12-12 15:01:55 -08:00
Tim Abbott 8a11c94a2d fakeldap: Add thumbnailPhoto/jpegPhoto for testing avatar syncing. 2018-12-12 11:23:33 -08:00
Roman Godov 9c8eeaed85 digest_email: Add endpoint for rendering digest to the web.
Adds "/digest/" endpoint for rendering content of digest email
to the web.

Fixes #9974
2018-12-11 13:38:30 -08:00
Tim Abbott 15d4b71e2e decorator: Skip rate limiting when accessing user uploads.
The code paths for accessing user-uploaded files are both (A) highly
optimized so as to not require a ton of work, and (B) a code path
where it's totally reasonable for a client to need to fetch 100+
images all at once (e.g. if it's the first browser open in a setting
with a lot of distinct senders with avatars or a lot of image
previews).

Additionally, we've been seeing exceptions logged in the production
redis configuration caused by this code path (basically, locking
failures trying to update the rate-limit data structures).

So we skip running our current rate limiting algorithm for these views.
2018-12-11 12:43:21 -08:00
Max Nussenbaum 536de666df register: Style avatar that shows when importing settings.
This styles the avatar and username that show when the registering
user is importing their settings from an existing Zulip account.

Tweaked by tabbott to fix the test/linter failures, a bit of styling,
and tag strings for translation.
2018-12-10 15:40:30 -08:00
Jack Zhang be9b6a6dee compatibility: Add a compatibility check to api_get_server_settings.
This should make it convenient for the mobile app to present errors of
the form "Your Zulip app is not new enough for this Zulip server".
2018-12-10 15:06:09 -08:00
Jack Zhang 8711fe7bae compatibility: Respond with HTTP 400 when no User-Agent header.
Prior to this commit, a request to /compatibility without a User-Agent
header would result in the server 500ing.
2018-12-10 15:04:34 -08:00
Tim Abbott a1d99adce7 auth: Handle SSO_APPEND_DOMAIN in remote_user SSO for mobile.
Apparently, while the main code path through
login_or_register_remote_user was correctly calling
remote_user_to_email(username) to get a proper email address for
situations where auth username != email (i.e. when SSO_APPEND_DOMAIN
is set), we neglected to do so in the mobile_flow_otp corner case.

Fixes #11005.
2018-12-10 10:36:15 -08:00
Marco Burstein ba46dc83c6 notifications: Add a setting for changing the notification sound.
Also, add a new notification sound, "ding". It comes from
https://freesound.org, where the original Zulip notification sound comes
from as well. In the future, new sounds can be added by adding audio
files to the `static/audio/notification_sounds` directory.

Tweaked significantly by tabbott:
* Avoided removing static/audio/zulip.ogg, because that file is
  checked for by old versions of the desktop app.
* Added a views check for the sound being valid + tests.
* Added additional tests.
* Restructured the test_events test to be cleaner.
* Removed check_bool_or_string.
* Increased max length of notification_sound.
* Provide available_notification_sounds in events data set if global
  notifications settings are requested.

Fixes #8051.
2018-12-09 21:25:30 -08:00
rht a1ff44a230 refactor: Add a helper function to create s3 buckets.
This refactor makes upgrading boto to boto3 easier.
Based on 43d2f6286c
2018-12-07 13:58:11 -08:00
Joshua Pan 441dca0eaf email: Support delivery email in email change system. 2018-12-06 17:43:26 -08:00
Tim Abbott 5c2e7a169f test_events: Use a better assertion when comparing event count. 2018-12-06 17:43:26 -08:00
Tim Abbott e603237010 email: Convert accounts code to use delivery_email.
A key part of this is the new helper, get_user_by_delivery_email.  Its
verbose name is important for clarity; it should help avoid blind
copy-pasting of get_user (which we'll also want to rename).
Unfortunately, it requires detailed understanding of the context to
figure out which one to use; each is used in about half of call sites.

Another important note is that this PR doesn't migrate get_user calls
in the tests except where not doing so would cause the tests to fail.
This probably deserves a follow-up refactor to avoid bugs here.
2018-12-06 16:21:38 -08:00
Tim Abbott 9d32857c37 realm: Add new Realm.email_address_visibility field.
This is preparatory work for settings controlling who can see user
emails; it includes the API-level support for editing it, but no code
to actually enforce the policy.
2018-12-06 15:56:30 -08:00
Tim Abbott 9da4b58c8e docs: Fix /help/ sidebar being accidentally missing.
Apparently, we didn't have a test for this, and it was broken in a
recent refactoring.
2018-12-06 10:13:51 -08:00