Commit Graph

3550 Commits

Author SHA1 Message Date
Eeshan Garg 020f2d0db5 send_message_backend: Add support for recipient user IDs.
Note that create_mirrored_message_users has not been updated to
use user IDs.

Tweaked by tabbott to add a unit test for the huddle case as well.
2019-01-08 16:17:20 -08:00
Eeshan Garg 82ace8ccc8 addressee: Add support for user IDs to Addressee.legacy_build().
This commit is a part of our efforts surrounding #9474.
2019-01-08 19:08:49 -03:30
Eeshan Garg 2f634f8c06 backend: Support user IDs for sending typing notifications.
This is a part of our efforts surrounding #9474.
2019-01-08 19:08:49 -03:30
Eeshan Garg 8f1dba6aad extract_recipients: Support user IDs.
This is a part of our efforts surrounding #9474.
2019-01-08 19:08:49 -03:30
sahil839 4dcccf32f8 zerver: Add feature for notification at rename of a stream.
Feature of sending notification to the stream using notification bot
is added. user_profile is also passed to do_rename_stream for using
the name of user who renamed the stream in notification.
Notification is sent to the stream using
internal_send_stream_message in do_rename_stream.

Fixes #11034.
2019-01-07 19:13:01 -08:00
Marco Burstein 9ddadd39f4 compose: Add support for using Zoom as the video chat provider.
This adds Zoom call properties to the `Realm` model, creates endpoints
for creating calls, adds a frontend and tests.

Fixes #10979.
2019-01-07 10:00:02 -08:00
Mateusz Mandera 87c95c6f41 email_mirror: Strip RE and FWD from email subject.
Fixes part 1 of #10612. We use a regex to remove RE:, FWD: (and similar
variations) from email subjects. Unit test is included, we add
subjects.json in fixtures containing various subjects to try the
stripping on.
2019-01-05 15:59:19 -08:00
Shubham Dhama efb9128aaa invite: Add option to invite user as guest.
This completes our basic guest user feature.

Fixes: #10818.
2019-01-05 14:52:51 -08:00
Shubham Dhama 42c262b807 invite: Replace `invite_as_admin` usage with `invite_as`.
Since we have already added the `invite_as` field to models, we can now
replace usage of `invite_as_admin` properly with its equivalent `invite_as
== PreregistrationUser.INVITE_AS['REALM_ADMIN']`.

Hence, also removed now redundant `invite_as`.
2019-01-05 14:46:38 -08:00
Tim Abbott 2adb0734d1 decorator: Set an error code for invalid API key errors.
This should make it easily for mobile/terminal apps to handle
situations like the user's API key changing.

Also fix the fact we were incorrectly using a 400, not 401, status
code for this case.
2019-01-05 11:29:34 -08:00
Tim Abbott 2ba24bdd09 settings: Fix flushing of API key cache.
The logic for flushing the API key has been broken every since we
added the cache, since we were incorrectly flushing the new API key,
not the old API key, from the cache after regeneration.
2019-01-04 17:26:10 -08:00
Tim Abbott bd5745ab35 test_settings: Extend caching section of API key test. 2019-01-04 17:26:05 -08:00
Rishi Gupta c1d40e22a7 emails: Update text for confirm_new_email. 2019-01-04 14:32:21 -08:00
Aditya Bansal 079dfadf1a camo: Add endpoint to handle camo requests.
This endpoint serves requests which might originate from an image
preview link which had an http url and the message holding the image
link was rendered before we introduced thumbnailing. In that case
we would have used a camo proxy to proxy http content over https and
avoid mix content warnings.

In near future, we plan to drop use of camo and just rely on thumbor
to serve such images. This endpoint helps maintain backward
compatibility for links which were already rendered.
2019-01-04 10:27:04 -08:00
Aditya Bansal 3ee69f3da9 thumbnails: Add setting THUMBNAIL_IMAGES.
This setting splits away part of responsibility from THUMBOR_URL.
Now on, this setting will be responsible for controlling whether
we thumbnail images or not by asking bugdown to render image links
to hit our /thumbnail endpoint. This is irrespective of what
THUMBOR_URL is set to though ideally THUMBOR_URL should be set
to point to a running thumbor instance.
2019-01-04 10:27:04 -08:00
Aditya Bansal 26c6ef1834 thumbnails: Fix bug with use of filters in thumbnail generation.
We used to add sharpen filter for all the image sizes whereas it was
intended for resized images only which would have been smoothened
out a bit by the resize operation.
This unnecessary use of the filter used to result in weird issues
with full size images.
For example: Image located at this url:-
http://arqex.com/wp-content/uploads/2015/02/trees.png
When rendered in full size would have just boundaries visible.
2019-01-04 19:06:01 +05:30
Tom Daff f2e06128c6 email_mirror: Add email address parsing.
When trying to find the email gateway address, use the
`email.util.getaddresses` function to deal with cases
where multiple recipients are included in the email header
or the stream address appears as an angle-addr with a
name given (e.g. if someone added it to their address book).

Added some other headers where the required address may
appear: "Resent" headers are sometimes used for forwarding,
and streams may also be found in CC. There is no way to find
the address if the email was recieved as a BCC.
2019-01-03 13:34:20 -08:00
Steve Howell a2614956d5 status: Add away_user_ids to page_params.
(Also, any client that registers for 'user_status'
will see `away_user_ids`.)
2019-01-02 09:16:31 -08:00
Steve Howell 423db23c36 status: Add endpoint to toggle "away" status. 2019-01-02 09:16:30 -08:00
Steve Howell a8301ca14a status: Add UserStatus model and core library for away status. 2019-01-02 09:12:03 -08:00
Hemanth V. Alluri e3aed0f7bc custom profile fields: Markdown rendering for custom profile field values.
This makes it possible it include our standard markdown formatting in
one's custom profile fields, allowing for links, emphasis, emoji, etc.

Fixes #10131.
2019-01-01 21:06:21 -08:00
Vishnu Ks e5f7d65231 signup: Set oldest account as default option in import settings.
Fixes: #11018
2018-12-29 15:01:09 -08:00
Steve Howell c35afb942f Accept stream_id for muting endpoints. 2018-12-27 20:26:30 -08:00
Eeshan Garg 488f558d49 test_docs: Print better error messages for failed responses. 2018-12-27 20:13:08 -08:00
Eeshan Garg 8a02e177e3 bugdown: Trigger test failure for invalid Markdown include statements.
This commit adds a custom Markdown include extension which is
identical to the original except when a macro file can't
be found, it raises a custom JsonableError exception, which
we can catch and then trigger an appropriate test failure.

Fixes: #10947
2018-12-27 20:13:08 -08:00
Rishi Gupta e7220fd71f billing: Do subscription management in-house instead of with Stripe Billing.
This is a major rewrite of the billing system. It moves subscription
information off of stripe Subscriptions and into a local CustomerPlan
table.

To keep this manageable, it leaves several things unimplemented
(downgrading, etc), and a variety of other TODOs in the code. There are also
some known regressions, e.g. error-handling on /upgrade is broken.
2018-12-22 13:39:30 -08:00
Rishi Gupta bf8db6f634 emails: Update text for confirm_registration. 2018-12-20 16:26:19 -08:00
Rishi Gupta 0844c9a6ec emails: Update text for password_reset.
Note that a pretty common use case for this is a realm admin sending this to
everyone after an import from HipChat or Slack. So this adds the realm_name
to the title (so that there is something they might recognize) and kept the
wording generic enough to accommodate the user not having clicked anything
to get this email.

Also strengthens the tests a bit to better test the complicated template
logic.
2018-12-20 16:26:19 -08:00
Rishi Gupta 2da0cb7a44 emails: Update text of realm_reactivation. 2018-12-20 16:26:19 -08:00
Rishi Gupta fb56720054 emails: Update text for invitation and invitation_reminder. 2018-12-20 16:26:19 -08:00
Tim Abbott 9c3f38a564 docs: Automatically construct OpenAPI metadata for help center.
This is somewhat hacky, in that in order to do what we're doing, we
need to parse the HTML of the rendered page to extract the first
paragraph to include in the open graph description field.  But
BeautifulSoup does a good job of it.

This carries a nontrivial performance penalty for loading these pages,
but overall /help/ is a low-traffic site compared to the main app, so
it doesn't matter much.

(As a sidenote, it wouldn't be a bad idea to cache this stuff).

There's lots of things we can improve in this, largely through editing
the articles, but we can deal with that over time.

Thanks to Rishi for writing all the tests.
2018-12-19 10:18:20 -08:00
Tim Abbott 9d058f9193 test_auth_backends: Eliminate manual lists of authentication backends.
This should dramatically reduce the manual work involved with
correctly adding a new authentication backend to Zulip with this test
suite.
2018-12-18 16:36:45 -08:00
Joshua Pan ad1df0ebeb settings: Add support for customizing the top-left logo.
This adds a new realm_logo field, which is a horizontal-format logo to
be displayed in the top-left corner of the webapp, and any other
places where we might want a wide-format branding of the organization.

Tweaked significantly by tabbott to rebase, fix styling, etc.

Fixing the styling of this feature's loading indicator caused me to
notice the loading indicator for the realm_icon feature was also ugly,
so I fixed that too.

Fixes #7995.
2018-12-18 12:44:52 -08:00
Tim Abbott d469c14d11 docs: Advertise arbitrary org GitHub patterns for linkifiers.
Now that we support this, we should make it really obvious that we do.

Also, link to the Help Center article.
2018-12-17 12:58:52 -08:00
Tim Abbott c87a533b3b realm filters: Make validation error messages more useful. 2018-12-17 12:25:12 -08:00
Tim Abbott 34e39248fc linkifiers: Add validation support for multiple items.
This is a simple change to our validation, to allow multiple copies of
the main linkifier syntax, which lets us support things like generic
GitHub URLs.

Fixes #10914.
2018-12-17 12:19:59 -08:00
Tim Abbott b29d66f34b i18n: Rename email subject files so i18n detects them.
Apparently, when we renamed these files to no longer have a .txt
extension, we accidentally removed them from the set of strings for
translation, because `manage.py makemessages` by default only
processes .txt and .html files under the templates/ directory.

Fix this by adding a .txt extension.
2018-12-17 10:00:49 -08:00
Tim Abbott b2fc017671 i18n: Use the recipient's language when sending outgoing emails.
It appears that our i18n logic was only using the recipient's language
for logged-in emails, so even properly tagged for translation and
translated emails for functions like "Find my team" and "password
reset" were being always sent in English.

With great work by Vishnu Ks on the tests and the to_emails code path.
2018-12-17 09:49:36 -08:00
Tim Abbott e9f889d57e portico: Fix totally broken ToS re-accept form styling.
This form isn't actively used, which is how it ended up broken, but it
basically didn't display its content properly at all.

Convert it to use our standard white-box framework.

This still doesn't look great in various ways, but it's at least not
obviously totally busted now.
2018-12-17 07:28:29 -08:00
Tim Abbott 8a90441d2f slack import: Import long-inactive users as long-term idle.
This avoids creating UserMessage rows for long-inactive users in
organizations with many thousands of users.
2018-12-16 18:52:20 -08:00
Tim Abbott 142d9cb63f report: Allow error-reporting views from unauthed users.
This should make it possible for blueslip error reports to be sent on
our logged-out portico pages, which should in turn make it possible to
debug any such issues as they occur.
2018-12-16 15:44:48 -08:00
ishanrai05 4105fb683b notifications: Optimize push notifications code path in tests.
This checks if push_notification_enabled() is set to false in
handle_push_notification and adds an early return statement.

This is a significant performance optimization for our unit tests
because the push notifications code path does a number of database
queries, and this migration means we don't end up doing those queries
the hundreds of times we send PMs or mentions in our tests where we're
not trying to test the push notifications functionality.

This should also have a small message sending scalability improvement
for any Zulip servers without push notifications enabled.

Tweaked by tabbott to fix a few small issues.

Fixes #10895.
2018-12-15 11:12:43 -08:00
Tim Abbott a63eae48cc test_push_notifications: Fix leak that can leak to test flakes.
While reviewing #11012, I discovered a nondeterministic result for
test_signup, which I tracked down to specifically this triple of tests
failing when run in this order:

test-backend GCMSuccessTest \
  zerver.tests.test_push_notifications.TestAPNs.test_get_apns_client \
  zerver.tests.test_signup.LoginTest.test_register

with a query count mismatch like this:

expected length: 73
actual length: 79

Comparing the list of queries, it's clear that test_register was
seeing `push_notifications_enabled()` returning True in this test order.

It's not clear why GCMSuccessTest was required here (it was!), but
further debugging determined the problem was that
`test_get_apns_client` left the _apns_client initialization system in
a state where get_apns_client would return a non-None value, resulting
in push_notifications_enabled() returning True for future tests.

The immediate fix is to just reset the `_apns_client` and
`_apns_client_initializedstate` state properly after the test runs;
but arguably we should do a larger refactor to make this less
fragile.
2018-12-15 11:12:43 -08:00
Rishi Gupta 111eda604b portico: Add /atlassian for Zulip users migrating from HipChat. 2018-12-14 23:42:47 -08:00
Tim Abbott 8c30c36006 realm filters: Expand set of characters allowed in prefixes.
Our list of allowed characters in realm filter patterns has long been
too string; fix this by extending the pattern.

Also, extend the tests to have examples of actual strings one would
use with the patterns, for clarity.

Fixes #10953, fixes #6835.
2018-12-14 11:33:13 -08:00
Tim Abbott 626e191201 ldap: Add support for automatic user deactivation/reactivation.
As part of this, extend our documentation on synchronizing data from
Active Directory.
2018-12-13 16:24:15 -08:00
Tim Abbott 0a5221a819 ldap: Extract dev_ldap_directory.py.
This gets what is fundamentally unit testing code out of backends.py.
2018-12-13 16:24:15 -08:00
Tim Abbott 5dd646f33f ldap: Add support for syncing avatar images from LDAP.
This should make life a lot more convenient for organizations that use
the LDAP integration and have their avatars in LDAP already.

This hasn't been end-to-end tested against LDAP yet, so there may be
some minor revisions, but fundamentally, it works, has automated
tests, and should be easy to maintain.

Fixes #286.
2018-12-13 13:39:22 -08:00
Rishi Gupta 8a95526ced billing: Always transition to Realm.LIMITED via do_change_plan_type.
Fixes a bug in import_realm where secondary attributes like message
visibility weren't being set, and also makes bugs like this less likely in
the future.

Also, putting the plan_type change at the end of import_realm, so that
future restrictions to LIMITED realms don't affect the import process.
2018-12-13 13:26:24 -08:00
Rishi Gupta b245c661da billing: Change do_change_plan_type to take a realm instead of a user.
More often than not, changes in plan type are not directly due to user
action.
2018-12-13 13:26:24 -08:00
Vishnu Ks 8a1794caa3 message: Store the value of first_visible_message_id in Realm table.
This eliminates a bunch of potentially buggy caching code, with no
material negative side effects.
2018-12-12 15:11:17 -08:00
Tim Abbott ee901ac8b1 django: Remove the Django contrib Sites app to fix 2FA QR codes.
Apparently, Django's get_current_site function (used, e.g., in
django-two-factor to look up the domain to use in QR codes) first
tries to use the Sites framework, and if unavailable, does the right
thing (namely, using request.get_host()).

We don't use the Sites framework for anything in Zulip, so the correct
fix is to just remove it.

Fixes #11014.
2018-12-12 15:01:55 -08:00
Tim Abbott 8a11c94a2d fakeldap: Add thumbnailPhoto/jpegPhoto for testing avatar syncing. 2018-12-12 11:23:33 -08:00
Roman Godov 9c8eeaed85 digest_email: Add endpoint for rendering digest to the web.
Adds "/digest/" endpoint for rendering content of digest email
to the web.

Fixes #9974
2018-12-11 13:38:30 -08:00
Tim Abbott 15d4b71e2e decorator: Skip rate limiting when accessing user uploads.
The code paths for accessing user-uploaded files are both (A) highly
optimized so as to not require a ton of work, and (B) a code path
where it's totally reasonable for a client to need to fetch 100+
images all at once (e.g. if it's the first browser open in a setting
with a lot of distinct senders with avatars or a lot of image
previews).

Additionally, we've been seeing exceptions logged in the production
redis configuration caused by this code path (basically, locking
failures trying to update the rate-limit data structures).

So we skip running our current rate limiting algorithm for these views.
2018-12-11 12:43:21 -08:00
Max Nussenbaum 536de666df register: Style avatar that shows when importing settings.
This styles the avatar and username that show when the registering
user is importing their settings from an existing Zulip account.

Tweaked by tabbott to fix the test/linter failures, a bit of styling,
and tag strings for translation.
2018-12-10 15:40:30 -08:00
Jack Zhang be9b6a6dee compatibility: Add a compatibility check to api_get_server_settings.
This should make it convenient for the mobile app to present errors of
the form "Your Zulip app is not new enough for this Zulip server".
2018-12-10 15:06:09 -08:00
Jack Zhang 8711fe7bae compatibility: Respond with HTTP 400 when no User-Agent header.
Prior to this commit, a request to /compatibility without a User-Agent
header would result in the server 500ing.
2018-12-10 15:04:34 -08:00
Tim Abbott a1d99adce7 auth: Handle SSO_APPEND_DOMAIN in remote_user SSO for mobile.
Apparently, while the main code path through
login_or_register_remote_user was correctly calling
remote_user_to_email(username) to get a proper email address for
situations where auth username != email (i.e. when SSO_APPEND_DOMAIN
is set), we neglected to do so in the mobile_flow_otp corner case.

Fixes #11005.
2018-12-10 10:36:15 -08:00
Marco Burstein ba46dc83c6 notifications: Add a setting for changing the notification sound.
Also, add a new notification sound, "ding". It comes from
https://freesound.org, where the original Zulip notification sound comes
from as well. In the future, new sounds can be added by adding audio
files to the `static/audio/notification_sounds` directory.

Tweaked significantly by tabbott:
* Avoided removing static/audio/zulip.ogg, because that file is
  checked for by old versions of the desktop app.
* Added a views check for the sound being valid + tests.
* Added additional tests.
* Restructured the test_events test to be cleaner.
* Removed check_bool_or_string.
* Increased max length of notification_sound.
* Provide available_notification_sounds in events data set if global
  notifications settings are requested.

Fixes #8051.
2018-12-09 21:25:30 -08:00
rht a1ff44a230 refactor: Add a helper function to create s3 buckets.
This refactor makes upgrading boto to boto3 easier.
Based on 43d2f6286c
2018-12-07 13:58:11 -08:00
Joshua Pan 441dca0eaf email: Support delivery email in email change system. 2018-12-06 17:43:26 -08:00
Tim Abbott 5c2e7a169f test_events: Use a better assertion when comparing event count. 2018-12-06 17:43:26 -08:00
Tim Abbott e603237010 email: Convert accounts code to use delivery_email.
A key part of this is the new helper, get_user_by_delivery_email.  Its
verbose name is important for clarity; it should help avoid blind
copy-pasting of get_user (which we'll also want to rename).
Unfortunately, it requires detailed understanding of the context to
figure out which one to use; each is used in about half of call sites.

Another important note is that this PR doesn't migrate get_user calls
in the tests except where not doing so would cause the tests to fail.
This probably deserves a follow-up refactor to avoid bugs here.
2018-12-06 16:21:38 -08:00
Tim Abbott 9d32857c37 realm: Add new Realm.email_address_visibility field.
This is preparatory work for settings controlling who can see user
emails; it includes the API-level support for editing it, but no code
to actually enforce the policy.
2018-12-06 15:56:30 -08:00
Tim Abbott 9da4b58c8e docs: Fix /help/ sidebar being accidentally missing.
Apparently, we didn't have a test for this, and it was broken in a
recent refactoring.
2018-12-06 10:13:51 -08:00
Vishnu Ks 18f90754da portico: Change wording of finding organization in /accounts/go. 2018-12-06 09:24:22 -08:00
Tim Abbott 9e06ab45bf webhooks: Fix HttpResponse with notify_bot_owner_on_invalid_json.
Apparently, there was a bug in notify_bot_owner_on_invalid_json, where
we didn't reraise the JsonableError.

We fix this with a refactoring that makes the exception layering
clearer as well.
2018-12-05 15:17:31 -08:00
Greg Price 32c6b81b3a compatibility: Add test data from real server logs.
In a quick scan of today's nginx logs on chat.zulip.org, there
were 20 distinct user-agents that begin with 'ZulipMobile/'.
Here's a representative sampling of them, such that the rest
were all boringly similar to one of these.

First, to make room for these without an excess of copy-paste and
overlong lines, convert this test to a data-oriented style.  The
existing, synthetic cases appear in the new data followed by the
seen-in-the-wild cases.

Happily, the code being tested passes all these new cases unchanged.
2018-12-05 14:58:13 -08:00
Greg Price fb7bfbe9ab compatibility: Require version 16.2.96+ for our Android app.
This release is from 2018-08-22, a little over 100 days ago.

It was the first release with the important fix so that when the
server advises it to stop displaying a notification because the user
has read the message (as the SEND_REMOVE_PUSH_NOTIFICATIONS server
setting enables), the app doesn't instead replace the notification
with a broken one reading "null".  We have that setting running now
on chat.zulip.org, and intend to roll it out more broadly soon.

The `# take 0` thing is a slightly absurd workaround for the fact
that our funky out-of-line way of marking lines to ignore doesn't
work right if there are multiple such lines in a given file that
are equal modulo leading and trailing whitespace.
2018-12-05 14:58:13 -08:00
Greg Price 0fa4fdcff9 compatibility: Rewrite verbose test for compact legibility.
This will facilitate adding a bunch more test cases shortly.
2018-12-05 14:58:13 -08:00
Greg Price 2bac6de6c6 compatibility: Add and test a helper find_mobile_os. 2018-12-05 14:58:13 -08:00
Greg Price 1859faccb7 compatibility: Treat empty version as unknown, not old. 2018-12-05 14:58:13 -08:00
Greg Price ebfa4b746d compatibility: Make 1.2 < 1.2.3; rewrite version_lt internals. 2018-12-05 14:58:13 -08:00
Greg Price 557aca2aa7 compatibility: Implement a version comparator. 2018-12-05 14:58:13 -08:00
rht e59ff6e6db slack import: Eliminate need to load all messages into memory.
This works by yielding messages sorted based on timestamp.  Because
the Slack exports are broken into files by date, it's convenient to do
a 2-layer sorting process, where we open all the files for a given
day, and then sort their messages by timestamp before yielding them.

Fixes #10930.
2018-12-05 12:20:50 -08:00
Tim Abbott 380231af9d push_notifications: Add tests for BrokenPipeError case.
This was missing in d723dbfef7.
2018-12-05 10:44:25 -08:00
Tim Abbott b47535d8bb push notifications: Fix exception when handling deleted messages.
If a user deletes message between when it triggered a potential push
notification for another user, and when that notification was actually
sent, we'd end up with a situation where the `Message` table didn't
have an entry for the requested message ID.

The clean fix for this is to still throw an exception in the event
that the message doesn't exist at all, but if it exists in
ArchivedMessage, don't throw a user-facing exception.
2018-12-05 10:38:37 -08:00
Tim Abbott 196ec78261 logging_handlers: Fix unnecessary broken import. 2018-12-04 18:18:07 -08:00
Anders Kaseorg 9ba860b4f2 Avoid double redirects to /login and then to /login/.
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2018-12-04 16:12:58 -08:00
Vishnu Ks fc4b6d8f5e auth: Redirect login to realm_redirect page instead of find_accounts. 2018-12-04 09:35:35 -08:00
Vishnu Ks 788b98d041 portico: Add page for redirecting to a realm subdomain. 2018-12-04 09:35:35 -08:00
Raymond Akornor 92dc3637df send_email: Add support for multiple recipients.
This adds a function that sends provided email to all administrators
of a realm, but in a single email. As a result, send_email now takes
arguments to_user_ids and to_emails instead of to_user_id and
to_email.

We adjust other APIs to match, but note that send_future_email does
not yet support the multiple recipients model for good reasons.

Tweaked by tabbott to modify `manage.py deliver_email` to handle
backwards-compatibily for any ScheduledEmail objects already in the
database.

Fixes #10896.
2018-12-03 15:12:11 -08:00
Tim Abbott 0fddf9a610 docs: Fix 500s with super long fake /help/ article URLs.
While this would never happen for a real article, this prevents a 500
in this case for a situation which is definitely user error and should
be a 40x (in this case, 404).

As part of this, we refactor the main view code to do validation in a
single code path, since the semi-duplicated-in-3-places logic was
getting pretty buggy.
2018-12-03 14:43:45 -08:00
Vishnu Ks f3033207b9 emails: Cover all LDAP cases when emailing login details.
This provides a nice user experience for folks where we do know what
their LDAP credentials are.

Though we need to fix #10917 before the content in the email with be
correct.
2018-11-29 12:14:22 -08:00
Rohitt Vashishtha c4e50a34d3 bugdown: Refactor get_user to get_user_by_name.
Also adds a warning against the use of this function.
2018-11-29 10:19:08 -08:00
Hemanth V. Alluri 6b1d724f5c zerver: Add bugdown rendering for text custom profile fields.
This is the first step of letting users use Zulip markdown in their
SHORT_TEXT and LONG_TEXT custom profile fields, so that they can
include emphasis, links, etc.

This doesn't include any frontend logic yet, however.
2018-11-29 09:47:32 -08:00
Rohitt Vashishtha 681368b937 bugdown: Refactor get_possible_mentions_info and related functions.
This commit changes the return type of get_possible_mentions_info  to a
list instead of a dict, thus disposing off the hacky logic of storing
users with duplicate full names with name|id keys that made the code
obfuscated.

The other functions continue to use the dicts as before, however, there
are minor variable changes where needed in accordance with the updated
definition of get_possible_mentions_info.
2018-11-28 14:07:52 -08:00
Rohitt Vashishtha ccdf893af7 bugdown: Rename get_full_name_info to get_possible_mentions_info. 2018-11-28 14:04:50 -08:00
Rishi Gupta dd0126ff1b settings: Improve error message when deactivating the last user.
This PR was originally started by Rishi Gupta (see #10383).
2018-11-27 12:49:43 -08:00
Eeshan Garg 7fb674cc58 messages: Add Addressee.for_user_ids().
This commit is a part of our efforts surrounding #9474.
2018-11-27 12:09:41 -08:00
Eeshan Garg 91b90460ee lib/actions: Add helper recipient_for_user_ids().
This function is equivalent to recipient_for_emails, but fetches
user_profiles by IDs, not by emails.

This commit is a part of our efforts surrounding #9474, but is
more primarily geared towards adding support for sending typing
notifications by user IDs.
2018-11-27 11:59:11 -08:00
Tim Abbott 6a9e6503b7 python: Remove unnecessary NonBinaryStr TypeVar.
This was a potentially useful value (later superceded by use of Text)
when first introduced, but now that we're on Python 3, it should just
be str.
2018-11-27 11:44:09 -08:00
Raymond Akornor d00b889402 auth: Add an organization reactivation flow with admin confirmation.
This adds a web flow and management command for reactivating a Zulip
organization, with confirmation from one of the organization
administrators.

Further work is needed to make the emails nicer (ideally, we'd send
one email with all the admins on the `To` line, but the `send_email`
library doesn't support that).

Fixes #10783.

With significant tweaks to the email text by tabbott.
2018-11-27 10:58:34 -08:00
Tim Abbott 7b930124d9 push notifications: Add a logger (default-off in tests).
This should suppress some spammy logging output about push
notifications that we were seeing in a large number of unit tests.
2018-11-27 09:45:45 -08:00
Tim Abbott 38a6003472 push notifications: Improve logging for missing configuration.
While it could make sense to print these logging statements at WARN
level on server startup, it doesn't make sense to do so on every
message (though it perhaps did make sense to do so before more recent
changes added good ways to discover you forgot to configure push
notifications).

Instead, we now just do a WARN log on queue processor startup, and
then at DEBUG level for individual messages.

Fixes #10894.
2018-11-27 09:37:57 -08:00
Steve Howell c2e9f5eb0a hipchat: Limit messages in slim mode.
For messages with strange senders, we don't import
messages.  Basically, we only import a message if
it has sender with an id that maps to a non-deleted
user.
2018-11-26 16:37:30 -08:00
Marco Burstein 6f569719c9 integrations: Change the truncation marker for long messages.
Change the truncation marker from `...` to `\n[message truncated]`
when receiving messages from the API or through e-mail. Also, update
tests to account for the new change.

Fix #10871.
2018-11-26 11:09:39 -08:00
Tim Abbott 043baa2af8 realm filters: Add a test for a useful thing to support. 2018-11-20 19:34:28 -08:00
Eeshan Garg b79213d260 webhooks: Notify bot owner on invalid JSON.
There are only a handful of non-JSON webhooks that wouldn't
benefit from the notify_bot_owner_on_invalid_json feature.

Specifically, these are the webhooks where the third-party product
uses another format, whether it be HTML form-encoded, XML, or
something else.

Tweaked by tabbott to correc the list of excluded webhooks.
2018-11-20 15:59:09 -08:00
Vishnu Ks ba640bf89d emails: Don't send day2 email if user already has an account. 2018-11-19 11:17:31 -08:00
Vishnu Ks 3258ce1fa4 emails: Cleanup followup day1 email. 2018-11-19 11:16:36 -08:00
Vishnu Ks 2e04cdbe5e billing: Show estimated subscription revenue on /activity.
[Substantial edits by Rishi Gupta]
2018-11-16 13:30:16 -08:00
Shubham Padia c1d1378125 auth: Add email_data option to github_oauth2_test.
Removes email_not_verified option. That option was used to assign
email_data a different set of emails for a test. Instead of that,
this refactor allows to specify the email_data itself in the function
which calls github_oauth2_test. Flags like email_not_verified are
generally used in one test. This is a preparatory refactor for
choose email screen which may have introduced multiple flags otherwise.
2018-11-16 12:18:23 -08:00
Shubham Padia d95364b94f auth: GitHubAuthBackend.get_verified_emails returns user's all emails.
The email_list returned has the primary email as the first element.
Testing: The order of the emails in the test was changed to put a
verified email before the primary one. The tests would fail without
this commit's change after the changes in the order of test emails.
2018-11-16 12:16:21 -08:00
Tim Abbott 209dd5db67 actions: Add a function for changing realm subdomains.
This is initial work, which will help us establish habits of using a
well-tested approach for renaming a Zulip organization (since as part
of https://github.com/zulip/zulip-mobile/issues/3142, we'll likely
need to make this function do more).
2018-11-15 14:39:14 -08:00
Tim Abbott 9e4058efd3 test_realm: Use update_fields when saving realm objects.
This is a slight cache performance optimization, and more importantly
matches our standard style for modifying objects.
2018-11-15 14:39:14 -08:00
Steve Howell f69a4f11e3 subject -> topic: Fix narrow tests.
This fixes the last remaining "subject" references except
for uses of it in expected SQL statements.
2018-11-14 23:24:06 -08:00
Steve Howell cc33e4cd0c digest: Eliminate unneeded queries for hot convos.
We can easily keep track of messages by bucket from the
original loop through messages.
2018-11-14 23:24:06 -08:00
Steve Howell f961408782 digest: Reduce queries using select_related().
We use the message a lot for the query modified
here, so I think it's worth taking the up-front
hit of getting bulkier objects to avoid O(N)
hops back to the database.
2018-11-14 23:24:06 -08:00
Steve Howell 51aa313833 test coverage: Add test_multiple_stream_senders().
This exercises a loop in digest.gather_hot_conversations().
2018-11-14 23:24:06 -08:00
Steve Howell 5a539d44b0 subject -> topic: Fix notification tests.
The URLs changed here are artificial, so we can convert
subject -> topic without breaking anything.
2018-11-14 23:24:06 -08:00
Steve Howell 8cf93a25c5 Use email_subject as var in test_notifications.
We change subject -> email_subject for all local vars
here, so it's easier to skip past this code if you
are sweeping subject -> topic.
2018-11-14 23:24:06 -08:00
Steve Howell 27d79352da topic -> subject: Extract get_topic_from_message_info().
This changes files where it's safe to just assume caller
may use either "topic" or "subject", and we prefer "topic"
but support "subject".
2018-11-14 23:24:06 -08:00
Steve Howell ea26372083 hipchat: Make conversion work with UUID ids from Stride.
Normal hipchat exports use integer ids for their
users and "rooms," which we just borrowed during
conversion.

Atlassian Stride uses stride UUIDs for these instead, but otherwise
has the same export format.

We now introduce IdMapper to handle external ids
that aren't integer.  The IdMapper will map UUID
ids to ints and remember them.  For ints it just
leaves them alone.

Fixes #10805.
2018-11-14 23:22:40 -08:00
Eeshan Garg d9958610a4 webhook_decorator: Support notifying bot owner on invalid JSON.
Our webhook-errors.log file is riddled with exceptions that are
logged when a webhook is incorrectly configured to send data in
a non-JSON format. To avoid this, api_key_only_webhook_view
now supports an additional argument, notify_bot_owner_on_invalid_json.
This argument, when True, will send a PM notification to the bot's
owner notifying them of the configuration issue.
2018-11-14 22:35:56 -08:00
Steve Howell a86e7fbd9f subject -> topic: Fix message tests. 2018-11-12 15:47:11 -08:00
Steve Howell 394cd4c6b6 subject -> topic: Fix user group tests. 2018-11-12 15:47:11 -08:00
Steve Howell ee9d321d9f subject -> topic: Fix tornado tests. 2018-11-12 15:47:11 -08:00
Steve Howell 2fe78ffe21 subject -> topic: Fix widget tests. 2018-11-12 15:47:11 -08:00
Steve Howell d86dd165da gitter/slack/hipchat: Remove "subject" from conversions.
We (lexically) remove "subject" from the conversion code.  The
`build_message` helper calls `set_topic_name` under the hood,
so things still have "subject" in the JSON.

There was good code coverage on `build_message`.
2018-11-12 15:47:11 -08:00
Steve Howell 710095920f subject -> topic: Fix push notification tests.
We also make the topic name more clear.
2018-11-12 15:47:11 -08:00
Steve Howell 8e639543fb subject -> topic: Fix webhook system tests. 2018-11-12 15:47:11 -08:00
Steve Howell 8f8ceab062 subject -> topic: Fix external tests. 2018-11-12 15:47:11 -08:00
Steve Howell cd76d4c308 subject -> topic: Fix events tests. 2018-11-12 15:47:11 -08:00
Steve Howell 4b38260a27 tests: Fix message.subject references. 2018-11-12 15:47:11 -08:00
Steve Howell 630dd8ac1b subject -> topic: Fix bugdown tests. 2018-11-12 15:47:11 -08:00
Steve Howell 3133998af9 test_messages: Fix a lot of subject -> topic_name. 2018-11-12 15:47:11 -08:00
Steve Howell e55fc144b9 Add alias support to REQ helpers for subject/topic.
The alias support is generic here, and we use it for
migrating subject -> topic in REQ_topic().
2018-11-12 15:47:11 -08:00
Steve Howell b46fb5eaad Rename subject_matches -> topic_matches.
This is always either a var or a SQL label,
so it's safe to change.
2018-11-12 15:47:11 -08:00
Steve Howell bb959acbaf Use modern huddle URLs in digest emails. 2018-11-12 12:01:09 -08:00
Steve Howell e79e0018f7 Use modern PM URLs in digest emails. 2018-11-12 12:01:09 -08:00
Steve Howell 6a89446e80 tests: Add test for huddle digest urls. 2018-11-12 12:01:09 -08:00
Steve Howell c08a3833e2 tests: Start to check urls in digest emails. 2018-11-12 12:01:09 -08:00
Roger Souza 6179bba2bd emails: Add one-click unsubscribe link to new login emails.
Fixes: #10547.
2018-11-08 14:39:33 -08:00
Tim Abbott 17be6e9165 confirmation: Move one_click_unsubscribe_link out of notifications.py.
This helps prevent a problematic import sequence when we start using
it from signals.py.
2018-11-08 14:39:33 -08:00
Steve Howell d05f731c1c Eliminate the use of arguments.db_data.
We now attach zulip_db_data to the markdown engines
for classes that need it.  This was the last remaining
global we had, so we remove `arguments.py` here.
2018-11-07 10:44:49 -08:00
Steve Howell b66304e167 refactor: Pass db_data down to helpers.
This mostly preps for the next commit.
2018-11-07 10:44:49 -08:00
Steve Howell fa6f642c9c refactor: Remove global argument.current_realm. 2018-11-07 10:44:48 -08:00
Steve Howell e1113c7011 refactor: Remove the global arguments.current_message.
The Markdown processor makes it fairly simple for
the helper classes to access the `md` engine.  We
now write `_md_engine.zulip_message` to avoid having
the current message in the global namespace.

Note that we do reuse engines for multiple messages,
but each engine is specific to a realm.  And we therefore
avoid even the theoretical possibility of leaking message
data between realms.
2018-11-07 10:44:48 -08:00
Steve Howell ffa4daf936 bugdown: Reduce overhead of building link regexes.
We were building the same link regex every time
we build a Markdown engine, which happens twice
per realm.  It's an expensive operation due to
the complexity of the regex and us reading a file.
2018-11-07 10:33:11 -08:00
Steve Howell 659c9dde00 bugdown: Avoid unnecessary realm queries.
We now keep realm in the arguments variable,
which avoids some lookups.

We also test settings before even trying to
get realms.
2018-11-07 10:08:46 -08:00
Steve Howell ff60055fa4 Use topic_match_sa() for topic searches.
Note this introduce literal(), which makes the way
we handle topic mutes more consistent with general
topic searches.
2018-11-07 10:03:53 -08:00
Steve Howell af1acf9239 Rename constant to MAX_TOPIC_NAME_LENGTH. 2018-11-07 10:03:53 -08:00
Steve Howell 32232377f7 Rename bugdown.subject_links -> topic_links. 2018-11-07 10:03:53 -08:00
Steve Howell 56ecd227ff Add topic-related helpers for ScheduledMessage. 2018-11-07 10:03:53 -08:00
Rohitt Vashishtha 652477daea bugdown: Refactor name to mention_text in mention.py.
This commit renames some variables to make it clear that a mention's
text is not only the name of the user in all cases.
2018-11-07 10:02:53 -08:00
Tim Abbott 6ef64fc752 narrow: Throw a clear error for invalid id: narrows.
This fixes an issue where searching for `id:` (with no operand) would
cause the server to 500, rather than returning a user-facing error.
2018-11-06 15:53:02 -08:00
Jack Zhang 5902a573be push_notifications: Play 'default' sound for iOS notifications.
Fixes zulip/zulip-mobile#2651.

This was tested on an iPhone 7 running iOS 12.
2018-11-02 17:01:50 -07:00
Tim Abbott ec065e92ee tornado: Store port on SockJS connection object.
This will make it available for use inside our websockets code.
2018-11-02 16:55:33 -07:00
Tim Abbott ea1ec68899 events: Pass a realm object into send_event.
This is a preparator refactor for supporting hosting different Tornado
processes on different servers; to look up which Tornado server we
should be sending the event to, we'll need the realm object.
2018-11-02 16:47:39 -07:00
Tim Abbott 75e48459b5 tornado: Support using a port-aware file for dumping event queues.
This should make it possible for there to safely be multiple Tornado
processes running on different ports on the same system.

It may also fix a rare race bug in development, where previously, it
was possible for the Tornados processes for Casper and the main
development server to interfere; I haven't investigated whether this
was a real bug or not, but now those two services will use independent
Tornado files.

We still need to add something to direct traffic between the different
Tornado processes.
2018-11-02 16:47:39 -07:00
Yashashvi Dave 4f37127ba9 /json/users/: Add user profile data variable to update values.
This commit adds user profile data variable in `/json/users/`
api view. This allows user to update user profile data
along with user's name and type.
2018-10-31 15:36:44 -07:00
Rishi Gupta 33d6baa529 portico: Add security doc to /security. 2018-10-30 19:52:24 -07:00
Steve Howell 30c493ed24 slack import: Generate message_id/reaction_id with NEXT_ID.
This avoids the need to pass tuples of ints around, which
is pretty brittle.
2018-10-29 13:24:50 -07:00
Steve Howell 2f58eb1057 slack import: Extract process_message_files().
This is mostly an extraction, but it does change the
way we calculate `content`.  We append the markdown
links from ALL files to any content that came in the
message itself.

Separating this out also allows us to add more
test coverage for the extracted code.
2018-10-29 13:24:50 -07:00
Steve Howell 00f822a26a conversion: Generate attachment_ids with helpers. 2018-10-29 13:24:50 -07:00
Steve Howell 5cb60f7bea conversions: Use subscriber_map for Slack/Gitter.
We now use subscriber_map for building UserMessage
rows in Slack/Gitter conversions.

This is mostly designed to simplify the code, rather
than having to scan the entire subscribers for each
message.

I am guessing this will improve performance for most
conversions.  We sort small lists on every message,
in order to be deterministic, but the sorting cost
is probably more than offset by avoiding the O(N)
scans across all subscriptions.  Also, it's probably
negligible in the grand scheme of things, compared
to JSON parsing, file I/O, etc.

This commits also fixes some typos with mentioned_users_id ->
mentioned_user_ids and cleans up a test a bit as well.
2018-10-29 13:24:50 -07:00
Steve Howell 5194701787 conversions: Use NEXT_ID for usermessage_id.
This is mostly complicated due to the way that the
Slack import passes around tuples of ids to maintain
four different parallel sequences.
2018-10-29 13:24:50 -07:00
Steve Howell fd62e71737 Clean up URLs sent by outgoing webhooks.
When you send a message to a bot that wants
to talk via an outgoing webhook, and there's
an error (e.g. server is down), we send a
message to the bot's owner that links to the
message that triggered the error.

The code to produce those links was out of
date.

Now we move the important code to the
`url_encoding.py` library and fix the PM
links to use the more modern style (user_ids
instead of emails).  We also replace "subject"
with "topic" in the stream urls.
2018-10-29 12:57:15 -07:00
Pragati Agrawal d5df0377cc settings_users: Support guest user in admin-user-table.
This supports guest user in the user-info-form-modal as well as in the
role section of the admin-user-table.

With some fixes by Tim Abbott and Shubham Dhama.
2018-10-29 12:33:35 -07:00
Vishnu Ks 553c50ebfb messages: Add history_limited flag to messages response.
The purpose of this commit is to pass information
to the frontend whether the message response recieved
has been limited due to plan restrictions or not.

To implement this, the backend for limiting the message
history had to be rewritten as we used to fetch
only the message rows whose id was greater than
first_visible_message_id. The filtered rows gives us
no information on whether the message history was
limited or not. So the backend was rewritten to not
do any restriction of limiting the message rows while
making the query. The limiting of rows is now done in
post_process_limited_query which will also return back
the value of history_limited flag.

Tweaked by tabbott to note a few cases where the results are
incorrect.  I'm merging this despite those, because those cases don't
impact the correctness of the feature, and it may have tricky
performance implications to fix correctly.
2018-10-26 17:41:07 -07:00
Tim Abbott dbc412b8eb test_narrow: Fix missing found_oldest assert in tests.
Apparently, we weren't actually checking that found_oldest had the
correct value; fortunately, this didn't actually result in a problem,
because the values were always correct.  But this will be important as
we start extending this test.
2018-10-26 17:33:13 -07:00
Vishnu Ks cc8dd0e971 billing: Set message_visibility_limit to 10000 for Limited. 2018-10-26 17:10:38 -07:00
Aditya Bansal f90f701f03 camo: Change CAMO_URI setting value for test suite.
This is a preparatory commit which will help us with removing camo.
In the upcoming commits we introduce a new endpoint which is based
out on the setting CAMO_URI. Since camo could have been hosted on
a different server as well from the main Zulip server, this change
will help us realise in tests how that scenerio might be dealt with.
2018-10-26 16:51:54 -07:00
Jack Zhang f116aba490 push notifications: Reword APNs payload alert titles.
Also, rename get_alert_from_message to get_gcm_alert.

With the implementation of the and get_apns_alert_title and
get_apns_alert_subtitle, the logic within get_alert_from_message
is only relevant to the GCM payload, so we adjust the name
accordingly.

Progresses #9949.
Resolves https://github.com/zulip/zulip-mobile/issues/1316.
2018-10-26 16:02:04 -07:00
Jack Zhang 92a100798c push notifications: Remove tests for get_alert_from_message.
The string that is returned from get_alert_from_message is
dependent upon the same message that is passed into get_apns_payload
and get_gcm_payload. The contents of those payloads that are tested via
TestGetAPNsPayload and TestGetGCMPayload, which makes the tests for
get_alert_from_message redundant.

Also, simplify the logic by removing the last elif conditional.
2018-10-26 15:55:26 -07:00
Steve Howell e2ee455314 outgoing webhooks: Support widget content.
If we use an outgoing webhook and the web server
responds with `widget_content` in the payload, we
include that in what we send through the send-message
codepath.

This makes outgoing webhook bots more consistent with
generic bots.
2018-10-26 12:08:05 -07:00
Rishi Gupta 92d0511c0b portico: Make a few wording edits to the top of /hello. 2018-10-25 18:25:20 -07:00
Steve Howell 4b4f27fffb tests: Fix flaky test by using sets, not lists.
The test named `test_archiving_messages_with_attachment`
started flaking recently.  We use sets for comparison
instead of lists to avoid arbitrary sorting differences.
2018-10-25 13:47:37 -05:00
Steve Howell 551fc7f165 bots: Prevent bots from having duplicate full names.
Bots are not allowed to use the same name as
other users in the realm (either bot or human).

This is kind of a big commit, but I wanted to
combine the post/patch (aka add/edit) checks
into one commit, since it's a change in policy
that affects both codepaths.

A lot of the noise is in tests.  We had good
coverage on the previous code, including some places
like event testing where we were expediently
not bothering to use different names for
different bots in some longer tests.  And then
of course I test some new scenarios that are relevant
with the new policy.

There are two new functions:

    check_bot_name_available:
        very simple Django query

    check_change_bot_full_name:
        this diverges from the 3-line
        check_change_full_name, where the latter
        is still used for the "humans" use case

And then we just call those in appropriate places.

Note that there is still a loophole here
where you can get two bots with the same
name if you reactivate a bot named Fred
that was inactive when the second bot named
Fred was created.  Also, we don't attempt
to fix historical data.  So this commit
shouldn't be considered any kind of lockdown,
it's just meant to help people from
inadvertently creating two bots of the same
name where they don't intend to.  For more
context, we are continuing to allow two
human users in the same realm to have the
same full name, and our code should generally
be tolerant of that possibility.  (A good
example is our new mention syntax, which disambiguates
same-named people using ids.)

It's also worth noting that our web app client
doesn't try to scrub full_name from its payload in
situations where the user has actually only modified other
fields in the "Edit bot" UI.  Starting here
we just handle this on the server, since it's
easy to fix there, and even if we fixed it in the web
app, there's no guarantee that other clients won't be
just as brute force.  It wasn't exactly broken before,
but we'd needlessly write rows to audit tables.

Fixes #10509
2018-10-24 16:59:57 -07:00
Tim Abbott 5cec566cb9 queue_processors: Rewrite MissedMessageWorker to always wait.
Previously, MissedMessageWorker used a batching strategy of just
grabbing all the events from the last 2 minutes, and then sending them
off as emails.  This suffered from the problem that you had a random
time, between 0s and 120s, to edit your message before it would be
sent out via an email.

Additionally, this made the queue had to monitor, because it was
expected to pile up large numbers of events, even if everything was
fine.

We fix this by batching together the events using a timer; the queue
processor itself just tracks the items, and then a timer-handler
process takes care of ensuring that the emails get sent at least 120s
(and at most 130s) after the first triggering message was sent in Zulip.

This introduces a new unpleasant bug, namely that when we restart a
Zulip server, we can now lose some missed_message email events;
further work is required on this point.

Fixes #6839.
2018-10-24 14:43:36 -07:00
Rishi Gupta 458169928c billing: Rename Zulip Premium to Zulip Standard. 2018-10-24 10:42:16 -07:00
Steve Howell 409e2b4134 hipchat import: Support sender_id == 0 use case. 2018-10-23 17:27:37 -05:00
Steve Howell 48b5b2f5d5 hipchat import: Start adding tests.
These test are for the handling of HipChat
sender info.  The data formats are somewhat
inconsistent and sometimes require us to
generate "mirror" users, so this is potentially
fragile code if we don't cover it well.
2018-10-23 17:27:37 -05:00
Steve Howell bd9e4ef0c8 import: Use pub_date to sort message ids.
When we create new ids for message rows, we
now sort the new ids by their corresponding
pub_date values in the rows.

This takes a sizable chunk of memory.

This feature only gets turned on if you
set sort_by_date to True in realm.json.
2018-10-23 17:27:37 -05:00
Rishi Gupta b26e8eca62 portico: Redirect /plans to zulipchat.com/plans for self-hosted realms. 2018-10-23 14:47:00 -07:00
Rishi Gupta fbd884f934 billing: Rename MAX_INVITES_PREMIUM.
Largely to have parallel structure with INVITES_DEFAULT_REALM_DAILY_MAX.
2018-10-23 14:47:00 -07:00
Rishi Gupta a7fe6b1c34 billing: Fix max_invites for PREMIUM_FREE.
We could migrate all the current PREMIUM_FREE organizations to have more
invites, but this setting mainly affects orgs right as they are starting, so
it's probably fine.
2018-10-23 14:47:00 -07:00
Aditya Bansal a16bf34c7f thumbnailing: Fix oversharpening of thumbnails.
We seemed to have been doing too much of sharpening on the thumbnails.
The purpose of sharpening here was to just counter the softening
effects of a resize on an image but overdoing it is bad.

Value sharpen(0.5,0.2,true) seems to look good for achieving the
best results here on different displays as revealed in the manual
hit and trial based testing.

Thanks to @borisyankov for pointing out the issue and suggesting
the values.
2018-10-22 22:28:04 +05:30
Pragati Agrawal b665b0a9a5 users: Avoid O(N) query checking for is_admin.
The main benefit is that we don't do O(N) queries for checking admin
thus leading to less processing of UserProfile.
2018-10-19 15:51:12 -07:00
Tim Abbott 39ea471cf1 error reports: Ensure we filter API keys from query strings.
For some webhook endpoints where the third-party API requires us to do
this, the user's API key might appear in error emails through
appearing in the `QUERY_STRING` parameter.  Fix that by filtering any
actual content from those; what we usually need for debugging is just
what set of parameters were provided.
2018-10-19 15:03:14 -07:00
Yashashvi Dave 41fbb16cdf org settings: Fix error when admin update realm users full names.
Currently, if there is only one admin in realm and admin tries
to updates any non-adminuser's full name it throws error,
"Cannot remove only realm admin". Because in `/json/users/<user_id>`
api check_if_last_admin_is_changed is checked even if property
is_admin is not changed.

This commit fix this issue and add tests for it.
2018-10-19 12:34:20 -07:00
Tim Abbott 695d8d0bd1 get_apns_payload: Require a UserProfile object for the recipient.
This is preparatory work for being able to display an unread count
badge on iOS, in which case we need to know who the current user is.
2018-10-18 15:09:18 -07:00
Tim Abbott 2eebacf2dc push_notifications: Lazily import APNS libraries.
The APNS client libraries (especially the hyper.http20 one) were
determined via profiling to take significant time during the import
process, so we move them to be lazily imported in order to optimize
the overall Zulip import process.  This save up to about 100ms in
import time.

These libraries are only used in certain Django processes inside
zulipchat.com, and so are unnecessary both in development as well as
for self-hosted Zulip servers.
2018-10-17 11:59:33 -07:00
Aditya Bansal 8324e2c976 thumbnails: Return original path if url is not supposed to be thumbnailed. 2018-10-16 16:00:47 -07:00
Aditya Bansal 9a411a5765 thumbnails: Stop thumbnailing urls other than external or user_uploads.
We are basically adding a check for url's to be external (belonging
to some 3rd party web site hosting the image) or be one of the
user uploaded files. User uploaded files are served by a separate
endpoint which is /user_uploads/. Any other local url such as
/user_avatars/ or /static/ should never be sent to thumbor for
thumbnailing.
Not sending /user_avatars/ to thumbor for thumbnailing makes sense
because they are already properly thumbnailed and stored properly.
/static/ urls host very few images we use for demo and can be safely
be excluded from thumbnailing.
2018-10-16 16:00:47 -07:00
Vishnu Ks 14a6a8db1a logs: Mock out warning logs in test_scrub_realm. 2018-10-16 11:16:47 -07:00
Vishnu Ks 57e0a24020 management: Scrub delivery email as well while running scrub_realm. 2018-10-16 11:16:46 -07:00
Kevin Lin b2c29274f3 presence: Allow bots to fetch realm presence data.
Before, presence information for an entire realm could only be queried via
the `POST /api/v1/users/me/presence` endpoint. However, this endpoint also
updates the presence information for the user making the request. Therefore,
bot users are not allowed to access this endpoint because they don't have
any presence data.

This commit adds a new endpoint `GET /api/v1/realm/presence` that just
returns the presence information for the realm of the caller.

Fixes #10651.
2018-10-15 12:51:28 -07:00
Steve Howell 76deb30312 preview: Hash cache keys for preview urls.
We don't want really long urls to lead to truncated
keys, or we could theoretically have two different
urls get mixed up previews.

Also, this suppresses warnings about exceeding the
250 char limit.

Finally, this gives the key a proper prefix.
2018-10-14 09:28:57 -07:00
Tim Abbott 68ab71eb8b push: Fix exceptions when removing push notifications.
Now that we allow multiple users to have registered the same token, we
need to configure calls to unregister tokens to only query the
targeted user_id.

We conveniently were already passing the `user_id` into the push
notification bouncer for the remove API, so no migration for older
Zulip servers is required.
2018-10-12 11:19:23 -07:00
Steve Howell 0f7628280f narrow: Handle spurious emails in pm-with searches.
If cordelia searches on pm-with:iago@zulip.com,cordelia@zulip.com,
we now properly treat that the same way as pm-with:iago@zulip.com.

Before this fix, the query would initially go through the
huddle code path.  The symptom wasn't completely obvious, as
eventually a deeper function would return a recipient id
corresponding to a single PM with @iago@zulip.com, but we would
only get messages where iago was the recipient, and not any
messages where he was the sender to cordelia.

I put the helper function for this in zerver/lib/addressee, which
is somewhat speculative.  Eventually, we'll want pm-with queries
to allow for user ids, and I imagine there will be some shared
logic with other Addressee code in terms of how we handle these
strings.  The way we deal with lists of emails/users for various
endpoints is kind of haphazard in the current code, although
granted it's mostly just repeating the same simple patterns.  It
would be nice for some of this code to converge a bit.  This
affects new messages, typing indicators, search filters, etc.,
and some endpoints have strange legacy stuff like supporting
JSON-encoded lists, so it's not trivial to clean this up.

Tweaked by tabbott to add some additional tests.
2018-10-12 10:18:30 -07:00
Steve Howell 51bd36e448 tests: Add coverage to get_service_interface_class(). 2018-10-11 16:12:07 -07:00
Steve Howell 8379aeee15 outgoing bots: Fix header for generic servers.
For our bots that use GenericOutgoingWebhookService
(which are basically Zulip style bots), we now
include a "content-type" header of "application/json".

We accomplish this by having the service classes
implement their own custom method called
`send_data_to_server`. For the Slack-related
code, we just extracted code from `do_rest_call`,
and then for the Zulip-related code, we added
a `headers` parameter.
2018-10-11 16:12:07 -07:00
Steve Howell 8226e13e9c bot tests: Replace use of MockServiceHandler.
We'll just use a real class here, since the service
handlers are pretty lightweight and just munge data.
2018-10-11 16:12:07 -07:00
Steve Howell 31597cf33e Remove timeout parameter in do_rest_call().
Nobody was setting it.
2018-10-11 16:12:07 -07:00
Steve Howell 69ee84bb14 refactor: Extract build_bot_request().
This fixes a couple things:

    * process_event() is a pretty vague name
    * returning tuples should generally be avoided
    * we were producing the same REST parameters in both
      subclasses
    * relative_url_path was always blank
    * request_kwargs was always empty

Now process_event() is called build_bot_request(),
and it only returns request data,
not a tuple of `rest_operation` and `request_data`.

By no longer returning `rest_operation`, there are
fewer moving parts.  We just have `do_rest_call` make
a POST call.
2018-10-11 16:12:07 -07:00
Steve Howell 16eff75e49 refactor: Simplify how we use base_url.
Before this change, we instantiated base_url into a superclass
of subclasses that returned base_url into a dictionary that
gets returned to our caller.

Now we just pull base_url out of service when we need to make
the REST call.
2018-10-11 16:12:07 -07:00
Steve Howell b89a94f730 Improve errors when we can't connect to a bot server.
We don't overwhelm people with error info when bots
fail to connect or time out.
2018-10-11 16:12:07 -07:00