Commit Graph

8695 Commits

Author SHA1 Message Date
Steve Howell 737e02a2e6 hipchat import: Fix PM messages.
Before this fix, we were creating two copies of every
PM Message in zerver_message with only corresponding
UserMessage row.

Now we only create one PM Message per message, which
we accomplish by making sure we only use imported
messages from the sender's history.json file.  And
then we write UserMessage rows for both participants
by making sure to include sender_id in the set of
user_ids that feeds into making UserMessage.  For
the case where you PM yourself, there's just one
UserMessage row.

It does not appear that we need to support huddles
yet.
2018-10-23 17:27:37 -05:00
Steve Howell 53436b4b41 import: Rename id_maps -> ID_MAP. 2018-10-23 17:27:37 -05:00
Steve Howell bd9e4ef0c8 import: Use pub_date to sort message ids.
When we create new ids for message rows, we
now sort the new ids by their corresponding
pub_date values in the rows.

This takes a sizable chunk of memory.

This feature only gets turned on if you
set sort_by_date to True in realm.json.
2018-10-23 17:27:37 -05:00
Steve Howell d1ff903534 refactor: Rename build_user -> build_user_profile.
This makes greps less confusing.
2018-10-23 17:27:37 -05:00
Rishi Gupta b26e8eca62 portico: Redirect /plans to zulipchat.com/plans for self-hosted realms. 2018-10-23 14:47:00 -07:00
Rishi Gupta fbd884f934 billing: Rename MAX_INVITES_PREMIUM.
Largely to have parallel structure with INVITES_DEFAULT_REALM_DAILY_MAX.
2018-10-23 14:47:00 -07:00
Rishi Gupta a7fe6b1c34 billing: Fix max_invites for PREMIUM_FREE.
We could migrate all the current PREMIUM_FREE organizations to have more
invites, but this setting mainly affects orgs right as they are starting, so
it's probably fine.
2018-10-23 14:47:00 -07:00
Eeshan Garg 4c0890e8b0 webhooks/github: Handle empty 'requested_reviewers' key.
We recently received a bug report that implied that for certain
payloads, the `requested_reviewers` key was empty whereas a
singular `requested_reviewer` key containing one reviewer's
information was present in its stead. Naturally, this raised
some not so pretty IndexError exceptions.

After some investigation and generating a few similar payloads,
I discovered that in every case both the `requested_reviewers`
and the `requested_reviewer` keys were correctly populated, so I
had to manually edit the payload to reproduce the error on my end.

My guess is that this anomaly goes back to when GitHub's reviewer
request feature was new and didn't support requesting multiple
reviewers, and that the singular `requested_reviewer` key could
possibly just be there for backwards compatibility or might just
be mere oversight. Either way, the solution here is to look for the
plural `requested_reviewers` key, and if that is empty, fall back
to the singular `requested_reviewer` key.
2018-10-23 14:41:24 -07:00
Rishi Gupta 1c264dedc5 api docs: Change Shakespeare quote.
The previous quote doesn't come off well if you don't know the reference
(which the majority of our users will not).
2018-10-22 15:44:36 -07:00
Eeshan Garg 3a47a1a627 webhooks/greenhouse: Ignore ping events.
"ping" type events are usually just Greenhouse's way of indicating
that a webhook has been set up.
2018-10-22 12:27:48 -07:00
Aditya Bansal a16bf34c7f thumbnailing: Fix oversharpening of thumbnails.
We seemed to have been doing too much of sharpening on the thumbnails.
The purpose of sharpening here was to just counter the softening
effects of a resize on an image but overdoing it is bad.

Value sharpen(0.5,0.2,true) seems to look good for achieving the
best results here on different displays as revealed in the manual
hit and trial based testing.

Thanks to @borisyankov for pointing out the issue and suggesting
the values.
2018-10-22 22:28:04 +05:30
Pragati Agrawal b665b0a9a5 users: Avoid O(N) query checking for is_admin.
The main benefit is that we don't do O(N) queries for checking admin
thus leading to less processing of UserProfile.
2018-10-19 15:51:12 -07:00
Tim Abbott 39ea471cf1 error reports: Ensure we filter API keys from query strings.
For some webhook endpoints where the third-party API requires us to do
this, the user's API key might appear in error emails through
appearing in the `QUERY_STRING` parameter.  Fix that by filtering any
actual content from those; what we usually need for debugging is just
what set of parameters were provided.
2018-10-19 15:03:14 -07:00
Yashashvi Dave 41fbb16cdf org settings: Fix error when admin update realm users full names.
Currently, if there is only one admin in realm and admin tries
to updates any non-adminuser's full name it throws error,
"Cannot remove only realm admin". Because in `/json/users/<user_id>`
api check_if_last_admin_is_changed is checked even if property
is_admin is not changed.

This commit fix this issue and add tests for it.
2018-10-19 12:34:20 -07:00
Rishi Gupta aad5c3df48 user docs: Fix typo causing broken gear icon. 2018-10-18 16:30:42 -07:00
Tim Abbott 695d8d0bd1 get_apns_payload: Require a UserProfile object for the recipient.
This is preparatory work for being able to display an unread count
badge on iOS, in which case we need to know who the current user is.
2018-10-18 15:09:18 -07:00
Tim Abbott a34b79a3f4 python: Avoid importing the mock module in production.
These lazy imports save a significant amount of time on Zulip's core
import process, because mock imports pbr, which in turn import
pkgresources, which is in turn incredibly slow to import.

Fixes part of #9953.
2018-10-17 15:28:48 -07:00
Steve Howell ff61c56f47 hipchat import: Add NotificationMessage support. 2018-10-17 12:11:08 -07:00
Tim Abbott f9b6eeb488 import: Migrate from json to ujson for better perf.
We expect to get better memory performace from
ujson than json.

We also do a better job of closing file handles.

This likely fixes #10377.
2018-10-17 12:11:08 -07:00
Tim Abbott 2eebacf2dc push_notifications: Lazily import APNS libraries.
The APNS client libraries (especially the hyper.http20 one) were
determined via profiling to take significant time during the import
process, so we move them to be lazily imported in order to optimize
the overall Zulip import process.  This save up to about 100ms in
import time.

These libraries are only used in certain Django processes inside
zulipchat.com, and so are unnecessary both in development as well as
for self-hosted Zulip servers.
2018-10-17 11:59:33 -07:00
Tim Abbott 704967faa4 email_mirror: Don't import talon unless we're using it.
Talon is an expensive import; on my system, deferring this import
saves 28ms on the import time for Zulip.
2018-10-17 11:25:38 -07:00
Rishi Gupta d76f728dda api docs: Rename webhook-walkthrough to incoming-webhooks-walkthrough. 2018-10-16 21:07:00 -07:00
Tim Abbott 78a15dd715 slack import: Fix obscure email address for Slackbot.
Since we know what slackbot is, we don't need to give it a crazy hash
as its email address.
2018-10-16 16:33:41 -07:00
Aditya Bansal 8324e2c976 thumbnails: Return original path if url is not supposed to be thumbnailed. 2018-10-16 16:00:47 -07:00
Aditya Bansal 5d68bd92ad thumbnails: Extract user_uploads_or_external as a function.
This is a prepartory commit for the upcoming changes. It was meaningful
to extract this one out because this function is essentially a condition
check on whether a given url is one of the user_uploads or an external
one. Based on its value we decide whether a url must be thumbnailed or
not and thus this function will also be used in an upcoming commit
patching lib/thumbnail.py to do the same check before thumbnail url
generation.
2018-10-16 16:00:47 -07:00
Aditya Bansal 9a411a5765 thumbnails: Stop thumbnailing urls other than external or user_uploads.
We are basically adding a check for url's to be external (belonging
to some 3rd party web site hosting the image) or be one of the
user uploaded files. User uploaded files are served by a separate
endpoint which is /user_uploads/. Any other local url such as
/user_avatars/ or /static/ should never be sent to thumbor for
thumbnailing.
Not sending /user_avatars/ to thumbor for thumbnailing makes sense
because they are already properly thumbnailed and stored properly.
/static/ urls host very few images we use for demo and can be safely
be excluded from thumbnailing.
2018-10-16 16:00:47 -07:00
Tim Abbott e4813e462b tornado: Rename async_request_{restart,stop} to mention timer.
Previously, these timer accounting functions could be easily mistaken
for referring to starting/stopping the request.  By adding timer to
the name, we make the code easier for the casual observer to read and
understand.
2018-10-16 15:39:10 -07:00
Eeshan Garg 0292ffe04b webhooks/taiga: Add test for when a task is moved between stories.
This commit adds a test for the payload that is generated when
a Task is moved from one user story to another on Taiga's Sprint
Taskboard UI.

This commit also gets up this webhook's test coverage up to 100%.
2018-10-16 13:06:20 -07:00
Eeshan Garg 48fe5ae5b2 webhooks/taiga: Stop catching KeyErrors for 'change' events.
I generated multiple payloads and verified that there are no
`change` event payloads that will not contain the values in
question, so it is useless to catch these KeyErrors. If there are
any anomalies still, it is better to be notified about them than
to silently ignore them.
2018-10-16 13:06:20 -07:00
Yago González 029d815be7 api docs: Document the /dev_fetch_api_key endpoint. 2018-10-16 12:47:12 -07:00
Yago González 3eeec94d03 api docs: Document the POST /users/me/subscriptions/properties endpoint. 2018-10-16 12:38:27 -07:00
Tim Abbott 8481a2fd2d api docs: Fix confusing discussion of "dev server".
The Zulip API is to be used on both development and production
servers, and really we just need to talk about zuliprc files.

There's a similar issue for the JS docs, but we need to fix the
copy/paste issues with those as well.
2018-10-16 12:23:23 -07:00
Yago González 3d2b3f8fa4 api docs: Document the POST /messages/flags endpoint. 2018-10-16 12:19:24 -07:00
Yago González 7a4103eee6 api docs: Document POST /mark_(all|stream|topic)_as_read. 2018-10-16 12:12:05 -07:00
Tim Abbott d13bd6e8d3 linkifiers: Rename articles to current feature name.
These were previously called "linkification filters", but we now want
to consistently refer to them as "linkifiers" in the UI and API docs.
2018-10-16 12:01:17 -07:00
Yago González 3bdc8f9946 api: Document the GET /realm/emoji endpoint. 2018-10-16 11:51:48 -07:00
Yago González 7a7b507e86 api docs: Document DELETE /realm/filters/<filter_id>. 2018-10-16 11:51:48 -07:00
Yago González a7c48acc8e api docs: Document GET /realm/filters. 2018-10-16 11:51:48 -07:00
Vishnu Ks 14a6a8db1a logs: Mock out warning logs in test_scrub_realm. 2018-10-16 11:16:47 -07:00
Vishnu Ks 57e0a24020 management: Scrub delivery email as well while running scrub_realm. 2018-10-16 11:16:46 -07:00
Kevin Lin b2c29274f3 presence: Allow bots to fetch realm presence data.
Before, presence information for an entire realm could only be queried via
the `POST /api/v1/users/me/presence` endpoint. However, this endpoint also
updates the presence information for the user making the request. Therefore,
bot users are not allowed to access this endpoint because they don't have
any presence data.

This commit adds a new endpoint `GET /api/v1/realm/presence` that just
returns the presence information for the realm of the caller.

Fixes #10651.
2018-10-15 12:51:28 -07:00
Steve Howell b1dd9a251b hipchat import: Break messages into smaller batches.
Even individual "room" files from hipchat can be large,
so we process only 1000 messages at a time
within each file, which produces smaller JSON files.
2018-10-15 10:54:23 -07:00
Steve Howell 6650bb2240 minor: Move fix_mentions() closer to caller. 2018-10-15 10:54:23 -07:00
Steve Howell 219ff0f749 hipchat import: Extract UserHandler class. 2018-10-15 10:54:23 -07:00
Steve Howell 2d523fd668 hipchat import: Extract make_user_messages(). 2018-10-15 10:54:23 -07:00
Steve Howell ca0495cbe6 hipchat import: Support attachments. 2018-10-15 10:54:23 -07:00
Steve Howell 2d4b09f59d utils: Add process_list_in_batches(). 2018-10-15 10:54:23 -07:00
Steve Howell d71f3eb1bf hipchat import: Add some more logging. 2018-10-14 09:29:04 -07:00
Steve Howell 76deb30312 preview: Hash cache keys for preview urls.
We don't want really long urls to lead to truncated
keys, or we could theoretically have two different
urls get mixed up previews.

Also, this suppresses warnings about exceeding the
250 char limit.

Finally, this gives the key a proper prefix.
2018-10-14 09:28:57 -07:00
Steve Howell d933779477 hipchat import: Support PrivateUserMessage data.
We now import PM data from HipChat.
2018-10-13 16:47:44 -07:00