Commit Graph

1188 Commits

Author SHA1 Message Date
Jack Weatherilt d857f26cd3 refactoring: Remove unused assignment on ensure_stream.
There were two instances of `ensure_stream` being called and assigned to
a variable with the variable not being used elsewhere. pyflakes picked
up on this (where it didn't in the previous version likely due to tuple
unpacking), so the the variable assignment has been replaced with a call
to `ensure_stream`.
2018-03-21 16:47:52 -07:00
Jack Weatherilt 3396cfc2ef refactoring: Replaced occurences of create_stream_if_needed.
Issue #2088 asked for a wrapper to be created for
`create_stream_if_needed` (called `ensure_stream`) for the 25 times that
`create_stream_if_needed` is called and ignores whether the stream was
created. This commit replaces relevant occurences of
`create_stream_if_needed` with `ensure_stream`, including imports.

The changes weren't significant enough to add any tests or do any
additional manual testing.

The refactoring intended to make the API easier to use in most cases.
The majority of uses of `create_stream_if_needed` ignored the second
parameter.

Fixes: #2088.
2018-03-21 16:47:36 -07:00
YJDave 11c995b70f custom profile data: Send event to active user on update.
On update of custom profile fields, send an event to all
active users of realm.
2018-03-21 16:08:12 -07:00
Harshit Bansal a49655e0d4 emoji: Migrate realm emoji to be addressed by `id` rather than `name`.
This commit migrates realm emoji to be addressed by their `id` rather
than their name. This fixes a long standing issue which was causing
an error on uploading an emoji with same name as a deactivated realm
emoji.

Fixes: #6977.
2018-03-20 22:24:44 -07:00
Tim Abbott 51018e12f5 message_edit: Remove obsolete block for diff highlighting.
We disabled the original "colorized HTML edit-history" feature way
back in 2013 in c51056ff8e.

That original feature involved showing what had been edited inline in
message bodies, so one could easily see what had been changed.

That old feature has since been replaced with the "view edit history"
menu option, and we're unlikely to ever want the old feature back.
So, we can just remove its code.  There's a few supporting variables
that were created to help implement this; we can clean those up and
simplify the `update_message` code now that this feature is fully
removed.
2018-03-20 15:22:53 -07:00
Greg Price 22071a44a7 antispam: Add a sitewide ratelimit on invites by new realms.
This applies only on a server open for anyone to create a realm.
Moreover, if the server admins have granted any given realm a
max_invites greater than the default, that realm is exempt too.
2018-03-16 18:00:11 -07:00
Tim Abbott c147d2e140 notifications: Fix missed-message emails for presence-idle users.
This fixes an unpleasant regression in
f5edeb01ae, where we stopped correctly
filtering users who have an open browser session that's idle.  These
users are tagged as "UserPresence.IDLE" with an current timestamp in
the database, and should be treated as idle for presence purposes.

As a result, if you had an open Zulip browser session, you incorrectly
wouldn't get missed-message emails for PMs and mentions before this fix.
2018-03-16 16:30:23 -07:00
YJDave 0281079a39 stream settings: Send all private streams subs to realm admins on load.
Tweaked by tabbott to simplify the conditionals in actions.py.

Fixes #8695.
2018-03-16 12:26:56 -07:00
YJDave 72a440a86d stream settings: Fix error in real time sync in subs add/remove event.
Currently, when other private stream subscriber add realm admin to
stream, new copy private stream is created in realm admin's streams.
Which resulted in error, cause there are two similar stream element
in stream settings.

If new subscriber is added to private stream, we first send them
stream `create` event, cause private stream are not visible until
user don't get subscribed at least once. But realm admins can now
always access private stream, so when realm admin is subscribed to
stream, realm admin get stream `create` event even if stream already
exist in on realm admin client side.

Fix this by extracting realm admins from stream `create` event on
`add` subscription operation and sending private stream `create`
event to all realm admins on stream creation operation.

Fixes #8695
2018-03-16 12:22:06 -07:00
Eeshan Garg af56df7723 webhooks: Enable custom topics and default PM notifications.
This commit adds a generic function called check_send_webhook_message
that does the following:
* If a stream is specified in the webhook URL, it sends a stream
  message, otherwise sends a PM to the owner of the bot.
* In the case of a stream message, if a custom topic is specified
  in the webhook URL, it uses that topic as the subject of the
  stream message.

Also, note that we need not test this anywhere except for the
helloworld webhook. Since helloworld is our default example for
webhooks, it is here to stay and it made sense that tests for a
generic function such as check_send_webhook_message be tested
with an actual generic webhook!

Fixes #8607.
2018-03-16 11:34:20 -07:00
Vishnu Ks 951b88dd30 models: Make email_allowed_for_realm raise exception. 2018-03-15 14:35:24 -07:00
Tim Abbott 1a3ae8a376 stream settings: Fix inadvertently merged broken whitespace.
I'm not sure why the pre-commit linter didn't flag this for me; maybe
it did and I just missed it.
2018-03-13 15:12:45 -07:00
YJDave 2031118545 stream settings: Allow realm admins to access all private stream subs.
This will allow realm admins to access subscribers of unsubscribed
private stream.  This is a preparatory commit for letting realm admins
remove those users.
2018-03-13 14:59:09 -07:00
neiljp (Neil Pilgrim) 9e1dbde82d mypy: Final small migrations to python3.5 annotations in many files. 2018-03-12 11:23:30 -07:00
Tim Abbott ef92fcbe2b topic history: Fix fetching topic history of public streams.
Apparently, we did essentially all the work to support showing full
topic history to newly subscribed users from a data flow perspective,
but didn't actually enable this feature by having the topic history
endpoint grant access to historical topics.  This fixes that gap.

I'm not altogether happy with how the code and tests read for this
feature; the code itself has more duplication than I'd like, and the
tests do too, but it works.
2018-03-11 20:59:20 -07:00
YJDave 382ac24151 create stream: Fix real time sync bug in private stream creation.
If new private stream is created by realm admin without realm admin
subscribed to it, then it doesn't automatically add created stream to
realm admin's stream list. We have to reload the browser to get newly
created stream in stream list. Cause private stream creation event is
only sent to the subscribed users to private stream, so even if realm
admin is acting user, they don't get creation event.

We should send private stream creation event to realm admin users along
with subscribed user to stream, as realm admins can access unsubscribed
private streams.

Tweaked by tabbott to fix various typos and clean up the code.
2018-03-09 18:35:12 -08:00
Shubham Padia 32e38d36aa bots: Send add/delete event on bot ownership change.
Adds realm_bot delete event. On bot ownership change, add event is
sent to the bot_owner(if not admin) and delete event to the
previous bot owner(if not admin). For admin, update event is sent.
2018-03-08 07:54:19 -08:00
Tim Abbott c3964dff6e i18n: Fix a last few strings mentioning realms. 2018-03-07 17:15:29 -08:00
Tim Abbott 9acf30b3d3 display_settings: Fix real-time sync of default language.
We apparently weren't sending to the frontend the data needed to
update the display of the current language.
2018-03-04 19:23:58 -08:00
Yago González b475e74c47 i18n: Fix string interpolation. 2018-03-04 15:44:12 -08:00
Tim Abbott 038579b840 streams: Fix counts using only one data point.
We had the aggregation query for StreamCount done slightly wrong.
2018-03-04 14:06:29 -08:00
Robert Hönig e69af3011e Add basic test for FeedbackBot.
This sets up a new test class with a simple
test, mostly for increasing coverage. The class
should in the future be extended to properly
verify the handle_feedback() logic.
2018-03-04 13:31:33 -08:00
Archana BS c7650c0d9d streams: send data for recent streams traffic to frontend.
Significantly edited by Tim Abbott and Vishnu Ks.
2018-03-04 13:24:53 -08:00
Robert Hönig 4c7dd971d0 embedded bots: Dispatch all config data on update event.
Previously, when a user updated the config data of an
embedded bot, only the updated fields were dispatched
back to the client. Dispatching all config data fields
makes the client updating code less brittle.
2018-03-04 13:11:23 -08:00
Greg Price 6d35a697af antispam: Don't let new accounts in open realms immediately send invites.
We haven't had a problem with this yet, but this should help prevent it.
2018-03-01 14:56:19 -08:00
rht 71ff8c370e django-2.0: Don't assign directly to Many-to-Many field.
The old pattern of setting the value and then using .save() here has
been deprecated.  set() also saves the record.
2018-03-01 08:49:35 -08:00
Robert Hönig 48b2e4eb66 backend: Add bot config data patching. 2018-03-01 08:25:43 -08:00
Robert Hönig 312cabb9a6 models.py: Move two long functions with circular imports to actions.py.
models.py should only contain thin wrapper functions. Furthermore,
this move allows us to remove the circular imports. The two moved
functions are interdependent and are thus moved in one commit.
2018-03-01 08:25:43 -08:00
Aastha Gupta d124597f4b org-settings: Add setting to turn off welcome emails.
This adds an organization-level setting to provide an option to turn
off the welcome emails.

Fixes #8000.
2018-02-28 12:39:01 -08:00
Aditya Bansal f50e325075 RealmAuditLog: Fill subscription events with event_last_message_id=None.
Creating the very first organization administrator user and
subscribing them to streams before any messages were sent resulted in
RealmAuditLog entries being created with a `event_last_message_id` of
None, because that's the maximum ID in the empty set.

We correct this by fixing the incorrectly created RealmAuditLog
entries, both for new servers and also fixing old broken entries on
existing servers.

This fixes an issue where if a user setup a Zulip server with just the
organization administrator, and then forgot about it (so that the
initial user became soft-deactivated), trying to sign in 3 weeks later
would throw an exception.

This fixes the issue reported here:

https://chat.zulip.org/#narrow/stream/9-issues/subject/500.20error.20on.20login/near/511981
2018-02-22 09:45:22 -08:00
Shubham Dhama 9feae472f8 org settings: Add button to deactivate organization.
This adds button under "Organization profile" settings, which
deactivates the organization and sends an "event" to all the
active user and log out them.

Fixes: #8212.
2018-02-18 10:20:38 -08:00
neiljp (Neil Pilgrim) ec829ea832 mypy: Add 2 Optional[StreamTopicTarget] annotations in actions.py. 2018-02-13 11:40:52 -08:00
neiljp (Neil Pilgrim) 79582a59f1 mypy: Move assertion into similar branch in get_recipient_info. 2018-02-13 11:40:52 -08:00
neiljp (Neil Pilgrim) 23cc7e1671 mypy: Allow acting_user of do_change_full_name() to be None.
This is based on usage in bulk_change_user_names.py, and that
the RealmAuditLog acting_user field is Optional[UserProfile].

This could be more meaningfully changed in future, perhaps to
indicate that the command was run by a specific zulip user.
2018-02-13 11:40:51 -08:00
Vishnu Ks 81f18bd69a actions: Create do_create_multiuse_invite_link function. 2018-02-13 09:31:45 -08:00
Greg Price ca1129dea3 Revert "Streams: Disallow certain markdown characters in stream names."
This reverts commit dd1b14378b.

Deleted a conflicted test entirely.
2018-02-05 16:55:21 -08:00
rht 9161f8c39b zerver/lib: Remove u prefix from strings. 2018-02-05 12:12:58 -08:00
Shubham Dhama dbce49ac53 refactor: Move topic name checks to Addressee class.
In order to get test coverage on topic name checks, we
do them in Addressee, so that we don't hit an assertion
first. The assertion in question is in Addressee.topic(),
and it was added partly to appease mypy.
2018-01-25 06:21:01 -05:00
Weronika Grzybowska 7ac7100a1d messages: Make checking for status message consistent with backend.
Adds a check for newline that was present on backend, but missing in the
frontend markdown implementation. Updating messages uses is_me_message flag
received from server instead of its own partial test. Similarly, rendering
previews uses markdown code.

Fixes #6493.
2018-01-23 09:26:41 -05:00
Robert Hönig 9a15c4e3ff Add bot services to page_params.
This is the first step for allowing users
to edit a bot's service entries, name the
outgoing webhook configuration entries. The
chosen data structures allow for a future
with multiple services per bot; right now,
only one service per bot is supported.
2018-01-23 07:29:00 -05:00
YJDave 8905744bb1 stream settings: Add period at the end of error messages. 2018-01-19 13:05:43 -05:00
Aditya Bansal f272ea9087 scheduledmessages: Start using/expecting delivery_type as a param. 2018-01-19 11:33:11 -05:00
Aditya Bansal c770bdaa3a reminder_bot: Add infra for adding reminder bot to every realm. 2018-01-19 11:33:11 -05:00
Aditya Bansal c64766e55e scheduledmessages: Add funcs to verify and schedule messages.
We add two functions:

1.) check_schedule_message(): This function is responsible for
doing the essential initial checkes to verify the validity of
the message. These checkes include things like if user is
allowed to send messages to some stream or not or if the user is
a super_user. All this is basically done by further calling
check_message() with appropriate parameters. This is on the same
lines as is check_send_message().

2.) do_schedule_messages(): This function is responsible for
creating ScheduleMessage table rows for a list of messages that
are to be scheduled. This basically accumulates the ScheduleMessage
objects in a list and then bulk creates the rows.
2018-01-10 09:18:02 -05:00
YJDave fa44d2ea69 settings: Remove autoscroll_forever setting.
Fixes #6845
2018-01-02 10:35:49 -05:00
rht 2e8adbde12 mypy: Use Python 3 type syntax in zerver/lib/actions.py. 2017-12-30 07:33:07 -05:00
snlkapil 1589c597d2 settings: Show local time instead of UTC in invitations tab. 2017-12-20 07:08:46 -05:00
Rishi Gupta 27a8ef86e4 actions: Add realm deactivation and reactivation to RealmAuditLog. 2017-12-19 17:46:36 -08:00
Rishi Gupta 4261307e27 emails: Clear ScheduledEmail emails on realm deactivation.
This only affects the invitation reminder emails; clear_scheduled_emails was
already clearing the other two types.
2017-12-19 17:46:36 -08:00
Vishnu Ks bc37317914 invites: Differentiate users invited as admin by showing bolt icon. 2017-12-13 15:12:21 -06:00
Rhea Parekh dd1b14378b Streams: Disallow certain markdown characters in stream names. 2017-12-12 10:04:31 -06:00
Greg Price e9a7173119 actions: Cut some redundant types in docstrings.
We have a better way of writing types now!

Also move a comment that should have been a docstring,
to being a docstring.
2017-12-11 19:29:11 -08:00
Shreyansh Dwivedi 47fcb27e39 invitations: Remove custom_body.
Fixes #7672
2017-12-11 19:23:54 -08:00
Vishnu Ks a30152cb94 models: Use is_cross_realm_bot_email whenever possible.
Fixes #7557
2017-12-07 14:55:06 -08:00
Rishi Gupta 6787fbd547 invitations: Change rate limiting to use invites_sent::day.
[greg: small tweaks]
2017-12-06 20:35:50 -08:00
Rishi Gupta fbd8dde1f8 invitations: Add LoggingCountStat to keep track of sent invitations. 2017-12-06 20:35:50 -08:00
Rishi Gupta cb5eb47470 invitations: Move resend/revoke error handling out of actions.py. 2017-12-06 20:35:50 -08:00
Rishi Gupta 968aae167b invitations: Remove get_prereg_user_by_email.
The original logic is buggy now that emails can belong to (and be
invited to) multiple realms.

The new logic in the `invites` queue worker also avoids the bug where
when the PreregistrationUser was gone by the time the queue worker got
to the invite (e.g., because it'd been revoked), we threw an exception.

[greg: fix upgrade-compatibility logic; add test; explain
revoked-invite race above]
2017-12-06 20:35:50 -08:00
Rishi Gupta 8baadc6b15 invitations: Add check for invite limit to resend invite pathway. 2017-12-06 20:30:42 -08:00
Greg Price b3482429f1 invitations: Adjust a few blank lines for clarity and compactness. 2017-12-06 20:30:42 -08:00
Rishi Gupta cb527fd3c6 invitations: Move spammer check from do_invite_users into its own function. 2017-12-06 20:30:42 -08:00
Rishi Gupta e48bc9136a invites: Change resend invitation pathway to use existing code.
This code changes frequently enough that errors are bound to creep in.  The
main change is that this sends the original invitation email instead of the
reminder email, but I think that's fine.
2017-12-06 20:30:42 -08:00
Rishi Gupta 7d1c88f0fb invitations: Limit realms to 100 invites per day.
To guard against using zulip invites as a vector for spam. Stopgap measure
until we figure out something better.
2017-11-29 22:18:05 -08:00
Tim Abbott 6bc1a682de notifications: Separate push and email notifications checks.
This is an early step in a larger refactor to properly decouple the
email and push notification code paths.
2017-11-28 17:51:18 -08:00
rht ee546a33a3 zerver/lib: Use python 3 syntax for typing.
Edited by tabbott to improve various line-wrapping decisions.
2017-11-28 17:15:14 -08:00
rht 229a8b38c0 zerver/lib: Use Python 3 syntax for typing for several files.
This adds a number of annotations that had been missed in previous
passes.
2017-11-28 17:02:24 -08:00
Tim Abbott 586e18b237 auth: Allow accounts with the same email in different realms.
[Modified by greg to (1) keep `USERNAME_FIELD = 'email'`,
(2) silence the corresponding system check, and (3) ban
reusing a system bot's email address, just like we do in
realm creation.]
2017-11-28 16:23:10 -08:00
Tim Abbott e6f460f511 auth: Replace user_email_is_unique validator.
As we migrate to allow reuse of the same email with multiple realms,
we need to replace the old "no email reuse" validators.  Because
stealing the email for a system bot would be problematic, we still ban
doing so.

This commit only affects the realm creation logic, not registering an
account in an existing realm.
2017-11-28 16:23:10 -08:00
Vishnu Ks 985768b2fd registration: Check realm against PreregistrationUser realm.
We would allow a user with a valid invitation for one realm to use it
on a different realm instead.  On a server with multiple realms, an
authorized user of one realm could use this (by sending invites to
other email addresses they control) to create accounts on other
realms. (CVE-2017-0910)

With this commit, when sending an invitation, we record the inviting
user's realm on the PreregistrationUser row; and when registering a
user, we check that the PregistrationUser realm matches the realm the
user is trying to register on.  This resolves CVE-2017-0910 for
newly-sent invitations; the next commit completes the fix.

[greg: rewrote commit message]
2017-11-27 14:58:26 -08:00
Tim Abbott 8b935f4e99 settings: Add setting for SYSTEM_BOT_REALM.
This fixes some subtle JavaScript exceptions we've been getting in
zulipchat.com, caused by the system bot realm there not being "zulip"
interacting with get_cross_realm_users.
2017-11-27 14:46:07 -08:00
Tim Abbott 25fd4c5508 bulk_get_users: Edit the cache keys to make them more unique.
While at this point I was to rewrite this function, this at least
plugs the issues for now.
2017-11-27 14:41:31 -08:00
Tim Abbott 339b67f7af get_cross_realm_dicts: Harden against caching bug in last commit.
This should help protect us from future issues with the way that
`bulk_get_users` does caching.

It's likely that we'll want to further restructure `bulk_get_users` to
not have this base_query code path altogether (since it's kinda
buggy), but I'm going to defer that for a time when we have another
user.
2017-11-27 14:35:50 -08:00
Tim Abbott 649b7b8b9f internal_send_message: Enforce being passed cross-realm bots.
We include ERROR_BOT in this set, even though it's not technically
cross-realm (it just lives in the admin realm).

This code path does not correctly handle emails that correspond to
multiple accounts (because `get_system_bot` does not).  Since it's
intended to only be used by system bots, we add an appropriate
assertion to ensure it is only used for system bots.
2017-11-26 17:14:23 -08:00
Tim Abbott ae8e203717 actions: Merge internal_prep_message into internal_send_message.
We also document the fact that only system bots are supported here.
2017-11-26 17:14:23 -08:00
Tim Abbott 5306a9634d email_mirror: Rewrite to not use internal_send_message.
This was causing problems with the fact that `get_system_bot` now only
works for actual system bot users.
2017-11-26 17:14:23 -08:00
Tim Abbott 2210f627a5 signup: Switch active mirror-dummy users to an AssertionError.
Previously, this was a ValidationError, but that doesn't really make
sense, since this condition reflects an actual bug in the code.

Because this happened to be our only test coverage the ValidationError
catch on line 84 of registration.py, we add nocoverage there for now.
2017-11-26 16:13:45 -08:00
Robert Hönig 0e0a8a2b14 queue processor tests: Call consume by default.
This significantly improves the API for queue_json_publish to not be
overly focused on what the behavior of this function should be in our
unit tests.
2017-11-26 11:45:34 -08:00
Vishnu Ks 766511e519 actions: Mark all messages as read when user unsubscribes from stream.
This fixes a bug where, when a user is unsubscribed from a stream,
they might have unread messages on that stream leak.  While it might
seem to be a minor problem, it can cause significant problems for
computing the `unread_msgs` data structures, since it means we need to
add an extra filter for whether the user is still subscribed, either
in the backend or in the UI.

Fixes #7095.
2017-11-21 20:09:17 -08:00
Vishnu Ks 34689370cd settings: Create UI and backend for setting signups stream. 2017-11-21 17:39:51 -08:00
Vishnu Ks 3d2c9c6098 models: Replace core team with Realm.INITIAL_PRIVATE_STREAM_NAME. 2017-11-21 17:39:51 -08:00
Vishnu Ks a0275a6257 actions: Send new user messages to signup_notifications_stream. 2017-11-21 17:39:50 -08:00
Vishnu Ks 3d17897563 onboarding: Move signup notifications stream creation to do_create_realm. 2017-11-21 17:39:50 -08:00
Tim Abbott 25ddba99f6 lint: Ban general use of user_profile.save().
This often can cause minor caching problems.

Obviously, it'd be better if we had access to the AST and thus could
do this rule for UserProfile objects in general.
2017-11-20 10:57:08 -08:00
Tim Abbott 81adc92db6 actions: Use get_system_bot for fetching WELCOME_BOT. 2017-11-17 17:25:56 -08:00
Harshit Bansal d9c2f613e3 api: Add new endpoint for reactions.
This endpoint will allow us to add/delete emoji reactions whose emoji
got renamed during various emoji infra changes. This was also a
required change for realm emoji migration.

This commit was tweaked significantly by tabbott for greater clarity
(with no changes to the actual logic).
2017-11-16 20:52:15 -08:00
Rhea Parekh 8c4651d716 zerver/lib/actions.py: Fix translation bugs.
Fixes #7396.
2017-11-16 11:05:49 -08:00
Umair Khan 98be0cc502 user-groups: Send delete user group event.
Fixes #7380
2017-11-15 22:35:43 -08:00
Umair Khan cc76f7deac user-groups: Create check_delete_user_group. 2017-11-15 22:35:43 -08:00
Umair Khan 79f4a7627f user-groups: Send remove members event. 2017-11-15 22:35:23 -08:00
Umair Khan cec3f19366 user-groups: Send add members event. 2017-11-15 22:35:23 -08:00
Umair Khan eb48fab495 user-groups: Send description update event. 2017-11-15 22:35:23 -08:00
Umair Khan 5571122120 user-groups: Send name update event. 2017-11-15 22:35:23 -08:00
Umair Khan 912505317a user-groups: Send create group event. 2017-11-15 22:35:23 -08:00
Umair Khan 31efe951b7 user-groups: Fix function argument bug; pass group.
In remove_members_from_group_backend, we are passing user group to
remove_members_from_user_group. In remove_members_from_user_group,
expect user_group_id.
2017-11-15 22:35:23 -08:00
Umair Khan 66179ea295 bulk_add_members_to_user_group: Pass user group.
Previously we passed user group id. If we pass user group, send event is
easier.
2017-11-15 22:35:23 -08:00
Tim Abbott f02e5b90f6 cross_realm: Use bulk_get_users to fix handling of missing users.
This fixes a regression in ae5ba7f4fd,
where Zulip would 500 if the newly added system bots didn't exist on
the server.

This also fixes a moderate size performance problem where we'd fetch 5
users from memcached or the database in a loop.
2017-11-15 21:24:51 -08:00
Tim Abbott c7a975e4df users: Move check_change_full_name to actions.py.
This avoids an import loop in the next commit, and better matches our
usual code structure.
2017-11-15 17:39:09 -08:00
Vishnu Ks 6f955fbf34 actions: Replace stream.add with stream.append in process_new_human_user.
stream is a list not a set. It used to be set
and was changed to stream before getting merged.
2017-11-15 17:06:43 -08:00
rht 09af29b051 zerver/lib: Text-wrap long lines exceeding 110. 2017-11-15 10:58:03 -08:00
Tim Abbott 5aa6751e7a lint: Fix long lines in newly added features. 2017-11-15 10:58:03 -08:00
Vishnu Ks d2555e8769 api: Add support to change name of default stream groups. 2017-11-14 14:41:42 -08:00
Vishnu Ks 2af249dd5d api: Add description attribute to default stream group. 2017-11-14 14:41:42 -08:00
Vishnu Ks f45ba7de93 api: Pass group id instead of name to default stream group api. 2017-11-14 14:41:42 -08:00
rht 23a481c5d2 zerver/lib/actions.py: Text-wrap long lines exceeding 110. 2017-11-10 16:27:22 -08:00
Vishnu Ks f44b60a150 Implement API for default stream groups. 2017-11-10 16:05:36 -08:00
Umair Khan b1603d289c user-groups: Add update members API endpoint.
Significantly modified by tabbott to fix some bugs.
2017-11-09 17:35:37 -08:00
Umair Khan eeaf0f1742 user-groups: Add edit API endpoint.
Significantly modified by tabbott to fix some bugs.
2017-11-09 17:35:22 -08:00
Umair Khan 1bbe84af49 user-groups: Add create API endpoint.
Significantly modified by tabbott for better security structure.
2017-11-09 17:26:14 -08:00
Umair Khan 18aecb82d7 user-groups: Get members from membership. 2017-11-09 16:18:40 -08:00
Steve Howell 5d3ecf8f31 Remove force_str() in log_event(). 2017-11-09 10:32:14 -08:00
Harshit Bansal 65838bb825 email_gateway: Disable code block processor for email gateway.
Generally emails are not written with markdown in mind and hence
sometimes render in strange ways. This commit fixes a particular
issue that was causing whitespace before paragraphs to be treated
as code block due to which email content was being rendered in a
box that scrolls in right direction a lot.

Fixes: #7045.
2017-11-09 09:56:35 -08:00
rht 6cce0e346e refactor: Remove six.moves.filter import. 2017-11-07 10:51:44 -08:00
rht 80a8d4f9f3 refactor: Remove six.moves.map import. 2017-11-07 10:46:42 -08:00
rht e311842a1b zerver/lib: Remove inheritance from object. 2017-11-06 08:53:48 -08:00
rht fef7d6ba09 zerver/lib: Remove u prefix from strings.
License: Apache-2.0
Signed-off-by: rht <rhtbot@protonmail.com>
2017-11-03 15:34:37 -07:00
Steve Howell 9767029211 Allow generic bots to be mentioned.
The original PR to allow generic bots to be mentioned had
some merge issues that we detected about a week after the
fact.  This commit restores the logic from the original PR.

The reason we didn't detect this bug earlier is that the
merge issues didn't break any existing behavior.  Instead,
they made it so that only UserMessage rows got written for
bots, but no events were being set.  The part of the commit
that got lost is restored here, so now events get sent as
well.

Thanks to @derAnfaenger for reporting this and being patient
as we tracked it down.

Fixes #7140
2017-11-03 07:45:18 -07:00
Harshit Bansal 44bff43c86 actions.py: Rename `do_remove_reaction()` to `do_remove_reaction_legacy()`. 2017-10-31 15:42:35 -07:00
Harshit Bansal 233595b2f7 actions.py: Rename `do_add_reaction()` to `do_add_reaction_legacy()`. 2017-10-31 15:42:35 -07:00
Umair Khan 5914afc018 cleanup: Use a single loop in do_send_messages. 2017-10-31 12:56:14 -07:00
Steve Howell faba34dae4 Simplify bulk_remove_subscriptions().
We extract get_bulk_stream_subscriber_info() from this
function to remove some of the complexity.  Also, in that
new function we avoid a hop to the database by querying
on stream ids instead of recipient ids.  The query that
gets changed here does require a join to the recipient
table (to get the stream id), so it's a little bit of a
tradeoff.
2017-10-30 16:33:50 -07:00
Steve Howell 48d13257b6 Make `our_realm` explicit in `bulk_remove_subscriptions`.
There's an implicit assumption in bulk_remove_subscriptions
that all users belong to the same realm.  We use the realm
for things like comparing occupied streams before and
after our main operation of deactivating streams.

Before this change, we just used the user_profile variable
that leaked from some prior loop to look up the realm, which
was super brittle.

Now we're a bit more explicit.
2017-10-30 16:29:43 -07:00
Steve Howell 08ad26f913 refactor: Extract get_stream_subscriptions_for_users(). 2017-10-29 18:36:35 -07:00
Steve Howell b3192d17ab refactor: Extract get_stream_subscriptions_for_user(). 2017-10-29 18:36:35 -07:00
Steve Howell a2747517a3 Extract StreamTopicTarget.get_active_subscriptions().
Note that this code leads to a slightly different query, because
we join to one row in the small Recipient table to match
stream_id to recipient.type_id.
2017-10-29 18:36:35 -07:00
Steve Howell 8e0b417bd9 Extract get_active_subscriptions_for_stream_ids(). 2017-10-29 18:36:35 -07:00
Steve Howell 126e14d1de Add zerver/lib/stream_subscription.py.
The first method we extract to this library is
get_active_subscriptions_for_stream_id().

We also move num_subscribers_for_stream_id() to here, which
is slightly annoying (having the method on Stream was nice)
but avoids some circular dependency issues.
2017-10-29 18:36:35 -07:00
Steve Howell 138568f4f4 refactor: Extract get_stream_recipients(). 2017-10-29 18:36:35 -07:00
Steve Howell f64c780d5c Extract get_huddle_user_ids(). 2017-10-29 18:36:35 -07:00
Steve Howell 8302689789 Extract Message.is_stream_message().
This sets us up to denormalize on to Message some notion of
whether a message is for a stream (without having to hit the
Recipient table).
2017-10-28 17:57:39 -07:00
Steve Howell 8b012c6210 Extract get_personal_recipient(). 2017-10-28 17:57:39 -07:00
Steve Howell 64211512f8 Avoid mutation in get_recipient_from_user_ids.
We shouldn't mutate parameters in a `get_foo` function.
2017-10-28 17:57:39 -07:00
Steve Howell 370ffe389c Extract get_huddle_recipient().
This extraction moves all the huddle logic into models.py, which
hopefully can reduce friction for things like re-organizing our
caches (there are two cache entries for every huddle) and/or
just putting huddle_id on Message directly.
2017-10-28 17:57:39 -07:00
Steve Howell a28841e8aa Extract get_stream_recipient().
Do you call get_recipient(Recipient.STREAM, stream_id) or
get_recipient(stream_id, Recipient.STREAM)?  I could never
remember, and it was not very type safe, since both parameters
are integers.
2017-10-28 17:57:39 -07:00
Steve Howell d132c30c24 Remove `active` parameter from `do_create_user()`.
Almost all callers to do_create_user were trying to
create active users, except for one test.  The
active=False codepath was kind of broken (things
like sending welcome messages had sort of undefined
behavior there), so instead of trying to maintain it,
we just update the one test (`test_people`) to flip the
`is_active` flag manually.

Fixes #7197
2017-10-28 10:32:01 -07:00
Tim Abbott fa55d7ed33 retry_event: Remove requirement of failed_tries being already there.
This fixes a bug where retries in the signups queue threw an exception.
2017-10-27 18:15:53 -07:00
Henrik Pettersson 09cd47c6ad Add UI for viewing and cancelling open Zulip invitations.
Lets administrators view a list of open(unconfirmed) invitations and
resend or revoke a chosen invitation.

There are a few changes that we can expect for the future:

  * It is currently possible to invite an email that you have already
    invited, it might make sense to change this behavior.

  * Resend currently sends an invite reminder instead of resending the
    original invite, this is because 'custom_body' was not stored when
    the first invite was sent.

Tweaked in various minor ways, primarily in the backend, by tabbott,
mostly for style consistency with the rest of the codebase.

Fixes: #1180.
2017-10-27 13:07:43 -07:00
Vishnu Ks 8c68a167fe backend: Allow Administrators to invite new users as admins.
Tweaked by tabbott to have the field before the invitation is
completed be called invite_as_admins, not invited_as_admins, for
readability.

Fixes #6834.
2017-10-27 11:19:38 -07:00
Steve Howell e5c5d10375 Rename subject vars in do_update_message(). 2017-10-27 10:48:11 -07:00
Steve Howell fbe5f93141 Rename subject -> topic_name in internal_send_message. 2017-10-27 10:48:11 -07:00
Steve Howell 973ee97612 Rename subject -> topic_name in internal_prep_message(). 2017-10-27 10:48:11 -07:00
Steve Howell f04550d143 Rename subject -> topic_name as var in check_message().
There were two variables, subject and subject_name, that
really had no clear distinction, so now they're just
topic_name.
2017-10-27 10:48:11 -07:00
Steve Howell 70b05998b1 Rename subject_name -> topic_name in check_send_message(). 2017-10-27 10:48:11 -07:00
Tim Abbott cab39dc6eb get_service_bot_events: Add some future-proofing.
The comment is the most important part here, but the new code
structure should help, too.
2017-10-26 22:16:47 -07:00
Tim Abbott b897c763f8 get_recipient_info: Simplify active_user_ids construction. 2017-10-26 22:16:47 -07:00
Steve Howell a70ede6c75 Allow "default" bots to see mentions on all streams.
This change allows normal bots to get UserMessage rows when
they are mentioned on a stream, even if they are not actually
subscribed to the stream.

Fixes #7140.
2017-10-26 22:16:47 -07:00
Steve Howell 53dbcbe2a4 refactor: Handle service bot mentions earlier in the code.
We now find all (possibly) relevant service bots for a message
in the call to get_recipient_info.  This allows us to eliminate
some code that would patch them after we rendered.

The get_service_bot_events() function will ignore any service
bots that weren't actually mentioned in the message (due to
backticks) or part of the active user ids.
2017-10-26 22:16:47 -07:00
Steve Howell 8ac26dfb9b refactor: Introduce bugdown.MentionData class.
We now have a MentionData class that encapsulates
the users who are possibly mentioned in a message.

Not that the rendering code may not keep all the mentions,
since things like backticks will suppress the mention.

We populate this now in do_send_messages, so that we can use
the info earlier in the message-sending process.  This info
now gets passed down the call stack as an optional parameter.

Note that bugdown.convert() still populates the data when its
callers decline to pass in a MentionData object.

This is mostly a preparatory commit, as we don't take advantage
of the data yet in do_send_messages.
2017-10-26 22:16:47 -07:00
Steve Howell 635675fe48 Reduce queries needed for sending messages.
In do_send_messages, we only produce one dictionary for
the event queues, instead of different flavors for text
vs. html.  This prevents two unnecessary queries to the
database.

It also means we only put one dictionary on the "message"
event queue instead of two, albeit a wider one that has
some values that won't be sent to the actual clients.

This wider dictionary from MessageDict.wide_dict is also
used for the `feedback_messages` queue and service bot
queues.  Since the extra fields are possibly useful down
the road, and they'll just be ignored for now, we don't
bother to remove them.  Also, those queue processors won't
have access to `content_type`, which they shouldn't need.

Fixes #6947
2017-10-26 16:35:28 -07:00
Steve Howell df93a99b50 Cache only one row per message.
Before this change, we populated two cache entries for each
message that we sent.  The entries were largely redundant,
with the only difference being whether we sent the content
as raw markdown or as the rendered HTML.

This commit makes it so we only have one cache entry per
message, and it includes both content and rendered_content.

One legacy source on confusion here is that `content`
changes meaning when you're on the front end.  Here is the
situation going forward:

    database:
        content = raw
        rendered_contented = rendered

    cache entry:
        content = raw
        rendered_contented = rendered

    payload for the frontend:
        content = raw (for apply_markdown=False)
        content = rendered (for apply_markdown=True)
2017-10-26 16:35:28 -07:00
derAnfaenger 1792dcbd09 tests: Call real consume method of queue processors.
This switches to more real tests for a first batch of
queue_json_publish() calls that don't cause trouble when
used with call_consume_tests=True.
2017-10-26 14:58:03 -07:00
Tim Abbott b8658c6901 default stream groups: Use cleaner system for error handling.
Wherever possible, we always want to move checking for error
conditions to the views code, so that we don't need to worry about
handling failures with (in this case) a user that's half-created
because a DefaultStreamGroup doesn't exist.
2017-10-26 11:38:27 -07:00
Vishnu Ks eecdc5bb61 registration: Add option to choose default stream groups.
This effectively implements the feature of default stream groups,
except for a UI, nice styling, etc.

Note that we're careful to not have this do anything in an
organization that doesn't have any default stream groups.
2017-10-26 11:34:06 -07:00
derAnfaenger ce4ba9c178 bot services: Use call_consume_in_tests. 2017-10-25 15:56:12 -07:00
Vishnu Ks 1d94119d31 actions: Call send_initial_pms from process_new_human_user. 2017-10-25 14:14:59 -07:00
Steve Howell 74081ff2d5 Add assert() on stream_topic in get_recipient_info(). 2017-10-24 16:02:33 -07:00
Steve Howell 6ed2a9b9f2 refactor: Rename user_ids -> message_to_user_ids.
This renaming sets the stage for the next change (and passes
all tests).
2017-10-24 14:32:17 -07:00
Tim Abbott bc3569f6ab actions: Move user activate/reactivate closer to creation.
This is just to create greater locality of this batch of code in the file.
2017-10-24 09:05:55 -07:00
Steve Howell b851020b1e Excluded topic-muted users from stream push notifications.
Fixes #7059
2017-10-23 17:26:15 -07:00
Tim Abbott 1698f979d0 invites: Call confirmation worker consumer in tests.
This makes the automated tests for this queue publish more faithful.
2017-10-19 14:02:44 -07:00
Tim Abbott 63489ebfa5 subdomains: Avoid exception when creating realm on root domain. 2017-10-18 23:51:58 -07:00
Tim Abbott 31afdd2ff9 internal_prep_message: Include traceback with sending errors.
This should make it easier to debug problems.
2017-10-18 21:08:41 -07:00
rht 691598a88b py3: Remove "from six.moves import range".
This is no longer required, since in Python 3, this is what the range
built-in does.
2017-10-17 23:28:14 -07:00
Tim Abbott 024c27dd3e notifications: Fix sending push notifications on muted streams.
Apparently, the stream muting feature did not take priority over the
stream_push_notify feature.  This fixes that gap.

Fixes zulip/zulip-mobile#1314.
2017-10-17 22:38:54 -07:00
derAnfaenger 87468f46ae queue processors: Test flow through UserActivityIntervalWorker.consume(). 2017-10-16 23:20:13 -07:00
derAnfaenger ad0407578d queue processors: Test flow through UserPresenceWorker.consume(). 2017-10-16 23:20:13 -07:00
Robert Hönig e749deb136 onboarding: Add welcome-bot response to initial user message.
Fixes #6030.
2017-10-11 20:45:42 -07:00
Steve Howell a6ad9a6d7c Add is_zephyr to the Stream model.
Add this field to the Stream model will prevent us from having
to look at realm data for several types of stream operations, which
can be prone to either doing extra database lookups or making
our cached data bloated.

Going forward, we'll set stream.is_zephyr to True whenever the
realm's string id is "zephyr".
2017-10-11 16:15:56 -07:00
Steve Howell c1d7fc6e80 Only require stream_id in private_stream_user_ids(). 2017-10-08 20:18:34 -07:00
Steve Howell 7dbea8a2bf Only require stream_id in subscribed_to_stream().
Since subscribed_to_stream is only doing an id lookup
on the Stream model to find out if a user is subscribed to
a stream, there's no reason to require a full Stream object.

It's currently the case that all callers do have full Stream
objects handy to pass in to this function, but it's still a
good practice to have functions only ask for objects that they
need.
2017-10-08 20:18:34 -07:00
Tim Abbott d215ea1e37 actions: Rename all_subs_by_stream to all_subscribers_by_stream.
The previous name sounded a bit too much like they were subcription
objects.
2017-10-08 12:33:53 -07:00
Steve Howell 3e6bfe1b23 Use user_ids, not emails, for bulk stream operations.
We now return user_ids for subscribers to streams in add-stream
events.  This allows us to eliminate the UserLite class for
both bulk adds and bulk removes.  It also simplifies some JS
code that already wanted to use user_ids, not emails.

Fixes #6898
2017-10-08 12:31:12 -07:00
Steve Howell 10a30bece1 Rename presence_idle_userids -> presence_idle_user_ids. 2017-10-07 12:16:45 -07:00
Steve Howell fbaef43ac3 Rename bot_owner_userids -> bot_owner_user_ids. 2017-10-07 12:16:45 -07:00
Steve Howell a331b4f64d Optimize query_all_subs_by_stream().
Using lightweight objects will speed up adding new users
to realms.

We also sort the query results, which lets us itertools.groupby
to more efficiently build the data structure.

Profiling on a large data set shows about a 25x speedup for this
function, and before the optimization, this function accounts
for most of the time spend in bulk_add_subscriptions.

There's a lot less memory to allocate.  I didn't measure
the memory difference.

When we test-deployed this to chat.zulip.org, we got about a 6x
speedup.
2017-10-06 11:03:44 -07:00
Steve Howell f5ddc40d14 Have get_peer_user_ids_for_stream_change() use user_ids. 2017-10-06 11:03:44 -07:00
Steve Howell aae0b2a826 Notify offline users about edited stream messages.
We now do push notifications and missed message emails
for offline users who are subscribed to the stream for
a message that has been edited, but we short circuit
the offline-notification logic for any user who presumably
would have already received a notification on the original
message.

This effectively boils down to sending notifications to newly
mentioned users.  The motivating use case here is that you
forget to mention somebody in a message, and then you edit
the message to mention the person.  If they are offline, they
will now get pushed notifications and missed message emails,
with some minor caveats.

We try to mostly use the same techniques here as the
send-message code path, and we share common code with the
send-message path once we get to the Tornado layer and call
maybe_enqueue_notifications.

The major places where we differ are in a function called
maybe_enqueue_notifications_for_message_update, and the top
of that function short circuits a bunch of cases where we
can mostly assume that the original message had an offline
notification.

We can expect a couple changes in the future:

    * Requirements may change here, and it might make sense
      to send offline notifications on the update side even
      in circumstances where the original message had a
      notification.

    * We may track more notifications in a DB model, which
      may simplify our short-circuit logic.

In the view/action layer, we already had two separate codepaths
for send-message and update-message, but this mostly echoes
what the send-message path does in terms of collecting data
about recipients.
2017-10-03 15:57:06 -07:00
Tim Abbott 654562b942 check_message: Reject null bytes in message content.
Postgres doesn't like them, we don't have an obvious way to escape
them, and they tend to be sent by buggy tools where it'd be better for
the user to get an error.

This fixes a 500 we were getting occasionally.
2017-10-03 15:32:04 -07:00
Tim Abbott 83a226ad3d do_send_message: Replace get_user_profile_by_email.
get_system_bot is the new way to do this sort of thing.
2017-10-02 16:15:36 -07:00
Eeshan Garg 763437cc9c backend: Introduce check_send_private_message.
check_send_private_message utilizes Addressee.for_user_profile()
to send a private message to a user_profile.
2017-10-02 15:27:26 -07:00
Steve Howell 2be713a7e4 Rename get_userids_for_missed_messages().
We rename this function to get_active_presence_idle_userids().
2017-10-02 15:19:28 -07:00
Steve Howell e660428c21 Rename missed_message_userids to presence_idle_userids. 2017-10-02 15:19:28 -07:00
Steve Howell 32d5e297a3 Rename get_idle_userids to filter_presence_idle_userids.
We have two different concepts of "idle", and this function
is based on the "presence" aspect of idleness.  There is also
idleness in terms of a user having no current client
descriptors accepting messages, and we check that later in
the process for things like sending missed message emails.
2017-10-02 15:19:28 -07:00
Eeshan Garg 24aff0d0a2 backend: Introduce check_send_stream_message.
check_send_stream_message is a simpler version of
check_send_message for sending messages where the addressee is
a stream. Instead of relying on Addressee.legacy_build,
check_send_stream_message uses Addressee.for_stream. Consequently,
it eschews many of check_send_message's kwargs that aren't needed
when the intended recipient of a message is a stream.
2017-09-30 17:48:55 -07:00
Steve Howell aaaaa66d4c refactor: Move default_sending_stream logic to Addressee.
Having Addressee take care of setting stream_name to
sender.default_sending_stream.name makes us able to have
the invariant that stream_name is never None when the
message type is 'stream', which will help for mypy, among
other things.

One thing to be aware of is that Addressee does do a little
bit of validation work, and this adds yet another JsonableError
exception.  I don't view this as a bad thing, just something to
know.
2017-09-28 12:14:08 -07:00
rht 035ed93111 zerver/lib: remove `import six`. 2017-09-27 19:10:28 -07:00
rht 2e12fe5e2e zerver/lib: Remove print_function. 2017-09-27 18:05:45 -07:00
Steve Howell de0b47fd4e Always notify service bots about stream mentions.
Before this change, we were only triggering service bots
for stream mentions when the bot was subscribed to the
stream.
2017-09-27 17:22:12 -07:00
Tim Abbott 28eaf5620e emails: Use common_context for email change notifications.
This also lets us remove `realm_uri`.
2017-09-27 16:48:18 -07:00
Steve Howell 1b518f1983 Return mentioned users in get_user_info_for_message_updates().
The dictionary result for get_user_info_for_message_updates()
now has a `mention_user_ids` field that is a set of user ids
who were mentioned in a message.
2017-09-27 16:01:50 -07:00
Steve Howell 646abb57b7 refactor: Extract get_user_info_for_message_updates.
We'll want to expand this to get users that were mentioned in
the prior message, but this commit is just a refactoring.
2017-09-27 16:01:50 -07:00
Steve Howell 5abf52de71 Extract get_idle_userids().
This function will help us send missed-message mails for
updates, in a future commit.
2017-09-27 16:01:50 -07:00
rht f43e54d352 zerver/lib: Remove absolute_import. 2017-09-27 10:00:39 -07:00
Steve Howell b340b28055 Extract get_service_bot_events().
There are several reasons to extract this function:

    * It's easy to unit test without extensive mocking.
    * It will show up when we profile code.
    * It is something that you can mostly ignore for
      most messages.

The main reason to extract this, though, is that we are about
to do some fairly complex splicing of data for the use case
of mentioning service bots on streams they are not subscribed to,
and we want to localize the complexity.
2017-09-26 18:49:03 -07:00
Tim Abbott c11b1623dd addressee: Accept a realm object in legacy_build.
This fixes a bug where the internal_prep_message code path would
incorrectly ignore the `realm` that was passed into it.  As a result,
attempts to send messages using the system bots with this code path
would crash.

As a sidenote, we really need to make our test system consistent with
production in terms of whether the user's realm is the same as the
system realm.
2017-09-25 14:06:29 -07:00
Tim Abbott 14c1660a55 addressee: Pass realm into get_user_profiles.
We don't access any attributes of the sender other than the realm, and
as it turns out, we in some cases want to use a different realm than
the sender's.
2017-09-25 14:00:46 -07:00
Tim Abbott 8e2c91b09c actions: Use internal_send_private_message. 2017-09-25 13:59:04 -07:00
Tim Abbott 1edd137263 RealmAuditLog: Pass acting_user to do_reactivate_user. 2017-09-22 07:33:02 -07:00
Rishi Gupta 6ec3595b77 emails: Change enqueue_welcome_emails to take a user rather than user_id. 2017-09-22 06:20:33 -07:00
Rishi Gupta a7c8770f97 emails: Move enqueue_welcome_emails outside of signups queue.
The only thing this queue should do is sign you up for the newsletter, since
it is only populated if newsletter_data is not None.
2017-09-22 06:20:33 -07:00
Tim Abbott f706f657c0 signup: Fix invitation emails not being cleared properly.
Previously, invitation reminder emails were only being cleared after a
successful signup if newsletter_data was available, since that was the
circumstance in which we were calling the relevant queue processor
code.  Now, we (1) clear them when a human user finishes signing up
and (2) correctly clear them using the 'address' field of
ScheduleEmail, not user_id.
2017-09-21 06:15:11 -07:00
Steve Howell 428d3027c2 Only require ids for finding DefaultStream objects.
We don't need full Realm objects to find DefaultStream
objects for a realm.  So now a few functions related to
adding/removing default streams use realm_id for lookups.

Similarly, we don't need a full Stream object to find
out if a stream exists in DefaultStream, so we do id
lookups there as well.

This sets us up to use thinner objects in callers.
2017-09-20 10:31:33 -07:00
Steve Howell fd58d472a5 Use update_fields in do_deactivate_stream.
We are generally explicit about which fields we save.
2017-09-20 10:31:33 -07:00
Steve Howell 7b2340decd Extract stream_name_in_use().
Checking to see if a stream exists is more idiomatic
if we just use exists() from Django.  We encapsulate it
for case insensitivity purposes.
2017-09-20 10:31:33 -07:00
Steve Howell 9046efb71a Use `prereg.users.set()` in do_invite_users.
This is a bit more idiomatic for many-to-many relationships.
2017-09-20 10:31:33 -07:00
Steve Howell 8ad7133351 Cache active_user_ids() more directly.
We now have a dedicated cache for active_user_ids() that only
stores a list of user_ids.

Before this commit, active_user_ids() used a cache of UserProfile
dictionaries, so it incurred unnecessary deserialization costs for
all the user fields that it sliced away in a list comprehension.

Because the cache is skinnier here, we also need to invalidate it
less frequently.  Basically, all we care about is new users, realm
deactivations, and user deactivations.

It's hard to measure how much this will improve performance, because
the speedup for any operation here is pretty minor, but we use this
function a lot, so hopefully it will make the overall system more
healthy.
2017-09-20 10:31:33 -07:00
Steve Howell cad3a35b6a Only require realm_id for active_user_ids().
This is mostly a preparatory commit for an upcoming optimization
related to stream data, but it probably does save us an
occasional DB hop to the realm table.
2017-09-20 10:31:33 -07:00
Steve Howell 26735eeeac Only require realm_id for get_active_user_dicts_in_realm().
This is a preparatory commit that will eventually allow us
to avoid fetching realm info that we don't need, in other
parts of the codebase.
2017-09-20 10:31:33 -07:00
Steve Howell 0966bf1a48 Simplify get_stream_cache_key().
Before this commit, we could pass in either a Realm object
or a realm_id to get_stream_cache_key().  Now we consistently
pass it a realm_id.
2017-09-20 10:31:33 -07:00
Greg Price 0c7dbd2e8a message send: Cut is_active from the values query in get_recipient_info.
This is unused since the query started filtering on is_active=True, in
51d4f16fe "Ignore inactive users in get_recipient_info()."
2017-09-19 20:08:39 -07:00
Steve Howell c4b17f2f80 Optimize get_recipient_info() with get_ids_for().
The get_ids_for() function produces a 2x speedup for
1000 users.
2017-09-16 03:07:13 -07:00
Steve Howell 84041d3195 Use itertools.groupby in bulk_get_subscriber_user_ids().
This results in about a 20% speedup by making more O(N)
things happen in C vs. Python.
2017-09-15 10:44:32 -07:00
Steve Howell 24b9f72b22 Use raw SQL in bulk_get_subscriber_user_ids().
This leads to more than a 2x speedup when tested with
20k+ total subscribers.  (For large realms with lots of default
streams, this function deals with LOTS of data, so it is important
to optimize.)
2017-09-15 10:44:32 -07:00
Steve Howell 1553dc00e0 Introduce StreamRecipient class.
This class encapsulates the mapping of stream ids to
recipient ids, and it is optimized for bulk use and
repeated use (i.e. it remembers values it already fetched).

This particular commit barely improves the performance
of gather_subscriptions_helper, but it sets us up for
further optimizations.

Long term, we may try to denormalize stream_id on to the
Subscriber table or otherwise modify the database so we
don't have to jump through hoops to do this kind of mapping.
This commit will help enable those changes, because we
isolate the mapping to this one new class.
2017-09-15 10:44:32 -07:00
Steve Howell fc2e485ca7 Sort emails in gather_subscriptions().
This helps makes the tests more deterministic.
2017-09-15 10:44:32 -07:00
Steve Howell 51d4f16fe0 Ignore inactive users in get_recipient_info().
We were mostly excluding inactive users before this fix, but
now we completely ignore them.

This potentially changes some of the data we return from
get_recipient_info(), but the extra user ids before this fix
were effectively ignored by the caller.
2017-09-15 03:08:52 -07:00
Steve Howell 1759137e4f Don't queue feedback unless the bot is active.
The prior code would queue up feedback messages even if the
feedback bot was deactivated, which was just due to oversight
most likely.  (People probably rarely disable the feedback bot,
but they should have that option.)
2017-09-15 03:08:52 -07:00
Tim Abbott 5722237f59 push: Rename received_pm to private_message.
This is a clearer name for this now more broadly used interface.
2017-09-14 05:41:37 -07:00
Sarah c3a8138f74 user_settings: Add push notifications for all stream messages.
Add setting to enable push notifications for all stream messages.
2017-09-14 05:41:37 -07:00
Steve Howell 41e3a819da Inline get_recipient_user_ids() into two callers.
This sets us up a subsequent commit where we need more data
from the Subscription table to build recipient info, so the
function boundary doesn't work any more for get_recipient_info,
which is part of the heavily optimized send-message
path.

We used to share code here with typing notifications, but
typing notifications need a lot less data than the
send-message path, so it's useful to decouple these two
things.  The idioms that are duplicated here are pretty simple
one-liners.
2017-09-14 05:13:58 -07:00
Steve Howell 6c90940f84 performance: Add UserMessageLite class.
This speeds up sending messages significantly.

For 1000 users, this speeds up create_user_messages from
0.652s to 0.0558s, so basically a 10x speedup.
2017-09-12 04:22:55 -07:00
Steve Howell 811fcf51ee Extract create_user_messages.
The logic to create UserMessage rows when you create a message
is very self-contained, and it's helpful to be able to profile it.
2017-09-12 04:22:55 -07:00
Steve Howell 7fbffb8e30 Optimize bulk inserts for UserMessage rows.
Avoiding ORM overhead makes inserting UserMessage rows
about 15 times faster.
2017-09-12 04:22:55 -07:00
Steve Howell d723be125a Optimize get_recipient_info() for sending messages.
This commit makes get_recipient_info() faster by never creating
Django ORM objects.  We use the ORM to create a values query
instead, and then we iterate over the rows to create various
collections of ids.

In order to avoid lots of code duplication, this commit unifies
how we query UserProfile for PMs and streams.  Prior to this
commit we were getting "wide" UserProfile objects out of
our memcached cache.  Now we just go to the database with our
list of userids.  The new approach at worst adds one hop to the
database for PMs, which aren't really a performance bottleneck
(compared to streams).  And the new approach actually saves a
hop when both partners aren't in cache (plus we don't pay the
penalty of hitting the cache itself).

The performance improvement here is easy to measure for messages
to streams with many users, even with all the other activity
that goes on inside do_send_messages().  I took test_performance()
in test_messages.py, set num_extra_users to 3000, and consistently
measured a ~20% speedup in do_send_messages().

This commit also eliminates fetching of emails.  We probably
could have done that in a prior commit, but in this commit it
is very explicit that we don't need it.  While removing email
from the query is a no-brainer, it actually had a negigible
impact on performance.  Almost all the savings here comes from
not create UserProfile objects.
2017-09-12 04:22:55 -07:00
Steve Howell d00c001b5f Create get_recipient_info().
This function returns a summary of recipient data for a message
that's being sent.  It's mostly just moving code into the
old function called get_recipient_user_profiles().
2017-09-12 04:22:55 -07:00
Steve Howell b562dedb53 Avoid using email to detect that the feedback bot is addressed.
This commit is necessary to prevent bringing back emails from the
DB for all N recipients of a message just to see if the feedback
bot is being invoked.
2017-09-12 04:22:55 -07:00
Steve Howell 6f0289ae79 do_send_messages(): Extract internal push_notify_user_ids set.
This is one more step toward not needing UserProfile objects.
2017-09-12 04:22:55 -07:00
Steve Howell 82b2bd8b65 Take user_ids in get_userids_for_missed_messages().
This helps us phase out the need for getting lots of UserProfile
objects.
2017-09-12 04:22:55 -07:00
Steve Howell 06c388774f do_send_messages(): Clean up service bot code.
We calculate `service_bot_tuples` earlier in the function, so that
we don't need "full" UserProfile objects later in the function.

This is part of consolidating code that basically just needs to
triage user_ids.
2017-09-12 04:22:55 -07:00
Steve Howell a22a22966f do_send_messages(): Create UserMessage objects with user_id.
This starts to phase out the need for UserProfile objects in
do_send_messages().  UserProfile objects are expensive to create
for large streams with lots of users.  The objects in the code
before this commit aren't even full UserProfile objects.

This change mostly sets up future performance improvements, but
we also get a minor speedup here when we run a test with 3000
stream subscribers.
2017-09-12 04:22:55 -07:00
Steve Howell ba397b5109 Use user_ids, not full objects, in render path.
There is no reason for either render_incoming_message() or
render_markdown() to require full UserProfile objects just to
triage alert words.

By only asking for user_ids, we save extra queries in two
callpaths and we make it easier to start using user_ids in
do_send_messages().
2017-09-12 04:22:55 -07:00
Steve Howell 9e8c24168d Extract get_typing_user_profiles().
This function is essentially a copy of get_recipient_user_profiles,
which is about to go away. The new function enforces the contract of
typing indicators, which is that they don't apply to streams, which
allows us to use a relatively simple approach for getting user
profile objects.

We are diverging this code, because the send-message path needs
more optimizations.
2017-09-12 04:22:55 -07:00
Steve Howell c87cc1447f Extract get_recipient_user_ids. 2017-09-12 04:22:55 -07:00
Steve Howell 56a552eec3 Get UserProfile objects directly for stream messages.
This change introduces an extra hop to the database, but it is
generally faster due to nuances of the DB and the ORM.  It
also sets us up to optimize get_recipient_user_profiles() by
avoiding creating ORM objects.

I measured the impact of this using a stream with 3000
subscribers, half of whom were idle, and it speeds things up
by 10%.
2017-09-12 04:22:55 -07:00
Steve Howell f5edeb01ae Calculate idle users more efficiently when sending messages.
Usually a small minority of users are eligible to receive missed
message emails or mobile notifications.

We now filter users first before hitting UserPresence to find idle
users.  We also simply check for the existence of recent activity
rather than borrowing the more complicated data structures that we
use for the buddy list.
2017-09-07 06:59:44 -07:00
Steve Howell 97c5f085e7 minor: Extract locals in do_send_messages().
This is a prepartory commit for another refactoring.
2017-09-07 06:59:44 -07:00
Steve Howell 4ac6bc46c7 Add MutedTopic model.
This commit completely switches us over to using a
dedicated model called MutedTopic to track which topics
a user has muted.

This includes the necessary migrations to create the
table and populate it from legacy data in UserProfile.

A subsequent commit will actually remove the old field
in UserProfile.
2017-09-02 09:19:51 -07:00
Steve Howell 0501570cd1 Remove POST-based API for setting topic mutes. 2017-08-29 16:53:38 -04:00
Steve Howell 73c30774cb admins: Add private streams to never_subscribed.
Admins need to know about private streams to delete them, even
if they are not subscribed.  We send the minimal info possible
to the client to allow them to have a UI for that.
2017-08-27 19:08:04 -07:00
Tim Abbott 133f005530 markdown: Remove is_me_message UserMessage flags.
This never made sense to be a flag on the UserMessage table, since
it's not per-user state.  And in fact it doesn't need to be in a
database at all, since it's easily computed from content anyway.

Fixes #1099.
2017-08-27 09:34:24 -07:00
Aditya Bansal f0c4f4d4bc soft-deactivation: Stop creating flag list for soft-deactivated users. 2017-08-26 13:48:16 -07:00
Tim Abbott f3c73e4aa4 actions: Add assertion for invalid flag operations. 2017-08-25 00:34:56 -07:00
Tim Abbott 35f1bbb968 process_new_human_user: Simplify prereg_user logic.
We don't need the old code paths now that we don't have an MitUser
model anymore.
2017-08-25 00:34:06 -07:00
Tim Abbott 2aab6e0f49 forms: Replace is_inactive with more comprehensive check.
While we're at it, we clean up the old confusing error messages.
2017-08-24 23:16:31 -07:00
Tim Abbott b3dbba3ad4 actions: Extract validate_email_for_realm helper. 2017-08-24 23:16:31 -07:00
Steve Howell 87c4961597 Add zerver/lib/topic_mutes.py
This is mostly pure code extraction.

It also removes some dead code in update_muted_topic, where
were updating muted_topics spuriously before calling
do_update_muted_topic.
2017-08-24 14:20:35 -07:00
Tim Abbott d3e3c704d4 do_create_realm: Remove unnecessary second return value.
Unlike creating a stream, there's really no reason one would want to
call the function to create a realm while uncertain whether that realm
already existed.
2017-08-23 20:07:17 -07:00
Vishnu Ks 58ad61a4ac actions: Fix errors when notifications_stream is deactivated. 2017-08-23 17:53:52 -07:00
Rishi Gupta 7c3f20d2ba registration: Set is_realm_admin on user creation.
This makes it easier for later parts of the user creation/onboarding process
to condition on whether the user is a realm admin.

No change in behavior.
2017-08-22 14:26:17 -07:00
Steve Howell 81e3f489f2 Use sender realm in user_profiles_from_unvalidated_emails.
This change is mostly based on a similar commit from hackerkid
in a feature branch.  It borrows both code and ideas.  Some of
it's my own stuff, as I was working on a newer branch.

We now call get_user_including_cross_realm_email() inside of
user_profiles_from_unvalidated_emails(), instead of using
get_user_profile_by_email.

This requires a few of our callers to pass down sender into us.

One consequence of this change is that we change the symptoms
for trying to send to emails outside of your realm.  In some
cases, we simply raise an error that an email is invalid to us
instead of getting into the deeper validate_recipient_user_profiles
check.
2017-08-22 10:42:15 -07:00
Steve Howell 54edecd510 Replace adddressee.for_email() with for_user_profile().
This requires us to change not just the immediate caller, but
also some of their callers, to pass user_profile objects around
instead of emails.
2017-08-22 10:42:15 -07:00