Commit Graph

575 Commits

Author SHA1 Message Date
Steve Howell 828459a24b Extract build_topic_mute_checker into topic_mutes.py.
We had two duplicate versions of this function, and one
of them was broken with respect to case insensitivity.
2017-08-29 16:53:38 -04:00
Yago González 659bff1ffb i18n: Fix URLs misparsing in translation tags.
The double forward slash (//) after the protocol in URLs was being
mistakenly considered the beginning of an inline JS comment, causing
internationalization strings being cut unexpectedly.

Now the check for inline JS comments is only run in .js files.
2017-08-28 13:54:17 -07:00
Aditya Bansal d9c9bfe7f6 logger: Add new create_logger abstraction to simplify logging.
This deduplicates a ton of Python logger-creation code to use a single
standard implementation, so we can avoid copy-paste problems.
2017-08-27 18:31:53 -07:00
Vishnu Ks 8fc8ac0799 management: Override CommandError to mention --entire-server argument. 2017-08-27 12:34:23 -07:00
Aditya Bansal b232563e12 soft-deactivation: Add cron job for weekly soft deactivating users. 2017-08-27 11:33:06 -07:00
Aditya Bansal 9d7e23c100 softdeactivation/management: Make specifying realm an optional arg. 2017-08-27 11:33:06 -07:00
Preston Hansen e8a608f733 management: Move enqueue_digest_email handler to digest. 2017-08-27 10:13:11 -07:00
Preston Hansen 9a4b17cf9b management: Move queue_digest_recipient to digest. 2017-08-27 10:13:11 -07:00
Preston Hansen 2aabf4fc67 management: Move should_process_digest to digest. 2017-08-27 10:13:11 -07:00
Preston Hansen 25a40806df management: Move inactive_since to digest. 2017-08-27 10:13:11 -07:00
Greg Price d02101a401 APNs: Rip out the existing, broken implementation.
This code empirically doesn't work.  It's not entirely clear why, even
having done quite a bit of debugging; partly because the code is quite
convoluted, and because it shows the symptoms of people making changes
over time without really understanding how it was supposed to work.

Moreover, this code targets an old version of the APNs provider API.
Apple deprecated that in 2015, in favor of a shiny new one which uses
HTTP/2 to meet the same needs for concurrency and scale that the old
one had to do a bunch of ad-hoc protocol design for.

So, rip this code out.  We'll build a pathway to the new API from
scratch; it's not that complicated.
2017-08-26 14:16:05 -07:00
Tim Abbott eeabed9119 models: Add new get_user_profile_by_api_key helper.
This results in a slight performance increase.
2017-08-24 23:17:08 -07:00
Tim Abbott a3bc041894 send_password_reset_email: Simplify argument parsing. 2017-08-24 14:41:45 -07:00
Vishnu Ks 2b076ef822 management: Move all_users option to ZulipBaseCommand. 2017-08-24 14:35:18 -07:00
Vishnu Ks 9d043e7fb0 management: Use self.style attribute for colored output. 2017-08-24 14:29:32 -07:00
Vishnu Ks f4df74d968 management: Use add_user_list_args in send_password_reset_email. 2017-08-24 14:29:32 -07:00
Vishnu Ks 65a20b3b99 management: Use add_user_list_args in remove_users_from_stream. 2017-08-24 14:29:32 -07:00
Vishnu Ks 0949924ea3 management: Use add_user_list_args in add_users_to_streams. 2017-08-24 14:29:32 -07:00
Vishnu Ks a82b3f1ea5 management: Use add_user_list_args in turn_off_digests. 2017-08-24 14:29:32 -07:00
Tim Abbott 8cbdaca9f5 fix_unreads: Fix handling of muted_topics on missing streams.
If the stream no longer exists, we can just ignored the muted_topics
entry.
2017-08-23 18:14:11 -07:00
Tim Abbott 7918551fa9 fix_unreads: Help postgres figure out the most efficient query.
In theory, this shouldn't make a difference, but in practice, this
helps postgres make better use of our unread messages index.
2017-08-23 18:00:33 -07:00
Tim Abbott 75d9f66bf0 fix_unreads: Fix error handling with no recipient_ids.
Previously, this would send an invalid database query to postgres.
2017-08-23 17:15:33 -07:00
Steve Howell bc61fae6d2 fix_unreads: Actually update pre-pointer unread counts.
This fixes an oversight in the original version.  (We have only
used this in controlled experiments.)
2017-08-23 13:13:23 -07:00
Tim Abbott 669b1b8adb mypy: Fix an annotation. 2017-08-22 17:39:24 -07:00
Steve Howell 110cfcb640 Support --all options in fix_unreads.
(We also changed the email metavar to accept multiple emails.)
2017-08-22 20:06:30 -04:00
Steve Howell 752bf14443 fix_unreads: Improve cursor handling.
Before adding the context manager, we were leaking cursors, which
was a non-issue for updating individual users, but it's easy to
fix.
2017-08-22 20:06:26 -04:00
Steve Howell c5e083857b fix_unreads: Mark pre-pointer messages as unread (unless muted). 2017-08-22 20:03:18 -04:00
Steve Howell 8b16c6060c Split out fix_unreads management command.
Most of the code in show_unreads is for diagnosising unread
counts issues, and we may not use that often.

We're creating a dedicated fix_unreads management command with
less clutter.
2017-08-22 19:38:44 -04:00
Preston Hansen c31ef34491 enqueue_digest_emails: Only send digest emails weekly, on Tuesday.
Fixes: #6114.
2017-08-22 15:07:53 -07:00
Greg Price a099e698e2 py3: Switch almost all shebang lines to use `python3`.
This causes `upgrade-zulip-from-git`, as well as a no-option run of
`tools/build-release-tarball`, to produce a Zulip install running
Python 3, rather than Python 2.  In particular this means that the
virtualenv we create, in which all application code runs, is Python 3.

One shebang line, on `zulip-ec2-configure-interfaces`, explicitly
keeps Python 2, and at least one external ops script, `wal-e`, also
still runs on Python 2.  See discussion on the respective previous
commits that made those explicit.  There may also be some other
third-party scripts we use, outside of this source tree and running
outside our virtualenv, that still run on Python 2.
2017-08-16 17:54:43 -07:00
Tim Abbott d43f5ceeec database: Add database index for wildcard mentions. 2017-08-16 13:28:04 -07:00
Tim Abbott 8bb812c8a9 database: Add database index for alert words. 2017-08-16 12:39:01 -07:00
Steve Howell 3a825ef22c Add create_large_indexes management command.
This management command creates the same indexes as migrations
82, 83, and 95, which are all indexes on the huge UserMessage
table. (*)

This command quickly no-ops with clear messaging when the
indexes already exist, so it's idempotent in that regard. (If
somebody somehow creates an index by the same name incorrectly,
they can always drop it in dbshell and re-run this command.)

If any of the migrations have not been run, which we detect simply
by the existence of the indexes, then we create them using a
`CREATE INDEX CONCURRENTLY` command.  This functionality in
postgres allows you to create indexes against large tables
without disrupting queries against those tables.  The tradeoff
here is that creating indexes concurrently takes significantly
longer than doing them non-concurrently.

Since most tables are small, we typically just use regular
Django migrations and run them during a brief interval while
the app is down.

For indexes on big tables, we will want to run this command
as part of the upgrade process, and we will want to run
it while the app is still up, otherwise it's pointless.

All the code in create_indexes() is literally copy/pasted
from the relevant migrations, and that scheme should work
going forward.  (It uses a different implementation of
create_index_if_not_exist than the migrations use, but the
code is identical lexically in the function.)

If we ever do major restructuring of our large tables, such
as UserMessage, and we end up droppping some of these indexes,
then we will need to make this command migrations-aware.  For
now it's safe to assume that indexes are generally additive in
nature, and the sooner we create them during the upgrade process,
the better.

(*) UserMessage is huge for large installations, of course.
2017-08-16 12:38:51 -07:00
Aditya Bansal 0716702558 management: Log number of soft deactivated/reactivated users. 2017-08-15 22:05:19 -07:00
Aditya Bansal 5f5d99550e soft-deactivation/management: Raise exception for non existant emails. 2017-08-15 22:05:19 -07:00
Aditya Bansal cd11cbea14 soft-deactivation/management: Rename variable to make more sense.
We rename variable user_ids_to_deactivate to users_to_deactivate
since it was storing UserProfiles anyway.
2017-08-15 22:05:19 -07:00
Aditya Bansal a33790946b management: Add soft_deactivate_users command. 2017-08-15 08:33:16 -07:00
Vishnu Ks f0bb11ef40 management: Use self.get_user in convert_bot_to_outgoing_webhook. 2017-08-08 14:13:19 -07:00
Vishnu Ks d1bece2ec2 management: Use self.get_realm in create_stream. 2017-08-08 14:13:19 -07:00
Vishnu Ks 9fb5381acb management: Use self.get_realm in set_default_streams. 2017-08-08 14:13:19 -07:00
Vishnu Ks 3bc475d04f management: Use self.get_realm in rename_stream. 2017-08-08 14:13:19 -07:00
Vishnu Ks 4e7dbff197 management: Use self.get_realm in realm_filters. 2017-08-08 14:13:19 -07:00
Vishnu Ks 7859ec76cf management: Use self.get_realm in realm_emoji. 2017-08-08 14:13:19 -07:00
Vishnu Ks af1c1861da management: Use self.get_realm in realm_domain. 2017-08-08 14:13:19 -07:00
Vishnu Ks 1b0b135bfc management: Remove unused imports from process_queue. 2017-08-08 14:13:19 -07:00
Vishnu Ks 0532b35c43 management: Remove unused imports from print_email_delivery_backlog. 2017-08-08 14:13:19 -07:00
Vishnu Ks 164badde13 management: Use self.get_realm in merge_streams.
The variable encoding is removed becuase it is not used
anywhere.
2017-08-08 14:13:19 -07:00
Vishnu Ks 27936fcfd4 management: Remove unused imports from makemessages. 2017-08-08 14:13:19 -07:00
Vishnu Ks cf61349563 management: Use self.get_realm in logout_all_users. 2017-08-08 14:13:19 -07:00
Vishnu Ks e28a007487 management: Use self.get_user in knight command. 2017-08-08 14:13:19 -07:00
Vishnu Ks 3a2edc66f1 management: Use get_system_bot in initialize_voyager_db. 2017-08-08 14:13:19 -07:00
Vishnu Ks f45b80aafe management: Remove unused imports from initialize_voyager. 2017-08-08 14:13:19 -07:00
Vishnu Ks 05af206392 management: Remove unused imports from import command. 2017-08-08 14:13:19 -07:00
Vishnu Ks f58efdc3cb management: Remove unused imports from export_usermessage_batch. 2017-08-08 14:13:19 -07:00
Vishnu Ks 28e76c990a management: Remove unused imports from email_mirror command. 2017-08-08 14:13:19 -07:00
Vishnu Ks 6687b236bf management: Remove unused imports from deliver_email command. 2017-08-08 14:13:19 -07:00
Vishnu Ks b8dacf9bf6 management: Remove unused import from checkconfig command. 2017-08-08 14:13:19 -07:00
Vishnu Ks 5ce0d85ccf management: Remove unused import from check_redis command. 2017-08-08 14:13:19 -07:00
Vishnu Ks 19f8da4995 management: Use self.get_realm in export command. 2017-08-08 14:13:19 -07:00
Vishnu Ks 4e9dff09fd management: Use self.get_realm in dump_messages. 2017-08-08 14:13:19 -07:00
Vishnu Ks bc48c68c09 management: Use self.get_realm in reactivate_realm command. 2017-08-08 14:13:19 -07:00
Vishnu Ks 279af33ea2 management: Use self.get_realm in deactivate_realm. 2017-08-08 14:13:19 -07:00
Vishnu Ks 349d604ad5 management: Use self.get_realm in create_user. 2017-08-08 14:13:19 -07:00
Rishi Gupta ddbf8b9c3c management: Remove create_realm.py.
This no longer does the correct thing (in terms of onboarding emails,
default streams, etc), and is tempting for new server admins to use.

Once we remove it we'll also have the invariant that we can't have a realm
without a user, which will simplify accounts_register a bit.
2017-08-05 10:17:14 -07:00
Vishnu Ks e7f5da83ac management: Create list realms command.
Fixes #5917.
2017-08-04 14:25:03 -07:00
Steve Howell bc6543a236 Use do_mark_all_as_read() in bankrupt_users command. 2017-08-04 14:10:46 -07:00
Steve Howell c5e4ce5278 Optimize show_unreads --fix approach.
This now breaks the process of cleaning up unread counts for
non-active streams into a three step process.

This allows us to use our unread message flags index, at least
in testing on dev.  Here is the relevant excerpt from explain
analyze:

    Bitmap Index Scan on zerver_usermessage_unread_message_id
2017-08-04 11:04:43 -07:00
Greg Price cc30226af0 deliver_email: When disabled, sleep forever rather than exit.
This makes supervisor see the service as cheerfully running
and let it alone, rather than constantly retry starting it.
Because the crash/restart loop means repeatedly spending a
couple of seconds loading Django and the app, separated by
brief periods while supervisor notices the crash and acts
on it, it was actually consuming about 30-50% CPU on the
zulipchat.com staging server.
2017-08-03 16:28:19 -07:00
Steve Howell e994ff36b5 Add option to fix unread messages for inactive streams. 2017-08-03 13:17:04 -07:00
Steve Howell dad1c9326e Add show_unreads management command.
This should help us troubleshoot anomalies with old unread
messages.
2017-08-03 13:17:04 -07:00
neiljp (Neil Pilgrim) 1c7d0c932a mypy: Ensure type of trim_func is consistent in check_redis.py. 2017-08-03 11:03:14 -07:00
Umair Khan 9e33917d25 rate_limiter: Upgrade max_api_calls to generic API. 2017-08-02 18:01:39 -07:00
Umair Khan ca9ac1948e rate_limiter: Upgrade max_api_window to generic API. 2017-08-02 18:01:39 -07:00
Umair Khan 9c08b62448 rate_limiter: Upgrade unblock_user to generic API.
Change name to unblock_access as well.
2017-08-02 18:01:39 -07:00
Umair Khan 23be0875a1 rate_limiter: Upgrade block_user to generic API.
Change name to block_access as well.
2017-08-02 18:01:39 -07:00
rht ca960da58f create_user: Receive password input from a file instead of shell arg.
This is to be used for the case of container orchestration instead of
shell arg to prevent snooping by any user account on the server via `ps
-ef` or any superuser with read access to the user\'s bash history.
2017-07-27 18:08:52 -07:00
Umair Khan 758dbec9e2 tornado: Move setup_tornado_rabbitmq to application.py 2017-07-21 09:55:25 +05:00
Tim Abbott 240f37e8c4 management: Add script to merge two streams.
This is intended to be used occasionally when there's a really good
reason to do so.  While it does work, it hasn't been extensively tested.
2017-07-18 21:08:24 -07:00
Rishi Gupta 0f4b71b766 confirmation: Liberate get_link_for_object from ConfirmationManager. 2017-07-17 23:18:47 -07:00
Rishi Gupta aa845e7f60 models: Replace ScheduledJob with ScheduledEmail.
ScheduledJob was written for much more generality than it ended up being
used for. Currently it is used by send_future_email, and nothing
else. Tailoring the model to emails in particular will make it easier to do
things like selectively clear emails when people unsubscribe from particular
email types, or seamlessly handle using the same email on multiple realms.
2017-07-17 16:05:38 -07:00
Rishi Gupta 898269bbac email: Change send_email to raise exception on failure.
More in line with how we do error handling in the rest of Zulip.
2017-07-16 16:56:39 -07:00
Rishi Gupta 3d24d12ba1 emails: Change reset password emails to use to_user_id. 2017-07-16 16:56:39 -07:00
Rishi Gupta 154d37afd2 emails: Add to_user_id argument to send_email.
Both the queue processor and ScheduledJob emails need to sometimes pass a
to_user_id and sometimes pass a to_email, and it's more convenient to just
have one function that they can call that can handle either.

Also removes the now redundant send_email_to_user.
2017-07-16 16:56:39 -07:00
James Rowan 69f3ca7870 emails: Make password reset emails come from 'Zulip Account Security.' 2017-07-13 14:50:36 -07:00
Vishnu Ks 48617c5915 send_password_reset_email: Replace get_user_profile_by_email with get_user. 2017-07-11 11:48:21 -07:00
Vishnu Ks aca4d18f23 generate_invite_links: Replace get_user_profile_by_email with get_user.
The extra refactoring is because the realm is made a compulsary
argument.
2017-07-11 11:48:21 -07:00
Umair Khan c31beec9e6 makemessages: Support only Django version >= 1.11. 2017-07-10 11:32:10 -07:00
Vishnu Ks 8dd2268c3b set_message_flags: Replace get_user_profile_by_email with get_user. 2017-07-08 14:40:55 -07:00
Vishnu Ks 0e76d95b9f export_single_user: Replace get_user_profile_by_email with get_user. 2017-07-08 14:38:45 -07:00
Vishnu Ks afc41dfb6b rate_limit: Replace get_user_profile_by_email with get_user. 2017-07-08 14:38:45 -07:00
Vishnu Ks 1ce6b8db8c remove_users_from_stream: Replace get_user_profile_by_email with get_user. 2017-07-08 14:38:45 -07:00
Vishnu Ks 038f2d669a turn_off_digests: Replace get_user_profile_by_email with get_user. 2017-07-08 14:38:45 -07:00
Vishnu Ks fdf3df4154 bulk_change_user_name: Replace get_user_profile_by_email with get_user. 2017-07-08 14:38:45 -07:00
Vishnu Ks 3b6d5fd80e bankrupt_users: Replace get_user_profile_by_email with get_user. 2017-07-08 14:38:45 -07:00
Vishnu Ks a5d2576801 deactivate_users: Replace get_user_profile_by_email with get_user. 2017-07-08 14:38:45 -07:00
Tim Abbott b01e6d750f management: Remove obsolete update_permissions management command.
This was part of the old Django/guardian-based permissions system.
2017-07-08 14:37:25 -07:00
Tim Abbott 3395f38bf7 management: Remove obsolete gravatar_to_user_avatar management command.
This was used for a 1-time migration in 2013, and definitely isn't
needed anymore.
2017-07-08 14:35:53 -07:00
Vishnu Ks 36972f611c add_users_to_streams: Remove unused imports. 2017-07-08 14:35:42 -07:00
Vishnu Ks 4340b94a2d create_user: Separate password help into multiple lines.
This line is too long and ./tools/lint --pep8 is failing because
of this.
2017-07-08 14:18:49 -07:00
rht 7a9de7766b Add password argument to `manage.py create_user`.
Tweaked by tabbott to provide clear --help output recommending against
setting passwords this way.
2017-07-08 10:59:02 -07:00