Commit Graph

706 Commits

Author SHA1 Message Date
Tim Abbott b7c3b4df0c migration_status: Switch to passing the path in Python.
This is an important prerequisite to changing these paths used in
provisioning.
2017-10-17 21:15:58 -07:00
Tim Abbott 15c0380a33 export: Fix exporting with an output file at a relative path.
Fixes #7051.
2017-10-17 20:47:29 -07:00
Tim Abbott 66a3f514d4 zilencer: Remove long-obsolete Deployment model.
This was a precursor to RemoteZulipServer, which is no longer used for
anything, and was cluttering the codebase.
2017-10-11 23:48:01 -07:00
Tim Abbott 49274a060c send_webhook_fixture_message: Fix to work with subdomains.
We apparently failued to update this tool when we switched the
development environment to require a subdomain.
2017-10-08 20:14:14 -07:00
rht fbd24669a4 Cleanup: Remove shebang and executable flag from these files. 2017-10-08 15:38:15 -07:00
derAnfaenger 747c283180 lint: Replace `pk` with `id` where they are equivalent. 2017-10-06 12:52:12 -07:00
Tim Abbott 0f2e4a55c0 soft deactivation: Shorten management command name.
This command is really for soft deactivation; there's just an undo
feature.
2017-10-06 08:48:43 -07:00
Tim Abbott ca59b28fc1 import: Fix positional argument parsing. 2017-10-06 08:45:22 -07:00
Tim Abbott bfedb82097 import: Fix argparse usage message breakage. 2017-10-06 08:41:01 -07:00
Umair Khan 326a6f6b4f i18next: Don't allow empty string as valid translation.
Previously we used to mark a key as unstranlated if its value was equal
to it in translations.json. This had an issue because it didn't allow
otherwise valid cases where key was equal to the value.

This commit solves the problem by disallowing an empty string as a valid
translation and then using the empty string as the value for all the
unstranslated keys.

Fixes #5261
2017-10-05 22:44:34 -07:00
Umair Khan b464fd7643 makemessages: Change type of translation_strings to List.
In the code this variable is only used as a list, however, we
create it as dictionary which uses more memory.
2017-10-05 22:44:34 -07:00
Tim Abbott 13bb546ddd subdomains: Hardcode can_add_realm_domains=True.
It was already always True if REALMS_HAVE_SUBDOMAINS, and now that's
the only supported configuration.
2017-10-02 16:32:10 -07:00
derAnfaenger d1afab7199 Replace deprecated Logging.warn calls with Logging.warning. 2017-10-02 11:11:42 +02:00
rht 6dd94becb3 zerver/management: Remove unused optparse import. 2017-09-30 09:22:08 -07:00
rht f22daeba1a Cleanup rabbitmq queues when doing dev db rebuild.
Modified by tabbott to cleanup the argument handling in the modified
purge_queue management command.

Fixes #1335.
2017-09-28 18:18:54 -07:00
rht 32650c8fdc zerver/management: Remove print_function. 2017-09-27 18:05:45 -07:00
rht 0fa3c1d876 zerver/management: Remove `import six`. 2017-09-27 17:06:16 -07:00
rht e239e97351 zerver/management: Remove absolute_import. 2017-09-27 10:00:39 -07:00
neiljp (Neil Pilgrim) ab4cbf81f0 mypy: Add assertions of get_realm results when parser requires realm. 2017-09-26 13:49:01 -07:00
Greg Price 7737326cf0 mypy: Add an Optional in fix_unreads command. 2017-09-26 13:49:01 -07:00
Greg Price c10aca1bbf soft deactivation: Simplify logic slightly in management command.
The `get_realm` method works properly regardless of whether the
'realm_id' option was set, so checking for that option first is
redundant.
2017-09-26 13:49:01 -07:00
neiljp (Neil Pilgrim) acf7e2a9ee mypy: Set conditional to be against result of get_realm, not input.
[Tweaked by greg to use explicit `is not None`.]
2017-09-26 13:49:01 -07:00
Vishnu Ks f3b4de822b command: Create multiuse user invite generation command. 2017-09-22 19:48:46 -07:00
Tim Abbott 8badd10a6c send_test_email: Remind users to restart the server after updates. 2017-09-22 13:32:22 -07:00
Umair Khan 1f93c06b76 i18n: Optimize get_language_list().
compilemessages command now does all the heavy lifting by creating a
language_name_map.json file under locale directory. This file is used
by get_language_list to retrieve the require information.

Fixes: #6486
2017-09-14 02:28:58 -07:00
Steve Howell 69203c1c81 fix_unreads: Remove commit() call in fix().
The commit() call in fix() breaks migrations and tests (unless you
mock) due to outer transactions.

We now explicitly call commit() from the management command.
2017-09-07 07:06:03 -07:00
Steve Howell 8cc8e87daf fix_unreads: Use logging instead of print. 2017-09-07 07:06:03 -07:00
Steve Howell a2fe4178be Extract zerver/lib/fix_unreads.py.
This is a pure code move.
2017-09-07 07:06:03 -07:00
Umair Khan f7d8db792c makemessages: Allow whitespaces after comma in i18n.
We allow such patterns:

```
i18n.t('Test __variable__',
        {variable: "script"})
```
2017-09-06 07:01:43 -07:00
Steve Howell 4ac6bc46c7 Add MutedTopic model.
This commit completely switches us over to using a
dedicated model called MutedTopic to track which topics
a user has muted.

This includes the necessary migrations to create the
table and populate it from legacy data in UserProfile.

A subsequent commit will actually remove the old field
in UserProfile.
2017-09-02 09:19:51 -07:00
Steve Howell 828459a24b Extract build_topic_mute_checker into topic_mutes.py.
We had two duplicate versions of this function, and one
of them was broken with respect to case insensitivity.
2017-08-29 16:53:38 -04:00
Yago González 659bff1ffb i18n: Fix URLs misparsing in translation tags.
The double forward slash (//) after the protocol in URLs was being
mistakenly considered the beginning of an inline JS comment, causing
internationalization strings being cut unexpectedly.

Now the check for inline JS comments is only run in .js files.
2017-08-28 13:54:17 -07:00
Aditya Bansal d9c9bfe7f6 logger: Add new create_logger abstraction to simplify logging.
This deduplicates a ton of Python logger-creation code to use a single
standard implementation, so we can avoid copy-paste problems.
2017-08-27 18:31:53 -07:00
Vishnu Ks 8fc8ac0799 management: Override CommandError to mention --entire-server argument. 2017-08-27 12:34:23 -07:00
Aditya Bansal b232563e12 soft-deactivation: Add cron job for weekly soft deactivating users. 2017-08-27 11:33:06 -07:00
Aditya Bansal 9d7e23c100 softdeactivation/management: Make specifying realm an optional arg. 2017-08-27 11:33:06 -07:00
Preston Hansen e8a608f733 management: Move enqueue_digest_email handler to digest. 2017-08-27 10:13:11 -07:00
Preston Hansen 9a4b17cf9b management: Move queue_digest_recipient to digest. 2017-08-27 10:13:11 -07:00
Preston Hansen 2aabf4fc67 management: Move should_process_digest to digest. 2017-08-27 10:13:11 -07:00
Preston Hansen 25a40806df management: Move inactive_since to digest. 2017-08-27 10:13:11 -07:00
Greg Price d02101a401 APNs: Rip out the existing, broken implementation.
This code empirically doesn't work.  It's not entirely clear why, even
having done quite a bit of debugging; partly because the code is quite
convoluted, and because it shows the symptoms of people making changes
over time without really understanding how it was supposed to work.

Moreover, this code targets an old version of the APNs provider API.
Apple deprecated that in 2015, in favor of a shiny new one which uses
HTTP/2 to meet the same needs for concurrency and scale that the old
one had to do a bunch of ad-hoc protocol design for.

So, rip this code out.  We'll build a pathway to the new API from
scratch; it's not that complicated.
2017-08-26 14:16:05 -07:00
Tim Abbott eeabed9119 models: Add new get_user_profile_by_api_key helper.
This results in a slight performance increase.
2017-08-24 23:17:08 -07:00
Tim Abbott a3bc041894 send_password_reset_email: Simplify argument parsing. 2017-08-24 14:41:45 -07:00
Vishnu Ks 2b076ef822 management: Move all_users option to ZulipBaseCommand. 2017-08-24 14:35:18 -07:00
Vishnu Ks 9d043e7fb0 management: Use self.style attribute for colored output. 2017-08-24 14:29:32 -07:00
Vishnu Ks f4df74d968 management: Use add_user_list_args in send_password_reset_email. 2017-08-24 14:29:32 -07:00
Vishnu Ks 65a20b3b99 management: Use add_user_list_args in remove_users_from_stream. 2017-08-24 14:29:32 -07:00
Vishnu Ks 0949924ea3 management: Use add_user_list_args in add_users_to_streams. 2017-08-24 14:29:32 -07:00
Vishnu Ks a82b3f1ea5 management: Use add_user_list_args in turn_off_digests. 2017-08-24 14:29:32 -07:00
Tim Abbott 8cbdaca9f5 fix_unreads: Fix handling of muted_topics on missing streams.
If the stream no longer exists, we can just ignored the muted_topics
entry.
2017-08-23 18:14:11 -07:00
Tim Abbott 7918551fa9 fix_unreads: Help postgres figure out the most efficient query.
In theory, this shouldn't make a difference, but in practice, this
helps postgres make better use of our unread messages index.
2017-08-23 18:00:33 -07:00
Tim Abbott 75d9f66bf0 fix_unreads: Fix error handling with no recipient_ids.
Previously, this would send an invalid database query to postgres.
2017-08-23 17:15:33 -07:00
Steve Howell bc61fae6d2 fix_unreads: Actually update pre-pointer unread counts.
This fixes an oversight in the original version.  (We have only
used this in controlled experiments.)
2017-08-23 13:13:23 -07:00
Tim Abbott 669b1b8adb mypy: Fix an annotation. 2017-08-22 17:39:24 -07:00
Steve Howell 110cfcb640 Support --all options in fix_unreads.
(We also changed the email metavar to accept multiple emails.)
2017-08-22 20:06:30 -04:00
Steve Howell 752bf14443 fix_unreads: Improve cursor handling.
Before adding the context manager, we were leaking cursors, which
was a non-issue for updating individual users, but it's easy to
fix.
2017-08-22 20:06:26 -04:00
Steve Howell c5e083857b fix_unreads: Mark pre-pointer messages as unread (unless muted). 2017-08-22 20:03:18 -04:00
Steve Howell 8b16c6060c Split out fix_unreads management command.
Most of the code in show_unreads is for diagnosising unread
counts issues, and we may not use that often.

We're creating a dedicated fix_unreads management command with
less clutter.
2017-08-22 19:38:44 -04:00
Preston Hansen c31ef34491 enqueue_digest_emails: Only send digest emails weekly, on Tuesday.
Fixes: #6114.
2017-08-22 15:07:53 -07:00
Greg Price a099e698e2 py3: Switch almost all shebang lines to use `python3`.
This causes `upgrade-zulip-from-git`, as well as a no-option run of
`tools/build-release-tarball`, to produce a Zulip install running
Python 3, rather than Python 2.  In particular this means that the
virtualenv we create, in which all application code runs, is Python 3.

One shebang line, on `zulip-ec2-configure-interfaces`, explicitly
keeps Python 2, and at least one external ops script, `wal-e`, also
still runs on Python 2.  See discussion on the respective previous
commits that made those explicit.  There may also be some other
third-party scripts we use, outside of this source tree and running
outside our virtualenv, that still run on Python 2.
2017-08-16 17:54:43 -07:00
Tim Abbott d43f5ceeec database: Add database index for wildcard mentions. 2017-08-16 13:28:04 -07:00
Tim Abbott 8bb812c8a9 database: Add database index for alert words. 2017-08-16 12:39:01 -07:00
Steve Howell 3a825ef22c Add create_large_indexes management command.
This management command creates the same indexes as migrations
82, 83, and 95, which are all indexes on the huge UserMessage
table. (*)

This command quickly no-ops with clear messaging when the
indexes already exist, so it's idempotent in that regard. (If
somebody somehow creates an index by the same name incorrectly,
they can always drop it in dbshell and re-run this command.)

If any of the migrations have not been run, which we detect simply
by the existence of the indexes, then we create them using a
`CREATE INDEX CONCURRENTLY` command.  This functionality in
postgres allows you to create indexes against large tables
without disrupting queries against those tables.  The tradeoff
here is that creating indexes concurrently takes significantly
longer than doing them non-concurrently.

Since most tables are small, we typically just use regular
Django migrations and run them during a brief interval while
the app is down.

For indexes on big tables, we will want to run this command
as part of the upgrade process, and we will want to run
it while the app is still up, otherwise it's pointless.

All the code in create_indexes() is literally copy/pasted
from the relevant migrations, and that scheme should work
going forward.  (It uses a different implementation of
create_index_if_not_exist than the migrations use, but the
code is identical lexically in the function.)

If we ever do major restructuring of our large tables, such
as UserMessage, and we end up droppping some of these indexes,
then we will need to make this command migrations-aware.  For
now it's safe to assume that indexes are generally additive in
nature, and the sooner we create them during the upgrade process,
the better.

(*) UserMessage is huge for large installations, of course.
2017-08-16 12:38:51 -07:00
Aditya Bansal 0716702558 management: Log number of soft deactivated/reactivated users. 2017-08-15 22:05:19 -07:00
Aditya Bansal 5f5d99550e soft-deactivation/management: Raise exception for non existant emails. 2017-08-15 22:05:19 -07:00
Aditya Bansal cd11cbea14 soft-deactivation/management: Rename variable to make more sense.
We rename variable user_ids_to_deactivate to users_to_deactivate
since it was storing UserProfiles anyway.
2017-08-15 22:05:19 -07:00
Aditya Bansal a33790946b management: Add soft_deactivate_users command. 2017-08-15 08:33:16 -07:00
Vishnu Ks f0bb11ef40 management: Use self.get_user in convert_bot_to_outgoing_webhook. 2017-08-08 14:13:19 -07:00
Vishnu Ks d1bece2ec2 management: Use self.get_realm in create_stream. 2017-08-08 14:13:19 -07:00
Vishnu Ks 9fb5381acb management: Use self.get_realm in set_default_streams. 2017-08-08 14:13:19 -07:00
Vishnu Ks 3bc475d04f management: Use self.get_realm in rename_stream. 2017-08-08 14:13:19 -07:00
Vishnu Ks 4e7dbff197 management: Use self.get_realm in realm_filters. 2017-08-08 14:13:19 -07:00
Vishnu Ks 7859ec76cf management: Use self.get_realm in realm_emoji. 2017-08-08 14:13:19 -07:00
Vishnu Ks af1c1861da management: Use self.get_realm in realm_domain. 2017-08-08 14:13:19 -07:00
Vishnu Ks 1b0b135bfc management: Remove unused imports from process_queue. 2017-08-08 14:13:19 -07:00
Vishnu Ks 0532b35c43 management: Remove unused imports from print_email_delivery_backlog. 2017-08-08 14:13:19 -07:00
Vishnu Ks 164badde13 management: Use self.get_realm in merge_streams.
The variable encoding is removed becuase it is not used
anywhere.
2017-08-08 14:13:19 -07:00
Vishnu Ks 27936fcfd4 management: Remove unused imports from makemessages. 2017-08-08 14:13:19 -07:00
Vishnu Ks cf61349563 management: Use self.get_realm in logout_all_users. 2017-08-08 14:13:19 -07:00
Vishnu Ks e28a007487 management: Use self.get_user in knight command. 2017-08-08 14:13:19 -07:00
Vishnu Ks 3a2edc66f1 management: Use get_system_bot in initialize_voyager_db. 2017-08-08 14:13:19 -07:00
Vishnu Ks f45b80aafe management: Remove unused imports from initialize_voyager. 2017-08-08 14:13:19 -07:00
Vishnu Ks 05af206392 management: Remove unused imports from import command. 2017-08-08 14:13:19 -07:00
Vishnu Ks f58efdc3cb management: Remove unused imports from export_usermessage_batch. 2017-08-08 14:13:19 -07:00
Vishnu Ks 28e76c990a management: Remove unused imports from email_mirror command. 2017-08-08 14:13:19 -07:00
Vishnu Ks 6687b236bf management: Remove unused imports from deliver_email command. 2017-08-08 14:13:19 -07:00
Vishnu Ks b8dacf9bf6 management: Remove unused import from checkconfig command. 2017-08-08 14:13:19 -07:00
Vishnu Ks 5ce0d85ccf management: Remove unused import from check_redis command. 2017-08-08 14:13:19 -07:00
Vishnu Ks 19f8da4995 management: Use self.get_realm in export command. 2017-08-08 14:13:19 -07:00
Vishnu Ks 4e9dff09fd management: Use self.get_realm in dump_messages. 2017-08-08 14:13:19 -07:00
Vishnu Ks bc48c68c09 management: Use self.get_realm in reactivate_realm command. 2017-08-08 14:13:19 -07:00
Vishnu Ks 279af33ea2 management: Use self.get_realm in deactivate_realm. 2017-08-08 14:13:19 -07:00
Vishnu Ks 349d604ad5 management: Use self.get_realm in create_user. 2017-08-08 14:13:19 -07:00
Rishi Gupta ddbf8b9c3c management: Remove create_realm.py.
This no longer does the correct thing (in terms of onboarding emails,
default streams, etc), and is tempting for new server admins to use.

Once we remove it we'll also have the invariant that we can't have a realm
without a user, which will simplify accounts_register a bit.
2017-08-05 10:17:14 -07:00
Vishnu Ks e7f5da83ac management: Create list realms command.
Fixes #5917.
2017-08-04 14:25:03 -07:00
Steve Howell bc6543a236 Use do_mark_all_as_read() in bankrupt_users command. 2017-08-04 14:10:46 -07:00
Steve Howell c5e4ce5278 Optimize show_unreads --fix approach.
This now breaks the process of cleaning up unread counts for
non-active streams into a three step process.

This allows us to use our unread message flags index, at least
in testing on dev.  Here is the relevant excerpt from explain
analyze:

    Bitmap Index Scan on zerver_usermessage_unread_message_id
2017-08-04 11:04:43 -07:00
Greg Price cc30226af0 deliver_email: When disabled, sleep forever rather than exit.
This makes supervisor see the service as cheerfully running
and let it alone, rather than constantly retry starting it.
Because the crash/restart loop means repeatedly spending a
couple of seconds loading Django and the app, separated by
brief periods while supervisor notices the crash and acts
on it, it was actually consuming about 30-50% CPU on the
zulipchat.com staging server.
2017-08-03 16:28:19 -07:00
Steve Howell e994ff36b5 Add option to fix unread messages for inactive streams. 2017-08-03 13:17:04 -07:00
Steve Howell dad1c9326e Add show_unreads management command.
This should help us troubleshoot anomalies with old unread
messages.
2017-08-03 13:17:04 -07:00
neiljp (Neil Pilgrim) 1c7d0c932a mypy: Ensure type of trim_func is consistent in check_redis.py. 2017-08-03 11:03:14 -07:00
Umair Khan 9e33917d25 rate_limiter: Upgrade max_api_calls to generic API. 2017-08-02 18:01:39 -07:00
Umair Khan ca9ac1948e rate_limiter: Upgrade max_api_window to generic API. 2017-08-02 18:01:39 -07:00
Umair Khan 9c08b62448 rate_limiter: Upgrade unblock_user to generic API.
Change name to unblock_access as well.
2017-08-02 18:01:39 -07:00
Umair Khan 23be0875a1 rate_limiter: Upgrade block_user to generic API.
Change name to block_access as well.
2017-08-02 18:01:39 -07:00
rht ca960da58f create_user: Receive password input from a file instead of shell arg.
This is to be used for the case of container orchestration instead of
shell arg to prevent snooping by any user account on the server via `ps
-ef` or any superuser with read access to the user\'s bash history.
2017-07-27 18:08:52 -07:00
Umair Khan 758dbec9e2 tornado: Move setup_tornado_rabbitmq to application.py 2017-07-21 09:55:25 +05:00
Tim Abbott 240f37e8c4 management: Add script to merge two streams.
This is intended to be used occasionally when there's a really good
reason to do so.  While it does work, it hasn't been extensively tested.
2017-07-18 21:08:24 -07:00
Rishi Gupta 0f4b71b766 confirmation: Liberate get_link_for_object from ConfirmationManager. 2017-07-17 23:18:47 -07:00
Rishi Gupta aa845e7f60 models: Replace ScheduledJob with ScheduledEmail.
ScheduledJob was written for much more generality than it ended up being
used for. Currently it is used by send_future_email, and nothing
else. Tailoring the model to emails in particular will make it easier to do
things like selectively clear emails when people unsubscribe from particular
email types, or seamlessly handle using the same email on multiple realms.
2017-07-17 16:05:38 -07:00
Rishi Gupta 898269bbac email: Change send_email to raise exception on failure.
More in line with how we do error handling in the rest of Zulip.
2017-07-16 16:56:39 -07:00
Rishi Gupta 3d24d12ba1 emails: Change reset password emails to use to_user_id. 2017-07-16 16:56:39 -07:00
Rishi Gupta 154d37afd2 emails: Add to_user_id argument to send_email.
Both the queue processor and ScheduledJob emails need to sometimes pass a
to_user_id and sometimes pass a to_email, and it's more convenient to just
have one function that they can call that can handle either.

Also removes the now redundant send_email_to_user.
2017-07-16 16:56:39 -07:00
James Rowan 69f3ca7870 emails: Make password reset emails come from 'Zulip Account Security.' 2017-07-13 14:50:36 -07:00
Vishnu Ks 48617c5915 send_password_reset_email: Replace get_user_profile_by_email with get_user. 2017-07-11 11:48:21 -07:00
Vishnu Ks aca4d18f23 generate_invite_links: Replace get_user_profile_by_email with get_user.
The extra refactoring is because the realm is made a compulsary
argument.
2017-07-11 11:48:21 -07:00
Umair Khan c31beec9e6 makemessages: Support only Django version >= 1.11. 2017-07-10 11:32:10 -07:00
Vishnu Ks 8dd2268c3b set_message_flags: Replace get_user_profile_by_email with get_user. 2017-07-08 14:40:55 -07:00
Vishnu Ks 0e76d95b9f export_single_user: Replace get_user_profile_by_email with get_user. 2017-07-08 14:38:45 -07:00
Vishnu Ks afc41dfb6b rate_limit: Replace get_user_profile_by_email with get_user. 2017-07-08 14:38:45 -07:00
Vishnu Ks 1ce6b8db8c remove_users_from_stream: Replace get_user_profile_by_email with get_user. 2017-07-08 14:38:45 -07:00
Vishnu Ks 038f2d669a turn_off_digests: Replace get_user_profile_by_email with get_user. 2017-07-08 14:38:45 -07:00
Vishnu Ks fdf3df4154 bulk_change_user_name: Replace get_user_profile_by_email with get_user. 2017-07-08 14:38:45 -07:00
Vishnu Ks 3b6d5fd80e bankrupt_users: Replace get_user_profile_by_email with get_user. 2017-07-08 14:38:45 -07:00
Vishnu Ks a5d2576801 deactivate_users: Replace get_user_profile_by_email with get_user. 2017-07-08 14:38:45 -07:00
Tim Abbott b01e6d750f management: Remove obsolete update_permissions management command.
This was part of the old Django/guardian-based permissions system.
2017-07-08 14:37:25 -07:00
Tim Abbott 3395f38bf7 management: Remove obsolete gravatar_to_user_avatar management command.
This was used for a 1-time migration in 2013, and definitely isn't
needed anymore.
2017-07-08 14:35:53 -07:00
Vishnu Ks 36972f611c add_users_to_streams: Remove unused imports. 2017-07-08 14:35:42 -07:00
Vishnu Ks 4340b94a2d create_user: Separate password help into multiple lines.
This line is too long and ./tools/lint --pep8 is failing because
of this.
2017-07-08 14:18:49 -07:00
rht 7a9de7766b Add password argument to `manage.py create_user`.
Tweaked by tabbott to provide clear --help output recommending against
setting passwords this way.
2017-07-08 10:59:02 -07:00
Rishi Gupta b3654c7b7e confirmation: Remove vestiges of the MitUser table.
The MitUser table was removed in df525ad.

confirm_mituser.html could have been accessed through the last few lines of
confirmation/views.py:
    templates.insert(0, 'confirmation/confirm_%s.html'
                     % (obj._meta.model_name,))

The commit message on df525ad suggests there was another way
confirm_mituser.html could have been called, but I don't currently see
evidence for it in the code.
2017-07-07 18:53:00 -07:00
Rishi Gupta 8fed9eeb75 confirmation: Make host a required argument in get_link_for_object.
Removes some lines of test from test_email_change.py. The relevant code path
was never utilized by the code itself, just by the tests.
2017-07-07 18:53:00 -07:00
Greg Price 207bf2e1fb i18n: Recognize a little more Handlebars syntax in makemessages.
Usually we write translation expressions as `{{t ... }}`, but `{{ t ... }}`
is equally valid as far as Handlebars is concerned, and it matches how we
usually write simple variable substitutions, as `{{ ... }}`.  So occasionally
someone writes `{{ t ... }}`; currently there are two examples of this
in the codebase, in `settings/bot-{settings,list-admin}.handlebars`.

Probably it'd be good to pick a style and enforce it uniformly, but
until we do, the other style shouldn't break translation.
2017-07-07 17:02:39 -07:00
Umair Khan 0e8231d0f1 process_queue: Recover gracefully after PostgreSQL restart.
- For threaded workers:
Django's autoreloader catches SIGQUIT(3) to reload the program. If
a process being watched by autoreloader exits with status code 3,
reloader will restart the process. To reload, we send SIGUSR1(10)
signal from consumers to a handler in process_queue which then
exits with status code 3.

- For single worker per process:
Catch the SIGUSR1 and quit; supervisorctl will restart the worker
automatically.

Fixes #5512
2017-07-07 16:33:15 -07:00
Vaida d5517bae36 Delete the old zulip.com "referrals" system.
This system hasn't been in active use for several years, and had some 
problems with it's design.  So it makes sense to just remove it to declutter
the codebase.

Fixes #5655.
2017-07-07 14:59:18 -07:00
Tim Abbott dbabe3fd37 add_users_to_streams: Fix check for all_users option. 2017-07-07 13:57:57 -07:00
Tim Abbott 982667acf5 management: Add new library for specifying realms.
This new library is intended to make it easy for management commands
to access a realm or a user in a realm without having to duplicate any
of the annoying parsing/extraction code.
2017-07-07 13:57:57 -07:00
Rishi Gupta a26703109e settings: Change all uses of ZULIP_ADMINISTRATOR to FromAddress.SUPPORT.
Make it less likely that further development will break compatibility with
ZULIP_ADMINISTRATORs of the form "name <email>".

Note that the suggested value for this setting has been
'zulip-admin@example.com' for a while, so hopefully this commit causes no
change for most installations.
2017-07-05 15:33:01 -07:00
Rishi Gupta 577e201df0 emails: Replace DEFAULT_FROM_EMAIL with ZULIP_ADMINISTRATOR.
We're getting rid of DEFAULT_FROM_EMAIL from prod_settings_template, since
we only really need one of {DEFAULT_FROM_EMAIL, ZULIP_ADMINISTRATOR}.
2017-06-29 17:54:33 -07:00
Rishi Gupta 36cd122905 models: Change default org_type to CORPORATE.
Once we implement org_type-specific features, it'll be easy to change a
corporate realm to a community realm, but hard to go the other way. The main
difference (the main thing that makes migrating from a community realm to a
corporate realm hard) is that you'd have to make everyone sign another terms
of service.
2017-06-29 15:14:58 -07:00
Rishi Gupta 78790ce825 send_password_reset_email: Remove spurious send_mail import.
Make grepping for and otherwise understanding our email pathways slightly
easier.
2017-06-29 10:51:12 -07:00
Rishi Gupta a56d703bdc i18n: Remove translation tags in management commands. 2017-06-25 11:28:28 -04:00
Umair Khan b74cc576ae Django 1.11: Make makemessages command compatible.
Now REs have moved to template module. This commit adds a condition
to use trans_real module if the Django version is less than 1.11 else
use template module.
2017-06-13 15:14:05 -07:00
Harshit Bansal e30e244b29 emoji: Moved `unified_reactions.json` file.
Moved `unified_reactions.json` file from `tools/setup/emoji` to
`zerver/management/data` so that it can be used in production
environment.
2017-06-09 16:38:58 -07:00
Rishi Gupta 769c5ab105 emails: Send password reset emails through zerver.lib.send_email.
Previously, the password reset email behaved differently from all the other
email Zulip sends.
2017-06-06 23:22:22 -07:00
rht 940cf9db3b Run queue processors multithreaded in production if system memory <3.5GB.
While running queue processors multithreaded will limit the
performance available to very small systems, it's easy to fix that by
adding more RAM, and previously, Zulip didn't work on such systems at
all, so this is unambiguously an improvement there.

Fixes #32.
Fixes #34.

(Commit message expanded significantly by tabbott.)
2017-06-03 12:19:58 -07:00
Aditya Bansal 15ea059aad pep8: Add compliance with rule E261 makemessages.py. 2017-05-31 17:07:15 -07:00
Aditya Bansal 717e5ae393 pep8: Add compliance with rule E261 commands/create_realm.py. 2017-05-31 17:07:15 -07:00
Maxim Averin 685fb16c39 Switch change_full_name to use RealmAuditLog.
This requires adding an `acting_user` parameter to the
`do_change_bot_owner` function.
2017-05-29 15:22:08 -07:00
Elliott Jin 5b4d2832fc bots: Add management command for making outgoing webhook bot. 2017-05-25 09:45:54 -07:00
Tim Abbott e9968a7a09 change_user_email: Use new get_user_for_mgmt function. 2017-05-24 15:29:59 -07:00
Matt Long 19363b2b77 notification_settings: Refactor notification preference settings.
Previously, all notification preference setting had a dedicated test
and setter. Now, all are handled through a modular function using the
property_types framework.
2017-05-23 14:47:46 -07:00
Aditya Bansal 8a6617e304 pep8: Add compliance with rule E261 to set_default_streams.py. 2017-05-07 23:21:50 -07:00
Aditya Bansal 7271fc8a7b pep8: Add compliance with rule E261 to import.py. 2017-05-07 23:21:50 -07:00
Aditya Bansal d8db632d21 pep8: Add compliance with rule E261 to enqueue_file.py. 2017-05-07 23:21:50 -07:00
Aditya Bansal 6a53c53588 pep8: Add compliance with rule E261 to email_mirror.py. 2017-05-07 23:21:50 -07:00
Aditya Bansal f1b2f10574 pep8: Add compliance with rule E261 to dump_messages.py. 2017-05-07 23:21:50 -07:00
Rishi Gupta 6b9f25a58a Remove duplicated code between send_future_email and send_email pathway.
Note that this change restricts the context for any template that uses
send_future_email to be jsonable.
2017-05-05 14:38:25 -07:00
Rishi Gupta 68c172192b email.py: Change recipients argument of send_future_email to to_email. 2017-05-05 14:38:25 -07:00
Rishi Gupta d70e09b41d notifications: Change sender arg of send_future_email to from_email.
This commit is a step towards the goal of replacing most of the
send_future_email pathway with a call to send_email.

Note that this commit changes the default value of sender from "Zulip
<NOREPLY_EMAIL_ADDRESS>" to "NOREPLY_EMAIL_ADDRESS". NOREPLY_EMAIL_ADDRESS
will soon be changed to have the Zulip in front.
2017-05-05 14:38:25 -07:00
Rishi Gupta e46cbaffa2 email: Remove Mandrill pathways and dependency.
Everything it was doing (send_future_email) can now be done using
ScheduledJob.
2017-05-05 14:38:23 -07:00
Rishi Gupta 925ee8c0f1 Add a send_email function that takes a template_prefix and context.
This commit replaces all uses of django.core.mail.send_mail with send_email,
other than in the password reset flow, since that code looks like it is just
a patch to Django's password reset code.

The send_email function is in a new file, since putting it in
zerver.lib.notifications would create an import loop with confirmation.models.

send_future_email will soon be moved into email.py as well.
2017-05-05 14:20:32 -07:00
Rishi Gupta 92dd76822f email: Move and rename password reset templates. 2017-05-03 23:26:14 -07:00
Tim Abbott 4a67ba241e realm_domains: Restructure library for getting realm domains.
* Remove duplicate list_of_domains_for_realm of get_realm_domains.
* Move get_realm_domains from actions.py.
2017-04-28 21:11:56 -07:00
K.Kanakhin e3e52e7284 email-mirror: Move postfix email mirror integration to separate script.
This fixes a performance problem where we were previously starting up
a full Django process (~0.7s even on a fast machine) every time a new
email came in, potentially allowing users to accidentally DoS a Zulip
server.  Now, we just post over HTTPS, allowing the existing thread
pool support to do its job.

- Add script wrapper to communicate postfix pipe with django web server
  over HTTP(S). It uses shared_secret authentication mode.
- Add django view to process messages from email mirror server.
- Clean management command `email-mirror`. Left just functional
  for cron email processing.
- Add routes for new tornado view.
- Change pipe script in master process postfix config template
  based on updated script.
- Add tests.

Tweaked by tabbott to adjust the directory and set better defaults.

Fixes #2421.
2017-04-24 21:24:23 -07:00
hackerkid b2504084ab Replace timezone.now with timezone_now. 2017-04-16 12:28:56 -07:00
hackerkid 55c3d12078 Replace timezone.utc with timezone_utc. 2017-04-16 12:28:56 -07:00
Yago González e710110a9e i18n: Automatically strip Handlebars strings.
Some Handlebars strings contained whitespaces characters at their ends.
With this, such characters are removed, as well as multiple spaces
(like the ones produced by code indentation).

This also includes a couple of fixes that removes spaces that were
intentionally placed before/after the string to translate.
2017-04-14 17:37:25 -07:00
Yago González 31c92cdcbc refactor: Keep sub replacements' format uniform. 2017-04-14 17:37:25 -07:00
Umair Khan 9dec5306ce makemessages.py: Handle unicode strings.
This commit applies to Python 2. It seems that sorted()
cannot handle non-ascii character unless the strings are
marked as Unicode.
2017-04-11 22:32:06 -07:00
Abhijeet Kaur 8f88b045a4 Rename "Administration" to "Organization" in the settings UI.
This better sets expectatations for the fact that in Zulip, the
Organization settings UI is available read-only to non-administrator
users.

Tweaked by tabbott to update some additional references.
2017-04-07 17:32:56 -07:00
Umair Khan 846e8686c4 makemessages: Exclude hidden files.
Fixes #3981.
2017-04-05 11:09:32 -07:00
Harshit Bansal 664b23c495 commands: Rename `realm_alias` command to `realm_domain`. 2017-04-04 15:48:03 -07:00
Harshit Bansal 362ee482c0 commands/realm_alias.py: Remove any occurrences of `alias`. 2017-04-04 15:48:03 -07:00
Harshit Bansal 983225612d actions.py: Rename `get_realm_aliases()` to `get_realm_domains()`. 2017-04-04 15:48:03 -07:00
Harshit Bansal 92c512d679 zerver/models.py: Rename can_add_alias() to can_add_realm_domain(). 2017-04-04 15:48:03 -07:00
Harshit Bansal 6f0b46d84f create_realm.py: Remove unnecessary imports. 2017-04-04 15:48:03 -07:00
Harshit Bansal ac2172e233 models: Rename RealmAlias model to RealmDomain.
Includes a migration.
2017-04-04 15:48:03 -07:00
Rishi Gupta 30024d0a8f models: Remove Realm.domain. 2017-03-25 19:55:48 -07:00
Rishi Gupta caef5332d5 mailchimp: Fix error in add_users_to_mailing_list.py.
Old behavior is a holdover from development testing.
2017-03-21 16:13:00 -07:00
Rishi Gupta 92dd767519 management commands: Change help text to say realm instead of domain. 2017-03-14 17:17:42 -07:00
Rishi Gupta 0c032adbde create_realm.py: Remove --domain option.
Realm domains can now be managed via Admin Settings in the web app.
2017-03-14 17:17:42 -07:00
Rishi Gupta 11346e50bb management commands: Use Realm.string_id instead of domain in print statements. 2017-03-13 09:44:32 -07:00
Tim Abbott ca50e10569 sessions: Move session deletion code to sessions.py.
This new organization feels more thematically appropriate, and of
course shrinks the amount of stuff unnecessarily in actions.py
2017-03-08 02:48:18 -08:00
Tim Abbott e90df029bf create_realm: Eliminate obsolete deployment code.
Since we're not using this model anymore, we certainly don't need it
when creating realms.
2017-03-06 00:07:45 -08:00
Tim Abbott 8815a598d8 digest: Remove obsolete deployment code.
This system was quite complicated, and never had great semantics.

Eventually, we'll want some other system for gating which server
should generate digest emails for which realm controlled via the
database.
2017-03-06 00:05:21 -08:00
sinwar 6f0564e9f4 python: Fix remaining bare excepts in codebase.
Fixes #2862.
2017-03-05 16:17:04 -08:00
Tim Abbott d95412425a generate_realm_creation_link: Fix line-wrapping of help text. 2017-03-04 19:02:01 -08:00
feorlen 3c954096f6 generate_realm_creation_link: Check for an uninitialized database.
This provides a nice error message to users who miss a step in the
instructions.

Fixes #3672.
2017-03-04 19:01:01 -08:00
Raghav Jajodia a3a03bd6a5 mypy: Added Dict, List and Set imports.
Fixed mypy errors associated with the upgrade.
2017-03-04 14:33:44 -08:00
adnrs96 9eb47f108c Refactor: Change upload_avatar_image to accept two user profiles.
In this commit we just change the upload_avatar_image function to accept
two user_profiles acting_user_profile and target_user_profile. Basically
email param is dropped for a target_user_profile so that avatar's could
be moved lateron to user id based storage.
2017-03-03 18:15:15 -08:00
Umair Khan 87c5ace24d i18n: Use deterministic order in translations.json.
Fixes: #3897
2017-03-03 12:32:17 +05:00
Rishi Gupta 3d07ac0c49 Change timezone-naive datetimes to use timezone.now() where safe to do so.
Change timezone-naive datetimes to use timezone.now() in cases where there
is no change in behavior.
2017-03-01 22:54:28 -08:00
Rishi Gupta c388858e53 Fix timezone errors in ScheduledJob and digest creation filters.
datetime.utcnow() is a timezone-naive datetime. The Django ORM interprets it
in the settings.TIME_ZONE timezone (e.g. 'America/New_York' in the
development server). We perhaps haven't noticed errors yet since with
'America/New_York' all it means is that emails are sent 5 hours early, or a
slightly different set of messages are included in the digest.
2017-03-01 22:54:28 -08:00
Rishi Gupta 562bc6429c Replace datetime.now() with timezone.now() in Django ORM queries.
When you pass a naive datetime to the Django ORM, it uses settings.TIME_ZONE
for the time zone. In the development environment, both settings.TIME_ZONE
and datetime.now() use 'America/New_York', so there is no change in behavior
there. (fromtimestamp with no tz argument uses the same timezone as
datetime.now)

We are soon going to change settings.TIME_ZONE to UTC, so need to remove
naive datetimes from queries to the ORM.
2017-03-01 22:54:28 -08:00
Rishi Gupta 2b2be8120f Change datetime.now(tz=X) to timezone.now().
datetime.now with a timezone set is equivalent to timezone.now() if it's
never being printed out, but the latter is cleaner and more idiomatic.
2017-03-01 22:54:28 -08:00
Tim Abbott cf96b1b873 generate_realm_creation_link: Clean up instructions. 2017-02-21 20:19:16 -08:00
Tim Abbott 32bfebeb7a mypy: Fix inconsistencies in use of *args/**kwargs. 2017-02-18 18:39:44 -08:00
Umair Khan 00f8239563 Clean code of send_test_email command.
* Derive from sendtestemail command of Django.
* Remove unwanted imported.
* Allow test email to admin and managers.
2017-02-10 16:01:43 -08:00
Umair Khan ef0d2a4bb5 logging: Use django.server to filter 200 and 304.
Previously, we were monkey patching the runserver command
in zerver/management/commands/rundjango.py for this.
2017-02-10 15:55:17 -08:00