Commit Graph

645 Commits

Author SHA1 Message Date
Umair Khan c6bfac0e18 language_name_map: Add newline at the end. 2018-01-26 16:55:40 -08:00
Aditya Bansal f272ea9087 scheduledmessages: Start using/expecting delivery_type as a param. 2018-01-19 11:33:11 -05:00
Aditya Bansal c770bdaa3a reminder_bot: Add infra for adding reminder bot to every realm. 2018-01-19 11:33:11 -05:00
Robert Hönig 93f47fa700 outgoing webhooks: Rename "Base URL" to "Endpoint URL".
Based on user feedback. "Endpoint URL" gives a clearer
idea of the field's purpose.
2018-01-16 08:11:51 -05:00
Aditya Bansal ec1297c1e8 schedulemessages: Add delivery system for scheduled message. 2018-01-10 09:18:02 -05:00
Rhea Parekh ed4054d110 Import script: Check and add system bots after every import.
This checks for the existing system bots and adds them if they
aren't included in the import.
2017-12-27 07:52:45 -05:00
Rhea Parekh 62aec0688f Slack importer: Add slack to zulip data conversion script as a manage.py command. 2017-12-27 07:50:53 -05:00
Xavier Cooney 63142385ca mypy: Use Python 3 type syntax in 'send_password_reset_email.py'. 2017-12-26 08:34:38 -05:00
Greg Price b830b446f1 logging: Reduce `create_logger` to new `log_to_file`.
The name `create_logger` suggests something much bigger than what this
function actually does -- the logger doesn't any more or less exist
after the function is called than before.  Its one real function is to
send logs to a specific file.

So, pull out that logic to an appropriately-named function just for
it.  We already use `logging.getLogger` in a number of places to
simply get a logger by name, and the old `create_logger` callsites can
do the same.
2017-12-12 17:17:08 -08:00
Greg Price b6491fd52a logging: Eliminate log_level from create_logger.
Because calls to `create_logger` generally run after settings are
configured, these would override what we have in `settings.LOGGING` --
which in particular defeated any attempt to set log levels in
`test_settings.py`.  Move all of these settings to the same place in
`settings.py`, so they can be overridden in a uniform way.
2017-12-12 17:17:08 -08:00
Greg Price ebcf0b4876 logging: Stop having `create_logger` force loglevels to INFO.
This is already the loglevel we set on the root logger, so this has no
effect -- except in tests, where `test_settings.py` attempts to set
some of these same loggers to higher loglevels.  Because the
`create_logger` call generally runs after we've configured settings,
it clobbers that effect.

The code in `test_settings.py` that tries to suppress logs only works
because it also sets `propagate=False`, which has nothing to do with
loglevels but does cause logs at this logger (and descendants) to be
dropped completely unless we've configured handlers for this logger
(or one of its relevant descendants.)
2017-12-12 17:17:07 -08:00
Rhea Parekh 460e9b4634 Scripts: Add script for resetting postgres sequences for imports. 2017-12-01 06:38:08 -08:00
Greg Price fe979af822 purge_queue: Allow purging Tornado-consumed queues.
These are the exceptions to the rule that our queues correspond to
queue-processor workers.

Purging `notify_tornado` in particular is a useful workaround right
now for some error spew in the dev environment.
2017-11-29 14:44:53 -08:00
Rhea Parekh 84847b811b requirements: Upgrade mypy to 0.550.
We have to add a few type: ignores due to mypy 0.550 having some bugs
in the new imaplib stubs in typeshed.

Fixes #7387.
2017-11-25 10:06:27 -08:00
Vishnu Ks f99fd20f8f management: Add option to set description of default stream groups. 2017-11-15 17:06:43 -08:00
Tim Abbott f04d6ed19e python: Sort imports in management commands. 2017-11-15 15:43:47 -08:00
rht bafba95eb7 Text-wrap the remaining long lines exceeding 110. 2017-11-15 10:58:03 -08:00
rht a93b7c40a7 zerver: Text-wrap long lines exceeding 110. 2017-11-10 16:29:32 -08:00
rht d6aa7f7738 zerver/management: Remove unused imports (F401). 2017-11-07 16:36:54 -08:00
rht 6b12fe24b6 refactor: Remove six.moves.input import. 2017-11-07 10:51:44 -08:00
rht 80a8d4f9f3 refactor: Remove six.moves.map import. 2017-11-07 10:46:42 -08:00
rht ec5120e807 refactor: Remove six.moves.zip import. 2017-11-07 10:46:42 -08:00
neiljp (Neil Pilgrim) 952f034136 mypy: Set explicit Any-like parameters for handle_callback_exception in runtornado.py. 2017-11-04 19:47:45 -07:00
rht 839b427ea5 zerver/management: Remove u prefix from strings.
License: Apache-2.0
Signed-off-by: rht <rhtbot@protonmail.com>
2017-11-02 11:01:47 -07:00
rht c4fcff7178 refactor: Replace super(.*self) with Python 3-specific super().
We change all the instances except for the `test_helpers.py`
TimeTrackingCursor monkey-patching, which actually needs to specify
the base class.
2017-10-30 14:30:25 -07:00
Steve Howell a28841e8aa Extract get_stream_recipient().
Do you call get_recipient(Recipient.STREAM, stream_id) or
get_recipient(stream_id, Recipient.STREAM)?  I could never
remember, and it was not very type safe, since both parameters
are integers.
2017-10-28 17:57:39 -07:00
Tim Abbott 8e2cdedf9a lint: Fix lines in Python codebase longer than 120 characters. 2017-10-26 17:47:30 -07:00
Tim Abbott b936e8c24b lint: Fix lines in Python codebase longer than 125 characters. 2017-10-26 17:36:54 -07:00
Steve Howell df93a99b50 Cache only one row per message.
Before this change, we populated two cache entries for each
message that we sent.  The entries were largely redundant,
with the only difference being whether we sent the content
as raw markdown or as the rendered HTML.

This commit makes it so we only have one cache entry per
message, and it includes both content and rendered_content.

One legacy source on confusion here is that `content`
changes meaning when you're on the front end.  Here is the
situation going forward:

    database:
        content = raw
        rendered_contented = rendered

    cache entry:
        content = raw
        rendered_contented = rendered

    payload for the frontend:
        content = raw (for apply_markdown=False)
        content = rendered (for apply_markdown=True)
2017-10-26 16:35:28 -07:00
rht a311678190 zerver/management: Use python 3 syntax for typing. 2017-10-26 15:24:56 -07:00
derAnfaenger 1792dcbd09 tests: Call real consume method of queue processors.
This switches to more real tests for a first batch of
queue_json_publish() calls that don't cause trouble when
used with call_consume_tests=True.
2017-10-26 14:58:03 -07:00
Vishnu Ks 25cb6af32c mangement: Create command for creating default stream groups. 2017-10-26 11:38:48 -07:00
Umair Khan 19ccff060a i18n: Extract language options from tracked locales.
Now we use 'git ls-files' to get the list of locales that we actually
track. Previously we were using os.listdir to get the contents of the
static/locale directory. This could also return locales which were
present in the directory but are not supported by us, e.g. zh_CN.
2017-10-23 22:58:44 -07:00
Umair Khan 2db1cba9cc i18n: Use language code instead of locale.
We have been assigning locale to language code. Mostly code and locale
are same but for languages like zh-Hans, locale is zh_Hans and code is
zh-hans.

After this commit, compilemessages command should be run.
2017-10-20 11:21:26 -07:00
Umair Khan df13ac570a i18n: Add localized name in language_name_map.json. 2017-10-20 11:20:22 -07:00
Umair Khan 3769972fc2 i18n: Clean up code to create language_options.json.
Previously we were using regexes to extract the language from our
locale files. Now we use LANG_INFO data structure provided by Django
to do the same and fallback to PO files only when language code is not
present in the Django data structure.
2017-10-20 11:20:22 -07:00
Tim Abbott 03d72a1cd6 send_webhook_fixture_message: Print useful error messages.
Previously, if this failed, it didn't tell you why.
2017-10-20 09:03:05 -07:00
derAnfaenger 3264e4f24a Remove superfluous queue_json_publish imports. 2017-10-19 13:57:54 -07:00
Tim Abbott 3983ebce9a test_fixtures: Move test_db_status state under UUID_VAR_PATH.
This should mean that maintaining two Zulip development environments
using the same Git checkout no longer has caching problems keeping
track of the migration status.
2017-10-17 21:15:58 -07:00
Tim Abbott b7c3b4df0c migration_status: Switch to passing the path in Python.
This is an important prerequisite to changing these paths used in
provisioning.
2017-10-17 21:15:58 -07:00
Tim Abbott 15c0380a33 export: Fix exporting with an output file at a relative path.
Fixes #7051.
2017-10-17 20:47:29 -07:00
Tim Abbott 66a3f514d4 zilencer: Remove long-obsolete Deployment model.
This was a precursor to RemoteZulipServer, which is no longer used for
anything, and was cluttering the codebase.
2017-10-11 23:48:01 -07:00
Tim Abbott 49274a060c send_webhook_fixture_message: Fix to work with subdomains.
We apparently failued to update this tool when we switched the
development environment to require a subdomain.
2017-10-08 20:14:14 -07:00
rht fbd24669a4 Cleanup: Remove shebang and executable flag from these files. 2017-10-08 15:38:15 -07:00
derAnfaenger 747c283180 lint: Replace `pk` with `id` where they are equivalent. 2017-10-06 12:52:12 -07:00
Tim Abbott 0f2e4a55c0 soft deactivation: Shorten management command name.
This command is really for soft deactivation; there's just an undo
feature.
2017-10-06 08:48:43 -07:00
Tim Abbott ca59b28fc1 import: Fix positional argument parsing. 2017-10-06 08:45:22 -07:00
Tim Abbott bfedb82097 import: Fix argparse usage message breakage. 2017-10-06 08:41:01 -07:00
Umair Khan 326a6f6b4f i18next: Don't allow empty string as valid translation.
Previously we used to mark a key as unstranlated if its value was equal
to it in translations.json. This had an issue because it didn't allow
otherwise valid cases where key was equal to the value.

This commit solves the problem by disallowing an empty string as a valid
translation and then using the empty string as the value for all the
unstranslated keys.

Fixes #5261
2017-10-05 22:44:34 -07:00
Umair Khan b464fd7643 makemessages: Change type of translation_strings to List.
In the code this variable is only used as a list, however, we
create it as dictionary which uses more memory.
2017-10-05 22:44:34 -07:00
Tim Abbott 13bb546ddd subdomains: Hardcode can_add_realm_domains=True.
It was already always True if REALMS_HAVE_SUBDOMAINS, and now that's
the only supported configuration.
2017-10-02 16:32:10 -07:00
derAnfaenger d1afab7199 Replace deprecated Logging.warn calls with Logging.warning. 2017-10-02 11:11:42 +02:00
rht 6dd94becb3 zerver/management: Remove unused optparse import. 2017-09-30 09:22:08 -07:00
rht f22daeba1a Cleanup rabbitmq queues when doing dev db rebuild.
Modified by tabbott to cleanup the argument handling in the modified
purge_queue management command.

Fixes #1335.
2017-09-28 18:18:54 -07:00
rht 32650c8fdc zerver/management: Remove print_function. 2017-09-27 18:05:45 -07:00
rht 0fa3c1d876 zerver/management: Remove `import six`. 2017-09-27 17:06:16 -07:00
rht e239e97351 zerver/management: Remove absolute_import. 2017-09-27 10:00:39 -07:00
neiljp (Neil Pilgrim) ab4cbf81f0 mypy: Add assertions of get_realm results when parser requires realm. 2017-09-26 13:49:01 -07:00
Greg Price 7737326cf0 mypy: Add an Optional in fix_unreads command. 2017-09-26 13:49:01 -07:00
Greg Price c10aca1bbf soft deactivation: Simplify logic slightly in management command.
The `get_realm` method works properly regardless of whether the
'realm_id' option was set, so checking for that option first is
redundant.
2017-09-26 13:49:01 -07:00
neiljp (Neil Pilgrim) acf7e2a9ee mypy: Set conditional to be against result of get_realm, not input.
[Tweaked by greg to use explicit `is not None`.]
2017-09-26 13:49:01 -07:00
Vishnu Ks f3b4de822b command: Create multiuse user invite generation command. 2017-09-22 19:48:46 -07:00
Tim Abbott 8badd10a6c send_test_email: Remind users to restart the server after updates. 2017-09-22 13:32:22 -07:00
Umair Khan 1f93c06b76 i18n: Optimize get_language_list().
compilemessages command now does all the heavy lifting by creating a
language_name_map.json file under locale directory. This file is used
by get_language_list to retrieve the require information.

Fixes: #6486
2017-09-14 02:28:58 -07:00
Steve Howell 69203c1c81 fix_unreads: Remove commit() call in fix().
The commit() call in fix() breaks migrations and tests (unless you
mock) due to outer transactions.

We now explicitly call commit() from the management command.
2017-09-07 07:06:03 -07:00
Steve Howell 8cc8e87daf fix_unreads: Use logging instead of print. 2017-09-07 07:06:03 -07:00
Steve Howell a2fe4178be Extract zerver/lib/fix_unreads.py.
This is a pure code move.
2017-09-07 07:06:03 -07:00
Umair Khan f7d8db792c makemessages: Allow whitespaces after comma in i18n.
We allow such patterns:

```
i18n.t('Test __variable__',
        {variable: "script"})
```
2017-09-06 07:01:43 -07:00
Steve Howell 4ac6bc46c7 Add MutedTopic model.
This commit completely switches us over to using a
dedicated model called MutedTopic to track which topics
a user has muted.

This includes the necessary migrations to create the
table and populate it from legacy data in UserProfile.

A subsequent commit will actually remove the old field
in UserProfile.
2017-09-02 09:19:51 -07:00
Steve Howell 828459a24b Extract build_topic_mute_checker into topic_mutes.py.
We had two duplicate versions of this function, and one
of them was broken with respect to case insensitivity.
2017-08-29 16:53:38 -04:00
Yago González 659bff1ffb i18n: Fix URLs misparsing in translation tags.
The double forward slash (//) after the protocol in URLs was being
mistakenly considered the beginning of an inline JS comment, causing
internationalization strings being cut unexpectedly.

Now the check for inline JS comments is only run in .js files.
2017-08-28 13:54:17 -07:00
Aditya Bansal d9c9bfe7f6 logger: Add new create_logger abstraction to simplify logging.
This deduplicates a ton of Python logger-creation code to use a single
standard implementation, so we can avoid copy-paste problems.
2017-08-27 18:31:53 -07:00
Vishnu Ks 8fc8ac0799 management: Override CommandError to mention --entire-server argument. 2017-08-27 12:34:23 -07:00
Aditya Bansal b232563e12 soft-deactivation: Add cron job for weekly soft deactivating users. 2017-08-27 11:33:06 -07:00
Aditya Bansal 9d7e23c100 softdeactivation/management: Make specifying realm an optional arg. 2017-08-27 11:33:06 -07:00
Preston Hansen e8a608f733 management: Move enqueue_digest_email handler to digest. 2017-08-27 10:13:11 -07:00
Preston Hansen 9a4b17cf9b management: Move queue_digest_recipient to digest. 2017-08-27 10:13:11 -07:00
Preston Hansen 2aabf4fc67 management: Move should_process_digest to digest. 2017-08-27 10:13:11 -07:00
Preston Hansen 25a40806df management: Move inactive_since to digest. 2017-08-27 10:13:11 -07:00
Greg Price d02101a401 APNs: Rip out the existing, broken implementation.
This code empirically doesn't work.  It's not entirely clear why, even
having done quite a bit of debugging; partly because the code is quite
convoluted, and because it shows the symptoms of people making changes
over time without really understanding how it was supposed to work.

Moreover, this code targets an old version of the APNs provider API.
Apple deprecated that in 2015, in favor of a shiny new one which uses
HTTP/2 to meet the same needs for concurrency and scale that the old
one had to do a bunch of ad-hoc protocol design for.

So, rip this code out.  We'll build a pathway to the new API from
scratch; it's not that complicated.
2017-08-26 14:16:05 -07:00
Tim Abbott eeabed9119 models: Add new get_user_profile_by_api_key helper.
This results in a slight performance increase.
2017-08-24 23:17:08 -07:00
Tim Abbott a3bc041894 send_password_reset_email: Simplify argument parsing. 2017-08-24 14:41:45 -07:00
Vishnu Ks 2b076ef822 management: Move all_users option to ZulipBaseCommand. 2017-08-24 14:35:18 -07:00
Vishnu Ks 9d043e7fb0 management: Use self.style attribute for colored output. 2017-08-24 14:29:32 -07:00
Vishnu Ks f4df74d968 management: Use add_user_list_args in send_password_reset_email. 2017-08-24 14:29:32 -07:00
Vishnu Ks 65a20b3b99 management: Use add_user_list_args in remove_users_from_stream. 2017-08-24 14:29:32 -07:00
Vishnu Ks 0949924ea3 management: Use add_user_list_args in add_users_to_streams. 2017-08-24 14:29:32 -07:00
Vishnu Ks a82b3f1ea5 management: Use add_user_list_args in turn_off_digests. 2017-08-24 14:29:32 -07:00
Tim Abbott 8cbdaca9f5 fix_unreads: Fix handling of muted_topics on missing streams.
If the stream no longer exists, we can just ignored the muted_topics
entry.
2017-08-23 18:14:11 -07:00
Tim Abbott 7918551fa9 fix_unreads: Help postgres figure out the most efficient query.
In theory, this shouldn't make a difference, but in practice, this
helps postgres make better use of our unread messages index.
2017-08-23 18:00:33 -07:00
Tim Abbott 75d9f66bf0 fix_unreads: Fix error handling with no recipient_ids.
Previously, this would send an invalid database query to postgres.
2017-08-23 17:15:33 -07:00
Steve Howell bc61fae6d2 fix_unreads: Actually update pre-pointer unread counts.
This fixes an oversight in the original version.  (We have only
used this in controlled experiments.)
2017-08-23 13:13:23 -07:00
Tim Abbott 669b1b8adb mypy: Fix an annotation. 2017-08-22 17:39:24 -07:00
Steve Howell 110cfcb640 Support --all options in fix_unreads.
(We also changed the email metavar to accept multiple emails.)
2017-08-22 20:06:30 -04:00
Steve Howell 752bf14443 fix_unreads: Improve cursor handling.
Before adding the context manager, we were leaking cursors, which
was a non-issue for updating individual users, but it's easy to
fix.
2017-08-22 20:06:26 -04:00
Steve Howell c5e083857b fix_unreads: Mark pre-pointer messages as unread (unless muted). 2017-08-22 20:03:18 -04:00
Steve Howell 8b16c6060c Split out fix_unreads management command.
Most of the code in show_unreads is for diagnosising unread
counts issues, and we may not use that often.

We're creating a dedicated fix_unreads management command with
less clutter.
2017-08-22 19:38:44 -04:00
Preston Hansen c31ef34491 enqueue_digest_emails: Only send digest emails weekly, on Tuesday.
Fixes: #6114.
2017-08-22 15:07:53 -07:00
Greg Price a099e698e2 py3: Switch almost all shebang lines to use `python3`.
This causes `upgrade-zulip-from-git`, as well as a no-option run of
`tools/build-release-tarball`, to produce a Zulip install running
Python 3, rather than Python 2.  In particular this means that the
virtualenv we create, in which all application code runs, is Python 3.

One shebang line, on `zulip-ec2-configure-interfaces`, explicitly
keeps Python 2, and at least one external ops script, `wal-e`, also
still runs on Python 2.  See discussion on the respective previous
commits that made those explicit.  There may also be some other
third-party scripts we use, outside of this source tree and running
outside our virtualenv, that still run on Python 2.
2017-08-16 17:54:43 -07:00
Tim Abbott d43f5ceeec database: Add database index for wildcard mentions. 2017-08-16 13:28:04 -07:00