Commit Graph

669 Commits

Author SHA1 Message Date
Tim Abbott 9f844ff681 tornado: Fix logging of tornado activity level.
This logging was apparently broken when sorting imports; it's a fairly
unique thing in our codebase that this would be a problem.  Prevent
future regressions by adding this exception explicitly to the isort
configuration.
2018-04-17 15:59:01 -07:00
Rhea Parekh f6b6aa1e75 slack import: Implement threading as a management command. 2018-04-15 19:53:02 +05:30
Rhea Parekh ccefaf7b26 scripts: Remove the depreciated script 'postgres-reset-sequences'. 2018-04-10 13:07:14 -07:00
Rhea Parekh 2baa9bc16e Import: Add subdomain in the import script.
Also remove user input of subdomain in the slack data
conversion script.
2018-04-06 09:12:56 -07:00
neiljp (Neil Pilgrim) 090b47ed19 mypy: Add explicit Optional for default=None parameters in various files. 2018-03-28 12:31:51 -07:00
Tim Abbott c94deff920 mypy: Remove some now-unnecessary type: ignores. 2018-03-28 10:39:05 -07:00
Umair Khan d1c7a400ee i18n: Ignore var directory.
Previously, makemessages command was also searching var directory for
translatable strings. This commit ignores that directory.

Fixes #8751
2018-03-25 09:08:37 -07:00
neiljp (Neil Pilgrim) ac09687fa2 mypy: Raise in commands/create_user.py if initial_password returns None. 2018-03-25 08:59:08 -07:00
neiljp (Neil Pilgrim) de76297f31 mypy.ini: Move purge_queue exclusion into re-architecting section. 2018-03-25 08:59:08 -07:00
Jack Weatherilt 3396cfc2ef refactoring: Replaced occurences of create_stream_if_needed.
Issue #2088 asked for a wrapper to be created for
`create_stream_if_needed` (called `ensure_stream`) for the 25 times that
`create_stream_if_needed` is called and ignores whether the stream was
created. This commit replaces relevant occurences of
`create_stream_if_needed` with `ensure_stream`, including imports.

The changes weren't significant enough to add any tests or do any
additional manual testing.

The refactoring intended to make the API easier to use in most cases.
The majority of uses of `create_stream_if_needed` ignored the second
parameter.

Fixes: #2088.
2018-03-21 16:47:36 -07:00
Vishnu Ks 951b88dd30 models: Make email_allowed_for_realm raise exception. 2018-03-15 14:35:24 -07:00
Tim Abbott c63b8983a0 models: Delete legacy convert_bot_to_outgoing_webhook.
We also delete a couple helper functions that were only used there.

This management command was primarily used before we had a UI for
creating outgoing webhook bots.
2018-03-12 11:52:39 -07:00
neiljp (Neil Pilgrim) 9e1dbde82d mypy: Final small migrations to python3.5 annotations in many files. 2018-03-12 11:23:30 -07:00
Harshit Bansal 089fede9b0 commands: Delete 'realm_emoji' management command.
We no longer accept URLs while creating emoji; so this management
command was probably left out while migrating realm emoji
infrastructure to upload backend.

We could fix this to work properly today, but the command was
originally written in a context when Zulip didn't have a UI for
managing realm emoji at all.  Now that we do have such a UI, it
doesn't have a compelling use case, and work on migrating the realm
emoji schema demonstrates that this does have a maintenance cost.

So, we simply remove this command.
2018-03-11 16:23:20 -07:00
rht 71ff8c370e django-2.0: Don't assign directly to Many-to-Many field.
The old pattern of setting the value and then using .save() here has
been deprecated.  set() also saves the record.
2018-03-01 08:49:35 -08:00
Robert Hönig b9c6cd19f7 translation: Mark error strings in django.po files.
In `django.po`, strings are marked with comments specifying the
type of string.
2018-02-15 11:12:42 -08:00
Greg Price 7bfd22a0b0 mypy: Give types to some kwargs in makemessages.
Based on types identified by Neil Pilgrim.
2018-02-13 11:40:52 -08:00
Vishnu Ks 81f18bd69a actions: Create do_create_multiuse_invite_link function. 2018-02-13 09:31:45 -08:00
Rhea Parekh a0b7976f77 slack importer: Fix output_dir path and remove sys.exit(0).
Also, as the exception is being handled in 'slack_data_to_zulip_data'
remove it from the manage.py command 'convert_slack_data'.
2018-02-09 12:17:10 -08:00
Rhea Parekh c0e30079f6 slack importer: Get user data from a get request to slack users api.
The fresh imported data shows that the users emails are not included
in the data. However, the data received from the older method of slack
(which is using legacy tokens) contains the email data of the users.
2018-02-09 12:17:10 -08:00
Greg Price c80802ff1e bots: Clean up create_realm_internal_bots.
This code duplicated the code in setup_realm_internal_bots, with some
added logic to avoid trying to create the same bot twice.  That logic
was buggy so that it would never work at all -- it subtracted a set of
UserProfile objects from a set of email strings -- so it looked like
the command might blow up when run after the users already existed.

In fact, the buggy logic wasn't necessary, because the work the
command does after it is idempotent -- in particular `create_users`,
within its subroutine `bulk_create_users`, already filters out users
that already exist.  So just cut the buggy stuff out, deduplicate the
rest with `setup_realm_internal_bots`, and document that invariant on
the latter.

While we're here, in the common case bail early without doing any
per-realm work in Python, since we're running this on every upgrade.
2018-02-06 18:59:57 -08:00
Greg Price b21cc8103f confirmation: Presume email valid when admin creates realm.
A followup commit will actually act on this presumption.
2018-02-05 12:59:12 -08:00
Umair Khan 8b2efcbf27 i18n: Include mobile stats in percent translated.
Fixes #7499
2018-01-29 12:49:51 -08:00
Umair Khan 151b3e9f1a language_options: Add newline at the end. 2018-01-26 16:55:40 -08:00
Umair Khan c6bfac0e18 language_name_map: Add newline at the end. 2018-01-26 16:55:40 -08:00
Aditya Bansal f272ea9087 scheduledmessages: Start using/expecting delivery_type as a param. 2018-01-19 11:33:11 -05:00
Aditya Bansal c770bdaa3a reminder_bot: Add infra for adding reminder bot to every realm. 2018-01-19 11:33:11 -05:00
Robert Hönig 93f47fa700 outgoing webhooks: Rename "Base URL" to "Endpoint URL".
Based on user feedback. "Endpoint URL" gives a clearer
idea of the field's purpose.
2018-01-16 08:11:51 -05:00
Aditya Bansal ec1297c1e8 schedulemessages: Add delivery system for scheduled message. 2018-01-10 09:18:02 -05:00
Rhea Parekh ed4054d110 Import script: Check and add system bots after every import.
This checks for the existing system bots and adds them if they
aren't included in the import.
2017-12-27 07:52:45 -05:00
Rhea Parekh 62aec0688f Slack importer: Add slack to zulip data conversion script as a manage.py command. 2017-12-27 07:50:53 -05:00
Xavier Cooney 63142385ca mypy: Use Python 3 type syntax in 'send_password_reset_email.py'. 2017-12-26 08:34:38 -05:00
Greg Price b830b446f1 logging: Reduce `create_logger` to new `log_to_file`.
The name `create_logger` suggests something much bigger than what this
function actually does -- the logger doesn't any more or less exist
after the function is called than before.  Its one real function is to
send logs to a specific file.

So, pull out that logic to an appropriately-named function just for
it.  We already use `logging.getLogger` in a number of places to
simply get a logger by name, and the old `create_logger` callsites can
do the same.
2017-12-12 17:17:08 -08:00
Greg Price b6491fd52a logging: Eliminate log_level from create_logger.
Because calls to `create_logger` generally run after settings are
configured, these would override what we have in `settings.LOGGING` --
which in particular defeated any attempt to set log levels in
`test_settings.py`.  Move all of these settings to the same place in
`settings.py`, so they can be overridden in a uniform way.
2017-12-12 17:17:08 -08:00
Greg Price ebcf0b4876 logging: Stop having `create_logger` force loglevels to INFO.
This is already the loglevel we set on the root logger, so this has no
effect -- except in tests, where `test_settings.py` attempts to set
some of these same loggers to higher loglevels.  Because the
`create_logger` call generally runs after we've configured settings,
it clobbers that effect.

The code in `test_settings.py` that tries to suppress logs only works
because it also sets `propagate=False`, which has nothing to do with
loglevels but does cause logs at this logger (and descendants) to be
dropped completely unless we've configured handlers for this logger
(or one of its relevant descendants.)
2017-12-12 17:17:07 -08:00
Rhea Parekh 460e9b4634 Scripts: Add script for resetting postgres sequences for imports. 2017-12-01 06:38:08 -08:00
Greg Price fe979af822 purge_queue: Allow purging Tornado-consumed queues.
These are the exceptions to the rule that our queues correspond to
queue-processor workers.

Purging `notify_tornado` in particular is a useful workaround right
now for some error spew in the dev environment.
2017-11-29 14:44:53 -08:00
Rhea Parekh 84847b811b requirements: Upgrade mypy to 0.550.
We have to add a few type: ignores due to mypy 0.550 having some bugs
in the new imaplib stubs in typeshed.

Fixes #7387.
2017-11-25 10:06:27 -08:00
Vishnu Ks f99fd20f8f management: Add option to set description of default stream groups. 2017-11-15 17:06:43 -08:00
Tim Abbott f04d6ed19e python: Sort imports in management commands. 2017-11-15 15:43:47 -08:00
rht bafba95eb7 Text-wrap the remaining long lines exceeding 110. 2017-11-15 10:58:03 -08:00
rht a93b7c40a7 zerver: Text-wrap long lines exceeding 110. 2017-11-10 16:29:32 -08:00
rht d6aa7f7738 zerver/management: Remove unused imports (F401). 2017-11-07 16:36:54 -08:00
rht 6b12fe24b6 refactor: Remove six.moves.input import. 2017-11-07 10:51:44 -08:00
rht 80a8d4f9f3 refactor: Remove six.moves.map import. 2017-11-07 10:46:42 -08:00
rht ec5120e807 refactor: Remove six.moves.zip import. 2017-11-07 10:46:42 -08:00
neiljp (Neil Pilgrim) 952f034136 mypy: Set explicit Any-like parameters for handle_callback_exception in runtornado.py. 2017-11-04 19:47:45 -07:00
rht 839b427ea5 zerver/management: Remove u prefix from strings.
License: Apache-2.0
Signed-off-by: rht <rhtbot@protonmail.com>
2017-11-02 11:01:47 -07:00
rht c4fcff7178 refactor: Replace super(.*self) with Python 3-specific super().
We change all the instances except for the `test_helpers.py`
TimeTrackingCursor monkey-patching, which actually needs to specify
the base class.
2017-10-30 14:30:25 -07:00
Steve Howell a28841e8aa Extract get_stream_recipient().
Do you call get_recipient(Recipient.STREAM, stream_id) or
get_recipient(stream_id, Recipient.STREAM)?  I could never
remember, and it was not very type safe, since both parameters
are integers.
2017-10-28 17:57:39 -07:00
Tim Abbott 8e2cdedf9a lint: Fix lines in Python codebase longer than 120 characters. 2017-10-26 17:47:30 -07:00
Tim Abbott b936e8c24b lint: Fix lines in Python codebase longer than 125 characters. 2017-10-26 17:36:54 -07:00
Steve Howell df93a99b50 Cache only one row per message.
Before this change, we populated two cache entries for each
message that we sent.  The entries were largely redundant,
with the only difference being whether we sent the content
as raw markdown or as the rendered HTML.

This commit makes it so we only have one cache entry per
message, and it includes both content and rendered_content.

One legacy source on confusion here is that `content`
changes meaning when you're on the front end.  Here is the
situation going forward:

    database:
        content = raw
        rendered_contented = rendered

    cache entry:
        content = raw
        rendered_contented = rendered

    payload for the frontend:
        content = raw (for apply_markdown=False)
        content = rendered (for apply_markdown=True)
2017-10-26 16:35:28 -07:00
rht a311678190 zerver/management: Use python 3 syntax for typing. 2017-10-26 15:24:56 -07:00
derAnfaenger 1792dcbd09 tests: Call real consume method of queue processors.
This switches to more real tests for a first batch of
queue_json_publish() calls that don't cause trouble when
used with call_consume_tests=True.
2017-10-26 14:58:03 -07:00
Vishnu Ks 25cb6af32c mangement: Create command for creating default stream groups. 2017-10-26 11:38:48 -07:00
Umair Khan 19ccff060a i18n: Extract language options from tracked locales.
Now we use 'git ls-files' to get the list of locales that we actually
track. Previously we were using os.listdir to get the contents of the
static/locale directory. This could also return locales which were
present in the directory but are not supported by us, e.g. zh_CN.
2017-10-23 22:58:44 -07:00
Umair Khan 2db1cba9cc i18n: Use language code instead of locale.
We have been assigning locale to language code. Mostly code and locale
are same but for languages like zh-Hans, locale is zh_Hans and code is
zh-hans.

After this commit, compilemessages command should be run.
2017-10-20 11:21:26 -07:00
Umair Khan df13ac570a i18n: Add localized name in language_name_map.json. 2017-10-20 11:20:22 -07:00
Umair Khan 3769972fc2 i18n: Clean up code to create language_options.json.
Previously we were using regexes to extract the language from our
locale files. Now we use LANG_INFO data structure provided by Django
to do the same and fallback to PO files only when language code is not
present in the Django data structure.
2017-10-20 11:20:22 -07:00
Tim Abbott 03d72a1cd6 send_webhook_fixture_message: Print useful error messages.
Previously, if this failed, it didn't tell you why.
2017-10-20 09:03:05 -07:00
derAnfaenger 3264e4f24a Remove superfluous queue_json_publish imports. 2017-10-19 13:57:54 -07:00
Tim Abbott 3983ebce9a test_fixtures: Move test_db_status state under UUID_VAR_PATH.
This should mean that maintaining two Zulip development environments
using the same Git checkout no longer has caching problems keeping
track of the migration status.
2017-10-17 21:15:58 -07:00
Tim Abbott b7c3b4df0c migration_status: Switch to passing the path in Python.
This is an important prerequisite to changing these paths used in
provisioning.
2017-10-17 21:15:58 -07:00
Tim Abbott 15c0380a33 export: Fix exporting with an output file at a relative path.
Fixes #7051.
2017-10-17 20:47:29 -07:00
Tim Abbott 66a3f514d4 zilencer: Remove long-obsolete Deployment model.
This was a precursor to RemoteZulipServer, which is no longer used for
anything, and was cluttering the codebase.
2017-10-11 23:48:01 -07:00
Tim Abbott 49274a060c send_webhook_fixture_message: Fix to work with subdomains.
We apparently failued to update this tool when we switched the
development environment to require a subdomain.
2017-10-08 20:14:14 -07:00
rht fbd24669a4 Cleanup: Remove shebang and executable flag from these files. 2017-10-08 15:38:15 -07:00
derAnfaenger 747c283180 lint: Replace `pk` with `id` where they are equivalent. 2017-10-06 12:52:12 -07:00
Tim Abbott 0f2e4a55c0 soft deactivation: Shorten management command name.
This command is really for soft deactivation; there's just an undo
feature.
2017-10-06 08:48:43 -07:00
Tim Abbott ca59b28fc1 import: Fix positional argument parsing. 2017-10-06 08:45:22 -07:00
Tim Abbott bfedb82097 import: Fix argparse usage message breakage. 2017-10-06 08:41:01 -07:00
Umair Khan 326a6f6b4f i18next: Don't allow empty string as valid translation.
Previously we used to mark a key as unstranlated if its value was equal
to it in translations.json. This had an issue because it didn't allow
otherwise valid cases where key was equal to the value.

This commit solves the problem by disallowing an empty string as a valid
translation and then using the empty string as the value for all the
unstranslated keys.

Fixes #5261
2017-10-05 22:44:34 -07:00
Umair Khan b464fd7643 makemessages: Change type of translation_strings to List.
In the code this variable is only used as a list, however, we
create it as dictionary which uses more memory.
2017-10-05 22:44:34 -07:00
Tim Abbott 13bb546ddd subdomains: Hardcode can_add_realm_domains=True.
It was already always True if REALMS_HAVE_SUBDOMAINS, and now that's
the only supported configuration.
2017-10-02 16:32:10 -07:00
derAnfaenger d1afab7199 Replace deprecated Logging.warn calls with Logging.warning. 2017-10-02 11:11:42 +02:00
rht 6dd94becb3 zerver/management: Remove unused optparse import. 2017-09-30 09:22:08 -07:00
rht f22daeba1a Cleanup rabbitmq queues when doing dev db rebuild.
Modified by tabbott to cleanup the argument handling in the modified
purge_queue management command.

Fixes #1335.
2017-09-28 18:18:54 -07:00
rht 32650c8fdc zerver/management: Remove print_function. 2017-09-27 18:05:45 -07:00
rht 0fa3c1d876 zerver/management: Remove `import six`. 2017-09-27 17:06:16 -07:00
rht e239e97351 zerver/management: Remove absolute_import. 2017-09-27 10:00:39 -07:00
neiljp (Neil Pilgrim) ab4cbf81f0 mypy: Add assertions of get_realm results when parser requires realm. 2017-09-26 13:49:01 -07:00
Greg Price 7737326cf0 mypy: Add an Optional in fix_unreads command. 2017-09-26 13:49:01 -07:00
Greg Price c10aca1bbf soft deactivation: Simplify logic slightly in management command.
The `get_realm` method works properly regardless of whether the
'realm_id' option was set, so checking for that option first is
redundant.
2017-09-26 13:49:01 -07:00
neiljp (Neil Pilgrim) acf7e2a9ee mypy: Set conditional to be against result of get_realm, not input.
[Tweaked by greg to use explicit `is not None`.]
2017-09-26 13:49:01 -07:00
Vishnu Ks f3b4de822b command: Create multiuse user invite generation command. 2017-09-22 19:48:46 -07:00
Tim Abbott 8badd10a6c send_test_email: Remind users to restart the server after updates. 2017-09-22 13:32:22 -07:00
Umair Khan 1f93c06b76 i18n: Optimize get_language_list().
compilemessages command now does all the heavy lifting by creating a
language_name_map.json file under locale directory. This file is used
by get_language_list to retrieve the require information.

Fixes: #6486
2017-09-14 02:28:58 -07:00
Steve Howell 69203c1c81 fix_unreads: Remove commit() call in fix().
The commit() call in fix() breaks migrations and tests (unless you
mock) due to outer transactions.

We now explicitly call commit() from the management command.
2017-09-07 07:06:03 -07:00
Steve Howell 8cc8e87daf fix_unreads: Use logging instead of print. 2017-09-07 07:06:03 -07:00
Steve Howell a2fe4178be Extract zerver/lib/fix_unreads.py.
This is a pure code move.
2017-09-07 07:06:03 -07:00
Umair Khan f7d8db792c makemessages: Allow whitespaces after comma in i18n.
We allow such patterns:

```
i18n.t('Test __variable__',
        {variable: "script"})
```
2017-09-06 07:01:43 -07:00
Steve Howell 4ac6bc46c7 Add MutedTopic model.
This commit completely switches us over to using a
dedicated model called MutedTopic to track which topics
a user has muted.

This includes the necessary migrations to create the
table and populate it from legacy data in UserProfile.

A subsequent commit will actually remove the old field
in UserProfile.
2017-09-02 09:19:51 -07:00
Steve Howell 828459a24b Extract build_topic_mute_checker into topic_mutes.py.
We had two duplicate versions of this function, and one
of them was broken with respect to case insensitivity.
2017-08-29 16:53:38 -04:00
Yago González 659bff1ffb i18n: Fix URLs misparsing in translation tags.
The double forward slash (//) after the protocol in URLs was being
mistakenly considered the beginning of an inline JS comment, causing
internationalization strings being cut unexpectedly.

Now the check for inline JS comments is only run in .js files.
2017-08-28 13:54:17 -07:00
Aditya Bansal d9c9bfe7f6 logger: Add new create_logger abstraction to simplify logging.
This deduplicates a ton of Python logger-creation code to use a single
standard implementation, so we can avoid copy-paste problems.
2017-08-27 18:31:53 -07:00
Vishnu Ks 8fc8ac0799 management: Override CommandError to mention --entire-server argument. 2017-08-27 12:34:23 -07:00
Aditya Bansal b232563e12 soft-deactivation: Add cron job for weekly soft deactivating users. 2017-08-27 11:33:06 -07:00
Aditya Bansal 9d7e23c100 softdeactivation/management: Make specifying realm an optional arg. 2017-08-27 11:33:06 -07:00
Preston Hansen e8a608f733 management: Move enqueue_digest_email handler to digest. 2017-08-27 10:13:11 -07:00