Commit Graph

10152 Commits

Author SHA1 Message Date
Anders Kaseorg 68dd8e4ec8 mypy: Migrate from mypy_extensions to typing_extensions.
This gives us access to typing_extensions.Deque, which was not added
to typing until 3.5.4.

(PROVISION_VERSION is not bumped because the transitive dependency set
in dev.txt hasn’t changed.)

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-05 17:24:09 -07:00
Anders Kaseorg 86a7fdddd7 events: Check last_event_id for validity, take 2.
This verifies that the client passed a last_event_id that actually
came from the queue instead of making up an ID from the future.  It
turns out one of our tests was making up such an ID, but legitimate
clients are expected not to do so.

The previous version of this commit (commit
e00d4be6d5, #12888) had to be reverted
(commit b86c5cc490) because it was
missing the `to_dict`/`from_dict` migration code.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-05 17:18:49 -07:00
Tim Abbott 441bb844d2 models: Improve comments about get_user_profile_by_*_email.
This should help avoid folks accidentally using methods they should
not.
2019-08-05 17:10:58 -07:00
Tim Abbott ba67a9b76a message: Use RawUnreadMessageResult typing more.
This is a cleaner implementation given we have the TypedDict defined.
2019-08-02 17:34:27 -07:00
Tim Abbott 8f7a86f5ec events: Move remove_message_id_from_unread_mgs to message.py.
It belongs there, next to the related code for adding an event.
2019-08-02 17:26:13 -07:00
Tim Abbott 2c7511023d events: Simplify arguments for remove_message_id_from_unread_mgs.
This should make it reusable by the event_queue.py code path.
2019-08-02 17:26:13 -07:00
Rohitt Vashishtha a7f2bedb15 markdown: Enable hashheadings syntax.
Our implementation requires at least 1 space after the
'#' not not break existing linkifiers like '#123', etc.
that generally follow the convention we show in linkifier
examples.

- [valid]  : # Hello
- [valid]  : #  Hello
- [invalid]: #Hello

For the frontend, we have taken the code from v0.7.0 of
upstream marked and made minor changes to avoid having
to refactor a significant part of our marked code.

For the backend, we merely have to change the regex to
force require spaces after #, and add hashheader to our
list of blockparsers.

Fixes #11418.
2019-08-02 15:15:34 -07:00
Tim Abbott b86c5cc490 Revert "events: Check last_event_id for validity."
This isn't correct without a proper migration for existing queues,
which may not be implementable.

This reverts commit e00d4be6d5.
2019-08-02 14:44:35 -07:00
Tim Abbott 2738e909f8 event_queue: Expand testing of event queue save/restore.
This verifies that every valid state of our event queue system will be
properly save/restored via the from_dict methods.
2019-08-02 14:43:47 -07:00
Tim Abbott 3a6e5dad53 tests: Move EventQueueTest to test_event_queue.py.
This seems more appropriate, especially as we expand this library.
2019-08-02 14:43:12 -07:00
Tim Abbott 8be3df0e29 tornado: Fix bugs in Tornado autoreload library.
This fixes two issues:

* The syntax check logic we had for zerver.tornado.autoreload would
  end up clearing _reload_hooks if one of the files that had changed
  was zerver.tornado.autoreload itself (because we'd had re-imported
  the current module), which could be incredibly confusing when trying
  to test the autoreload logic.  It seems better to just not run the
  syntax check for syntax errors in this file.

  Similarly, because reloading event_queue.py would destroy the state
  in the queues, we avoid that as well.

* We make sure to flush stdout after running and reload hooks, to make
  sure their output reaches the user.
2019-08-02 12:47:49 -07:00
Tim Abbott a43b2f7a43 tornado: Fix incorrect import of autoreload library.
We were apparently not running our own forked Tornado autoreload
library when adding reload hooks, which meant that our autoreload
hooks didn't run at all.

This fixes an issue that made dump_event_queues never run and thus the
local development environment difficult to use for testing event queues.
2019-08-02 12:47:49 -07:00
Mateusz Mandera d3d62f99e4 tests: Adjust to new changes in populate_db. 2019-07-31 16:01:37 -07:00
Mateusz Mandera 9b2b24c617 production: Rename the 'zulip' system bot realm to 'zulipinternal'. 2019-07-31 16:01:36 -07:00
Tim Abbott e1d9fe84ad clubhouse: Type payload as Optional.
Apparently if you post json.dumps(None), one will end up with this
None case being possible.
2019-07-31 15:31:57 -07:00
Tim Abbott 78df15ef08 Revert "clubhouse: Remove dead code."
This reverts commit 48ea427e87.
2019-07-31 15:31:57 -07:00
Wyatt Hoodes b005c7567a messages: Fix typing errors.
We remove a redundant term conversion test, as well as
add appropriate typing for the `anchor` variable in
`get_messages_backend`.
2019-07-31 12:42:29 -07:00
Tim Abbott 48ea427e87 clubhouse: Remove dead code. 2019-07-31 12:40:01 -07:00
Tim Abbott 9599d311a5 logging_util: Add an extra type: ignore. 2019-07-31 12:40:01 -07:00
Wyatt Hoodes c787f0a550 webhooks/jira: Fix useless check for subject being None.
This wasn't possible since a recent rewrite of the webhook code.
2019-07-31 12:27:04 -07:00
Tim Abbott acd52be820 pivotal: Remove useless conditionals.
It isn't actually possible for these values to be None.
2019-07-31 12:26:21 -07:00
Wyatt Hoodes b807c4273e middleware: Fix exception typing.
Mypy seems to have trouble understanding `Exception` inheritance
here, so we create a `Union` for the only `Exception` we are
looking for.
2019-07-31 12:23:20 -07:00
Wyatt Hoodes 167d0bad61 test_runner: Unionize suite type.
Mypy cannot see the class hierarchy pertaining to django's
ParallelTestSuite. To deal with this, we simply create a
`Union` of our two types.
2019-07-31 12:20:12 -07:00
Wyatt Hoodes 4beec5c6b9 typing: Use TYPE_CHECKING when dealing with cyclic dependencies. 2019-07-31 12:19:39 -07:00
Wyatt Hoodes fa227d79af typing: Fix options typing in management/commands.
We simply state that certain options are `Optional`.

The following files are affected:

add_users_to_mailing_list
send_to_email_mirror
fill_memcached_caches
client_activity
2019-07-31 12:13:56 -07:00
Wyatt Hoodes 457842ccd6 add_users_to_mailing_list: Prepare for using Optional typing.
When typing `**options` as an `Optional[str]` we will see errors
in the from of `None type has no attribute 'split'`.  This change
allows mypy to effectively handle the `None` case.
2019-07-31 12:13:56 -07:00
Tim Abbott 997fd714d1 openapi: Fix some errors caught by online editor.
This fixes several values that had invalid types recorded.
2019-07-30 16:36:50 -07:00
Tim Abbott 782cb2e13c openapi: Remove duplicate announce section. 2019-07-30 15:08:59 -07:00
Tim Abbott 9827801569 slack import: Improve readability of user recipient object allocation.
This loop management tweak makes it a bit more obvious what's
happening in this block of code.
2019-07-30 14:46:14 -07:00
Vishnu KS ff3871fc63 slack_import: Clean up return values of channels_to_zerver_stream.
This commits reduces the number of values returned by
channel_to_zerver_stream function by setting the values
directly in realm dict and returning it instead.
2019-07-30 14:46:14 -07:00
Vishnu Ks 6110f495df slack_import: Support importing pms. 2019-07-30 14:46:14 -07:00
Mateusz Mandera 8f6f78b912 requirements: Upgrade redis from 2.10.6 to 3.2.1.
Fixes #11209.

This requires changing how zadd is used in rate_limiter.py:
In redis-py >= 3.0 the pairs to ZADD need to be passed as a dictionary,
not as *args or **kwargs, as described at
https://pypi.org/project/redis/3.2.1/ in the section
"Upgrading from redis-py 2.X to 3.0".

The rate_limiter change has to be in one commit with the redis upgrade,
because the dict format is not supported before redis-py 3.0.
2019-07-29 15:34:05 -07:00
Wyatt Hoodes ecb1b1a1e1 openapi: Fix typing for self.data. 2019-07-29 15:24:51 -07:00
Wyatt Hoodes b01ba48369 migration_0060: Remove unreachable return statement.
This was introduced in a past refactoring.
2019-07-29 15:23:34 -07:00
Wyatt Hoodes f22b2c7e9d archive: Fix typing for prev_sender variable.
We know that via the `AbstractMessage` class that `sender`
is of the type `UserProfile`.  We type this as `Optional`
to tell mypy that the operands to the right of the first
`or` can indeed be evaluated within the following `for` loop.
2019-07-29 15:23:10 -07:00
Wyatt Hoodes 99bd55917e zephyr: Fix typing for cred parameter. 2019-07-29 15:23:10 -07:00
Wyatt Hoodes 0834514668 auth: Fix typing for authenticate_remote_user. 2019-07-29 15:23:10 -07:00
Wyatt Hoodes 4149bac696 actions: Fix typing of query_filter. 2019-07-29 15:23:10 -07:00
Wyatt Hoodes 1706e06884 bugdown/init: Fix typing for fence variable. 2019-07-29 15:23:10 -07:00
Wyatt Hoodes 22481f63bf upload: Fix typing for key variable. 2019-07-29 15:23:10 -07:00
Wyatt Hoodes a75abe096b migration_0064: Fix typing for file_key. 2019-07-29 15:23:10 -07:00
Wyatt Hoodes e106f95c8b migration_0064: Remove usage of Text type. 2019-07-29 15:23:10 -07:00
Wyatt Hoodes 8107230b35 domains: Fix typing for validate_domain. 2019-07-29 15:18:24 -07:00
Wyatt Hoodes a109508e34 typing: Remove now-unnecessary conditional import.
As a result of dropping support for trusty, we can remove our old
pattern of putting `if False` before importing the typing module,
which was essential for Python 3.4 support, but not required and maybe
harmful on newer versions.

cron_file_helper
check_rabbitmq_consumers
hash_reqs
check_zephyr_mirror
check_personal_zephyr_mirrors
check_cron_file
zulip_tools
check_postgres_replication_lag
api_test_helpers
purge-old-deployments
setup_venv
node_cache
clean_venv_cache
clean_node_cache
clean_emoji_cache
pg_backup_and_purge
restore-backup
generate_secrets
zulip-ec2-configure-interfaces
diagnose
check_user_zephyr_mirror_liveness
2019-07-29 15:18:22 -07:00
Anders Kaseorg e00d4be6d5 events: Check last_event_id for validity.
This verifies that the client passed a last_event_id that actually
came from the queue instead of making up an ID from the future.  It
turns out one of our tests was making up such an ID, but legitimate
clients are expected not to do so.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-07-26 17:18:28 -07:00
Wyatt Hoodes bbbea9ec87 events: Rewrite system for managing realm exports.
This feature is intended to cover all of our ways of exporting a
realm, not just the initial "public export" feature, so we should name
things appropriately for that goal.

Additionally, we don't want to include data exports in page_params;
the original implementation was actually buggy and would have.
2019-07-26 16:38:52 -07:00
Wyatt Hoodes b1900c406a public_export: Add logic for deleting the export tarball.
The path to the uploaded tarball is reconstructed via the relative url and
removed with the canonical methods in `upload.py`.
2019-07-26 15:52:03 -07:00
Wyatt Hoodes ef02de4834 public_export: Add endpoint for returning all REALM_EXPORTED objects. 2019-07-26 15:52:02 -07:00
Wyatt Hoodes d070f27359 queue_processors: Change the extra_data field to a relative url path.
A better approach as compared to saving the full public url.
2019-07-26 15:50:02 -07:00
Wyatt Hoodes 266ed9ff75 public_export: Add acting_user to RealmAuditLog object.
An acting_user attribute is required to correctly reference who
initiated a realm_exported event.
2019-07-26 15:49:47 -07:00