Commit Graph

4493 Commits

Author SHA1 Message Date
Tim Abbott 8f7a86f5ec events: Move remove_message_id_from_unread_mgs to message.py.
It belongs there, next to the related code for adding an event.
2019-08-02 17:26:13 -07:00
Tim Abbott 2c7511023d events: Simplify arguments for remove_message_id_from_unread_mgs.
This should make it reusable by the event_queue.py code path.
2019-08-02 17:26:13 -07:00
Rohitt Vashishtha a7f2bedb15 markdown: Enable hashheadings syntax.
Our implementation requires at least 1 space after the
'#' not not break existing linkifiers like '#123', etc.
that generally follow the convention we show in linkifier
examples.

- [valid]  : # Hello
- [valid]  : #  Hello
- [invalid]: #Hello

For the frontend, we have taken the code from v0.7.0 of
upstream marked and made minor changes to avoid having
to refactor a significant part of our marked code.

For the backend, we merely have to change the regex to
force require spaces after #, and add hashheader to our
list of blockparsers.

Fixes #11418.
2019-08-02 15:15:34 -07:00
Tim Abbott 9599d311a5 logging_util: Add an extra type: ignore. 2019-07-31 12:40:01 -07:00
Wyatt Hoodes 167d0bad61 test_runner: Unionize suite type.
Mypy cannot see the class hierarchy pertaining to django's
ParallelTestSuite. To deal with this, we simply create a
`Union` of our two types.
2019-07-31 12:20:12 -07:00
Wyatt Hoodes 4beec5c6b9 typing: Use TYPE_CHECKING when dealing with cyclic dependencies. 2019-07-31 12:19:39 -07:00
Mateusz Mandera 8f6f78b912 requirements: Upgrade redis from 2.10.6 to 3.2.1.
Fixes #11209.

This requires changing how zadd is used in rate_limiter.py:
In redis-py >= 3.0 the pairs to ZADD need to be passed as a dictionary,
not as *args or **kwargs, as described at
https://pypi.org/project/redis/3.2.1/ in the section
"Upgrading from redis-py 2.X to 3.0".

The rate_limiter change has to be in one commit with the redis upgrade,
because the dict format is not supported before redis-py 3.0.
2019-07-29 15:34:05 -07:00
Wyatt Hoodes ecb1b1a1e1 openapi: Fix typing for self.data. 2019-07-29 15:24:51 -07:00
Wyatt Hoodes 4149bac696 actions: Fix typing of query_filter. 2019-07-29 15:23:10 -07:00
Wyatt Hoodes 1706e06884 bugdown/init: Fix typing for fence variable. 2019-07-29 15:23:10 -07:00
Wyatt Hoodes 22481f63bf upload: Fix typing for key variable. 2019-07-29 15:23:10 -07:00
Wyatt Hoodes 8107230b35 domains: Fix typing for validate_domain. 2019-07-29 15:18:24 -07:00
Wyatt Hoodes a109508e34 typing: Remove now-unnecessary conditional import.
As a result of dropping support for trusty, we can remove our old
pattern of putting `if False` before importing the typing module,
which was essential for Python 3.4 support, but not required and maybe
harmful on newer versions.

cron_file_helper
check_rabbitmq_consumers
hash_reqs
check_zephyr_mirror
check_personal_zephyr_mirrors
check_cron_file
zulip_tools
check_postgres_replication_lag
api_test_helpers
purge-old-deployments
setup_venv
node_cache
clean_venv_cache
clean_node_cache
clean_emoji_cache
pg_backup_and_purge
restore-backup
generate_secrets
zulip-ec2-configure-interfaces
diagnose
check_user_zephyr_mirror_liveness
2019-07-29 15:18:22 -07:00
Wyatt Hoodes bbbea9ec87 events: Rewrite system for managing realm exports.
This feature is intended to cover all of our ways of exporting a
realm, not just the initial "public export" feature, so we should name
things appropriately for that goal.

Additionally, we don't want to include data exports in page_params;
the original implementation was actually buggy and would have.
2019-07-26 16:38:52 -07:00
Wyatt Hoodes b1900c406a public_export: Add logic for deleting the export tarball.
The path to the uploaded tarball is reconstructed via the relative url and
removed with the canonical methods in `upload.py`.
2019-07-26 15:52:03 -07:00
Wyatt Hoodes ef02de4834 public_export: Add endpoint for returning all REALM_EXPORTED objects. 2019-07-26 15:52:02 -07:00
Hemanth V. Alluri ede73ee4cd streams: New stream messages should be auto marked as read by creator.
When a person creates a new realm, they'll likely want to create a
bunch of initial streams at once. When doing so, it could be annoying
to have to mark all of the new stream notification messages as read.

Thus to make this process smoother, we should automatically mark
the messages generated by the Notification Bot in the notifications
(announcements) stream, as well as in the newly created stream itself
as read by the stream creator.

Fixes #12765.
2019-07-26 15:10:09 -07:00
Wyatt Hoodes 1c8106fc67 timezone: Fix return type in get_timezone.
While it's true `datetime` is implicit via `pytz`, it makes sense
that mypy should now complain about the semantics of calling our
return type `pytz.datetime.tzinfo`, when such a type doesn't
actually exist.
2019-07-25 17:41:01 -07:00
Tim Abbott b8a1050fc4 rest: Disable caching for all REST API endpoints.
Investigation into #12876, a mysterious bug where users were seeing
messages reappear as unread, determined that the root cause was
missing headers to disable client-side caching for Zulip's REST API
endpoints.

This manifested, in particular, for `GET /messages`, which is
essentially the only API GET endpoint used by the webapp at all.  When
using the `Ctrl+Shift+T` feature of browsers to restore a recently
closed tab (and potentially other code paths), the browser would
return from its disk cache a cached copy of the GET /messages results.

Because we include message flags on messages fetched from the server,
this in particular meant that those tabs would get a stale version of
the unread flag for the batches of the most recent ~1200 messages that
Zulip fetches upon opening a new browser tab.

The issue took same care to reproduce as well, in large part because
the arguments to those initial GET /messages requests will vary as one
reads messages (because the `pointer` moves forward) and then enters
the "All messages" view; the disk cache is only used for GET requests
with the exact same URL parameters.

We will probably still want to merge the events error-handling changes
we had previously proposed for this, but the conclusion of this being
a straightforward case of missing cache-control headers is much more
satisfying than the "badly behaving Chrome" theory discussed in the
issue thread.

Fixes #12876.
2019-07-25 12:42:46 -07:00
Anders Kaseorg 2b33822de9 storage: Stop using django-pipeline.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-07-24 17:40:31 -07:00
Anders Kaseorg fd7803e7f4 settings: Unset STATIC_ROOT in development.
Django’s default FileSystemFinder disallows STATICFILES_DIRS from
containing STATIC_ROOT (by raising an ImproperlyConfigured exception),
because STATIC_ROOT is supposed to be the result of collecting all the
static files in the project, not one of the potentially many sources
of static files.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-07-24 17:40:31 -07:00
Anders Kaseorg a97a2612bb integrations: Get logo URLs from staticfiles.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-07-24 17:40:31 -07:00
Tim Abbott daca742e9f email_gateway: Use html2text directly.
In the rare case that Zulip receives an email with only an HTML
format, we originally (code dating to 2013) shelled out to
html2markdown/python-html2text in order to convert the HTML into
markdown.

We long since added html2text as a reasonably managed Python
dependency of Zulip; we should just use it here.
2019-07-24 12:42:17 -07:00
Mateusz Mandera 11862e5ce0 test_classes: Use subdomain kwarg in api_* functions instead of realm.
It's more appropriate for the kwarg to be named subdomain. We also
update the functions, so that this kwarg is used in all of them
consistently.
2019-07-23 15:05:39 -07:00
Mateusz Mandera 7f24e27610 test_classes: Rename first argument of api_* functions to "identifier".
This was the name in api_post already - we rename it in the other
functions, for consistency.
2019-07-23 15:05:39 -07:00
Eeshan Garg 8e10ab282a webhooks: Log unexpected webhook events separately.
This change serves to declutter webhook-errors.log, which is
filled with too many UnexpectedWebhookEventType exceptions.

Keeping UnexpectedWebhookEventType in zerver/lib/webhooks/common.py
led to a cyclic import when we tried to import the exception in
zerver/decorators.py, so this commit also moves this exception to
another appropriate module. Note that our webhooks still import
this exception via zerver/lib/webhooks/common.py.
2019-07-22 18:20:53 -07:00
Conner Bondurant 4d4b6b8319 email: Convert send_email to use formataddr.
Changed the requirements for UserProfile in order to allow use of
the formataddr function in send_mail.py.

Converted send_email to use formataddr in conjunction with the commit
that strengthened requirements for full_name, such that they can now be
used in the to field of emails.

Fixes #4676.
2019-07-22 18:15:58 -07:00
Conner Bondurant c25dcf048d models: Enforce stricter requirements on the full_name field.
This changes the requirements for UserProfile to disallow some
additional characters, with the overall goal of being able to use
formataddr in send_mail.py.

We don't need to be particularly careful in the database migration,
because user full_names are not required to be unique.
2019-07-22 18:13:34 -07:00
neiljp (Neil Pilgrim) ba7a0934e3 requirements: Upgrade mypy to 0.711.
This comes with it a big performance improvement; mypy is now only
barely our slowest linter even if it wasn't previously running.

Fixes: #12058
2019-07-22 17:12:50 -07:00
neiljp (Neil Pilgrim) 80a4504784 mypy: Clean up typing for CustomProfileField code. 2019-07-22 17:10:50 -07:00
Wyatt Hoodes e4133e687f test_runner: Ignore inheritance issues in TestSuite.
Again, `result` of type `TestResult` is considered by mypy to
not have these _attributes acquired through inheritance. So,
we ignore them.
2019-07-22 17:06:41 -07:00
Wyatt Hoodes 5686821150 middleware: Change write_log_line to publish as a dict.
We were seeing errors when pubishing typical events in the form of
`Dict[str, Any]` as the expected type to be a `Union`.  So we instead
change the only non-dictionary call, to pass a dict instead of `str`.
2019-07-22 17:06:41 -07:00
Wyatt Hoodes dd19e2df1b test_runner: Ignore self.stream in TextTestResult class.
Per the import line:

`from unittest import loader, runner  # type: ignore  # Mypy cannot pick
these up.`

Because `TextTestResult` inherits from `runner.TextTestResult`, mypy
doesn't see `self` as having an attribute `stream`, so we ignore these
instead of cluttering with `casts` or `isinstances`.
2019-07-22 16:45:21 -07:00
Tim Abbott d3f59db8ce test_runner: Cleanup typing for TestResult objects.
There's two possible types here.
2019-07-22 16:45:21 -07:00
Wyatt Hoodes 76e8dc1657 test_runner: Change type to unittest.TestSuite.
In `deserialize_suite`, it passes a test as being of the
super type, where just having TestSuite assumes the type of
`zerver.Runner.TestSuite`.
2019-07-22 16:27:39 -07:00
Wyatt Hoodes 3509fe07b5 test_runner: Fix incorrect type for enforce_timely_test_completion.
Our TestResult objects are always actually TextTestResults.
2019-07-22 16:27:39 -07:00
Tim Abbott 02d55928ea import: Fix importing slack avatars into S3_UPLOAD_BACKEND.
Apparently, a subtle mismatch between the filename/URL formats for our
upload codebases meant that importing Slack avatars into systems using
S3_UPLOAD_BACKEND would end up with the avatars having the wrong URLs.
2019-07-21 21:25:31 -07:00
Hemanth V. Alluri d62b75fc4c openapi: Remove /users/me/subscriptions from pending_endpoints.
In commit 7c71e98, we added a special exception for the
/users/me/subscriptions endpoint in the automatic validation test.
By adding some extra documentation, we now remove this extra code,
as well as the endpoint from the list of pending endpoints.
2019-07-21 18:39:53 -07:00
Rishi Gupta 1a6f8cdf17 emails: Remove unsubscribe link from missed message emails.
This allows the footer to be two lines long instead of three.
2019-07-21 15:10:48 -07:00
Tim Abbott 21c8a7ec36 i18n: Fix strings in send_pm_if_empty_stream.
The previous iteration did not properly handle languages with a
different word order than English.

Discovered via warning output in `manage.py makemessages`.
2019-07-21 14:43:50 -07:00
Mateusz Mandera f1b135bd16 email_mirror: Rename include-quotations to include-quotes. 2019-07-20 15:53:43 -07:00
Mateusz Mandera 519ed41803 email_mirror: Display stream email address with . as the separator. 2019-07-20 15:53:43 -07:00
Mateusz Mandera 8bbd1d005d email_mirror: Support both + and . as separators in the address. 2019-07-20 15:53:42 -07:00
Wyatt Hoodes e331a758c3 python: Migrate open statements to use with.
This is low priority, but it's nice to be consistently using the best
practice pattern.

Fixes: #12419.
2019-07-20 15:48:52 -07:00
Mateusz Mandera 842d9e944a test_classes: Prepare for moving system bots to zulipinternal. 2019-07-20 14:55:21 -07:00
Rishi Gupta 38eeb3e183 notification bot: Add constant for STREAM_EVENTS_NOTIFICATION_TOPIC. 2019-07-20 14:39:15 -07:00
Rishi Gupta c9f194be99 notification bot: Update message for stream name changes.
Also properly internationalizes string formatting arguments for the stream
created message.
2019-07-20 14:39:15 -07:00
Rohitt Vashishtha 726d5003e1 bugdown: Force absolute urls in topic links.
If a url doesn't have a scheme, browsers would treat it as a relative
url and open something like: https://chat.zulip.org/google.com instead.

This PR fixes the issue on the backend; the frontend implementation
remains out of sync and the user sending the message wouldn't see
any linkification for urls without a scheme.

Fixes #12791.
2019-07-19 12:02:52 -07:00
Anders Kaseorg afa251de5d dev_urls: Serve static files with staticfiles_urlpatterns.
The test_docs change is because Django runs test cases with DEBUG =
False, which ordinarily means it doesn’t serve /static during tests.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-07-19 11:47:51 -07:00
Anders Kaseorg a7eb3faf96 storage: Move the staticfiles.json hack to ZulipStorage.
There’s no reason to monkey-patch something that we were already
subclassing.

Removing the PRODUCTION conditional causes us to generate
staticfiles.json in the right place to begin with so we don’t need to
move it later.  It also allows Django to find staticfiles.json if
running the dev server with PIPELINE_ENABLED.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-07-19 11:47:51 -07:00