Commit Graph

33052 Commits

Author SHA1 Message Date
Wyatt Hoodes f623540409 data export: Add UI to trigger data export.
This commit serves as the frontend piece for the "public export"
webapp feature.

Fixes: #11930
2019-08-12 18:21:38 -07:00
Tim Abbott 8ac8247dbe data export: Add limits on exporting large organizations.
It seems possible that attempting to export large organizations could
result in high resource consumption that justifies having a technician
manage the exports manually.
2019-08-12 18:21:09 -07:00
Tim Abbott 1d63f129bf data export: Add summary logging of runtime.
This should help us investigate cases where this runs a very long
time.
2019-08-12 18:21:08 -07:00
Wyatt Hoodes 896886b412 realm_export: Add endpoint for export deletion. 2019-08-12 17:51:46 -07:00
Wyatt Hoodes 777dcdd272 test_realm_export: Clean up export testing.
* Added comments.
* Refactored `export_object` to `audit_log_entry`.
* Cleaned up white space.
* Removed uses of `getattr`.
2019-08-12 17:51:46 -07:00
Wyatt Hoodes 7c118096e7 test_events: Remove print statement from self.client_post.
Leftover from prior debugging.
2019-08-12 17:51:46 -07:00
Wyatt Hoodes 7d178bbb0f queue_processors: Clean up the extra_data dict code.
We don't want to add a `deleted_timestamp` key until
the export is actually deleted.
2019-08-12 17:51:46 -07:00
Wyatt Hoodes 6f6efa516d exports: Refactor extra_data to export_data. 2019-08-12 17:51:46 -07:00
Wyatt Hoodes 7a2a1f29ad exports: Refactor event_time to export_time timestamp.
The time of the event was incorrectly being sent
as a datetime object.
2019-08-12 17:51:46 -07:00
Anders Kaseorg e0cf3d0e76 zerver: Accept HEAD requests wherever GET requests are accepted.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-12 16:47:41 -07:00
Anders Kaseorg ead13c11e5 run-dev: Don’t rewrite the Content-Length header.
A HEAD response has a Content-Length but no body; it’s not correct in
that case to let Tornado default Content-Length to 0.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-12 16:47:41 -07:00
Hemanth V. Alluri d425ab5ae5 test_templates: Add a verbose error messages for rendering failures.
If a template fails to render in test_templates because of a missing
context variable, we should better explain the reason for the failure.
2019-08-12 16:24:08 -07:00
Anders Kaseorg 09e7a688b0 Revert "vagrant: Add NFS backend for file synchronization for OSX."
Reportedly the NFS synced folder type doesn’t work on macOS with the
Docker provider, while the default type does.  And it seems likely
that four years of Ubuntu updates will have resolved the guest crashes
that motivated the prior switch to NFS.

https://chat.zulip.org/#narrow/stream/21-provision-help/topic/experimental.20docker.20setup

This reverts commit 9e8ae68dfb (#7520).

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-12 16:04:00 -07:00
Anders Kaseorg 72655611ce requirements: Use maintained fork django-sendfile2 of django-sendfile
The original seems to be unmaintained
(johnsensible/django-sendfile#65).  Notably, this fixes a bug in the
filename parameter, which perviously showed the Python 3 repr of a
byte string (johnsensible/django-sendfile#49).

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-12 15:40:08 -07:00
Anders Kaseorg 263d71bf2b nginx: Add CORS headers to /user_uploads.
Fixes: #12980.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-12 15:35:35 -07:00
Anders Kaseorg 73330f3136 rest_dispatch: Remove invalid Content-Length: 0 from 204 responses.
There’s an apparent contradiction between RFC 7230 §3.3.2
Content-Length:

“A server MUST NOT send a Content-Length header field in any response
with a status code of 1xx (Informational) or 204 (No Content).”

and RFC 7231 §4.3.7 OPTIONS:

“A server MUST generate a Content-Length field with a value of "0" if
no payload body is to be sent in the response.”

The only resolution within the existing language would be to disallow
all 204 responses to OPTIONS requests.  However, I don’t think that
was the intention, so I submitted this erratum report:

https://www.rfc-editor.org/errata/eid5806

and updated the code accordingly.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-12 15:34:42 -07:00
Anders Kaseorg 2e57f3ffae puppet: “Resolve” puppet-lint warnings.
Introduced by #12966.

puppet/zulip/manifests/base.pp - WARNING: double quoted string containing no variables on line 93
puppet/zulip/manifests/base.pp - WARNING: string containing only a variable on line 93

scanf doesn’t accept a number as input, so uh, add a dummy space
character.

What.  You can’t give me a bad language and then complain when I write
bad programs in it.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-12 15:29:23 -07:00
Tim Abbott 7953e23d49 event_queue: Actually fix missing copy for edit-message events.
Apparently, our edit-message events did not guarantee that the outer
wrapper dictionary, which is intended to be unique for each client,
was unique for every client (instead only ensuring it was unique for
each user).

This led to clients unexpectedly getting last_event_id validation
errors in this code path when a user had multiple connected clients,
because the linear ordering of event IDs within a given queue was
corrupted.

In fd2a63b049, we accidentally fixed
this issue with a different set of userdata events, without fixing the
edit-message event bug.  This commit fixes the remaining issue.
2019-08-12 15:17:10 -07:00
Anders Kaseorg 58adac5663 zthumbor: Clean up type ignores.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-09 17:42:33 -07:00
Anders Kaseorg d56a5bdd88 dev_settings: Clean up type ignores.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-09 17:42:33 -07:00
Anders Kaseorg 9650188dfe backends: Clean up type ignores.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-09 17:42:33 -07:00
Anders Kaseorg 5d063910ff zilencer: Clean up type ignores.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-09 17:42:33 -07:00
Anders Kaseorg 7bf09067d1 event_queue: Clean up type ignores.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-09 17:42:33 -07:00
Anders Kaseorg 7cd11d0559 test_users: Clean up type ignores.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-09 17:42:33 -07:00
Anders Kaseorg 75a01810e3 test_outgoing_webhook_interfaces: Clean up type ignores.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-09 17:42:33 -07:00
Anders Kaseorg 1ffbc0009d test_openapi: Clean up type ignores.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-09 17:42:33 -07:00
Anders Kaseorg 9d33200fb2 test_messages: Clean up type ignores.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-09 17:42:33 -07:00
Anders Kaseorg 4671e4ca5c test_i18n: Clean up type ignores.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-09 17:42:33 -07:00
Anders Kaseorg c453c4b602 test_events: Clean up type ignores.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-09 17:42:33 -07:00
Anders Kaseorg d05578817e soft_deactivate_users: Clean up type ignores.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-09 17:42:33 -07:00
Anders Kaseorg 6d875dd211 topic: Clean up type ignores.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-09 17:42:33 -07:00
Priyank Patel 72bf7e2181 test_narrow: Rename stream_id -> stream_recipient_id.
It looks like this isn't actually a stream id, doing
get_stream_by_id_in_realm(stream_id, stream_messages[0].get_realm())
throws error.
2019-08-09 17:31:53 -07:00
Anders Kaseorg b0859f4b1e linter_lib: Fix mypy errors.
tools/linter_lib/pyflakes.py:35: error: Argument 3 to "run_pyflakes" has incompatible type "List[Tuple[bytes, bytes]]"; expected "List[Tuple[str, str]]"
    tools/linter_lib/custom_check.py:110: error: Argument "rules" to "RuleList" has incompatible type "List[Dict[str, Any]]"; expected "List[Rule]"
    tools/linter_lib/custom_check.py:214: error: Argument "rules" to "RuleList" has incompatible type "List[Dict[str, Any]]"; expected "List[Rule]"
    tools/linter_lib/custom_check.py:214: error: Argument "shebang_rules" to "RuleList" has incompatible type "List[Dict[str, Any]]"; expected "List[Rule]"
    tools/linter_lib/custom_check.py:502: error: Argument "rules" to "RuleList" has incompatible type "List[Dict[str, Any]]"; expected "List[Rule]"
    tools/linter_lib/custom_check.py:502: error: Argument "shebang_rules" to "RuleList" has incompatible type "List[Dict[str, Any]]"; expected "List[Rule]"
    tools/linter_lib/custom_check.py:519: error: Argument "rules" to "RuleList" has incompatible type "List[Dict[str, Any]]"; expected "List[Rule]"
    tools/linter_lib/custom_check.py:706: error: Argument "rules" to "RuleList" has incompatible type "List[Dict[str, Any]]"; expected "List[Rule]"
    tools/linter_lib/custom_check.py:728: error: Argument "rules" to "RuleList" has incompatible type "List[Dict[str, Any]]"; expected "List[Rule]"
    tools/linter_lib/custom_check.py:738: error: Argument "rules" to "RuleList" has incompatible type "List[Dict[str, Any]]"; expected "List[Rule]"
    tools/linter_lib/custom_check.py:779: error: Argument "rules" to "RuleList" has incompatible type "List[Dict[str, Any]]"; expected "List[Rule]"
    tools/linter_lib/custom_check.py:779: error: Argument "length_exclude" to "RuleList" has incompatible type "Set[str]"; expected "List[str]"
    tools/linter_lib/custom_check.py:803: error: Argument "length_exclude" to "RuleList" has incompatible type "Set[str]"; expected "List[str]"
    tools/linter_lib/custom_check.py:805: error: Unsupported operand types for + ("List[Rule]" and "List[Dict[str, Any]]")
    tools/linter_lib/custom_check.py:819: error: Argument "rules" to "RuleList" has incompatible type "List[Dict[str, Any]]"; expected "List[Rule]"

These were missed the `zulint` package was missing PEP 561 type
annotation markers, and if it’d had them, mypy daemon mode would’ve
required us to set `follow_imports = skip` for it.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-09 17:22:45 -07:00
Anders Kaseorg a0f4c99dad test_runner: Clean up type ignores.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-09 16:39:16 -07:00
Anders Kaseorg aa384c03a6 push_notifications: Clean up type ignores.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-09 16:39:16 -07:00
Anders Kaseorg 95658a2a54 logging_util: Clean up type ignores.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-09 16:39:16 -07:00
Anders Kaseorg df1fbbfee3 emoji: Clean up type ignores.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-09 16:39:16 -07:00
Anders Kaseorg bb5507009f camo: Clean up type ignores.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-09 16:39:16 -07:00
Anders Kaseorg 9b392917c5 addressee: Clean up type ignores.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-09 16:39:16 -07:00
Anders Kaseorg e417d3a040 slack_message_conversion: Clean up type ignores.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-09 16:39:16 -07:00
Anders Kaseorg 17acf95437 update-authors-json: Clean up type ignores.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-09 16:39:16 -07:00
Anders Kaseorg 906af61d20 pythonrc: Clean up type ignores.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-09 16:39:16 -07:00
Anders Kaseorg b320b8c37f email-mirror-postfix: Clean up type ignores.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-09 16:39:16 -07:00
Anders Kaseorg 490ce993f4 analytics: Clean up type ignores.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-09 16:39:16 -07:00
Anders Kaseorg 820165e4da Merge pull request #12968 from andersk/ffdhe2048
nginx: Use fixed ffdhe2048 DH parameter (RFC 7919)
2019-08-09 16:29:10 -07:00
Anders Kaseorg 4e9fb05c4f puppet: Use built-in memorysize_mb fact.
Fixes this warning:

    Warning: The string '8167976' was automatically coerced to the numerical value 8167976 (file: /root/zulip/puppet/zulip/manifests/base.pp, line: 93, column: 19)

Fixes #9682.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-09 16:28:09 -07:00
Anders Kaseorg cd1306c8d9 test-install: Add bionic.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-09 16:27:03 -07:00
meiomorphism 1af72a2745 subscriptions: Avoid sending unneeded subscriber information.
The `users/me/subscriptions` endpoint accidentally started returning
subscriber information for each stream.  This is convenient, but
unnecessarily costly for those clients which either don't need it
(most API apps) or already acquire this information via /register
(including Zulip's apps).

This change removes that data set from the default response.  Clients
which had come to rely on it, or would like to rely on it in future,
may still access it via an additional documented API parameter.

Fixes #12917.
2019-08-09 16:03:21 -07:00
Anders Kaseorg 671844d667 provision: Add default-jre-headless for vnu-jar.
This was previously pulled in by yui-compressor.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-08 17:28:20 -07:00
Tim Abbott fb01f2208d api: Verify that all registered OpenAPI examples are called.
This should help make this test suite robust against potential
mistakes when updating it.
2019-08-08 11:45:05 -07:00