Commit Graph

4695 Commits

Author SHA1 Message Date
Mateusz Mandera 3ba0a37a92 types: Define UserDisplayRecipient type using TypedDict.
Since the display_recipients dictionaries corresponding to users are
always dictionaries with keys email, full_name, short_name, id,
is_mirror_dummy - instead of using the overly general Dict[str, Any]
type, we can define a UserDisplayRecipient type,
using an appropriate TypedDict.

The type definitions are moved from display_recipient.py to types.py, so
that they can be imported in models.py.

Appropriate type adjustments are made in various places in the code
where we operate on display_recipients.
2019-08-20 12:15:30 -07:00
Mateusz Mandera c779bb1959 messages: Don't use display_recipient values from cached message_dicts.
The user information in display_recipient in cached message_dicts
becomes outdated if the information is changed in any way.

In particular, since we don't have a way to find all the message
objects that might contain PMs after an organization toggles the
setting to hide user email addresses from other users, we had a
situation where client might see inaccurate cached data from before
the transition for a period of up to hours.

We address this by using our generic_bulk_cached_fetch toolchain to
ensure we always are fetching display_recipient data from the database
(and/or a special recipient_id -> display_recipient cache, which we
can flush easily).

Fixes #12818.
2019-08-20 12:15:30 -07:00
Pragati Agrawal 48efd46bc6 realm_logo: Remove redundant `realm_logo_url` function.
This function is nothing but a redundant layer of call to
`get_realm_logo_url`.
2019-08-20 12:07:20 -07:00
Pragati Agrawal 7326acdc2b realm_logo: Refactor `get_realm_logo_url` function. 2019-08-20 12:07:20 -07:00
Pragati Agrawal 1da5001875 realm_logo: Refactor/Deduplicate the code to sent the events.
This also creates a function to get source and url for logo.
2019-08-20 12:07:20 -07:00
Tim Abbott e02806ccc5 request: Fix other syntax error on Xenial Python. 2019-08-19 15:58:12 -07:00
Tim Abbott 0435556fd5 request: Fix syntax error on Xenial Python. 2019-08-19 15:47:40 -07:00
Tim Abbott 44db59bbe0 openapi: Fix handling of parameters passed via the URL/path.
Previously, our OpenAPI documentation validation was failing for some
endpoints because it didn't account for the `in: path` type of
parameter, resulting in a mismatch between what was declared via REQ
and what was declared in the OpenAPI docs.

We fix this by excluding the path type parameters in both places from
what's considered by documentation using the `path_only` flag.

I doubt this is the correct long-term fix; in particular, I don't
think we're actually running the validators for these path-only
parameters.  The examples that exist today are all IDs with validators
for being non-negative numbers, but longer-term I think we'll want to
do something different (possibly at the REQ layer, see the TODO).
2019-08-19 15:06:08 -07:00
Tim Abbott 26ac3ebd3e openapi: Suggest the -sS options to curl.
These options prevent curl from doing the downloading progress bar
(which is clutter).
2019-08-17 11:35:08 -07:00
Tim Abbott 2366490ffc openapi: Pass api_url to curl example generation.
Our new curl example generation logic was broken, in that it hardcoded
localhost:9991 (without an HTTP method or anything) as the API URL.

It requires a bit of plumbing to make this possible.
2019-08-17 11:35:08 -07:00
Vishnu KS f32382f7b7 emails: Show preheader block only if preheader is present. 2019-08-17 11:32:28 -07:00
Priyank Patel 3680393b47 messages: Support passing user ID for stream operator.
ok_to_include_history fuction was updated to expect stream ID.

Fixes part of #9474.
2019-08-17 11:19:12 -07:00
Priyank Patel 5eab5bbc3e streams: Refactor can_access_stream_history.
This refactor extracts the code logic of checking if user can access
stream history into it's own function: can_access_stream_history
that takes in user_profile and stream. Then we make seperate function
can_access_stream_by_name that takes in stream_name and retrives stream
and pass it to can_access_stream_history. This will make it easily to later
add a function that does the same thing with stream ID.
2019-08-17 11:10:00 -07:00
Priyank Patel 463ecb375f refactor: Add get_stream_by_narrow_operand_access_unchecked.
This let's us clean up the linter that excludes the use of get_stream
and by adding the access_unchecked in the name we make it clear that
it should be used with caution.

Refactoring idea by Tim Abbott.
2019-08-17 11:10:00 -07:00
Gloria Elston 092662d83e api: Use new endpoint for handling reactions. 2019-08-16 14:36:15 -07:00
Mateusz Mandera c6b3d0212d models: Move some display_recipient code to display_recipient.py. 2019-08-15 17:15:32 -07:00
Mateusz Mandera cb2c9b04b3 generic_bulk_cached_fetch: Only call query_function if necessary. 2019-08-15 17:14:02 -07:00
Tim Abbott 27a0e307b6 cache: Fix typing for generic_bulk_cached_fetch.
The typing for generic_bulk_cached_fetch is complicated, and was
recorded incorrectly previously for the case where a cache_transformer
function is required.  We fix this by adding the new CacheItemT, and
additionally add comments explaining what's going on with these types
for future reference.

Thanks to Mateusz Mandera for raising this issue.
2019-08-14 11:00:40 -07:00
Tim Abbott 2ada0a9bad management: Fix password reset emails being sent to deactivated users.
Apparently, the filters written for the send_password_reset_email (and
some other management commands) didn't correctly consider the case of
deactivated users.

While some commands, like syncing LDAP data (which can include whether
a user should be deactivated) want to process all users, other
commands generally only want to interact with active users.  We fix
this and add some tests.
2019-08-14 10:50:45 -07:00
Wyatt Hoodes 896886b412 realm_export: Add endpoint for export deletion. 2019-08-12 17:51:46 -07:00
Wyatt Hoodes 6f6efa516d exports: Refactor extra_data to export_data. 2019-08-12 17:51:46 -07:00
Wyatt Hoodes 7a2a1f29ad exports: Refactor event_time to export_time timestamp.
The time of the event was incorrectly being sent
as a datetime object.
2019-08-12 17:51:46 -07:00
Anders Kaseorg e0cf3d0e76 zerver: Accept HEAD requests wherever GET requests are accepted.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-12 16:47:41 -07:00
Anders Kaseorg 73330f3136 rest_dispatch: Remove invalid Content-Length: 0 from 204 responses.
There’s an apparent contradiction between RFC 7230 §3.3.2
Content-Length:

“A server MUST NOT send a Content-Length header field in any response
with a status code of 1xx (Informational) or 204 (No Content).”

and RFC 7231 §4.3.7 OPTIONS:

“A server MUST generate a Content-Length field with a value of "0" if
no payload body is to be sent in the response.”

The only resolution within the existing language would be to disallow
all 204 responses to OPTIONS requests.  However, I don’t think that
was the intention, so I submitted this erratum report:

https://www.rfc-editor.org/errata/eid5806

and updated the code accordingly.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-12 15:34:42 -07:00
Anders Kaseorg 6d875dd211 topic: Clean up type ignores.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-09 17:42:33 -07:00
Anders Kaseorg a0f4c99dad test_runner: Clean up type ignores.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-09 16:39:16 -07:00
Anders Kaseorg aa384c03a6 push_notifications: Clean up type ignores.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-09 16:39:16 -07:00
Anders Kaseorg 95658a2a54 logging_util: Clean up type ignores.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-09 16:39:16 -07:00
Anders Kaseorg df1fbbfee3 emoji: Clean up type ignores.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-09 16:39:16 -07:00
Anders Kaseorg bb5507009f camo: Clean up type ignores.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-09 16:39:16 -07:00
Anders Kaseorg 9b392917c5 addressee: Clean up type ignores.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-09 16:39:16 -07:00
meiomorphism 1af72a2745 subscriptions: Avoid sending unneeded subscriber information.
The `users/me/subscriptions` endpoint accidentally started returning
subscriber information for each stream.  This is convenient, but
unnecessarily costly for those clients which either don't need it
(most API apps) or already acquire this information via /register
(including Zulip's apps).

This change removes that data set from the default response.  Clients
which had come to rely on it, or would like to rely on it in future,
may still access it via an additional documented API parameter.

Fixes #12917.
2019-08-09 16:03:21 -07:00
Anders Kaseorg 7e92ff9d0a request: Delete request.pyi and make request.py type check.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-07 15:37:21 -07:00
Anders Kaseorg 13a2ac7b8e request: Remove ExtractRecipients type safety hole on REQ.
It was allowing us to get away with wrong types on a few functions:
`check_send_typing_notification` and `send_notification_backend` can be
(and are) called with a list of `int` as `notification_to`, not just a
list of `str`.

The problem it was working around already had a better solution using
the dummy `type` argument.  Use that.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-07 15:26:59 -07:00
Wyatt Hoodes 22842dab34 events: Rename notify_export_completed.
notify_realm_export is more reasonable for the context of doing
deletion events as well.
2019-08-07 14:18:27 -07:00
Wyatt Hoodes f93efcb89c test_helpers: Refactor dummy file creation for broader use. 2019-08-07 12:05:44 -07:00
Wyatt Hoodes 11db0c23fb exports: Update extra_data field to a JSON structure.
We add the `deleted_timestamp` key to the new `extra_data`
dictionary.
2019-08-07 12:04:28 -07:00
Anders Kaseorg becef760bf cleanup: Delete leading newlines.
Previous cleanups (mostly the removals of Python __future__ imports)
were done in a way that introduced leading newlines.  Delete leading
newlines from all files, except static/assets/zulip-emoji/NOTICE,
which is a verbatim copy of the Apache 2.0 license.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-06 23:29:11 -07:00
neiljp (Neil Pilgrim) 12f3ed12c4 mypy: Refactor extract_recipients in actions.py to remove type ignore. 2019-08-06 23:24:56 -07:00
neiljp (Neil Pilgrim) 5ab64daecc mypy: Remove type ignore by defining ProfileDataElement using TypedDict. 2019-08-06 23:24:56 -07:00
neiljp (Neil Pilgrim) 62a1c3443d mypy: Remove type ignores for python 3 stubs now present in typeshed.
Namely:
* posix stubs
* readfp stub
2019-08-06 23:24:56 -07:00
neiljp (Neil Pilgrim) 5f673f5820 mypy: Remove type ignores after boto stub improvements. 2019-08-06 23:24:56 -07:00
Hemanth V. Alluri 5af753d940 api_code_examples: Parse kwargs/mods for example code generation.
Now we can also include extra keyword arguments to specify
modifications in how the example code should be generated
in the generate_code_example template tag.

E.g. generate_code_example(curl, exclude=["param1", "param2"])
2019-08-05 21:27:23 -07:00
Hemanth V. Alluri b20cf095e7 curl: Add code to auto generate cURL examples from OpenAPI docs.
This commit extends api_code_examples.py to support automatically
generating cURL examples from the OpenAPI documentation. This way
work won't have to be repeated and we can also drastically reduce
the chance of introducing faulty cURL examples (via. an automated
test which can now be easily created).
2019-08-05 21:13:14 -07:00
Hemanth V. Alluri f280e9cf84 lib: Rename lib/api_test_helpers.py to openapi/python_examples.py
This will make the contained code easier to find.
2019-08-05 21:06:19 -07:00
Hemanth V. Alluri 748192d58c openapi: Replace add-subscriptions from TEST_FUNCTIONS.
Migrate the add-subscriptions line to the OpenAPI equivalent.
2019-08-05 21:06:19 -07:00
Hemanth V. Alluri 14c0a387cf openapi: Remove /users/me from pending endpoints.
This commit progress our efforts to reduce pending_endpoints
as well as to migrate away from templates/zerver/api/fixtures
and towards our OpenAPI documentation.

Similar to commit d62b75fc.
2019-08-05 21:06:19 -07:00
Rishi Gupta 2c80224e2d custom profile: Rename check_url_pattern.
validator.py has validators for the whole project, so we need a more
specific name.
2019-08-05 20:59:47 -07:00
Rishi Gupta b37c6a8644 custom profile: Update a few error messages. 2019-08-05 20:59:47 -07:00
Rishi Gupta 8167b5b645 email: Remove special integration doc behavior when gateway not set.
The current code looks like it's trying to redirect /integrations/doc/email
to /integrations when EMAIL_GATEWAY_PATTERN is not set.

I think it doesn't currently do this. The test for that pathway has a bug:
self.get_doc('integrations/doc-html/email', subdomain='zulip') needs a
leading slash, and putting the slash back in results in the test failing.

This redirection is not really desired behavior -- better is to
unconditionally show that the email integration exists, and just point the
user to https://zulip.readthedocs.io/en/latest/production/email-gateway.html
(this is done in a child commit).
2019-08-05 17:33:05 -07:00
Anders Kaseorg 68dd8e4ec8 mypy: Migrate from mypy_extensions to typing_extensions.
This gives us access to typing_extensions.Deque, which was not added
to typing until 3.5.4.

(PROVISION_VERSION is not bumped because the transitive dependency set
in dev.txt hasn’t changed.)

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-05 17:24:09 -07:00
Tim Abbott ba67a9b76a message: Use RawUnreadMessageResult typing more.
This is a cleaner implementation given we have the TypedDict defined.
2019-08-02 17:34:27 -07:00
Tim Abbott 8f7a86f5ec events: Move remove_message_id_from_unread_mgs to message.py.
It belongs there, next to the related code for adding an event.
2019-08-02 17:26:13 -07:00
Tim Abbott 2c7511023d events: Simplify arguments for remove_message_id_from_unread_mgs.
This should make it reusable by the event_queue.py code path.
2019-08-02 17:26:13 -07:00
Rohitt Vashishtha a7f2bedb15 markdown: Enable hashheadings syntax.
Our implementation requires at least 1 space after the
'#' not not break existing linkifiers like '#123', etc.
that generally follow the convention we show in linkifier
examples.

- [valid]  : # Hello
- [valid]  : #  Hello
- [invalid]: #Hello

For the frontend, we have taken the code from v0.7.0 of
upstream marked and made minor changes to avoid having
to refactor a significant part of our marked code.

For the backend, we merely have to change the regex to
force require spaces after #, and add hashheader to our
list of blockparsers.

Fixes #11418.
2019-08-02 15:15:34 -07:00
Tim Abbott 9599d311a5 logging_util: Add an extra type: ignore. 2019-07-31 12:40:01 -07:00
Wyatt Hoodes 167d0bad61 test_runner: Unionize suite type.
Mypy cannot see the class hierarchy pertaining to django's
ParallelTestSuite. To deal with this, we simply create a
`Union` of our two types.
2019-07-31 12:20:12 -07:00
Wyatt Hoodes 4beec5c6b9 typing: Use TYPE_CHECKING when dealing with cyclic dependencies. 2019-07-31 12:19:39 -07:00
Mateusz Mandera 8f6f78b912 requirements: Upgrade redis from 2.10.6 to 3.2.1.
Fixes #11209.

This requires changing how zadd is used in rate_limiter.py:
In redis-py >= 3.0 the pairs to ZADD need to be passed as a dictionary,
not as *args or **kwargs, as described at
https://pypi.org/project/redis/3.2.1/ in the section
"Upgrading from redis-py 2.X to 3.0".

The rate_limiter change has to be in one commit with the redis upgrade,
because the dict format is not supported before redis-py 3.0.
2019-07-29 15:34:05 -07:00
Wyatt Hoodes ecb1b1a1e1 openapi: Fix typing for self.data. 2019-07-29 15:24:51 -07:00
Wyatt Hoodes 4149bac696 actions: Fix typing of query_filter. 2019-07-29 15:23:10 -07:00
Wyatt Hoodes 1706e06884 bugdown/init: Fix typing for fence variable. 2019-07-29 15:23:10 -07:00
Wyatt Hoodes 22481f63bf upload: Fix typing for key variable. 2019-07-29 15:23:10 -07:00
Wyatt Hoodes 8107230b35 domains: Fix typing for validate_domain. 2019-07-29 15:18:24 -07:00
Wyatt Hoodes a109508e34 typing: Remove now-unnecessary conditional import.
As a result of dropping support for trusty, we can remove our old
pattern of putting `if False` before importing the typing module,
which was essential for Python 3.4 support, but not required and maybe
harmful on newer versions.

cron_file_helper
check_rabbitmq_consumers
hash_reqs
check_zephyr_mirror
check_personal_zephyr_mirrors
check_cron_file
zulip_tools
check_postgres_replication_lag
api_test_helpers
purge-old-deployments
setup_venv
node_cache
clean_venv_cache
clean_node_cache
clean_emoji_cache
pg_backup_and_purge
restore-backup
generate_secrets
zulip-ec2-configure-interfaces
diagnose
check_user_zephyr_mirror_liveness
2019-07-29 15:18:22 -07:00
Wyatt Hoodes bbbea9ec87 events: Rewrite system for managing realm exports.
This feature is intended to cover all of our ways of exporting a
realm, not just the initial "public export" feature, so we should name
things appropriately for that goal.

Additionally, we don't want to include data exports in page_params;
the original implementation was actually buggy and would have.
2019-07-26 16:38:52 -07:00
Wyatt Hoodes b1900c406a public_export: Add logic for deleting the export tarball.
The path to the uploaded tarball is reconstructed via the relative url and
removed with the canonical methods in `upload.py`.
2019-07-26 15:52:03 -07:00
Wyatt Hoodes ef02de4834 public_export: Add endpoint for returning all REALM_EXPORTED objects. 2019-07-26 15:52:02 -07:00
Hemanth V. Alluri ede73ee4cd streams: New stream messages should be auto marked as read by creator.
When a person creates a new realm, they'll likely want to create a
bunch of initial streams at once. When doing so, it could be annoying
to have to mark all of the new stream notification messages as read.

Thus to make this process smoother, we should automatically mark
the messages generated by the Notification Bot in the notifications
(announcements) stream, as well as in the newly created stream itself
as read by the stream creator.

Fixes #12765.
2019-07-26 15:10:09 -07:00
Wyatt Hoodes 1c8106fc67 timezone: Fix return type in get_timezone.
While it's true `datetime` is implicit via `pytz`, it makes sense
that mypy should now complain about the semantics of calling our
return type `pytz.datetime.tzinfo`, when such a type doesn't
actually exist.
2019-07-25 17:41:01 -07:00
Tim Abbott b8a1050fc4 rest: Disable caching for all REST API endpoints.
Investigation into #12876, a mysterious bug where users were seeing
messages reappear as unread, determined that the root cause was
missing headers to disable client-side caching for Zulip's REST API
endpoints.

This manifested, in particular, for `GET /messages`, which is
essentially the only API GET endpoint used by the webapp at all.  When
using the `Ctrl+Shift+T` feature of browsers to restore a recently
closed tab (and potentially other code paths), the browser would
return from its disk cache a cached copy of the GET /messages results.

Because we include message flags on messages fetched from the server,
this in particular meant that those tabs would get a stale version of
the unread flag for the batches of the most recent ~1200 messages that
Zulip fetches upon opening a new browser tab.

The issue took same care to reproduce as well, in large part because
the arguments to those initial GET /messages requests will vary as one
reads messages (because the `pointer` moves forward) and then enters
the "All messages" view; the disk cache is only used for GET requests
with the exact same URL parameters.

We will probably still want to merge the events error-handling changes
we had previously proposed for this, but the conclusion of this being
a straightforward case of missing cache-control headers is much more
satisfying than the "badly behaving Chrome" theory discussed in the
issue thread.

Fixes #12876.
2019-07-25 12:42:46 -07:00
Anders Kaseorg 2b33822de9 storage: Stop using django-pipeline.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-07-24 17:40:31 -07:00
Anders Kaseorg fd7803e7f4 settings: Unset STATIC_ROOT in development.
Django’s default FileSystemFinder disallows STATICFILES_DIRS from
containing STATIC_ROOT (by raising an ImproperlyConfigured exception),
because STATIC_ROOT is supposed to be the result of collecting all the
static files in the project, not one of the potentially many sources
of static files.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-07-24 17:40:31 -07:00
Anders Kaseorg a97a2612bb integrations: Get logo URLs from staticfiles.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-07-24 17:40:31 -07:00
Tim Abbott daca742e9f email_gateway: Use html2text directly.
In the rare case that Zulip receives an email with only an HTML
format, we originally (code dating to 2013) shelled out to
html2markdown/python-html2text in order to convert the HTML into
markdown.

We long since added html2text as a reasonably managed Python
dependency of Zulip; we should just use it here.
2019-07-24 12:42:17 -07:00
Mateusz Mandera 11862e5ce0 test_classes: Use subdomain kwarg in api_* functions instead of realm.
It's more appropriate for the kwarg to be named subdomain. We also
update the functions, so that this kwarg is used in all of them
consistently.
2019-07-23 15:05:39 -07:00
Mateusz Mandera 7f24e27610 test_classes: Rename first argument of api_* functions to "identifier".
This was the name in api_post already - we rename it in the other
functions, for consistency.
2019-07-23 15:05:39 -07:00
Eeshan Garg 8e10ab282a webhooks: Log unexpected webhook events separately.
This change serves to declutter webhook-errors.log, which is
filled with too many UnexpectedWebhookEventType exceptions.

Keeping UnexpectedWebhookEventType in zerver/lib/webhooks/common.py
led to a cyclic import when we tried to import the exception in
zerver/decorators.py, so this commit also moves this exception to
another appropriate module. Note that our webhooks still import
this exception via zerver/lib/webhooks/common.py.
2019-07-22 18:20:53 -07:00
Conner Bondurant 4d4b6b8319 email: Convert send_email to use formataddr.
Changed the requirements for UserProfile in order to allow use of
the formataddr function in send_mail.py.

Converted send_email to use formataddr in conjunction with the commit
that strengthened requirements for full_name, such that they can now be
used in the to field of emails.

Fixes #4676.
2019-07-22 18:15:58 -07:00
Conner Bondurant c25dcf048d models: Enforce stricter requirements on the full_name field.
This changes the requirements for UserProfile to disallow some
additional characters, with the overall goal of being able to use
formataddr in send_mail.py.

We don't need to be particularly careful in the database migration,
because user full_names are not required to be unique.
2019-07-22 18:13:34 -07:00
neiljp (Neil Pilgrim) ba7a0934e3 requirements: Upgrade mypy to 0.711.
This comes with it a big performance improvement; mypy is now only
barely our slowest linter even if it wasn't previously running.

Fixes: #12058
2019-07-22 17:12:50 -07:00
neiljp (Neil Pilgrim) 80a4504784 mypy: Clean up typing for CustomProfileField code. 2019-07-22 17:10:50 -07:00
Wyatt Hoodes e4133e687f test_runner: Ignore inheritance issues in TestSuite.
Again, `result` of type `TestResult` is considered by mypy to
not have these _attributes acquired through inheritance. So,
we ignore them.
2019-07-22 17:06:41 -07:00
Wyatt Hoodes 5686821150 middleware: Change write_log_line to publish as a dict.
We were seeing errors when pubishing typical events in the form of
`Dict[str, Any]` as the expected type to be a `Union`.  So we instead
change the only non-dictionary call, to pass a dict instead of `str`.
2019-07-22 17:06:41 -07:00
Wyatt Hoodes dd19e2df1b test_runner: Ignore self.stream in TextTestResult class.
Per the import line:

`from unittest import loader, runner  # type: ignore  # Mypy cannot pick
these up.`

Because `TextTestResult` inherits from `runner.TextTestResult`, mypy
doesn't see `self` as having an attribute `stream`, so we ignore these
instead of cluttering with `casts` or `isinstances`.
2019-07-22 16:45:21 -07:00
Tim Abbott d3f59db8ce test_runner: Cleanup typing for TestResult objects.
There's two possible types here.
2019-07-22 16:45:21 -07:00
Wyatt Hoodes 76e8dc1657 test_runner: Change type to unittest.TestSuite.
In `deserialize_suite`, it passes a test as being of the
super type, where just having TestSuite assumes the type of
`zerver.Runner.TestSuite`.
2019-07-22 16:27:39 -07:00
Wyatt Hoodes 3509fe07b5 test_runner: Fix incorrect type for enforce_timely_test_completion.
Our TestResult objects are always actually TextTestResults.
2019-07-22 16:27:39 -07:00
Tim Abbott 02d55928ea import: Fix importing slack avatars into S3_UPLOAD_BACKEND.
Apparently, a subtle mismatch between the filename/URL formats for our
upload codebases meant that importing Slack avatars into systems using
S3_UPLOAD_BACKEND would end up with the avatars having the wrong URLs.
2019-07-21 21:25:31 -07:00
Hemanth V. Alluri d62b75fc4c openapi: Remove /users/me/subscriptions from pending_endpoints.
In commit 7c71e98, we added a special exception for the
/users/me/subscriptions endpoint in the automatic validation test.
By adding some extra documentation, we now remove this extra code,
as well as the endpoint from the list of pending endpoints.
2019-07-21 18:39:53 -07:00
Rishi Gupta 1a6f8cdf17 emails: Remove unsubscribe link from missed message emails.
This allows the footer to be two lines long instead of three.
2019-07-21 15:10:48 -07:00
Tim Abbott 21c8a7ec36 i18n: Fix strings in send_pm_if_empty_stream.
The previous iteration did not properly handle languages with a
different word order than English.

Discovered via warning output in `manage.py makemessages`.
2019-07-21 14:43:50 -07:00
Mateusz Mandera f1b135bd16 email_mirror: Rename include-quotations to include-quotes. 2019-07-20 15:53:43 -07:00
Mateusz Mandera 519ed41803 email_mirror: Display stream email address with . as the separator. 2019-07-20 15:53:43 -07:00
Mateusz Mandera 8bbd1d005d email_mirror: Support both + and . as separators in the address. 2019-07-20 15:53:42 -07:00
Wyatt Hoodes e331a758c3 python: Migrate open statements to use with.
This is low priority, but it's nice to be consistently using the best
practice pattern.

Fixes: #12419.
2019-07-20 15:48:52 -07:00
Mateusz Mandera 842d9e944a test_classes: Prepare for moving system bots to zulipinternal. 2019-07-20 14:55:21 -07:00
Rishi Gupta 38eeb3e183 notification bot: Add constant for STREAM_EVENTS_NOTIFICATION_TOPIC. 2019-07-20 14:39:15 -07:00
Rishi Gupta c9f194be99 notification bot: Update message for stream name changes.
Also properly internationalizes string formatting arguments for the stream
created message.
2019-07-20 14:39:15 -07:00
Rohitt Vashishtha 726d5003e1 bugdown: Force absolute urls in topic links.
If a url doesn't have a scheme, browsers would treat it as a relative
url and open something like: https://chat.zulip.org/google.com instead.

This PR fixes the issue on the backend; the frontend implementation
remains out of sync and the user sending the message wouldn't see
any linkification for urls without a scheme.

Fixes #12791.
2019-07-19 12:02:52 -07:00
Anders Kaseorg afa251de5d dev_urls: Serve static files with staticfiles_urlpatterns.
The test_docs change is because Django runs test cases with DEBUG =
False, which ordinarily means it doesn’t serve /static during tests.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-07-19 11:47:51 -07:00
Anders Kaseorg a7eb3faf96 storage: Move the staticfiles.json hack to ZulipStorage.
There’s no reason to monkey-patch something that we were already
subclassing.

Removing the PRODUCTION conditional causes us to generate
staticfiles.json in the right place to begin with so we don’t need to
move it later.  It also allows Django to find staticfiles.json if
running the dev server with PIPELINE_ENABLED.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-07-19 11:47:51 -07:00
Rishi Gupta bd52ec9f95 notification bot: Remove checks for whether bot exists. 2019-07-17 17:54:16 -07:00
Rishi Gupta f4397166c4 notify_new_user: Remove call to statsd.gauge.
This has been deprecated by our analytics system.
2019-07-17 17:54:16 -07:00
Rishi Gupta b520df18ee notify_new_user: Move clear_scheduled_invitation_emails out of function.
notify_new_user is confusingly named, but it should be read as "notify
others that a new user was created", not "notify the new user".
2019-07-17 17:54:16 -07:00
Rishi Gupta c030411fec email visibility: Use delivery_email to clear scheduled invitation emails.
This fixes a user-visible bug, where users signing up for realms with
restricted email visibility get reminder emails 1 week later, whether or not
they created an account.
2019-07-17 17:54:16 -07:00
Vishnu Ks 3a6fc09a4d emails: Make sender name go in-line with message body.
Making sender name go in-line with message body only if
the html starts with <p> tag since it won't look good
if the message starts with a code snippet, ul, etc.
If message starts with p tag we can safely assume that
it can go in-line with sender name.
2019-07-17 17:50:04 -07:00
Priyank Patel c067c155aa messages: Support passing user ID to sender operator. 2019-07-17 16:09:12 -07:00
Wyatt Hoodes c218735b2c test_runner: Add error handling for test directory clean up.
It was discovered that errors such as:
    `OSError: [Errno 16] Device or resource busy`
potentially arise when running in serial mode, or with
explicit test cases passed to `test-backend`.
2019-07-15 14:15:19 -07:00
Tim Abbott 7c71e987ac openapi: Fix endpoints incorrectly tagged as documentation_pending. 2019-07-15 13:39:29 -07:00
Tim Abbott 588a24b661 cache: Fix caching of system bots.
In the unlikely event that someone edited the properties of a system
bot and then saved the result, we were still caching the old version
indefinitely in the get_system_bot cache.

This led to a confusing case where a newly installed Zulip server
didn't have is_api_super_user properly set on its EMAIL_GATEWAY_BOT in
memcached.

Co-authored-by: Mateusz Mandera <mateusz.mandera@protonmail.com>
2019-07-15 12:32:29 -07:00
Rishi Gupta 15cd2903dd notification bot: Refactor arguments to send_pm_if_empty_stream.
Previously we sent "" for stream_name where we should have sent None, which
made this function harder to understand. The "" value was never used.

This also reorders the arguments to be match the order of the arguments in
the two callers.
2019-07-13 15:53:36 -07:00
Rishi Gupta 628d9ad67d notification bot: Update error messages in send_pm_if_empty_stream. 2019-07-13 15:53:36 -07:00
Rishi Gupta b92f936dad notification bot: Refactor send_pm_if_empty_stream to simplify logic. 2019-07-13 15:53:36 -07:00
David Wood 9bace3f2cd notifications: Allow only notifiable in unread count.
This commit adds a new setting to the user's notification settings that
will change the behaviour of the unread count in the title bar and
desktop application.

When enabled, the title bar will show the count of unread private messages
and mentions. When disabled, the title bar will act as before, showing
the total number of unread messages.

Fixes #1736.
2019-07-13 15:49:04 -07:00
Hariom Verma 107da5402c url preview: Replace YouTube URLs with their titles.
Modified by punchagan to:
* Replace URLs with titles only if the inline url embed previews are turned on
* Add a test for youtube titles replacing URLs

The titles for the videos are fetched asynchronously after the message has been
sent via the code that fetches metadata for open graph previews. So, the URLs
are replaced with titles only if the inline embed url previews feature is
enabled.

Ideally, YouTube previews should be shown only if inline url previews are
enabled, but this feature is in beta, while YouTube previews are pretty stable.
Once this feature is out of beta, YouTube previews should be shown only if the
url previews feature is turned on.

YouTube preview image is calculated as soon as the message is sent, while the
title needs to be fetched using a network request. This means that the URL is
replaced only after the data has been fetched from the request, and happens a
couple of seconds after the message has been rendered.

Closes #7549
2019-07-12 19:14:19 -07:00
Puneeth Chaganti 865bc24f67 url preview: Avoid showing previews for URLs in blockquotes.
Messages with links embedded in blockquotes turn out to be replies to
messages with links, more often than not. Showing previews for links in
replies seems like clutter, and it seems reasonable to turn off previews for
such links.
2019-07-12 19:14:00 -07:00
Puneeth Chaganti ad98f536bf bugdown: Fix incorrect type annotation for ElementPair. 2019-07-12 19:14:00 -07:00
Wyatt Hoodes 0e3fddbc6e test_fixtures: Add logic for removing stale test directories.
Similarly to how stale database removal is handled, we add a check for
stale test run directories at the end of the `test-backend` script.
2019-07-12 17:42:18 -07:00
Wyatt Hoodes 62566f02d7 tests: Use TEST_WORKER_DIR for file access. 2019-07-12 17:42:18 -07:00
Wyatt Hoodes 5f20caa6e0 test_upload: Refactor test_upload output to new filepath.
We write a function to set the `LOCAL_UPLOADS_DIR` path depending
on whether tests are being run in parallel or serial mode.
2019-07-12 17:35:44 -07:00
Wyatt Hoodes ff156c1f8a test_runner: Create file structure for test-backend filepath rework.
We create a path structure in the from:
 `/var/<uuid>/test-backend/run_1234567/worker_N/`

A settings attribute, `TEST_WORKER_DIR`, was created as a worker's
directory for a given `test-backend` run's file storage.  The
appropirate path is created in `setup_test_environment`, while each
workers subdirectory is created within `init_worker`.

This allows a test class to write to `settings.TEST_WORKER_DIR`,
populating the appropirate directory of a given worker.  Also
providing the long-term approach to clean up filesystem access
in the backend unit tests.
2019-07-12 17:27:18 -07:00
Rohitt Vashishtha 0ba332bcc0 topic-mention: Add Bugdown implementation as StreamTopicPattern. 2019-07-11 14:53:10 -07:00
Rishi Gupta 5400eb62be notification bot: Inline prep_stream_welcome_message.
This will make it easier to have access to the stream creator.

The indirection also isn't really adding anything, especially given that the
announce message is inlined just above.
2019-07-11 14:16:56 -07:00
Rishi Gupta 7de7b6872b notification bot: Modify initial stream creation message.
Discussion at
https://chat.zulip.org/#narrow/stream/137-feedback/topic/hello.20topic

The "by @**X**" part in the message content is in a later commit.
2019-07-11 14:16:56 -07:00
Aayush Agrawal 54584f6c16 url preview: Create a single preview for each URL in a message.
Modified by punchagan to:
* Add a separate markdown test for de-duplicating inline previews
* Check for number of unique URLs to see if per limit message is crossed
* Use a set for processed URLs instead of a list

Fixes #8379.
2019-07-11 13:37:15 -07:00
Hemanth V. Alluri ffd2bccd4e openapi: Standardize URL format to use {var_name} syntax.
The previous code for the validator test was fairly messy due to
checking for both formats of the openapi url, one with
<variable_name> and the other with {variable_name}. To eliminate
this, we have standardized the format and restricted it to
{variable_name} as per the official format at:
https://swagger.io/docs/specification/describing-parameters.
2019-07-11 11:01:40 -07:00
Wyatt Hoodes 0f31053b0c export.py: Call upload_backend only when we know which backend to use.
Importing at the top of the file causes conflictions on the
backend when deciding whether to go down the S3 path, or the
`LOCAL_UPLOADS_DIR` path.
2019-07-10 17:48:54 -07:00
Yashashvi Dave d7ee2aced1 models: Add `external_account` in custom profile field types.
Add new custom profile field type, External account.
External account field links user's social media
profile with account. e.g. GitHub, Twitter, etc.

Fixes part of #12302
2019-07-09 17:21:54 -07:00
Vishnu Ks c4d28777ec emails: Include stream topic details in missed message subject. 2019-07-09 11:51:20 -07:00
Mateusz Mandera 58754830fd email_mirror: Rename "include-footers" option to "include-footer". 2019-07-08 20:10:21 -07:00
Mateusz Mandera d1c2185c81 retention: Archive cross-realm personal messages.
We can simply archive cross-realm personal messages according to the
retention policy of the recipient's realm. It requires adding another
message-archiving query for this case however.

What remains is to figure out how to treat cross-realm huddle messages.
2019-07-08 20:03:20 -07:00
Mateusz Mandera 24ec1c7aa1 Revert "retention: Delete objects tied to a Message in one query with archiving."
This reverts commit 8f15884c7d. Using the
WITH ( ) ... DELETE method leads to a small performance drop, while
probably not offering many positives, so it seems appropriate to go to
the simpler case of just letting things get cleaned up by CASCADE.
2019-07-08 16:35:53 -07:00
Mateusz Mandera b55f24e07c retention: Avoid "SELECT realm" being run by Django ORM for each stream.
The  way the code changed in this commit was written caused Django to
fetch stream.realm from the database for every stream, leading to
redundant, identical queries. Each stream's realm is already known, so
we use that information.
2019-07-08 16:32:38 -07:00
Hemanth V. Alluri cecdec3097 openapi: Add test to see if code exists for documented endpoints.
In addition to the test which checks to to see if each endpoint in
code (urls.py) is documented in the openapi documentation (and with
the right arugments). We now also have a test to see if every
endpoint in the openapi documentation is a legitimate endpoint
also existing in code.

We do this by piggy-backing on the work done be the former test and
using set operations. This method avoid the need for an extra loop
and it uses set operations for additional speed and ease of reading.
2019-07-08 12:58:21 -07:00
Hemanth V. Alluri 2738b09044 openapi: Remove a few "buggy" endpoints in the validation test.
By importing a few view modules in the validation test itself we
can remove a few endpoints which were marked as buggy. What was
happening was that the view functions weren't imported and hence
the arguments map was not filled. Thus the test complained that
there was documentation for request parameters that seemed to be
missing in the code. Also, for the events register endpoint, we
have renamed one of the documented request parameters from
"stream" to "topic" (the API itself was not modified though).

We add a new "documentation_pending" attribute to req variables
so that any arguments not currently documented but should be
documented can be properly accounted for.
2019-07-08 12:34:31 -07:00
Wyatt Hoodes af4eb8c0d5 export/upload: Refactor tarball upload logic to upload_backend.
The conditional block containing the tarball upload logic for both S3
and local uploads was deconstructed and moved to the more appropriate
location within `zerver/lib/upload.py`.
2019-07-03 15:40:35 -07:00
Wyatt Hoodes 8efb7b903b export.py: Have do_export_realm handle the export tarball.
This change is preliminary refactoring in order to improve the test
mocking strategy related to `test_realm_export.py`.

What this allows is the ability to simply mock a return value from
`do_export_realm`.  We can then use that value as a dummy url to
ensure a file has been served and can be retrieved.
2019-07-03 15:40:35 -07:00
Mohit Gupta 00cdba2090 actions: Add allow_deactivated option when fetching recipients.
Preparatory commit to allow viewing group PM with deactivated
users.
2019-07-03 15:30:15 -07:00
Puneeth Chaganti b10fc1d896 url preview: Don't show a message embed if there's no image. 2019-07-03 14:38:19 -07:00
Anders Kaseorg 3ded6f53c3 storage: Transfer responsibility for hash-naming webpack files to webpack.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-07-03 14:16:17 -07:00
Anders Kaseorg b0a30aa01a storage: Remove RemoveUnminifiedFilesMixin.
This can be done much more easily with the right options to
collectstatic.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-07-03 14:16:07 -07:00
Anders Kaseorg f47f98e2f1 storage: Remove AddHeaderMixin.
We should be doing this inside webpack, which would be a lot less
hacky.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-07-03 14:15:46 -07:00
Anders Kaseorg 079ddae4c8 minify-js: Remove; everything has been migrated to Webpack.
min/sockjs-0.3.4.min.js is not used.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-07-03 13:58:21 -07:00
vinitS101 04f3fce761 ldap: Fix LDAP avatar synchronization to check if avatar has changed.
When "manage.py sync_ldap_user_data" is run, user avatars are now only
updated if they have changed in LDAP.

Fixes #12381.
2019-07-02 17:52:48 -07:00
Mateusz Mandera 89ba6d7941 retention: Improve the placement of logging in run_archiving_in_chunks. 2019-07-02 17:33:53 -07:00
Mateusz Mandera 17c5398703 retention: Filter also by transaction type when restoring by realm. 2019-07-02 17:31:07 -07:00
Mateusz Mandera 627238ebe1 retention: Replace all LEFT JOIN IS NULL with ON CONFLICT handling.
Duplicate handling when INSERTing is switched from "LEFT JOIN ... id IS
NULL" approach to "ON CONFLICT (id) DO NOTHING", since we now have
postgresql 9.5. The ON CONFLICT approach is more natural as well as also
potentially being faster,
2019-07-02 17:25:31 -07:00
Mateusz Mandera 9f452a6e9a retention: Add logging to the restoring and archive cleaning functions. 2019-07-02 17:25:31 -07:00
Anders Kaseorg 5f3d36ae87 tex: Bundle KaTeX CLI using Webpack.
We don’t need a hacked copy anymore.  We run the installed version out
of node_modules in development, and a Webpack-bundled version of that
in production.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-07-02 16:50:03 -07:00
Hemanth V. Alluri e5cb3cabef openapi: Add intentionally_undocumentated parameter to REQ.
This will allow us to mark a REQ variable as intentionally
undocumented. With this, we can remove some of the endpoints marked
as "buggy" even though they're not actually buggy, we just needed to
specify certain parameters as intentionally undocumented (e.g. the
stream_id for the /users/me/subscriptions/muted_topics endpoint.)

Any REQ variable with intentionally_undocumentated set to True
will not be added to the arguments_map data structure.

For some of the other "buggy" endpoints, we would want to mark the
entire endpoint as being undocumented intentionally via. the urls.py
file.
2019-07-02 16:30:33 -07:00
Anders Kaseorg bbb56df6b0 i18n: Move static/locale back to locale.
As of commit cff40c557b (#9300), these
files are no longer served directly to the browser.  Disentangle them
from the static asset pipeline so we can refactor it without worrying
about them.

This has the side effect of eliminating the accidental duplication of
translation data via hash-naming in our release tarballs.

This reverts commit b546391f0b (#1148).

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-07-02 14:57:55 -07:00
Tim Abbott e0522b6395 api: Fix broken API documentation tests.
We had gotten to the right documented content, but the previous
chaining of object creation/deletion wasn't quite right.

Fix this in a way that reduces the amount by which tests are dependent
on what other tests are doing.
2019-06-28 18:16:34 -07:00
sameerchoubey 4dcc1b744f api_docs: Add documentation for remove_user_group. 2019-06-28 16:34:39 -07:00
sameerchoubey d351564798 api_docs: Add documentation for update_user_group. 2019-06-28 16:33:54 -07:00
sameerchoubey 1b2f67f1bf api_docs: Add documentation for create_user_group. 2019-06-28 16:28:56 -07:00
sameerchoubey f84e84d6c7 api_docs: Add documentation for update_stream. 2019-06-28 16:26:06 -07:00
sameerchoubey 0ceea2713c api_docs: Add documentation for delete_stream. 2019-06-28 16:18:22 -07:00
Tim Abbott bf9e0c3239 messages: Improve variable names and comments around mirroring.
This should help make this part of the codebase easier to understand.
2019-06-28 12:42:19 -07:00
Vishnu Ks a5c69fb707 emails: Simplify content styling of missed_message emails.
This is a dramatic redesign of the look and feel of our missed-message
emails, designed to decrease the feeling of clutter and just provide
the content users care about in a clear, visible fashion.
2019-06-28 11:58:52 -07:00
Vishnu Ks 3a6cc4214f emails: Improve the reply wording in missed_message emails.
This cleans up the reply_warning feature in favor of a more coherent
explanation of whether or not one can reply.

(Also, critically, it now advertises the ability to enable
missed-message email replies with some administrative configuration
work.)
2019-06-28 11:53:52 -07:00
Vishnu Ks 66ee5b870a emails: Extract get_narrow_url into a function.
Reorganized a bit by tabbott to avoid doing a bunch of database
queries twice.
2019-06-28 11:38:17 -07:00
Rohitt Vashishtha 047086b81c markdown: Make raw urls in topic names navigable.
We reuse the link regexes we use elsewhere inn markdown
for parsing links in topic names and add a button to open
them in new tabs similar to our behavior with linkifiers
in topic names.

Fixes #12391.
2019-06-27 15:18:42 -07:00
Rohitt Vashishtha 96d7c1f3b0 markdown: Test escaping of topic_links and document. 2019-06-27 15:18:30 -07:00
Mateusz Mandera 7950aaea1e retention: Add code for deleting old archive data. 2019-06-26 12:24:47 -07:00
Mateusz Mandera 3ac11a3fc5 retention: Use ON CONFLICT DO UPDATE to handle re-archiving properly.
When archiving Messages, we stop relying on LEFT JOIN ... IS NULL to
avoid duplicates when INSERTing. Instead we use ON CONFLICT DO UPDATE
(added in postgresql 9.5) to, in case of archiving a Message that
already has a corresponding archived objects (this happens if a Message
gets archived, restored and then archived again), re-assign the existing
ArchivedMessage to the new transaction.

This also allows us to fix test_archiving_messages_second_time, which
was temporarily disable a few commits before.
2019-06-26 12:05:59 -07:00
Mateusz Mandera 7b2b4435ed retention: Combine run_message_batch_query and run_archiving_in_chunks.
We combine run_message_batch_query and run_archiving_in_chunks
functions, which makes the code simpler and more readable - we get rid
of hacky generator usage, for example.
In the process, move_expired_messages_* functions are adjusted, and now
they archive Messages as well as their related objects.
Appropriate adjustments in reaction to this are made in the main
archiving functions which call move_expired_messages_* (they no longer
need to call move_related_objects_to_archive).
2019-06-26 12:05:59 -07:00
Mateusz Mandera 6e46c6d752 retention: Add functions for restoring archived data.
Functions for restoring archived data are added and existing tests are
expanded to restore data they archived and check correctness.
2019-06-26 12:05:59 -07:00
Mateusz Mandera 9acd3b0f46 retention: Rewrite move_messages_to_archive to use existing functions.
Instead of having a bunch of custom code in the function, we make it use
run_message_batch_query and run_archiving_in_chunks to do the necessary
operations in a consistent way, using the same codepaths as the rest of
the archiving system.
This breaks test_archiving_messages_second_time temporarily, but we will
fix it and re-enable the test in the next commits, where we'll address
various other issues with re-archiving of messages.

We also remove the @transaction.atomic wrapper, because atomicity is
handled by the logic inside run_archiving_in_chunks.
2019-06-26 12:05:59 -07:00
Mateusz Mandera 80b834dd1b retention: Update move_rows() function code.
We make minor changes to the move_rows() function to allow its use in
the code for restoring from the archive.
2019-06-26 12:05:59 -07:00
Mateusz Mandera e3fe66a084 retention: Set savepoint=False on atomic wrapper on move_rows().
Savepoints create unnecessary overhead, and there's no benefit from
them, with the way we use this function.
2019-06-26 12:05:59 -07:00
Mateusz Mandera 5d8d5910a8 retention: Log archive_transaction id and information. 2019-06-26 12:05:59 -07:00
Mateusz Mandera a2cce62c1c retention: Use new ArchiveTransaction model.
We add a new model, ArchiveTransaction, to tie archived objects together
in a coherent way, according to the batches in which they are archived.
This enables making a better system for restoring from archive, and it
seems just more sensible to tie the archived objects in this way, rather
the somewhat vague setting of archive_timestamp to each object using
timezone_now().
2019-06-26 12:05:59 -07:00
Anders Kaseorg 6112d020f4 migrate, create_large_indexes: Use CREATE INDEX IF NOT EXISTS.
We no longer support Postgres versions missing this syntax.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-06-26 11:01:30 -07:00
Hemanth V. Alluri ef52aa0fc1 webhooks: Eliminate the usage of a headers.py file.
For storing HTTP headers as a function of fixture name, previously
we required that the fixture_to_headers method should reside in a
separate module called headers.py.

However, as in many cases, this method will only take a few lines,
we decided to move this function into the view.py file of the
integration instead of requiring a whole new file called headers.py

This commit introduces the small change in the system architecture,
migrates the GitHub integration, and updates the docs accordingly.
2019-06-26 10:35:14 -07:00
Hemanth V. Alluri 4691028097 webhooks: Extract the '__' method of mapping to headers.
In the GitHub integration we established that for many integrations,
we can directly map the fixture filename to the set of required
headers and by following a simple naming convention we can greatly
ease the logic involved in fixture_to_headers method required .

So to prevent the need for duplicating the logic used by the GitHub
integration, we created a method called `get_http_headers_from_filename`
which will take the name of the HTTP header (key) and then return a
corresponding method (in a decorator-like fashion) which could then be
equated to fixture_to_headers in headers.py.

The GitHub integration was modified to use this method and the docs
were updated to suggest using this when possible.
2019-06-26 10:32:34 -07:00
Yago González 4dc206ad2e api docs: Document PATCH /settings/notifications.
With significant tweaks by tabbott after rebasing.
2019-06-24 18:42:44 -07:00
Hemanth V. Alluri e2549b3b84 webhooks: Replace headers parsing logic with a common source.
When parsing custom HTTP headers in the integrations dev panel, http
headers from fixtures system and the send_webhook_fixture_message
we now use a singular source of logic: standardize_headers which
will take care of converting a dictionary of input headers into a
standard form that Django expects.
2019-06-24 13:23:21 -07:00
Wyatt Hoodes fb58b54293 test-backend: Move `test_messages.json` to `var/<uuid>/test-backend`. 2019-06-24 12:23:09 -07:00
Hemanth V. Alluri 7c145c3f68 webhooks: Add a system for storing http headers for integrations.
Using this system, we can now associate any fixture of any integration
with a particular set of HTTP headers. A helper method called
determine_http_headers was introduced, and the test suite was upgraded
to use determine_http_headers.

Comments and documentation significantly edited by tabbott.
2019-06-20 16:36:38 -07:00
Tim Abbott 1817f657ee models: Rename get_admin_users to get_admin_users_and_bots.
This provides much more clarity that administrative bot users are
included in this query as well.
2019-06-20 14:36:15 -07:00
Tim Abbott 4eb9d67b70 models: Extract get_human_admin_users function.
This function is an alternative to get_admin_users that we use in all
places where we explicitly want only human administrative users (not
administrative bots).  The following commits will rename
get_admin_users for better clarity.
2019-06-20 14:32:30 -07:00
Tim Abbott fd25ced43c import: Fix check for whether data-user-id is present.
Apparently, the `in` keyword in Beautiful Soup does something different.
2019-06-18 11:13:32 -07:00
Tim Abbott 649e363ee3 import: Fix handling of legacy and wildcard mentions.
Our recently-added code for rewriting user IDs on data import didn't
correctly handle wildcard mentions and mentions generated by very old
versions of Zulip (pre data-user-id).
2019-06-18 10:35:01 -07:00
Tim Abbott 2538f84447 import: Fix bad database query for first_message_id.
The previous query ended up doing an awkward join that did not
guarantee use of the Recipient index on zerver_message, turning a very
fast query into something that could take much longer for a single
stream than the rest of the import combined.
2019-06-18 10:25:00 -07:00
Priyank Patel d7b4de2348 messages: Add support for passing user IDs for pm-with clause.
We also document support for user IDs in the pm-with narrow operator.

Edited by tabbott to document on /api rather than in the /help page.

Fixes part of #9474.
2019-06-17 17:38:51 -07:00
Priyank Patel e3431976ce validator: Add validator for checking string or integer list.
This validator will be used to check if string or user IDs list is
passed in.
2019-06-17 17:33:58 -07:00
Tim Abbott 544f9c74ce export: Use outbox emoji for managing who is exported.
This is a little more unambiguous.
2019-06-17 16:10:28 -07:00
Wyatt Hoodes dd339a57af test-backend: Move import/export ouput into `var/<uuid>/test-backend`. 2019-06-17 13:51:40 -07:00
Hemanth V. Alluri 5a1043237c realm_plan_type: Add wide organization logo related fields to state.
Namely, here we add the "plan_includes_wide_organization_logo" and
"upgrade_text_for_wide_organization_logo" to the page_params (which
is set in zerver/lib/events.py).

"plan_includes_wide_organization_logo" is True if the plan is not of
the Realm.LIMITED type. We need to add this extra boolean parameter
instead of just using "realm_plan_type" to make things a lot easier
to work with on the frontend side, especially considering that
handlebars won't allow checking for equality in its {{#if}} blocks.
2019-06-14 15:59:28 -07:00
Hemanth V. Alluri 5d8050f822 realm_plan_type: Send an event when the realm's plan type is updated.
When a realm's plan type is updated using "do_change_plan_type" we
notify active users of the realm. This way certain plan features
could be enabled instantaneously for active users.
2019-06-14 15:59:28 -07:00
Wyatt Hoodes 6b033c7909 test-backend: Add steps to deal with potential database leaks.
A function was written in `test_fixtures.py` to drop a test database
template if the corresponding database id doesn't belong to a file.
Alongside this fact, every file that is written is removed after 60
minutes.  Meaning any potential database template can never exist
longer than one hour.

This follow-up work was added to deal with the potential race
conditions when running `test-backend`.  Ensuring that all templates
are properly dealt with.

Essentially rewritten by tabbott for cleanliness.

Fixes the remainder of #12426.
2019-06-14 15:23:25 -07:00
Wyatt Hoodes 0b05d91e62 test_runner: Write database ids to file for reference in clean up.
The ids that will be used for each particular run of the test suite are
written to a unique file. Each file will then be used as a time
reference of when the suite was ran.

This change sets up the ability for a complete clean up of potentially
leaked database templates.

Tweaked by tabbott to remove these files after successful database
cleanup.
2019-06-14 15:23:20 -07:00
Wyatt Hoodes 900d9d3e49 test_runner: Add comment explaining abnormal database destruction.
When running the test-backend suite in serial mode, `destroy_test_db`
double appends the database id number to the template if passed an
argument for `number`.  The comment here explains this behavior.
2019-06-14 14:35:39 -07:00
vinitS101 a6eda858d0 ldap: Fix avatar sync not working with the S3 backend.
This fixes an issue that caused LDAP synchronization to fail for
avatars.  The problem occurred due to the lack of a 'name' attribute
on the BytesIO object that we pass to the upload backend (which is
only used in the S3 backend for computing Content-Type).

Fixes #12411.
2019-06-13 15:12:13 -07:00
Mateusz Mandera 8f15884c7d retention: Delete objects tied to a Message in one query with archiving.
Rather than relying on the CASCADING property of the ForeignKey to the
Message table to clean up these objects, we delete them in the same
query as we archive them - since it's guaranteed that any of these
objects that we archive will be deleted due to their Message being
deleted later.
We don't have this guarantee for Attachment objects, which is why we
can't apply this scheme to them.
2019-06-13 11:18:11 -07:00
Mateusz Mandera 25810752fe retention: Fully process each Message chunk in a transaction.
To ensure the database retains a consistent state if archiving gets
interrupted, we process each Messages chunk together with related
objects in a single atomic transaction.
2019-06-13 11:17:54 -07:00
Mateusz Mandera 55eb46433b retention: Use yield when batching instead of returning a list of lists.
This generator architecture will be cleaner for supporting the
transactionality model we want.
2019-06-13 11:11:34 -07:00
Mateusz Mandera 37a22844b9 retention: Clean up code of move_messages_to_archive(). 2019-06-13 11:02:11 -07:00
Mateusz Mandera a68c460a14 retention: Clean up code for archiving attachment_messages.
We had two duplicate functions for archiving zerver_attachment_messages
rows, doing the same thing - archiving by message_id. One of them had a
redundant INNER JOIN, so we get rid of that too.
2019-06-13 11:02:11 -07:00