Commit Graph

39236 Commits

Author SHA1 Message Date
Alex Vandiver dc58dec231 restart-server: Start services in opposite order from stop.
`supervisorctl` starts and stops its arguments sequentially, in the
order they are passed[1].  Start them in the opposite order from the
order in which they were stopped -- this puts the dependencies first,
and the most core services (`zulip-django`) last.

While the only "dependency" here is currently thumbor, this sets us up
in case others are added later.

[1] https://github.com/Supervisor/supervisor/blob/master/supervisor/supervisorctl.py#L782
2020-09-14 16:27:15 -07:00
Alex Vandiver 8adf530400 puppet: Generate sharding in puppet, then refresh-sharding-and-restart.
This supports running puppet to pick up new sharding changes, which
will warn of the need to finalize them via
`refresh-sharding-and-restart`, or simply running that directly.
2020-09-14 16:27:15 -07:00
Alex Vandiver 0de356c2df puppet: Move generation of tornado nginx upstreams into tornado_sharding.
This puts the creation of the upstreams referenced by
`nginx_sharding.conf` adjacent to their use.
2020-09-14 16:27:15 -07:00
Alex Vandiver bf029d99f1 sharding: Also mark sharding.json 644 for consistency.
There is no reason to limit this to 640; mark it 644 for consistency
with the other file.
2020-09-14 16:27:15 -07:00
Alex Vandiver b5bcff04e5 sharding: Consistent mode for nginx sharding file.
This disagreed between `tornado_sharding.pp` in puppet and
`scripts/refresh-sharding-and-restart`.
2020-09-14 16:27:15 -07:00
Anders Kaseorg a659542d84 parallel: Remove run_parallel.
This was a broken abstraction that returned to its caller within
multiple forked processes on exceptions, and encouraged ignoring the
error code (as all of its callers did).

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-09-14 16:22:23 -07:00
Anders Kaseorg 0f16df2f13 transfer: Migrate from run_parallel to multiprocessing.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-09-14 16:22:23 -07:00
Anders Kaseorg 7f410ff0de import_realm: Migrate from run_parallel to multiprocessing.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-09-14 16:22:23 -07:00
Anders Kaseorg 9a2aad58d0 import_util: Migrate from run_parallel to multiprocessing.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-09-14 16:22:23 -07:00
Anders Kaseorg 73eb0aa891 test_runner: Monkey-patch to allow using multiprocessing in tests.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-09-14 16:22:23 -07:00
Tim Abbott f751acbec5 openapi: Add comments on top of various OpenAPI doc files.
This is part of our standard approach for trying to make it easy for
folks to find relevant documentation on a system they're trying to
understand.
2020-09-14 15:25:46 -07:00
Aman Agrawal 2bc3924672 move_topic_to_stream: Allow moving to/between/from private streams.
Fixes #16284.

Most of the work for this was done when we implemented correct
behavior for guest users, since they treat public streams like private
streams anyway.

The general method involves moving the messages to the new stream with
special care of UserMessage.

We delete UserMessages for subs who are losing access to the message.
For private streams with protected history, we also create UserMessage
elements for users who are not present in the old stream, since that's
important for those users to access the moved messages.
2020-09-14 15:00:55 -07:00
Alex Vandiver cba1722129 webhooks: Do log non-"unsupported" errors to main logfiles.
This undoes a small part of b8a2e6b5f8; namely, logs to
`zulip.zerver.webhooks`, which are all exceptions from webhooks except
UnsupportedWebhookEventType, should still be logged to the main error
loggers.  This maintains the property that exceptions generating 500's
are all present in `errors.log`.
2020-09-14 12:58:16 -07:00
Ryan Rehman 20775e5429 minor: Rename `previously_selected` of message selected event.
This is done since we have logic in place which compares `event.id`
and `event.previously_selected`, both of which point to message ids.
2020-09-14 10:56:37 -07:00
Priyansh Garg 4d38cab9b9 frontend_tests: Fix typo in node_tests/general.js 2020-09-13 21:01:59 -07:00
Anders Kaseorg fd59db292f requirements: Upgrade Python requirements.
Fixes #16335.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-09-13 20:59:09 -07:00
Anders Kaseorg ddf8ec33df upload: Strip leading slash from deleted S3 export paths.
Previously, S3UploadBackend.delete_export_tarball failed to strip the
leading ‘/’ from the export path.  This mistake is now caught by Moto
1.3.15.  I expect it caused deletion failures in the real S3, although
I haven’t verified this.

We store export_path in the audit log with a leading ‘/’, but the
actual S3 keys do not have a leading ‘/’.  Changing either system
would require a migration.  So the new convention is that the
variables named ‘export_path’ have a leading ‘/’, while variables
named ‘path_id’ or ‘key’ do not.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-09-13 20:59:09 -07:00
Anders Kaseorg 42d038f09b migrations: Fix 0301 to replace a Python loop with SQL.
The previous code is correctly flagged by semgrep 0.23 as a violation
of our sql-format rule.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-09-13 20:59:09 -07:00
Hemanth V. Alluri f57196b1d0 webhooks/gitlab: Add fixture for MR opened with multiple assignees.
Fixes #16173.

Signed-off-by: Hemanth V. Alluri <hdrive1999@gmail.com>
2020-09-13 20:54:14 -07:00
Hemanth V. Alluri 54aba8d402 webhooks/gitlab: Use information about all assignees.
Previously, the GitLab webhook code, namely the `get_objects_assignee`
method first tried to get a single assignee and if that failed then it
looks for multiple assignees and then it would return the first
assignee that it found (there's actually a code smell here - a loop
which would always return on the first iteration).

Instead, this commit will change that behavior to first check for
multiple assignees first then for a single assignee if we can't find
multiple assignees. Ultimately it will return a list of all of the
assignees (however many that might be [0, n]). This method has then
aptly been renamed to `get_assignees`.

Finally, we tweked the code using this method to always use it's
output as an "assignees" parameter to templates (there's also an
assignee parameter which we want to avoid here for consistency).

Signed-off-by: Hemanth V. Alluri <hdrive1999@gmail.com>
2020-09-13 20:54:14 -07:00
Hemanth V. Alluri bbe7a54171 webhooks/gitlab: Make permissions consistent across all fixtures.
For some reasons, some of the fixtures had the +x bit set, while
some didn't. What this commit does is make sure that no fixture
is marked as "executable" (for anyone).

Signed-off-by: Hemanth V. Alluri <hdrive1999@gmail.com>
2020-09-13 20:54:14 -07:00
Anders Kaseorg dfab09b17d markdown: Replace hyperlink requirement with urllib.parse.
The previous code only worked by accident and hyperlink 20.0.0 breaks
it.

>>> hyperlink.parse("example.com").replace(scheme="https")
DecodedURL(url=URL.from_text('https:example.com'))

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-09-13 15:37:28 -07:00
Anders Kaseorg f39d4cf1f0 sentry: Pass format arguments to logging.
semgrep 0.23 correctly flags this as a violation of our logging-format
rule.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-09-13 15:37:28 -07:00
Mateusz Mandera 70bcc7a8e9 migrations: Add case-insensitive unique index on realm and stream name.
This is closely analogical to b9b146c809.
2020-09-13 15:36:43 -07:00
Mateusz Mandera cf5d544c82 ldap: Fix inaccuracy in django_to_ldap_username comment. 2020-09-13 18:39:00 +02:00
Anders Kaseorg 463929f349 urls: Migrate re_path routes to path.
Django treats path("<name>") like re_path(r"(?P<name>[^/]+)") and
path("<path:name>") like re_path(r"(?P<name>.+)").

This is more readable and consistent than the mix of slightly
different regexes we had before, and fixes various bugs:

• The r'apps/(.*)$' regex was missing a start anchor ^, so it
  incorrectly matched all URLs that included apps/ as a substring
  anywhere.
• The r'accounts/login/(google)/$' regex was missing a start anchor ^,
  so it incorrectly matched all URLs that ended with
  accounts/login/google/.
• The type annotation of zerver.views.realm_export.delete_realm_export
  takes export_id as an int, but it was previously passed as a string.
• The type annotation of zerver.views.users.avatar takes medium as a
  bool, but it was previously passed as a string.
• The [0-9A-Za-z]+ pattern for uidb64 was missing the - and _
  characters that can validly be part of a base64url encoded
  string (although I think the id is actually a decimal integer here,
  in which case only 012345ADEIMNOQTUYcgjkwxyz are present in its
  base64url encoding).

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-09-12 14:29:26 -07:00
Aryan Shridhar 4e8067aadc
message_edit: Add hide event for clipboard tooltip after copy.
The clipboard tooltip for the "copy and close" button was incorrectly staying visible after the
copy event.  Fix this by explicitly hiding tooltips in the click handler.

Fixes #16328.
2020-09-12 12:04:24 -07:00
Anders Kaseorg c21818e093 check-openapi: Disallow siblings of $ref.
The specification says “any sibling elements of a $ref are ignored”,
so their presence, although not invalid, indicates a mistake.  yamole
incorrectly merges these siblings into the referenced object, but we
should not rely on this nonstandard behavior.

https://swagger.io/docs/specification/using-ref/#sibling

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-09-12 11:57:13 -07:00
Anders Kaseorg 7c17bdb9c5 openapi: Remove ‘example’ $ref siblings.
$ref siblings are ignored according to the OpenAPI specification, and
the referenced definitions already have examples.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-09-12 11:57:13 -07:00
Anders Kaseorg caa08d76b5 openapi: Inline parameter references to avoid ‘required’ $ref siblings.
$ref siblings are ignored according to the OpenAPI specification.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-09-12 11:57:13 -07:00
Thomas Güttler c567661015
docs: Added PyCharm to the list of editors.
PyCharm is great for writing Python.
2020-09-12 11:42:22 -07:00
Thomas Güttler 740a4a1eff
docs: Spell "front end" as "frontend" consistently.
I think it makes sense avoid diversity here.
2020-09-12 11:41:35 -07:00
Tim Abbott 45cd979d46 tools: Fix webpack wrapper logic for finding config file to watch. 2020-09-12 11:31:44 -07:00
Thomas Güttler c5a7103b4d docs: Use zulip.com rather than zulip.org (which just redirects).
I guess it make sense to use zulip.com in the first place.
2020-09-12 11:22:08 -07:00
Anders Kaseorg b41165fd01 dev_urls: Give the user_avatars URLPattern a name.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-09-12 11:21:40 -07:00
Anders Kaseorg 1c80188648 urls: Elide str converter, which is the default.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-09-12 11:21:40 -07:00
palash 2632317b06 test_message_send: Refactor mock.patch to assertLogs.
Replaced mock.patch with assertLogs for testing log outputs
in file zerver/tests/test_message_send.py
2020-09-12 11:04:51 -07:00
palash f2f8034b76 test_markdown: Refactor mock.patch to assertLogs.
Replaced mock.patch with assertLogs for testing log outputs
in file zerver/tests/test_markdown.py
2020-09-12 11:04:51 -07:00
palash 588fff236d test_service_bot_system: Refactor mock.patch with assertLogs.
Replaced mock.patch with assertLogs for testing log outputs
in file zerver/tests/test_service_bot_system.py
2020-09-12 10:59:36 -07:00
palash 4e2769c31c test_decorators: Refactor mock.patch to assertLogs.
Replaced mock.patch with assertLogs for testing log outputs
in file zerver/tests/test_decorators.py
2020-09-12 10:59:36 -07:00
palash c3fee5059f test_email_mirror: Refactor mock.path to assertLogs.
Replaced mock.patch with assertLogs for testing log outputs
in file zerver/tests/test_email_mirror.py
2020-09-12 10:59:36 -07:00
palash 34003fc7f6 email_mirror: Change root logger to zerver.lib.email_mirror. 2020-09-12 10:53:56 -07:00
palash 60f9f87c5d test_external: Refactor mock.patch to assertLogs.
Replaced mock.patch with assertLogs for testing log outputs
in file zerver/tests/test_external.py
2020-09-12 10:53:56 -07:00
palash 485d740fe8 apple auth: Change logger to zulip.auth.apple from root logger.
Replace default root logger with zulip.auth.apple for apple auth
in file zproject/backends.py and update the test cases
accordingly in file zerver/tests/test_auth_backends.py
2020-09-12 10:53:56 -07:00
palash 04f08f45e7 test_auth_backends: Refactor mock.patch to assertLogs.
Replaced mock.patch with assertLogs for testing log outputs
in file test_auth_backends.py.

This change requires adjusting
test_log_into_subdomain_when_email_is_none to use an explicit token
since that appears in the log output.
2020-09-12 10:53:37 -07:00
Alex Vandiver 1c17583ad5 puppet: Restrict postfix incoming addresses to postmaster and zulip.
This removes the possibility of local user enumeration via RCPT TO.
2020-09-11 18:49:22 -07:00
Aman Agrawal ed8796f23f report: Allow anonymous queries to report (un)narrow times. 2020-09-11 16:57:27 -07:00
Alex Vandiver 3f6e4ff303 webhooks: Move the extra logging information into a formatter.
This clears it out of the data sent to Sentry, where it is duplicative
with the indexed metadata -- and potentially exposes PHI if Sentry's
"make this issue public" feature is used.
2020-09-11 16:43:29 -07:00
Alex Vandiver b8a2e6b5f8 webhooks: Configure webhook loggers in zproject/computed_settings.py.
This limits the webhook errors to only go to their respective log
files, and not to the general server logs.
2020-09-11 16:43:29 -07:00
Alex Vandiver 006875e7d0 sentry: Add additional context (client, fallback realm) to Sentry tags. 2020-09-11 16:43:29 -07:00