Commit Graph

38752 Commits

Author SHA1 Message Date
Steve Howell 3a710ab996 webhook tests: Remove stream_name parameter.
Having an optional stream_name parameter makes
it confusing to read the code if you know your
webhook is sending private messages.

And then the other two callers are already
checking topics, so they might as well check
stream names, too.

We also have the two stream-oriented callers
make their own call to "subscribe".  And we
future-proof this by making sure the exception
for no-message-being-sent calls out that gotcha.

Somewhat in passing, we now assert that
self.STREAM_NAME is not None in the main
helper.  This is partly to satisfy mypy, but
it's also a good sanity check.

This also sets the stage for the next commit,
where I'll add an assert_stream_message helper.
2020-08-24 12:34:46 -07:00
Steve Howell 2fdf966d9f webhook tests: Rename helper to send_webhook_payload.
Not all webhook payloads are json, so send_json_payload was a
bit misleading.

In passing I also remove "bytes" from the Union type for
"payload" parameter.
2020-08-24 12:34:46 -07:00
Steve Howell 388053db6b webhook tests: Rename main helper to check_webhook.
Almost all webhook tests use this helper, except a few
webhooks that write to private streams.

Being concise is important here, and the name
`self.send_and_test_stream_message` always confused
me, since it sounds you're sending a stream message,
and it leaves out the webhook piece.

We should consider renaming `send_and_test_private_message`
to something like `check_webhook_private`, but I couldn't
decide on a great name, and it's very rarely used.  So
for now I just made sure the docstrings of the two
sibling functions reference each other.
2020-08-24 12:34:46 -07:00
Steve Howell 00001a396b webhook tests: Shorten topic constants.
The "EXPECTED_" prefix and "_EVENTS" suffix
usually provided more noise than signal.

We also use module constants to avoid the "self."
noise.  It also makes it a bit more clear which
constants actually have to be in the class (e.g.
"FIXTURE_DIR_NAME") to do their job.
2020-08-24 12:34:46 -07:00
Steve Howell 0f970c6971 minor: Avoid do_test_message.
This function is a bad idea, as it leads to a possible situation
where you aren't actually testing anything:

    def do_test_message(self, msg: Message, expected_message: Optional[str]) -> None:
        if expected_message is not None:
            self.assertEqual(msg.content, expected_message)

Unfortunately, it's called deep in the stack in some places, but
we can safely replace it with assertEqual here.
2020-08-24 12:34:46 -07:00
Steve Howell fbef6e3c03 zendesk tests: Fix anti-patterns.
We had optional parameters for expected_topic and
expected_message, which are trivial to eliminate,
since the integration is really simple.

And we were doing strange things trying to reset
class variables at the end of tests.  Now we just
set them explicitly in the tests.
2020-08-24 12:34:46 -07:00
Steve Howell 3de2e4f034 dialogflow test: Fix nonsensical tests.
The test helper here was taking an "expected_topic"
parameter that it just ignored, and then the
dialogflow tests were passing in expected messages
in that slot, so the actual "expected_message" var
was "None" and was ignored.  So the tests weren't
testing anything.

Now we eliminate the crufty expected_topic parameter
and require an actual value for "expected_message".

I also clean up the mypy type for content_type,
and I remove the `content_type is None` check,
since all callers either pass in a str content
type or default to "application/json".
2020-08-24 12:34:46 -07:00
Sutou Kouhei ebf4048dd4
create-db.sql: Ensure using en_US.UTF-8 encoding.
PostgreSQL packages for Ubuntu run "initdb" without specifying locale
on installation. It means that the default template
database (template1) is created by the system default locale. If the
system default locale is non UTF-8 compatible encoding such as
en_US.ISO-8859-15, "zulip" database is also created non UTF-8
compatible encoding such as LATIN9.

You can reproduce this case by running the following script:

    apt update

    apt install -y locales
    locale-gen en_US.ISO-8859-15
    update-locale LANG=en_US.ISO-8859-15 LANGUAGE=en_US:

    apt install -y wget
    wget https://www.zulip.org/dist/releases/zulip-server-latest.tar.gz
    tar xf zulip-server-latest.tar.gz
    zulip-server-*/scripts/setup/install \
      --hostname=zulip-test.example.com \
      --email=zulip-test-admin@example.com \
      --self-signed-cert

scripts/setup/install is failed with the following error:

    + ./manage.py migrate --noinput
    Operations to perform:
      Apply all migrations: analytics, auth, confirmation, contenttypes, otp_static, otp_totp, sessions, social_django, two_factor, zerver
    Running migrations:
      Applying contenttypes.0001_initial... OK
      Applying auth.0001_initial... OK
      Applying zerver.0001_initial...Traceback (most recent call last):
      File "/srv/zulip-venv-cache/b4a27188142d80b2eeb64f5d5c05b1d94cc6b7b9/zulip-py3-venv/lib/python3.8/site-packages/django/db/backends/utils.py", line 82, in _execute
        return self.cursor.execute(sql)
      File "/home/zulip/deployments/2020-08-19-05-57-10/zerver/lib/db.py", line 33, in execute
        return wrapper_execute(self, super().execute, query, vars)
      File "/home/zulip/deployments/2020-08-19-05-57-10/zerver/lib/db.py", line 20, in wrapper_execute
        return action(sql, params)
    psycopg2.errors.UntranslatableCharacter: character with byte sequence 0xe2 0x80 0x99 in encoding "UTF8" has no equivalent in encoding "LATIN9"
    CONTEXT:  line 4 of configuration file "/usr/share/postgresql/12/tsearch_data/en_us.affix"

    The above exception was the direct cause of the following exception:

    Traceback (most recent call last):
      File "./manage.py", line 50, in <module>
        execute_from_command_line(sys.argv)
      File "/srv/zulip-venv-cache/b4a27188142d80b2eeb64f5d5c05b1d94cc6b7b9/zulip-py3-venv/lib/python3.8/site-packages/django/core/management/__init__.py", line 381, in execute_from_command_line
        utility.execute()
      File "/srv/zulip-venv-cache/b4a27188142d80b2eeb64f5d5c05b1d94cc6b7b9/zulip-py3-venv/lib/python3.8/site-packages/django/core/management/__init__.py", line 375, in execute
        self.fetch_command(subcommand).run_from_argv(self.argv)
      File "/srv/zulip-venv-cache/b4a27188142d80b2eeb64f5d5c05b1d94cc6b7b9/zulip-py3-venv/lib/python3.8/site-packages/django/core/management/base.py", line 323, in run_from_argv
        self.execute(*args, **cmd_options)
      File "/srv/zulip-venv-cache/b4a27188142d80b2eeb64f5d5c05b1d94cc6b7b9/zulip-py3-venv/lib/python3.8/site-packages/django/core/management/base.py", line 364, in execute
        output = self.handle(*args, **options)
      File "/srv/zulip-venv-cache/b4a27188142d80b2eeb64f5d5c05b1d94cc6b7b9/zulip-py3-venv/lib/python3.8/site-packages/django/core/management/base.py", line 83, in wrapped
        res = handle_func(*args, **kwargs)
      File "/srv/zulip-venv-cache/b4a27188142d80b2eeb64f5d5c05b1d94cc6b7b9/zulip-py3-venv/lib/python3.8/site-packages/django/core/management/commands/migrate.py", line 232, in handle
        post_migrate_state = executor.migrate(
      File "/srv/zulip-venv-cache/b4a27188142d80b2eeb64f5d5c05b1d94cc6b7b9/zulip-py3-venv/lib/python3.8/site-packages/django/db/migrations/executor.py", line 117, in migrate
        state = self._migrate_all_forwards(state, plan, full_plan, fake=fake, fake_initial=fake_initial)
      File "/srv/zulip-venv-cache/b4a27188142d80b2eeb64f5d5c05b1d94cc6b7b9/zulip-py3-venv/lib/python3.8/site-packages/django/db/migrations/executor.py", line 147, in _migrate_all_forwards
        state = self.apply_migration(state, migration, fake=fake, fake_initial=fake_initial)
      File "/srv/zulip-venv-cache/b4a27188142d80b2eeb64f5d5c05b1d94cc6b7b9/zulip-py3-venv/lib/python3.8/site-packages/django/db/migrations/executor.py", line 245, in apply_migration
        state = migration.apply(state, schema_editor)
      File "/srv/zulip-venv-cache/b4a27188142d80b2eeb64f5d5c05b1d94cc6b7b9/zulip-py3-venv/lib/python3.8/site-packages/django/db/migrations/migration.py", line 124, in apply
        operation.database_forwards(self.app_label, schema_editor, old_state, project_state)
      File "/srv/zulip-venv-cache/b4a27188142d80b2eeb64f5d5c05b1d94cc6b7b9/zulip-py3-venv/lib/python3.8/site-packages/django/db/migrations/operations/special.py", line 105, in database_forwards
        self._run_sql(schema_editor, self.sql)
      File "/srv/zulip-venv-cache/b4a27188142d80b2eeb64f5d5c05b1d94cc6b7b9/zulip-py3-venv/lib/python3.8/site-packages/django/db/migrations/operations/special.py", line 130, in _run_sql
        schema_editor.execute(statement, params=None)
      File "/srv/zulip-venv-cache/b4a27188142d80b2eeb64f5d5c05b1d94cc6b7b9/zulip-py3-venv/lib/python3.8/site-packages/django/db/backends/base/schema.py", line 137, in execute
        cursor.execute(sql, params)
      File "/srv/zulip-venv-cache/b4a27188142d80b2eeb64f5d5c05b1d94cc6b7b9/zulip-py3-venv/lib/python3.8/site-packages/django/db/backends/utils.py", line 67, in execute
        return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
      File "/srv/zulip-venv-cache/b4a27188142d80b2eeb64f5d5c05b1d94cc6b7b9/zulip-py3-venv/lib/python3.8/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
        return executor(sql, params, many, context)
      File "/srv/zulip-venv-cache/b4a27188142d80b2eeb64f5d5c05b1d94cc6b7b9/zulip-py3-venv/lib/python3.8/site-packages/django/db/backends/utils.py", line 84, in _execute
        return self.cursor.execute(sql, params)
      File "/srv/zulip-venv-cache/b4a27188142d80b2eeb64f5d5c05b1d94cc6b7b9/zulip-py3-venv/lib/python3.8/site-packages/django/db/utils.py", line 89, in __exit__
        raise dj_exc_value.with_traceback(traceback) from exc_value
      File "/srv/zulip-venv-cache/b4a27188142d80b2eeb64f5d5c05b1d94cc6b7b9/zulip-py3-venv/lib/python3.8/site-packages/django/db/backends/utils.py", line 82, in _execute
        return self.cursor.execute(sql)
      File "/home/zulip/deployments/2020-08-19-05-57-10/zerver/lib/db.py", line 33, in execute
        return wrapper_execute(self, super().execute, query, vars)
      File "/home/zulip/deployments/2020-08-19-05-57-10/zerver/lib/db.py", line 20, in wrapper_execute
        return action(sql, params)
    django.db.utils.DataError: character with byte sequence 0xe2 0x80 0x99 in encoding "UTF8" has no equivalent in encoding "LATIN9"
    CONTEXT:  line 4 of configuration file "/usr/share/postgresql/12/tsearch_data/en_us.affix"
2020-08-24 12:24:38 -07:00
Priyansh Garg ba119a2f20
help: Fix typo in change-organization-url. 2020-08-24 12:22:17 -07:00
Dinesh 77bea3de7a puppeteer: Fix test flake in admin tests.
This flake occurs because the Verona dropdown menu couldn't be
clicked, in rare cases, because puppeteer would click it too
quickly before it appears and then fails. To fix this we wait for it
to fully appear and then click it. Around 1000 runs
passed without a failure.

The error the flake caused was:
TimeoutError: waiting for selector
"#org-submit-notifications[data-status="unsaved"]" ...
2020-08-21 07:41:28 -04:00
Steve Howell 25c3f052cd node fixture: Fix data for realm_user__remove. 2020-08-20 08:40:59 -04:00
Alex Vandiver ad8943a64a url_preview: Only extract img tags with an `src`.
Some `<img>` tags do not have an SRC, if they are rewritten using JS
to have one later.  Attempting to access `first_image['src']` on these
will raise an exception, as they have no such attribute.

Only look for images which have a defined `src` attribute on them.  We
could instead check if `first_image.has_attr('src')`, but this seems
only likely to produce fewer valid images.
2020-08-18 14:26:21 -04:00
Alex Vandiver 5b74de7be7 markdown: Add another twitter code to retry-later.
Error code 131 is documented to be an arbitrary server error on
Twitter's side; add it to the retry list.
2020-08-18 10:32:24 -07:00
Alex Vandiver 092ed87ae3 markdown: Cache Twitter 403 responses that are semi-permanent.
03ca3afbc2 added more codes that are equivalent to 404's; this adds to
the list of cache-as-None codes a couple which are equivalent to
403's.  It does not comprise _all_ possible 403-like codes -- many of
them are "the client is not OK," which is relevant to log as an error
still.
2020-08-18 10:32:24 -07:00
Alex Vandiver 13e0c7454e docs: Update API-doc-writing docs to use @openapi_test_function.
c2f9227892 switched from a long hard-coded list in `TEST_FUNCTIONS` to
using a new `@openapi_test_function` decorator; update the
documentation about writing API docs accordingly.
2020-08-18 10:31:00 -07:00
Alex Vandiver aa5f98907c sentry: Deal more gracefully with missing stacktraces.
Not all languages produce exceptions with context or filenames.  See
https://develop.sentry.dev/sdk/event-payloads/stacktrace/
2020-08-18 11:17:42 -04:00
Alex Vandiver 0db311ddad sentry: Stacks are returned most-recent last.
Per [1], the sentry API returns frames sorted from oldest to newest.
As such, matching against the first filename that matches is most
likely not the right frame.

Match against the last frame with the guilty filename.

[1] https://develop.sentry.dev/sdk/event-payloads/stacktrace/
2020-08-18 11:17:42 -04:00
Sumana Harihareswara 47fe7b7d13 docs: Capitalize proper noun.
Signed-off-by: Sumana Harihareswara <sh@changeset.nyc>
2020-08-18 11:16:22 -04:00
Alex Vandiver 0078f16f06 tests: Add an explicit order_by to prevent flakes.
Without an order_by, this can fail spuriously.
2020-08-18 11:15:35 -04:00
Steve Howell b38353f3ff node fixtures: Add author_id key. 2020-08-18 10:43:42 -04:00
Steve Howell 2d5096ea8f minor: Make local var for test_user. 2020-08-18 10:43:42 -04:00
Steve Howell 4f5b214104 minor: Revise disclaimer on node fixtures data. 2020-08-18 10:43:42 -04:00
Steve Howell 33d7a22685 test_events: Fix loop structure in settings test.
The original commit was broken here:
b553507412

The intention was to run the same loop for all
settings, but instead, we did a funny loop of
just resetting schema_checker, and then we only
actually tested the last value of the loop.
2020-08-17 13:00:47 -04:00
Steve Howell 35dd6ef93b node fixtures: Use better data for reactions. 2020-08-17 13:00:47 -04:00
Steve Howell 31fd872781 node fixtures: Use better data for subscription update.
The dispatch test here really only cares that values
get passed on.

Note that the dispatch code ignores the email field, because
we only send subscription/update events to the user
whose subscription has changed.
2020-08-17 13:00:47 -04:00
Steve Howell 60b16a92e3 node tests: Add has_zoom_token to event.fixtures. 2020-08-16 11:37:06 -04:00
Steve Howell f33a314a4d openapi: Add enum for stream role. 2020-08-16 08:35:18 -04:00
Tim Abbott 2d70a73902 docs: Fix spelling of audit_fix_indexes.
Fixes #16140.
2020-08-14 16:56:35 -07:00
Mohit Gupta 4167517a6f filters: Fix has:image and avoid future issues for other has filters.
This fixes a bug with the original frontend-side implementation for
has: filters, where it would incorrectly not match content in cases
where the message's nesting structure did not have an outer tag.

Bug was introduced in 02ea52fc18.

Fixes #16118.
2020-08-14 16:51:26 -07:00
sahil839 f046c9c58a streams: Add role field to Subscription objects passed to clients.
This commit adds "role" field to the Subscription objects passed to
clients.  This is important preparation for being able to work on the
frontend for this feature.
2020-08-14 16:33:11 -07:00
Aman 88821b960a vagrant: Add parallels as a provider.
There are file sharing issues with the macOS 10.15.6 and
vagrant. var/remote_cache_prefix was an empty file when using
VirtualBox and Docker on macOS.
Using parallels as a provider for vagrant fixes the issue.

Use --watch-poll which makes webpack to recompile
automatically on file changes, since inotify is not
working here too.
2020-08-14 16:03:51 -07:00
shanukun 5c6d42c6a5 navbar_alerts: Change HTML ordering for obvious tab order.
Fixes #16079.
2020-08-14 15:59:44 -07:00
shanukun aa6f2b1310 navbar_alerts: Fix navbar alerts inaccessibility.
Part of #16079.
2020-08-14 15:57:48 -07:00
Gittenburg 2545703b58 compose: Add role=button to message control buttons.
Fixes #9143. Part of #15910.
2020-08-14 15:49:14 -07:00
Gittenburg 05e792ec1a compose: Move send button after textarea in DOM.
This restores the Tab + Enter shortcut to send.

We are floating the send button to the right so that it still looks like
before. Instead of moving the button we could have also given every
message control button a tabindex, but these would be cumbersome to
maintain.

Tweaked by tabbott to add a comment recording the reasoning behind
the somewhat unusual CSS here.

Part of #15910.
2020-08-14 15:48:49 -07:00
Gittenburg 3b0694693b compose: Make message control buttons accessible.
Previously the emoji picker, the formatting help, the button to attach
files, the video call button, the Drafts button and the Press Enter to
send checkbox were all inaccessible from the keyboard.

This does break the Tab + Enter workflow for sending messages, which is
fixed in the next commit by moving the Send button to be the first
element after the textarea.

Part of #15910.
2020-08-14 15:46:50 -07:00
Tim Abbott f94a8adf9e export: Remove duplicate 'analytics' zerver_realm object.
This fixes a harmless duplication of data in the Zulip data export
format.
2020-08-14 15:45:11 -07:00
arpit551 7568f6f9a8 export: Renamed zerver_analytics to zerver_realm.
While exporting analytics data we were using wrong table name
'zerver_analytics' in analytics config. Renamed it with
correct table name 'zerver_realm'.
2020-08-14 15:45:11 -07:00
arpit551 0d6047840b decorator: Updated user_passes_test function from Django 2.2.
Since bug https://bugs.python.org/issue3445 was resolved in Python
3.3, we can avoid the use of assigned=available_attrs(view_func) in
wraps decorator (which we were only using because we'd copied code
that handled that from Django).

Also available_attrs is now depreciated from Django 3.0 onwards.
2020-08-14 11:40:13 -07:00
arpit551 af3a34fbca cache: Used lru_cache from functools instead of django.utils.lru_cache.
Django 3.0 removed private Python 2 compatibility APIs
so used lru_cache() directly from functools.

We cast lru_cache to Any to avoid attr-defined error in mypy since we
are adding extra field, 'key_prefix', to this object later.
2020-08-14 11:34:04 -07:00
Alex Vandiver 2b6989a40f queue: Remove a no-longer-correct comment.
This comment stopped being true in 5686821150, and very much stopped
being relevant in dd40649e04 when the middleware entirely stopped
publishing to a queue.
2020-08-14 11:30:13 -07:00
Tim Abbott f2c9ee8000 migrations: Fix migration 0283 with orjson.
This function now matches the copy in zerver/lib/actions.py.

This is the same migration as
b250e42f61c525029bd2b3bbb8f4ea93ece62072; orjson enforces that we
don't use integers as keys in JSON dictionaries.
2020-08-14 11:22:03 -07:00
Tim Abbott b494b16791 meta tags: Use open graph titles/descriptions.
Apparently, we were incorrectly using constants for title/description
rather than the nice non-constant values from og:title and
og:description in our meta tags.
2020-08-14 10:58:18 -07:00
Sumanth V Rao cc93044cd1 stream_creation_form: Remove `optional` from stream description input.
The idea behind this change is to encourage users to enter a stream
description rather than take the easy way out and leave the 'optional'
field empty. This solution intends to solve the same issue as #15997
but in a more indirect way.

Discussion in this thread:
https://chat.zulip.org/#narrow/stream/2-general/topic/Stream.20description.20mandatory.20setting.20.2315997
2020-08-14 10:30:56 -07:00
Steve Howell 9b6da158ad openapi: Add enum values to role node in specs.
We will eventually want to improve descriptions here
too.
2020-08-14 10:40:29 -04:00
Steve Howell f4181ea300 node fixtures: Improve realm_user__add.
We also improve the relevant dispatch test a bit.
2020-08-14 10:40:29 -04:00
Steve Howell 8b34a2f755 node fixtures: Use better data for presence.
The dispatch for presence is a trivial one-liner,
so the test just makes sure three important parameters
get passed along.

We will eventually want to use the fixtures data in
other presence-related tests, but for now the only
goal is to make it pass the schema checks.
2020-08-14 10:40:29 -04:00
Steve Howell 196df919f2 minor: Improve error message for check-node-fixtures.
This is helpful now, but it will also reduce confusion
when we extend this tool to also look at openapi schemas.
2020-08-14 10:40:29 -04:00
Steve Howell 5931ebffd2 openapi: Use enum for message_type. 2020-08-14 10:40:29 -04:00
Tim Abbott 201df3432d help: Document Full Members more extensively.
Also reorder the Billing Administrator feature to not interrupt the
owner/admin/member/guest sequence, and add some important cross-links.
2020-08-13 16:42:12 -07:00