Commit Graph

39677 Commits

Author SHA1 Message Date
Steve Howell 1afca3d430 minor: Extract local for stream. 2020-10-12 16:40:37 -07:00
Steve Howell 84aa1389d8 Extract bulk_add_subs_to_db_with_logging.
This is a trivial code extraction.
2020-10-12 16:40:37 -07:00
Steve Howell 3ff9ce78ea refactor: Extract send_peer_add_events. 2020-10-12 16:40:37 -07:00
Areeb Beigh (Euphinx) 5d6f3eef29
docs: Fix typos in events-system.md.
Fixes 2 typos in events-system.md:
it's handler -> its handler
Event Queue Server server -> Event Queue Server
2020-10-12 12:10:50 -07:00
shanukun 537051e4f1 api docs: Fix JavaScript example for /streams api endpoint.
This commit fixes the JavaScript tab in 'Usage example'
for 'Get all streams' that currently shows python code.
2020-10-11 23:12:33 -07:00
Alex Vandiver f3ba227614 create_user: Strip whitespace from initial password file.
Fixes #12144.
2020-10-11 16:29:00 -07:00
Cody Piersall 5dab6e9d31 emoji-upload: Fix transparency issues on GIF emoji upload.
This preserves the alpha layer on GIF images that need to be resized
before being uploaded.  Two important changes occur here:

1. The new frame is a *copy* of the original image, which preserves the
   GIF info.
2. The disposal method of the original GIF is preserved.  This
   essentially determines what state each frame of the GIF starts from
   when it is drawn; see PIL's docs:
   https://pillow.readthedocs.io/en/stable/handbook/image-file-formats.html#saving
   for more info.

This resolves some but not all of the test cases in #16370.
2020-10-11 16:23:07 -07:00
Anders Kaseorg b7a94be152 python: Catch BaseException when we need to clean something up.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-10-11 16:16:16 -07:00
Anders Kaseorg 7f69c1d3d5 python: Catch specific exceptions from requests.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-10-11 16:11:41 -07:00
Anders Kaseorg 17ac17286c python: Catch specific exceptions from subprocess.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-10-11 16:11:41 -07:00
Anders Kaseorg aabef3d9be python: Catch specific exceptions from orjson.
Followup to #16120.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-10-11 16:11:41 -07:00
Anders Kaseorg a109d68af0 create-test-api-docs: Remove unhelpful exception handler.
encode returns bytes, which cannot possibly be written to a text file.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-10-11 16:11:35 -07:00
Anders Kaseorg 234f7245cf export_usermessage_batch: Use os.rename.
This avoids an extra stat call to check whether the target is a
directory.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-10-11 16:11:35 -07:00
Anders Kaseorg 83eca256a4 compilemessages: Use polib for get_name_from_po_file.
This also corrects the name of zh_TW from “Chinese” to
“Chinese (Taiwan)”.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-10-11 16:11:35 -07:00
Anders Kaseorg 1346c5397a zephyr: Use correct shell quoting for ssh.
ssh always runs its command through a shell (after naïvely joining
multiple arguments with spaces), so it needs an extra level of shell
quoting.  This should have no effect because we already validated user
with a regex, but it’s better for escaping to be locally correct in
case the context changes.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-10-11 16:11:35 -07:00
Anders Kaseorg 82593338ba report: Show Git commit in a way that works for merges.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-10-11 16:11:35 -07:00
Anders Kaseorg c9fec8f021 deliver_scheduled_messages: Don’t do_send_messages inside a transaction.
do_send_messages has side effects outside the database and may not
work reliably if its database effects are reordered by being inside a
transaction.

This also fixes a bug where we were doing the update incorrectly on
the Message table.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-10-11 16:09:22 -07:00
Alex Vandiver c2132a4f9c queue: Drop register_json_consumer / json_drain_queue interface.
Now that all callsites use the same interface, drop the now-unused
ones, and their tests.
2020-10-11 14:19:42 -07:00
Alex Vandiver 5477b9d9a1 queue: Switch tests to start_json_consumer interface. 2020-10-11 14:19:42 -07:00
Alex Vandiver 179c387409 tornado: Switch to start_json_consumer interface. 2020-10-11 14:19:42 -07:00
Alex Vandiver f0b23b0752 queue: Switch non-batch consumer to also use start_json_consumer.
This has no effect on consumption rate, but unifies the codepaths.
Before:
```
$ ./manage.py queue_rate --count 50000
Purging queue...
Enqueue rate: 11187 / sec
Dequeue rate: 4158 / sec
```

After:
```
$ ./manage.py queue_rate --count 50000
Purging queue...
Enqueue rate: 11010 / sec
Dequeue rate: 4113 / sec
```
2020-10-11 14:19:42 -07:00
Alex Vandiver 45c9c3cc30 queue: Monitor user_activity queue, now that it has a consumer.
Since this was using repead individual get() calls previously, it
could not be monitored for having a consumer.  Add it in, by marking
it of queue type "consumer" (the default), and adding Nagios lines for
it.

Also adjust missedmessage_emails to be monitored; it stopped using
LoopQueueProcessingWorker in 5cec566cb9, but was never added back
into the set of monitored consumers.
2020-10-11 14:19:42 -07:00
Alex Vandiver f9358d5330 queue: Switch batch interface to use the channel.consume iterator.
This low-level interface allows consuming from a queue with timeouts.
This can be used to either consume in batches (with an upper timeout),
or one-at-a-time.  This is notably more performant than calling
`.get()` repeatedly (what json_drain_queue does under the hood), which
is "*highly discouraged* as it is *very inefficient*"[1].

Before this change:
```
$ ./manage.py queue_rate --count 10000 --batch
Purging queue...
Enqueue rate: 11158 / sec
Dequeue rate: 3075 / sec
```

After:
```
$ ./manage.py queue_rate --count 10000 --batch
Purging queue...
Enqueue rate: 11511 / sec
Dequeue rate: 19938 / sec
```

[1] https://www.rabbitmq.com/consumers.html#fetching
2020-10-11 14:19:40 -07:00
Alex Vandiver 571f8b8664 queue: Use low-level queue_purge to empty at the end of tests.
This is O(1) at the RabbitMQ API level, and doesn't rely on the code
under test to function correctly during test cleanup.
2020-10-09 20:43:49 -07:00
Alex Vandiver ac0ba21c2c tests: Stop reusing a variable name.
`loopworker_sleep_mock` is a file-level variable used to mock out the
sleep() call in LoopQueueProcessingWorker; don't reuse the variable
name for something else.
2020-10-09 20:42:20 -07:00
Alex Vandiver 754638f673 tests: Refactor test_queue_worker to separate queues. 2020-10-09 20:42:12 -07:00
Alex Vandiver 2547bdbf4a queue: Rename consume_wrapper to a better name. 2020-10-09 20:40:51 -07:00
Alex Vandiver d5a6b0f99a queue: Rename queue_size, and update for all local queues.
Despite its name, the `queue_size` method does not return the number
of items in the queue; it returns the number of items that the local
consumer has delivered but unprocessed.  These are often, but not
always, the same.

RabbitMQ's queues maintain the queue of unacknowledged messages; when
a consumer connects, it sends to the consumer some number of messages
to handle, known as the "prefetch."  This is a performance
optimization, to ensure the consumer code does not need to wait for a
network round-trip before having new data to consume.

The default prefetch is 0, which means that RabbitMQ immediately dumps
all outstanding messages to the consumer, which slowly processes and
acknowledges them.  If a second consumer were to connect to the same
queue, they would receive no messages to process, as the first
consumer has already been allocated them.  If the first consumer
disconnects or crashes, all prior events sent to it are then made
available for other consumers on the queue.

The consumer does not know the total size of the queue -- merely how
many messages it has been handed.

No change is made to the prefetch here; however, future changes may
wish to limit the prefetch, either for memory-saving, or to allow
multiple consumers to work the same queue.

Rename the method to make clear that it only contains information
about the local queue in the consumer, not the full RabbitMQ queue.
Also include the waiting message count, which is used by the
`consume()` iterator for similar purpose to the pending events list.
2020-10-09 20:40:39 -07:00
Alex Vandiver a1ce1aca3b queue: Update comment to be more accurate about import errors. 2020-10-09 20:40:32 -07:00
Alex Vandiver 2d71ca1fb8 email: Remove unused `log_digest_event` function.
Its last callsite was removed in e46cbaffa2.

Also ref #6786.
2020-10-08 20:35:53 -07:00
Anders Kaseorg f0ae2a88f2 activity: Show server time rather than browser time.
This gets rid of an inline <script> that was blocking #16451.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-10-08 19:17:35 -07:00
Sumanth V Rao 264bf1fc2a css: Hide bottom-right resizable box in view-source textarea. 2020-10-08 17:13:50 -07:00
Sumanth V Rao adfe4cacc6 css: Add vertical scrollbar styling for view-source textarea.
This styles the vertical scrollbar similar to the horizontal
one for <pre/> (which can be seen in narrow windows). Strictly
speaking, this change shouldn't go in rendered_markdown.scss, but
placing it there helps unify the two scrollbar stylings rather than
duplicating them.
2020-10-08 17:13:50 -07:00
Sumanth V Rao ab9a011f89 css: Modify button styling to adjust for svg size change.
This builds on the previous commit and adds final touches to
both the button positioning and style.
2020-10-08 17:13:50 -07:00
Sumanth V Rao b0c37713e5 templates: Extract and tweak copy_to_clipboard_svg from both buttons.
Since both the original button svg's have their own individual height
and width, this commit tweaks it to values height=20 and width=16 which
works well for both buttons.
2020-10-08 17:13:50 -07:00
Sumanth V Rao 413df1881e templates: Extract and tweak copy_message_button from message_edit_form.
Small tweak in the extracted text which adds aria-lable to the button.
2020-10-08 17:13:50 -07:00
Sumanth V Rao 792029cbb8 css: Refactor copy_message button to use copy_button_base class styling.
Z-index is added to the base class. This doesn't affect copy_code_button
in any way.

Attributes dropped/changed:
    - background-color
    - Base class on-hover property is now used.
    - height, width, padding is now the base classes.

We can also remove the TODO now.
2020-10-08 17:13:50 -07:00
Sumanth V Rao 26ff6db751 css: Extract copy_button_base from copy_codeblock class.
The base class will contain common styling which is used by both
copy_codeblock and copy_message buttons. This sets us up nicely
for following commit(s) which aims to unify the two button styling.
2020-10-08 17:13:50 -07:00
sahil839 4e8f5b5b31 streams: Change access_stream_for_delete_or_update to also return sub.
We modify access_stream_for_delete_or_update function to return
Subscription object also along with stream. This change will be
helpful in avoiding an extra query to get subscription object in
code for updating subscription role.
2020-10-08 17:07:30 -07:00
Aman Agrawal 623fb63406 recent_topics: Go to topic if user clicks anywhere in topic col. 2020-10-08 17:05:06 -07:00
Aman Agrawal d3f4f64b0e timerender: Extract func to calculate days_old from time. 2020-10-08 17:05:06 -07:00
Aman Agrawal c7d66421f9 sidebars: Fill search boxes to full width.
After our bootstrap upgrade, search boxes were not full width on
when sidebars acted as popovers on small widths.
2020-10-08 17:03:56 -07:00
Aman Agrawal 8b419c93e4
message_send: Fix old guests being treated as full members.
For streams in which only full members are allowed to post,
we block guest users from posting there.

Guests users were blocked from posting to admin only streams
already. So now, guest users can only post to
STREAM_POST_POLICY_EVERYONE streams.

This is not a new feature but a bugfix which should have
happened when implementing full member stream policy / guest users.
2020-10-08 11:30:11 -07:00
Aman Agrawal 6fe82620df login: Fix alignment of error message in login form. 2020-10-07 16:18:54 -07:00
Anders Kaseorg f81a5e87ed run-dev: Wait for children to exit on Ctrl+C after killing them.
In addition to being generally more correct, this works around a bug
in Node.js that causes webpack-dev-server to corrupt the terminal
state when exiting as a background process.

https://github.com/nodejs/node/issues/35536

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-10-07 16:15:19 -07:00
Anders Kaseorg 1c7db528bd test_server: Wait for run-dev after terminating it.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-10-07 16:15:19 -07:00
Anders Kaseorg e1a3aa9c1a deliver_scheduled_messages: Exit cleanly on Ctrl+C.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-10-07 16:15:19 -07:00
Anders Kaseorg 363374346c management: Use signal.pause to sleep forever.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-10-07 16:15:19 -07:00
Anders Kaseorg 81d21068b5 eslint: Fix no-useless-concat.
https://eslint.org/docs/rules/no-useless-concat

And add some escaping to static/js/markdown.js while I’m here.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-10-07 16:00:33 -07:00
Anders Kaseorg 043c34d944 eslint: Fix no-throw-literal.
https://eslint.org/docs/rules/no-throw-literal

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-10-07 16:00:33 -07:00