Commit Graph

42 Commits

Author SHA1 Message Date
Anders Kaseorg 97e4e9886c python: Replace universal_newlines with text.
This is supported in Python ≥ 3.7.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-01-23 22:16:01 -08:00
Anders Kaseorg 702ce071f4 python: Accept Optional[FrameType] in signal handlers.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-12-28 09:31:55 -08:00
Anders Kaseorg 684dad8145 tools: Use root-based absolute import for tools.lib, etc.
Mypy can’t follow absolute imports based on directories other than the
root.  This was hiding some type errors due to ignore_missing_imports.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-07-05 12:21:52 -07:00
Anders Kaseorg 544bbd5398 docs: Fix capitalization mistakes.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-05-10 09:57:26 -07:00
Anders Kaseorg 6e4c3e41dc python: Normalize quotes with Black.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-02-12 13:11:19 -08:00
Anders Kaseorg 11741543da python: Reformat with Black, except quotes.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-02-12 13:11:19 -08:00
Mateusz Mandera 716df658fa queue_processors: Don't run test queues with run-dev.py. 2020-10-18 14:07:31 -07:00
Mateusz Mandera 269faa342a test-queue-worker-reload: Fix expected worker count.
We didn't bump this when adding some additional test queues, that run
with run-dev.py as of this commit.
2020-10-18 14:07:31 -07:00
Anders Kaseorg 365fe0b3d5 python: Sort imports with isort.
Fixes #2665.

Regenerated by tabbott with `lint --fix` after a rebase and change in
parameters.

Note from tabbott: In a few cases, this converts technical debt in the
form of unsorted imports into different technical debt in the form of
our largest files having very long, ugly import sequences at the
start.  I expect this change will increase pressure for us to split
those files, which isn't a bad thing.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-11 16:45:32 -07:00
Anders Kaseorg 8dd83228e7 python: Convert "".format to Python 3.6 f-strings.
Generated by pyupgrade --py36-plus --keep-percent-format, but with the
NamedTuple changes reverted (see commit
ba7906a3c6, #15132).

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-08 15:31:20 -07:00
Mateusz Mandera dd40649e04 queue_processors: Remove the slow_queries queue.
While this functionality to post slow queries to a Zulip stream was
very useful in the early days of Zulip, when there were only a few
hundred accounts, it's long since been useless since (1) the total
request volume on larger Zulip servers run by Zulip developers, and
(2) other server operators don't want real-time notifications of slow
backend queries.  The right structure for this is just a log file.

We get rid of the queue and replace it with a "zulip.slow_queries"
logger, which will still log to /var/log/zulip/slow_queries.log for
ease of access to this information and propagate to the other logging
handlers.  Reducing the amount of queues is good for lowering zulip's
memory footprint and restart performance, since we run at least one
dedicated queue worker process for each one in most configurations.
2020-05-11 00:45:13 -07:00
Anders Kaseorg d3c55c166e requirements: Upgrade mypy from 0.761 to 0.770.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-04-18 13:09:51 -07:00
Mateusz Mandera 4c5a8e6f0c queue: Remove missedmessage_email_senders. 2020-01-31 12:13:51 -08:00
Mateusz Mandera a583da67b3 test-queue-worker-reload: Fix expected number of workers. 2020-01-31 12:13:51 -08:00
Anders Kaseorg 40f4ead738 mypy: Upgrade from 0.720 to 0.730.
Fixes #13269.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-11-13 12:38:45 -08:00
Anders Kaseorg 6678a95b20 test-queue-worker-reload: Use a pipe rather than polling a log file.
This avoids unnecessary polling delays, quadratic string operations,
and hardcoded paths in /tmp.

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2019-01-15 16:00:48 -08:00
Anders Kaseorg 19f37fb3d4 tools/test-queue-worker-reload: Avoid shelling out for touch.
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2018-07-25 16:54:46 -07:00
Aditya Bansal addfde7374 tools: Change use of typing.Text to str. 2018-05-13 17:17:32 -07:00
Greg Price 4475950ddf queue: Restore prematurely-cut upgrade path.
Revert c8f034e9a "queue: Remove missedmessage_email_senders code."
As the comment in the code says, it ensures a smooth upgrade path
from 1.7.x; we can delete it in master after 1.8.0 is released.
The removal commit was merged early due to a communication failure.
2018-02-28 11:15:53 -08:00
Umair Khan c8f034e9a0 queue: Remove missedmessage_email_senders code.
After 68513952fb, all emails are sent through email_senders queue.
This commit removes code related to the legacy queue.
2018-02-21 16:43:56 -08:00
Umair Khan 764cd977d6 test-queue-worker-reload: Increase delay to 1.3s.
See 625939 for more information. In short, the purpose of this delay is
to give autoreload code enough time to touch every watched file at least
once before the change is made.
2018-02-06 10:35:20 -08:00
Umair Khan 6259390f40 test-queue-worker-reload: Add wait to avoid race condition.
The autoreload code of Django works by looping over the files associated
with all the loaded modules. This loop is run after every 1 second. If
the file is found for the first time by the loop, it is assumed that the
file is new and is not modified between the time it is loaded and is
checked by the loop. This assumption is the source of a race condition.

We can either implement a more sensitive version of the loop or we can
just allow enough time to the Django loop to touch every file at least
once.

For the time being, we are going with the second option.
2018-01-05 08:24:57 -05:00
Umair Khan e68adab75d test-queue-worker-reload: Poll for 60 seconds. 2018-01-05 08:24:57 -05:00
Umair Khan 21eabf4a60 travis: Fix test-queue-worker-reload.
Previously, there were following problems with the implmentation:
* Same file handle was being used to read and write. We used to do
  `seek(0)` and then `read()`. This had a chance to overwrite
  file data. Now we use different file handles to read and write data.
* We were using text streams. Text streams cannot be used with
  `bufferring=0`. Now we use binary streams without buffering so that
  data is available for reading without any delay.

This commit also updates the key(s) that we search in the logfile.
Previously, launch of all queues was announced in the log, now we only
anounce the number of threads that were launched.

This commit also makes sure that we always exit after gracefull shutting
down the development server.
2018-01-05 08:24:57 -05:00
Umair Khan 68513952fb email-worker: Create EmailSendingWorker.
This commit just copies all the code from MissedMessageSendingWorker
class to a new EmailSendingWorker class. All the logic to send an email
through a queue was already there. This commit only makes the logic
generic. It does so by creating a special purpose queue called
'email_senders' to send any type of email. To make
MissedMessageSendingWorker still work we derive it from
EmailSendingWorker. All the tests that were testing
MissedMessageSendingWorker now run against EmailSendingWorker.
2017-12-20 19:36:27 -08:00
Greg Price 137c0e65bb tools: Revert to Python 2 typing syntax for now.
This reverts commit 66261f1cc.  See parent commit for reason; here,
provision worked but `tools/run-dev.py` would give errors.

We need to figure out a test that reproduces these issues, then make a
version of these changes that keeps that test working, before we
re-merge them.
2017-12-13 10:38:15 -08:00
rht 66261f1cc3 tools: Use Python 3 syntax for typing in many files. 2017-12-12 17:42:57 -08:00
Vishnu Ks 766511e519 actions: Mark all messages as read when user unsubscribes from stream.
This fixes a bug where, when a user is unsubscribed from a stream,
they might have unread messages on that stream leak.  While it might
seem to be a minor problem, it can cause significant problems for
computing the `unread_msgs` data structures, since it means we need to
add an extra filter for whether the user is still subscribed, either
in the backend or in the UI.

Fixes #7095.
2017-11-21 20:09:17 -08:00
neiljp (Neil Pilgrim) a5d1fdf0d6 mypy: Add type parameter for IO in test-queue-worker-reload. 2017-11-04 19:47:44 -07:00
rht 691598a88b py3: Remove "from six.moves import range".
This is no longer required, since in Python 3, this is what the range
built-in does.
2017-10-17 23:28:14 -07:00
rht f15bdce90e tools: Remove print_function.
Tweaked by tabbott to exclude the linter libraries.
2017-09-29 15:44:56 -07:00
rht bf4eda7374 tools: Remove absolute_import in most tools.
Tweaked by tabbott to not remove it from lister.py, linter_lib, and
friends, since those are intended to support both Python 2 and 3
(we're planning to extract them from the repository).
2017-09-29 12:28:43 -07:00
Greg Price a099e698e2 py3: Switch almost all shebang lines to use `python3`.
This causes `upgrade-zulip-from-git`, as well as a no-option run of
`tools/build-release-tarball`, to produce a Zulip install running
Python 3, rather than Python 2.  In particular this means that the
virtualenv we create, in which all application code runs, is Python 3.

One shebang line, on `zulip-ec2-configure-interfaces`, explicitly
keeps Python 2, and at least one external ops script, `wal-e`, also
still runs on Python 2.  See discussion on the respective previous
commits that made those explicit.  There may also be some other
third-party scripts we use, outside of this source tree and running
outside our virtualenv, that still run on Python 2.
2017-08-16 17:54:43 -07:00
K.Kanakhin 6a801db1c2 missed-emails-sending: Move email sending to separate queue worker.
- Add new 'missedmessage_email_senders' queue for sending missed messages emails.
- Add the new worker to process 'missedmessage_email_senders' queue.
- Split aggregation missed messages and sending missed messages email
  to separate queue workers.
- Adapt tests for sending missed emails to the new logic.

Fixes #2607
2017-03-07 20:08:40 -08:00
Tim Abbott d6bbcd2737 travis: Automate updates to production-helper Nagios test.
This list was likely to end up out of date quickly, since it wasn't
documented that you need to update it when adding a queue.  The best
solution is to just not require it to be updated.
2017-02-19 16:19:53 -08:00
sinwar eab355b0cd tools: Create more consistent checks for venv.
This helps make the Zulip development environment somewhat more robust
to new contributors, since it will give them a nice warning if they
try running any of our development tools outside the Zulip virtualenv.

Fixes #3468.
2017-02-06 21:50:32 -08:00
JefftheBest1 110398bf36 Fixed typos in test-queue-worker-reload 2017-01-12 04:51:27 -08:00
Tim Abbott 0c5f419be0 Annotate test-queue-worker-reload. 2016-10-15 22:36:10 -07:00
Umair Khan 740b5634ac Fix and re-enable test-queue-worker-reload.
It seems that we no longer get the message, 'zerver/lib/actions.py
modified; restarting server', but the server reloads successfully
nonetheless.

Fixes: #1341.
2016-08-10 16:17:23 -07:00
Tim Abbott 30dd8cfd4a test-queue-worker-reload: Increase sleep resolution. 2016-07-21 10:21:29 -07:00
Umair Khan 7d687b0f79 Increase the timeout in test-queue-worker-reload.
May fix #1341.
2016-07-21 10:19:49 -07:00
rahuldeve 3ca53df152 Add test for worker queue autoreloading. 2016-06-26 20:12:37 -07:00