mirror of https://github.com/zulip/zulip.git
3efed5f1e6
Python's behaviour on `sys.exit` is to wait for all non-daemon threads to exit. In the context of the missedmessage_emails worker, if any work is pending, a non-daemon Timer thread exists, which is waiting for 5 seconds. As soon as that thread is serviced, it sets up another 5-second Timer, a process which repeats until all ScheduledMessageNotificationEmail records have been handled. This likely takes two minutes, but may theoretically take up to a week until the thread exits, and thus sys.exit can complete. Supervisor only gives the process 30 seconds to shut down, so something else must prevent this endless Timer. When `stop` is called, take the lock so we can mutate the timer. However, since `stop` may have been called from a signal handler, our thread may _already_ have the lock. As Python provides no way to know if our thread is the one which has the lock, make the lock a re-entrant one, allowing us to always try to take it. With the lock in hand, cancel any outstanding timers. A race exists where the timer may not be able to be canceled because it has finished, maybe_send_batched_emails has been called, and is itself blocked on the lock. Handle this case by timing out the thread join in `stop()`, and signal the running thread to exit by unsetting the timer event, which will be detected once it claims the lock. |
||
|---|---|---|
| .. | ||
| __init__.py | ||
| queue_processors.py | ||