Mirror
/
zulip
mirror of https://github.com/zulip/zulip.git
1
0
Fork 0
Code Issues Projects Releases Wiki Activity
2,243 Commits 28 Branches 143 Tags 606 MiB
Commit Graph

147 Commits

Author SHA1 Message Date
Tim Abbott ef992fc8ba zephyr-mirror-crontab: Fix truncation races. 
(imported from commit f50f34b3a739e67f2dffb442ac01e50eb7c9e4b8)
 2012-12-03 17:56:38 -05:00
Keegan McAllister ae0d007e56 mirroring: Fix import by putting 'api' path first 
(imported from commit b35d962d6456f01b97deeeaa328faa03b65494f8)
 2012-12-03 17:50:29 -05:00
Keegan McAllister 0f0ff53527 zephyr_mirror_backend: Rename variable 'humbug' 
(imported from commit 2e278e4d2452f750cb874510a762fc9616e7dfb3)
 2012-12-03 17:49:27 -05:00
Keegan McAllister 25f0eb67bf gcal-bot: Rename variable 'humbug' 
It's weird to clobber the module, although it works.

(imported from commit 160d6e1c596ba6618c3da9c8cdc2f5cd3ca43dc6)
 2012-12-03 17:49:27 -05:00
Tim Abbott 021a897786 Add zmirror's crontab to git. 
(imported from commit 37c96933a507c661c2ca252c78e5cd9e8a699ba2)
 2012-12-03 16:20:19 -05:00
Tim Abbott 0fe3d93fbc zephyr_mirror: Fix case issue in subscriptions syncing. 
(imported from commit e87ddab3684f4acae940ee831caadbc346458a11)
 2012-12-03 16:20:19 -05:00
Keegan McAllister 7a080c7911 gcal-bot: Remove .humbug-api-key reading 
This behavior is part of api/humbug.py now.

(imported from commit 8a9722c8ad05391160cbb98387f34ef9924d81de)
 2012-12-03 15:54:54 -05:00
Keegan McAllister 9125489c20 Rename class HumbugAPI to Client 
(imported from commit 4270f31fc5febcd9c444d0d133a1dad3860618f0)
 2012-12-03 15:54:54 -05:00
Keegan McAllister bd7b7adb6b Rename api.common to humbug 
Fixes #482.

(imported from commit 1bd6a7fd993d8d5e225e0311c288dbce0c369a40)
 2012-12-03 15:54:54 -05:00
Tim Abbott 70a8e51351 Rewrite check-mirroring to be more robust and flexible. 
Features include:
* Not forking into two processes (shells out to zwrite to send
  instead).  This makes life easier since we're not doing concurrent
  programming.
* Eliminated a lot of hard-to-read or unnecessary debugging output.
* Adding explanatory test suggesting the likely problem for some
  common sets of received messages.
* Much less code duplication.
* Support for testing a sharded zephyr_mirror script (--sharded).
* Use of the logging module to print timestamps -- makes debugging
  some issues a lot.
* Only one sleep, and for only 10 seconds, between sending the
  outgoing messages and checking that they were received.
* Support for running two copies of this script at the same time, so that
  running it manually doesn't screw up Nagios.
* Passed running 100 tests run in a row.

(imported from commit a3ec02ac1d1a04972e469ca30fec1790c4fb53bc)
 2012-11-30 17:57:52 -05:00
Tim Abbott 4dc47bcd33 zephyr_mirror: Move class name canonicalization earlier. 
We should be canonicalizing stream names to class names in
update_subscriptions_from_humbug, before we even decide which classes
to subscribe to; otherwise deduplication and tracking of which classes
we're already subscribed to won't work.

(imported from commit a751b6fca1022390a087516a0730ff77f13d7edf)
 2012-11-29 13:57:06 -05:00
Tim Abbott b699ad6cde zephyr_mirror: Add options to run class mirror without sharding for debugging. 
(imported from commit 4fcc403504d93e30aca6046cba9f1fe5342faac6)
 2012-11-29 13:57:06 -05:00
Tim Abbott b3c1f08b3b zephyr_mirror: Don't leak zephyr_mirror processes on send_humbug failures. 
(imported from commit be005018672369683d9e3bdbf144f96f58af6b6c)
 2012-11-29 13:57:06 -05:00
Tim Abbott b54e5eb3c0 zephyr_mirror: Replace bare excepts with 'except Exception'. 
(imported from commit 857212e0e5116471e418d37845550108c0a47d2b)
 2012-11-29 13:57:05 -05:00
Tim Abbott de3799589b zephyr_mirror: Compare zephyr subs from server after normalization. 
(imported from commit eea290698e83c7d26f23ab3bb0201914417f0e09)
 2012-11-28 22:26:10 -05:00
Tim Abbott 18aace499c zephyr_mirror: Add two more restricted access classes. 
(imported from commit 06ff0e19c1522ade018aea66f152fdfa55c18e1d)
 2012-11-28 22:26:10 -05:00
Tim Abbott a1d8e9d30f get_updates: Replace the "mirror" argument with checking "client". 
(imported from commit bd84c2c7a456de8ea7ec35eaf76b5bf8647ca61a)
 2012-11-28 17:41:38 -05:00
Tim Abbott f58898fba7 zephyr_mirror: Pass an encoded stream to hashlib for sharding. 
(imported from commit cc57b3956198b66c9cfc84423434a71b8200d819)
 2012-11-28 15:44:35 -05:00
Tim Abbott 9505b38004 zephyr_mirror: Validate zephyr subscriptions happened successfully. 
(imported from commit 494895b87eb9b555175c9890c5c2046cf4ce40ac)
 2012-11-28 15:44:35 -05:00
Tim Abbott 84ab466ff3 zephyr_mirror: Fix display of personals on non-message classes. 
(imported from commit 42ff030d2d78c530428dcede406d46fc71521fef)
 2012-11-27 14:12:19 -05:00
Keegan McAllister e4b7fcc2d9 send-nagios-notification: Expand "\n" escape sequences that we get from Nagios 
(imported from commit 3e8efbfe8bbd5a2f13a8e02d9f44442b035113ba)
 2012-11-27 14:08:49 -05:00
Tim Abbott 8cf9012977 zephyr_mirror: Run the class mirror in 16 parallel shards. 
(imported from commit cb17ef999e94cc57d5b01114479728b04475c9c4)
 2012-11-27 12:09:57 -05:00
Tim Abbott af40020ff9 zephyr_mirror: Add a --no-forward-from-humbug option. 
We'll use this to only have one of our shards forward from Humbug.

(imported from commit a6f3c2180b49c1fa826baf1ae9e7e3ba252a5bfa)
 2012-11-27 12:09:57 -05:00
Tim Abbott cdabd2b90a zephyr_mirror: Clean up unnecessary logic in process_notice. 
(imported from commit 28599789016fd1006c2de4d5ddc3cf1f10baa47f)
 2012-11-27 12:09:57 -05:00
Tim Abbott 4b6f771be1 zephyr_mirror: Fix the check for whether a Zephyr is a personal. 
(imported from commit 609b6339fa82ecbd6435a386d24bf555e75a8120)
 2012-11-27 12:09:57 -05:00
Tim Abbott f05967034b zephyr_mirror: Reorder process_notice to log fewer "Skipping" messages. 
Previously, we were sending "Skipping message we got from Humbug!"
for messages we wouldn't have forwarded anyway.

(imported from commit 36df85a61336ac00e3d7913d5a417d6b42764350)
 2012-11-27 12:09:57 -05:00
Tim Abbott a9b938d7ac zephyr_mirror: Don't display/log some useless "Skipping" notices. 
In this case, if we're configured to not forward personals, there's no
point in logging a decision not to forward one.

(imported from commit 62c37591c6a70afb6235de626b0c6a3502cbcb27)
 2012-11-27 12:09:56 -05:00
Tim Abbott 09c5a5c06f zephyr_mirror: Simplify optparse usage. 
(imported from commit d7cce4236dae6f369ea1cc4e12f89c2011718b7f)
 2012-11-27 12:09:56 -05:00
Tim Abbott 9c7878164c check-mirroring: Fix some issues with startup process timing. 
It seems that check-mirroring was reporting a lot of spurious failures
due to it sometimes taking more than 3 seconds for the setup phase of
check-mirroring's receive path to run.  So fix this:

(1) Wait a bit more than 3 seconds for the receiver to subscribe to
messages

(2) Subscribe to Humbug messages before forking (we can't do this with
zephyr.init() because python-zephyr gets totally messed up if you use
it from multiple processes with a shared initialization)

(3) Get rid of the old time.sleep(0.x) values that were intended to
make messages arrive in order -- since we're now checking that
messages correctly arrived using set(), they aren't needed.

(4) Use a single request to subscribe to both zephyr classes we need
to subscribe to (saves 1 RTT).

(imported from commit d96aef05405ce43e9a4a549de189da9a2e393875)
 2012-11-26 17:10:15 -05:00
Tim Abbott 369f5e9d80 zephyr_mirror: Don't use @(@color(blue)) in zsigs. 
(imported from commit e4b594e3c22c25fa4f87baaad09227396bb92a0d)
 2012-11-26 16:19:09 -05:00
Tim Abbott 789a5c5e2d zephyr_mirror: Use default format to indicate what came from Humbug. 
(imported from commit 5c2fdccb08d605a37d6c87c37f023c06a0a9a760)
 2012-11-26 16:18:05 -05:00
Tim Abbott e16e5f7e1a zephyr_mirror: Refactor zwrite_args handling a bit. 
(imported from commit 4923c90dc546022d48f226c2a204e9d09717b563)
 2012-11-26 14:58:34 -05:00
Keegan McAllister d7b3afef6b Send Nagios alerts to Humbug 
Fixes #385.

(imported from commit 7dac013debd6ccff031fc4da0dd7185e198b4498)
 2012-11-26 14:42:55 -05:00
Tim Abbott 9c0e0844ef check-mirroring: Add subscriptions inside IOError check. 
(imported from commit d3b00ce9d91a0bcffcef58436a4b90770155fce9)
 2012-11-26 12:25:39 -05:00
Tim Abbott 3cc5973511 api: Add copyright and MIT license notices to distributed tools. 
(imported from commit 9f93050f988549dc363f5a7c798c8a9fa1c01d29)
 2012-11-26 12:25:38 -05:00
Tim Abbott 6ac25fb9ba zephyr_mirror: Don't break long words/urls when mirroring. 
(imported from commit df2b6a6969fc54016eeee4996df317e2843844c8)
 2012-11-21 15:55:18 -05:00
Tim Abbott f72427d83c zephyr_mirror: Fix running --sync-subscriptions repeatedly. 
(imported from commit 15bead790590c91cd4a1d794799f94341d31c181)
 2012-11-20 17:17:40 -05:00
Jessica McKellar f1fc271258 nagios: Increase the timeout on MIT zephyr forwarding functioning. 
Sometimes messages are delayed passed the allowed 10 seconds. We may
eventually decide that that amount of latency is unacceptable, but the
latency is not what this script is supposed to be checking.

(imported from commit d83a6a83d60e9eac13b3b87fb31de7f9881acabf)
 2012-11-20 15:32:06 -05:00
Tim Abbott 5587b502a2 zephyr_mirror: Add exception handling around all main loop code. 
(imported from commit 5dc46decf882302a6c58f3bb788362f82f293b35)
 2012-11-19 12:44:12 -05:00
Tim Abbott ab9858ba65 zephyr_mirror: Use minimal wrapper script that is hard to accidentally break. 
(imported from commit 45967b8b806df616b0480b217dd5db82c5a2cabd)
 2012-11-19 12:44:06 -05:00
Tim Abbott c0f649559c zephyr_mirror: Move parse_args into its own function. 
(imported from commit b56ca627e3d34b8dcefead2cfecfa77bb428b0ec)
 2012-11-19 12:30:15 -05:00
Tim Abbott 235654e022 zephyr_mirror: Get rid of extra_mirror code -- it is no longer used. 
(imported from commit ab2a9cfcfa6a8c7615523ebb16864e9a4db05f3a)
 2012-11-19 12:11:07 -05:00
Tim Abbott 2045bdae77 zephyr_mirror: Improve robustness when killing child processes. 
(imported from commit f212b87ffb1fc814d289c14d3854fb1f2f6e2cf3)
 2012-11-19 11:46:05 -05:00
Tim Abbott e9bc9ac3c2 zephyr_mirror: Fix restart_stamp functionality. 
(imported from commit a79db110d3372ab88fabfc0b7013c5fd3ad07f8e)
 2012-11-19 11:20:33 -05:00
Tim Abbott eac3ae2ed1 zephyr_mirror: Improve zwrite error handling. 
(imported from commit a6b5b9c0c9bd6ffde0b2b783c87528bdcae03566)
 2012-11-16 17:08:51 -05:00
Tim Abbott afc62a8e16 [manual] Uniformize the subscriptions API method names. 
This requires a zephyr_mirror deployment when it is pushed to prod.

(imported from commit 6543441fb264b518f8705d7989d068a1d50ce5d6)
 2012-11-16 17:06:01 -05:00
Jessica McKellar b48671ae43 nagios: print more information on check-mirroring errors. 
(imported from commit 6ea149d939ae309745ce8c5b2ab7ae2237007d97)
 2012-11-16 11:25:22 -05:00
Tim Abbott 411a7f6b4f [manual] send_message: Rename recipient/stream fields to 'to'. 
This commit changes APIs and requires and update of all zephyr
mirroring bots to deploy properly.

(imported from commit 2672d2d07269379f7a865644aaeb6796d54183e1)
 2012-11-15 15:30:06 -05:00
Tim Abbott 9e7802cd51 check-mirroring: Use consistent ordering for keys. 
(imported from commit 7deaf55ccb4b47a11dbd30c9239893a42eb39e00)
 2012-11-15 14:56:10 -05:00
Tim Abbott a11a4dceb7 zephyr_mirror: Improve error handling getting public streams list. 
(imported from commit 06f9c00f19056bfe7db3c02955e3a29ad270de6e)
 2012-11-15 14:51:05 -05:00