dbaeb9b1d5
zephyr_mirror: Add comment explaining why we drop the @ATHENA.MIT.EDU in zwrite -C.
...
(imported from commit b591e2e0d6c607ef04e3c0005091078cfe23e3ff)
2012-12-10 17:05:04 -05:00
fb574966e1
zephyr_mirror: Improve debugging output on outgoing messages.
...
Previously all that we logged was the sender, which turns out to be a
constant for humbug=>zephyr mirroring.
(imported from commit 527a3ac4b9b815a2b4d6b63c3ad92d9d5ad99a6e)
2012-12-10 17:05:04 -05:00
0532d4b49e
check_mirroring: Retry automatically on 'Detected server failure' errors.
...
(imported from commit 2f834218bd492876749dd8953e522725935a7a1d)
2012-12-10 17:05:04 -05:00
635c5f3d1f
check-mirroring: Use zwrite -n to not send PINGs.
...
(imported from commit e345f1b86c7f8850ac4eb4ae092cc83717e2cbe8)
2012-12-10 17:05:04 -05:00
2d1e072896
zephyr_mirror: Fix handling of Humbug=>Zephyr personals.
...
(imported from commit 149f8aaf9076583499392bcb09a0250b05c3dea6)
2012-12-08 13:05:44 -05:00
17a0246b0b
zephyr_mirror: Use a variable for humbug_account_email.
...
(imported from commit 90da406a5eab67d20b42243e96b8afb3e5b5f342)
2012-12-08 13:00:22 -05:00
c241713332
zephyr_mirror: Fix detection of personals.
...
(imported from commit 1fc9050bad36dc15e76f65bdb6532e3e7e601a40)
2012-12-05 12:50:14 -05:00
b00f23e21f
Don't send zwrite pings.
...
(imported from commit 8dd7894eda6315ab52e61489e21ec1d201e86cc9)
2012-12-05 12:04:18 -05:00
7e4776322d
[manual] Unify huddles and personals into private messages on the receive path
...
feedback-bot and zephyr_mirror will need to be updated and restarted
when this is deployed to prod.
(imported from commit fe2b524424c174bcb1b717a851a5d3815fda3f69)
2012-12-04 18:01:51 -05:00
62fff3b900
Add a Tddium post-build notification task for CUSTOMER1
...
Resolves #502 .
(imported from commit b88fa55ea6561b601a3d6519c84d56ba3098c3fe)
2012-12-04 14:25:44 -05:00
ef992fc8ba
zephyr-mirror-crontab: Fix truncation races.
...
(imported from commit f50f34b3a739e67f2dffb442ac01e50eb7c9e4b8)
2012-12-03 17:56:38 -05:00
ae0d007e56
mirroring: Fix import by putting 'api' path first
...
(imported from commit b35d962d6456f01b97deeeaa328faa03b65494f8)
2012-12-03 17:50:29 -05:00
0f0ff53527
zephyr_mirror_backend: Rename variable 'humbug'
...
(imported from commit 2e278e4d2452f750cb874510a762fc9616e7dfb3)
2012-12-03 17:49:27 -05:00
25f0eb67bf
gcal-bot: Rename variable 'humbug'
...
It's weird to clobber the module, although it works.
(imported from commit 160d6e1c596ba6618c3da9c8cdc2f5cd3ca43dc6)
2012-12-03 17:49:27 -05:00
021a897786
Add zmirror's crontab to git.
...
(imported from commit 37c96933a507c661c2ca252c78e5cd9e8a699ba2)
2012-12-03 16:20:19 -05:00
0fe3d93fbc
zephyr_mirror: Fix case issue in subscriptions syncing.
...
(imported from commit e87ddab3684f4acae940ee831caadbc346458a11)
2012-12-03 16:20:19 -05:00
82475afe82
Fix module and class name in api/README
...
(imported from commit c5157d52479ef0e1f26c471df79ffb9c5ea7f51e)
2012-12-03 15:54:54 -05:00
7a080c7911
gcal-bot: Remove .humbug-api-key reading
...
This behavior is part of api/humbug.py now.
(imported from commit 8a9722c8ad05391160cbb98387f34ef9924d81de)
2012-12-03 15:54:54 -05:00
9125489c20
Rename class HumbugAPI to Client
...
(imported from commit 4270f31fc5febcd9c444d0d133a1dad3860618f0)
2012-12-03 15:54:54 -05:00
bd7b7adb6b
Rename api.common to humbug
...
Fixes #482 .
(imported from commit 1bd6a7fd993d8d5e225e0311c288dbce0c369a40)
2012-12-03 15:54:54 -05:00
70a8e51351
Rewrite check-mirroring to be more robust and flexible.
...
Features include:
* Not forking into two processes (shells out to zwrite to send
instead). This makes life easier since we're not doing concurrent
programming.
* Eliminated a lot of hard-to-read or unnecessary debugging output.
* Adding explanatory test suggesting the likely problem for some
common sets of received messages.
* Much less code duplication.
* Support for testing a sharded zephyr_mirror script (--sharded).
* Use of the logging module to print timestamps -- makes debugging
some issues a lot.
* Only one sleep, and for only 10 seconds, between sending the
outgoing messages and checking that they were received.
* Support for running two copies of this script at the same time, so that
running it manually doesn't screw up Nagios.
* Passed running 100 tests run in a row.
(imported from commit a3ec02ac1d1a04972e469ca30fec1790c4fb53bc)
2012-11-30 17:57:52 -05:00
4dc47bcd33
zephyr_mirror: Move class name canonicalization earlier.
...
We should be canonicalizing stream names to class names in
update_subscriptions_from_humbug, before we even decide which classes
to subscribe to; otherwise deduplication and tracking of which classes
we're already subscribed to won't work.
(imported from commit a751b6fca1022390a087516a0730ff77f13d7edf)
2012-11-29 13:57:06 -05:00
1bc98a6f8a
call_on_each_message: Remove dead 'first' parameter.
...
(imported from commit 06885bf8049238184226d9fc7b64345e6848931a)
2012-11-29 13:57:06 -05:00
735857008a
api: Don't mutate the original request.
...
This causes e.g. call_on_each_message to switch to dont_block mode
after the first error.
(imported from commit b6a5a10970c987faf8017f0ddae4e0b64a513c6f)
2012-11-29 13:57:06 -05:00
7914bb78c6
do_api_query: Reduce error handling code duplication.
...
Also fixes bugs where the retry code wouldn't work correctly if
verbose wasn't set, prints out which API query had the error, and is
more consistent about printing something to end the "..." sequences.
(imported from commit 15c1e0e4a14c5c5559b43bafe4ec268451ee04f5)
2012-11-29 13:57:06 -05:00
0a278b55b9
api: Only retry up to 10 times on connection errors.
...
(imported from commit f395370120d5e7a635eec7b27e4d5ed0c48b774d)
2012-11-29 13:57:06 -05:00
b699ad6cde
zephyr_mirror: Add options to run class mirror without sharding for debugging.
...
(imported from commit 4fcc403504d93e30aca6046cba9f1fe5342faac6)
2012-11-29 13:57:06 -05:00
b3c1f08b3b
zephyr_mirror: Don't leak zephyr_mirror processes on send_humbug failures.
...
(imported from commit be005018672369683d9e3bdbf144f96f58af6b6c)
2012-11-29 13:57:06 -05:00
b54e5eb3c0
zephyr_mirror: Replace bare excepts with 'except Exception'.
...
(imported from commit 857212e0e5116471e418d37845550108c0a47d2b)
2012-11-29 13:57:05 -05:00
de3799589b
zephyr_mirror: Compare zephyr subs from server after normalization.
...
(imported from commit eea290698e83c7d26f23ab3bb0201914417f0e09)
2012-11-28 22:26:10 -05:00
18aace499c
zephyr_mirror: Add two more restricted access classes.
...
(imported from commit 06ff0e19c1522ade018aea66f152fdfa55c18e1d)
2012-11-28 22:26:10 -05:00
d23a83ee81
get_updates: Replace "failures" with the new dont_block option.
...
(imported from commit 2b4ecb35c4f14b6c408323662ef6f39c6485c62d)
2012-11-28 17:41:38 -05:00
a1d8e9d30f
get_updates: Replace the "mirror" argument with checking "client".
...
(imported from commit bd84c2c7a456de8ea7ec35eaf76b5bf8647ca61a)
2012-11-28 17:41:38 -05:00
f58898fba7
zephyr_mirror: Pass an encoded stream to hashlib for sharding.
...
(imported from commit cc57b3956198b66c9cfc84423434a71b8200d819)
2012-11-28 15:44:35 -05:00
9505b38004
zephyr_mirror: Validate zephyr subscriptions happened successfully.
...
(imported from commit 494895b87eb9b555175c9890c5c2046cf4ce40ac)
2012-11-28 15:44:35 -05:00
84ab466ff3
zephyr_mirror: Fix display of personals on non-message classes.
...
(imported from commit 42ff030d2d78c530428dcede406d46fc71521fef)
2012-11-27 14:12:19 -05:00
e4b7fcc2d9
send-nagios-notification: Expand "\n" escape sequences that we get from Nagios
...
(imported from commit 3e8efbfe8bbd5a2f13a8e02d9f44442b035113ba)
2012-11-27 14:08:49 -05:00
2f198a50c1
send_message: Accept subject and message arguments.
...
(imported from commit 93aca5e750fa74d7ee95d8a00adbf6d4e61b2fc8)
2012-11-27 14:02:24 -05:00
8cf9012977
zephyr_mirror: Run the class mirror in 16 parallel shards.
...
(imported from commit cb17ef999e94cc57d5b01114479728b04475c9c4)
2012-11-27 12:09:57 -05:00
af40020ff9
zephyr_mirror: Add a --no-forward-from-humbug option.
...
We'll use this to only have one of our shards forward from Humbug.
(imported from commit a6f3c2180b49c1fa826baf1ae9e7e3ba252a5bfa)
2012-11-27 12:09:57 -05:00
cdabd2b90a
zephyr_mirror: Clean up unnecessary logic in process_notice.
...
(imported from commit 28599789016fd1006c2de4d5ddc3cf1f10baa47f)
2012-11-27 12:09:57 -05:00
4b6f771be1
zephyr_mirror: Fix the check for whether a Zephyr is a personal.
...
(imported from commit 609b6339fa82ecbd6435a386d24bf555e75a8120)
2012-11-27 12:09:57 -05:00
f05967034b
zephyr_mirror: Reorder process_notice to log fewer "Skipping" messages.
...
Previously, we were sending "Skipping message we got from Humbug!"
for messages we wouldn't have forwarded anyway.
(imported from commit 36df85a61336ac00e3d7913d5a417d6b42764350)
2012-11-27 12:09:57 -05:00
a9b938d7ac
zephyr_mirror: Don't display/log some useless "Skipping" notices.
...
In this case, if we're configured to not forward personals, there's no
point in logging a decision not to forward one.
(imported from commit 62c37591c6a70afb6235de626b0c6a3502cbcb27)
2012-11-27 12:09:56 -05:00
09c5a5c06f
zephyr_mirror: Simplify optparse usage.
...
(imported from commit d7cce4236dae6f369ea1cc4e12f89c2011718b7f)
2012-11-27 12:09:56 -05:00
08bbf889ed
api: Extend documentation a bit.
...
(imported from commit fd435286e12ffebad8476cc8005303b2d8d70fd9)
2012-11-27 12:09:56 -05:00
9c7878164c
check-mirroring: Fix some issues with startup process timing.
...
It seems that check-mirroring was reporting a lot of spurious failures
due to it sometimes taking more than 3 seconds for the setup phase of
check-mirroring's receive path to run. So fix this:
(1) Wait a bit more than 3 seconds for the receiver to subscribe to
messages
(2) Subscribe to Humbug messages before forking (we can't do this with
zephyr.init() because python-zephyr gets totally messed up if you use
it from multiple processes with a shared initialization)
(3) Get rid of the old time.sleep(0.x) values that were intended to
make messages arrive in order -- since we're now checking that
messages correctly arrived using set(), they aren't needed.
(4) Use a single request to subscribe to both zephyr classes we need
to subscribe to (saves 1 RTT).
(imported from commit d96aef05405ce43e9a4a549de189da9a2e393875)
2012-11-26 17:10:15 -05:00
369f5e9d80
zephyr_mirror: Don't use @(@color(blue)) in zsigs.
...
(imported from commit e4b594e3c22c25fa4f87baaad09227396bb92a0d)
2012-11-26 16:19:09 -05:00
789a5c5e2d
zephyr_mirror: Use default format to indicate what came from Humbug.
...
(imported from commit 5c2fdccb08d605a37d6c87c37f023c06a0a9a760)
2012-11-26 16:18:05 -05:00
e16e5f7e1a
zephyr_mirror: Refactor zwrite_args handling a bit.
...
(imported from commit 4923c90dc546022d48f226c2a204e9d09717b563)
2012-11-26 14:58:34 -05:00
9274ff20d5
examples/send-message: Remove default sender
...
(imported from commit 30ada075ac4f1b759fe787eef6a93275dbea584c)
2012-11-26 14:57:20 -05:00
128302a7ff
api/examples: Simplify optparse use
...
- The default 'default' is None
- The default 'dest' is the option name, with - replaced with _
- The default 'action' is 'store'
Not coincidentally, these defaults are correct for most of our existing code.
(imported from commit 9c6078bd778324e08e1ca214a97281a7bdce08c2)
2012-11-26 14:57:20 -05:00
ca723aed43
api/common.py: Ensure that the API key file is closed when we're done with it
...
(imported from commit 87a0feb52952aa76bec041d032027e7a272156fc)
2012-11-26 14:57:20 -05:00
a09c9de17c
api/examples: Rework sys.path computation
...
(imported from commit 6e4c2a9b5f5ab6f6680704a1e06bd17fd95bd038)
2012-11-26 14:57:20 -05:00
c1994f3886
api/common.py: Reduce code duplication between API call methods
...
(imported from commit 4a0f1a8479c21366bf552edf98401b03bad72d9c)
2012-11-26 14:57:20 -05:00
8576446791
Use a new-style class for HumbugAPI
...
(imported from commit a88366e0129dadb40b058ddd0ede64db3b6b206f)
2012-11-26 14:57:20 -05:00
d7b3afef6b
Send Nagios alerts to Humbug
...
Fixes #385 .
(imported from commit 7dac013debd6ccff031fc4da0dd7185e198b4498)
2012-11-26 14:42:55 -05:00
c86fd609e2
Add tool to build an API tarball for distribution.
...
(imported from commit eeaf9f526f243cc628eede603fe1b1691c34977f)
2012-11-26 12:25:39 -05:00
9c0e0844ef
check-mirroring: Add subscriptions inside IOError check.
...
(imported from commit d3b00ce9d91a0bcffcef58436a4b90770155fce9)
2012-11-26 12:25:39 -05:00
f7101aa3ef
api: Add an extra send_message example.
...
(imported from commit fe5ab550fa862f587cdf7987c5b238dffab45672)
2012-11-26 12:25:39 -05:00
32f91f4ab0
api: Don't use 127.0.0.1:8000 as an example.
...
At least the zephyr.humbughq.com case is one our users might find
useful. I imagine we'll end up hiding --site once we integrate the
zephyr and main site tornado servers.
(imported from commit a3a28492cf3f32b78eab6bbd48ba66be2f38a5f2)
2012-11-26 12:25:39 -05:00
3cc5973511
api: Add copyright and MIT license notices to distributed tools.
...
(imported from commit 9f93050f988549dc363f5a7c798c8a9fa1c01d29)
2012-11-26 12:25:38 -05:00
e3f6a6a752
api: Remove hardcoded API keys from examples.
...
(imported from commit 64bf22298452a436d0aec796973a77f10d5e0bf9)
2012-11-26 12:25:38 -05:00
7314c12527
api: Read the API key from ~/.humbug-api-key by default.
...
(imported from commit 309469a0955969eafd78fbdf89d9651cb530010d)
2012-11-26 12:25:38 -05:00
6ac25fb9ba
zephyr_mirror: Don't break long words/urls when mirroring.
...
(imported from commit df2b6a6969fc54016eeee4996df317e2843844c8)
2012-11-21 15:55:18 -05:00
f72427d83c
zephyr_mirror: Fix running --sync-subscriptions repeatedly.
...
(imported from commit 15bead790590c91cd4a1d794799f94341d31c181)
2012-11-20 17:17:40 -05:00
f1fc271258
nagios: Increase the timeout on MIT zephyr forwarding functioning.
...
Sometimes messages are delayed passed the allowed 10 seconds. We may
eventually decide that that amount of latency is unacceptable, but the
latency is not what this script is supposed to be checking.
(imported from commit d83a6a83d60e9eac13b3b87fb31de7f9881acabf)
2012-11-20 15:32:06 -05:00
5587b502a2
zephyr_mirror: Add exception handling around all main loop code.
...
(imported from commit 5dc46decf882302a6c58f3bb788362f82f293b35)
2012-11-19 12:44:12 -05:00
ab9858ba65
zephyr_mirror: Use minimal wrapper script that is hard to accidentally break.
...
(imported from commit 45967b8b806df616b0480b217dd5db82c5a2cabd)
2012-11-19 12:44:06 -05:00
c0f649559c
zephyr_mirror: Move parse_args into its own function.
...
(imported from commit b56ca627e3d34b8dcefead2cfecfa77bb428b0ec)
2012-11-19 12:30:15 -05:00
235654e022
zephyr_mirror: Get rid of extra_mirror code -- it is no longer used.
...
(imported from commit ab2a9cfcfa6a8c7615523ebb16864e9a4db05f3a)
2012-11-19 12:11:07 -05:00
2045bdae77
zephyr_mirror: Improve robustness when killing child processes.
...
(imported from commit f212b87ffb1fc814d289c14d3854fb1f2f6e2cf3)
2012-11-19 11:46:05 -05:00
e9bc9ac3c2
zephyr_mirror: Fix restart_stamp functionality.
...
(imported from commit a79db110d3372ab88fabfc0b7013c5fd3ad07f8e)
2012-11-19 11:20:33 -05:00
eac3ae2ed1
zephyr_mirror: Improve zwrite error handling.
...
(imported from commit a6b5b9c0c9bd6ffde0b2b783c87528bdcae03566)
2012-11-16 17:08:51 -05:00
6026c80de6
API: Add unsubscribe function.
...
(imported from commit 6dc55e9030770500770ce3921a4e77499d64f2d6)
2012-11-16 17:06:01 -05:00
f5ccde78e3
[manual] Use "subscriptions" for {add,list}_subscriptions consistently.
...
This change requires a zephyr_mirror deployment when it is pushed to prod.
(imported from commit a31d6efd2db4d4617c7c6b00326be3f07c7263da)
2012-11-16 17:06:01 -05:00
afc62a8e16
[manual] Uniformize the subscriptions API method names.
...
This requires a zephyr_mirror deployment when it is pushed to prod.
(imported from commit 6543441fb264b518f8705d7989d068a1d50ce5d6)
2012-11-16 17:06:01 -05:00
b48671ae43
nagios: print more information on check-mirroring errors.
...
(imported from commit 6ea149d939ae309745ce8c5b2ab7ae2237007d97)
2012-11-16 11:25:22 -05:00
18a6014fa3
send-message: Add sending to streams and some code cleanup.
...
(imported from commit 806b17bd959f643a7422f9fcf5d24d6f945a18c2)
2012-11-15 15:31:48 -05:00
411a7f6b4f
[manual] send_message: Rename recipient/stream fields to 'to'.
...
This commit changes APIs and requires and update of all zephyr
mirroring bots to deploy properly.
(imported from commit 2672d2d07269379f7a865644aaeb6796d54183e1)
2012-11-15 15:30:06 -05:00
9e7802cd51
check-mirroring: Use consistent ordering for keys.
...
(imported from commit 7deaf55ccb4b47a11dbd30c9239893a42eb39e00)
2012-11-15 14:56:10 -05:00
a11a4dceb7
zephyr_mirror: Improve error handling getting public streams list.
...
(imported from commit 06f9c00f19056bfe7db3c02955e3a29ad270de6e)
2012-11-15 14:51:05 -05:00
56c6d93e8a
zephyr_mirror: Send Humbugs in child processes.
...
This improves the throughput of mirroring a large number of zephyrs in
a row from about 1.5/second to about 9/second, which is basically
satisfactory.
(imported from commit 5f72680d6290eaa02ef8ced5b3792fb3efc1db41)
2012-11-15 14:50:59 -05:00
11c7d0e841
Ask user to reload Humbug after running sync-subscriptions.
...
(imported from commit d374d5b8eefd4240b8b11f83080fcdf2ff3cadaf)
2012-11-15 10:40:10 -05:00
c447a7ba94
If a zwrite fails, retry unauthenticated and warn the user.
...
(imported from commit a8a516d59f4af72ed2eaebb6bfa65e2f359fc2a6)
2012-11-14 17:38:42 -05:00
3cf6b27ec4
Make send-message API example program more flexible
...
(imported from commit e1fa90169500f84b088f1697c31641c24ed93bc8)
2012-11-14 17:35:38 -05:00
50b8195bf3
api: Fix type error updating the failures count.
...
(imported from commit 5733f252f676549ff817df3a7560274ecec0cd2a)
2012-11-14 15:03:49 -05:00
aecb8bd849
zephyr_mirror: Don't use the /mit symlink for use on systems without automounters.
...
(imported from commit b376e6da6c971d887f2d75194a8ed0d3adaddfea)
2012-11-14 15:03:49 -05:00
c00628d149
zephyr_mirror: Don't try subscribing to an empty stream list.
...
(imported from commit c8155459b768dd1fc192246c1f584abb30a7bc02)
2012-11-14 15:03:49 -05:00
2e11624b2f
zephyr_mirror: Don't send fullname data to server.
...
(imported from commit 743d0109f422fea5c4642e83448cd179f4e60827)
2012-11-14 15:03:49 -05:00
268492c45d
gcal-bot: Get API key from ~/.humbug-api-key
...
(imported from commit 63d63ad0d299c300d28de61069ada8ea2c8eb839)
2012-11-14 10:38:36 -05:00
0d5c67c792
gcal-bot: Fix sys.path for new location under api/bots/
...
(imported from commit 9f5b90d5e0cd2134f2f0210980ad49fd8062a9c4)
2012-11-14 10:38:36 -05:00
0a930cd87a
zephyr_mirror: forward messages in test mode if the site is not the default one
...
(imported from commit 7d504a6d555866f12f1657b522b4e6eb380c64a3)
2012-11-13 20:22:37 -05:00
195bdb07c9
Unify huddles and personals into private messages on the send path
...
Personals are now just private messages between two people (which
sometimes manifests as a private message with one recipient). The
new message type on the send path is 'private'. Note that the receive
path still has 'personal' and 'huddle' message types.
(imported from commit 97a438ef5c0b3db4eb3e6db674ea38a081265dd3)
2012-11-13 15:40:53 -05:00
33c23c0113
Use a JSON array for recipients in send_message
...
(imported from commit e2184f92b708cc2e8ef3e9ae79ee4241c0aa12a1)
2012-11-13 15:40:53 -05:00
375062d14c
zephyr_mirror: Extend logging on subscribe failures.
...
(imported from commit ce54549a74935bb95cb68dd1e6db2a3eec6166c5)
2012-11-13 12:47:59 -05:00
04d18d2461
zephyr_mirror: Tweak linewrapping algorithm.
...
(imported from commit 1165a7680251f6bb365f44a60612fffe0fbbae42)
2012-11-13 12:43:37 -05:00
832688d521
zephyr_mirror: Use python logging module.
...
(imported from commit 3cd9506d46eb56fbd5a48c789160a45846332633)
2012-11-13 12:43:14 -05:00
ef99a53465
api: Generically convert non-string values to JSON in python bindings
...
(imported from commit 820bc20d469832b5785b434726c81e59a3f67ea5)
2012-11-09 18:08:31 -05:00
7cafd7c6d0
zephyr_mirror: Support whitespace-only zephyr instances.
...
Previously messages sent to zephyr instance " " wouldn't be forwarded
zephyr=>humbug because the target instance is the null instance; I
learned this when we got a few tracebacks in the zephyr_mirror log, so
clearly this does happen.
(imported from commit 08bd7470e75ac6af24ac83696b6cf68d70654664)
2012-11-09 17:32:15 -05:00
Tim Abbott
Zev Benjamin
Keegan McAllister
Jessica McKellar
Waseem Daher