Previously, we would return a JSONDecodeError to the user in the event
that the server returned a 500 error (or other non-JSON content).
(imported from commit 1624dfec6ac65d34216f4de91e33116a54e414fa)
urlparse.urljoin(base_url, url) will drop any path inside base_url if
either the url has a leading "/" or base_url doesn't have a trailing
"/". So adjust our API bindings to ensure that doesn't happen.
(imported from commit c080ee8c04b89127888609da28afc8b388af1911)
This must be deployed after we update our running nginx configuration
to serve api.humbughq.com.
(imported from commit b5c34ebdd595f55eecd6dca6a18a37f105107bd5)
Since in the future we might want requests to add subscriptions to
include things like colors, in_home_view, etc., we're changing the
data format for the add_subscriptions API call to pass each stream as
a dictionary, giving a convenient place to put any added options.
The manual step required here is updating the API version in AFS
available for use with the zephyr_mirror.py system.
(imported from commit 364960cca582a0658f0d334668822045c001b92c)
I believe this should require no special work on deploy, since some
grepping of logs suggests we are not currently using this API query.
(imported from commit 240086f900c6680cbc90bf6a2f334a9e1f172df6)
The --site= option is really only for internal developer use, so I
don't think we gain anything from being strict about it.
(And it doesn't help that the error message one gets pre-this-patch is
super confusing). Fixes Trac #937.
(imported from commit 8d699982aa6830f9eae2bccd6d0c7a1e0e53dd56)
Give better examples, and rewrite options parsing to be more consistent across examples.
Make it more obvious that you can use "--user" and "--api-key" with our python examples.
This bumps our python bindings to v0.1.9
(imported from commit 297468088f864b7d585e567dc45523ea681f1856)
Previously our receive API bindings were broken in our API tarballs
because we weren't including the receive API bindings which they used.
This requires our deploying the built API tarball to the prod server
when we deploy it so that the link on /api isn't broken.
(imported from commit 14ecaab34556f4e29c72f4f567d8af73c89d6297)
Currently the interface for editing messages is limited to a
command-line API tool; it's great for testing with e.g.:
./api/examples/edit-message --message=348135 --content="test $(date +%s)" --site=http://localhost:9991 --subject="test"
The next commit will add a user interface for actually doing the editing.
(imported from commit bdd408cec2946f31c2292e44f724f96ed5938791)
This post-commit hook depends on pysvn. After a transaction is completed,
a Humbug is sent to a configurable stream with the repo modified, actor,
and commit message.
(imported from commit 75cab82d5fe993ea7c4c05be07a7b61e770aff81)
Now that we're distributing these examples, we shouldn't be promoting
the use of the "--site" option.
(imported from commit 01ded4a851dc799fa6c7e902e937c4331ed92bf8)
The previous version of our code only worked with python-requests <
1.0 (as is the case on our servers), the new version will work with
any python-requests new enough to have a .json at all.
(imported from commit 77ffe3e0d890fe88776c313e0e3289aee1bb30ea)
Previously we sent it always as "data", which caused problems for GET
requests where there is no request body.
(imported from commit 20084d1da1b8228cc484536ca4d6f77f547a9d78)
We also switch the Python client to use a client string of "API: Python"
to allow us to determine more easily which bindings our users are using.
(imported from commit 7216c3d150b371835f14d1bc8d81979a92e44925)
After this commit, we built an API tarball and sent it to
CUSTOMER4, and then promptly reverted the commit so that
we could continue as we had been before.
(imported from commit 662519a79edd508e7c115b451a7ec6fbdf1fc0a4)
To incorporate the site parsing fix from a couple weeks ago.
Before deploying this to prod we need to run build-api-tarball and
deploy the code to humbughq.com as for usual API releases.
(imported from commit f6711f5cc07d174c30866029032a595ecee785a3)
At Ksplice we used /usr/bin/python because we shipped dependencies as Debian /
Red Hat packages, which would be installed against the system Python. We were
also very careful to use only Python 2.3 features so that even old system
Python would still work.
None of that is true at Humbug. We expect users to install dependencies
themselves, so it's more likely that the Python in $PATH is correct. On OS X
in particular, it's common to have five broken Python installs and there's no
expectation that /usr/bin/python is the right one.
The files which aren't marked executable are not interesting to run as scripts,
so we just remove the line there. (In general it's common to have libraries
that can also be executed, to run test cases or whatever, but that's not the
case here.)
(imported from commit 437d4aee2c6e66601ad3334eefd50749cce2eca6)
We need to run build-api-tarball and release it on prod when pushing
this commit to prod.
(imported from commit 09e86500d2d208b1972c87444b4c2d56faafc8e6)
Before pushing this to prod, we need to build the 0.1.2 API tarball
and deploy it to the appropriate place on our servers.
(imported from commit ec1a07b3cc2a3e360dac32823ff7cd9de9de1da2)
Previously we were installing data files to e.g. /usr/local/example
and /usr/local/integrations, which is really not OK.
(imported from commit 0efb50412f93efabfe55443d5cac57a8ebb9fe06)
This works much better for working with staging, since rather than
needing to tell each individual tool that you're using staging, you
just specify that along with your API (which at the moment implies
whether you should be using staging or prod).
(imported from commit c1de8e72c24f35ef2160bce5339a5f03c6e1da95)
It doesn't matter for this script, but I worry the code will be
copy-pasted into other new plugins.
(imported from commit 0fe5280af5aa05a7efc3d146f1570f9a72c62027)
This should fix the symptoms of the problem we've been having where a
few API clients using the MIT Zephyr mirroring system sometimes seem
to end up with a too-old value of last.
(imported from commit 9f2426fa6a7e8365e8d3443bfd2cce3238cc9510)
I would prefer to be testing the attribute itself rather than the
version, but it's not easy to access without an actual request object,
and I'd prefer to compute this once-and-for-all on startup, rather
than on each request, since the latter just seems fragile.
(imported from commit dd74cadb1b2359faeb3e1b482faeee4003dfad77)
Bots are not part of what we distribute, so put them in the repo root.
We also updated some of the bots to use relative path names.
(imported from commit 0471d863450712fd0cdb651f39f32e9041df52ba)
My previous commit (fbdc092029bbafea716e27fbb99fec58a6f24392)
incorrectly specified that you must have a version of python-requests
greater than 0.12.1, when it should be a >= relation since 0.12.1 is
sufficient.
(imported from commit 9f716af6dfe0ce17d982fc22d507f144e9543bec)
Previously, if users of our code put the API folder in their pyshared
they would have to import it as "humbug.humbug". By moving Humbug's API
into a directory named "humbug" and moving the API into __init__, you
can just "import humbug".
(imported from commit 1d2654ae57f8ecbbfe76559de267ec4889708ee8)
The old-style api.humbug is deprecated, so we rewrite the path for things
to work correctly.
(imported from commit 360282b70fab1ad9d9d517157fc4fe06f9acd6a2)
This was causing about 10% of the time, personals being forwarded by
tabbott/extra@ATHENA.MIT.EDU to get this error:
zwrite: Field too long for buffer while sending notice to
tabbott/extra@ATHENA.MIT.EDU
We don't fully understand the cause of the problem, but this fixes it.
(imported from commit 22c39a1061cde9ad6973ef07aee7227623a3d47d)
Sometimes the very first message we send isn't received by
python-zephyr; this could be because of our growing a 17-message queue
of zephyrs to service, so let's not do that.
(imported from commit 281bf1807442b6335b05c803b1a47e0a162bef4e)
This ensures that Humbug receives the verbatim text of the Nagios
body, rather than a slightly modified version. Tested by running
manually.
(imported from commit 7e0eea0b230fd8c5552860acfb7372099598f036)
The refactoring that we did a couple weeks ago to make the zephyr
mirror script restart itself automatically (by splitting it into
zephyr_mirror.py and zephyr_mirror_backend.py) had a poor interaction
with our code for killing old zephyr_mirror processes (to prevent
double-mirroring). If you manually ran two copies of the outer
mirroring script (zephyr_mirror.py), then the second one would on
startup kill the first one's zephyr_mirror_backend.py children (for
being duplicate zephyr_mirror_backend.py processes that would result
in double-mirroring). However, importantly, it did not kill the first
mirroring script's zephyr_mirror.py parent process, so the first
mirroring script would then proceed to startup up new children. The
process then repeats, with the two scripts' roles reversed.
This issue didn't affect the sharded mirroring case, where I had been
doing the testing of the kill code with that refactoring, because we
don't have a version of the outer zephyr_mirror.py loop for that
situation (a consequence of it being hard to restart the threads
properly with the run_parallel API that we're using to spawn all the
children).
(imported from commit d4886ac77312a6b0ebd0d612f6fb084970bf23a2)
This is probably a lot more annoying to use, in that e.g. there are
separate log files for the two directions of the mirror, but we
haven't used these logs for much, so whatever.
(imported from commit d3bc407d90099214d242526c01cd3d3cd9d9d9bd)
Previously all that we logged was the sender, which turns out to be a
constant for humbug=>zephyr mirroring.
(imported from commit 527a3ac4b9b815a2b4d6b63c3ad92d9d5ad99a6e)
feedback-bot and zephyr_mirror will need to be updated and restarted
when this is deployed to prod.
(imported from commit fe2b524424c174bcb1b717a851a5d3815fda3f69)
Features include:
* Not forking into two processes (shells out to zwrite to send
instead). This makes life easier since we're not doing concurrent
programming.
* Eliminated a lot of hard-to-read or unnecessary debugging output.
* Adding explanatory test suggesting the likely problem for some
common sets of received messages.
* Much less code duplication.
* Support for testing a sharded zephyr_mirror script (--sharded).
* Use of the logging module to print timestamps -- makes debugging
some issues a lot.
* Only one sleep, and for only 10 seconds, between sending the
outgoing messages and checking that they were received.
* Support for running two copies of this script at the same time, so that
running it manually doesn't screw up Nagios.
* Passed running 100 tests run in a row.
(imported from commit a3ec02ac1d1a04972e469ca30fec1790c4fb53bc)
We should be canonicalizing stream names to class names in
update_subscriptions_from_humbug, before we even decide which classes
to subscribe to; otherwise deduplication and tracking of which classes
we're already subscribed to won't work.
(imported from commit a751b6fca1022390a087516a0730ff77f13d7edf)
This causes e.g. call_on_each_message to switch to dont_block mode
after the first error.
(imported from commit b6a5a10970c987faf8017f0ddae4e0b64a513c6f)
Also fixes bugs where the retry code wouldn't work correctly if
verbose wasn't set, prints out which API query had the error, and is
more consistent about printing something to end the "..." sequences.
(imported from commit 15c1e0e4a14c5c5559b43bafe4ec268451ee04f5)
Previously, we were sending "Skipping message we got from Humbug!"
for messages we wouldn't have forwarded anyway.
(imported from commit 36df85a61336ac00e3d7913d5a417d6b42764350)
In this case, if we're configured to not forward personals, there's no
point in logging a decision not to forward one.
(imported from commit 62c37591c6a70afb6235de626b0c6a3502cbcb27)
It seems that check-mirroring was reporting a lot of spurious failures
due to it sometimes taking more than 3 seconds for the setup phase of
check-mirroring's receive path to run. So fix this:
(1) Wait a bit more than 3 seconds for the receiver to subscribe to
messages
(2) Subscribe to Humbug messages before forking (we can't do this with
zephyr.init() because python-zephyr gets totally messed up if you use
it from multiple processes with a shared initialization)
(3) Get rid of the old time.sleep(0.x) values that were intended to
make messages arrive in order -- since we're now checking that
messages correctly arrived using set(), they aren't needed.
(4) Use a single request to subscribe to both zephyr classes we need
to subscribe to (saves 1 RTT).
(imported from commit d96aef05405ce43e9a4a549de189da9a2e393875)
- The default 'default' is None
- The default 'dest' is the option name, with - replaced with _
- The default 'action' is 'store'
Not coincidentally, these defaults are correct for most of our existing code.
(imported from commit 9c6078bd778324e08e1ca214a97281a7bdce08c2)
At least the zephyr.humbughq.com case is one our users might find
useful. I imagine we'll end up hiding --site once we integrate the
zephyr and main site tornado servers.
(imported from commit a3a28492cf3f32b78eab6bbd48ba66be2f38a5f2)
Sometimes messages are delayed passed the allowed 10 seconds. We may
eventually decide that that amount of latency is unacceptable, but the
latency is not what this script is supposed to be checking.
(imported from commit d83a6a83d60e9eac13b3b87fb31de7f9881acabf)
This commit changes APIs and requires and update of all zephyr
mirroring bots to deploy properly.
(imported from commit 2672d2d07269379f7a865644aaeb6796d54183e1)
This improves the throughput of mirroring a large number of zephyrs in
a row from about 1.5/second to about 9/second, which is basically
satisfactory.
(imported from commit 5f72680d6290eaa02ef8ced5b3792fb3efc1db41)
Personals are now just private messages between two people (which
sometimes manifests as a private message with one recipient). The
new message type on the send path is 'private'. Note that the receive
path still has 'personal' and 'huddle' message types.
(imported from commit 97a438ef5c0b3db4eb3e6db674ea38a081265dd3)
Previously messages sent to zephyr instance " " wouldn't be forwarded
zephyr=>humbug because the target instance is the null instance; I
learned this when we got a few tracebacks in the zephyr_mirror log, so
clearly this does happen.
(imported from commit 08bd7470e75ac6af24ac83696b6cf68d70654664)
This is useful for trying out new versions of the forwarding code
while the mirroring bot is still running in production.
(imported from commit bcdaf91fed55ac0974b1efe31dd13ed006b6fd06)
This happens sometimes (especially when our server is restarting), and
isn't a _real_ problem -- it's much more important that we have a
completely reliable test that we can put a Nagios alert against.
(imported from commit 0add0b3dfc5447307014bbb9137366bd7141ade0)
Previously we were spending 15 seconds on linerva (and more like 2.5
minutes on the not-yet-operational zmirror.humbughq.com) to subscribe
to all of our streams.
(imported from commit c36cb1c26868f142683d9c92d4875fcd4931886e)
Steve Howell
Tim Abbott
Scott Feeney
Scott Feeney
acrefoot
Jessica McKellar
Luke Faraone
Zev Benjamin
Leo Franchi
Waseem Daher
Keegan McAllister
Reid Barton