Commit graph

1653 commits

Author SHA1 Message Date
Tim Abbott 3285d914c5 do_api_query: Reduce error handling code duplication.
Also fixes bugs where the retry code wouldn't work correctly if
verbose wasn't set, prints out which API query had the error, and is
more consistent about printing something to end the "..." sequences.

(imported from commit 15c1e0e4a14c5c5559b43bafe4ec268451ee04f5)
2012-11-29 13:57:06 -05:00
Tim Abbott 70d9281f7a api: Only retry up to 10 times on connection errors.
(imported from commit f395370120d5e7a635eec7b27e4d5ed0c48b774d)
2012-11-29 13:57:06 -05:00
Tim Abbott 8683613e76 zephyr_mirror: Add options to run class mirror without sharding for debugging.
(imported from commit 4fcc403504d93e30aca6046cba9f1fe5342faac6)
2012-11-29 13:57:06 -05:00
Tim Abbott 18bbd02074 zephyr_mirror: Don't leak zephyr_mirror processes on send_humbug failures.
(imported from commit be005018672369683d9e3bdbf144f96f58af6b6c)
2012-11-29 13:57:06 -05:00
Tim Abbott 074316105b zephyr_mirror: Replace bare excepts with 'except Exception'.
(imported from commit 857212e0e5116471e418d37845550108c0a47d2b)
2012-11-29 13:57:05 -05:00
Tim Abbott ab68503fcf zephyr_mirror: Compare zephyr subs from server after normalization.
(imported from commit eea290698e83c7d26f23ab3bb0201914417f0e09)
2012-11-28 22:26:10 -05:00
Tim Abbott c61567b67a zephyr_mirror: Add two more restricted access classes.
(imported from commit 06ff0e19c1522ade018aea66f152fdfa55c18e1d)
2012-11-28 22:26:10 -05:00
Tim Abbott a644e4bc41 get_updates: Replace "failures" with the new dont_block option.
(imported from commit 2b4ecb35c4f14b6c408323662ef6f39c6485c62d)
2012-11-28 17:41:38 -05:00
Tim Abbott 13ebd229c1 get_updates: Replace the "mirror" argument with checking "client".
(imported from commit bd84c2c7a456de8ea7ec35eaf76b5bf8647ca61a)
2012-11-28 17:41:38 -05:00
Tim Abbott 2b2398b712 zephyr_mirror: Pass an encoded stream to hashlib for sharding.
(imported from commit cc57b3956198b66c9cfc84423434a71b8200d819)
2012-11-28 15:44:35 -05:00
Tim Abbott de14210a4d zephyr_mirror: Validate zephyr subscriptions happened successfully.
(imported from commit 494895b87eb9b555175c9890c5c2046cf4ce40ac)
2012-11-28 15:44:35 -05:00
Tim Abbott 121c3cb9c8 zephyr_mirror: Fix display of personals on non-message classes.
(imported from commit 42ff030d2d78c530428dcede406d46fc71521fef)
2012-11-27 14:12:19 -05:00
Keegan McAllister 30f234d42d send-nagios-notification: Expand "\n" escape sequences that we get from Nagios
(imported from commit 3e8efbfe8bbd5a2f13a8e02d9f44442b035113ba)
2012-11-27 14:08:49 -05:00
Tim Abbott faa2470ed5 send_message: Accept subject and message arguments.
(imported from commit 93aca5e750fa74d7ee95d8a00adbf6d4e61b2fc8)
2012-11-27 14:02:24 -05:00
Tim Abbott 305a8ce569 zephyr_mirror: Run the class mirror in 16 parallel shards.
(imported from commit cb17ef999e94cc57d5b01114479728b04475c9c4)
2012-11-27 12:09:57 -05:00
Tim Abbott bfec56661e zephyr_mirror: Add a --no-forward-from-humbug option.
We'll use this to only have one of our shards forward from Humbug.

(imported from commit a6f3c2180b49c1fa826baf1ae9e7e3ba252a5bfa)
2012-11-27 12:09:57 -05:00
Tim Abbott 975c8ef5b0 zephyr_mirror: Clean up unnecessary logic in process_notice.
(imported from commit 28599789016fd1006c2de4d5ddc3cf1f10baa47f)
2012-11-27 12:09:57 -05:00
Tim Abbott dbeb107aaf zephyr_mirror: Fix the check for whether a Zephyr is a personal.
(imported from commit 609b6339fa82ecbd6435a386d24bf555e75a8120)
2012-11-27 12:09:57 -05:00
Tim Abbott b081d2fa5a zephyr_mirror: Reorder process_notice to log fewer "Skipping" messages.
Previously, we were sending "Skipping message we got from Humbug!"
for messages we wouldn't have forwarded anyway.

(imported from commit 36df85a61336ac00e3d7913d5a417d6b42764350)
2012-11-27 12:09:57 -05:00
Tim Abbott f45a8bebf2 zephyr_mirror: Don't display/log some useless "Skipping" notices.
In this case, if we're configured to not forward personals, there's no
point in logging a decision not to forward one.

(imported from commit 62c37591c6a70afb6235de626b0c6a3502cbcb27)
2012-11-27 12:09:56 -05:00
Tim Abbott 333573f06b zephyr_mirror: Simplify optparse usage.
(imported from commit d7cce4236dae6f369ea1cc4e12f89c2011718b7f)
2012-11-27 12:09:56 -05:00
Tim Abbott ebf009b232 api: Extend documentation a bit.
(imported from commit fd435286e12ffebad8476cc8005303b2d8d70fd9)
2012-11-27 12:09:56 -05:00
Tim Abbott ca34a2a02f check-mirroring: Fix some issues with startup process timing.
It seems that check-mirroring was reporting a lot of spurious failures
due to it sometimes taking more than 3 seconds for the setup phase of
check-mirroring's receive path to run.  So fix this:

(1) Wait a bit more than 3 seconds for the receiver to subscribe to
messages

(2) Subscribe to Humbug messages before forking (we can't do this with
zephyr.init() because python-zephyr gets totally messed up if you use
it from multiple processes with a shared initialization)

(3) Get rid of the old time.sleep(0.x) values that were intended to
make messages arrive in order -- since we're now checking that
messages correctly arrived using set(), they aren't needed.

(4) Use a single request to subscribe to both zephyr classes we need
to subscribe to (saves 1 RTT).

(imported from commit d96aef05405ce43e9a4a549de189da9a2e393875)
2012-11-26 17:10:15 -05:00
Tim Abbott ba33df97bb zephyr_mirror: Don't use @(@color(blue)) in zsigs.
(imported from commit e4b594e3c22c25fa4f87baaad09227396bb92a0d)
2012-11-26 16:19:09 -05:00
Tim Abbott 135f7f1b57 zephyr_mirror: Use default format to indicate what came from Humbug.
(imported from commit 5c2fdccb08d605a37d6c87c37f023c06a0a9a760)
2012-11-26 16:18:05 -05:00
Tim Abbott 111a9bc0e6 zephyr_mirror: Refactor zwrite_args handling a bit.
(imported from commit 4923c90dc546022d48f226c2a204e9d09717b563)
2012-11-26 14:58:34 -05:00
Keegan McAllister 185ab66797 examples/send-message: Remove default sender
(imported from commit 30ada075ac4f1b759fe787eef6a93275dbea584c)
2012-11-26 14:57:20 -05:00
Keegan McAllister b65ce435c4 api/examples: Simplify optparse use
- The default 'default' is None
- The default 'dest' is the option name, with - replaced with _
- The default 'action' is 'store'

Not coincidentally, these defaults are correct for most of our existing code.

(imported from commit 9c6078bd778324e08e1ca214a97281a7bdce08c2)
2012-11-26 14:57:20 -05:00
Keegan McAllister dd80275354 api/common.py: Ensure that the API key file is closed when we're done with it
(imported from commit 87a0feb52952aa76bec041d032027e7a272156fc)
2012-11-26 14:57:20 -05:00
Keegan McAllister 4a62308ab9 api/examples: Rework sys.path computation
(imported from commit 6e4c2a9b5f5ab6f6680704a1e06bd17fd95bd038)
2012-11-26 14:57:20 -05:00
Keegan McAllister 602ec5cfbf api/common.py: Reduce code duplication between API call methods
(imported from commit 4a0f1a8479c21366bf552edf98401b03bad72d9c)
2012-11-26 14:57:20 -05:00
Keegan McAllister c1cc221cb6 Use a new-style class for HumbugAPI
(imported from commit a88366e0129dadb40b058ddd0ede64db3b6b206f)
2012-11-26 14:57:20 -05:00
Keegan McAllister 84d73db4f3 Send Nagios alerts to Humbug
Fixes #385.

(imported from commit 7dac013debd6ccff031fc4da0dd7185e198b4498)
2012-11-26 14:42:55 -05:00
Tim Abbott 1ff5440f28 Add tool to build an API tarball for distribution.
(imported from commit eeaf9f526f243cc628eede603fe1b1691c34977f)
2012-11-26 12:25:39 -05:00
Tim Abbott fcf24e30f5 check-mirroring: Add subscriptions inside IOError check.
(imported from commit d3b00ce9d91a0bcffcef58436a4b90770155fce9)
2012-11-26 12:25:39 -05:00
Tim Abbott 4903843151 api: Add an extra send_message example.
(imported from commit fe5ab550fa862f587cdf7987c5b238dffab45672)
2012-11-26 12:25:39 -05:00
Tim Abbott 03dc2b2af1 api: Don't use 127.0.0.1:8000 as an example.
At least the zephyr.humbughq.com case is one our users might find
useful.  I imagine we'll end up hiding --site once we integrate the
zephyr and main site tornado servers.

(imported from commit a3a28492cf3f32b78eab6bbd48ba66be2f38a5f2)
2012-11-26 12:25:39 -05:00
Tim Abbott 9d6ce4ee0b api: Add copyright and MIT license notices to distributed tools.
(imported from commit 9f93050f988549dc363f5a7c798c8a9fa1c01d29)
2012-11-26 12:25:38 -05:00
Tim Abbott c5a140303c api: Remove hardcoded API keys from examples.
(imported from commit 64bf22298452a436d0aec796973a77f10d5e0bf9)
2012-11-26 12:25:38 -05:00
Tim Abbott afae2a016e api: Read the API key from ~/.humbug-api-key by default.
(imported from commit 309469a0955969eafd78fbdf89d9651cb530010d)
2012-11-26 12:25:38 -05:00
Tim Abbott 27d597d686 zephyr_mirror: Don't break long words/urls when mirroring.
(imported from commit df2b6a6969fc54016eeee4996df317e2843844c8)
2012-11-21 15:55:18 -05:00
Tim Abbott 27b02176bc zephyr_mirror: Fix running --sync-subscriptions repeatedly.
(imported from commit 15bead790590c91cd4a1d794799f94341d31c181)
2012-11-20 17:17:40 -05:00
Jessica McKellar 34abe48f62 nagios: Increase the timeout on MIT zephyr forwarding functioning.
Sometimes messages are delayed passed the allowed 10 seconds. We may
eventually decide that that amount of latency is unacceptable, but the
latency is not what this script is supposed to be checking.

(imported from commit d83a6a83d60e9eac13b3b87fb31de7f9881acabf)
2012-11-20 15:32:06 -05:00
Tim Abbott 244ff8459e zephyr_mirror: Add exception handling around all main loop code.
(imported from commit 5dc46decf882302a6c58f3bb788362f82f293b35)
2012-11-19 12:44:12 -05:00
Tim Abbott 36280c2769 zephyr_mirror: Use minimal wrapper script that is hard to accidentally break.
(imported from commit 45967b8b806df616b0480b217dd5db82c5a2cabd)
2012-11-19 12:44:06 -05:00
Tim Abbott 70e7c9fd14 zephyr_mirror: Move parse_args into its own function.
(imported from commit b56ca627e3d34b8dcefead2cfecfa77bb428b0ec)
2012-11-19 12:30:15 -05:00
Tim Abbott 99f90762d3 zephyr_mirror: Get rid of extra_mirror code -- it is no longer used.
(imported from commit ab2a9cfcfa6a8c7615523ebb16864e9a4db05f3a)
2012-11-19 12:11:07 -05:00
Tim Abbott ff80f7df71 zephyr_mirror: Improve robustness when killing child processes.
(imported from commit f212b87ffb1fc814d289c14d3854fb1f2f6e2cf3)
2012-11-19 11:46:05 -05:00
Tim Abbott ce279d48ae zephyr_mirror: Fix restart_stamp functionality.
(imported from commit a79db110d3372ab88fabfc0b7013c5fd3ad07f8e)
2012-11-19 11:20:33 -05:00
Tim Abbott 5411e5c360 zephyr_mirror: Improve zwrite error handling.
(imported from commit a6b5b9c0c9bd6ffde0b2b783c87528bdcae03566)
2012-11-16 17:08:51 -05:00