Commit graph

96 commits

Author SHA1 Message Date
Luke Faraone d8f13837d9 Move files around in the API.
Bots are not part of what we distribute, so put them in the repo root.

We also updated some of the bots to use relative path names.

(imported from commit 0471d863450712fd0cdb651f39f32e9041df52ba)
2013-02-01 15:52:28 -05:00
Luke Faraone f2f4a2f8bd Move the API into a subdirectory for ease of imports.
Previously, if users of our code put the API folder in their pyshared
they would have to import it as "humbug.humbug". By moving Humbug's API
into a directory named "humbug" and moving the API into __init__, you
can just "import humbug".

(imported from commit 1d2654ae57f8ecbbfe76559de267ec4889708ee8)
2013-01-16 16:55:22 -05:00
Tim Abbott 8683613e76 zephyr_mirror: Add options to run class mirror without sharding for debugging.
(imported from commit 4fcc403504d93e30aca6046cba9f1fe5342faac6)
2012-11-29 13:57:06 -05:00
Tim Abbott 305a8ce569 zephyr_mirror: Run the class mirror in 16 parallel shards.
(imported from commit cb17ef999e94cc57d5b01114479728b04475c9c4)
2012-11-27 12:09:57 -05:00
Tim Abbott 9d6ce4ee0b api: Add copyright and MIT license notices to distributed tools.
(imported from commit 9f93050f988549dc363f5a7c798c8a9fa1c01d29)
2012-11-26 12:25:38 -05:00
Tim Abbott 27b02176bc zephyr_mirror: Fix running --sync-subscriptions repeatedly.
(imported from commit 15bead790590c91cd4a1d794799f94341d31c181)
2012-11-20 17:17:40 -05:00
Tim Abbott 36280c2769 zephyr_mirror: Use minimal wrapper script that is hard to accidentally break.
(imported from commit 45967b8b806df616b0480b217dd5db82c5a2cabd)
2012-11-19 12:44:06 -05:00
Tim Abbott 70e7c9fd14 zephyr_mirror: Move parse_args into its own function.
(imported from commit b56ca627e3d34b8dcefead2cfecfa77bb428b0ec)
2012-11-19 12:30:15 -05:00
Tim Abbott 99f90762d3 zephyr_mirror: Get rid of extra_mirror code -- it is no longer used.
(imported from commit ab2a9cfcfa6a8c7615523ebb16864e9a4db05f3a)
2012-11-19 12:11:07 -05:00
Tim Abbott ff80f7df71 zephyr_mirror: Improve robustness when killing child processes.
(imported from commit f212b87ffb1fc814d289c14d3854fb1f2f6e2cf3)
2012-11-19 11:46:05 -05:00
Tim Abbott ce279d48ae zephyr_mirror: Fix restart_stamp functionality.
(imported from commit a79db110d3372ab88fabfc0b7013c5fd3ad07f8e)
2012-11-19 11:20:33 -05:00
Tim Abbott 5411e5c360 zephyr_mirror: Improve zwrite error handling.
(imported from commit a6b5b9c0c9bd6ffde0b2b783c87528bdcae03566)
2012-11-16 17:08:51 -05:00
Tim Abbott 584743eca8 [manual] Uniformize the subscriptions API method names.
This requires a zephyr_mirror deployment when it is pushed to prod.

(imported from commit 6543441fb264b518f8705d7989d068a1d50ce5d6)
2012-11-16 17:06:01 -05:00
Tim Abbott 6727e9dbd9 [manual] send_message: Rename recipient/stream fields to 'to'.
This commit changes APIs and requires and update of all zephyr
mirroring bots to deploy properly.

(imported from commit 2672d2d07269379f7a865644aaeb6796d54183e1)
2012-11-15 15:30:06 -05:00
Tim Abbott 4ca1572283 zephyr_mirror: Improve error handling getting public streams list.
(imported from commit 06f9c00f19056bfe7db3c02955e3a29ad270de6e)
2012-11-15 14:51:05 -05:00
Tim Abbott c751cc97db zephyr_mirror: Send Humbugs in child processes.
This improves the throughput of mirroring a large number of zephyrs in
a row from about 1.5/second to about 9/second, which is basically
satisfactory.

(imported from commit 5f72680d6290eaa02ef8ced5b3792fb3efc1db41)
2012-11-15 14:50:59 -05:00
Waseem Daher ca351f3ddc Ask user to reload Humbug after running sync-subscriptions.
(imported from commit d374d5b8eefd4240b8b11f83080fcdf2ff3cadaf)
2012-11-15 10:40:10 -05:00
Jessica McKellar c571c997b3 If a zwrite fails, retry unauthenticated and warn the user.
(imported from commit a8a516d59f4af72ed2eaebb6bfa65e2f359fc2a6)
2012-11-14 17:38:42 -05:00
Tim Abbott 7514390190 zephyr_mirror: Don't use the /mit symlink for use on systems without automounters.
(imported from commit b376e6da6c971d887f2d75194a8ed0d3adaddfea)
2012-11-14 15:03:49 -05:00
Tim Abbott e182bfcb98 zephyr_mirror: Don't try subscribing to an empty stream list.
(imported from commit c8155459b768dd1fc192246c1f584abb30a7bc02)
2012-11-14 15:03:49 -05:00
Tim Abbott 1a5f0648c5 zephyr_mirror: Don't send fullname data to server.
(imported from commit 743d0109f422fea5c4642e83448cd179f4e60827)
2012-11-14 15:03:49 -05:00
Zev Benjamin 1a4f97a874 zephyr_mirror: forward messages in test mode if the site is not the default one
(imported from commit 7d504a6d555866f12f1657b522b4e6eb380c64a3)
2012-11-13 20:22:37 -05:00
Zev Benjamin 73daa73887 Unify huddles and personals into private messages on the send path
Personals are now just private messages between two people (which
sometimes manifests as a private message with one recipient).  The
new message type on the send path is 'private'.  Note that the receive
path still has 'personal' and 'huddle' message types.

(imported from commit 97a438ef5c0b3db4eb3e6db674ea38a081265dd3)
2012-11-13 15:40:53 -05:00
Zev Benjamin 8a76560970 Use a JSON array for recipients in send_message
(imported from commit e2184f92b708cc2e8ef3e9ae79ee4241c0aa12a1)
2012-11-13 15:40:53 -05:00
Tim Abbott 4c2d56c10b zephyr_mirror: Extend logging on subscribe failures.
(imported from commit ce54549a74935bb95cb68dd1e6db2a3eec6166c5)
2012-11-13 12:47:59 -05:00
Tim Abbott dd07236eec zephyr_mirror: Tweak linewrapping algorithm.
(imported from commit 1165a7680251f6bb365f44a60612fffe0fbbae42)
2012-11-13 12:43:37 -05:00
Tim Abbott defaeb935e zephyr_mirror: Use python logging module.
(imported from commit 3cd9506d46eb56fbd5a48c789160a45846332633)
2012-11-13 12:43:14 -05:00
Tim Abbott c38587191e zephyr_mirror: Support whitespace-only zephyr instances.
Previously messages sent to zephyr instance " " wouldn't be forwarded
zephyr=>humbug because the target instance is the null instance; I
learned this when we got a few tracebacks in the zephyr_mirror log, so
clearly this does happen.

(imported from commit 08bd7470e75ac6af24ac83696b6cf68d70654664)
2012-11-09 17:32:15 -05:00
Tim Abbott 2735c8410e zephyr_mirror: Catch IOError from zephyr.init().
(imported from commit d657fede53ee3754e76edf66b10bef09ad1f9577)
2012-11-09 16:56:14 -05:00
Tim Abbott 35fbf350ae Fix sending to cross-realm Zephyr users.
(imported from commit 1eebf286e149aa1242d6d26096003eb696fb18f2)
2012-11-09 16:56:13 -05:00
Tim Abbott c3c3045e3d zephyr_mirror: Forward messages sent to the empty instance more clearly.
(imported from commit 1fb5e4f8ce9941b3ca89688821d325d7ae49aee5)
2012-11-08 17:48:15 -05:00
Tim Abbott 8d27bcbad7 zephyr_mirror: Add a test mode that just prints messages.
This is useful for trying out new versions of the forwarding code
while the mirroring bot is still running in production.

(imported from commit bcdaf91fed55ac0974b1efe31dd13ed006b6fd06)
2012-11-08 17:48:15 -05:00
Tim Abbott 1dba0c0344 zephyr_mirror: Support subscribing to non-ascii Zephyr classes.
(imported from commit 85437f09c4738df11208e623db566a9409e501a3)
2012-11-07 16:42:06 -05:00
Keegan McAllister 3db339da35 zephyr_mirror: Don't open /dev/null to disable logging
(imported from commit 804e627d6d70bbf2ae1ee2f467b79ed0582ced0f)
2012-11-07 16:36:40 -05:00
Keegan McAllister 6329bb0a46 zephyr_mirror: Always pass a tuple to string %
(imported from commit 64a59e23c3aa3fceb350ed348812258c90ff09da)
2012-11-07 16:36:36 -05:00
Tim Abbott d3066a46d3 The empty string shouldn't be a valid stream name.
(imported from commit 1fa878d7d51d3c3444ac75edf08b32f886683964)
2012-11-07 15:42:32 -05:00
Tim Abbott 1c1519109a zephyr_mirror: Use zephyr's bulk subscription functionality.
Previously we were spending 15 seconds on linerva (and more like 2.5
minutes on the not-yet-operational zmirror.humbughq.com) to subscribe
to all of our streams.

(imported from commit c36cb1c26868f142683d9c92d4875fcd4931886e)
2012-11-07 15:37:55 -05:00
Tim Abbott 43b9510abd zephyr_mirror: Eliminate the confusing "subs" global.
zephyr.Subscriptions() is a singleton class, intended to be accessed
as it is used.

(imported from commit f76c1b1129adb19ba96ed17f441492edf859a748)
2012-11-07 15:37:55 -05:00
Tim Abbott 4b2d45e9a9 zephyr_mirror: Fix missing api key file error message.
(imported from commit 3efe7fb8b2d593418c37dbab9389a8df9c27d003)
2012-11-07 15:37:55 -05:00
Tim Abbott c945d60d16 zephyr_mirror: Make the logging options require the log path as an argument.
This is for supporting running the mirroring bot on a server without
AFS.

(imported from commit 400bf32f17aca984fa40c3b1222893214497eaa0)
2012-11-07 15:37:55 -05:00
Tim Abbott 6caf52867f zephyr_mirror: Use os.path.join for constructing the root_path.
(imported from commit d894ab68430e42233a7aa6b289f01e790f8c8586)
2012-11-07 15:37:46 -05:00
Tim Abbott 1e50224ca2 zephyr_mirror: Add option to change the root path.
This is relevant for running not out of AFS on Linerva.

(imported from commit 28549830c99ad5a52349fbcc9e53dcb3c5ef25a1)
2012-11-07 15:35:26 -05:00
Jessica McKellar c29eede9d0 For now, allow all characters in stream names.
We've had multiple requests from MIT zephyr users to allow
non-alphanumeric stream names, and we haven't decided what we want to
allow, so for now allow everything.

Note that the web client and mirror script limit stream names to 30
characters, which is our database limit.

(imported from commit 2acb5ee04e5ee7c40031ac831e12d09d04bbb2e6)
2012-11-07 15:12:03 -05:00
Tim Abbott 0489154cd7 zephyr_mirror: don't crash when the zephyr server returns SERVNAK errors.
Zephyr has this great property where a small fraction of the time, the
Zephyr server rejects your attempt to subscribe to something for no
particular reason, returning a "SERVNAK" error.

(imported from commit 6d5ed033d46d77a5b02539a816453724740f8fb0)
2012-11-07 13:56:21 -05:00
Tim Abbott 66640aefc6 zephyr_mirror: Remove remaining AFS reads when restarting automatically.
This does introduce a small security issue, in that a shell with
expired AFS tokens on the machine running the mirroring script will be
able to read the Humbug API key using /proc/pid/environ.  I think this
is fine -- you can steal the API key from a running process using
ptrace anyway.

(imported from commit c6fdb798294fb32d640823b409f3e46274ca01f4)
2012-11-06 15:10:11 -05:00
Tim Abbott 45057fbf5d zephyr_mirror: Zev's proposed fix to latest linewrapping problem.
(imported from commit 90fc4774e10255d0237a37b8af753059a66efb2d)
2012-11-06 11:31:37 -05:00
Tim Abbott ff85226fee zephyr_mirror: Add support for importing as a library.
(imported from commit 91dc7e8fae89ba8ade8dd98b747a25b46fca13e7)
2012-11-06 11:31:37 -05:00
Tim Abbott a728f7936d zephyr_mirror: Speed up retrying the restart process.
(imported from commit 7f7a2aaa8a7f872180400f57ad09c065ea2ababa)
2012-11-05 18:39:02 -05:00
Tim Abbott a70fc5c5c1 zephyr_mirror: Also linewrap with extremely long strings.
(imported from commit cffa80f283fdac341942beb56ada9013cb8f7c39)
2012-11-05 18:39:02 -05:00
Tim Abbott 42e94b2684 zephyr_mirror: Only strip trailing whitespace.
(imported from commit f349778200e0c2fb410e5e2644c82ba4105f3634)
2012-11-05 18:39:02 -05:00