Commit graph

128 commits

Author SHA1 Message Date
Tim Abbott 97be689411 zephyr_mirror: Auto-detect Python version for pyzephyr path.
(imported from commit e2aca3a08f8aa12b43d71acffcc0e2249feb7ca7)
2013-08-20 15:27:23 -04:00
Tim Abbott 264651c7b4 zephyr_mirror: De-zcrypt messages for which we have the key.
For now we only support the AES encryption type since the DES one is
probably not used anymore.

(imported from commit 222606db9f704917e74159e7d07a110187a236e6)
2013-08-20 15:02:07 -04:00
Tim Abbott 7f626d2bba zephyr_mirror: Handle syncing subs with invite-only streams.
(imported from commit ec8d0c0d13ff0a81fa4a5e1225e94e6dad997356)
2013-08-20 15:02:04 -04:00
Tim Abbott 9d55822cfb zephyr_mirror: Add option to control path to streams file.
This makes it easy to test the class mirror on just a single test
stream.

(imported from commit 3decb1ced54507683f1c0634df14f73e693104ef)
2013-08-20 14:57:13 -04:00
Tim Abbott 084847b0d7 Rename the 'humbug' API module to 'zulip'.
(imported from commit b3a3d7c05459cbb0110cd0fbe2197d779f3a6264)
2013-08-08 10:22:31 -04:00
Tim Abbott 4741258ca4 Change humbug => zulip in zephyr_mirror_backend internal variables.
(imported from commit 7cc5d78bfb5db8c9d5a44ed862da304d9e44ff0e)
2013-08-07 10:00:08 -04:00
Tim Abbott 84aa7d4230 Change Humbug => Zulip in zephyr_mirror states.
(imported from commit 67542cd2ba93ff4b03c99b59645b7d22ed937337)
2013-08-07 10:00:07 -04:00
Tim Abbott 4cd4c6897b Change Humbug => Zulip in text/comments.
(imported from commit 2f9d73431ae40e1b9e9e11bc2f4f62f566ae758a)
2013-08-07 10:00:07 -04:00
Tim Abbott e0c77efb3f Escape @ when mirroring content back to Zephyr.
(imported from commit 0ab58b5e8a30c66fb6aeaccaebbe498092de1cd5)
2013-08-01 14:01:58 -04:00
Tim Abbott 22e5a15bc4 [manual] Use api.zulip.com to access the Humbug API.
This needs to be deployed after the nginx configuration is deployed.

(imported from commit 167a3d9d27595d40883bb400ebcc44c4ad9b6a2f)
2013-07-25 17:27:46 -04:00
Tim Abbott d063efa8a0 More miscellaneous humbughq.com=>zulip.com updates.
(imported from commit 9a3dbc4268f251c2c0b20d8b6a7e1dc672f1b385)
2013-07-25 17:27:45 -04:00
Tim Abbott c9fab96e5d zephyr_mirror: Use Humbug full name as zsig.
(imported from commit 8cd24c4ff6c265084764ab367b46e4cb64e8c796)
2013-07-18 13:58:16 -04:00
Jessica McKellar e64d390c99 zephyr mirror backend: Switch from Humbug to Zulip in user-visible text.
(imported from commit cc4da47373362babb2f5f57493831946b29d32cf)
2013-07-15 17:49:00 -04:00
Tim Abbott 6b67e42d1a zephyr_mirror: you no longer need to reload Humbug after subbing.
(imported from commit 23cb4e5c4b3300953e58c029150d4a980bac626e)
2013-06-28 15:57:29 -04:00
Tim Abbott 5e2979b870 [manual] Change API URLs to be based on api.humbughq.com/api.
This must be deployed after we update our running nginx configuration
to serve api.humbughq.com.

(imported from commit b5c34ebdd595f55eecd6dca6a18a37f105107bd5)
2013-06-28 15:57:27 -04:00
Tim Abbott 58f4ffe088 [manual] API add_subscriptions: Change arguments to support options.
Since in the future we might want requests to add subscriptions to
include things like colors, in_home_view, etc., we're changing the
data format for the add_subscriptions API call to pass each stream as
a dictionary, giving a convenient place to put any added options.

The manual step required here is updating the API version in AFS
available for use with the zephyr_mirror.py system.

(imported from commit 364960cca582a0658f0d334668822045c001b92c)
2013-06-25 16:34:45 -04:00
Tim Abbott c849ae3470 zephyr_mirror: Fix incorrect check for already running mirror bots.
(imported from commit 76e758697ac19362e79d4a4995b576f10d229514)
2013-05-20 23:29:53 -04:00
Zev Benjamin 223ac22a13 Fix use of logging instead of logger
(imported from commit cd25471e842e089d5032b07bcd83c6fe44d03a2d)
2013-05-09 13:50:21 -04:00
Tim Abbott fcb6336642 zephyr_mirror: Use select to determine whether new messages are available.
(imported from commit a91f45aa9f3913f38285c050e574bf562b2af2df)
2013-05-09 10:35:49 -04:00
Tim Abbott 5a0e86c6d4 zephyr_mirror: Use json if simplejson is not available.
(imported from commit 4b5f6ff71067cd2697c864c715afbb3b9dd19302)
2013-05-09 10:35:47 -04:00
Tim Abbott 5af166e063 zephyr_mirror: Limit logging of errors when checking restart stamps.
(imported from commit 194b30d757a5f20d6d20bc8aed9154c6aece9837)
2013-05-09 10:35:47 -04:00
Keegan McAllister 396861feb1 Consistently use #!/usr/bin/env python
At Ksplice we used /usr/bin/python because we shipped dependencies as Debian /
Red Hat packages, which would be installed against the system Python.  We were
also very careful to use only Python 2.3 features so that even old system
Python would still work.

None of that is true at Humbug.  We expect users to install dependencies
themselves, so it's more likely that the Python in $PATH is correct.  On OS X
in particular, it's common to have five broken Python installs and there's no
expectation that /usr/bin/python is the right one.

The files which aren't marked executable are not interesting to run as scripts,
so we just remove the line there.  (In general it's common to have libraries
that can also be executed, to run test cases or whatever, but that's not the
case here.)

(imported from commit 437d4aee2c6e66601ad3334eefd50749cce2eca6)
2013-02-20 16:02:30 -05:00
Tim Abbott 67ee231ff9 zmirror: Add backoff-retry code for the Zephyr=>Humbug initialization.
(imported from commit bde30d87d6dae75bbfdcd49b30a1f08b3c585ec7)
2013-02-15 17:28:59 -05:00
Jessica McKellar cec2dfa52e zephyr mirror: remove unused imports.
(imported from commit 3ba0629f0a9009087b90fb6f795e855b569c2a67)
2013-02-06 14:22:17 -05:00
Jessica McKellar a31f443baa zephyr mirror: update subscription reporting to accomodate new results format.
(imported from commit 94a680d509f727962a07c18bc89edcb1146d8772)
2013-02-06 14:22:17 -05:00
Jacob Hurwitz 4448ba595a Change our default format to be more descriptive
This adds two characters to the length of our default format field, but
based on a conversation I had with kcr, I think this should probably be
okay. If it's a problem, the symptom we'll see is that certain people
will be unable to send zephyrs with this default format (so, certain
Humbug users will have their forwarding consistently fail).

We need to remember to, in a future commit (once everyone has started
using the updated version), remove the:
> or notice.format.endswith("@(@color(blue))")

(imported from commit 703ef60f524646bca8d5099c9066efabd365be43)
2013-02-05 14:27:56 -05:00
Jacob Hurwitz 630b7ae521 zephyr_mirror: Fix logging messages to multiple files under /tmp.
Also fixes the fact that certain output was not logged at all.

(imported from commit 0e63a9ef4105921ad7cbe3f7565ce7832c5db25e)
2013-02-05 14:27:55 -05:00
Jacob Hurwitz 9ebfa84385 zmirror: Allow duplicate zmirror processes to die gracefully.
Fixes #602.

I replaced the SIGKILL with a SIGINT, and then catch SIGINT with a
handler.  This handler calls cancelSubs if necessary, and can later be
edited to perform other clean-up operations, too. I thought about, in
this same commit, changing the SIGTERM in
maybe_restart_mirroring_script to a SIGINT, but after tracing out the
code paths, I realized that isn't necessary. (The SIGTERM is
necessarily performed on a process that has not subscribed to any
zephyr classes, so cancelSubs is unnecessary. If we do think that we
may want to add additional clean-up operations in the future, though,
then it might be worth investigating changing this SIGTERM.)

(imported from commit 692b295be6cb40b0e4ec2ca0bc58c58056ed9bd9)
2013-02-05 14:27:55 -05:00
Tim Abbott bd3bd8ca44 zephyr_mirror: Read list of public streams from sync-public-streams ouput.
(imported from commit c91828534bfed4047c16a98b926335f4beded005)
2013-02-05 14:27:55 -05:00
Luke Faraone d8f13837d9 Move files around in the API.
Bots are not part of what we distribute, so put them in the repo root.

We also updated some of the bots to use relative path names.

(imported from commit 0471d863450712fd0cdb651f39f32e9041df52ba)
2013-02-01 15:52:28 -05:00
Luke Faraone f2f4a2f8bd Move the API into a subdirectory for ease of imports.
Previously, if users of our code put the API folder in their pyshared
they would have to import it as "humbug.humbug". By moving Humbug's API
into a directory named "humbug" and moving the API into __init__, you
can just "import humbug".

(imported from commit 1d2654ae57f8ecbbfe76559de267ec4889708ee8)
2013-01-16 16:55:22 -05:00
Tim Abbott 8704f6f9d0 zmirror: Fix running on 32 bit systems.
(This also needs pyzephyr to be compiled for 32 bit).

(imported from commit 06e33e17be517483213d402dd75ec2ac168395d1)
2013-01-02 14:20:24 -05:00
Tim Abbott a18796961f zephyr_mirror: Use modern default_format that discourages displaying it.
(imported from commit bfb2ffbf4ca7d424ffe54c4b2752378f4854db25)
2013-01-02 14:10:21 -05:00
Tim Abbott 1e91142e38 zmirror: Fix use of print rather than logging.foo in send_zephyr.
(imported from commit 7188f5364a864f2f144fb630d7a8c85d69219c42)
2013-01-02 14:10:21 -05:00
Tim Abbott 44c5315cd7 zmirror: Add another Zephyr class we can't subscribe to.
(imported from commit 6d43c901ecfd01574149efc06c782a49c5d64c9e)
2013-01-02 14:10:21 -05:00
Tim Abbott 1351fb5f8b zmirror: Use shorter default_format string.
This was causing about 10% of the time, personals being forwarded by
tabbott/extra@ATHENA.MIT.EDU to get this error:

zwrite: Field too long for buffer while sending notice to
tabbott/extra@ATHENA.MIT.EDU

We don't fully understand the cause of the problem, but this fixes it.

(imported from commit 22c39a1061cde9ad6973ef07aee7227623a3d47d)
2013-01-02 14:10:21 -05:00
Tim Abbott f65b130b45 zmirror: Fix printing of list of stream failures.
(imported from commit 0d573b412809263dbbacdf1b475adac440dae352)
2013-01-02 10:22:08 -05:00
Tim Abbott 2919ce2544 zephyr_mirror: Cancel Zephyr subscriptions on graceful restart.
(imported from commit 39737bbf7ea76181ab25a0eaacf230906cba3d87)
2012-12-21 12:44:19 -05:00
Tim Abbott 25bb86d2d0 zephyr_mirror: Fix killing duplicate zephyr_mirror processes.
The refactoring that we did a couple weeks ago to make the zephyr
mirror script restart itself automatically (by splitting it into
zephyr_mirror.py and zephyr_mirror_backend.py) had a poor interaction
with our code for killing old zephyr_mirror processes (to prevent
double-mirroring).  If you manually ran two copies of the outer
mirroring script (zephyr_mirror.py), then the second one would on
startup kill the first one's zephyr_mirror_backend.py children (for
being duplicate zephyr_mirror_backend.py processes that would result
in double-mirroring).  However, importantly, it did not kill the first
mirroring script's zephyr_mirror.py parent process, so the first
mirroring script would then proceed to startup up new children.  The
process then repeats, with the two scripts' roles reversed.

This issue didn't affect the sharded mirroring case, where I had been
doing the testing of the kill code with that refactoring, because we
don't have a version of the outer zephyr_mirror.py loop for that
situation (a consequence of it being hard to restart the threads
properly with the run_parallel API that we're using to spawn all the
children).

(imported from commit d4886ac77312a6b0ebd0d612f6fb084970bf23a2)
2012-12-21 12:44:19 -05:00
Tim Abbott a22f94633b zephyr_mirror: Fix insecure temporary file for log.
This is probably a lot more annoying to use, in that e.g. there are
separate log files for the two directions of the mirror, but we
haven't used these logs for much, so whatever.

(imported from commit d3bc407d90099214d242526c01cd3d3cd9d9d9bd)
2012-12-20 14:30:20 -05:00
Tim Abbott e609692e33 zephyr_mirror: Add a few more restricted-access classes.
(imported from commit 6690e5dcd6a313ac4e4ef2351e300306ca7fb3c5)
2012-12-19 08:49:14 -05:00
Waseem Daher a3fba04fdd Don't consider hyphens to be an acceptable place to wordwrap.
Otherwise, URLs like this one from catherio:
  http://www.sadiemagazine.com/issue-no-11/arts-letters/comic/i-think-i-am-in-friend-love-with-you
don't go to zephyr as one line, which makes them really
annoying to click on barnowl, etc.

(imported from commit d14f0af7d24a6cb8bc7cb0582f60116308f59923)
2012-12-18 11:03:18 -05:00
Tim Abbott 08f165e3ac zephyr_mirror: Add comment explaining why we drop the @ATHENA.MIT.EDU in zwrite -C.
(imported from commit b591e2e0d6c607ef04e3c0005091078cfe23e3ff)
2012-12-10 17:05:04 -05:00
Tim Abbott 8e0aabc730 zephyr_mirror: Improve debugging output on outgoing messages.
Previously all that we logged was the sender, which turns out to be a
constant for humbug=>zephyr mirroring.

(imported from commit 527a3ac4b9b815a2b4d6b63c3ad92d9d5ad99a6e)
2012-12-10 17:05:04 -05:00
Tim Abbott bb80b86e25 zephyr_mirror: Fix handling of Humbug=>Zephyr personals.
(imported from commit 149f8aaf9076583499392bcb09a0250b05c3dea6)
2012-12-08 13:05:44 -05:00
Tim Abbott 94a553f0f1 zephyr_mirror: Use a variable for humbug_account_email.
(imported from commit 90da406a5eab67d20b42243e96b8afb3e5b5f342)
2012-12-08 13:00:22 -05:00
Tim Abbott 28d98729a5 zephyr_mirror: Fix detection of personals.
(imported from commit 1fc9050bad36dc15e76f65bdb6532e3e7e601a40)
2012-12-05 12:50:14 -05:00
Tim Abbott 37c837904f Don't send zwrite pings.
(imported from commit 8dd7894eda6315ab52e61489e21ec1d201e86cc9)
2012-12-05 12:04:18 -05:00
Zev Benjamin b945113867 [manual] Unify huddles and personals into private messages on the receive path
feedback-bot and zephyr_mirror will need to be updated and restarted
when this is deployed to prod.

(imported from commit fe2b524424c174bcb1b717a851a5d3815fda3f69)
2012-12-04 18:01:51 -05:00
Keegan McAllister b971ce4683 mirroring: Fix import by putting 'api' path first
(imported from commit b35d962d6456f01b97deeeaa328faa03b65494f8)
2012-12-03 17:50:29 -05:00
Keegan McAllister ac9e3d6483 zephyr_mirror_backend: Rename variable 'humbug'
(imported from commit 2e278e4d2452f750cb874510a762fc9616e7dfb3)
2012-12-03 17:49:27 -05:00
Tim Abbott 24b8a7eccc zephyr_mirror: Fix case issue in subscriptions syncing.
(imported from commit e87ddab3684f4acae940ee831caadbc346458a11)
2012-12-03 16:20:19 -05:00
Keegan McAllister b9325a09fa Rename class HumbugAPI to Client
(imported from commit 4270f31fc5febcd9c444d0d133a1dad3860618f0)
2012-12-03 15:54:54 -05:00
Keegan McAllister f9f8a6e603 Rename api.common to humbug
Fixes #482.

(imported from commit 1bd6a7fd993d8d5e225e0311c288dbce0c369a40)
2012-12-03 15:54:54 -05:00
Tim Abbott abb95fca04 zephyr_mirror: Move class name canonicalization earlier.
We should be canonicalizing stream names to class names in
update_subscriptions_from_humbug, before we even decide which classes
to subscribe to; otherwise deduplication and tracking of which classes
we're already subscribed to won't work.

(imported from commit a751b6fca1022390a087516a0730ff77f13d7edf)
2012-11-29 13:57:06 -05:00
Tim Abbott 8683613e76 zephyr_mirror: Add options to run class mirror without sharding for debugging.
(imported from commit 4fcc403504d93e30aca6046cba9f1fe5342faac6)
2012-11-29 13:57:06 -05:00
Tim Abbott 18bbd02074 zephyr_mirror: Don't leak zephyr_mirror processes on send_humbug failures.
(imported from commit be005018672369683d9e3bdbf144f96f58af6b6c)
2012-11-29 13:57:06 -05:00
Tim Abbott 074316105b zephyr_mirror: Replace bare excepts with 'except Exception'.
(imported from commit 857212e0e5116471e418d37845550108c0a47d2b)
2012-11-29 13:57:05 -05:00
Tim Abbott ab68503fcf zephyr_mirror: Compare zephyr subs from server after normalization.
(imported from commit eea290698e83c7d26f23ab3bb0201914417f0e09)
2012-11-28 22:26:10 -05:00
Tim Abbott c61567b67a zephyr_mirror: Add two more restricted access classes.
(imported from commit 06ff0e19c1522ade018aea66f152fdfa55c18e1d)
2012-11-28 22:26:10 -05:00
Tim Abbott 13ebd229c1 get_updates: Replace the "mirror" argument with checking "client".
(imported from commit bd84c2c7a456de8ea7ec35eaf76b5bf8647ca61a)
2012-11-28 17:41:38 -05:00
Tim Abbott 2b2398b712 zephyr_mirror: Pass an encoded stream to hashlib for sharding.
(imported from commit cc57b3956198b66c9cfc84423434a71b8200d819)
2012-11-28 15:44:35 -05:00
Tim Abbott de14210a4d zephyr_mirror: Validate zephyr subscriptions happened successfully.
(imported from commit 494895b87eb9b555175c9890c5c2046cf4ce40ac)
2012-11-28 15:44:35 -05:00
Tim Abbott 121c3cb9c8 zephyr_mirror: Fix display of personals on non-message classes.
(imported from commit 42ff030d2d78c530428dcede406d46fc71521fef)
2012-11-27 14:12:19 -05:00
Tim Abbott 305a8ce569 zephyr_mirror: Run the class mirror in 16 parallel shards.
(imported from commit cb17ef999e94cc57d5b01114479728b04475c9c4)
2012-11-27 12:09:57 -05:00
Tim Abbott bfec56661e zephyr_mirror: Add a --no-forward-from-humbug option.
We'll use this to only have one of our shards forward from Humbug.

(imported from commit a6f3c2180b49c1fa826baf1ae9e7e3ba252a5bfa)
2012-11-27 12:09:57 -05:00
Tim Abbott 975c8ef5b0 zephyr_mirror: Clean up unnecessary logic in process_notice.
(imported from commit 28599789016fd1006c2de4d5ddc3cf1f10baa47f)
2012-11-27 12:09:57 -05:00
Tim Abbott dbeb107aaf zephyr_mirror: Fix the check for whether a Zephyr is a personal.
(imported from commit 609b6339fa82ecbd6435a386d24bf555e75a8120)
2012-11-27 12:09:57 -05:00
Tim Abbott b081d2fa5a zephyr_mirror: Reorder process_notice to log fewer "Skipping" messages.
Previously, we were sending "Skipping message we got from Humbug!"
for messages we wouldn't have forwarded anyway.

(imported from commit 36df85a61336ac00e3d7913d5a417d6b42764350)
2012-11-27 12:09:57 -05:00
Tim Abbott f45a8bebf2 zephyr_mirror: Don't display/log some useless "Skipping" notices.
In this case, if we're configured to not forward personals, there's no
point in logging a decision not to forward one.

(imported from commit 62c37591c6a70afb6235de626b0c6a3502cbcb27)
2012-11-27 12:09:56 -05:00
Tim Abbott 333573f06b zephyr_mirror: Simplify optparse usage.
(imported from commit d7cce4236dae6f369ea1cc4e12f89c2011718b7f)
2012-11-27 12:09:56 -05:00
Tim Abbott ba33df97bb zephyr_mirror: Don't use @(@color(blue)) in zsigs.
(imported from commit e4b594e3c22c25fa4f87baaad09227396bb92a0d)
2012-11-26 16:19:09 -05:00
Tim Abbott 135f7f1b57 zephyr_mirror: Use default format to indicate what came from Humbug.
(imported from commit 5c2fdccb08d605a37d6c87c37f023c06a0a9a760)
2012-11-26 16:18:05 -05:00
Tim Abbott 111a9bc0e6 zephyr_mirror: Refactor zwrite_args handling a bit.
(imported from commit 4923c90dc546022d48f226c2a204e9d09717b563)
2012-11-26 14:58:34 -05:00
Tim Abbott 9d6ce4ee0b api: Add copyright and MIT license notices to distributed tools.
(imported from commit 9f93050f988549dc363f5a7c798c8a9fa1c01d29)
2012-11-26 12:25:38 -05:00
Tim Abbott 27d597d686 zephyr_mirror: Don't break long words/urls when mirroring.
(imported from commit df2b6a6969fc54016eeee4996df317e2843844c8)
2012-11-21 15:55:18 -05:00
Tim Abbott 244ff8459e zephyr_mirror: Add exception handling around all main loop code.
(imported from commit 5dc46decf882302a6c58f3bb788362f82f293b35)
2012-11-19 12:44:12 -05:00
Tim Abbott 36280c2769 zephyr_mirror: Use minimal wrapper script that is hard to accidentally break.
(imported from commit 45967b8b806df616b0480b217dd5db82c5a2cabd)
2012-11-19 12:44:06 -05:00