Commit graph

87 commits

Author SHA1 Message Date
Tim Abbott 5e5e35924c zephyr_mirror: Subscribe to mail zephyrs even if we're not using them.
This should enabled us, in the future, to enable support for mail
zephyrs without requiring users to re-auth.

(imported from commit 2665743212da567fe85742d30cca42b902e41a0d)
2013-08-27 16:21:41 -04:00
Tim Abbott 0d59d15781 zephyr_mirror: Rename confusing log path variables.
(imported from commit a120f653c1b9283b91f339cccea52bd5ad89b69f)
2013-08-27 14:21:26 -04:00
Tim Abbott fd96a0b694 zmirror: Add monitoring for personals mirrors.
(imported from commit acb7f2222076d2f3884a2e52c7032cb48fde1757)
2013-08-26 18:17:26 -04:00
Tim Abbott 2249b18081 zephyr_mirror: Ignore expired Kerberos tickets for personals mirror.
(imported from commit 094755feacd299255f078432465f11a609df9aef)
2013-08-26 18:17:26 -04:00
Tim Abbott 9223f6b2e8 zephyr_mirror: Allow running multiple zephyr mirroring scripts under one user.
(imported from commit ecec50ee0e51736487358ef9f5847000d35f327a)
2013-08-26 18:17:26 -04:00
Tim Abbott 2ee0b3a079 zephyr_mirror: Add options to save/restore sessions.
The davidben-patched-for-roost Zephyr branch (available at
https://github.com/davidben/zephyr/tree/roost) adds Zephyr support for
these options.  We also patch python-zephyr to expose them.  These
basically let you save your Zephyr tickets and port number to a file,
so that you can later restore them (even potentially after the machine
rebooted).  Basically because Zephyr is UDP, the Zephyr server will
continue trying to deliver messages to a particular port number that
was registered for up to 20 minutes after getting an error; so we can
even have downtime and reboot and still get our packets so long as we
restore the sessions within 20 minutes.

(imported from commit 986cbb157ddfa57aa4b644cd826f8418e9876dc7)
2013-08-26 18:17:25 -04:00
Tim Abbott 81aebf897b zephyr_mirror: Finish outgoing zcrypt.
(imported from commit dfef23feb8e0f6a0a8ba775b6cccd391cb307adb)
2013-08-22 12:29:04 -04:00
Tim Abbott 2c39d17120 zephyr_mirror: Send outgoing messages encrypted when zcrypt would.
Our .crypt-table parsing code isn't quite correct, in that we don't
handle either the "zcrypt default" or "zcrypt by class/instance" pair
options (for sending messages in either direction) -- you have to be
zcrypting for an entire class.  I think this makes sense given that on
the Zulip end we can only enforce anything on a stream level.

(imported from commit a7901b1dc025a04a23ee71ecdd499e3f150ba614)
2013-08-20 17:09:09 -04:00
Tim Abbott a47b0e54c3 zephyr_mirror: Renew kerberos tickets when starting up class mirror.
(imported from commit 34b0a78558c6f20043efe986a16064387f36f1c0)
2013-08-20 15:27:24 -04:00
Tim Abbott 97be689411 zephyr_mirror: Auto-detect Python version for pyzephyr path.
(imported from commit e2aca3a08f8aa12b43d71acffcc0e2249feb7ca7)
2013-08-20 15:27:23 -04:00
Tim Abbott 264651c7b4 zephyr_mirror: De-zcrypt messages for which we have the key.
For now we only support the AES encryption type since the DES one is
probably not used anymore.

(imported from commit 222606db9f704917e74159e7d07a110187a236e6)
2013-08-20 15:02:07 -04:00
Tim Abbott 7f626d2bba zephyr_mirror: Handle syncing subs with invite-only streams.
(imported from commit ec8d0c0d13ff0a81fa4a5e1225e94e6dad997356)
2013-08-20 15:02:04 -04:00
Tim Abbott 9d55822cfb zephyr_mirror: Add option to control path to streams file.
This makes it easy to test the class mirror on just a single test
stream.

(imported from commit 3decb1ced54507683f1c0634df14f73e693104ef)
2013-08-20 14:57:13 -04:00
Tim Abbott 084847b0d7 Rename the 'humbug' API module to 'zulip'.
(imported from commit b3a3d7c05459cbb0110cd0fbe2197d779f3a6264)
2013-08-08 10:22:31 -04:00
Tim Abbott 4741258ca4 Change humbug => zulip in zephyr_mirror_backend internal variables.
(imported from commit 7cc5d78bfb5db8c9d5a44ed862da304d9e44ff0e)
2013-08-07 10:00:08 -04:00
Tim Abbott 84aa7d4230 Change Humbug => Zulip in zephyr_mirror states.
(imported from commit 67542cd2ba93ff4b03c99b59645b7d22ed937337)
2013-08-07 10:00:07 -04:00
Tim Abbott 4cd4c6897b Change Humbug => Zulip in text/comments.
(imported from commit 2f9d73431ae40e1b9e9e11bc2f4f62f566ae758a)
2013-08-07 10:00:07 -04:00
Tim Abbott e0c77efb3f Escape @ when mirroring content back to Zephyr.
(imported from commit 0ab58b5e8a30c66fb6aeaccaebbe498092de1cd5)
2013-08-01 14:01:58 -04:00
Tim Abbott 22e5a15bc4 [manual] Use api.zulip.com to access the Humbug API.
This needs to be deployed after the nginx configuration is deployed.

(imported from commit 167a3d9d27595d40883bb400ebcc44c4ad9b6a2f)
2013-07-25 17:27:46 -04:00
Tim Abbott d063efa8a0 More miscellaneous humbughq.com=>zulip.com updates.
(imported from commit 9a3dbc4268f251c2c0b20d8b6a7e1dc672f1b385)
2013-07-25 17:27:45 -04:00
Tim Abbott c9fab96e5d zephyr_mirror: Use Humbug full name as zsig.
(imported from commit 8cd24c4ff6c265084764ab367b46e4cb64e8c796)
2013-07-18 13:58:16 -04:00
Jessica McKellar e64d390c99 zephyr mirror backend: Switch from Humbug to Zulip in user-visible text.
(imported from commit cc4da47373362babb2f5f57493831946b29d32cf)
2013-07-15 17:49:00 -04:00
Tim Abbott 6b67e42d1a zephyr_mirror: you no longer need to reload Humbug after subbing.
(imported from commit 23cb4e5c4b3300953e58c029150d4a980bac626e)
2013-06-28 15:57:29 -04:00
Tim Abbott 5e2979b870 [manual] Change API URLs to be based on api.humbughq.com/api.
This must be deployed after we update our running nginx configuration
to serve api.humbughq.com.

(imported from commit b5c34ebdd595f55eecd6dca6a18a37f105107bd5)
2013-06-28 15:57:27 -04:00
Tim Abbott 58f4ffe088 [manual] API add_subscriptions: Change arguments to support options.
Since in the future we might want requests to add subscriptions to
include things like colors, in_home_view, etc., we're changing the
data format for the add_subscriptions API call to pass each stream as
a dictionary, giving a convenient place to put any added options.

The manual step required here is updating the API version in AFS
available for use with the zephyr_mirror.py system.

(imported from commit 364960cca582a0658f0d334668822045c001b92c)
2013-06-25 16:34:45 -04:00
Tim Abbott c849ae3470 zephyr_mirror: Fix incorrect check for already running mirror bots.
(imported from commit 76e758697ac19362e79d4a4995b576f10d229514)
2013-05-20 23:29:53 -04:00
Zev Benjamin 223ac22a13 Fix use of logging instead of logger
(imported from commit cd25471e842e089d5032b07bcd83c6fe44d03a2d)
2013-05-09 13:50:21 -04:00
Tim Abbott fcb6336642 zephyr_mirror: Use select to determine whether new messages are available.
(imported from commit a91f45aa9f3913f38285c050e574bf562b2af2df)
2013-05-09 10:35:49 -04:00
Tim Abbott 5a0e86c6d4 zephyr_mirror: Use json if simplejson is not available.
(imported from commit 4b5f6ff71067cd2697c864c715afbb3b9dd19302)
2013-05-09 10:35:47 -04:00
Tim Abbott 5af166e063 zephyr_mirror: Limit logging of errors when checking restart stamps.
(imported from commit 194b30d757a5f20d6d20bc8aed9154c6aece9837)
2013-05-09 10:35:47 -04:00
Keegan McAllister 396861feb1 Consistently use #!/usr/bin/env python
At Ksplice we used /usr/bin/python because we shipped dependencies as Debian /
Red Hat packages, which would be installed against the system Python.  We were
also very careful to use only Python 2.3 features so that even old system
Python would still work.

None of that is true at Humbug.  We expect users to install dependencies
themselves, so it's more likely that the Python in $PATH is correct.  On OS X
in particular, it's common to have five broken Python installs and there's no
expectation that /usr/bin/python is the right one.

The files which aren't marked executable are not interesting to run as scripts,
so we just remove the line there.  (In general it's common to have libraries
that can also be executed, to run test cases or whatever, but that's not the
case here.)

(imported from commit 437d4aee2c6e66601ad3334eefd50749cce2eca6)
2013-02-20 16:02:30 -05:00
Tim Abbott 67ee231ff9 zmirror: Add backoff-retry code for the Zephyr=>Humbug initialization.
(imported from commit bde30d87d6dae75bbfdcd49b30a1f08b3c585ec7)
2013-02-15 17:28:59 -05:00
Jessica McKellar cec2dfa52e zephyr mirror: remove unused imports.
(imported from commit 3ba0629f0a9009087b90fb6f795e855b569c2a67)
2013-02-06 14:22:17 -05:00
Jessica McKellar a31f443baa zephyr mirror: update subscription reporting to accomodate new results format.
(imported from commit 94a680d509f727962a07c18bc89edcb1146d8772)
2013-02-06 14:22:17 -05:00
Jacob Hurwitz 4448ba595a Change our default format to be more descriptive
This adds two characters to the length of our default format field, but
based on a conversation I had with kcr, I think this should probably be
okay. If it's a problem, the symptom we'll see is that certain people
will be unable to send zephyrs with this default format (so, certain
Humbug users will have their forwarding consistently fail).

We need to remember to, in a future commit (once everyone has started
using the updated version), remove the:
> or notice.format.endswith("@(@color(blue))")

(imported from commit 703ef60f524646bca8d5099c9066efabd365be43)
2013-02-05 14:27:56 -05:00
Jacob Hurwitz 630b7ae521 zephyr_mirror: Fix logging messages to multiple files under /tmp.
Also fixes the fact that certain output was not logged at all.

(imported from commit 0e63a9ef4105921ad7cbe3f7565ce7832c5db25e)
2013-02-05 14:27:55 -05:00
Jacob Hurwitz 9ebfa84385 zmirror: Allow duplicate zmirror processes to die gracefully.
Fixes #602.

I replaced the SIGKILL with a SIGINT, and then catch SIGINT with a
handler.  This handler calls cancelSubs if necessary, and can later be
edited to perform other clean-up operations, too. I thought about, in
this same commit, changing the SIGTERM in
maybe_restart_mirroring_script to a SIGINT, but after tracing out the
code paths, I realized that isn't necessary. (The SIGTERM is
necessarily performed on a process that has not subscribed to any
zephyr classes, so cancelSubs is unnecessary. If we do think that we
may want to add additional clean-up operations in the future, though,
then it might be worth investigating changing this SIGTERM.)

(imported from commit 692b295be6cb40b0e4ec2ca0bc58c58056ed9bd9)
2013-02-05 14:27:55 -05:00
Tim Abbott bd3bd8ca44 zephyr_mirror: Read list of public streams from sync-public-streams ouput.
(imported from commit c91828534bfed4047c16a98b926335f4beded005)
2013-02-05 14:27:55 -05:00
Luke Faraone d8f13837d9 Move files around in the API.
Bots are not part of what we distribute, so put them in the repo root.

We also updated some of the bots to use relative path names.

(imported from commit 0471d863450712fd0cdb651f39f32e9041df52ba)
2013-02-01 15:52:28 -05:00
Luke Faraone f2f4a2f8bd Move the API into a subdirectory for ease of imports.
Previously, if users of our code put the API folder in their pyshared
they would have to import it as "humbug.humbug". By moving Humbug's API
into a directory named "humbug" and moving the API into __init__, you
can just "import humbug".

(imported from commit 1d2654ae57f8ecbbfe76559de267ec4889708ee8)
2013-01-16 16:55:22 -05:00
Tim Abbott 8704f6f9d0 zmirror: Fix running on 32 bit systems.
(This also needs pyzephyr to be compiled for 32 bit).

(imported from commit 06e33e17be517483213d402dd75ec2ac168395d1)
2013-01-02 14:20:24 -05:00
Tim Abbott a18796961f zephyr_mirror: Use modern default_format that discourages displaying it.
(imported from commit bfb2ffbf4ca7d424ffe54c4b2752378f4854db25)
2013-01-02 14:10:21 -05:00
Tim Abbott 1e91142e38 zmirror: Fix use of print rather than logging.foo in send_zephyr.
(imported from commit 7188f5364a864f2f144fb630d7a8c85d69219c42)
2013-01-02 14:10:21 -05:00
Tim Abbott 44c5315cd7 zmirror: Add another Zephyr class we can't subscribe to.
(imported from commit 6d43c901ecfd01574149efc06c782a49c5d64c9e)
2013-01-02 14:10:21 -05:00
Tim Abbott 1351fb5f8b zmirror: Use shorter default_format string.
This was causing about 10% of the time, personals being forwarded by
tabbott/extra@ATHENA.MIT.EDU to get this error:

zwrite: Field too long for buffer while sending notice to
tabbott/extra@ATHENA.MIT.EDU

We don't fully understand the cause of the problem, but this fixes it.

(imported from commit 22c39a1061cde9ad6973ef07aee7227623a3d47d)
2013-01-02 14:10:21 -05:00
Tim Abbott f65b130b45 zmirror: Fix printing of list of stream failures.
(imported from commit 0d573b412809263dbbacdf1b475adac440dae352)
2013-01-02 10:22:08 -05:00
Tim Abbott 2919ce2544 zephyr_mirror: Cancel Zephyr subscriptions on graceful restart.
(imported from commit 39737bbf7ea76181ab25a0eaacf230906cba3d87)
2012-12-21 12:44:19 -05:00
Tim Abbott 25bb86d2d0 zephyr_mirror: Fix killing duplicate zephyr_mirror processes.
The refactoring that we did a couple weeks ago to make the zephyr
mirror script restart itself automatically (by splitting it into
zephyr_mirror.py and zephyr_mirror_backend.py) had a poor interaction
with our code for killing old zephyr_mirror processes (to prevent
double-mirroring).  If you manually ran two copies of the outer
mirroring script (zephyr_mirror.py), then the second one would on
startup kill the first one's zephyr_mirror_backend.py children (for
being duplicate zephyr_mirror_backend.py processes that would result
in double-mirroring).  However, importantly, it did not kill the first
mirroring script's zephyr_mirror.py parent process, so the first
mirroring script would then proceed to startup up new children.  The
process then repeats, with the two scripts' roles reversed.

This issue didn't affect the sharded mirroring case, where I had been
doing the testing of the kill code with that refactoring, because we
don't have a version of the outer zephyr_mirror.py loop for that
situation (a consequence of it being hard to restart the threads
properly with the run_parallel API that we're using to spawn all the
children).

(imported from commit d4886ac77312a6b0ebd0d612f6fb084970bf23a2)
2012-12-21 12:44:19 -05:00
Tim Abbott a22f94633b zephyr_mirror: Fix insecure temporary file for log.
This is probably a lot more annoying to use, in that e.g. there are
separate log files for the two directions of the mirror, but we
haven't used these logs for much, so whatever.

(imported from commit d3bc407d90099214d242526c01cd3d3cd9d9d9bd)
2012-12-20 14:30:20 -05:00
Tim Abbott e609692e33 zephyr_mirror: Add a few more restricted-access classes.
(imported from commit 6690e5dcd6a313ac4e4ef2351e300306ca7fb3c5)
2012-12-19 08:49:14 -05:00