zmirror: Use backoff-retry code for the outer loop.

Realistically, if the bot crashes once, it'll probably crash the next
time too, so I'm not convinced we need this loop at all, but in the
interests of avoiding churn on an extensively tested script, I'm going
to err on the side of the minimal change here.

(imported from commit e2bbd3700395ba4d0b181a4616e816e8f1231669)
This commit is contained in:
Tim Abbott 2013-02-14 12:04:45 -05:00
parent 67ee231ff9
commit 43acb4fdde

View file

@ -30,6 +30,7 @@ import traceback
import signal import signal
from zephyr_mirror_backend import parse_args from zephyr_mirror_backend import parse_args
from zephyr_mirror_backend import RandomExponentialBackoff
def die(signal, frame): def die(signal, frame):
# We actually want to exit, so run os._exit (so as not to be caught and restarted) # We actually want to exit, so run os._exit (so as not to be caught and restarted)
@ -59,10 +60,19 @@ if options.forward_class_messages and not options.noshard:
pass pass
sys.exit(0) sys.exit(0)
while True: backoff = RandomExponentialBackoff()
while backoff.keep_going():
print "Starting zephyr mirroring bot" print "Starting zephyr mirroring bot"
try: try:
subprocess.call(args) subprocess.call(args)
except: except:
traceback.print_exc() traceback.print_exc()
time.sleep(1) backoff.fail()
print ""
print ""
print "ERROR: The Zephyr mirroring bot is unable to continue mirroring Zephyrs."
print "This is often caused by failing to maintain unexpired Kerberos tickets"
print "or AFS tokens."
print ""
sys.exit(1)