Posted by Michael Krantz, Google Blog Team



-- via craigslist, and thanks for asking. Our engineers, though, tend to come by more varied, and occasionally odder, routes. Some get recruited out of grad school, or by friends or former colleagues. Others just send their resumes to jobs@google.com. For a few engineers, though, the path has been more interesting. Peter Bradshaw, for instance, built “a music playing system based on printed cards with barcodes and webcams. Includes lego!” (No, I don’t know what that means, either.) Over the next few weeks, we’re going to post some of their stories.



Like this one, from Systems Administrator Aaron Joyner:





My story started when I came into work one morning and was unable to look up something on Google. Being the sysadmin for my company at the time, it was my responsibility to resolve the problem, so I started poking around. It turned out that our DNS server [ed: all the jargony stuff you'll hear in this anecdote refers to the software that websites use to connect and talk to each other] was returning an error when trying to look up google.com, specifically a server failure error. Just as I’d convinced myself that it wasn't our problem but Google’s, the problem suddenly resolved itself. I promptly forgot about it and went back to my regular work.



But then I came in the next morning and had exactly the same problem, so I started looking at Google's DNS responses very closely. It became clear that the specific combination of delegations and glue records they were returning [ed: see note above] would result in an eventual error approximately once per day, and this would then take it about five minutes to give up and try again. Not entirely convinced that I should point the finger at Google yet, I posted a message to my local Linux Users Group asking if anyone had had problems with resolving google.com addresses and got a couple "Yeah, I did have a problem like that once recently" responses.



Thus reinforced, I headed over to Google.com, found the "Contact Us" page and the "Report a problem" link, chunked in a brief problem description and a link to the archived copy of the long technical description from that same mailing list thread, and thought to myself, "Gee, I'll never hear about that again." But then one afternoon a week later I get an email that said, basically, "We've received your problem report, and forwarded it on to the appropriate department, if they need any further information they’ll contact you. Thanks." Again, I thought, "Gee, how nice. I'll never hear about that again."



But that evening I got an email from Dave Presotto (the guy who wrote the DNS server for Plan9) saying that he was looking into it and would get back to me. Forty-five minutes later I got another email, this one describing how he believed they had accidentally fixed the problem earlier in the week due to general code cleanup, and asking what I thought of the solution. After I recovered my senses and stopped bouncing around the room, I had a few email exchanges with Dave, in the course of which I asked casually if they needed any good sysadmins out in Mountain View. He referred me, and the rest is history.