Yesterday I got the final answer from the hiring committee, I am not going to be a SRE (Site Reliability Engineer) at Google.
It all started 3 months ago when I got the first screening interview. I answered some basic questions (50% answers were correct, though) and I was notified that I will get the second phone interview with a real SRE.
During this interview I enjoyed the troubleshooting part the most. It was basically a role-playing game when you ask the interviewer the questions about the system or the commands you would run on the machine and the interviewer tells you what you see on the screen. It took me mere seconds to figure out the failing machine but then I started going back and forth around the different components. In the end I did manage to find the actual failure. The coding exercise was fun too. I learned that I forgot everything about hashes, dict.items()? dict.entries()? (actually it is dict.items()) but managed to write a properly working code from the first try in the shared Google document. There were also some basic UNIX questions, like how you can delete a file starting with a dash as well as basic networking - how does a switch work.
After a week or so I was notified that they would like me to come to Zurich, Switzerland for an on-site interview. I had 2 weeks to read the recommended books, play with the various systems and network configuration, get the Visa and come to Switzerland (the flights and the hotel was paid for by Google). I had to leave home quite early since we just had a huge snow storm, so I was quite sleepy and did not walk around much.
The Engimatt Hotel was within the walking distance from the Google Zurich office which looked quite... let's say "cost-effective".
I walked around the location to get familiar with the surroundings, found the city to be quite expensive, returned back to the hotel, and shut down.
On the 26th of March I woke up, went to the hotel restaurant for a breakfast being the only one who did not wear the suit (the recruiters explicitly told me to stay away from the suit and tie), and went to the Google office.
The inside of the office looked extremely different, it was joyful, interesting and well thought. I passed through the cafeteria, received my bottle of Google Water and was led to the conference room. Each interview lasts for 45 minutes and there are 5 interviews and a lunch break.
The interviews focused on the following:
- Linux internals.
- System administration.
- Python coding skills.
- Large scale systems design.
You will need to know how the malloc() works and how the memory allocator is implemented in, say, glibc, how the processes are started and pass the data between themselves on the low level. I believe I got something around 3/5 there, because I never jumped that far into the kernel for the memory allocation and I never had to find the reason why strace output is full of sbrk() and mmap() calls.
Having just finished building my little system for the kernel bisection with PXE I had the answer even before the interviewer finished the question and the overall experience was fun. I played with puppet and rsync in the past, so the questions about synchronizing machine data and configuration were simple. After all the prepared questions were answered we had a chat about ZFS and how awesome it is. I believe this is the only interview where I scored 5/5.
Python coding skills.
I started pretty well with explaining what's wrong with def foo(data=): ... but I failed miserably at implementing a simple calculator performing only additions and multiplications. For some reason I forgot all the python I knew ("elseif?", definitely not "else if?"... it is "elif", yes) and the rest 30 minutes I struggled to come up with something I believed to be working. After spending 3 years primarily on the little one-time scripts and support issues I got really rusty and uncertain in programming something w/o running the intermediate solution to see whether it is working or not. I feel that I got 2/5 there and when I returned back home I had a beautiful solution written in 5 minutes. The critical point is that you need to find out how you would do this task, don't try to think how the computer should be doing it immediately.
Oh this was fun. I managed to get through 2 tasks (I don't know how much more the interviewer has prepared), but they were great. I never dealt with so many failures as what I got at the interview while trying to do a simple "umount /usr", however I've spent more than I should on finding the reason of hanging terminal during the SSH connection. Locales, environment, libraries, networking MTU... everything should have been considered. I think I got 4/5 there but the score might have been lower.
Large scale systems design
I've never designed anything like this before and while I read a lot about this, I haven't had the hands-on experience. The task was humble, design the logging system for the whole Google. I had difficult time coming up with a design, used technologies and it was a completely failed interview. 1/5 or even 0/5.
I was escorted out of the building, went to the hotel, had dinner there, prepared for the upcoming flight and shut down. It was a pretty tiring experience but it was well worth it.
Usually it takes one or two weeks for the hiring committee to come up with a decision, however this year the H1B US visa cap was reached in mere 5 days, and while the original US hiring committee decision was negative, the recruiters decided to forward the interview results to the Europe-based committee too. The overall review process took almost 4 weeks and yesterday I finally received an answer, "No". The committee has noted that I have a potential though.
I feared I would face the people who would be all "I work for Google, you are nobody". Instead, everybody remained positive even when I was obviously failing. The office internals are interesting. You do want to spend the day inside. I sometimes visit my wife at Global Logic Kiev and now I know where they tried to copy the environment from. One of the interviewers brought the Chromebook Pixel to take the notes (and guess what, I did not ask to look at the screen, double fail!), Lenovo Thinkpads and Apple MacBooks were spotted as well. The food is free for all the employees and there are 4 or 5 restaurants which prevent the over-crowding during the lunch hours, there is a gym and a workshop where you can build something (as in, the real, physical workshop with the power tools etc.), there is a room full of trees and plants and you definitely feel that there is more oxygen there. Every room has a natural light source, i.e. a window while all the stairs are located in the center of the building. The office felt really bright and light.
I saw a Hipster Dog image pinned to the corkboard.
It's definitely a place one would like working at.
You would want to read UNIX and Linux System Administration Handbook for system administration. It took me a week to get through all the linux things and it structured my administration knowledge. It is fun and easy to read.
As per recommended reading by Google itself:
- TCP Illustrated; Volume 3 by W. Richard Stevens, but you don’t really need to dive into the details of TCP for Transactions, which turned out to have security flaws. Understanding of IP, TCP/IP and UDP/IP is a must.
- The Design and Implementation of the 4.4BSD Operating System by Marshall Kirk McKusick
- The UNIX Programming Environment by Brian Kernighan and Rob Pike
- The Practice of System and Network Administration by Tom Limoncelli, Christina Hogan and Strata R Chalup