Another problem is the disconnect between "recruiting is extremely important to the company" and "the interview process should be as thorough and regimented as possible", as if the latter followed from the former.
There is a reason for the regimented process that you're talking about.
The challenge for any large company is to delay, for as long as possible, the inevitable reversion to hiring the mean. Under normal circumstances, different teams will have different standards, and by the time you realize this you'll have a lot of teams whose standards have slipped, and you will have a lot of marginal employees.
Therefore smart large companies try to create a minimal process that aims to create uniform process across the whole company. The process tries to be lightweight as possible on individual people, but fights the good fight against entropy.
Unfortunately for Rachel, Google's process centers around the idea of having well-calibrated interviewers whose interview characteristics had been so well tested against others that hiring committees know exactly what to make of their interview feedback. (What absolute scale you grade on does not matter much. How your scale compares to everyone else, and how well it predicts everyone else, does.) Once an employee winds up as a well-calibrated interviewer, HR would like to use that person as much as possible (and each use does a better job calibrating the employee, and increases their desirability as an interviewer).
This is inconvenient for the employee to whom it happens. But, on average across all developers, the process is quite light-weight.
Instead, the formal interview processes I've seen are gauntlets of trivia and case studies and successive whiteboarding exercises that exhaust everybody. They don't generate value proportional to their cost. And they're unsustainable, so that whatever marginal effectiveness they have is compromised totally within a year when everyone who participates gets sick of it and starts phoning things in.
Google's process has been sustained company-wide for a number of years now. There is much that is imperfect about it, but evidence suggests that it is sustainable.
So first, I should be more careful. Google is operating at a scale that I don't understand. My experience tops out at ~100 employees. Let's stipulate that Google has a hiring problem that very few other companies have, and let me shift the focus back on to startup hiring, which is what I care about.
And then: I want my point to stand: for a given role, there are good hires and there are bad hires (where "bad" includes "meh"). There is not a lot of value in grading past that. But lots of hiring processes are premised on doing exactly that; they take candidates who can demonstrably perform the task required for the role and then beat them up with trivia questions, with puzzles, with nerd dominance quizzes, and with subjective stuff.
I point that out because if you can find a way to accept that, you can streamline a lot of recruiting process and make interviews less painful for everyone. The way I finally managed to make that leap of faith was to reflect back on everyone I've had a hand in hiring that didn't work out, and then consider the process that hired them. Then I think back on everyone I've hired who has surprised me with their awesomeness (some of them are even people I'd negged). I am now sold on the fact that most tech interviews are voodoo rituals.
What I would do/am doing instead:
* Get people out of HR/phone screening as quickly as possible.
* Have an explicit up-front call with me or someone senior in the hiring process to level-set people, try to defuse some of the tension, explain the process, and answer questions. Things go faster and more smoothly when candidates are convinced they want the role. The amount of sales effort it takes to get a candidate into the pipeline is not enough; you need to engage them on what the job is like, let them ask what the day-to-day is, and get them revved up before they interview.
* Minimize phone screening, both in length and in depth. I am on a mission to get rid of them altogether but am not convinced I'll succeed.
In the interim, what I'd like to do is take all the effort and expense we'd sink into hour-long phone screens and redirect it to coming up with a small rotating battery of really, really good questions that we standardize on.
* Replace phone screens with challenge problems. Calibrate them for the length of time a series of elaborate phone screens would take. Try to make them fun. Here I admit I'm not sure what I'd do back in my pure dev jobs to make this work, but I'm sure I could come up with something.
* Give the on-site interview team a full dossier of objective facts about what our up-front process learned about the candidate, so that nobody is trying to read tea leaves out of resumes. Candidly: I barely glance at resumes anymore, and I wince when I read interview reports that go "I read XXX on the candidate's resume and so asked YYY".
The most important thing I think I've learned about recruiting in the last year --- apart from getting better at getting people in the door --- is to generate sets of facts that are easy to compare between candidates. I'm learning a lot by reviewing the history of how people do on challenges, and I really very much want to be in the same place by the end of the year with all person-to-person screening interviews. A nice side effect I'm hoping for is for prep time on phone screens to approach zero, and for actual phone time for interviews to approach ~20-30 minutes. We'll see.
I don't know if you have read Thinking Fast and Slow but your observations are in strong agreement with research.
From what I have read (mostly Kahneman) most interviews as performed are useless and suffer from the illusion of validity. Instead, structured interviews and scoring that are the same for all candidates are strongly recommended. Which is, draw up a short list ~= 6 independent factors which from experience are reflective of the job and whose answers you can easily judge objectively. For each factor come up with a small number of questions with a score, say from 1 - 5 and then simply add up the scores. No fancy weighting required - research shows that regression does not add much to this simple model. To avoid the "halo effect" do not jump question categories. This will vastly improve upon the more common unstructured interview and though still far from perfect you will be hard pressed to improve on that short of an IQ or equivalent test.
Interestingly, and reminiscent to bagging in machine learning, unstructured interviews can be brought nearly up to par by averaging multiple independently scoring interviewer opinions.
I agree that there is a pretty binary "yes/no" choice to be made. However the fit between interviewer and interviewee is hugely variable, and you're never getting away from it. My suggestion for a small team, therefore, is to have a group interview. This probably does not reduce the interviewer/interviewee fit issue, but it does tend to leave people much more in agreement on candidates than when they interview separately, and therefore goes a long ways towards removing arguments over whether certain hires should have happened.
On standardization, there is one gotcha. It is much easier to calibrate a standardized interview process and compare this candidate to that one. However at some point you'll have a candidate who goes through your interview, and blogs the whole interview when they get home. At that point you'll find that every candidate who bothers to do a basic search is curiously well-prepared for your interview - at which point your interview's value drops sharply. At Google's size this happens constantly. For a random startup, not so often. But a hot startup should expect it.
Therefore I would recommend that to the rest of your list you add a monthly calendar entry to see what a search can turn up about your interview process so that you notice when it needs to change.
> demonstrably perform the task required for the role and then beat them up with trivia questions, with puzzles, with nerd dominance quizzes, and with subjective stuff
In this world, what happens when a person wants an internal transfer to another team with more rigorous requirements? Does he have to re-interview? Who controls the transfer process--HR or the other team? Will re-interviews be fair, given that the interviewee is being interviewed by people he already knows? What if it happens because of a re-org? Do you get fired if you fail the re-interview?
Companies like google interview to a higher standard that may be required for your initial job with the understanding that you may have to transfer, and firing/re-interviewing is much harder than just setting a higher bar to begin with.
Self-selection can potentially be powerful for a startup (although maybe less so recently), meaning that it may be possible to spend significantly more time considering each candidate. Skipping initial recruiter calls for at least a subset of candidates, for example, and having a hiring manager reach out and work on expectations setting and sell is something that might happen less often in a larger company.
Similarly, self-deselection is something you want to avoid - ie, you need to prove yourself to them a lot more. Making sure people you are interested in see the sorts of information they might need to consider working for you is important - maybe it's talking about your product/opportunity, your development/product/management/business philosophy, or even just mentioning who works with you and what they've done (although focusing on the VC team, or just saying "A bunch of former employees at Google, Apple, Microsoft, Facebook, Dropbox, ..." is probably not enough).
One thing I've found is that certain people are better at asking certain types of questions. I love "exploring" conversational-style interviews that are highly interactive and where not knowing something is an opportunity for the candidate to figure it out themselves or to learn a new thing. One colleague does an awesome debugging scenario that I sometimes think could continue for hours if he wanted to. I assume that certain candidates like particular interview styles - finding some way to match those two together would be great. (A recent Facebook start and I spent an additional half-hour during his interview because we both enjoyed the subject we were discussing. Making each interview that enjoyable...)
I'm not entirely sure what you mean by challenge problems - is this a technical phone interview with a specific set of broader topics/challenges, or is this something that gets sent to the candidate and they get some time to return a result?
I prefer the term "phone interview" when a non-recruiter/sourcer is evaluating the quality of the response to some question or challenge, and "phone screen" for the initial recruiting discussions where wholly unsuitable candidates are weeded out due to skill or experience mismatch to the role based more on treating commentary provided by the candidate as fact (although it does go a lot further than that).
Minimize phone screening, both in length and in depth. I am on a mission to get rid of them altogether
Phone screenings can be very useful to the candidate. I've taken the trouble to travel (via public transportation, so it took quite a few hours out of my day) to a potential employer only to find out within 10 minutes that there is no fit, and the lack of fit would have been obvious within the same 10 minutes of a phone interview.
There is a reason for the regimented process that you're talking about.
The challenge for any large company is to delay, for as long as possible, the inevitable reversion to hiring the mean. Under normal circumstances, different teams will have different standards, and by the time you realize this you'll have a lot of teams whose standards have slipped, and you will have a lot of marginal employees.
Therefore smart large companies try to create a minimal process that aims to create uniform process across the whole company. The process tries to be lightweight as possible on individual people, but fights the good fight against entropy.
Unfortunately for Rachel, Google's process centers around the idea of having well-calibrated interviewers whose interview characteristics had been so well tested against others that hiring committees know exactly what to make of their interview feedback. (What absolute scale you grade on does not matter much. How your scale compares to everyone else, and how well it predicts everyone else, does.) Once an employee winds up as a well-calibrated interviewer, HR would like to use that person as much as possible (and each use does a better job calibrating the employee, and increases their desirability as an interviewer).
This is inconvenient for the employee to whom it happens. But, on average across all developers, the process is quite light-weight.
Instead, the formal interview processes I've seen are gauntlets of trivia and case studies and successive whiteboarding exercises that exhaust everybody. They don't generate value proportional to their cost. And they're unsustainable, so that whatever marginal effectiveness they have is compromised totally within a year when everyone who participates gets sick of it and starts phoning things in.
Google's process has been sustained company-wide for a number of years now. There is much that is imperfect about it, but evidence suggests that it is sustainable.