AI killed the tech interview. Now what?

> i.e. if you wag your tail chasing a bad end for 15 mins, this is a fail in an interview

In all pair programming interviews I have run (which I will admit have been only a few) I would fail myself as an interviewer if I was not able to guide the interviewee away from a dead end within 15 minutes.

If the candidate wasn't able to understand the hints I was giving them, or just kept driving forward, then they would fail.

That's definitely up to the interviewer, in which a lot of discretion and trust has been placed. I think a lot of it also comes down to the culture of the company—whether they're cutthroat or supportive. As you get better people into the company, hopefully this improves over time. I know that when we did it, it was never about nailing it on the first try, it was literally about proving you knew how to program and were not an asshole. So, not the equivalent of reversing a binary tree on a whiteboard. The kinds of problems we worked on in the interviews weren't leetcode type problems, they were real tickets from our current project. Sometimes it was just doing stuff like making a new component or closing a bug, but those were the things we really did, so it felt like a better test.

Hiring doesn't scale, period; deal with it.

> i.e. if you wag your tail chasing a bad end for 15 mins, this is a fail in an interview

That’s an assumption. Perhaps following a dead end for a while, realizing it, pivoting, etc is a valuable, positive, signal?

I do 1hr pair programming interviews for my company and you have to strike a balance between letting candidates think through the problem even when you think it won't work (to see their thought process and maybe be surprised at their approach working/see how quickly they can self-correct) and keeping them on track so that the interview still provides a good signal for candidates who are less familiar with that specific task/stack.

I'm also not actually testing for pair programming ability directly, moreso ability to complete practical tasks / work in a specific area, collaborate, and communicate. If you choose a problem that is general/expandable enough that good candidates for the position are unlikely to go down bad rabbit holes (eg for a senior fullstack role, create a minimal frontend and api server that talk to each other) it works just fine. Actually with these kinds of problems it's kind of good if your candidates end up "studying" them like with leetcode, because it means they are just learning how to do the things that they'll do on the job.

> maybe even on an actual feature

I don't think this would work unless the feature were entirely self-contained. If your workaround is to give the candidate an OSS project they need to study beforehand, I think that would bias candidates' performance in ways that aren't aligned with who you want to hire (eg how desperate are they for the role and how much time outside of work are they willing to put into your interview).

Another problem is it is difficult to compare candidates whose interviews involved working on completely different problems.

> if you wag your tail chasing a bad end for 15 mins, this is a fail in an interview

Eh, if it's a reasonable bad end and you communicate through it, I wouldn't see it as a fail. Particularly if it's something I would have tried, too. (If it's something I wouldn't have thought of and it's a good idea, you're hired.)

I did a couple of rounds of this with my manager as the interviewer. Personally I really liked the process, and the feedback I got from the candidates was positive (but then again it always would be).

What worked well for me was that I made it very clear to my manager, a man who I trust, that I would not be able to provide him with a boolean pass/fail result. I couldn't provide him any objective measure of their ability or performance. What I could do was hang out with the canditates for an hour while we discussed some concepts I thought were important in my position. From that conversation I would be able to provide him a ranking along with a personal evaluation on whether I would personally like to work with the candidate.

I prepared some example problems that I worked for myself a bit. Then I went into the interviews with those problems and let them direct those same explorations of the problem. Some of them took me on detours I hadn't taken on my own. Some of them needed a little nudge at times. I never took specific notes, but just allowed my brain to get a natural impression of the person. I was there to get to know them, not administer an exam.

I feel like the whole experience worked super well. It felt casual, but also very telling. It was almost like a focused date. Afterwards I discussed my impression of the candidates with my manager to ensure the things I was weighing was somewhat commutable to what he desired.

All in all it was a very human process. It must have taken an enormous amount of trust from my manager to allow me the discretion to make a subjective judgment. I was extremely surprised at how clearly I was able to delineate the people, but also how that delineation shifted depending on which axis we evaluated. A simple pass/fail technical interview would have missed that image of a full person.

I hear this all the time, but I have yet to experience it. It may be because the small companies that I interview with are all startups, but I have yet to be able to get a call back from any other kind of small company. And the startups I do interview with have a full FAANG interview loops.

There seems to be a weird selection bias that if you're FAANG or FAANG adjacent these small companies aren't interested.

What exactly does "scalable" mean here?

If a startup can spend 20 man-hours filling a single position, why can't a big company spend 1000 man-hours filling 50 positions?

> small companies and startups are great at interviewing

Small companies have the benefit of the pressure to fill a role to get work done, the lack of bureaucratic baggage to "protect" the company from bad hires, and generally don't have enough staff to suffer empire-building.

Somewhere along the line the question changes from "can this candidate do the job that other people in this office are already doing?" to "can this candidate do the job of this imaginary archetype I've invented with seemingly impossible qualities".

You need to run through hundreds of candidates to find someone marginally qualified. I am not exaggerating.

Lightly ironic indeed! Though I’m not sure “broken” is exactly the word I’d choose.

I can only speak for myself, but I imagine myself as a candidate approaching a “couple of hours” project or relationship differently than I would a “few minutes” speed round. For that matter I can think of people I know professionally who I only know through their behavior “on stage” in structured and stylized meetings of a half hour or an hour—and I don’t feel like I have any sense at all of how they would be as day-to-day coworkers.

If we sat down to work together, you’d probably have a sense in the first few minutes of whether or not we were going to work out—but that would be contingent on us going into it with the attitude that we were working together as colleagues toward a goal.

That's mainly because there were multiple pairing sessions, and even if you knew the person was going to pass, there are still a couple more people who need to meet them, and a schedule to make sure they're available to do that. Plus due diligence, etc.

Nor am I saying it was a perfect system, just the best I've seen in terms of results.

Anecdote:

I've been interviewing recently and got through to the last round (of five...) with an interesting company. I knew the interview involved pairing, but I didn't expect: two people sitting behind me while I sat on a wobbly stool at a standing desk, trying to use a sticky wired mouse and a non-UK keyboard, and a very bright monitor (I normally use dark mode). They had a screen recorder running, and a radio was on in the next room.

I totally bombed it. I suspect just the two people behind me would have been enough though.

I would find trying to solve such problems with known people in known environments to be somewhat stressful too.

We didn't do assessments (if by that you mean take home assignments). This was partly a solution to that, since nobody thought they were a good idea. If you mean the phone screen, I think that would be a problem, yep, but it wasn't an issue back in 2016. Having them pair would weed out cheaters, but we would have to figure out a way to weed them out during the screening, I agree.

We also did not require the employers doing the interview to be our most senior team members. They probably did it more often than most people, but often because they volunteered to do it. Anyone on the team would be part of the loop, which helped with scheduling. And, remember, we were working on actual tickets, so in a lot of cases it actually helped having the candidate there as a pairing partner.

For a little extra detail, the way we actually did it was to have 2-3 pairing sessions of up to 2 hours apiece. At the end of the day, all the team members who paired with the candidate had to give them the thumbs up.

Use another AI, of course!

Idk if I'm even being sarcastic here.

You can do code review exercises without larger context.

An example from the interview: the code included a python web API and SQL schema. Some obvious points I noticed were no input validation, string concatenating for the database access (SQL injection), no input scrubbing (XSS), based on the call pattern there were some missing indices, a few bad data type choices (e.g. integer for user ID), a possible infinite loop in one case.

You might be thinking about it in the wrong way; what you want to see is that someone can spot these types of logic errors that either a human or AI copilot might produce regardless of the larger context.

The juniors will find formatting and obvious bad practices; the senior and staff will find the real gems. This format works really well for stratification.

It's not so hard. One of the interview stages I did somewhere well known used this.

Here's the neural net model your colleague sent you. They say it's meant to do ABC, but they found limitation XYZ. What is going on? What changes would you suggest and why?

Was actually a decent combined knowledge + code question.

100% it is more conducive to a conversational exchange that actually gives you better insight into how a developer thinks much more so than leetcode.

Coding for me is an intensely focused activity and I work from home to boot so most of the time, I'm coding in complete silence. It's very awkward to be talking about my thought process while I'm coding, but not talking is just as awkward!

Even more relevant now that we are in the age of code generation as the powertool for productivity.

Honestly this sounds like a successful "bad fit" signal (assuming that they work with go and payment systems mostly).

Language and domain experience are things id like to know after an interview process.

    > A cold code review on a codebase they never saw

So yeah, I think it's the opposite: explicitly testing for their ability to read code is probably kinda important.

Screen Studio on Mac.

(OBS Elements other times)

> Is there a site where one could review some code and see what many others say about it and their experience level?

https://codereview.stackexchange.com

This could be like rap genius for code!

Not that I know of!

It would be interesting, but I agree it would need to be content moderated to some extent.

github and issues page

Yup, that's just one of the many ways to do a code-review interview wrong.

Each code sample should have multiple things wrong. The best people will find most (not necessarily all) of them. The mediocre will find a few.

Yeah it's really tempting when you discover an interesting fact to think "that would make an interesting interview question" and turn the interview into some kind of pub quiz. Happens with all forms of technical interview though. I mean 90% of leetcode questions are "this one weird trick".

>hiring has been shifted to HR

Well, unless you know sufficiently senior people. But I suspect that is a deeply unsatisfactory answer to many people in this forum.

My long term last, only technically-adjacent, job came through a combination of knowing execs, having gone to the same school as my ultimate manager, and knowing various other people involved. (And having a portfolio of public work.)

I saw this at the big corporate (not faang/tech) place I work at. Engineers run and score interviews, but we don't make the final decision. That goes to HR and the hiring manager who usually has no technically background.

HR are experts in HR, which is to say they have a broader view of the institutional needs and legal requirements of hiring staffing than you do. It's always annoying when that clashes with your vision, but dismissing their entire domain is unlikely to help you avoid running into that dynamic again and again

> hiring has been shifted to HR

Not everywhere. At my company, HR owns the process but we -- the hiring tech team -- own the content of interviews and the outcomes.

I've never seen hiring completely in the domain of HR. HR filters incoming candidates and checks for culture fit etc, but technical competency is checked by engineers/ML folks. I can't imagine an HR person checking if someone understands neural networks.

> In an hour-long interview, I have to be able to state the problem in a way the candidate can understand, within 10 minutes or so. We don't have time for a lecture on the intricacies of voucher calculation and global sales tax law.

If you send me a rubric I can pre-load whatever you want to talk about. If you tell me what you're trying to build and what you need help with, I can show up with a game plan.

You need to make time for a conversation on the intricacies of voucher calculation and global sales tax law if you want to find people jazzed about the problem space.

> In an hour-long interview, I have to be able to state the problem in a way the candidate can understand, within 10 minutes or so. We don't have time for a lecture on the intricacies of voucher calculation and global sales tax law.

Proving if they are technically capable of a job seems rather silly. Look at their resume, look at their online works, ask them questions about it. Use probing questions to understand the depths of their knowledge. I don't get why we are over-engineering interviews. If I have 10+ years of experience with some proof through chatting that I am, in fact, a professional software engineer, isn't that enough?

>Interview coding questions aren't like the day-to-day job, because of the nature of an interview.

You have missed his point. If the interview questions are such that AI can solve them, they are the wrong questions being asked, by definition. Unless that company is trying to hire a robot, of course.

“Real problems” aren’t something that can be effectively discussed in the time span of an interview, so companies concoct unreal problems that are meant to be good indicators.

This is the answer.

Let's not pretend 95% of companies are asking asinine interview questions (though I understand the reasons why) that LLMs can easily solve.

Plenty of companies administer IQ tests. The reason everyone doesn't is that it doesn't work well.

A lot of companies have IQ like tests, in particular big consulting companies like McKinsey and so on.

I work for a faang subsidiary. We pay well below average salary and equity. We finally got one nice perk, a very good 401k match. A few months later it was announced that the 401k match would be scaled back "to come in line with what our parent company offers". I thought about asking "will be getting salaries or equity in line with what our parent company offers?" but that would have been useless. Management doesn't care. I'm job hunting.

Oh man I needed that in the clip for like a dozen interviews a decade ago.

This zinger I have to remember for the next time someone tries this whiteboard BS on me!

"How many google shares do I get?"

> We CANT test you effectively on your programming skillbase. So we test on a more relevant job skill, like can you have a real conversation (with a whiteboard to help) about how to solve the problem.

Except, that's not what happens. In basically every coding interview in my life, it's been a gauntlet: code this leetcode medium/hard problem while singing and tapdancing backwards. Screw up in any way -- or worse (and also commonly) miss the obscure trick that brings the solution to the next level of algorithmic complexity -- and your interview day is over. And it's only gotten worse over time, in that nowadays, interviewers start with the leetcode medium as the "warmup exercise". That's nuts.

It's not a one off. The people doing these interviews either don't know what they're supposed to be looking for, or they're at a big tech company and their mandate is to be a severe winnowing function.

> It isn't that your interviewer knew all the languages, but that the language didn't matter.

I've done enough programming interviews to know that using even a marginally exotic language (like, say, Ruby) will drastically reduce your success rate. You either use a language that your interviewer knows well, or you're adding a level of friction that will hurt you. Interviewers love to say that language doesn't matter, but in practice, if they can't know that you're not making up the syntax, then it dials up the skepticism level.

> can you have a real conversation (with a whiteboard to help) about how to solve the problem

And do you frame the problem like that when giving interviews? Or the candidates are led to believe working code is expected?

> The goal isn't to "solve the puzzle" but instead to demonstrate you can reason about it effectively, communicate your knowledge and communicate as part of problem solving.

...while being closely monitored in a high-stakes performance in front of an audience of strangers judging them critically.

> So we test on a more relevant job skill, like can you have a real conversation (with a whiteboard to help) about how to solve the problem.

Everybody says that, but reality is they don't imho. If you don't pass the pet question quiz "they don't know how to program" or are a "faker", etc.

I've seen this over and over and if you want to test a real conversation you can ask about their experience. (I realize the challenge with that is young interviewers aren't able to do that very well with more experienced people.)

+1 to all this. It still surprises me how many people, even after being in the industry for years, think the goal of any interview is to “write the best code” or “get the right answer”.

What I want to know from an interview is if you can be presented an abstract problem and collaboratively work with others on it. After that, getting the “right” answer to my contrived interview question is barely even icing on the cake.

If you complain about having to have a discussion about how to solve the problem, I no longer care about actually solving the problem, because you’ve already failed the test.

And my axe…

Hehe, I have to remember to bring one of my custom Forths to the next interview.

And if the Principal is good, they should stand up and say exactly why the code is bad. If there's a reason to laugh because it is cliche bad, they should say so.

If someone gave me code with

if (x = 7) { ... } as part of a C eval.

Yeah, you'll get a sarcastic response back because I know it is testing code.

What I think people ignore is that personality matters. Especially at the higher levels. If you are a Principal SWE you have to be able to stand up to a CEO and say "No, sir. I think you are wrong. This is why." In a diplomatic way. Or sometimes. Less than diplomatic, depending on the CEO.

One manager that hired me was trying to figure me out. So he said (and I think he was honest at the time). "You got the job as long an you aren't an axe murderer."

To which I replied deadpan: "I hope I hid the axe well." (To be clear to all reading, I have never killed someone, nevermind with an axe! Hi FBI, NSA, CIA and pals!)

Got the job, and we got along great, I operated as his right hand.

I’ve worked at a company with 150,000 employees. The interview process was pretty much as described here. There is absolutely no reason a Big Co needs to operate any differently.

With each Sprint, presumably.

Most people have still written code for school or a hobby project. Maybe I'm missing empathy, but I cannot understand how some developers have no code to show.

If that's the case however, just let them make a small project over the weekend and then do another interview where you ask stuff about what they've made. It's not that deep

My worst code is always what I wrote yesterday. Often what’s missing is context, unless I comment ad nauseam. Sure I didn’t write complete test, obey open closed principles abstract into factory functions. The code I send from my hobby projects is likely a mess, because finishing on my own time by my own unpaid constraints wills it to be so

Maybe you forked a library because of reasons. You can tour the original repo and explain the problems. I have at least one of those examples for each time the legal or confidentiality department stepped in.

> We do this too, works fine. We ask open ended questions like, "What's your favorite thing you've done in your career and why?" and "What was the most challenging project in your career and why?" If you listen, you can get a lot of insight from just those two questions. If they don't give enough detail, we'll probe a little.

The problem is: there is a very negative incentive to give honest answers. If I were to answer these questions honestly, I'd bring up some very interesting theorems (related to some deep algorithmic topics) that I proved in my PhD thesis. Yes, I would have loved to stay in academia, but I switched to industry because of the bad job prospects in academia - this is not what interviewers want to hear. :-(

> "What's the difference between a function and a procedure." It's a one sentence answer

The terminology here differs quite a lot in different "programming communities". For example

> https://en.wikipedia.org/w/index.php?title=Procedure&oldid=1...

says: "Procedure (computer science), also termed a subroutine, function, or subprogram",

i.e. there is no difference. On the other hand, Pascal programmers strongly distinguish between functions and procedures; here functions return a value, but procedures don't. Programmers who are more attracted to type theory (think Haskell) would rather consider "procedures" to be functions returning a unit type. If you rather come from a database programming background, (stored) procedures vs functions are quite different concepts.

I could go on and on. What I want to point out is that this topic is much more subtle than a "one sentence answer".

Here's an interesting thought on your "gotcha" - I'm 57 years old, been programming as a career for over 30 years, a lot of languages and I have no idea what the difference is.

> Our "gotcha," which doesn't apply to most languages anymore is, "What's the difference between a function and a procedure."

My answer would be along the lines of "It's 2025, no one has talked about procedures for 20+ years"

Yes that's a tricky one.

When I'm interviewing a candidate, I'm often asking myself if this question is just something I happen to know therefor expect the candidate to know too, or if it's crucial to doing the job?

Sometimes it may not be fair to expect a random developer to be familiar with a specific concept. But at the same time it might be critical to the kind of work we're doing.

Or maybe join forces and show them how it's really done?

Not that I mind growing tomatoes, quite the opposite :)

Part of the problem is there just aren't a lot of people out there who can correctly judge that level of experience, and looking up the spectrum tends to simply look weird.

> My experience differs a lot. Many insanely skilled people are somewhat "weird" (including possibly

I am actually a bit weird myself, so I can relate.

> What you consider to be "skilled people" is what I would rather call "skilled self-promoters". "Skilled people" and "skilled self-promoter" are quite different breeds of people.

I don't mean that they have told me that they are skilled, or that their resume has implied it. I mean that they actually have the skills. Self-promoters that don't know the information always look good on paper, but after a few minutes of talking to them you can tell that they don't quite match.

Before IT, I was a live sound engineer TV, theater, music. There was also a entertainment university starting up around the same time. They were pumping out tons of "trained" engineers that looked good on paper but couldn't mix for shit. I think we can blame them for the shitification of pop music.

Thank you. I took it as both. :)

busybox: so many tools bundled into one binary

We hired the guy who said it was a dump question, he became our manager. He then decided to retire and we had to replace him. One of the candidates answered the 5 cli question with terminal eye candy, not functional commands. He was not hired for the job.

Candidate would be compensated, obviously. That's why it's expensive.

You don't need him to become efficient. Also I don't think it is always necessary to have such long onboarding. I'll never understand why a new hire (at least in senior position) can't start contributing after a week.

If you work with Boring Technology, your onboarding process has no reason to be longer than a week, unless you're trying to make the non-tech parts of the role too interesting.

Sure, psychological research is hard because of this, but that's not what I'm proposing - I'm talking about just having some data on predictive validity of the hiring process. If there's some coding test: is it reliable and valid? Aren't some items redundant because they're too easy or too hard? Which items have the best discrimination parameter? How the total scores correlate with e.g. length of the test takers tenures?

Sure, the confidence intervals will be wide, but it doesn't matter, even noisy data are better than no data.

Maybe some companies already do this, but I didn't see it (though my sample is small).