this post was submitted on 20 Jun 2024
471 points (89.7% liked)
Technology
59674 readers
3085 users here now
This is a most excellent place for technology news and articles.
Our Rules
- Follow the lemmy.world rules.
- Only tech related content.
- Be excellent to each another!
- Mod approved content bots can post up to 10 articles per day.
- Threads asking for personal tech support may be deleted.
- Politics threads may be removed.
- No memes allowed as posts, OK to post as comments.
- Only approved bots from the list below, to ask if your bot can be added please contact us.
- Check for duplicates before posting, duplicates may be removed
Approved Bots
founded 1 year ago
MODERATORS
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
I interviewed a candidate for a senior role, and they asked if they could use AI tools. I told them to use whatever they normally would, I only care that they get a working answer and that they can explain the code to me.
The problem was fairly basic, something like randomly generate two points and find the distance between them, and we had given them the details (e.g. distance is a straight line). They used AI, which went well until it generated the Manhattan distance instead of the Pythagorean theorem. They didn't correct it, so we pointed it out and gave them the equation (totally fine, most people forget it under pressure). Anyway, they refactored the code and used AI again to make the same mistake, didn't catch it, and we ended up pointing it out again.
Anyway, at the end of the challenge, we asked them how confident they felt about the code and what they'd need to do to feel more confident (nudge toward unit testing). They said their code was 100% correct and they'd be ready to ship it.
They didn't pass the interview.
And that's generally my opinion about AI in general, it's probably making you stupider.
Wait wait wait so... this person forgot the pythagorean theorem?
Like that is the most basic task. It's
d = sqrt((x1 - x2)^2 + (y1 - y2)^2), right?That was off the top of my head, this person didn't understand that? Do I get a job now?
I have seen a lot of programmers talk about how much time it saves them. It's entirely possible it makes them very fast at making garbage code. One thing I've known for a long time is that understanding code is much harder than writing it, and so asking an LLM to generate your code sounds like it's just creating harder work for you, unless you don't care about getting it right.
Yup, you're hired as whatever position you want. :)
Our instructions were basically:
It was technically different (we phrased it as a top-down game, but same gist). AI generated manhattan distance (
abs(x2 - x1) + abs(x2 - x1)) probably due to other clues in the text, but the instructions were clear. The candidate didn't notice what it was doing, we pointed it out, then they asked for the algorithm, which we provided.Our better candidates remember the equation like you did. But we don't require it, since not all applicants finished college (this one did). We're more concerned about code structure, asking proper questions, and software design process, but math knowledge is cool too (we do a bit of that).
College? Pythagorean Theorem is mid-level high school math.
I did once talk to a high school math teacher about a graphics program I was hacking away on at the time, and she was surprised that I actually use the stuff she teaches. Which is to say that I wouldn't expect most programmers to know it exactly off the top of their head, but I would expect they've been exposed to it and can look it up if needed. I happen to have it pretty well ingrained in my brain.
Yes, you learn it in the context of finding the hypotenuse of a triangle, but:
So when I'm interviewing, I try to provide things like algorithms that they probably know but are likely to space on, and focus on the part I care about: can they reason their way through a problem and produce working code, and then turn around and review their code. Programming is mostly googling stuff (APIs, algorithms, etc), I want to know if they can google the right stuff.
And yeah, we let applicants look stuff up, we just short circuit the less important stuff so they have time to show us the important parts. We dedicate 20-30 min to coding (up to an hour if they rocked at questions and are struggling on code), and we expect a working solution and for them to ask questions about vague requirements. It's a software engineering test, not a math test.
Yeah, that's absolutely fair, and it's a bit snobby of me to get all up in arms about forgetting a formula - although it is high school level where I live. But to be handed the formula, informed that there's an issue and still not fix it is the really hard part to wrap my head around, given it's such a basic formula.
I guess I'm also remembering someone I knew who got a programming job off the back of someone else's portfolio, who absolutely couldn't program to save their life and revealed that to me in a glaring way when I was trying to help them out. It just makes me think of that study that was done that suggested that there might be a "programmer brain" that you either have or you don't. They ended up costing that company a lot to my knowledge.