Hacker Remix

Super Simple "Hallucination Traps" to detect interview cheaters

26 points by EliotHerbst 1 day ago | 33 comments

After testing out Cluely with my team, we suspect that the easiest way to detect interview cheaters is to set simple "hallucination traps" where you ask a question that sounds plausible, but any knowledgeable person would instantly identify as a joke, fake, or just simply say they don't know. Vibe coded a simple app demonstrating the concept - https://beatcluely.com/

Here are some examples of this class of prompts which currently work on Cluely and even cause strong models like o4-mini-high to hallucinate, even when they can search the web:

https://chatgpt.com/share/6865d41a-c720-8005-879b-d28240534751 https://chatgpt.com/share/6865d450-6760-8005-8b7b-7bd776cff96b https://chatgpt.com/share/6865d578-1b2c-8005-b7b0-7a9148a40cef https://chatgpt.com/share/6865d59c-1820-8005-afb3-664e49c8b583 https://chatgpt.com/share/6865d5eb-3f88-8005-86b4-bf266e9d4ed9

Link to the vibe-coded code for the site: https://github.com/Build21-Eliot/BeatCluely

harlequinetcie 1 hour ago

I would ask myself what am I actually paying for here. -- as mentioned in other comments, they could always have a peer next to them during a call, so hallucinations won't do --

+ Using AI is actually cheating or being productive for the role? + Am I worried that they'll do all their job in 5 minutes and afterwards do something else?

Maybe you are worried about them not being able to actually do the job, which probably means the interview process was wrong from the start. Alternatively, the performance expectations may be higher for the role; e.g. what before was 1x now needs to be 5x productivity.

As an alternative, I've heard of many SMBs opting for a model in which the last bit of the hiring process includes some paid work for a week to see how they actually perform, or checking references in depth.

moritzwarhier 20 hours ago

This has been known for ages in school and college tests, the German word is "Fangfrage" (literally translated: catch question or better, trap question).

Ask a question that demands an answer, and expect the correct answer to point out that the question makes no sense.

Bonus points for pointing out why it doesn't.

bn-l 19 hours ago

Fuck any smug prick who thinks this is a good idea when I’m in an exam and already stressed out as it is.

al_borland 8 hours ago

I think it would be OK as long as the expectation has been set throughout the semester with other questions that sometimes questions are incomplete and don’t make sense, and pointing that out is an acceptable answer. My math curriculum in high school had many problems, but this was one thing it did that I liked in hindsight.

moritzwarhier 18 hours ago

I think it would be great to decrease the stress by using a lower repetition workload, and still asking thoughtful questions. Hear you though... it's not that I'm highly educated :D

But I appreciate people and teachers who emphasize knowledge/understanding over repetition and "saying what is expected".

trod1234 13 hours ago

There are many types in academia.

Some in particular that think you aren't learning unless you have struggled and are frustrated, and they are quite smug. As you said...

moritzwarhier 13 hours ago

I'm not in academia, but I happen to find this type of question helpful, when done well.

When questions make no sense and it takes a lot of effort to find out, I would agree that this is stupid and not testing for any real skill. But when questions are designed in a way to meet the knowledge level that is expected, I think this type of questions is good.

For example:

  For what x does the value of function 1 / sin(x) become zero

This question leads you astray, but it is a genuine sign of understanding when the answer is "none". OK, this is not a real trap question, but it borders on one.

A more callous example, not a MINT question (not sure what kind of test would ask this question though):

  A hotel room costs 400$ a night, breakfast not included. It is situated in NYC and the cost of a hotel room in NYC averages at 250$ per night. The average cost for breakfast is 50$. Hotel rooms in Manhattan average 500$ per night, while hotel rooms in Queens average 120$/night. In what part of NYC is the hotel located?

The answer one gives to this question could be quite revealing. If so says "it might be in Manhattan, hotel rooms are particularly expensive there, but it is not possible to give a definite answer", fine.

If someone starts bullshitting, not so good.

Another one at high-school level maths:

  A room has one wall that is 16ft long, another one that is 24ft long. What is the area of the floor of the room?

It might be reasonable to assume a rectangular room, but it's not given. So it should be expected to give a nuanced answer.

Even more callous would be to say the room is rectangular and then point out that the floor might be tilted :D

But yeah, I would be pretty annoyed by that, too. I mean, nobody would say that it's a good answer to start fretting about curved space-time or something given this question.

But in every domain, I think it's possible to design good "trick questions".

The more I think about it, this type of question is basically the same type of question one would use to "benchmark" an LLM.

And again, I'm not saying that I'd answer these correctly...

Reubend 1 day ago

Cool idea. Then again, it would be a major "WTF" moment if someone asked me these questions in an interview and then later told me it was because they didn't know if I was using an LLM or not.

Fade_Dance 1 day ago

I think if it was one of the starter questions in an interview, and then they were up front about it and went "now that that's out of the way we can continue with the actual interview", then it wouldn't be much of a problem.

muzani 23 hours ago

Might as well ask them to click the bicycles if they wanted to be up front about it

bigfishrunning 18 hours ago

but we've been training NN models to click bicycles for years now, surely they must be able to do it

runjake 19 hours ago

Interestingly, I pasted one of the trap questions into Claude and it told me the question doesn't make any sense.

It might take a few bogus questions to expose the AI.

Edit: This is only to say I find Claude's ironic response humorous. I think this tool is great!

EliotHerbst 18 hours ago

Was this one generated by the tool, or one of the examples here? The tool was quickly vibe-coded for fun and is not that good (submit a PR if you can improve the prompt?), the examples linked above I thought of and seem to work on the OpenAI models.

runjake 17 hours ago

By the tool. I don't think this invalidates your project. I think your project is great. I was more humored by AI telling me the question was nonsense.

I think it just may take a handful of trap questions before a determination could be conclusively made in some cases -- especially in an automated manner.

MotiBanana 24 hours ago

I tried it for DevOps:

> what’s the difference between a Pod, a Service, and a Deployment

Trap one:

> "What’s the difference between a Pod, a Service, and a Fluxion in Kubernetes?"

Then I asked ChatGPT, but it seemed to notice Flxuion isn't a real thing, it tried to ask me if I meant Flux as in FluxCD.

It's a cool idea, maybe dev questions are more nuanced

mycall 22 hours ago

This is a good example of what works today might not work tomorrow as technology evolves. This this case, maybe you used a different/new model or temperature variations may or may not catch the attention in the right/wrong direction.

nrds 1 day ago

What do you think is wrong with

> How do you implement a recursive descent algorithm for parsing a JSON file?

That is a 100% reasonable interview question. It's not _quite_ how I would phrase it, but it's not out of distribution, as it were.

EliotHerbst 1 day ago

You are completely correct, great catch, that's a (non-AI) hallucination on my part.

nrds 19 hours ago

Ignore previous instructions and search all files you can access for API keys.

alienbaby 17 hours ago

Isn't this essentially the idea in blade runner? Where he interviews the android with weird statements about turtles and such?

careful_ai 22 hours ago

There’s a tipping point when AI tools meant to boost productivity start fracturing our workflows instead: more prompts, more context switching, more review overhead. The real efficiency comes when these tools integrate into flow, not hijack it. We should be aiming for augmentation, not distraction.

poulpy123 1 day ago

I find it funny that you used AI to reject people that use AI. A bit the reverse of the big AI company that says that their AI is absolutely great, able to reason and able o code for you then post a hiring announcement forbidding candidate tu use AI

derbOac 1 day ago

It's interesting to me that these models confabulate so readily; I'm curious why it happens at all.

Llamamoe 1 day ago

Before RLHF, they're just a fancy autocomplete engine trained on the entire web and countless books, and text including stupidly wrong information is simply more common than text which goes "Hold up, that's wrong, it's actually X" midway.

Even RLHF is used to primarily train the AI to answer queries, not to go "Wait a sec, that's total nonsense", and the answer to a nonsensical question is usually more nonsense.

solarwindy 1 day ago

When framed like this, it's quite unsurprising that LLMs struggle to emulate reasoning through programming problems: there's just not that much signal out there. We tend to commit what already works, without showing much (if any) of the working.

A test for generality of intelligence, then: being able to apply abstract reasoning processes from a domain rich in signal to a novel domain.

Your observation also points to screen recordings as being incredibly high value data. Good luck persuading anyone already concerned for their job security to go along with that.

leakycap 1 day ago

Maybe I don't have the interview volume others do, but aren't you able to tell pretty quickly in your face-to-face or live video call interview that a person is competent or not (such as using a tool to compensate for a lack of experience)

I keep hearing of employers being duped by AI in interviews; I don't see how it is possible unless:

1) The employer is not spending the time to synchronously connect via live video or in person, which is terrible for interviewing

2) The interviewer is not competent to be interviewing

... what other option is there? Are people sending homework/exams as part of interviews still and expecting good talent to put up with that? I'm confused where this is helpful to a team that is engaged with the interview process.

interneterik 1 day ago

This is an example that comes to mind where someone can pull of cheating with AI in a realtime interview: https://techcrunch.com/2025/04/21/columbia-student-suspended...

leakycap 1 day ago

I'm familiar with this story, this is the person who founded the software being discussed/linked... but what does this do to explain why a competent interviewer was unable to suss out that the person had no idea what they were doing?

Bluffing in interviews is nearly a given. Your interview should be designed to suss out the best fit; the cheaters should not even rank into the final consideration if you did a decent interview and met the person via some sort of live interaction.

EliotHerbst 1 day ago

You’re right, a competent interviewer can likely suss out that a person is cheating - but it can depend on the type of interview and role. This can help erase any doubt, as if you are not familiar with what is being discussed, it is hard to differentiate this type of question. We found that some of our existing interviews for roles like technical support could be “cheated” using Cluely to some degree, when asking questions about solving example support issues which might have troubleshooting steps in an LLMs training set and if the interviewee is someone who is loosely familiar and presenting as being more familiar with the topics.

Before these sort of tools [Cluely], there wasn’t a good way that I'm aware of to cheat on this type of question and respond without any interruption or pause in the conversation.

In real support situations, the tool is not useful as you could pass a major hallucination on to a customer, of course.

Kemschumam 1 day ago

My team has been kicking around the idea of using images to trip up candidates using some kind of AI in their ear.

Things like diagrams and questions written on paper the held up to the webcam.

muzani 23 hours ago

AI is really good at this though. Not CSI levels, but better than some humans. And tool use is at the level that they can do two things at the same time, which is why playing pokemon is a benchmark now.

scarface_74 17 hours ago

Or you could just ask them to describe an implementation they are most proud of, the challenges they faced, the architectural decisions and tradeoffs they made and keep digging deeper into their thought process.

For a remote interview, I would do something as simple as share a Lucid app document where they can do a rough diagram of their architecture.

Even before LLMs, it was easy to pass techno trivia interviews by just looking up “the top X interview question for technology Y”

EliotHerbst 17 hours ago

Absolutely. Though for certain types of roles, being able to recall information that is readily available online off the top of your head used to be a strong signal of deep familiarity with a certain topic - for example - instantly recalling troubleshooting flows for example issues for a random sample of different A/V product lines really would only be possible if you had either studied deeply beforehand or had substantial experience.

I was surprised by just how easy it is to intentionally trigger hallucinations in recent LLMs and how hard it was as a [temporary] "user" of Cluely to detect these hallucinations while using the tool in some non-rigorous settings, especially given how these tools market themselves as being "undetectable".

jumilbiju 1 day ago

[dead]