What to Do If Hiring Feels AI Capability Is Non-Negotiable Now. Most Leaders Are Testing for It Badly.

I want to talk about a question that keeps coming up in interviews across the Australian tech market, because I think most panels are asking it the wrong way.

The question is some version of “What AI tools did you use in that role?”

On the surface, it sounds sensible. AI capability has quietly become non-negotiable in senior tech hiring. Twelve months ago, it was a nice-to-have on a job spec. Now, when I take a brief for a senior engineering, QA or data role, the client almost always wants someone who can work with AI in some form. Fair enough. The market has moved.

But here is the problem, and it showed up on a screening call I ran recently.

I was speaking with an experienced quality engineer. Sharp, capable, honest. They told me about a recent interview where the panel asked exactly that question: which AI testing tools did you use in your last role? Their honest answer was that there were none. The environment did not allow them, so the work was done manually, and done well.

In a different role, at a different company, the same person was told the opposite. Leadership there said to fully embrace AI, do what you want. So they did. They used AI assistants to build out automation suites and moved faster than they ever had before.

Same person. Same capability. Two completely different answers to the tool question, depending entirely on the environment their employer created.

That is the bit most interview panels miss. When you ask “what AI tools did you use,” you are mostly measuring the previous employer’s budget, risk appetite and permission settings. You are not measuring the person sitting in front of you. And you are quietly screening out good people whose last workplace happened to be conservative.

The second thing I keep seeing makes this even more important.

Across my client conversations this year, more and more companies are building internal tools with AI coding assistants rather than buying off-the-shelf software. Small teams are standing up their own integrations, their own APIs, their own workflow tools. The build-versus-buy equation has genuinely shifted, and for a lot of mid-sized businesses, it makes sense.

But when I ask who is testing this stuff, the room usually goes quiet.

Because here is the uncomfortable truth: it is very difficult to use AI to test and verify software if you do not have the underlying knowledge yourself. I know plenty of people right now who know enough to be dangerous. They can prompt their way to something that works on the surface. What they cannot do is tell you whether it is secure, whether it will hold up under load, or where it will quietly fail. The scarce skill in this market is not prompting. It is the experience and judgment to know when the machine is wrong.

So AI capability is non-negotiable, yes. But the capability that matters is fundamentals plus AI leverage, not AI leverage on its own.

What does that mean practically, if you are hiring or leading a tech team?

First, change the question. Instead of “what AI tools have you used,” try “tell me about a time AI gave you something wrong, and how you caught it.” Or “what have you refused to ship that AI produced?” Those questions test judgment, and judgment travels between environments. Tool lists do not.

Second, remember that AI capability shows up where leaders give permission for it. If your own environment locks everything down, do not be surprised when your team looks flat-footed next to competitors. And do not penalise a candidate for a previous employer’s caution. Ask what they would do with permission, then listen to how quickly the ideas come.

Third, weight fundamentals more heavily, not less. The people who will get the most out of AI over the next few years are the ones who deeply understand the craft underneath it. A tester who understands how systems break. An engineer who understands security. A leader who knows what good looks like without needing the tool to tell them.

The market has decided AI capability is table stakes. The leaders who win from here will be the ones who can actually recognise it when it walks into the room.

If your interview panel is still asking for a tool list, that is worth a conversation this week.

See you next Friday.

James