Most HR teams have spent a decent amount of time building a governance framework around AI use. They’ve:
- Drafted acceptable use policies
- Set rules about what data employees can feed into large language models (LLMs)
- Considered the risk of bias in AI-assisted hiring decisions
- Explored how they’ll protect the security of sensitive data when it’s exposed to third-party tools.
That scepticism is well-placed. But that same scepticism is largely absent when AI shapes HR’s own purchasing decisions.
The governance gap nobody is talking about
When an HR leader asks ChatGPT to: ‘recommend the best HR and payroll software for a 700-person UK business’, they get a clean, confident shortlist in seconds.
It feels authoritative. Like research. Like the green light to make a confident buying decision.
But the same concerns HR leaders raise about the role of AI in people decisions apply directly here, too:
- Where did this recommendation come from?
- What data was the AI tool trained on?
- Is that training data current and valid?
- Could it be biased towards certain vendors?
- And who is accountable if this decision turns out to be wrong?
AI recommendations aren’t true research – they’re a probability output
AI search tools generate plausible-sounding answers based on what they were trained on and what they can retrieve. They don’t conduct an objective assessment of which HR or payroll software is the best fit for your organisation.
Research by SparkToro in 2025 asked 600 volunteers to run nearly 3,000 brand-recommendation prompts through ChatGPT, Claude and Google’s AI. There was less than a one-in-100 chance of getting the same list of recommended brands twice and roughly a one-in-1,000 chance of getting them in the same order.
But randomness is only part of the problem. AI tools also hallucinate – and in a procurement context, that has real consequences.
Here’s a scenario that might sound familiar. Ask an AI tool about the functionality of a particular HR system and you’ll get a list of confident-sounding answers. But if you explicitly instruct it not to make any guesses and assumptions, significant factual gaps become visible. The AI model has fabricated plausible-sounding details and presented it as fact.
Randomness is only part of the problem. AI tools also hallucinate – and in a procurement context, that has real consequences
The content trap: loud voices, not the best fit
There is a second, less obvious distortion. AI doesn’t always surface the best-fit vendor. It surfaces the vendor with the largest online presence.
Search-integrated AI tools – like Google’s AI Overviews – pull from what’s available across the web. Vendors who invest heavily in online content and review platforms appear consistently. Vendors who are genuinely excellent but less visible online may not appear at all.
This creates what might be called the ‘Google trap’. The summary card at the top of the search results feels comprehensive, so buyers stop scrolling. They don’t reach the vendor websites, comparison pages, or specialist directories lower down. And so the shortlist closes before it has fully opened.
The decision making accountability gap
When we trust AI to make a decision, who is held accountable to the result?
In AI-assisted hiring, HR has rightly fought for explainability. Decisions that affect people’s livelihoods should be traceable, auditable and defensible. HR has pushed back hard on “well, the algorithm said so” as a sufficient justification.
But “ChatGPT recommended it” is becoming an acceptable explanation for a software decision that will affect every person in an organisation for the next five to seven years.
And the consequences of getting it wrong are not abstract. Vendors are increasingly able to identify buyers who have let AI tools lead their searches, because these buyers arrive with lists of must-have features that don’t reflect actual business processes.
While AI might have given them a shortlist of systems to demo, it hasn’t provided a robust list of requirements. And it’s in these situations – where the requirements list doesn’t match with the requirements reality – that HR software implementation projects go wrong.
HR teams should treat AI shortlists the same way they’d treat any unaudited output: as a starting point that requires human verification
How to confidently use AI in the buying process
Used well, AI tools speed up the HR software shortlisting process. They do a great job of generating evaluation questions, transcribing demos and drafting comparison matrices.
The tool isn’t the problem: the issue is the absence of the same critical framework that HR applies everywhere else.
HR teams should treat AI shortlists the same way they’d treat any unaudited output: as a starting point that requires human verification, not a conclusion.
Refine your prompts by working in elements from your requirements list, and explicitly instruct the AI engine to consider HR systems’ ability to satisfy those requirements. Analyse the responses against your list of must-haves. Document who made the final decision and on what basis.
HR humans will need to do the foundational work that AI cannot do: defining requirements before you open a search tool, involving the right stakeholders and running a structured evaluation where vendors are assessed on the same criteria, in the same way.
The time saved at the research stage is quickly lost in a failed implementation.
Actionable insights
- Ensure you’re applying the same rigorous AI governance you apply to people decisions when purchasing software
- Treat AI outputs as a starting hypothesis, defining requirements before searching
- Ensure HR humans do the foundational work that AI cannot do
- Keep final decisions with people who can be held accountable for them.
Ciphr is a UK HR and payroll software company that helps mid-sized organisations connect the systems, data and people behind great work. Learn more at ciphr.com.
The post Why “ChatGPT recommended it” won’t cut it for HR software decisions appeared first on HRZone.
