In 2023, a major hiring platform faced a class-action lawsuit alleging that its system discriminated against millions of job applicants by screening them unfairly. The lawsuit cited concerns about Workdays's AI system which processes billions of job applications. The case arose partly because the company could not adequately explain how their AI made hiring decisions. This isn't an edge case or an anomaly. It is the reality for AI applications without proper infrastructure.
Many companies are making multi-million dollar decisions, and working with AI like it’s experimental new technology. Often trying to “move fast, and break things”, but this doesn’t work especially with intelligent systems that can make decisions on your behalf.
Everyone's in a hurry to deploy AI as quickly as possible, which creates tremendous opportunity but also significant risk. AI adoption isn’t like implementing microservices, or switching programming languages. LLMs can make autonomous decisions (within defined parameters), which means you not only use them, but you work along-side them now, and in some cases delegate to them.
AI can write production code, help classify data, screen CVs, and do many other things that humans have done for years, and gotten valuable experience doing. We have learned how to safely do these things, but more importantly, we have learned how to be accountable when we make mistakes.
Why Invest In More Robust Infrastructure?
AI without robust infrastructure can create multiple failure modes, and while deploying AI can be highly rewarding, it can also be highly risky. When agentic systems have a say in who gets a job, or who has access to credit, and other crucial decisions, then it’s imperative that we have robust systems in place to support them. This goes beyond just saying “it’s the right thing to do”, even though it is.
Some reasons to invest in robust infrastructure are highlighted below.
User Safety
AI systems making decisions about healthcare, financial services, or safety-critical applications can directly impact human wellbeing. Without proper safeguards, these systems can cause real harm to real people.
AI systems without proper safeguards can be manipulated to generate harmful content, bypass safety measures, or amplify dangerous misinformation. Recent examples include AI models generating offensive content, or AI coding tools being weaponised by hackers.
Product Predictability And Understanding
Not understanding your system's decisions creates cascading business problems. LLMs, while they are incredible at text prediction, they are not masters of consistency. Not being able to consistently explain the reasoning behind your product decision will have a knock-on effect for many other business processes. For example, if outcomes aren’t consistent, then users might be frustrated or product teams could waste time investigating transient issues.
Protection Against Litigation
Users have rights which are protected by the law. When these laws are violated, users will question it, and they might have the right to litigate. For example, when applicants are denied loans or jobs by AI systems without correct, fair, and clear reasoning, companies could face discrimination lawsuits that could have been prevented with proper audit trails.
AI bias litigation is becoming more commonplace with millions of dollars in confirmed settlements, and ongoing cases threatening even more damage. We have seen a court ruling that AI vendors can be held directly liable as "agents" of employers, and not just the companies using their tools.
These examples show just how crucial it is to get things right, and not deploy AI with flippancy. It can have very real-world, and very legal consequences.
Compliance And Regulation
New rules are coming into place that will require companies to get this right. The EU AI Act, which is a pioneering legal framework for regulating artificial intelligence, imposes comprehensive requirements on AI applications, with penalties up to €35 million or 7% of global turnover for non-compliance.
New York City's Local Law 144 has already taken effect, requiring annual third-party bias audits for any automated employment decision tools. Companies face daily fines of up to $1,500 for non-compliance.
These regulations have been implemented to create external accountability for companies using and deploying AI, and impose severe sanctions for non-compliance. Depending on your use-case, having robust AI infrastructure isn’t a nice-to-have, but a regulatory requirement.
Maintaining Customer Trust
Transparency is required to keep customer’s trust. Nowadays customers want to know how their data is being used, how decisions are made for them, and how your system is keeping them and their loved ones safe. Getting AI deployments right will go a long way to help achieving these things.
Brand Reputation
When companies deploy AI systems that produce biased or unfair outcomes, social media backlash can be swift and damaging. Recent controversies, for example the Apple Card investigation, show how quickly perception problems can escalate into brand crises.
As organizations become more responsible entities that people relate to, any perception of AI bias or unfairness can damage brand value that took years to build.
Retrofitting Cost
More expensive to start badly and fix later, as we all know tech debt lasts longer than we always think it would. Every decision also creates potential liability.
Techniques For More Robust AI
Here are some techniques every team deploying AI should consider
- Human Accountability Frameworks
- Audit Trails
- Operational Resilience
- Model Governance
Human Accountability Frameworks
Humans need to own and be accountable for decisions made by AI. During my time building in financial services, there were certain measures put in place to make sure staff (including engineers) owned the decisions they made and could be held accountable for them. These include bias trainings, security trainings, attestations, and so on. The same needs to be applied to AI applications. Frameworks need to be built such that humans are accountable for crucial decisions made by AI.
For instance, if an AI system denies a loan application, a loan officer should review the decision and be able to explain the reasoning to the customer.
Some examples of accountability frameworks could include:
- AI Trainings
- Decision Reviews and Approvals
Audit Trails
Audit trails are crucial for any mission critical system, but become more crucial for AI, where usually decisions are being made. Audit trails are great for knowing exactly what is happening, and trace how data is flowing through your systems. In the case of an external audit, these will also be valuable to show exactly what your system did in various situations. In fact, depending on your industry, and various other factors, they might be required by regulation.
Useful points to capture audit trails could include:
- User Input
- Model versions
- Thinking / Processing
- Validation / Feedback
- Decisions
Operational Resilience
Most times, technology problems require both technical and operational solutions. With critical systems, the ability to operate properly becomes equally important. For example, these should be considered:
- When there are increased errors in a deployed model, there should be reliable fallbacks, failsafes, or rollback procedures
- Observability so there is human oversight on system behaviour
- Escalation procedures
Model Governance
Models need to be validated, approved, and properly versioned to have a robust AI system. Rather than simply shipping whatever is new into production, the same considerations that are given to product deployments, need to be given to AI.
Conclusion
Companies deploying AI without proper infrastructure are building on quicksand. The technology is powerful, but power without accountability creates risk.
In my next article, I’ll look to explore infrastructure decisions for robust and safe AI.
What challenges are you facing with your AI deployment/applications? I'd love to hear your experiences. Feel free to connect on LinkedIn and send a message.
Cover Photo by Steve Johnson on Unsplash