Compliance & Regulation: What the EU AI Act Means for Software Development
Why law, ethics and code now have to go hand in hand
Figure 1: Image generated by Gemini (Google AI)
The new blueprint for the digital world: how the EU AI Act is changing software development
Whenever I think back to the early days of the modern tech giants, the famous (and infamous) motto comes to mind: “Move fast and break things.” Just get on with it, push code out into the world and see what happens. But when we look at the breakneck development of artificial intelligence over the past months, this approach reminds me more of the legendary line from Dr. Ian Malcolm (Jeff Goldblum) in Jurassic Park: “Your scientists were so preoccupied with whether or not they could that they didn’t stop to think if they should.” Artificial intelligence left the lab phase long ago. And this is exactly where the EU AI Act now steps in — the world’s first comprehensive attempt to impose on AI something like the robot laws Isaac Asimov devised, adapted for the 21st century. For those of us in software development, project management and testing, this marks a massive turning point: law, ethics and code now have to go hand in hand. Good compliance is no longer just a legal fig leaf, but a hard quality attribute. But what does that concretely mean for our everyday development work?
1. Classification: tell me what you do, and I’ll tell you your risk
Before we type even a single line of code into our keyboards in the future, a new hurdle awaits: legal classification. From now on, the technical architecture strictly follows regulation.
Software is no longer judged primarily by its cool features, but by its context of use. An example: we build an AI image-recognition system. Nothing earth-shattering, standard machine learning. We train the model to sort our private holiday photos on the smartphone into categories like “beach” and “mountains.” Legally speaking? Such systems fall into the “minimal” or “limited risk” category. Here we have no regulatory headaches; pure sorting causes us no problems. As an app for purely personal, non-professional use, that’s no issue. But now take the same algorithm — the same weights in the neural network, the same codebase — and deploy it to monitor employees at the workplace: then the red light immediately comes on — we’ve built a high-risk AI system!
The magic phrase for this new operational challenge is “compliance by design.” We have to check proactively where our software ends up. And here we run into the dual-use dilemma. What if our software, meant to be harmless, is repurposed by users? Like a kitchen knife, intended for chopping vegetables but which can also serve as a weapon. We can’t foresee every misuse, but regulation forces us to build smart “guardrails” (technical rails) into our design. We have to look at software from the very start through the lens of social responsibility. For everyday work this means: the imperative of “compliance by design” spells the end of the purely feature-driven sprint. Legal requirements now become part of the Definition of Done. A ticket cannot be closed if it isn’t clear which risk class the feature falls into. Security, privacy and now also AI-compliance checks move all the way to the left in the development cycle — the well-known “shift-left.”
2. Data governance: “Garbage In, Lawsuit Out”
In the big-data frenzy, the maxim often used to be: the more data we pump into the algorithm, the better. Its provenance? A minor matter. Today this Wild West is giving way to the strict obligation of “clean & fair data.”
Now there are clear limits. It’s not enough for data simply to be “there.” It has to be representative, error-free and — most importantly — checked for biases (prejudices). If we train a recruiting AI almost exclusively on male CVs, we’re simply building ourselves an automated chauvinist that systematically filters out applicants of other genders. That means both women and people who identify as diverse, inter, non-binary, gender-fluid, agender, pan or belonging to other genders.
And: how do you measure a philosophical concept like “fairness” in code? Here we have to be honest: the perfect, absolutely bias-free AI does not exist, because there is no absolutely bias-free world from which we draw the data. For data scientists and QA teams, the future is about process compliance. They have to prove that they systematically searched for biases and minimised them (e.g. via disparate impact ratio or equalized odds). Data cleaning is finally no longer a tiresome afterthought left to interns. Anyone who feeds in unclean data today reaps not only poor results but risks direct legal consequences. Garbage In, Garbage Out simply becomes: Garbage In, Lawsuit Out.
But what if the AI is hacked during training or in live operation? If deliberately manipulated data is smuggled into the training dataset? The answer according to the AI Act: “Garbage In, Lawsuit Out” gains a security dimension here. Companies have to demonstrate resilience against so-called “data poisoning.” This requires data provenance — that is, a gapless, cryptographically secured trace of who fed which data into the training pool and when. On top of that, algorithms for anomaly detection are needed already during the training phase, automatically flagging statistical outliers in the dataset. Under the AI Act, data governance means not only checking data for fairness and quality, but hardening it as highly sensitive, business-critical infrastructure against sabotage — both in the quiet chamber of training and out in live operation.
3. The end of the “black box”: transparency & explainability
Some may know the sketch from Little Britain: a customer asks for a loan, sits in front of a bank clerk who listlessly taps away at her keyboard, stares at the screen and says monotonously: “Computer says no.” When the customer then asks: “Why?”, nobody knows. Neither the clerk, nor the branch manager, nor the programmer.
That is exactly what must no longer happen today. When a high-risk AI makes a decision, it must not happen in the hidden depths of a mysterious algorithm — here we’re talking about Explainable AI (XAI). A system must not only spit out a mathematical result at breakneck speed, but also supply a logical justification that a human — or, in case of doubt, a court — can follow. Comprehensive log files become mandatory, as do crystal-clear “package inserts” for users that explain exactly where the AI excels and where it makes mistakes. With neural networks that have billions of parameters, we cannot trace every single computation. Instead, the industry uses methods like SHAP (Shapley Additive exPlanations) or LIME (Local Interpretable Model-agnostic Explanations). These tools don’t show the exact path of the algorithm, but they visualise which input factors (e.g. age or income in a dataset) had the strongest influence on the final decision.
Added to this is the important labelling obligation in the fight against deepfakes and disinformation: everything that was artificially generated (texts, chat, photos…) must be marked as an AI product.
4. Human-in-the-loop: designing against blind trust
As breathtaking as the technological efficiency gains are, the EU AI Act draws a red line: in highly sensitive areas the machine never decides fully autonomously. There always has to be the so-called “human-in-the-loop.” A medical AI or a credit-scoring software may only assist, never act as sole decision-maker. Someone has to keep the proverbial emergency stop button in hand.
For our UI/UX designers this is a fascinating challenge. Out of convenience, we humans tend towards automation bias — we trust the computer blindly and click “accept” without reading. When it comes to high-risk systems, Steve Krug’s credo “Don’t Make Me Think” has to be deliberately suspended. So how do we build user interfaces that literally force our users to pause, think critically, and actively take responsibility? We have to build deliberate friction points into the software so that the human is not degraded to a mere button-pusher. If a bank clerk rejects the AI’s suggestion (or accepts it uncritically), she has to justify the decision, e.g. with a short piece of text — this breaks the reflexive click-through. Moreover: the AI does not output absolute truths (“This is fraud”), but probabilities (“82% an anomaly”). That forces the human to mentally assess the remaining 18%. The point is to place the “friction point” exactly where the critical decision is made, and to keep the rest of the workflow flowing as smoothly as possible.
5. The open-source dilemma: who is liable for the code?
A particularly hotly debated topic concerns the open-source community. Who foots the bill when an open AI model is built by third parties into a lucrative commercial app and causes harm there? The EU AI Act attempts a balancing act here: pure research and hobby development remain largely spared. But as soon as the code becomes a commercial product, the liability trap snaps shut.
Software today is mostly assembled from existing modules, like Lego bricks. The legal situation essentially protects pure open-source development. The developer who puts the model on Hugging Face is generally not liable. But: as soon as you as a company take that free model, build it into your commercial app and bring it onto the EU market as a high-risk system, you legally become the operator or provider. The liability trap snaps shut on you. You have to answer for the compliance, the data quality and the logs of that model — no matter who originally wrote it. So there’s an obligation to check the entire “software supply chain” for legacy burdens.
Conclusion
Is the EU AI Act, then, a joyful vision of the future or a bureaucratic monster? Do we have to fear it will strangle the European tech scene? There is heated debate about whether innovative start-ups will now migrate abroad in frustration because they cannot withstand the requirements of the EU AI Act. But perhaps we will experience exactly the opposite: trust will become the most important currency of the AI era. When a piece of software bears the seal “EU AI Act Compliant,” customers worldwide know that this software has been checked for security, fairness and transparency. Just as the GDPR is today regarded globally as a quality attribute for data protection, this — admittedly painful — transformation process could give Europe’s tech sector a decisive unique selling point in the world.
Regulation (EU) 2024/1689 — EU AI Act
Classification as the first hurdle
- Unacceptable Risk (prohibitions): Article 5
- High-Risk: Article 6 in conjunction with Annex III
- Minimal/Limited Risk: Article 50
Data governance (data quality)
- Requirements for data: Article 10
Transparency & explainability (black box)
- Transparency for users: Article 13 (information for operators/deployers)
- Record-keeping obligations (logging): Article 12
- Technical documentation: Article 11 and Annex IV
Human-in-the-loop
- Human oversight: Article 14
Open source & liability
- Exemptions for open source: Article 2(12)
- GPAI (General Purpose AI): Article 53 and recitals 102 to 104
Source: Regulation (EU) 2024/1689 (EU AI Act) — eur-lex.europa.eu
Published in QualityNews H1/2026