The EU AI Act: Loopholes, Ambiguities, and Compliance Headaches

Introduction

With the EU AI Act (Regulation (EU) 2024/1689) (hereafter: AI Act) https://artificialintelligenceact.eu/  moving from theory to reality, businesses across sectors are working to understand how their AI systems fit into the new regulatory framework. Recent guidance from both the European Commission and the European Data Protection Board (EDPB) offers valuable insights, but also highlights open questions and evolving interpretations that companies will need to keep an eye on. This article takes a closer look at three recent developments shaping the compliance landscape:

1. Opinion 28/2024 from the EDPB, which explores how personal data can be lawfully processed within AI systems under GDPR (hereafter: EDPB Opinion) https://www.edpb.europa.eu/our-work-tools/our-documents/opinion-board-art-64/opinion-282024-certain-data-protection-aspects_en
2. The European Commission’s Guidelines on the Definition of an Artificial Intelligence System, which clarify key terminology and scope.  (hereafter: CIE Guidelines) https://digital-strategy.ec.europa.eu/en/library/commission-publishes-guidelines-ai-system-definition-facilitate-first-ai-acts-rules-application  
3. Article 111 of the AI Act, which outlines a grandfathering clause for high-risk AI systems already on the market (hereafter: Article 111) https://artificialintelligenceact.eu/article/111/

To help you make sense of these developments, we dive into some of the most pressing questions organizations need to answer, identifying loopholes and ambiguities:

1. Is there a potential regulatory gap for certain high-risk systems?
2. What falls outside the definition of an  “AI system”?
3. How does the regulatory framework distinguish between AI models and AI systems?
4. What specific components constitute an AI system under the regulation? 
5. Do current Large Language Models (LLMs) meet GDPR compliance requirements?
6. Who determines if an AI Model is anonymous?
   

The following analysis aims to provide clarity on these complex regulatory questions while identifying areas where further guidance may be needed.

Let’s dive in!

1. Is there a potential regulatory gap for certain high-risk systems?

The EU AI Act’s implementation reveals a significant potential compliance challenge in its grandfathering provisions for high-risk AI systems. Article 111 creates what may function as a regulatory gap by stipulating:

“…this Regulation shall apply to operators of high-risk AI systems, other than the systems referred to in paragraph 1 of this Article, that have been placed on the market or put into service before 2 August 2026, only if, as from that date, those systems are subject to significant changes in their designs.…”

This suggests that high-risk AI systems already on the market before August 2, 2026, may not have to comply with Chapter 3 requirements of the AI Act, as long as they do not change their intended purpose or system architecture.

For further guidance on ‘significant changes’ we make our way to Recital 177 which clarifies:

“…the concept of significant change should be understood as equivalent in substance to the notion of substantial modification, which is used with regard only to high-risk AI systems pursuant to this Regulation…”

And so we look into Article 3 (23) to understand what the AI act means with ‘substantial modification’. In Article 3(23) it is defined to mean a change to an AI system after its placing on the market or putting into service which is not foreseen or planned in the initial conformity assessment carried out by the provider.This would mean that for example an AI-powered medical device could avoid stricter obligations simply by remaining unchanged, even though it is classified as high-risk under the AI Act and will not need to comply with the AI Act’s requirements for high-risk AI systems unless significant design changes are made to the product.

This raises a critical concern: Will companies rush to push any significant changes before the deadline or only make incremental updates to secure a more favorable compliance position? Or simply not make updates due to the regulatory burden involved. And if so, does this create an unintended loophole that undermines long-term accountability?

2. What falls outside the definition of an “AI system”?

The question of what falls out of scope of the definition of AI Systems remains frustratingly vague.

Algorithm types. The latest CIE Guidelines  exclude AI systems that “infer in a narrow manner” (Paragraph 5.2). And in paragraph 48, it also excludes “simpler” rule-based systems and classical heuristics. Yet, in paragraph 39, it acknowledges that AI systems using predefined rules and logical inference can be in scope.

This inconsistency leaves companies struggling to determine whether widely used tools like spam filters, antivirus detection, or other machine-learning-powered services should be included in their AI inventories. The CIE Guidelines tells us to assess a system’s ability to infer, analyze, and adjust, but without clear and practical thresholds, this could turn into a compliance guessing game.

System Boundaries in Multi-Model Architectures. Modern AI applications often combine multiple models working together. The CIE Guidelines don’t specifically address whether a collection of interconnected models constitutes a single AI system, how to define boundaries when models from different providers are integrated nor the status of orchestration layers managing multiple models

Consider an example in a financial platform combining: an LLM (general-purpose AI) for document analysis, a credit-scoring model trained in-house, a fraud-detection model sourced from a vendor and other models, what exactly is “the AI system” under the AI Act? The guidelines offer no clear rules for defining system boundaries in composite solutions.

So, where does this leave businesses? Will regulators clarify these gray areas, or are we looking at a wave of fragmented interpretations across industries?

3. How does the regulatory framework distinguish between AI models and AI systems?

The AI Act explicitly regulates AI Systems, but does not clearly address standalone AI Models. These models can be core components of AI Systems, yet before integration, they exist in a gray area. Does this mean a trained model itself falls outside the AI Act’s regulatory scope?

Current guidance does not clearly define how standalone AI models should be governed. The European Data Protection Board (EDPB) distinguishes between AI Models and AI Systems (see paragraphs 19-26 of the EDPB Opinion) , but the AI Act does not provide explicit clarity on where standalone models fit. Without clear oversight, could this create a compliance gap where AI models, particularly those with high-risk potential, remain unregulated until deployed or integrated in a system. 

If standalone models are not considered AI Systems, this could lead to regulatory blind spots, allowing models with significant risks to be trained, developed, and even distributed without accountability. As the AI Act moves toward implementation, will regulators address this ambiguity, or is this a potential gap?

4. What specific components constitute an AI system under the regulation?

As organizations prepare for compliance with the AI Act, a critical question emerges: once an AI system falls under the regulation’s scope, what exact components are considered part of that system for regulatory purposes?

Why is this question important? The definition of an AI system’s components directly impacts several operational aspects of AI governance and cybersecurity practices:

• Compliance Scope Definition: Organizations need to know precisely what parts of their technology stack must comply with the AI Act’s requirements, especially for high-risk systems.
• Risk Assessment Boundaries: When conducting risk assessments, clearly defined system boundaries help determine what elements must be evaluated.
• Documentation Requirements: Technical documentation must cover all components of the AI system, requiring clarity on what constitutes the system itself.
• Accountability and Responsibility: Organizations must assign clear roles for maintaining different components. Without clear boundaries, responsibilities may be ambiguous.
• Reporting and Incident Response: For incident reporting purposes, organizations need to know what constitutes a failure or issue with the “AI system” as defined by the regulation.
• Supply Chain Management: Organizations need to identify which third-party components are part of their AI systems to manage vendor compliance.

The CIE Guidelines provide some clarity but also leave room for interpretation. Here’s what we know:

What is included:

1. Hardware Components: The physical components that provide the infrastructure for computation, including processing units, memory, storage devices, networking units, and input/output interfaces (Paragraph 11).
2. Software Components: Computer code, instructions, programs, operating systems, and applications that handle how the hardware processes data and performs tasks (Paragraph 11).
3. Components Throughout the Lifecycle: The components necessary for the AI system to function throughout its lifecycle, from training to deployment (Paragraphs 10-12).
4. Components Related to Internal Objectives: Components that help the system achieve its explicit or implicit internal objectives (Paragraph 24-25).

Besides the Commission’s efforts to provide clarity, there are many grey areas and unanswered questions left:

1. Data Processing Infrastructure – The CIE Guidelines don’t clearly address whether data processing infrastructure (beyond the immediate computing resources) should be considered part of the AI system. This includes: Data pipelines, ETL processes, data storage systems, and pre-processing workflows that transform data before it enters the model. Example – A real-time fraud detection system that relies on streaming data pipelines performing feature extraction before inference. Are those pipelines part of the AI system?
2. Human-Machine Interface Components – While the CIE Guidelines mention input/output interfaces as hardware components, they don’t specify whether the software interface layer through which humans interact with the system is considered part of the AI system. This includes: dashboard interfaces, API layers, user feedback mechanisms. Example: In a customer service chatbot, the user interface collects customer ratings (thumbs up/down), which feeds directly into ongoing training. Is the interface part of the AI system?
3. Monitoring and Feedback Systems – For adaptive AI systems that continue to learn after deployment, it’s unclear whether monitoring tools and feedback mechanisms should be considered part of the system. This includes: model performance monitoring tools, feedback collection mechanisms, retraining workflows.Example: A bank’s fraud detection system continuously updates its systems using new transactional data and analyst feedback. Each month, it retrains to adapt to emerging fraud patterns. Is the retraining pipeline, including feedback collection, model evaluation, and updates, considered part of an AI system?
4. Cloud-Based Components – For AI systems deployed in cloud environments, questions remain, such as: which cloud infrastructure elements fall within the system boundary, how to handle shared responsibility models between cloud providers and system operators, what is the status of serverless functions or containerized components? Example: If an AI model is deployed via AWS Lambda functions that manage inference, are those Lambda functions part of the AI system?
5. Other gray areas like shared computing resources and multipurpose sensors. Examples: If AI runs on a shared CPU alongside non-AI software, does the CPU as a whole fall under the AI Act? If some sensor data is used for AI and some is not, does the sensor as a whole (e.g. camera, microphone) fall under regulation? 

The definition of what constitutes an AI system from guidelines of the European Commission on AI systems provides important guidance, but organizations must still make practical judgments about where to draw system boundaries. Taking a comprehensive, risk-based approach to identifying system components will help ensure compliance while the regulatory landscape continues to evolve.

As implementation of the AI Act progresses over the coming months and years, we can expect additional clarity on these questions from regulators, court decisions, and emerging best practices. In the meantime, organizations should document their rationale for defining system boundaries to demonstrate due diligence in their compliance efforts.

5. GDPR’s Legitimate Interest Test for AI Systems: Compliance Challenges

The European Data Protection Board (EDPB) has in its EDPB Opinion  clarified how “legitimate interest” may justify personal data processing for AI under GDPR Article 6(1)(f). However, controllers must pass a strict three-step test, including a balancing test that requires assessing:

• Potential harm to data subjects;
• The Controller’s  reasonable expectations;
• The proportionality of data processing (the proportionality of data processing requires organizations to ensure that the scale, scope, and methods of collecting and using personal data for AI training are not excessive relative to their stated purposes)

This guidance raises critical compliance questions for AI development:

• Where did the training data originate?
• How was the data collected?
• Were data subjects aware their information could be used for AI training?
• What are the potential future uses of the AI model?
• Do individuals know their personal data is online and potentially used for training?

These specific requirements pose significant challenges for the current LLM development approach, which often relies on web scraping and processing vast amounts of publicly accessible personal data. The lack of transparency and limited user awareness raises serious concerns about whether such models can truly comply with GDPR’s principles of fairness, lawfulness, and transparency.

6. Who determines if an AI model is anonymous?

Proving that an AI model is truly anonymous is more complex than simply stating it was trained on (anonymized) personal data in a way that prevents re-identification. The EDPB Opinion  clarifies that a model is only considered anonymous if both the risk of extracting personal data and the likelihood of a successful attack are insignificant.

But how can a company prove that re-identification risks of the trained data are insignificant and that personal data cannot be extracted or attacked successfully? There is no clear methodology for conducting such an assessment. Does this require an independent review, and if so, who qualifies? Would this need to be conducted by an anonymization expert, or can anyone make the determination? And who will check this review, so who really decides if an AI model can be considered anonymous? Would that be a privacy legal expert? An external committee? Without standardized guidelines, companies may face inconsistent interpretations of compliance.

This lack of clarity creates uncertainty: Should organizations prioritize compliance with the EDPB Opinion, or is the AI Act the governing framework? Since these concerns are not explicitly addressed in the AI Act, companies risk navigating conflicting legal requirements. So who will take the lead in bridging this gap?

Conclusion

The EU AI Act aims to create a governance framework for artificial intelligence, but implementation challenges remain unresolved. Article 111’s grandfathering provisions may undermine accountability by creating compliance advantages for existing high-risk systems. The Commission’s guidelines leave substantial ambiguities regarding system boundaries and exemptions for rule-based systems. The EDPB’s Opinion 28/2024 introduces additional compliance considerations for data protection that aren’t explicitly addressed in the AI Act.

While some regulatory questions remain open, companies can get ahead by adopting a risk-based, well-documented compliance approach that demonstrates due diligence. Clear internal policies on identifying AI systems, defining their boundaries, and ensuring personal data use complies with both the AI Act and GDPR,  particularly in LLM training pipelines, will be essential to demonstrating due diligence.

As the regulatory framework continues to mature, businesses should expect further clarifications, potential court challenges, and evolving interpretations across both AI Act compliance and GDPR compatibility. For now, transparency, proactive documentation, and close collaboration between legal, privacy, and technical teams will position companies well to adapt as the regulatory landscape evolves.

Looking to read more?

Creating trustworthy and secure AI systems is a complex challenge that requires a multidisciplinary approach. Recognizing this, we have formed a collaboration that we refer to as the “Golden Triangle“, consisting of a legal privacy expert (Tanya Chib), a cybersecurity expert (Anna Hakkers, PhD) and an anonymization expert (Renate van Kempen). 

Each corner of the triangle, legal privacy, cybersecurity and anonymization, represents a critical domain of expertise that contributes to the overall trustworthiness and security of AI systems.

Together, we are excited to announce a forthcoming series of articles, where each of us will provide our unique insights from three distinct angels on the same critical topics. We invite you to follow our series and join us in exploring these essential aspects of AI.