AI Supply Chain Security: Why You Need an AI Bill of Materials Before Your Next Audit

Your organization probably has a Software Bill of Materials. You might even have a thorough one. But if you are deploying AI systems, whether large language models, machine learning pipelines, or AI-powered features in your product, your SBOM is not telling you the full story. The components that make AI systems dangerous are the ones that traditional software supply chain tools were never designed to track: pre-trained model weights downloaded from public repositories, training datasets scraped from unknown sources, fine-tuning data curated by third parties, and plugins that let your AI agent call APIs you have never audited.

This is the problem that OWASP identified when they placed Supply Chain Vulnerabilities at LLM03 in their Top 10 for Large Language Model Applications. It is the problem that the EU AI Act is now forcing organizations to address with documentation requirements and penalties. And it is the problem that most enterprises are still ignoring because they do not have the vocabulary, the frameworks, or the tools to deal with it.

The solution is an AI Bill of Materials, an AI-BOM. This article explains what an AI-BOM is, why you need one, what risks it addresses, and how to start building one before your next audit forces the issue.

OWASP LLM03: Supply Chain Vulnerabilities for AI Systems

The OWASP Top 10 for Large Language Model Applications is the most widely referenced framework for understanding AI-specific security risks. LLM03, Supply Chain Vulnerabilities, addresses the reality that modern AI systems are assembled from components sourced from a sprawling, often opaque ecosystem of model repositories, dataset platforms, and third-party service providers.

Traditional software supply chain attacks target code libraries and packages. The SolarWinds attack compromised a software update mechanism. The Log4j vulnerability lived in a widely used Java library. These are risks that SBOMs and Software Composition Analysis tools are designed to catch. But AI supply chains introduce categories of risk that have no parallel in traditional software:

• Pre-trained model weights downloaded from public repositories like Hugging Face, which may contain backdoors, biased training artifacts, or malicious serialization payloads.
• Training and fine-tuning datasets sourced from third parties, web scrapes, or community contributions, any of which could be poisoned to influence model behavior.
• Embedding models and vector databases used in Retrieval-Augmented Generation (RAG) systems, where compromised embeddings or injected documents can manipulate outputs.
• Third-party AI plugins and agent integrations that extend model capabilities by calling external APIs, often with minimal security review.
• Model serving infrastructure including inference APIs, quantization tools, and runtime environments that introduce their own vulnerabilities.

OWASP LLM03 recognizes that any of these components can be compromised, and that the downstream effects are uniquely dangerous because AI systems make decisions, generate content, and take actions that directly affect users and business operations. A poisoned model does not crash. It silently produces wrong or manipulated outputs that may go undetected for months.

What Is an AI Bill of Materials (AI-BOM)?

An AI Bill of Materials is a structured, machine-readable inventory of every component in an AI system's supply chain. It extends the concept of a Software Bill of Materials (SBOM) to cover the AI-specific components that SBOMs cannot track. A comprehensive AI-BOM documents:

Foundation and Fine-Tuned Models

Every model used in your system, including the base foundation model, any fine-tuned variants, and auxiliary models used for tasks like embeddings, classification, or moderation. For each model, the AI-BOM records the source repository, version or commit hash, model architecture, parameter count, quantization method, license, and known limitations or biases documented in the model card.

Training Data Sources

A catalog of every dataset used in pre-training and fine-tuning, including the source, collection methodology, date range, size, data types, and any known issues such as representation gaps or contamination. For models you did not train yourself, this means documenting whatever the model provider discloses about training data, and explicitly noting what is unknown.

Fine-Tuning Datasets

Detailed records of the datasets used to adapt a base model for your specific use case. This includes the source of the data, the annotation process, quality control measures, any personally identifiable information (PII) contained in the data, and the data processing pipeline used to prepare it for training.

RAG Sources and Vector Databases

For systems using Retrieval-Augmented Generation, the AI-BOM documents every data source indexed in the vector database, the embedding model used to create vector representations, the chunking and indexing strategy, update frequency, and access controls governing who can add or modify the indexed content.

Plugins, Tools, and Agent Integrations

Every external API, tool, or plugin that the AI system can invoke, including the provider, the data shared with each integration, the permissions granted, authentication mechanisms, and the scope of actions the integration can perform. For AI agents that can chain multiple tool calls, this includes documenting the full range of possible action sequences.

Infrastructure and Runtime Dependencies

The model serving framework, inference API, hardware accelerators, container images, and any preprocessing or postprocessing pipelines. This overlaps with traditional SBOMs but must be explicitly linked to the AI-specific components they support.

Why Traditional SBOMs Are Not Enough

Organizations that already maintain SBOMs sometimes assume they have AI supply chain visibility covered. They do not. The gap between what SBOMs track and what AI systems require is fundamental, not incremental.

An SBOM tells you that your application uses PyTorch version 2.3.1, the transformers library version 4.42.0, and numpy 1.26.4. It does not tell you that the model loaded by that PyTorch instance was downloaded from an unverified Hugging Face repository six months ago, was fine-tuned on a dataset containing scraped personal data that violates GDPR, and includes a pickle-serialized weights file that could execute arbitrary code on load.

The distinction matters because the most dangerous AI supply chain attacks do not target the software libraries. They target the artifacts those libraries consume: models and data. A perfectly patched, vulnerability-free software stack can still load a poisoned model or ingest compromised training data. Your SBOM will show green across the board while your AI system is silently compromised.

This is not a criticism of SBOMs. You absolutely need one. But treating your SBOM as sufficient AI supply chain documentation is like treating a building's electrical inspection as a complete structural assessment. It covers one critical system while leaving others entirely unexamined.

Real Risks: How AI Supply Chains Get Compromised

AI supply chain attacks are not theoretical. They are happening now, and the attack surface is growing as organizations accelerate AI adoption without proportional investment in supply chain security.

Model Poisoning via Public Repositories

Hugging Face hosts over 500,000 models, and it has become the de facto package manager for AI. Security researchers have repeatedly demonstrated that malicious models can be uploaded to the platform and downloaded by unsuspecting developers. In 2024, JFrog security researchers discovered over 100 malicious models on Hugging Face that contained hidden code execution payloads.^[1] These models looked legitimate, had plausible names and descriptions, and would function correctly for their stated purpose while silently executing attacker-controlled code on the developer's machine.

The attack vector is straightforward. An attacker creates a model repository with a name similar to a popular model, such as a misspelled variant or a version claiming to be an optimized fork. Developers searching for a specific model find the malicious version, download it, and load it into their pipeline. Because model files are large binary blobs, they are not subject to the same code review processes that software dependencies receive.

Malicious Model Serialization: The Pickle Problem

The most technically dangerous AI supply chain vulnerability is malicious model serialization, specifically through Python's pickle format. Pickle is the default serialization method for PyTorch models, and it is inherently unsafe. A pickle file can contain arbitrary Python code that executes automatically when the file is deserialized, meaning when the model is loaded.

This is not a bug. It is a fundamental design characteristic of the pickle format. When you load a pickle-serialized model file with torch.load(), you are executing whatever code the model creator embedded in that file. An attacker can craft a model file that functions correctly as a machine learning model while simultaneously establishing a reverse shell, exfiltrating environment variables and API keys, or installing persistent backdoors on the host system.^[2]

Safer serialization formats like SafeTensors exist and should be mandatory in any production AI pipeline. But adoption remains inconsistent, and many popular models are still distributed exclusively in pickle format. Your AI-BOM should track the serialization format of every model file and flag any use of pickle in production environments.

Compromised Fine-Tuning Data

Fine-tuning is where organizations customize a foundation model for their specific use case, and it is one of the most vulnerable points in the AI supply chain. Fine-tuning datasets are often smaller and more targeted than pre-training data, which means a relatively small amount of poisoned data can have an outsized effect on model behavior.

An attacker who can inject even a few hundred carefully crafted examples into a fine-tuning dataset can create a backdoor that triggers specific model behaviors under specific conditions. For example, a poisoned customer service chatbot might function normally for 99.9% of queries but provide incorrect refund instructions when a specific phrase is used, routing refund payments to an attacker-controlled account. The model passes all standard quality assurance tests because the trigger conditions are never present in test data.^[3]

The Model Provenance Problem

Perhaps the most pervasive AI supply chain risk is not a specific attack but a systemic lack of visibility: most organizations cannot answer basic questions about where their models came from and what they were trained on.

Consider a common scenario. Your engineering team downloads a pre-trained language model from Hugging Face. The model card says it was trained on "a large corpus of internet text." That description covers everything from Wikipedia to 4chan. You fine-tune this model on your company's data and deploy it to production. A customer asks what data the model was trained on because they need to verify compliance with their data processing agreement. You cannot answer the question.

This is not edge-case paranoia. This is a routine compliance requirement that most AI-deploying organizations cannot satisfy. The EU AI Act explicitly requires documentation of training data provenance for high-risk AI systems. If you cannot trace your model's lineage from pre-training data through fine-tuning to deployment, you have a compliance gap that an AI-BOM is designed to close.

Third-Party AI Plugin and Agent Risks

The emergence of AI agents, systems that can autonomously call APIs, browse the web, execute code, and chain multiple actions together, has introduced a supply chain risk category that did not exist two years ago. When your AI agent calls a third-party API to retrieve information, process data, or take an action, that API becomes part of your AI supply chain.

The risks are compounding. An AI agent might call a plugin that retrieves data from an external source, processes it through another third-party service, and uses the result to make a decision or generate a response. Each link in that chain is a potential point of compromise. A malicious or compromised plugin can:

• Exfiltrate sensitive data by collecting prompts, context, and user information passed to it during API calls.
• Inject malicious content into the agent's context window, manipulating subsequent reasoning and actions through indirect prompt injection.
• Escalate privileges by exploiting the permissions granted to the agent, using tool-calling capabilities to access systems the plugin should not reach.
• Poison the information supply chain by returning subtly incorrect data that the agent incorporates into its outputs without verification.

Most organizations deploying AI agents have no formal process for vetting the plugins and tools those agents can access. The agent's tool manifest, the list of APIs it can call and the permissions it has, is effectively an unaudited supply chain.

How to Build an AI-BOM

Building an AI-BOM is not a one-time project. It is an ongoing practice that must be integrated into your AI development and deployment lifecycle. Here is a practical framework for getting started.

Step 1: Inventory All AI Components

Start by cataloging every AI component in your organization. This includes models you trained, models you downloaded, models accessed through APIs, fine-tuning datasets, RAG data sources, plugins, and inference infrastructure. For many organizations, this initial discovery phase is the hardest step because AI adoption has been decentralized, with individual teams and developers deploying models without centralized oversight.

Step 2: Adopt Model Cards and Data Cards

Model cards, originally proposed by researchers at Google, are structured documents that describe a model's intended use, training data, performance metrics, limitations, and ethical considerations.^[4] Data cards serve the same purpose for datasets. Together, they form the documentation backbone of an AI-BOM.

For every model in your inventory, create or obtain a model card that documents:

• Model architecture, version, and parameter count
• Source repository and download hash
• Training data description and known limitations
• Fine-tuning data sources and methodology
• Serialization format (flag any pickle usage)
• Performance benchmarks and known failure modes
• License and usage restrictions
• Known biases and mitigation measures

For every dataset, create a data card that documents the source, collection methodology, annotation process, PII presence, consent status, and any known quality issues.

Step 3: Implement Dependency Tracking

Just as software dependency management tools like Dependabot and Snyk track library versions and known vulnerabilities, you need a system for tracking AI component versions and associated risks. This means:

• Pinning model versions to specific commit hashes or checksums, never using "latest" in production.
• Monitoring upstream changes to models and datasets you depend on, including license changes, retractions, and disclosed vulnerabilities.
• Scanning model files for known malicious patterns, particularly pickle deserialization exploits, using tools like Protect AI's ModelScan or Hugging Face's malware scanning.
• Auditing RAG data sources for unauthorized modifications, injection attempts, and data quality degradation.

Step 4: Establish a Plugin and Tool Review Process

Every third-party plugin, API, or tool that your AI systems interact with needs a security review before deployment. This review should assess the data shared with the integration, the permissions granted, the provider's security posture, and the potential impact of compromise. Maintain a living registry of approved integrations and review it quarterly.

Step 5: Automate and Integrate

The AI-BOM should not be a static document that gets updated once a quarter. It should be generated and updated automatically as part of your CI/CD pipeline. When a developer changes a model version, updates a fine-tuning dataset, or adds a new plugin, the AI-BOM should reflect the change immediately. Integrate AI-BOM generation into your deployment pipeline so that every production release includes an up-to-date inventory.

Compliance Implications: EU AI Act and Beyond

The regulatory landscape for AI supply chain transparency is tightening rapidly. Organizations that build AI-BOM capabilities now will be ahead of requirements that are becoming mandatory.

The EU AI Act

The EU AI Act, which began phased enforcement in 2025, is the most comprehensive AI regulation in the world. For high-risk AI systems, which include applications in healthcare, finance, employment, and critical infrastructure, the Act requires:

• Article 10: Detailed documentation of training, validation, and testing data, including data governance measures, data preparation processes, and data provenance.
• Article 11: Technical documentation that enables authorities to assess the AI system's compliance, including a description of the elements of the AI system and of the process for its development.
• Article 17: A quality management system that covers, among other things, supply chain management and data management procedures.

Non-compliance penalties reach up to 35 million euros or 7% of global annual turnover, whichever is higher. An AI-BOM is not explicitly named in the regulation, but it is the practical mechanism by which organizations satisfy these documentation requirements.^[5]

Upcoming US Regulations

The United States regulatory approach is evolving through a combination of executive orders, sector-specific guidance, and state-level legislation. The 2023 Executive Order on AI Safety directed NIST to develop guidelines for AI supply chain risk management, which are now being formalized. The 2025 NIST AI Risk Management Framework Companion explicitly addresses supply chain risks and recommends maintaining detailed inventories of AI components. Multiple state legislatures have introduced bills requiring AI transparency and documentation for specific use cases.^[6]

The trend is unmistakable. Whether through the EU AI Act, NIST frameworks, sector-specific regulations from the FDA or financial regulators, or state-level legislation, AI supply chain documentation is moving from best practice to legal requirement. Organizations that wait for mandates to build these capabilities will find themselves scrambling to comply with deadlines they cannot meet.

Practical Steps for Enterprises: Start Today

Building a comprehensive AI-BOM capability does not happen overnight, but you can take meaningful steps immediately to improve your AI supply chain security posture.

Audit Your AI Supply Chain

Conduct a discovery exercise across your organization to identify every AI system in production or development. For each system, document the models used, their sources, the training and fine-tuning data, and all third-party integrations. You will likely discover AI usage you did not know about, models downloaded from unverified sources, and undocumented data pipelines. This visibility is the foundation everything else builds on.

Implement Model Scanning

Deploy automated model scanning tools in your CI/CD pipeline. At minimum, scan every model file for pickle deserialization exploits before it enters your environment. Tools like Protect AI's ModelScan, Hugging Face's built-in malware scanning, and NVIDIA's Morpheus can detect known malicious patterns in model files. Make SafeTensors the required serialization format for all new model deployments and create a migration plan for existing pickle-based models.^[7]

Verify Data Provenance

For every dataset used in training or fine-tuning, establish and document the chain of custody. Where did the data come from? Who collected it? What consent was obtained? Has it been checked for PII? What quality controls were applied? If you are using third-party datasets, evaluate the provider's data governance practices as you would evaluate any other vendor's security posture. If you cannot verify provenance, treat the data as untrusted and assess the risk of using it.

Secure Your AI Agent Tool Chain

If you are deploying AI agents, audit every tool and API the agent can access. Apply the principle of least privilege: agents should only have access to the tools they need for their specific task, with the minimum required permissions. Implement monitoring for agent tool calls, logging every API interaction for anomaly detection and incident investigation. Require security review for any new tool integration before it is made available to agents.

Establish Governance and Ownership

Assign clear ownership for AI supply chain security. This might be your existing application security team, a dedicated AI security function, or a cross-functional working group. The key is that someone is responsible for maintaining the AI-BOM, reviewing supply chain changes, and escalating risks. Without ownership, AI supply chain security becomes everyone's concern and no one's responsibility.

Plan for Incident Response

Update your incident response plans to include AI supply chain compromise scenarios. What do you do if a model you are using in production is found to contain a backdoor? How do you respond if a training data source is discovered to be contaminated? What is the rollback procedure for a compromised model, and how do you assess the blast radius of outputs generated while the compromised model was active? These scenarios are different from traditional software supply chain incidents and require specific response procedures.

Lorikeet Security Team

Penetration Testing & Cybersecurity Consulting

We've completed 170+ security engagements across web apps, APIs, cloud infrastructure, and AI-generated codebases. Everything we publish here comes from patterns we see in real client work.