Adversarial Reconnaissance and Resource Acquisition Against ML Systems

AWS Certified AI Practitioner · Security, Compliance, and Governance for AI Solutions · By Super Admin

Adversaries first gather intelligence through public research, websites, and app stores, then acquire tools and infrastructure—including compromising the ML supply chain—as shown by the LLM package hallucination attack case study.

Adversarial techniques in ML can be broken into three phases—reconnaissance, resource acquisition, and execution—each building on the previous one to create a layered approach to compromising ML systems. Understanding the first two phases helps practitioners anticipate threats before an attack is launched.

Reconnaissance: Gathering Intelligence

Before launching an attack, adversaries invest significant time gathering information about their target, much like a thief entering a neighborhood to identify vulnerabilities and opportunities. Adversaries scour publicly available research materials such as academic papers, conference proceedings, and technical blogs, which can reveal details about the ML models and frameworks used by the target organization, enabling adversaries to craft realistic proxy models or tailor attacks. They also explore websites owned by the target organization, which often contain technical details about ML-enabled products, key employee names, departments, and even physical data center locations. Public app stores, such as Google Play or the iOS App Store, are another rich source; adversaries search for applications incorporating ML components, acquire and analyze them to understand implementation and identify entry points for exploitation.

Resource Acquisition: Building the Toolkit

With reconnaissance complete, adversaries move on to acquiring tools and infrastructure needed to execute attacks. They often search for publicly available ML artifacts such as datasets, models, and software stacks hosted on cloud storage or public repositories, which provide insight into the target's systems—for example, acquiring a dataset used by the victim to train a proxy model that simulates the target's environment offline. Adversaries may develop their own tools or repurpose existing open source libraries like CleverHans or the Adversarial Robustness Toolbox, originally designed for research, weaponizing them to create adversarial attacks. They may also acquire infrastructure such as cloud servers or consumer hardware to obtain the computational power needed for complex attacks. One of the most insidious tactics involves compromising the ML supply chain by introducing poisoned datasets or backdoored models into public repositories that are unknowingly used by the target organization, embedding vulnerabilities deep within victim systems, often without detection.

Case Study: LLM Package Hallucination Attack

In June 2024, researchers from Vulcan Cyber and Lasso Security identified a vulnerability in LLMs like ChatGPT enabling a new type of supply chain attack, where models frequently "hallucinate" nonexistent software package names when providing installation advice. By probing ChatGPT via its public API, they found the model incorrectly recommended a nonexistent huggingface-cli package instead of the legitimate huggingface-hub package. Researchers uploaded an empty package under the hallucinated name to PyPI and tracked over 30,000 downloads, revealing that multiple LLMs consistently produced identical hallucinations and many users blindly followed installation instructions. Malicious actors could publish harmful packages under commonly hallucinated names, and installing them could lead to arbitrary code execution, data theft, or system compromise—showing how AI hallucinations can propagate beyond misinformation into tangible security threats.

Key takeaways

  • Reconnaissance draws on public research papers, organizational websites, and app stores to gather target intelligence.
  • Resource acquisition involves obtaining public ML artifacts, weaponizing open source tools, and compromising the ML supply chain.
  • LLM package hallucinations created a real supply chain attack vector when researchers demonstrated a hallucinated package name received over 30,000 downloads.
0.0 (0 ratings)

Discussion (0)

No discussion on this article yet — be the first to comment.

Please sign in to join the discussion.