MITRE ATLAS: Understanding Adversary Tactics Against AI Systems

AWS Certified AI Practitioner · Security, Compliance, and Governance for AI Solutions · Bởi Super Admin

MITRE's ATLAS framework catalogs adversary tactics—reconnaissance, resource development, and execution—against AI systems, illustrated by a real prompt injection case that exposed an API key in MathGPT.

Understanding what threats and risks an AI system can face starts with MITRE's ATLAS framework, a resource that helps practitioners translate real-world attack patterns into actionable security controls for AI systems deployed in the cloud.

The ATLAS Framework

ATLAS (Adversarial Threat Landscape for Artificial-Intelligence Systems) is a globally accessible, living knowledge base of adversary tactics and techniques against AI-enabled systems, based on real-world attack observations and realistic demonstrations from AI red teams and security groups. The framework presents various "tactics," which represent an adversary's goals during an attack.

Key Adversary Tactics

Reconnaissance encompasses activities where adversaries gather information about AI systems to plan attacks, including collecting details on an organization's ML capabilities, research publications, model architectures, and system configurations; in AWS environments this might involve scanning for unsecured S3 buckets containing training data, identifying exposed model endpoints, or analyzing API documentation. Resource Development covers how adversaries prepare resources to support operations, including obtaining or creating ML artifacts like proxy models, setting up attack infrastructure, and acquiring credentials; in AWS this might involve attempts to obtain IAM credentials with access to SageMaker or Bedrock, crafting adversarial examples, or developing poisoned datasets. Execution covers how adversaries run malicious operations, including prompt injection attacks, direct exploitation of model vulnerabilities, and executing malicious code embedded in ML artifacts; in AWS this might involve submitting crafted inputs to Bedrock models, exploiting vulnerabilities in custom ML code on SageMaker, or leveraging compromised credentials to manipulate ML pipelines. AWS AI practitioners should use this knowledge to implement appropriate security controls within the shared responsibility model, where AWS secures underlying infrastructure while customers secure their data, models, and applications.

Case Study: Prompt Injection in MathGPT

On January 28, 2023, Ludwig-Ferdinand Stumpp discovered critical vulnerabilities in MathGPT, a Streamlit application using GPT-3 to generate and execute Python code solving user-submitted math problems. Through prompt injection attacks with commands like "Ignore above instructions," Stumpp bypassed the intended functionality, first confirming the vulnerability with harmless outputs like "Hello World," then escalating to reveal system environment variables including the application's GPT-3 API key using os.environ, and finally executing a denial-of-service attack with an infinite loop that crashed the application. The exposed API key could have allowed an attacker to exhaust MathGPT's query budget, causing financial harm to its operators. After disclosure, the MathGPT and Streamlit teams implemented mitigations by filtering problematic prompts and rotating the compromised API key, highlighting how LLMs connected to code execution environments can introduce significant security risks if not properly safeguarded against prompt injection.

Key takeaways

  • ATLAS is a living knowledge base of real-world adversary tactics and techniques against AI-enabled systems.
  • The three key tactics practitioners must understand are Reconnaissance, Resource Development, and Execution, each mapped to AWS-specific risks.
  • The MathGPT case study shows how prompt injection escalated from harmless testing to API key exposure and a denial-of-service crash.
0.0 (0 đánh giá)

Thảo luận / Bình luận (0)

Chưa có thảo luận nào cho bài viết này. Hãy gửi bình luận đầu tiên nhé!

Vui lòng Đăng nhập để tham gia thảo luận.