CWE-1426

Base Abstraction Level
Pillar — Highest-level weakness category
Class — Abstract, language-independent
Base — Specific enough to detect
Variant — Tied to specific technology
Compound — Requires multiple weaknesses Incomplete MITRE CWE Status
Stable — Fully reviewed and complete
Draft — Under development, may change
Incomplete — Partially defined by MITRE
Deprecated — No longer recommended
Obsolete — Replaced by another CWE
Improper Validation of Generative AI Output

Description

The product invokes a generative AI/ML component whose behaviors and outputs cannot be directly controlled, but the product does not validate or insufficiently validates the outputs to ensure that they align with the intended security, content, or privacy policy.

Consequences

Integrity — Execute Unauthorized Code or Commands, Varies by Context

Mitigations

Phase: Architecture and Design

Since the output from a generative AI component (such as an LLM) cannot be trusted, ensure that it operates in an untrusted or non-privileged space.

Phase: Operation

Use "semantic comparators," which are mechanisms that provide semantic comparison to identify objects that might appear different but are semantically similar.

Phase: Operation

Use components that operate externally to the system to monitor the output and act as a moderator. These components are called different terms, such as supervisors or guardrails.

Phase: Build and Compilation

During model training, use an appropriate variety of good and bad examples to guide preferred outputs.

Detection

Dynamic Analysis with Manual Results Interpretation

Use known techniques for prompt injection and other attacks, and adjust the attacks to be more specific to the model or system.

Dynamic Analysis with Automated Results Interpretation

Use known techniques for prompt injection and other attacks, and adjust the attacks to be more specific to the model or system.

Architecture or Design Review

Review of the product design can be effective, but it works best in conjunction with dynamic analysis.