CWE-1434

Base Abstraction Level
Pillar — Highest-level weakness category
Class — Abstract, language-independent
Base — Specific enough to detect
Variant — Tied to specific technology
Compound — Requires multiple weaknesses Draft MITRE CWE Status
Stable — Fully reviewed and complete
Draft — Under development, may change
Incomplete — Partially defined by MITRE
Deprecated — No longer recommended
Obsolete — Replaced by another CWE
Insecure Setting of Generative AI/ML Model Inference Parameters

Description

The product has a component that relies on a generative AI/ML model configured with inference parameters that produce an unacceptably high rate of erroneous or unexpected outputs.

Generative AI/ML models, such as those used for text generation, image synthesis, and other creative tasks, rely on inference parameters that control model behavior, such as temperature, Top P, and Top K. These parameters affect the model's internal decision-making processes, learning rate, and probability distributions. Incorrect settings can lead to unusual behavior such as text "hallucinations," unrealistic images, or failure to converge during training. The impact of such misconfigurations can compromise the integrity of the application. If the results are used in security-critical operations or decisions, then this could violate the intended security policy, i.e., introduce a vulnerability.

Consequences

Integrity, Other — Varies by Context, Unexpected State

The product can generate inaccurate, misleading, or nonsensical information.

Other — Alter Execution Logic, Unexpected State, Varies by Context

If outputs are used in critical decision-making processes, errors could be propagated to other systems or components.

Mitigations

Phase: Implementation, System Configuration, Operation

Develop and adhere to robust parameter tuning processes that include extensive testing and validation.

Phase: Implementation, System Configuration, Operation

Implement feedback mechanisms to continuously assess and adjust model performance.

Phase: Documentation

Provide comprehensive documentation and guidelines for parameter settings to ensure consistent and accurate model behavior.

Detection

Automated Dynamic Analysis

Manipulate inference parameters and perform comparative evaluation to assess the impact of selected values. Build a suite of systems using targeted tools that detect problems such as prompt injection (CWE-1427) and other problems. Consider statistically measuring token distribution to see if it is consistent with expected results.

Manual Dynamic Analysis

Manipulate inference parameters and perform comparative evaluation to assess the impact of selected values. Build a suite of systems using targeted tools that detect problems such as prompt injection (CWE-1427) and other problems. Consider statistically measuring token distribution to see if it is consistent with expected results.