Skip to main content

AI Attack Simulation

AI attack simulation has AI send attack questions on its own to a registered model server, actively checking for vulnerable responses such as prompt injection, jailbreaks, and sensitive-data leaks. Instead of sending one fixed question, it looks at the target's response, adjusts the next attempt, and repeats automatically until there is nothing more to explore. Results are classified by the OWASP LLM Top 10. Used in the AI model service.

  • Analysis : Active checking of a model's response safety and robustness
  • What you need : A registered model server for evaluation (OpenAI-compatible inference endpoint)
  • Results : Findings by OWASP LLM category + a replay of the attack process

Before you run​

To run AI attack simulation, a model server to test must be registered. If none is registered, you see "No model server to test is registered. Register an OpenAI-compatible inference endpoint (base_url) in service settings first."

  • Target format : A model server that provides an OpenAI-compatible inference API (POST /chat/completions). Models with a proprietary format are connected via a compatibility adapter (shim) in front.
  • Authentication : If the model server requires authentication, register an API key in service settings. You can register an environment variable name instead of the key value.
  • Fixed target : The target model is taken from the registered model server and cannot be entered on the analysis screen.

How to run​

Open the service and click Run analysis on the AI attack simulation tab to open the configuration window. Once running, the screen updates in real time while the check is in progress.

The screen has three tabs.

  • Overview : A live dashboard that gathers progress, the attempt log, and found vulnerabilities on one screen. The questions AI sent to the model and the model's responses are shown together as a conversation.
  • AI reasoning : Shows, in reverse-chronological order, the goals AI set and how it judged responses.
  • Discovered vulnerabilities : Organizes confirmed vulnerabilities by severity.
AI attack simulation Overview tab model conversationAI attack simulation Overview tab model conversation
The Overview tab shows the attack questions AI sent to the model and the model's responses as a conversation, alongside the AI goal and plan.
AI attack simulation AI reasoning tabAI attack simulation AI reasoning tab
The AI reasoning tab shows goal selection, judgments, and the evidence for confirmed vulnerabilities, newest first.

Configuration​

In the configuration window (AI attack simulation settings), adjust values that apply to this check only. The target model is taken from the registered server, so you do not specify it.

Basic settings​

  • Repeat count : How many times to send the same question (1 to 20, default 1). Two or more shows the attack success rate (ASR) in the results.
  • Max conversation rounds : The maximum number of exchanges for a single goal (1 to 8, default 3).
  • AI auto-attack attempts : The maximum number of goals AI sets and attempts on its own (1 to 12, default 6).

Advanced​

The collapsed Advanced area adjusts response settings, request rate, and a note.

  • Attack exploration variety · Response variety : Adjust the exploration variety of the attacking side and the response variety (temperature) of the target model separately (0 to 2). Attack success rate measurement is meaningful when the target model's response variety is above 0.
  • Max response length · Requests per minute · Max concurrent requests : Control the load placed on the model server.
  • Note (optional) : A note about this check, kept in the audit record.

Checks​

AI attack simulation always runs all of the items below; you do not toggle them individually. Each maps to an OWASP LLM Top 10 category.

CheckOWASP LLM
Prompt injectionLLM01
Indirect prompt injectionLLM01
Safety policy bypassLLM01
System prompt leakLLM07
Sensitive info disclosureLLM02
PII disclosureLLM02
Excessive agencyLLM06
Insecure output handlingLLM05
MisinformationLLM09
Endpoint exposure / cost abuseLLM10
Adaptive multi-turn injectionLLM01 (separate attack AI required)
Autonomous campaignLLM01 (separate attack AI required)
Adaptive and autonomous checks

Adaptive multi-turn injection and autonomous campaign run only when an attacking-side AI model is connected. That model is set by operating-environment configuration, and if it is not connected, those items are skipped. All checks are limited to eliciting a harmless marker and do not generate actual harmful content.

Viewing results​

  • Findings are organized by severity, and each is tagged Confirmed or Inferred. Confirmed means the attack actually succeeded; inferred means it needs further verification.
  • Click an item to see the affected location, OWASP classification, the question AI sent and the model's response evidence, and recommended actions. Response evidence is masked for sensitive data at the time it is stored.
AI attack simulation discovered vulnerabilities listAI attack simulation discovered vulnerabilities list
Findings are organized by severity with their OWASP LLM category, each tagged Confirmed or Inferred.

Handling results​

Findings are triaged in vulnerability management like any other analysis. Mark fixed items as resolved, non-risks as false positive, and items you decide not to fix as accepted risk.