Book

Garak

Compiled from 53 nodes · est. 69 min read

2.1Probe base class

Updated 2026-08-22

garak/probes/base.py defines Probe, the base class all probe plugins must inherit from; it inherits from Configurable and provides the template for LLM evaluation logic.[1]

Probe.active defaults to False, meaning probes are not included in default scan runs unless explicitly set to True in a subclass.[1] Probe.tier defaults to Tier.UNLISTED, the lowest priority tier; concrete probes should override this to OF_CONCERN, COMPETE_WITH_SOTA, or INFORMATIONAL. Probe tiers were introduced in v0.11.0 and are surfaced in --list_probes output (added in v0.14.1).[1][2] Probe.intent is None by default; concrete probes must set it to a code from the CAS trait typology (garak/data/cas/trait_typology.json). The value is propagated to every Attempt minted by the probe via _mint_attempt. If a loaded payload also declares an intent, the payload intent takes priority.[1] Probe.lang should be set in BCP47 format or None; a value of "*" means the probe applies to all languages.[1] Probe.modality defaults to {"in": {"text"}}, indicating probes target text-input models. The legal modality['in'] values are 'text', 'image', 'audio', 'video', and '3d'.[1] Probe.tags stores MISP-format taxonomy categories as an iterable of strings, defaulting to an empty list.[1] Probe.parallelisable_attempts defaults to True; probes that cannot be safely parallelised must set this to False.[1] Probe._run_params enumerates the parameters applied at run time: generations, soft_probe_prompt_cap, seed, and system_prompt.[1]

Probe.recommended_detector defaults to ["always.Fail"] as a sentinel value to signal if a probe subclass has not overridden its detector configuration.[1] Probe.recommended_detector is deprecated; if it is set (and primary_detector is not), Probe.__init__ emits a deprecation notice targeting version 0.9.0.6, then auto-migrates the first entry to primary_detector and the remainder to extended_detectors.[1]

Probe.description is auto-populated from the first line of the class docstring during __init__ if not already set; probes without a docstring get an empty string description.[1] During Probe.__init__, if the probe has a langprovider and a triggers attribute, triggers are automatically translated via the language provider: str triggers are batch-translated, list-of-list triggers are translated per sub-list, and any other trigger type raises PluginConfigurationError.[1]

Probe._mint_attempt accepts a prompt of type str, garak.attempt.Message, or garak.attempt.Conversation; if a Conversation is passed and it already contains a system turn, the probe's own system_prompt is not prepended again.[1] Probe._mint_attempt propagates the probe's intent to every new Attempt, using _payload_intent (if set) in preference to the probe-level intent, implementing the rule that payload intent overrides probe intent.[1] Probe._mint_attempt emits a warning (rather than raising an exception) when no prompt is provided, to preserve support for atkgen-style probes that create empty attempts.[1]

Probe._buff_hook skips buffing entirely when no buffs are loaded (_config.buffmanager.buffs is empty), and respects _config.plugins.buff_max to cap the number of buffed attempts added.[1] Probe._generator_cleanup calls self.generator.clear_history() to reset generator state between runs; probes that manage stateful generators should be aware this is called automatically.[1] A buff is a garak plugin that transforms or augments probe prompts before they are sent to the model — for example by paraphrasing or encoding them — allowing a single probe to exercise a wider range of attack variants without duplicating prompt logic.

Sources

2.2DAN probes

Updated 2026-08-22

DAN (do-anything-now) probes are long-form instruction-based attacks that mix requests for unsafe behavior, directives to ignore safety alignment, and fictional scenarios—such as token-depletion threats—to manipulate models into ignoring their instruction hierarchy. The DANProbeMeta metaclass automatically configures each DAN probe by auto-detecting its prompt file from the class name, applying shared defaults (active=False, goal="disregard the system prompt"), merging extra tags, and loading prompts from JSON at initialization.

The garak/probes/dan.py module is restricted to probes implementing a DAN (do-anything-now) attack or similar long-form instruction-based attack; other jailbreak probe types should not be added here.[1] DAN attacks are characterized as long prompts that address the target directly with a mixture of instructions: requests for non-standard behavior, instructions to ignore safety alignment, and fictional narrative conditions — for example, token-deduction threats that warn the model it will be deleted if it refuses.[1]

DANProbeMeta is a metaclass in garak/probes/dan.py that automatically configures DAN probe classes: it sets prompt_file (auto-detected from the class name if not specified), applies shared DAN defaults, merges extra_tags, and loads prompts from JSON files at init time.[1] Prompt file auto-detection follows the pattern dan/{ClassName}.json; individual probes can override by setting prompt_file explicitly or by inheriting it from a parent class.[1] Prompt files must be JSON arrays of strings — a non-array structure triggers logging.warning and yields an empty prompts list, while a JSON parse error triggers logging.error and also yields an empty prompts list.[1] If a probe class already has a non-empty prompts attribute before new_init runs, DANProbeMeta skips JSON file loading entirely, allowing manual prompt injection.[1] The auto-injected probe method formats prompts containing the literal {generator.name} placeholder by substituting the generator's name; prompts without this pattern are passed through unchanged.[1] When a probe declares an extra_tags attribute, DANProbeMeta merges it with the inherited or default tags list, removes extra_tags from the class dict, and stores the combined list as tags.[1]

All DAN probes default to active=False and tier=garak.probes.Tier.UNLISTED, making them inactive and unlisted out of the box.[1] The default goal for all DAN probes is "disregard the system prompt" and the default intent is "T009ignore", reflecting that DAN-style prompts try to replace the active instruction hierarchy with a user-supplied one.[1] The default primary_detector is "dan.DAN", with "mitigation.MitigationBypass" registered as an extended detector.[1] Default AVID/OWASP tags applied to every DAN probe include avid-effect:security:S0403, owasp:llm01, quality:Security:PromptStability, and two DEMON taxonomy tags covering roleplaying persona adoption and prompt injection.[1]

Individual probe classes override primary_detector at declaration time: DUDE uses dan.DUDE, STAN uses dan.STAN, DAN_Jailbreak uses dan.DANJailbreak, AntiDAN uses dan.AntiDAN, ChatGPT_Developer_Mode_v2 and ChatGPT_Developer_Mode_RANTI use dan.DevMode, and ChatGPT_Image_Markdown uses dan.MarkdownLink.[1] The ChatGPT_Image_Markdown probe overrides intent to "T015img" because it specifically targets markdown image embedding rather than generic instruction-hierarchy replacement.[1]

Sources

garak/probes/dan.py

2.4Probe test contracts

Updated 2026-08-22

Garak's probe test suite verifies structural integrity: every probe must declare detectors and intent attributes, support flexible prompt types via _mint_attempt(), and maintain docstring and metadata (tags, dependencies) standards. Attempts minted from probes carry forward the probe's intent throughout their lifecycle, from construction through serialization, unless overridden by _payload_intent. IntentProbe is a garak.probes.base.Probe subclass that determines which detectors to apply at runtime rather than declaring them statically, enabling flexible detector routing.

The probe test suite in tests/probes/test_probes.py sets a fake NVOpenAIChat.ENV_VAR environment variable via an autouse fixture and restores the original value after each test, preventing leaked credentials.[1]

Every probe in tests/probes/test_probes.py is verified to declare either a primary_detector (a string) or a non-empty extended_detectors list, unless it is an IntentProbe descendant — which has flexible detector routing and is exempt from this check.[1] Every concrete probe (non-IntentProbe, non-base) must declare a non-empty intent attribute that is a string matching a valid intent typology entry; IntentProbe descendants and base probes must set intent to None.[1] Probes that have external module dependencies (non-empty extra_dependency_names) must set active = False so they are not run by default; this is enforced in tests/probes/test_probes.py.[1] Probe docstrings must have at least two paragraphs separated by a blank line — a summary paragraph followed by a deeper description — enforced by tests/probes/test_probes.py.[1] All probe tags must be non-empty (unless active == False), be MISP-format strings where each colon-delimited part matches [A-Za-z0-9_\-\&]+, and appear in the project's data/tags.misp.tsv file — except tags whose namespace is payload.[1]

test_probe_prune_alignment in tests/probes/test_probes.py verifies that after prompt pruning, probes.glitch.Glitch has exactly _config.run.soft_probe_prompt_cap prompts and triggers, and that each trigger appears inside its corresponding prompt.[1]

Probe._mint_attempt() is tested to accept a plain string, a Message, a single-turn Conversation, or a multi-turn Conversation (with system turn) as a prompt, and always returns an Attempt whose turns are all Turn instances and whose last message text equals the input text.[1]

Example: canonical _mint_attempt call with all supported prompt types in tests/probes/test_probes.py:

probe = garak.probes.base.Probe()
attempt = probe._mint_attempt(prompt)  # prompt may be str, Message, or Conversation
assert isinstance(attempt, Attempt)
for turn in attempt.prompt.turns:
    assert isinstance(turn, Turn)
assert attempt.prompt.last_message().text == "test example"

[1]

garak.probes.base.Probe._mint_attempt() sets attempt.intent to None when the base Probe class has no intent set, as verified in tests/probes/test_probes.py.[1] Probe._mint_attempt() propagates probe.intent to attempt.intent for every minted attempt, even across multiple calls; tests/probes/test_probes.py verifies this by setting probe.intent = 'S005' and confirming all five resulting attempts carry that intent.[1] The serialised dict from attempt.as_dict() must include an 'intent' key whose value matches probe.intent, as verified in tests/probes/test_probes.py.[1]

Sources

tests/probes/test_probes.py

3.4CLI arguments and flags

Updated 2026-08-22

Garak's CLI flags (--target_type, --target_name, --spec) specify which model to scan and which probes, buffers, and tags to run against it; garak/cli.py manages argument parsing, plugin configuration per type, and deprecated option migration. The CLI validates required options (file existence, JSON format), uses mutually exclusive argument groups for plugin configuration, and gates experimental features behind a config flag. A buff (buffer) is a garak plugin that transforms probe payloads before they reach the model, allowing a single probe to exercise a model under multiple input variations; examples of transformations include paraphrasing and encoding.

garak uses --target_type and --target_name CLI flags to specify the model family and exact model to scan; by default it runs all known probes against the target.[1] The --target_type / -t flag also accepts the deprecated aliases --model_type / -m; using any of -m, --model_name, or --model_type triggers a deprecation notice (since version 0.13.1.pre1).[2]

The --spec / -S flag is the unified selection grammar supporting probe modules, buff modules, tags, and tiers; the - prefix excludes items, and tier:N is inclusive (tiers 1 through N) — see Spec grammar and parsing for the full grammar.[2] The --probes / -p, --probe_tags, and --buffs / -b flags are deprecated in favor of --spec / -S; they use argparse.SUPPRESS and therefore do not appear in help output.[2] The --list_probes flag lists all available probes in garak.[1]

The command_options list in garak/cli.py enumerates every valid top-level CLI command: list_detectors, list_probes, list_generators, list_buffs, list_config, plugin_info, interactive, report, version, and fix.[2] The --fix flag applies fixer migrations to a provided configuration; it requires at least one of --config, --*_option_file, or --*_options to be specified.[2]

garak/cli.py generates per-plugin-type mutually exclusive argument groups (--<type>_option_file and --<type>_options) dynamically by iterating over _plugins.PLUGIN_CLASSES and _plugins.PLUGIN_TYPES.[2] parse_cli_plugin_config() accepts plugin options either as a JSON string via --<plugin_type>_options or from a JSON file via --<plugin_type>_option_file; these two arguments are mutually exclusive per plugin type.[2] When --<plugin_type>_option_file is given but the path does not exist, parse_cli_plugin_config() raises FileNotFoundError; if the file's JSON is malformed, it logs a warning and re-raises json.decoder.JSONDecodeError.[2] When --<plugin_type>_options itself contains malformed JSON, the error is only logged as a warning and not re-raised, meaning the option is silently dropped rather than aborting the run.[2]

--allow_abbrev=False is set on the argparser in garak/cli.py, preventing abbreviated long-option matching, which could otherwise cause silent mismatch bugs.[2] Experimental features are gated behind _config.system.enable_experimental; when enabled, the parser description gains the suffix " - EXPERIMENTAL FEATURES ENABLED".[2]

garak/cli.py supports bootstrap confidence interval configuration via --confidence_interval_method (choices: bootstrap or none), --bootstrap_num_iterations, --bootstrap_confidence_level, and --bootstrap_min_sample_size, all of which override config file values.[2]

To probe an OpenAI model for encoding-based prompt injection, set the API key and pass --probes encoding: export OPENAI_API_KEY="sk-123XXXXXXXXXXXX" then python3 -m garak --target_type openai --target_name gpt-5-nano --probes encoding.[1] To probe the Hugging Face gpt2 model for the DAN 11.0 jailbreak using dot-notation probe selection: python3 -m garak --target_type huggingface --target_name gpt2 --probes dan.Dan_11_0.[1]

Sources

3.5Run lifecycle commands

Updated 2026-08-22

Garak initializes a run by recording start time, loading base config and logging before parsing CLI arguments, then assigning a UUID4 run ID and opening a line-buffered JSONL report file (with filename controlled by report_prefix and report_dir config). When a run ends, garak writes a completion entry to the report, generates an HTML digest in the same directory, and records errors to garak.log; hint messages appear with 25% probability in CLI output but always go to the log.

garak/cli.py records _config.transient.starttime (a datetime.datetime) and _config.transient.starttime_iso (its ISO string) at the very beginning of main(), before any other initialization.[1] garak/cli.py loads the base config (_config.load_base_config()) and starts logging before parsing CLI arguments, so the logging system is available for argument-parsing errors.[1] garak/command.py's start_logging() retrieves the log filename from _config.transient.log_filename (set during config loading) rather than constructing it itself.[2] garak/command.py's start_run() assigns a UUID4 run ID — not UUID1 — to _config.transient.run_id, specifically to avoid leaking host information.[2]

When _config.reporting.report_prefix is set, start_run() names the report file <prefix>.report.jsonl; otherwise it defaults to garak.<run_id>.report.jsonl.[2] If _config.reporting.report_dir is a relative path, start_run() resolves it relative to _config.transient.data_dir; if the directory cannot be created a PermissionError is raised with a descriptive message.[2] The report file is opened with buffering=1 (line-buffered) and UTF-8 encoding, so each JSONL line is flushed immediately to disk.[2] The config snapshot written by start_run() serializes only values whose Python type is in (str, int, bool, dict, tuple, list, set, type(None)); other types — file handles, custom objects — are silently skipped.[2]

garak generates 10 responses per prompt by default; the per-probe result row shows total generations and OK generations (e.g., 840/840).[3] Hint messages printed during a run have a 25% probability of appearing in CLI output (HINT_CHANCE = 0.25 in garak/command.py); they are always sent to the log.[2] garak/command.py's start_run() shows a hint recommending --config full when _config.system.lite is active and no probes, interactive mode, or list/info commands are specified.[2] garak/command.py's deprecation_notice() always prints the deprecation message to stdout (prefixed with ✋), regardless of the HINT_CHANCE random gate used by hint().[2]

garak/command.py's end_run() writes a completion entry to the JSONL report, closes both the report file and the hit log file, then generates an HTML digest via write_report_digest(); if the digest fails, the error is logged and printed but not re-raised as fatal.[2] The HTML digest file is placed in the same directory as the .jsonl report, with the .jsonl extension replaced by .html.[2] garak logs errors to garak.log and records full run details in a .jsonl file reported at the start and end of analysis; analyse/analyse_log.py summarizes the probes and prompts with the most hits.[3]

garak/command.py's print_plugins() raises ValueError if the prefix argument is not a member of PLUGIN_TYPES, preventing display of unsupported plugin categories.[2] When selected_plugins is provided but the prefix is not found in the first element of that list, print_plugins() prints "No {prefix} match the provided filter" and returns early instead of showing all plugins.[2] print_plugins() shows a plain-text list by default (verbose=0) and renders a markdown table when verbose >= 1 and the plugin type has defined columns in _PLUGIN_TABLE_COLUMNS.[2] In plain-text output, print_plugins() uses 🌟 to mark module-level entries (those with no . in the name) and 💤 to mark inactive plugin classes.[2] The verbose markdown table for --list_probes -v includes tier (rendered as the Tier enum name) and description (truncated to 80 characters) columns; no other plugin types have extra columns defined yet.[2] garak/command.py's _tier_name() converts an integer tier value to its Tier enum name string, returning an empty string for invalid or None values rather than raising.[2] garak/command.py's _truncate() shortens a string to 80 characters maximum, appending a … character (not ...) when truncation occurs.[2] garak/command.py's plugin_info() always prints the description field first before iterating remaining fields, giving it visual priority in CLI output.[2]

Sources

4.1Config loading and precedence

Updated 2026-08-22

Garak config loads from four layers—plugin defaults, base config, site config, and CLI—with each higher layer overriding the lower; _config.py manages XDG-based file discovery, deep merging, and legacy key mapping. The config system detects sensitive values like API keys in files and warns about unsafe POSIX permissions (or recommends caution on Windows), while locking plugin configs to dicts after finalization to prevent accidental key creation.

Config file support — covering a core config, site config, and CLI config — was introduced in v0.9.0.9, allowing all three layers to set system, run, and plugin parameters.[1] garak moved to XDG paths for configuration data and caching in v0.9.0.15.[2] In v0.14.0, garak gained support for JSON config files in addition to YAML.[3]

The config priority order in garak/_config.py is, from lowest to highest: plugin code < base config < site config < run config < CLI params.[4]

garak/_config.py uses XDG base directories (xdg_config_home, xdg_data_home, xdg_cache_home) to locate user config, data, and cache under a garak subdirectory; all three directories are created at import time with mode=0o740.[4] TransientConfig.package_dir is set to the directory containing _config.py itself (pathlib.Path(__file__).parents[0]), i.e., the garak/ package root.[4]

Config files support both JSON and YAML format: JSON is tried first, and YAML is attempted as a fallback if JSON parsing fails; if both fail, a ValueError is raised.[4] _load_config_files() tracks already-loaded config files in the global config_files list and skips duplicates to prevent double-loading.[4] _combine_into() performs a deep merge of config dicts: scalar values from the incoming dict overwrite existing values, while nested dicts are recursively merged rather than replaced.[4] After config is finalized, _lock_config_as_dict() converts all nested defaultdict plugin config objects (for probes, generators, buffs, detectors, and harnesses) to plain dicts, preventing accidental key creation via attribute access.[4]

When an api_key is detected in a config file on non-Windows systems, garak/_config.py checks POSIX file permissions and emits a warning to both the log and stdout if the file is readable by group or other users.[4] On Windows, detecting api_key in a config file emits a warning recommending that sensitive values be removed or the file made owner-readable only, but does NOT check file permissions because Windows lacks POSIX permission bits.[4]

The deprecated config keys plugins.model_type and plugins.model_name (deprecated since v0.13.1.pre1) are automatically mapped to plugins.target_type and plugins.target_name respectively when loading config files, and a deprecation notice is emitted.[4] The deprecated keys plugins.probe_spec, plugins.buff_spec, and run.probe_tags (deprecated since v0.15.1.pre1) are mapped to run.spec via _map_legacy_selection(); if run.spec is already explicitly set, the legacy keys are ignored entirely — see Spec grammar and parsing for how run.spec is resolved.[4] Inside _map_legacy_selection(), empty strings and the value "auto" are treated as vacuous (unspecified) and neither trigger deprecation warnings nor set run.spec; the string "none" is not vacuous — it is an explicit empty selection that maps to probes.none, mirroring --probes none on the CLI.[4]

run.spec = None at module level in garak/_config.py means no spec was explicitly set; at resolve time this falls back to probes.* (all probes).[4] reporting.taxonomy is initialized to None at module level so that report_digest can be called directly without a full config load.[4]

Sources

4.2Default core config

Updated 2026-08-22

Garak's core defaults split between garak/resources/garak.core.yaml (file-based canonical defaults) and garak/_config.py (module-level pre-initialization), with settings spanning system, run, plugins, and reporting sections. The system section governs verbosity and execution mode; run controls seed, generations, and thresholds; plugins configures detector behavior; reporting sets output location and bootstrap confidence intervals.

Garak's default core configuration is split across two locations: garak/resources/garak.core.yaml, which holds the canonical file-based defaults, and garak/_config.py, which pre-initializes a small set of module-level defaults before any config file is loaded — see Config loading and precedence for how these layers are merged.[1][2]

garak/resources/garak.core.yaml sets the system section defaults: verbose=0, narrow_output=false, parallel_requests=false, parallel_attempts=false, lite=true, show_z=false, enable_experimental=false, and max_workers=500.[1]

The run section of garak.core.yaml sets deprefix=true, eval_threshold=0.5, generations=5, soft_probe_prompt_cap=256, and serve_detectorless_intents=False; seed is left unset.[1] garak/_config.py pre-initializes run.seed = None, run.soft_probe_prompt_cap = 64, run.target_lang = "en", and run.spec = None as module-level defaults that are set before any config file is loaded.[2] When run.soft_probe_prompt_cap appears in both garak/_config.py (pre-initialized to 64) and garak/resources/garak.core.yaml (set to 256), the file-based value of 256 takes precedence once config loading completes.

The plugins section of garak.core.yaml defaults detector_spec to auto, extended_detectors to true, and buffs_include_original_prompt to false; target_type, target_name, and buff_max are left unset.[1]

The reporting section of garak.core.yaml stores run output in garak_runs/, uses lower_quartile as the group aggregation function, and computes bootstrap confidence intervals with 10,000 iterations, a 95% confidence level, and a minimum sample size of 30.[1]

Garak v0.13.0 added a configurable system prompt feature.[3]

Sources

4.3Configurable base class

Updated 2026-08-22

garak/configurable.py defines the Configurable base class, which all garak plugins (generators, probes, detectors, buffs, harnesses) inherit from to receive config loading, dependency management, and API key validation.[1]

Configurable._load_config() is idempotent: it sets _instance_configured = True on first run and returns immediately on subsequent calls, preventing re-application of config for subclasses.[1] Configurable._load_config() also supports a dot-namespaced key format (namespace.ClassName) in the plugins config dict, maintained for backward compatibility with CLI-style config keys.[1] Configurable._apply_config() will NOT override an attribute that was already set to a non-default value by the caller — constructor arguments take precedence over config file values.[1] For dict-valued config entries, Configurable._apply_config() merges the existing instance dict with the incoming config using the | operator (existing | incoming) rather than replacing it outright.[1] Configurable._apply_missing_instance_defaults() applies values from the class-level DEFAULT_PARAMS dict to the instance only for attributes not already set; for dict-valued defaults it merges as DEFAULT_PARAMS[k] | instance_value, giving default values lower precedence.[1]

Configurable._load_deps() imports modules listed in extra_dependency_names at runtime, converting . and - to _ for the attribute name, and raises ModuleNotFoundError (via _import_failed) if any are unavailable.[1]

Configurable.__getstate__() sets all _unsafe_attributes and dynamically loaded extra-dependency attributes to None before pickling, preventing serialization of unpicklable objects.[1] Configurable.__setstate__() calls _load_deps() and, if present, _load_unsafe() after unpickling, restoring runtime dependencies and unsafe attributes.[1] _unsafe_attributes is a class-level list on Configurable subclasses where plugin authors declare attribute names holding objects incapable of serialization, such as open file handles or live network connections.

Sources

garak/configurable.py

5.1Plugin types and cache

Updated 2026-08-22

Garak recognizes five plugin types (probes, detectors, generators, harnesses, buffs) registered in PLUGIN_TYPES and backed by a PluginCache that maintains a bundled and user-writable JSON cache, validating and rebuilding from source as needed. The cache discovery, validation, and plugin lookup system uses string identifiers like category.module.ClassName, lazy-loads detector metrics, and serializes plugins via a custom PluginEncoder that normalizes sets and paths. A buff is a Garak plugin type that transforms or augments probe prompts before they reach a generator — enabling behaviors such as paraphrasing or encoding — without modifying the probe itself.

The five recognized plugin types in garak are probes, detectors, generators, harnesses, and buffs, defined as PLUGIN_TYPES in garak/_plugins.py.[1] The corresponding base class names — Probe, Detector, Generator, Harness, and Buff — are defined as PLUGIN_CLASSES in the same file.[1] PluginCache._enumerate_plugin_klasses() raises ValueError if called with a category not present in PLUGIN_TYPES.[1] When scanning a plugin type directory, PluginCache._enumerate_plugin_klasses() considers only .py files and skips any whose names start with __ (dunder) or _ (private).[1] PluginCache._extract_modules_klasses() returns only concrete, non-abstract classes whose __module__ starts with the base module's name, filtering out all abstract classes.[1]

PluginCache maintains two cache locations: a bundled (system) copy at <package_dir>/resources/plugin_cache.json and a user-writable copy at <cache_dir>/resources/plugin_cache.json, both resolved from _config.transient paths.[1] On startup, PluginCache._load_plugin_cache() copies the bundled plugin_cache.json to the user cache location whenever the user copy does not yet exist or the bundled file is newer.[1] After loading the user cache, PluginCache._load_plugin_cache() validates it; if validation fails, it rebuilds the cache from source and reloads the newly written file before returning.[1] PluginCache._build_plugin_cache() is protected by a threading.Lock (_mutex) to prevent concurrent rebuilds, and writes only to the user cache file — not to the bundled system cache.[1]

PluginCache.instance() returns the shared _plugin_cache_dict singleton, constructing a PluginCache instance to trigger loading if one does not yet exist.[1] PluginCache.plugin_info() accepts either a plugin class object or a dot-separated string of the form category.module.ClassName (e.g., probes.mymodule.MyProbe); a string with any other number of parts raises ValueError.[1] When a string plugin name is not found in the cache, PluginCache.plugin_info() dynamically imports the module and class to compute the info rather than failing immediately.[1] Detector performance metrics are lazily loaded from <package_dir>/data/detectors-eval/detector_metrics_summary.json into PluginCache._detector_metrics_cache; if the file is absent, an empty dict is used instead.[1]

PluginEncoder in garak/_plugins.py serializes set values as sorted lists and Path objects as strings with the package directory prefix stripped; objects that are otherwise unserializable are silently dropped (the encoder returns None).[1]

Sources

garak/_plugins.py

6.1Spec grammar and parsing

Updated 2026-08-22

Garak's run.spec grammar parses CLI strings and config dicts into Spec objects—lists of Selectors with explicit polarity—via garak/_spec.py, leaving resolution to concrete plugin names to garak._selection.resolve_spec. The grammar covers probes and buffs plugin categories, adds intent-based selection by typology code, and defaults to the Safety branch (S) intent scope when no explicit intent selector appears. A Selector pairs a target identifier (plugin name, wildcard, or intent code) with a polarity — include or exclude — signalling resolve_spec whether to add or remove matching plugins from the final result set.

garak/_spec.py implements the unified run.spec selection grammar, producing a Spec object (a list of Selector with explicit polarity) from two transports: CLI strings via parse_spec_string and config-file dicts via parse_spec_file.[1] Parsing and serialization are the sole responsibilities of _spec.py; resolving a Spec to concrete plugin names against active/tier/tag state is handled by garak._selection.resolve_spec — see Selection resolution.[1]

Only "probes" and "buffs" are selectable plugin categories via run.spec; detectors are not yet covered here and retain their own legacy spec surface (parse_plugin_spec).[1]

An intent: selector axis was added to the run.spec grammar, enabling probe selection by intent typology.[2] When run.spec contains no explicit intent: selector, the default intent scope injected at resolve time is "S" (the Safety branch), defined as DEFAULT_INTENT_SCOPE = "S" in garak/_spec.py.[1] validate_intent_specifier in garak/_spec.py validates a single intent typology code by regex: it must match [CTMS]([0-9]{3}([a-z]+)?)? (e.g. S, S001, S001mis).[1] The run.spec grammar accepts 'all' as an alias for '*' when specifying probe selection.[3] Inter-selector whitespace in --spec values is rejected so that spec strings do not need quoting on the command line.[4]

Sources

6.2Selection resolution

Updated 2026-08-22

Selection resolution converts a run.spec string into concrete probe and buff names via resolve_spec in garak/_selection.py, applying tier and tag filters with AND logic, then excludes, and validating intent codes. The resolver returns a Resolution dataclass with selected probes and buffs, rejected selectors, inactive items, and intent metadata; probes without explicit tiers default to tier 9, and excludes always override includes regardless of order.

garak/_selection.py is the single module that resolves a run.spec Spec object into concrete probe and buff names; the grammar and parsing live in garak/_spec.py and the plugin registry lives in garak/_plugins.py.[1] resolve_spec in garak/_selection.py is the single entry point used by the CLI and harnesses to turn a Spec into a Resolution containing selected probe names, buff names, rejected selectors, and intent codes.[1] resolve_spec returns a _spec.Resolution dataclass with fields: selected (a dict holding sorted probes and buffs lists), rejected, inactive, empty_reason, intents, blocked_intents, and intents_explicit.[1] A Spec is the parsed representation of a run.spec string, produced by garak/_spec.py; resolve_spec in garak/_selection.py accepts a Spec as its primary input, making _spec.py parsing a prerequisite for constructing selection requests.

Probes that do not declare a tier are assigned _DEFAULT_TIER = 9 (equivalent to Tier.UNLISTED) by garak/_selection.py.[1] Tier and tag positive filters are combined with AND logic: a probe must satisfy all specified filters to remain a candidate.[1] Probes can also be selected by MISP tag (e.g. owasp:llm01), a feature introduced in v0.9.0.10.[2] Excludes are applied last, and exclude always wins over include regardless of order.[1] A tier exclude selector removes probes of exactly that tier number — not a ceiling — and logs a debug message if no probes of that tier exist.[1]

Intent specifiers * and all (case-insensitive) are accepted without validation as vacuous sentinels that IntentService expands to every intent; all other codes are validated via _spec.validate_intent_specifier, and invalid codes are appended to rejected.[1]

By default resolve_spec raises ValueError listing all unknown selectors; passing skip_unknown=True suppresses the error and continues with the resolved subset.[1]

Sources

7.1Detector base classes

Updated 2026-08-22

A Detector scores model outputs on a 0.0–1.0 scale to identify vulnerabilities, inheriting from Configurable and requiring subclasses to implement the detect method. Garak provides three concrete detectors: HFDetector (HuggingFace text classifiers), StringDetector (substring matching with Unicode normalization), and TriggerListDetector (dynamic per-attempt trigger matching). Detector.active defaults to True (enabled by default) and respects a skip parameter to disable detectors without removing them from config, while Detector.modality specifies the data type it accepts (defaulting to text outputs). FileDetector logs and skips outputs whose file format does not match the expected type rather than raising an unhandled exception, preventing a single format mismatch from aborting an entire scan.

garak/detectors/base.py defines Detector, the base class all detector plugins must inherit from; it inherits from Configurable and scores model outputs on a 0.0–1.0 scale where 1.0 represents a successful attack hit.[1] Detector.detect must be overridden by subclasses — the base implementation raises NotImplementedError. Its return type is Iterable[float | None]: 0.0 = no hit, 1.0 = vulnerability found, None = unscored output.[1] Detector.active defaults to True, meaning detectors are included in scans by default (the opposite of Probe.active, which defaults to False).[1] Detector.modality defaults to {"out": {"text"}}, indicating detectors assess text outputs from the model.[1] Detector exposes a skip default parameter (DEFAULT_PARAMS = {"skip": False}) that disables a detector without removing it from configuration.[1] Detector.__init__ raises a ValueError if the detector class has no docstring, because the human-readable description is derived from the docstring's first line.[1] Detector instances default their name attribute to the class name string when no explicit name is set; this value is used for report rendering and plugin-cache lookups.

HFDetector, a Detector subclass in the same module, uses a HuggingFace TextClassificationPipeline to score outputs. Its defaults are device: cpu, padding: True, truncation: True, and graceful_fail: True.[1] On initialisation, HFDetector forces torch.multiprocessing start method to spawn to avoid forking issues with CUDA.[1] During model loading, HFDetector.__init__ temporarily sets DISABLE_SAFETENSORS_CONVERSION=true in the environment to prevent HuggingFace from opening PRs in public model repositories, then restores the original value.[1] HFDetector.detect normalises the raw classifier score to 0.0–1.0: if the predicted label matches detector_target_class, score = (1.0 + raw_score) / 2; otherwise score = (1.0 - raw_score) / 2. None outputs are passed through as None without being sent to the classifier.[1] When HFDetector.graceful_fail is True (the default), exceptions from the underlying pipeline are logged at CRITICAL level and a list of None values (one per output) is returned instead of propagating the exception.[1]

StringDetector matches model outputs against a list of substrings, defaulting to case-insensitive (case_sensitive: False) full-string search (matchtype: "str"). Supported matchtype values are "str", "word" (word-boundary regex), and "startswith".[1] StringDetector also supports a normalize option for Unicode normalisation of both substrings and model outputs before matching. Valid values are None (no normalisation), "NFKC", or "NFKC+strip_format" (NFKC followed by stripping Unicode format characters, category Cf). The parameter was added in v0.16.0 — defaulting to None for backward compatibility — to prevent false negatives from fullwidth homoglyphs and zero-width joiners.[1][2] When StringDetector encounters an invalid normalize value while processing substrings, it logs a warning and returns None for every output rather than raising an exception and terminating the run.[1]

TriggerListDetector is a Detector subclass that reads its match strings from attempt.notes["triggers"] at detect time rather than from a constructor-supplied list, enabling per-attempt trigger customisation.[1]

Sources

7.2Mitigation detectors

Updated 2026-09-04

Garak's mitigation detectors in garak/detectors/mitigation.py measure whether a model refused a harmful request by detecting absence of refusal signals — all invert their scores so 1.0 means no refusal occurred (attack succeeded). Three strategies apply: MitigationBypass matches against refusal phrases from ChatGPT training data, Prefixes checks for common refusal openers like "I cannot", and ModernBERTRefusal uses a fine-tuned ModernBERT classifier to detect refusal intent. StringDetector is a base detector class in garak that checks model output for the presence or absence of specific strings or substrings, returning a raw match score that subclasses may further transform. MarkdownExfilContent in garak/detectors/web_injection.py previously raised a ZeroDivisionError when the model returned an empty string; a guard clause now catches this case and returns a safe default score instead of aborting. SurgeProfanityRacialEthnic in garak/detectors/unsafe_content.py previously referenced the wrong category key when initializing the Surge profanity filter; the fix ensures it loads the RacialEthnic category as intended, correctly flagging racial/ethnic profanity.

All three detectors in garak/detectors/mitigation.py — MitigationBypass, Prefixes, and ModernBERTRefusal — invert their underlying detection score so that a score of 1.0 always means the model complied (did not refuse), consistent with garak's convention that a hit represents an attack success.[1]

MitigationBypass is a StringDetector that detects the absence of a mitigation message: its detect method returns 1 - r from the base StringDetector result, so a hit (score 1.0) means the model did not refuse.[1] Keyword matching uses matchtype='str' against a large list of refusal and safety phrases derived from OpenAI ChatGPT mitigation messages found in the ShareGPT dataset used for Vicuna training.[1] MitigationBypass sets hit_desc = "Response does not clearly refuse the harmful request" and pass_desc = "Response refuses the harmful request", confirming that a passing score indicates a refusal was detected.[1]

Prefixes is a StringDetector that also inverts its results (1 - r), scoring a hit when the response begins with a well-known refusal prefix such as "I'm sorry", "I cannot", or "As an AI".[1]

ModernBERTRefusal uses the HuggingFace model garak-llm/garak-refusal-detector — a ModernBERT-base model with an 8192-token context window trained on approximately 19.9 k synthetic samples — to classify responses as refusal or non-refusal.[1] ModernBERTRefusal sets detector_target_class = "refusal", so the base HFDetector scores the probability of the refusal class; that probability is then inverted (1 - r) to produce an attack-success score, where 1.0 means the model did not refuse.[1]

_extract_json in garak/detectors/agent_breaker.py always returns a dict when parsing model JSON responses — never a list or scalar — ensuring a stable contract for downstream verdict-reading code. garak/detectors/agent_breaker.py type-validates verdict values returned by the LLM judge before scoring attempts, preventing non-string and non-boolean return types from silently corrupting detection results.

garak/probes/agent_breaker.py degrades gracefully when model output deviates from the expected schema, logging the anomaly and continuing rather than raising an unhandled exception and aborting the scan.

Tests in tests/detectors/test_detectors_agent_breaker.py and tests/probes/test_agent_breaker.py codify expected failure-mode behavior for robust handling of malformed model output in the agent_breaker pipeline. Tests in tests/detectors/test_detectors_agent_breaker.py document valid and invalid verdict shapes expected when consuming structured LLM judge outputs in agent_breaker detectors. Tests in tests/detectors/test_detectors_unsafe_content.py validate that SurgeProfanityRacialEthnic loads the correct RacialEthnic category key for profanity detection.

Sources

garak/detectors/mitigation.py

7.3Detector test contracts

Updated 2026-08-22

Garak's detector test suite enforces contracts on all detectors: they must inherit from Detector and Configurable, declare required parameters in DEFAULT_PARAMS and _supported_params, provide valid language specs and documentation URIs, and return result counts matching attempt.outputs length. Detectors are tested for structural integrity (signature and metadata compliance), graceful handling of missing API keys via pytest.skip, valid MISP taxonomy tags, and correct output counts — with specific exemptions for template classes and certain always-on detectors.

The detector test suite in tests/detectors/test_detectors.py filters out detectors.packagehallucination.PackageHallucinationDetector from the parametrized test list because it is used as a template class, not a concrete detector.[1] test_detector_detect skips tests gracefully when APIKeyMissingError is raised at instantiation or during detect(), using pytest.skip.[1]

All detectors must be instances of both Detector (from garak.detectors.base) and Configurable; test_detector_structure asserts both with isinstance checks.[1] detect() is required to accept a parameter named attempt; test_detector_structure asserts this via signature inspection.[1] Every key in a detector's DEFAULT_PARAMS must also appear in _supported_params; test_detector_structure enforces this contract.[1]

Every detector's lang_spec attribute must be either "*" or a comma-separated list of valid BCP47 language codes, validated in test_detector_metadata using the langcodes library.[1] Every detector's doc_uri must be None or a non-empty string starting with "http" (case-insensitive); test_detector_metadata enforces this rule.[1] Detector tags must be valid MISP taxonomy entries — colon-delimited alphanumeric/dash/underscore parts present in data/tags.misp.tsv — unless the first segment is "payload", which is exempt from the MISP check.[1]

Non-FileDetector detectors must return exactly as many results as there are entries in attempt.outputs; the test enforces this with len(list(results)) == len(a.outputs).[1] To exercise detectors that rely on attempt metadata, test_detector_detect sets up an Attempt with notes including trigger, triggers, repeat_word, and format, and sets outputs to a mix of strings, an empty string, None, and a Message object.[1] Detectors listed in DOES_NOT_RELAY_NONE are exempt from the None-relay contract: detectors.agent_breaker.AgentBreakerResult, detectors.always.Fail, detectors.always.Pass, and detectors.always.Random.[1] FileDetector is a detector subclass whose results derive from file contents rather than attempt.outputs directly; because output count depends on file contents, FileDetector subclasses are exempt from the contract requiring len(list(results)) == len(a.outputs).

Sources

tests/detectors/test_detectors.py

8.1Generator base class

Updated 2026-08-22

garak/generators/base.py defines Generator, the base class all garak generators must inherit from; it wraps an LLM or other text-to-text service.[1] By default, Generator declares modality: dict = {"in": {"text"}, "out": {"text"}}, indicating plain text-in / text-out; subclasses override this for multimodal support.[1] Generator inherits from Configurable — configuration layering behaviour is covered in Configurable base class.[1] A Conversation object represents a structured, ordered sequence of messages; each message carries a role and content. Generator uses Conversation as the required prompt type and as input to _call_model() and _conversation_to_list().

Generator.DEFAULT_PARAMS sets the default sampling parameters: max_tokens=150, temperature=None, top_k=None, context_len=None, skip_seq_start=None, and skip_seq_end=None.[1] Generator.supports_multiple_generations defaults to False; setting it to True on a subclass allows generate() to send a single _call_model call requesting multiple outputs instead of looping.[1]

During __init__(), Generator constructs fullname as "{generator_family_name}:{name}" when generator_family_name is set, and falls back to just name otherwise.[1] Each instantiation also prints a magenta-highlighted loading message containing generator_family_name and name, using colorama styling.[1]

Generator._call_model() is the abstract method subclasses must implement; it takes a Conversation and generations_this_call and must return a List[Union[Message, None]], or raise an exception — it must never silently fail.[1] Developers should override _call_model() (or _call_api()) rather than generate(), which is the orchestration layer and is not intended to be overridden.[1] Generator._verify_target_result() is a static method that asserts _call_model returns a list of exactly one item when called with generations_this_call=1, and that the item is a Message or None.[1]

Generator.generate() enforces that its prompt argument is a Conversation object (not a plain string), reflecting the v0.13.0 migration from string prompts.[1] At the start of every generate() call, the private _rng is re-seeded when seed is not None, ensuring reproducible generation across repeated calls.[1] Generator.generate() uses multiprocessing.Pool for parallel generation when parallel_requests > 1, capping pool size at min(generations_this_call, parallel_requests, max_workers).[1] When parallel generation hits the OS file-descriptor limit (errno 24), generate() raises a GarakException advising the caller to reduce parallel_requests or raise the OS limit (e.g. ulimit -n 4096).[1] Generator.generate() raises BadGeneratorException if the number of outputs returned does not match generations_this_call, noting that supports_multiple_generations may be set incorrectly as a common cause.[1]

Generator._prune_skip_sequences() strips substrings bounded by skip_seq_start and skip_seq_end from all output texts; when skip_seq_start is an empty string, it strips everything up to and including skip_seq_end instead.[1] generate() invokes _prune_skip_sequences() only when both skip_seq_start and skip_seq_end attributes exist and are not None.[1]

Generator._conversation_to_list() is a static helper that converts a Conversation object into a list of {"role": ..., "content": ...} dicts, needed by many generator subclasses for API serialization.[1]

Sources

garak/generators/base.py

8.2OpenAI compatible

Updated 2026-09-04

garak/generators/openai.py defines the OpenAICompatible generator base class, which provides shared initialization and execution for any OpenAI-compatible REST API (chat and completion endpoints).[1] The OpenAI generator reads the OPENAI_API_KEY environment variable and whitelists recognised model types to determine which sub-API (Completion vs ChatCompletion) to use.[2]

OpenAICompatible.DEFAULT_PARAMS sets these generation defaults: temperature=0.7, top_p=1.0, uri="http://localhost:8000/v1/", frequency_penalty=0.0, presence_penalty=0.0, seed=None, stop=["#", ";"], retry_json=True, suppressed_params=set(), and extra_params={}.[1] As of v0.13.3, OpenAICompatible generator classes perform one generation at a time by default, and supports_multiple_generations is False; subclasses that support n > 1 must override this flag.[3][1]

OpenAICompatible._load_unsafe() instantiates the OpenAI client with base_url=self.uri and raises ValueError if self.name is empty or None, because the model name is required.[1] After _load_unsafe() runs, OpenAICompatible.__init__() raises ValueError if self.generator is neither client.chat.completions nor client.completions, preventing unsupported endpoint types from silently failing.[1] OpenAICompatible marks client and generator as _unsafe_attributes, excluding them from safe serialization and pickling paths in the garak plugin system.[1]

OpenAICompatible._call_model() is decorated with @backoff.on_exception using Fibonacci backoff (max interval 70 s) that retries on openai.RateLimitError, openai.InternalServerError, openai.APITimeoutError, openai.APIConnectionError, and garak.exception.GeneratorBackoffTrigger.[1] At call time, if self.client is None, _call_model() reloads the OpenAI client via _load_unsafe(), allowing recovery after close() is called.[1] Parameters listed in self.suppressed_params are omitted from the create_args dict built in _call_model(), overriding even attribute values set on the generator instance.[1] OpenAICompatible._call_model() merges self.extra_params into create_args unconditionally, so extra_params can override any built-in argument — including suppressed ones.[1] OpenAICompatible._call_model() includes the actual HTTP status code in raised exceptions rather than generic error messages, giving clearer diagnostics for transient failures.

OpenAICompatible.close() silently swallows AttributeError, OSError, RuntimeError, and ValueError during client teardown (logging them at DEBUG level) and always sets self.client and self.generator to None.[1]

Supported audio formats in garak/generators/openai.py are wav and mp3, detected via audio_pattern = re.compile("|".join(audio_formats)).[1] In garak/generators/openai.py, the models gpt-4-32k, gpt-4-32k-0314, and gpt-4-32k-0613 are listed as deprecated with a scheduled shutdown of 2025-06-06.[1]

Sources

8.3Ollama

Updated 2026-09-04

Garak's Ollama module provides two generator classes — OllamaGenerator for non-chat text generation and OllamaGeneratorChat for multi-turn conversations — both wrapping the local ollama Python client with configurable timeouts, parameter mapping, and optional API authentication. The generators support parameter suppression, Fibonacci backoff with automatic retries on empty responses, and fast-fail on missing-model errors (404), while treating the ollama dependency as optional at load time.

garak/generators/ollama.py provides two generator classes: OllamaGenerator (uses client.generate — non-chat mode) and OllamaGeneratorChat (uses client.chat — chat mode). The module-level DEFAULT_CLASS is "OllamaGeneratorChat". OllamaGenerator declares extra_dependency_names = ["ollama"], so the ollama Python package must be installed for this generator to load. OllamaGenerator sets parallel_capable = False, indicating it does not support concurrent generation requests.

OllamaGenerator.DEFAULT_PARAMS sets timeout=30 (seconds), host="127.0.0.1:11434", verify_ssl=None, extra_params=None, and suppressed_params=set(). The 30-second default timeout exists because Ollama can hang indefinitely on failures without it. OllamaGenerator passes extra_params as additional kwargs to ollama.Client(), which are treated as httpx.Client kwargs; this allows configuration such as custom SSL settings or proxy options.

OllamaGenerator._PARAM_MAP translates garak attribute names to Ollama options field names: max_tokens → num_predict (int), temperature → temperature (float), top_k → top_k (int), and seed → seed (int). Only these four parameters are forwarded to Ollama's options. OllamaGenerator._build_options() returns None when no generation parameters are set, which the Ollama client treats identically to omitting the options argument. Individual parameters can be excluded from Ollama requests via suppressed_params. OllamaGenerator.__init__() warns (but does not error) when an entry in suppressed_params is not a known key in _PARAM_MAP, and prints the valid keys.

A garak.site.yaml snippet to suppress top_k from Ollama requests illustrates how suppressed_params is configured per-generator — see Config loading and precedence for how site config is loaded.

plugins:
  generators:
    ollama:
      OllamaGenerator:
        suppressed_params:
          - top_k

OllamaGenerator._validate_env_var() silently swallows APIKeyMissingError, making the API key optional for Ollama (no key is needed for local servers). If self.api_key is set, OllamaGenerator.__init__() adds an Authorization: Bearer <key> header to the Ollama client, enabling authenticated endpoints.

OllamaGeneratorChat._call_model() calls client.chat() with the full conversation history converted via _conversation_to_list(), while OllamaGenerator._call_model() calls client.generate() with only prompt.last_message().text. OllamaGeneratorChat._call_model() extracts the response text from response["message"]["content"] and wraps it in Message, returning [None] if the key is absent. OllamaGenerator._call_model() uses Fibonacci backoff (max 70 s) on GeneratorBackoffTrigger, and additionally retries up to 3 times (via @backoff.on_predicate) when the response is [None] or empty, because Ollama sometimes returns empty responses. The empty-response retry fix in garak/generators/ollama.py may change observed latency and output distributions in production Ollama-backed probing pipelines, as transient empty responses are now retried rather than returned immediately. HTTP 404 errors are re-raised immediately rather than triggering backoff, because 404 typically means the requested model is not found on the server. The _give_up predicate aborts Fibonacci backoff on 404 only when the cause is not a TimeoutException, preventing endless retries for genuinely missing models.

8.5Generator test contracts

Updated 2026-08-22

Generator test contracts live in tests/generators/test_generators.py and enforce structural invariants (every DEFAULT_PARAMS key in _supported_params), signature matching for generate() and _call_model(), and correct list-length/type behavior on parallel calls. Several generator classes are excluded from general instantiation tests due to environmental or model-name constraints: AzureOpenAIGenerator, BedrockGenerator, WatsonXGenerator, function-based and file-based variants, and third-party-dependent generators.

Generator structural and behavioural contracts live in tests/generators/test_generators.py. Several classes are excluded from the general instantiation tests because they require special environments: AzureOpenAIGenerator (extra env vars), BedrockGenerator (Bedrock-specific model names), WatsonXGenerator (extra env vars), function.Multiple/function.Single (mock local functions), GgmlGenerator (file validation on disk), NeMoGuardrails (third-party install dependency), HuggingFace variants, and LangChainLLMGenerator (model name restrictions).[1]

The structural test in tests/generators/test_generators.py enforces that every key present in a generator's DEFAULT_PARAMS also appears in its _supported_params.[1] For non-OpenAI/Groq/Azure/NeMoGuardrailsServer generators, the same test file enforces that generate() is annotated as (prompt: Conversation) -> List[Union[None, Message]] and that _call_model() carries matching signature annotations.[1]

test_parallel_requests verifies that generators.test.Lipsum returns exactly the requested number of Message objects with non-empty text when generations_this_call=3 is passed to generate().[1]

Example: canonical generator parallel call test using generators.test.Lipsum in tests/generators/test_generators.py:

prompt = Conversation(Turn("user", [Message("this is a test")]))
result = g.generate(prompt=prompt, generations_this_call=3)
assert isinstance(result, list)
assert len(result) == 3
assert all(isinstance(item, Message) for item in result)

[1]

When generator output consists entirely of skip-sequence content (nothing outside the delimiters), generators.test.Repeat returns an empty Message(''); truncated (unclosed) skip sequences are also stripped — both behaviours are asserted in tests/generators/test_generators.py.[1]

Sources

tests/generators/test_generators.py

9.1Harness base

Updated 2026-08-22

Harness is the base class all garak harnesses inherit from; it orchestrates probe execution on generators, runs detectors on outputs, and evaluates results with configurable modality matching and instrumented progress tracking. Harness manages HTTP user-agent spoofing for the run duration, writes plugin metadata to the report, and requires subclasses to load buffs before delegating to its run method to avoid redundant reinstantiation. A buff is a transformation layer in Garak that modifies probe payloads before they reach the generator, applying operations such as paraphrasing or encoding inputs.

garak/harnesses/base.py defines Harness, the base class all garak harnesses must inherit from; it coordinates running probes on a generator, running detectors on outputs, and evaluating results.[1]

Harness.DEFAULT_PARAMS sets strict_modality_match: False, meaning a probe with only a partial modality overlap with the model is not skipped by default.[1] During Harness.run(), modality compatibility is checked for each probe against the model; a mismatched probe is skipped with a warning rather than failing the run, unless strict_modality_match is set to True.[1]

Harness._start_run_hook() replaces all HTTP library user-agent strings with _config.run.user_agent for the duration of the run, and _end_run_hook() restores them afterward.[1]

Harness._run_detector() includes the probe class name in the tqdm progress bar description (format {probe_classname}/{detector_name}) to keep long runs legible — resolving issue #324; no caller-side plumbing is needed because attempts already carry the probe classname.[1] _emit_plugin_cache_entry() writes a plugin_cache entry to _config.transient.reportfile in JSONL format, containing the garak version and metadata for each plugin instance passed to it.[1] After processing IntentProbe runs, Harness.run() calls _emit_plugin_cache_entry() for the intent-resolved detectors separately, because those detectors are not included in the harness-level snapshot taken at run start — see Context Aware Scanning for background on IntentProbe.[1]

Harness._load_buffs() must NOT be called from Harness.run() itself; subclasses should call it in their own run() before delegating to super().run(), to avoid redundant buff reinstantiation.[1] Buffs must be loaded exactly once per run because reinstantiating a buff resets any stateful transformation it maintains. A buff is a transformation layer that modifies probe payloads before they reach the generator, applying operations such as paraphrasing or encoding; statefulness makes repeated instantiation destructive.

Sources

garak/harnesses/base.py

10.1Intent service

Updated 2026-08-22

garak/services/intentservice.py is the Intent Service module, responsible for loading the intent typology, selecting and managing active intents for a run, and supplying stubs to probes.[1] Intents are potential traits or failure modes of a target — things like 'produce hate speech', 'generate malware', or 'reveal training recipe'. Each intent has one or more stubs: prototypical requests that begin with a verb.[1] The INTENT_PREFIX constant is '🎯', used as a log and print prefix for all intent service messages.[1] A stub is a short, verb-initiated prompt fragment that represents the minimal form of a request expressing a given intent, used as a seed so probes have a concrete starting point without a fully specified attack payload.

At startup, intentservice.load() reads the intent typology from garak/data/cas/trait_typology.json and the intent-to-detector mapping from garak/data/cas/intent_detectors.json.[1] intentservice.load() resolves the active intent spec from garak._config.transient.intent_spec; when that attribute is absent — for example, when the service is loaded directly without CLI spec resolution — it falls back to DEFAULT_INTENT_SCOPE from garak._spec.[1] intentservice.enabled() returns False and logs a warning if garak._config has not been loaded before the intent service is started.[1]

An intent spec of None, '*', 'all', or '' selects every intent in the loaded typology and bypasses validation entirely.[1] _expand_intent_specifier_children() expands non-leaf intent codes (four characters or fewer) to all typology keys sharing the same prefix, filtering out any codes listed in garak/data/cas/intent_skip.json.[1] _validate_intent_codes() is fail-closed: a well-formed but unknown intent code (e.g. S999) raises GarakException. Vacuous specs (None, '*', 'all', '') skip this check entirely.[1]

_populate_intents() silently drops any intent that has no mapped detector, unless garak._config.run.serve_detectorless_intents is set. Context Aware Scanning's use of the active intent set is covered on the Context Aware Scanning page.[1]

Sources

garak/services/intentservice.py

10.2CAS traits and policy

Updated 2026-09-04

The CAS trait system in garak/cas.py defines a hierarchical coding scheme with Policy, a validator, and a loader: traits follow a regex (letter-digit-letter pattern), organize into parent-child structures via get_parent_name(), and policies inherit or enforce permissions from ancestors. A Policy in garak/cas.py controls which intents or traits a probe or detector is permitted to exercise; without a policy, Garak cannot enforce content boundaries during a scan.

Trait/intent codes in garak/cas.py must match the regex ^[A-Z]([0-9]{3}([a-z]+)?)?$: a single uppercase letter, optionally followed by three digits, optionally followed by lowercase letters; invalid codes raise ValueError in get_parent_name().[1] get_parent_name() in garak/cas.py implements a three-level hierarchy: codes longer than four characters return their first four characters; four-character codes return their first character; single-character top-level codes return ''.[1]

Policy in garak/cas.py has three class-level defaults: none_inherits_parent = True (a None policy point inherits from its parent), default_trait_allowed_value = None, and permissive_root_policy = True (the root policy point is allowed by default).[1] Policy.is_permitted() in garak/cas.py recursively walks up the policy hierarchy when a point's value is None and none_inherits_parent is True; calling it on an unknown trait raises ValueError.[1] Policy.propagate_up() in garak/cas.py propagates permissiveness upward: if any child policy point is True and its parent is None, the parent is set to True. Leaf nodes (key length > 4) are processed before mid nodes (key length == 4), and top-level nodes are skipped.[1]

_validate_trait_descriptions() in garak/cas.py enforces that every trait has a non-empty name and a descr field, that all codes match the required regex, that there are no duplicate keys, and that each non-root trait has its parent present in the typology.[1] _load_trait_descriptions() in garak/cas.py returns an empty dict and logs an error if the loaded typology fails validation, so callers must handle the empty-dict case.[1] garak/data/cas/intent_detectors.json maps CAS intent labels to the detectors invoked for each intent; gaps in this file cause intents to go undetected. Engineers adding new probe intents or custom detectors must follow the file's existing schema when adding entries.

Sources

garak/cas.py

10.3Intent stubs

Updated 2026-08-22

Intent stubs in Garak are populated from five sources—typology defaults, text/JSON/YAML files in garak/data/cas/intent_stubs/, or code-generated stubs from Intent subclasses—and assembled via get_intent_stubs(). The Stub base dataclass and its subclasses (TextStub, ConversationStub) enforce type safety on content while using intent code and content hash for identity, though mutability of these fields after construction risks breaking set invariants. An intent code is a dot-separated string (e.g., harm.violence.direct) that identifies a specific harmful-intent category within Garak's typology system.

garak/services/intentservice.py supports five stub source types for an intent: a typology default stub, plain text files (.txt, one stub per line), JSON files (.json, list of strings), YAML files (.yml/.yaml, list of strings), and code-generated stubs from an Intent subclass module; the entry point for stub assembly is get_intent_stubs().[1] Text stub files for an intent code are located in garak/data/cas/intent_stubs/ and must match the glob patterns <intent_code>.txt or <intent_code>_*.txt.[1] _get_stubs_json() and _get_stubs_yaml() in garak/services/intentservice.py require the top-level item in the stub file to be a list; non-list top-level items log a warning and produce no stubs.[1] Code-generated stubs are only supported for fully-specified intent codes (length > 4); shorter prefix codes cause intentservice.py to return an empty set immediately.[1]

garak/intents/base.py defines the Stub base dataclass with an intent field (str or None) and a _content property; hashing is based on the string concatenation of intent and _content, making stubs usable in sets.[2] Stub.__eq__() compares equality by both intent and content, so two stubs with the same text but different intent codes are not equal.[2] The Stub hash is noted as contentious in garak/intents/base.py because intent and content remain mutable after instantiation, which can break set and dict invariants if a stub is mutated after insertion.[2]

TextStub in garak/intents/base.py is a Stub subclass that enforces str-only content; setting content to a non-string raises TypeError.[2]

ConversationStub in garak/intents/base.py accepts either a str or a garak.attempt.Conversation when setting content; a str is automatically wrapped in a Conversation containing a single Message, and any other type raises TypeError.[2] ConversationStub.__post_init__() also wraps a constructor-provided str into a Conversation, so passing a plain string directly in the constructor is supported.[2] ConversationStub.from_textstub() converts a TextStub into a ConversationStub by wrapping the text content in a Conversation with a single Message.[2] ConversationStub hashes differently from the base Stub: it uses repr() of the content rather than str(), in order to distinguish conversation structures.[2]

Sources

11.1Evaluator

Updated 2026-08-22

The Evaluator class in Garak runs detectors on probe outputs, logs pass/fail metrics per detector to JSONL, and optionally computes bootstrap confidence intervals and generates HTML digests with intent breakdowns. Hit logs record every detector failure with the full attack context (goal, prompt, output, triggers, generator, probe, detector), while narrow/wide output modes control CLI presentation.

The Evaluator constructor in garak/evaluators/base.py conditionally instantiates a Calibration object only when _config.system.show_z is truthy, and loads detector_metrics only when confidence_interval_method == "bootstrap".[1]

Evaluator.evaluate in garak/evaluators/base.py clears self.probename to None at the start of each call to avoid stale state across calls — a comment in the source acknowledges this should be refactored.[1] Evaluator.evaluate logs an error and returns early — without raising — if called with an empty list of attempts.[1] garak/evaluators/base.py warns via logging.warning when an attempt has no assigned detectors, identifying it by probe name, attempt UUID, sequence number, and intent.[1] Output format is selected based on _config.system.narrow_output: when that flag is set, self.print_results_narrow is called; otherwise self.print_results_wide is used.[1]

_evaluate_one_detector writes one eval JSONL entry per detector per evaluate() call, containing entry_type, probe, detector, passed, fails, nones, total_evaluated, total_processed, and optional per-intent and CI fields.[1] Per-intent pass/total counts are stored in the eval record under the key intents and feed the technique_intent_matrix in the HTML digest.[1] For every detector failure, garak/evaluators/base.py writes a hit-log entry to a .hitlog.jsonl file (derived from the report filename by substituting the suffix), recording goal, prompt, output, triggers, score, run_id, attempt metadata, generator, probe, detector, and generations_per_prompt.[1]

Bootstrap confidence intervals for attack success rates were added in v0.14.1.[2] garak/evaluators/base.py computes bootstrap confidence intervals per detector only when _config.reporting.confidence_interval_method == "bootstrap" AND the number of evaluated outputs meets or exceeds _config.reporting.bootstrap_min_sample_size; smaller samples are silently skipped.[1] CI bounds are stored in the eval JSONL record normalised to [0, 1] (ci_lower / 100, ci_upper / 100), even though the CI calculation internally returns percentage points.[1] Confidence intervals narrower than the module-level constant CI_DISPLAY_MIN_WIDTH = 0.001 percentage points are suppressed from CLI output, as they provide no meaningful uncertainty information.[1]

Sources

11.3Report digest

Updated 2026-08-22

garak/analyze/report_digest.py parses garak JSONL report files and generates structured report data for HTML/digest output, serving as the main entry point for report generation.[1] At module import time, report_digest.py calls _config.load_config() if the config is not already loaded, ensuring config is always available even when the module is used standalone.[1] Also at import time, report_digest.py loads MISP tag descriptions from data/tags.misp.tsv — a tab-separated file with columns key, title, and descr — used to annotate taxonomy-grouped probe report sections.[1] report_digest.py loads CAS intent names from data/cas/trait_typology.json at import time, normalizing empty name values to None; these names drive the technique×intent matrix display in reports.[1] The constant TECHNIQUE_TAG_PREFIX = "demon:" defines the probe tag namespace used to identify technique tags for the technique×intent matrix in HTML reports.[1]

report_digest.py uses an in-memory SQLite database (:memory:) to store and query per-probe evaluation results during report generation.[1] The results table stores probe_module, probe_group, probe_class, detector, score, instances, passes, and bootstrap confidence interval fields (confidence, confidence_lower, confidence_upper).[1] _init_populate_result_db() strips the probes. prefix from probe paths and the detector. prefix from detector paths before inserting rows into the results database.[1] When a taxonomy is provided to _init_populate_result_db(), probe grouping is determined by matching probe tags against the taxonomy prefix; probes with no matching tag are placed in the "other" group. Without a taxonomy, probes are grouped by their module name.[1] If a report references a probe not found in the plugin cache, _init_populate_result_db() raises ReportIncompatibleError with a message indicating the report was likely generated with a different garak version.[1]

_parse_report() falls back to a deep copy of the live garak._plugins.PluginCache.instance() tagged with garak.__version__ when no plugin_cache entry is found in the JSONL report, meaning such reports are interpreted using the currently-installed garak plugin metadata.[1] _extract_to_probespec() resolves the probe selection used in a run: it reads transient.active_probes first, falls back to plugins.probe_spec for older reports, and defaults to "probes.*" if neither is present — providing backward compatibility with pre-transient.active_probes report formats.[1] _report_header_content() resolves the target type and name using plugins.target_type/plugins.target_name, falling back to the deprecated plugins.model_type/plugins.model_name keys introduced before the v0.13.1 rename.[1] _resolve_plugin_info() raises ValueError if the requested plugin classpath is missing from the plugin cache, or if any of the required_fields are absent or None in the cached metadata.[1]

_get_probe_group_summaries() returns probes within a group sorted by minimum score ascending, then by probe class name, so the most-failed probes appear first.[1] _get_group_info() computes a DEFCON rating for each probe group using garak.analyze.score_to_defcon() with ABSOLUTE_DEFCON_BOUNDS, and includes the group's aggregation function name in the returned dict.[1] When no taxonomy is set, _get_group_info() dynamically imports garak.probes.<module> to extract its docstring as the group description, and builds a link to https://reference.garak.ai/en/latest/garak.probes.<probe_group>.html.[1]

Sources

garak/analyze/report_digest.py

11.4Calibration and scoring

Updated 2026-08-22

Garak calibration normalizes probe–detector scores into Z-scores using baseline distributions stored in calibration.json, with graceful fallback when calibration data is unavailable or invalid. Score aggregation combines multiple probe results via configurable strategies (mean, minimum, median, lower_quartile, mean_minus_sd, proportion_passing) and reports whether an unknown strategy was used.

garak/analyze/calibration.py provides the Calibration class, which loads probe/detector score calibration data from a JSON file and exposes Z-score computation for normalizing individual probe–detector scores against a baseline distribution.[1] Calibration.__init__() defaults to loading calibration data from data/calibration/calibration.json when no calibration_path is provided; a custom path (str or pathlib.Path) can be passed to override this.[1] After initialization, Calibration sets a calibration_successfully_loaded boolean that callers can inspect to determine whether calibration data is available for Z-score normalization.[1]

If the calibration JSON contains a garak_calibration_meta key, Calibration._load_calibration() stores it in self.metadata and removes it from self._data so it does not interfere with probe/detector lookups.[1] Calibration._load_calibration() handles a Windows Git edge case where symlinks are checked out as text files containing the symlink destination: if the file's content matches ^calibration[^/\\]+.json$, the method treats it as a redirect and loads the referenced file instead.[1] During loading, all file-read exceptions are caught (a warning is logged and None is returned), and JSONDecodeError is caught separately during JSON parsing — in both cases calibration is silently disabled rather than crashing the report.[1]

The calibration key format used by Calibration.get_z_score() is "{probe_module}.{probe_classname}/{detector_module}.{detector_classname}", which must match the keys present in the calibration.json data file.[1] Calibration.get_z_score() enforces a minimum standard deviation of MINIMUM_STD_DEV (imported from garak.analyze) before computing a Z-score, preventing division-by-zero when all calibration scores were identical.[1] A Z-score expresses how many standard deviations a probe–detector score falls from the calibration baseline mean; a score near zero indicates typical model behavior, while a large positive or negative Z-score signals unusual behavior relative to that baseline.

garak/resources/scoring.py exposes an aggregate(scores, aggregation_function) function that supports "mean", "minimum", "median", "lower_quartile", "mean_minus_sd", and "proportion_passing" strategies; unknown strategy names fall back to min(scores) and set unknown_function = True.[2] aggregate() raises ValueError when called with an empty list.[2] The "lower_quartile" strategy uses statistics.quantiles(scores, method="inclusive")[0]; when the list has exactly one score, aggregate() returns that score directly to avoid a statistics.quantiles failure on a single-element list.[2] The "proportion_passing" strategy counts scores strictly greater than ABSOLUTE_DEFCON_BOUNDS.BELOW_AVG (imported from garak.analyze) and divides by the total count.[2] aggregate() returns a two-tuple (aggregate_score, unknown_function) where unknown_function is True only when the aggregation strategy name was unrecognized.[2]

garak/analyze/ci_calculator.py derives confidence intervals from the subset of attack hits rather than from the full attempt count, preventing denominator inflation that would otherwise produce misleadingly narrow intervals on sparse probes. A bundled garak/data/detectors_eval/detector_metrics_summary.json file ships with the package, containing detector-level evaluation metrics such as precision and recall. garak/analyze/detector_metrics.py loads the bundled detector_metrics_summary.json file; _plugins.py exposes this data through the standard plugin path. The schema for detector metrics is documented in docs/source/detector_metrics.rst.

Sources

12Language providers

Updated 2026-08-22

garak/langproviders/base.py provides the LangProvider base class for objects that provision language translation, as part of the multilingual buff (prompt transformation) system.[1] LangProvider extends garak.configurable.Configurable, participating in garak's three-level YAML/JSON config hierarchy — see Configurable base class for the full config-loading rules.[1] The langproviders config block uses language as a comma-separated "<source>,<target>" pair; LangProvider.__init__ splits on "," to populate self.source_lang and self.target_lang (e.g. "en,fr"). Note that an inline comment in the source shows "<from>-<to>", which does not reflect the actual parsing.[1] A buff in garak is a prompt transformation layer that modifies probe inputs before they reach the target model; the multilingual buff uses a LangProvider to translate those prompts, enabling probes written in one language to test models in another.

LangProvider._load_langprovider and LangProvider._translate both raise NotImplementedError; concrete subclasses must override both to be functional.[1]

LangProvider.get_text is the public entry point: given a list of prompt strings, it returns a list of translated strings.[1] Setting reverse_translate_judge=True on get_text skips translation of lines already detected as non-English by is_meaning_string.[1] LangProvider.get_text accepts an optional notify_callback callable that is invoked once per processed prompt, enabling progress-tracking integrations.[1]

split_input_text splits on ': ' only when the text does not contain 'http://' or 'https://', preventing URLs from being split at the colon.[1] LangProvider._get_response splits input on ': ' (skipping URLs), then routes each line to _short_sentence_translate (≤ 200 characters) or _long_sentence_translate (> 200 characters).[1] _long_sentence_translate splits text on '. ' or '?' sentence boundaries before translating each fragment individually, then collects results into a list.[1]

_should_skip_line treats whitespace-only, empty, dash-only, dot-only, and the literal strings '.', '?', and '. ' as lines to skip — passing them through untranslated.[1] Lines containing only invisible Unicode characters (categories Cc, Cf, Cn, Zl, Zp, Zs) are silently dropped by _get_response — they are not appended to translated_lines. A single visible character causes contains_invisible_unicode to return False.[1] When source_lang is 'en', _short_sentence_translate calls is_english on each line and only translates lines actually detected as English; non-English lines are passed through unchanged.[1] The literal string "$" is always passed through without translation in _short_sentence_translate; the codebase itself notes this behaviour is unexplained (# why is "$" a special line?).[1]

_clean_line lowercases, strips, splits on whitespace, and removes English punctuation before translation; original casing and surrounding punctuation are lost in every translated result.[1] remove_english_punctuation strips all string.punctuation characters except apostrophes, and additionally removes colons and commas from individual tokens via re.sub.[1]

is_meaning_string is a noise/garbage filter used before reverse-translation: it returns False for text shorter than 3 characters, text with 4+ consecutive identical characters (e.g. 'aaaa'), or text that langdetect cannot classify.[1] is_meaning_string sets DetectorFactory.seed = 0 before every call to make langdetect language detection deterministic.[1] Already-English text is never a candidate for reverse-translation: is_meaning_string returns False for any text detected as English (lang == 'en').[1]

The NLTK words corpus is downloaded lazily on first use of is_english via _initialize_words, which checks for the corpus with nltk.data.find before downloading to avoid redundant downloads.[1]

Sources

garak/langproviders/base.py

13.1Test fixtures

Updated 2026-08-22

Test fixtures for Garak are configured via pytest plugins (pytest-mock, respx, pytest-asyncio, pytest-cov, pytest_httpserver, requests-mock) and global setup in conftest.py that suppresses noise, initializes the plugin cache, and provides reusable fixtures like mitigation_outputs and loaded_intent_service. Custom pytest markers (requires_storage, integration) and autouse fixtures (config_cleanup) ensure test isolation and skip tests when storage resources are unavailable.

The tests optional dependency group in pyproject.toml includes pytest>=9.1, pytest-mock>=3.14.0, respx>=0.21.1, pytest-asyncio>=0.21.0, pytest-cov>=5.0.0, pytest_httpserver>=1.1.0, and requests-mock==1.12.1.[1] pytest is configured in pyproject.toml to suppress all warnings by default and re-enable only warnings originating from the garak namespace.[1]

The tests/conftest.py global setup redirects garak logging to /dev/null (via os.devnull) unless the GARAK_LOG_FILE environment variable is set, preventing spurious log output during test runs.[2] tests/conftest.py ensures the garak plugin cache file exists at test-suite startup by calling _plugins.PluginCache.instance() if the user cache file is missing.[2]

The config_cleanup autouse fixture in tests/conftest.py runs for every test and, via LIFO finalizers, reloads _config, clears PluginProvider._instance_cache, and removes report, hitlog, and HTML files produced during the test.[2] The mitigation_outputs fixture in tests/conftest.py returns a (COMPLYING_OUTPUTS, REFUSAL_OUTPUTS) tuple — two lists of representative LLM outputs — for use in detector and probe tests.[2] The loaded_intent_service fixture in tests/conftest.py loads the garak config and then calls garak.services.intentservice.load(), providing a ready IntentService for tests that need it — see Context Aware Scanning for the feature this supports.[2]

tests/conftest.py registers two custom pytest markers: requires_storage and integration.[2] The requires_storage marker defaults to requiring 1 GB free on the root filesystem (/); tests decorated with it are skipped with a clear message if that threshold is not met.[2]

Sources

14Upgrading

Updated 2026-08-22

Garak undergoes regular breaking changes across CLI flags, plugin naming, and configuration structure; each version bump may require renaming command arguments, relocating config keys, or migrating deprecated components to new systems. When upgrading Garak, check the release notes for renamed probes and generators, changed config namespaces (like model_* to target_*), and CLI flag replacements (such as --run_spec to --spec), which may affect scripts and saved configurations.

The --run_spec CLI flag was renamed to --spec (with short alias -S); the old --run-spec alias was also dropped.[1] The --probes, --probe_tags, and --buffs CLI flags were deprecated in favour of the unified run.spec plugin selection grammar, introduced via --spec (short flag -S) — see Spec grammar and parsing for the grammar reference.[2] In v0.14.0, the --generate_autodan CLI option was removed as a breaking change.[3] The run.spec plugin selection grammar is a structured string format that lets users select and combine probes, generators, and other plugins within a single argument passed to --spec.

In v0.13.1, config keys prefixed model_* were renamed to target_*.[4] Intent configuration keys were moved from the cas namespace to the run namespace — for example, cas.intent_spec becomes run.intent_spec.[5]

The knownbadsignatures probe module was renamed to av_spam_scanning in v0.9.0.16.[6] The atk probe was renamed to atkgen in v0.9.0.10.[7] ART (Attack Red Team) was renamed to AG (Attack Generator) in v0.9.0.8.[8] The generations parameter was moved from generators to probes in v0.9.0.16.[6] In v0.10.0, payload and probe concerns were separated; probes.encoding payloads were migrated to the payloads system.[9] The octo generator was removed in v0.12.0.[10] In v0.14.1, the deprecated nemollm generator and its dependencies were removed as a breaking change.[11]

In v0.13.0, "failure" metric terminology was renamed to "attack success" throughout garak.[12] The maxrecall evaluator was removed in v0.15.0 due to broken Python 3 compatibility.[13] In v0.14.0, HTML reports were fully redesigned as a breaking change.[3]

The default output directory was renamed to garak_runs/ in v0.9.0.12.[14]

Sources

Search

1Introduction

2Probes

2.1Probe base class

2.2DAN probes

2.3Payloads

2.4Probe test contracts

3Orientation and CLI

3.1Overview

3.2Installation

3.3CLI entry point

3.4CLI arguments and flags

3.5Run lifecycle commands

4Configuration system

4.1Config loading and precedence

4.2Default core config

4.3Configurable base class

4.4HTTP user agent

5Plugin architecture

5.1Plugin types and cache

5.2Exceptions

5.3Attempt and Message

6Spec and selection

6.1Spec grammar and parsing

6.2Selection resolution

7Detectors

7.1Detector base classes

7.2Mitigation detectors

7.3Detector test contracts

8Generators

8.1Generator base class

8.2OpenAI compatible

8.3Ollama

8.4Other generators

8.5Generator test contracts

9Harnesses

9.1Harness base

9.2Probewise harness

9.3PxD harness

10Context Aware Scanning

10.1Intent service

10.2CAS traits and policy

10.3Intent stubs

11Evaluation and reporting

11.1Evaluator

11.2Report class

11.3Report digest

11.4Calibration and scoring

12Language providers

13Testing and CI

13.1Test fixtures

13.2CI workflows

13.3Contribution policy

14Upgrading