Harness is the base class all garak harnesses inherit from; it orchestrates probe execution on generators, runs detectors on outputs, and evaluates results with configurable modality matching and instrumented progress tracking. Harness manages HTTP user-agent spoofing for the run duration, writes plugin metadata to the report, and requires subclasses to load buffs before delegating to its run method to avoid redundant reinstantiation. A buff is a transformation layer in Garak that modifies probe payloads before they reach the generator, applying operations such as paraphrasing or encoding inputs.
garak/harnesses/base.py defines Harness, the base class all garak harnesses must inherit from; it coordinates running probes on a generator, running detectors on outputs, and evaluating results.[1]
Harness.DEFAULT_PARAMS sets strict_modality_match: False, meaning a probe with only a partial modality overlap with the model is not skipped by default.[1] During Harness.run(), modality compatibility is checked for each probe against the model; a mismatched probe is skipped with a warning rather than failing the run, unless strict_modality_match is set to True.[1]
Harness._start_run_hook() replaces all HTTP library user-agent strings with _config.run.user_agent for the duration of the run, and _end_run_hook() restores them afterward.[1]
Harness._run_detector() includes the probe class name in the tqdm progress bar description (format {probe_classname}/{detector_name}) to keep long runs legible — resolving issue #324; no caller-side plumbing is needed because attempts already carry the probe classname.[1] _emit_plugin_cache_entry() writes a plugin_cache entry to _config.transient.reportfile in JSONL format, containing the garak version and metadata for each plugin instance passed to it.[1] After processing IntentProbe runs, Harness.run() calls _emit_plugin_cache_entry() for the intent-resolved detectors separately, because those detectors are not included in the harness-level snapshot taken at run start — see Context Aware Scanning for background on IntentProbe.[1]
Harness._load_buffs() must NOT be called from Harness.run() itself; subclasses should call it in their own run() before delegating to super().run(), to avoid redundant buff reinstantiation.[1] Buffs must be loaded exactly once per run because reinstantiating a buff resets any stateful transformation it maintains. A buff is a transformation layer that modifies probe payloads before they reach the generator, applying operations such as paraphrasing or encoding; statefulness makes repeated instantiation destructive.
Sources