The Evaluator class in Garak runs detectors on probe outputs, logs pass/fail metrics per detector to JSONL, and optionally computes bootstrap confidence intervals and generates HTML digests with intent breakdowns. Hit logs record every detector failure with the full attack context (goal, prompt, output, triggers, generator, probe, detector), while narrow/wide output modes control CLI presentation.
The Evaluator constructor in garak/evaluators/base.py conditionally instantiates a Calibration object only when _config.system.show_z is truthy, and loads detector_metrics only when confidence_interval_method == "bootstrap".[1]
Evaluator.evaluate in garak/evaluators/base.py clears self.probename to None at the start of each call to avoid stale state across calls — a comment in the source acknowledges this should be refactored.[1] Evaluator.evaluate logs an error and returns early — without raising — if called with an empty list of attempts.[1] garak/evaluators/base.py warns via logging.warning when an attempt has no assigned detectors, identifying it by probe name, attempt UUID, sequence number, and intent.[1] Output format is selected based on _config.system.narrow_output: when that flag is set, self.print_results_narrow is called; otherwise self.print_results_wide is used.[1]
_evaluate_one_detector writes one eval JSONL entry per detector per evaluate() call, containing entry_type, probe, detector, passed, fails, nones, total_evaluated, total_processed, and optional per-intent and CI fields.[1] Per-intent pass/total counts are stored in the eval record under the key intents and feed the technique_intent_matrix in the HTML digest.[1] For every detector failure, garak/evaluators/base.py writes a hit-log entry to a .hitlog.jsonl file (derived from the report filename by substituting the suffix), recording goal, prompt, output, triggers, score, run_id, attempt metadata, generator, probe, detector, and generations_per_prompt.[1]
Bootstrap confidence intervals for attack success rates were added in v0.14.1.[2] garak/evaluators/base.py computes bootstrap confidence intervals per detector only when _config.reporting.confidence_interval_method == "bootstrap" AND the number of evaluated outputs meets or exceeds _config.reporting.bootstrap_min_sample_size; smaller samples are silently skipped.[1] CI bounds are stored in the eval JSONL record normalised to [0, 1] (ci_lower / 100, ci_upper / 100), even though the CI calculation internally returns percentage points.[1] Confidence intervals narrower than the module-level constant CI_DISPLAY_MIN_WIDTH = 0.001 percentage points are suppressed from CLI output, as they provide no meaningful uncertainty information.[1]
Sources