Garak calibration normalizes probe–detector scores into Z-scores using baseline distributions stored in calibration.json, with graceful fallback when calibration data is unavailable or invalid. Score aggregation combines multiple probe results via configurable strategies (mean, minimum, median, lower_quartile, mean_minus_sd, proportion_passing) and reports whether an unknown strategy was used.
garak/analyze/calibration.py provides the Calibration class, which loads probe/detector score calibration data from a JSON file and exposes Z-score computation for normalizing individual probe–detector scores against a baseline distribution.[1] Calibration.__init__() defaults to loading calibration data from data/calibration/calibration.json when no calibration_path is provided; a custom path (str or pathlib.Path) can be passed to override this.[1] After initialization, Calibration sets a calibration_successfully_loaded boolean that callers can inspect to determine whether calibration data is available for Z-score normalization.[1]
If the calibration JSON contains a garak_calibration_meta key, Calibration._load_calibration() stores it in self.metadata and removes it from self._data so it does not interfere with probe/detector lookups.[1] Calibration._load_calibration() handles a Windows Git edge case where symlinks are checked out as text files containing the symlink destination: if the file's content matches ^calibration[^/\\]+.json$, the method treats it as a redirect and loads the referenced file instead.[1] During loading, all file-read exceptions are caught (a warning is logged and None is returned), and JSONDecodeError is caught separately during JSON parsing — in both cases calibration is silently disabled rather than crashing the report.[1]
The calibration key format used by Calibration.get_z_score() is "{probe_module}.{probe_classname}/{detector_module}.{detector_classname}", which must match the keys present in the calibration.json data file.[1] Calibration.get_z_score() enforces a minimum standard deviation of MINIMUM_STD_DEV (imported from garak.analyze) before computing a Z-score, preventing division-by-zero when all calibration scores were identical.[1] A Z-score expresses how many standard deviations a probe–detector score falls from the calibration baseline mean; a score near zero indicates typical model behavior, while a large positive or negative Z-score signals unusual behavior relative to that baseline.
garak/resources/scoring.py exposes an aggregate(scores, aggregation_function) function that supports "mean", "minimum", "median", "lower_quartile", "mean_minus_sd", and "proportion_passing" strategies; unknown strategy names fall back to min(scores) and set unknown_function = True.[2] aggregate() raises ValueError when called with an empty list.[2] The "lower_quartile" strategy uses statistics.quantiles(scores, method="inclusive")[0]; when the list has exactly one score, aggregate() returns that score directly to avoid a statistics.quantiles failure on a single-element list.[2] The "proportion_passing" strategy counts scores strictly greater than ABSOLUTE_DEFCON_BOUNDS.BELOW_AVG (imported from garak.analyze) and divides by the total count.[2] aggregate() returns a two-tuple (aggregate_score, unknown_function) where unknown_function is True only when the aggregation strategy name was unrecognized.[2]
garak/analyze/ci_calculator.py derives confidence intervals from the subset of attack hits rather than from the full attempt count, preventing denominator inflation that would otherwise produce misleadingly narrow intervals on sparse probes. A bundled garak/data/detectors_eval/detector_metrics_summary.json file ships with the package, containing detector-level evaluation metrics such as precision and recall. garak/analyze/detector_metrics.py loads the bundled detector_metrics_summary.json file; _plugins.py exposes this data through the standard plugin path. The schema for detector metrics is documented in docs/source/detector_metrics.rst.
Sources