Garak's detector test suite enforces contracts on all detectors: they must inherit from Detector and Configurable, declare required parameters in DEFAULT_PARAMS and _supported_params, provide valid language specs and documentation URIs, and return result counts matching attempt.outputs length. Detectors are tested for structural integrity (signature and metadata compliance), graceful handling of missing API keys via pytest.skip, valid MISP taxonomy tags, and correct output counts — with specific exemptions for template classes and certain always-on detectors.
The detector test suite in tests/detectors/test_detectors.py filters out detectors.packagehallucination.PackageHallucinationDetector from the parametrized test list because it is used as a template class, not a concrete detector.[1] test_detector_detect skips tests gracefully when APIKeyMissingError is raised at instantiation or during detect(), using pytest.skip.[1]
All detectors must be instances of both Detector (from garak.detectors.base) and Configurable; test_detector_structure asserts both with isinstance checks.[1] detect() is required to accept a parameter named attempt; test_detector_structure asserts this via signature inspection.[1] Every key in a detector's DEFAULT_PARAMS must also appear in _supported_params; test_detector_structure enforces this contract.[1]
Every detector's lang_spec attribute must be either "*" or a comma-separated list of valid BCP47 language codes, validated in test_detector_metadata using the langcodes library.[1] Every detector's doc_uri must be None or a non-empty string starting with "http" (case-insensitive); test_detector_metadata enforces this rule.[1] Detector tags must be valid MISP taxonomy entries — colon-delimited alphanumeric/dash/underscore parts present in data/tags.misp.tsv — unless the first segment is "payload", which is exempt from the MISP check.[1]
Non-FileDetector detectors must return exactly as many results as there are entries in attempt.outputs; the test enforces this with len(list(results)) == len(a.outputs).[1] To exercise detectors that rely on attempt metadata, test_detector_detect sets up an Attempt with notes including trigger, triggers, repeat_word, and format, and sets outputs to a mix of strings, an empty string, None, and a Message object.[1] Detectors listed in DOES_NOT_RELAY_NONE are exempt from the None-relay contract: detectors.agent_breaker.AgentBreakerResult, detectors.always.Fail, detectors.always.Pass, and detectors.always.Random.[1] FileDetector is a detector subclass whose results derive from file contents rather than attempt.outputs directly; because output count depends on file contents, FileDetector subclasses are exempt from the contract requiring len(list(results)) == len(a.outputs).
Sources