Garak's probe test suite verifies structural integrity: every probe must declare detectors and intent attributes, support flexible prompt types via _mint_attempt(), and maintain docstring and metadata (tags, dependencies) standards. Attempts minted from probes carry forward the probe's intent throughout their lifecycle, from construction through serialization, unless overridden by _payload_intent. IntentProbe is a garak.probes.base.Probe subclass that determines which detectors to apply at runtime rather than declaring them statically, enabling flexible detector routing.
The probe test suite in tests/probes/test_probes.py sets a fake NVOpenAIChat.ENV_VAR environment variable via an autouse fixture and restores the original value after each test, preventing leaked credentials.[1]
Every probe in tests/probes/test_probes.py is verified to declare either a primary_detector (a string) or a non-empty extended_detectors list, unless it is an IntentProbe descendant — which has flexible detector routing and is exempt from this check.[1] Every concrete probe (non-IntentProbe, non-base) must declare a non-empty intent attribute that is a string matching a valid intent typology entry; IntentProbe descendants and base probes must set intent to None.[1] Probes that have external module dependencies (non-empty extra_dependency_names) must set active = False so they are not run by default; this is enforced in tests/probes/test_probes.py.[1] Probe docstrings must have at least two paragraphs separated by a blank line — a summary paragraph followed by a deeper description — enforced by tests/probes/test_probes.py.[1] All probe tags must be non-empty (unless active == False), be MISP-format strings where each colon-delimited part matches [A-Za-z0-9_\-\&]+, and appear in the project's data/tags.misp.tsv file — except tags whose namespace is payload.[1]
test_probe_prune_alignment in tests/probes/test_probes.py verifies that after prompt pruning, probes.glitch.Glitch has exactly _config.run.soft_probe_prompt_cap prompts and triggers, and that each trigger appears inside its corresponding prompt.[1]
Probe._mint_attempt() is tested to accept a plain string, a Message, a single-turn Conversation, or a multi-turn Conversation (with system turn) as a prompt, and always returns an Attempt whose turns are all Turn instances and whose last message text equals the input text.[1]
Example: canonical _mint_attempt call with all supported prompt types in tests/probes/test_probes.py:
probe = garak.probes.base.Probe()
attempt = probe._mint_attempt(prompt) # prompt may be str, Message, or Conversation
assert isinstance(attempt, Attempt)
for turn in attempt.prompt.turns:
assert isinstance(turn, Turn)
assert attempt.prompt.last_message().text == "test example"
garak.probes.base.Probe._mint_attempt() sets attempt.intent to None when the base Probe class has no intent set, as verified in tests/probes/test_probes.py.[1] Probe._mint_attempt() propagates probe.intent to attempt.intent for every minted attempt, even across multiple calls; tests/probes/test_probes.py verifies this by setting probe.intent = 'S005' and confirming all five resulting attempts carry that intent.[1] The serialised dict from attempt.as_dict() must include an 'intent' key whose value matches probe.intent, as verified in tests/probes/test_probes.py.[1]
Sources