Garak recognizes five plugin types (probes, detectors, generators, harnesses, buffs) registered in PLUGIN_TYPES and backed by a PluginCache that maintains a bundled and user-writable JSON cache, validating and rebuilding from source as needed. The cache discovery, validation, and plugin lookup system uses string identifiers like category.module.ClassName, lazy-loads detector metrics, and serializes plugins via a custom PluginEncoder that normalizes sets and paths. A buff is a Garak plugin type that transforms or augments probe prompts before they reach a generator — enabling behaviors such as paraphrasing or encoding — without modifying the probe itself.
The five recognized plugin types in garak are probes, detectors, generators, harnesses, and buffs, defined as PLUGIN_TYPES in garak/_plugins.py.[1] The corresponding base class names — Probe, Detector, Generator, Harness, and Buff — are defined as PLUGIN_CLASSES in the same file.[1] PluginCache._enumerate_plugin_klasses() raises ValueError if called with a category not present in PLUGIN_TYPES.[1] When scanning a plugin type directory, PluginCache._enumerate_plugin_klasses() considers only .py files and skips any whose names start with __ (dunder) or _ (private).[1] PluginCache._extract_modules_klasses() returns only concrete, non-abstract classes whose __module__ starts with the base module's name, filtering out all abstract classes.[1]
PluginCache maintains two cache locations: a bundled (system) copy at <package_dir>/resources/plugin_cache.json and a user-writable copy at <cache_dir>/resources/plugin_cache.json, both resolved from _config.transient paths.[1] On startup, PluginCache._load_plugin_cache() copies the bundled plugin_cache.json to the user cache location whenever the user copy does not yet exist or the bundled file is newer.[1] After loading the user cache, PluginCache._load_plugin_cache() validates it; if validation fails, it rebuilds the cache from source and reloads the newly written file before returning.[1] PluginCache._build_plugin_cache() is protected by a threading.Lock (_mutex) to prevent concurrent rebuilds, and writes only to the user cache file — not to the bundled system cache.[1]
PluginCache.instance() returns the shared _plugin_cache_dict singleton, constructing a PluginCache instance to trigger loading if one does not yet exist.[1] PluginCache.plugin_info() accepts either a plugin class object or a dot-separated string of the form category.module.ClassName (e.g., probes.mymodule.MyProbe); a string with any other number of parts raises ValueError.[1] When a string plugin name is not found in the cache, PluginCache.plugin_info() dynamically imports the module and class to compute the info rather than failing immediately.[1] Detector performance metrics are lazily loaded from <package_dir>/data/detectors-eval/detector_metrics_summary.json into PluginCache._detector_metrics_cache; if the file is absent, an empty dict is used instead.[1]
PluginEncoder in garak/_plugins.py serializes set values as sorted lists and Path objects as strings with the package directory prefix stripped; objects that are otherwise unserializable are silently dropped (the encoder returns None).[1]
Sources