garak/langproviders/base.py provides the LangProvider base class for objects that provision language translation, as part of the multilingual buff (prompt transformation) system.[1] LangProvider extends garak.configurable.Configurable, participating in garak's three-level YAML/JSON config hierarchy — see Configurable base class for the full config-loading rules.[1] The langproviders config block uses language as a comma-separated "<source>,<target>" pair; LangProvider.__init__ splits on "," to populate self.source_lang and self.target_lang (e.g. "en,fr"). Note that an inline comment in the source shows "<from>-<to>", which does not reflect the actual parsing.[1] A buff in garak is a prompt transformation layer that modifies probe inputs before they reach the target model; the multilingual buff uses a LangProvider to translate those prompts, enabling probes written in one language to test models in another.
LangProvider._load_langprovider and LangProvider._translate both raise NotImplementedError; concrete subclasses must override both to be functional.[1]
LangProvider.get_text is the public entry point: given a list of prompt strings, it returns a list of translated strings.[1] Setting reverse_translate_judge=True on get_text skips translation of lines already detected as non-English by is_meaning_string.[1] LangProvider.get_text accepts an optional notify_callback callable that is invoked once per processed prompt, enabling progress-tracking integrations.[1]
split_input_text splits on ': ' only when the text does not contain 'http://' or 'https://', preventing URLs from being split at the colon.[1] LangProvider._get_response splits input on ': ' (skipping URLs), then routes each line to _short_sentence_translate (≤ 200 characters) or _long_sentence_translate (> 200 characters).[1] _long_sentence_translate splits text on '. ' or '?' sentence boundaries before translating each fragment individually, then collects results into a list.[1]
_should_skip_line treats whitespace-only, empty, dash-only, dot-only, and the literal strings '.', '?', and '. ' as lines to skip — passing them through untranslated.[1] Lines containing only invisible Unicode characters (categories Cc, Cf, Cn, Zl, Zp, Zs) are silently dropped by _get_response — they are not appended to translated_lines. A single visible character causes contains_invisible_unicode to return False.[1] When source_lang is 'en', _short_sentence_translate calls is_english on each line and only translates lines actually detected as English; non-English lines are passed through unchanged.[1] The literal string "$" is always passed through without translation in _short_sentence_translate; the codebase itself notes this behaviour is unexplained (# why is "$" a special line?).[1]
_clean_line lowercases, strips, splits on whitespace, and removes English punctuation before translation; original casing and surrounding punctuation are lost in every translated result.[1] remove_english_punctuation strips all string.punctuation characters except apostrophes, and additionally removes colons and commas from individual tokens via re.sub.[1]
is_meaning_string is a noise/garbage filter used before reverse-translation: it returns False for text shorter than 3 characters, text with 4+ consecutive identical characters (e.g. 'aaaa'), or text that langdetect cannot classify.[1] is_meaning_string sets DetectorFactory.seed = 0 before every call to make langdetect language detection deterministic.[1] Already-English text is never a candidate for reverse-translation: is_meaning_string returns False for any text detected as English (lang == 'en').[1]
The NLTK words corpus is downloaded lazily on first use of is_english via _initialize_words, which checks for the corpus with nltk.data.find before downloading to avoid redundant downloads.[1]
Sources