Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Migrate string output/input to Turn objects #1089

Open
wants to merge 92 commits into
base: main
Choose a base branch
from
Open
Changes from 1 commit
Commits
Show all changes
92 commits
Select commit Hold shift + click to select a range
fa193b0
start moving to abstracted prompt objects
leondz Jan 20, 2025
f6587fe
renaming Prompt -> Turn (incomplete)
leondz Jan 27, 2025
e4edeb6
attempt turns are now instances of Turn
leondz Jan 27, 2025
9f86eca
file reading helper
leondz Jan 27, 2025
e0aa817
use Turn() in turns
leondz Jan 27, 2025
c4a0e00
specify dict-based serialisation
leondz Jan 27, 2025
31f5393
support string use insetting attempt values
leondz Jan 27, 2025
08fd05f
base detectors operate on turn text
leondz Jan 27, 2025
ae77d0f
stringdetector only operates on output.text
leondz Jan 27, 2025
d4ecf95
stringdetector only operates on output.text
leondz Jan 27, 2025
6f429cf
serialise turns in eval logging
leondz Jan 27, 2025
895eaa4
migrate base buff tests, buffs.lowercase
leondz Jan 27, 2025
84a9754
migrate encoding buff
leondz Jan 27, 2025
e0fda46
migrate encoding buff
leondz Jan 27, 2025
25b8172
migrate leakreplay, detector test case
leondz Jan 27, 2025
887c18d
Turns can have text==None but cannot be Nones themselves
leondz Jan 27, 2025
5c4cc11
migrate detectors: continuation, fileformats, divergence, encoding, n…
leondz Jan 27, 2025
d519ff1
migrate many detectors
leondz Jan 27, 2025
3b65324
migrate HFDetectors
leondz Jan 27, 2025
512780d
migrate llmaj detector
leondz Jan 27, 2025
57ad823
migrate base detector tests
leondz Jan 27, 2025
1e1f147
migrate fileformats detectors and tests
leondz Jan 27, 2025
3db4bc4
migrate lrl buff
leondz Jan 27, 2025
47d912f
get None responsibility in the right place (turn content - can't repl…
leondz Jan 27, 2025
6147759
clarify fileformat typing wrt. Turn
leondz Jan 27, 2025
8fa359c
migrate function_single test
leondz Jan 27, 2025
0370260
Merge branch 'NVIDIA:main' into feature/turn_objects
leondz Jan 27, 2025
1c455d1
migrate base generator and base generator tests
leondz Jan 28, 2025
6a6ad47
test generators migrated
leondz Jan 28, 2025
d7b2e71
black
leondz Jan 28, 2025
4c11faf
migrate test, function generators to Turn
leondz Jan 28, 2025
8087442
migrate rest generator
leondz Jan 28, 2025
23dc5eb
migrate replicate generator
leondz Jan 28, 2025
3b257a9
migrate ollama generator
leondz Jan 28, 2025
8b0c557
migrate octo generator
leondz Jan 28, 2025
8152951
migrate nvcf generator
leondz Jan 28, 2025
98486b3
migrate nemo generator
leondz Jan 28, 2025
e5f6513
migrate langchain serve generator
leondz Jan 28, 2025
3aab2f4
migrate ggml generator
leondz Jan 28, 2025
0005245
migrate groq generator
leondz Jan 28, 2025
7c350da
migrate guardrails generator
leondz Jan 28, 2025
0ae2b81
migration on hf, litellm, octo, ollama
leondz Jan 29, 2025
6a3d66a
merge main
leondz Feb 13, 2025
ab9e01c
prune ConversationalPipeline
leondz Feb 13, 2025
5d7b697
update hf to using Turn
leondz Feb 13, 2025
98e50db
add Turn typechecking in base generator .generate() to help everyone …
leondz Feb 13, 2025
41aa3a0
update nvcf, octo, rest with Turn
leondz Feb 13, 2025
7c7b853
consider new pattern for turn extra components
leondz Feb 13, 2025
10ae255
add Turn to generator tests
leondz Feb 14, 2025
e897b8a
i am altering the Turn object. pray i do not alter it any further (pr…
leondz Feb 14, 2025
fef9632
update atkgen to take s instead of its local
leondz Feb 14, 2025
814bc92
map detectors.judge over to Turn
leondz Feb 14, 2025
9bcbc06
update openai to expect turn for single interactions
leondz Feb 14, 2025
41484bc
update xss rx constant name
leondz Feb 14, 2025
51b6e6c
catch a string return
leondz Feb 14, 2025
2130384
cast expected test results to Turn
leondz Feb 14, 2025
f4a644b
update Turn.__str__ to expect 1 part in text-only case
leondz Feb 14, 2025
49a9b17
move openai json to valid json
leondz Feb 17, 2025
957c1ec
Turn inherits from dict for serialisation
leondz Feb 17, 2025
c154a5c
migrate openaicompatible, and type-check its output
leondz Feb 17, 2025
c5691ce
move hf, watson json test files to valid json
leondz Feb 17, 2025
24a6852
migrate watson to Turn
leondz Feb 17, 2025
a69bf8c
migrate litellm
leondz Feb 17, 2025
6c54bed
migrate OllamaGeneratorChat to Turn
leondz Feb 17, 2025
6ee1d92
set expectations about Turn structure and serialisability
leondz Feb 17, 2025
38448e3
clarify docs on default turn part names; migrate nim.Vision; add imag…
leondz Feb 17, 2025
e8f988d
migrate llava
leondz Feb 17, 2025
4d75baa
add test vision generator
leondz Feb 17, 2025
e3a966b
probes.base.Probe.prompts init to empty list
leondz Feb 17, 2025
5b60ed7
move visual jailbreak load up into init, do inheritance correctly
leondz Feb 17, 2025
38324a5
change repr a bit
leondz Feb 17, 2025
1372191
refactor visual jailbreak, add tests
leondz Feb 17, 2025
3610cf6
migrate langchain, cohere
leondz Feb 17, 2025
dba5a20
Merge branch 'NVIDIA:main' into feature/turn_objects
leondz Feb 17, 2025
6316d27
migrate openai to Turn; add typechecking flag to bypass Turn check (u…
leondz Feb 17, 2025
9a30123
update to handle ollama.list() output type
leondz Feb 17, 2025
6d85cf6
migrate ollama & tests to Turn
leondz Feb 17, 2025
911c16a
give the 'not found' exception if the error's a not found one, but ca…
leondz Feb 17, 2025
af4b409
migrate groq to Turn
leondz Feb 17, 2025
3ea5834
adjust OpenAI o- message scope
leondz Feb 17, 2025
ff1331d
don't have opinions about init'ing probes base prompts
leondz Feb 17, 2025
1e84ad9
update exception pattern thrown when invalid openai litellm model req…
leondz Feb 18, 2025
e49e49f
type check generators for Turn patterns
leondz Feb 18, 2025
3ec028b
brief refactor in detector checking
leondz Feb 18, 2025
e86cf32
header docs for Turn
leondz Feb 18, 2025
3569f39
rm conversational pipeline cache entry
leondz Feb 19, 2025
2779129
pretty print test json data to reduce churn
leondz Feb 19, 2025
061dbcc
merge in skip seq feature
leondz Feb 21, 2025
393a22e
restore Turn-based tests for base generators
leondz Feb 21, 2025
45609b2
refactor generator tests, leaving placeholder for base
leondz Feb 21, 2025
916e962
type annotations
leondz Feb 26, 2025
57c13b8
Merge branch 'NVIDIA:main' into feature/turn_objects
leondz Mar 4, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
migrate fileformats detectors and tests
  • Loading branch information
leondz committed Jan 27, 2025
commit 1e1f147d87e7672a62c373576ddcb4bb1eb064b1
2 changes: 1 addition & 1 deletion garak/detectors/base.py
Original file line number Diff line number Diff line change
@@ -259,5 +259,5 @@ def detect(self, attempt: garak.attempt.Attempt) -> Iterable[float]:
continue

else:
test_result = self._test_file(local_filename)
test_result = self._test_file(local_filename.text)
yield test_result if test_result is not None else 0.0
16 changes: 8 additions & 8 deletions tests/detectors/test_detectors_fileformats.py
Original file line number Diff line number Diff line change
@@ -10,7 +10,7 @@

import pytest

from garak.attempt import Attempt
from garak.attempt import Attempt, Turn
import garak.detectors.base
import garak.detectors.fileformats

@@ -38,7 +38,7 @@ def test_possiblepickle_filenames(possible_pickle):
def test_fileispickled_invalid_format():
d = garak.detectors.fileformats.FileIsPickled()
plain_attempt = Attempt(prompt="")
plain_attempt.outputs = ["a", "b", "c"]
plain_attempt.outputs = [Turn(s) for s in ["a", "b", "c"]]
with pytest.raises(
ValueError
) as exc_info: # should not process attempts without correct "format" note
@@ -49,7 +49,7 @@ def test_fileispickled_valid_format():
d = garak.detectors.fileformats.FileIsPickled()
file_attempt = Attempt(prompt="")
file_attempt.notes["format"] = "local filename"
file_attempt.outputs = [""]
file_attempt.outputs = [Turn("")]
l = list(d.detect(file_attempt))
assert l == []

@@ -63,7 +63,7 @@ def test_fileispickled_nonpickle():
"Do you know what the sad part is, Odo? I'm a very good tailor."
)
nonpickle_file.close()
nonpickle_file_attempt.outputs = [nonpickle_file.name]
nonpickle_file_attempt.outputs = [Turn(nonpickle_file.name)]
l = list(d.detect(nonpickle_file_attempt))
assert l == [0.0]

@@ -76,7 +76,7 @@ def test_fileispickled_pickle_default():
test_data_struct = {"key": ["value", 37176]}
pickle.dump(test_data_struct, pickle_file)
pickle_file.close()
pickle_file_attempt.outputs = [pickle_file.name]
pickle_file_attempt.outputs = [Turn(pickle_file.name)]
l = list(d.detect(pickle_file_attempt))
assert l == [1.0]

@@ -93,7 +93,7 @@ def test_fileispickled_pickle_versioned(pickle_ver):
test_data_struct = {"key": ["value", 37176]}
pickle.Pickler(pickle_file, pickle_ver).dump(test_data_struct)
pickle_file.close()
pickle_file_attempt.outputs = [pickle_file.name]
pickle_file_attempt.outputs = [Turn(pickle_file.name)]
l = list(d.detect(pickle_file_attempt))
assert l == [1.0]

@@ -105,7 +105,7 @@ def test_fileisexectuable_nonexec():
with tempfile.NamedTemporaryFile(mode="w+", delete=False) as test_file:
test_file.write("I'm a very good tailor.")
test_file.close()
exec_attempt.outputs = [test_file.name]
exec_attempt.outputs = [Turn(test_file.name)]
results = d.detect(exec_attempt)
assert list(results) == [0.0]

@@ -143,6 +143,6 @@ def test_fileisexectuable_exec(decoded_filename):
exec_attempt.notes["format"] = "local filename"

d = garak.detectors.fileformats.FileIsExecutable()
exec_attempt.outputs = [decoded_filename]
exec_attempt.outputs = [Turn(decoded_filename)]
results = d.detect(exec_attempt)
assert list(results) == [1.0]