<name> | argument | required | Lowercase letters, digits, and - |
--from-file | path | | A local JSONL file |
--from-hf | string | | A Hugging Face dataset repo. @<rev> pins a branch, tag, or commit. |
--file | path | required with --from-hf | The JSONL file inside the repo |
--kind | string | required | sft for demonstrations, preference for preference pairs |
--input-key | string | input | The prompt column of an sft row |
--output-key | string | output | The answer column of an sft row |
--prompt-key | string | prompt | The prompt column of a preference row |
--chosen-key | string | chosen | The preferred answer of a preference row |
--rejected-key | string | rejected | The rejected answer of a preference row |
--anonymize | flag | off | Replace personal details before the rows are stored. Anonymization explains it. |
--entities | list | every built-in type | With --anonymize: the types to find. NER types: person, org, location, address. Pattern types: email, phone, ip, account, card, id |
--replace | placeholder, pseudonym, or redact | placeholder | With --anonymize: what a found value becomes |
--threshold | number from 0 to 1 | 0.5 | With --anonymize: the lowest NER confidence that counts as a find |
--keep | path | none | With --anonymize: a text file of terms that are never replaced, one for each line |
--patterns | path | none | With --anonymize: a YAML file that maps a custom type to a regular expression |
--preview | integer | off | With --anonymize: print this many rows before and after, and store nothing |
--json | flag | off | Print JSON on stdout |