Repository ·Public ·Research preview
tokenizer_config.json 119 B 9 lines json
1{2 "type": "sentencepiece",3 "file": "tokenizer.model",4 "pad_id": 0,5 "bos_id": 1,6 "eos_id": 2,7 "unk_id": 38}9
main