Extend base test suite with p50k_base and r50k_base token examples · openai/tiktoken@f86f5bc · GitHub

Commit

Extend base test suite with p50k_base and r50k_base token examples

Browse files

See:
* https://github.com/knuddelsgmbh/jtokkit/blob/main/lib/src/test/resources/p50k_base_encodings.csv
* https://github.com/knuddelsgmbh/jtokkit/blob/main/lib/src/test/resources/r50k_base_encodings.csv

Loading branch information

Lőrinc committed Jan 6, 2024

1 parent b683a61 commit f86f5bc