File tree Expand file tree Collapse file tree 1 file changed +1
-1
lines changed Expand file tree Collapse file tree 1 file changed +1
-1
lines changed Original file line number Diff line number Diff line change @@ -56,7 +56,7 @@ tokenizer is around 17x faster than the original tokenizer and 9.6x faster than
56
56
57
57
![ performance-comparison] ( data/performance-comparison.png )
58
58
59
- We updated the Rust RWKV world tokenizer to support multithreading for batch encoding. We ran the same comparison
59
+ We updated the Rust RWKV world tokenizer to support batch encoding with multithreading . We ran the same comparison
60
60
[ script] ( tools/test_tiktoken-huggingface-rwkv.py ) from the [ Huggingface Tokenizers] ( https://github.com/huggingface/tokenizers )
61
61
with the additional rwkv tokenizer. The result shows that the rwkv world tokenizer is significantly faster than
62
62
the Tiktoken and Huggingface tokenizers in all numbers of threads and document sizes (on average, its speed is ten times faster).
You can’t perform that action at this time.
0 commit comments