Skip to content

Commit 1337ec2

Browse files
committed
updated README.md
1 parent a2bc33f commit 1337ec2

File tree

1 file changed

+1
-1
lines changed

1 file changed

+1
-1
lines changed

README.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -56,7 +56,7 @@ tokenizer is around 17x faster than the original tokenizer and 9.6x faster than
5656

5757
![performance-comparison](data/performance-comparison.png)
5858

59-
We updated the Rust RWKV world tokenizer to support multithreading for batch encoding. We ran the same comparison
59+
We updated the Rust RWKV world tokenizer to support batch encoding with multithreading. We ran the same comparison
6060
[script](tools/test_tiktoken-huggingface-rwkv.py) from the [Huggingface Tokenizers](https://github.com/huggingface/tokenizers)
6161
with the additional rwkv tokenizer. The result shows that the rwkv world tokenizer is significantly faster than
6262
the Tiktoken and Huggingface tokenizers in all numbers of threads and document sizes (on average, its speed is ten times faster).

0 commit comments

Comments
 (0)