Language selection

[ArtyomZemlyak]

I'm glad you shared this implementation.
A steep increase in performance relative to the torch on the CPU.

It is possible that you already know, but found how to enable recognition of a certain language.
We just can put in line 2012 main.cpp this:

std::vector<whisper_vocab::id> prompt = { vocab.token_sot, vocab.token_lang, vocab.token_task };  

This 3 tokens formed here:
https://github.com/openai/whisper/blob/8cf36f3508c9acd341a45eb2364239a3d81458b9/whisper/tokenizer.py#L324-L331

For specific use in main.cpp, you can simply specify the desired index manually. But for regular users, it would be cool to specify which language they would prefer to see in the output.

[ggerganov]

Thanks for looking into this. I will definitely add support for language selection.
I wasn't 100% sure if it's just the starting tokens that I need to modify to make it work with other languages, but it looks like this is the case.

I will probably add a CLI argument to be able to select the language.

Regarding the performance:
Yes, I suppose the speed for the smaller models should be better compared to torch. For the bigger models however, my matrix multiplication cannot match the performance of the original implementation. I measured about 2-3 times slower performance for 1024 x 4096 matrix sizes on my M1 MacBook.

Also, my sampling strategy is very basic - this is another thing that makes whisper.cpp go faster, but of course, the results won't be as good as a proper beam search implementation.

[ArtyomZemlyak]

I can share the latest tests.

By accuracy:

• Little difference on "ideal" recordings.
• But a rather strong degradation for bad recordings with a lot of noise.

Device

• CPU: i7 11800H
• GPU: RTX 3080 Laptop
• Python: 3.8.12
• torch: 1.8.2

Torch

Model	Time, s	CPU/GPU	RAM, GB	VRAM, GB	DISK, GB
tiny	488	CPU	0.5		0.074
base	564	CPU	1		0.142
small	*3	CPU	2.5		0.472
medium	*20	CPU	6		1.492
large	*30	CPU	10		3.014
tiny	24	GPU	3.4	2.7	0.074
base	29	GPU	3.4	2.7	0.142
small	41	GPU	3.6	3.5	0.472
medium	89	GPU	4.3	6.1	1.492
large	-	GPU	-	-	3.014

C++ ggml

Model	Time, s	CPU/GPU	RAM, GB	VRAM, GB	DISK, GB
tiny	18	CPU	0.4		0.076
large	520	CPU	2.5		3.022

[ggerganov]

Much appreciated!

Few things:

• Your CPU has 8 cores, so make sure to use -t 8 to run whisper.cpp with 8 threads. By default it uses 4
• What does the *30 mean for the large / CPU run with Torch ?
• How long is the audio recording that you used?

[ArtyomZemlyak]

• Yes, -t 8 used for this test.
• Time=488*3, 488*20, 488*30. I did not want to wait for the end of the test, since it was already clear that the speed of processing was sooo low. And I just pointed out that it's about that much slower than the tiny model.
• 7 files, total 200 seconds (and for ggml I subtracted model loading time for this result)

[ggerganov]

I just added an option to be able to select a language.
For example, the following command will translate French audio to English using the small model:

./download-ggml-model.sh small
./main -m models/ggml-small.bin -f fr0.wav --language fr --translate

Additionally, I was able to reduce the memory usage at runtime using flash attention and 16-bit float key/value memory. Also the inference speed has also improved as a result.

[ArtyomZemlyak]

New test on my same device.

Model	time, s	CPU/GPU
tiny	16	CPU
base	32	CPU
large	472	CPU

~10 % for large model better

[frankiedrake]

@ggerganov Language question again :) Is it possible to add a possibility to specify the langs array for each file respectively? To be able to specify different languages for different files

[ggerganov]

It's relatively easy to add this functionality.
Feel free to open an issue with feature request