PoC:clean-room implementation of real-time AI subtitle for English online-TV(OTT TV)

[zhouwg]

whisper.cpp is an open-source and powerful device-side AI framework/lib/model for ASR(Automatic Speech Recognition, a sub-filed of AI).

I want to integrate the great and powerful whisper.cpp to KanTV for purpose of real-time English subtitle with English online TV on Xiaomi 14.

just looks like the following snapshots by Xiaomi 14's powerful proprietary 6B device-side AI model(aka XiaoAI, or Chinese "小爱") + Xiaomi 14's powerful mobile SoC------Qualcomm SM8650-AB Snapdragon 8 Gen 3 (4 nm).

[zhouwg]

I will start integrating the excellent and amazing whisper.cpp to project KanTV since March 5,2024 after v1.2.9 was released on March 4, 2024 and before that I had been spent about two weeks to migrate some local personal projects to github since Feb 22,2024.

background study:

GGML is a C library for machine learning (ML): https://github.com/rustformers/llm/blob/main/crates/ggml/README.md

Roadmap and FAQ: ggml-org/whisper.cpp#126

Android example app: ggml-org/whisper.cpp#283

whisper.cpp should support NNAPI on Android: ggml-org/whisper.cpp#1249

Android Inference is too slow: ggml-org/whisper.cpp#1070

Use Android NNAPI to accelerate inference on Android Devices: ggml-org/ggml#88

NPU support in whisper.cpp:ggml-org/whisper.cpp#1557

Support for realtime audio input: ggml-org/whisper.cpp#10

The Whisper model processes the audio in chunks of 30 seconds - this is a hard constraint of the architecture.

However, what seems to work is you can take for example 5 seconds of audio and pad it with 25 seconds of silence. This way you can process shorter chunks.

silence removal for transcription implemented: ggml-org/whisper.cpp#1649

Can real-time transcription be achieved?: ggml-org/whisper.cpp#1653

How to increase speech to text speed when using whisper cpp?:ggml-org/whisper.cpp#1635

Benchmark results: ggml-org/whisper.cpp#89

Whisper model files in custom ggml format: https://github.com/ggerganov/whisper.cpp/blob/master/models/README.md

GGUF file format specification:

• https://github.com/GGUF file format specification ggml-org/ggml#302
• gguf.md

Google Highway: https://github.com/google/highway

Tencent ncnn: https://github.com/Tencent/ncnn

updated on 03-13-2024:

SIGFPE on certain audio files:
ggml-org/whisper.cpp#39

Real-time identification of microphone has no result:
sandrohanea/whisper.net#155

How to handle real-time sound streams:
sandrohanea/whisper.net#25

updated on 03-20-2024,
Finetuning models for audio_ctx support(VERY important but bring side-effect: 4cd35dd)
ggml-org/whisper.cpp#1951

here are some strategies from original author to reduce repetition and hallucinations:
ggml-org/whisper.cpp#1507

[zhouwg]

integrate whisper.cpp to KanTV step1:(just migrate original Android sample in official whisper.cpp to KanTV and study something accordingly)

How to practise/play with this branch:

adb logcat | grep KANTV
(logs in Java layer / JNI layer / Native layer would be displayed with same prefix properly, so it's helpful for troubleshooting and tracking source codes in whisper.cpp)

[zhouwg]

move to #62 to avoid misunderstanding

[zhouwg]

I suddenly got an idea to implement PoC - stage1 after study source code in examples/bench/bench.cpp and examples/main/main.cpp.

PoC stage-1
• background study
• high-level study of whisper.cpp( structure of source code, build system of source code, .......)
• migrate original Android example to KanTV
• reuse code in kantv-core and add a simple event handling framework for ASRResearchFragment.java to make further work easier

If it works well as expected, I'll move to PoC - stage2(whispercpp inference with pre-loadded audio file by another method, referenced with original Android sample and examples/main/main.cpp)

PoC stage-2
• re-write(or "refine" would be accurate) Java part & JNI part of whisper.cpp JNI, referenced with original Android sample
• re-implement native part of whisper.cpp JNI by a new method base on PoC stage-1, some codes referenced with examples/main/main.cpp
• reuse code in kantv-core and spent some time to integrate customized FFmpeg within kantv-core to whisper.cpp and it would/might be heavily used/very helpful in the future

if it works well as expected,I'll merge previous works to master branch and create a new branch/baseline accordingly. then I'll move to PoC-stage3(ASR with Live stream -- aka online English TV). PoC - stage3 would be a real challenge for me so I would breakdown it like PoC-S31/PoC-S32/PoC-S33....accordingly

PoC stage-3
• PoC - S31 : investigate performance of mulmat and inference(study internal detail of whisper.cpp)
• PoC - S32 : high level design (HLD) of real-time subtitle by whisper.cpp in KanTV
• PoC - S33: coding work of data path: UI <----> JNI <----> whisper.cpp <----> kantv-play(original name is libkantv-ffmpeg.so, rename it to libkantv-play.so since v1.3.0 because the real codes of customized FFmpeg is statically linked in libkantv-core.so, so rename it to libkantv-play.so to avoid confusion) <----> kantv-core. this step is just like design/coding a pure virtual function and correspond virtual function by C++ and every node in the data path should works fine as expected
• PoC - S34: reuse code in kantv-core and coding work of implementation of audio only record mode
• PoC - S35: whispercpp inference/predication with live stream(online English TV) base on PoC-S33 & PoC-S34

if it works well as expected,I'll move to PoC-stage4(performance analysis and optimization on Android phone).

PoC stage-4
• build optimization
• code optimization
• arch/synthesis optimization
• algorithm optimization
• assemble optimization
• bug fix
• pre-alpha release (updated on 03-17-2024, done on master branch or referenced in branch v1.3.2)
• alpha release

the PoC stage-3 and PoC stage-4 might be taken place simultaneously.

the final goal is implement real-time English subtitle for English online-TV by KanTV + customized whisper.cpp and I'll demo it on Xiaomi 14(because Xiaomi 14 contains a very powerful mobile SoC and I personally purchased one for purpose of software development).

of course, source code of customized whisper.cpp will be found in this Android turn-key project. if it's considered well and accepted by upstream whisper.cpp, I'll submit a PR accordingly.

[zhouwg]

it works as expected(PoC stage1 was finished).

[dinvlad]

How's the performance? Can this be used for real-time transcription on a reasonably old device?

[zhouwg]

How's the performance? Can this be used for real-time transcription on a reasonably old device?

Benchmark of whisper.cpp/GGML's mulmat(matrix computing) seems not good on low-end Android phone.

But the performance of whisper.cpp/GGML on iOS is very good because the original author of whisper.cpp/GGML(the great Georgi Gerganov) spent much time to optimize them with Apple's dedicated machine learning library(just similar to SSE2/SSE3/AVX optimization on X86 architecture).

I think whisper.cpp could be used for real-time subtitle with online TV on high-end Android phone(such as Xiaomi 14, I will demo it later after finish this PoC successfully) and this is the goal of this PoC(this opening issue).

Whisper.cpp/GGML might be not reasonable for old device because complicated math computing need powerful SoC or highly optimized code(just like what Georgi did for iOS/Mac platform).

@liam-mceneaney has provide an Android example to demo transcription.

BTW, the following loop from GGML's official website would be helpful for more information:

ggml-org/whisper.cpp#283

[zhouwg]

it works as expected(PoC stage2 was finished).

the above screenshots can't illustrate any exciting progress in this commit(I'd like to say this is a big milestone and express my sincerely thanks for the great whisper.cpp/GGML again at the moment:I have to say that the more I understand/familiar from whisper.cpp the more feeling I think we all should thanks for the great whisper.cpp/GGML).

or built the APK from source code(branch kantv-poc-with-whispercpp) by Android Studio IDE accordingly.