Forward audio

[rom1v]

Last month, @yume-chan submitted a working PoC to capture audio on Android. Big thanks for this discovery!

Since then I have worked to properly integrate audio support in scrcpy (capture, encode, transmit, record, decode, resample, play…), and my branch reached a working state (at least on my machine) 🎉

By default, audio is enabled (if supported):

• On Android >= 12, it works out-of-the-box;
• On Android 11, the device screen must be unlocked (because of a dirty workaround to make the system think the app is foreground, again thank you to @yume-chan for the tip);
• On Android <= 10, it does not work, audio is automatically disabled.

If audio capture fails, then mirroring continues with video only (since audio is enabled by default, it is not acceptable to make scrcpy fail if audio is not available), unless --require-audio is set.

To disable audio:

scrcpy --no-audio

If audio is enabled, it is also recorded when recording is enabled:

scrcpy --record=file.mp4
scrcpy --record=file.mkv

To play audio in real-time, the audio player implements drift compensation (it attempts to keep audio sample buffering to a certain level, quite low to get low-latency, but not too low to avoid underflow). It is experimental (I just tinkered with this thing today) (I've made a lot of improvements since the initial MR, and it should work pretty fine), parameters and behavior are hardcoded, some configuration variables might need to be exposed in the future (or not).

If you record, the timestamps are computed on the device, so they are not impacted by any jitter/underflow (this is the same as for the video), so the recording is always clean (if you use --record of course, not if you capture your audio output on the computer).

Like the video, it works over TCP/IP (typically Wifi).

The following options have been renamed:

• --codec -> --video-codec
• --bit-rate -> --video-bitrate
• --codec-options -> --video-codec-options
• --encoder -> --video-encoder

And similar options for audio have been added:

• --audio-codec
• --audio-bit-rate
• --audio-codec-options
• --audio-encoder

Three audio formats are supported:

• Opus: scrcpy --audio-codec=opus
• AAC: scrcpy --audio-codece=aac
• Raw (uncompressed) PCM 16-bit LE: scrcpy --audio-codec=raw (more bandwidth, typically just for testing)

It is possible to list the encoders available on the device (video and audio):

scrcpy --list-encoders

If you need a specific encoder, use --audio-encoder:

scrcpy --audio-codec=aac --audio-encoder='OMX.google.aac.encoder'

When recording, raw audio is not possible (it automatically switches to OPUS). In the future, I might add audio encoding on the client-side specifically for recording, but I don't plan to make it for this audio feature.

I also implemented --audio-buffer= similar to existing --display-buffer= and --v4l2-buffer=.

This branch is in working state.

Since I have refactored A LOT of code, I may have broken things (please tell me!).

Please test, review (good luck, that's a big PR 😉), and report any problem :)

Fixes #14

---

old

Here is a release build for the current MR for Windows (audio.115):

• scrcpy-win64-PR3757-1.zip SHA-256: 8a46774889f0d6602e7f94c1e664b7184fdc0f0ccfe4d114045d2a8eefb408f6
• scrcpy-win32-PR3757-1.zip SHA-256: 7b07819377fa87f1a94f5a424b6757590492b32309d59457fb4eb1e543902bd4

Here is a release build for the current MR for Windows (audio.145):

• scrcpy-win64-PR3757-3.zip SHA-256: 6ef3c45292f5c3b713628e593bbf695be586506cd38a26f038d135cd88d08033
• scrcpy-win32-PR3757-3.zip SHA-256: 06fc50e289494554aa1d86a783960b5fd42268ac18395f224b0ea355aa377dd5

[parkerlreed]

Are the BUILD instructions outdated?

(deck@steamdeck build)$ git clone https://github.com/Genymobile/scrcpy
Cloning into 'scrcpy'...
remote: Enumerating objects: 23178, done.
remote: Counting objects: 100% (3736/3736), done.
remote: Compressing objects: 100% (921/921), done.
remote: Total 23178 (delta 2443), reused 3435 (delta 2189), pack-reused 19442
Receiving objects: 100% (23178/23178), 4.74 MiB | 9.34 MiB/s, done.
Resolving deltas: 100% (14698/14698), done.
(deck@steamdeck build)$ cd scrcpy/
(deck@steamdeck scrcpy)$ git checkout audio
branch 'audio' set up to track 'origin/audio'.
Switched to a new branch 'audio'
(deck@steamdeck scrcpy)$ meson setup x --buildtype=release --strip -Db_lto=true
The Meson build system
Version: 1.0.1
Source dir: /home/deck/build/scrcpy
Build dir: /home/deck/build/scrcpy/x
Build type: native build
Project name: scrcpy
Project version: 1.25
C compiler for the host machine: cc (gcc 12.2.0 "cc (GCC) 12.2.0")
C linker for the host machine: cc ld.bfd 2.39.0
Host machine cpu family: x86_64
Host machine cpu: x86_64
Found pkg-config: /usr/bin/pkg-config (1.8.0)
Run-time dependency libavformat found: YES 59.27.100
Run-time dependency libavcodec found: YES 59.37.100
Run-time dependency libavutil found: YES 57.28.100
Run-time dependency libswresample found: YES 4.7.100
Run-time dependency sdl2 found: YES 2.24.1
Run-time dependency libavdevice found: YES 59.7.100
Run-time dependency libusb-1.0 found: YES 1.0.26
Checking for function "strdup" : YES 
Checking for function "asprintf" : YES 
Checking for function "vasprintf" : YES 
Checking for function "nrand48" : YES 
Checking for function "jrand48" : YES 
Header "sys/socket.h" has symbol "SOCK_CLOEXEC" : YES 
Configuring config.h using configuration

app/meson.build:226:0: ERROR: File src/util/average.c does not exist.

A full log can be found at /home/deck/build/scrcpy/x/meson-logs/meson-log.txt

[parkerlreed]

Seems to be nowhere in the sources? I don't even see a src folder

(deck@steamdeck scrcpy)$ find . -name "average.c"
(deck@steamdeck scrcpy)$ 

EDIT: Oh the folder is there just not that file in app/src/util

[rom1v]

app/meson.build:226:0: ERROR: File src/util/average.c does not exist.

Oops, it was not committed, sorry. Fixed.

[parkerlreed]

ninja: Entering directory `x'
[48/59] Generating server/scrcpy-server with a custom command

FAILURE: Build failed with an exception.

* What went wrong:
Could not open settings generic class cache for settings file '/home/deck/build/scrcpy/settings.gradle' (/home/deck/.gradle/caches/7.5/scripts/4g6ucengw406w5p3xne7pf3a4).
> BUG! exception in phase 'semantic analysis' in source unit '_BuildScript_' Unsupported class file major version 63

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 752ms
[57/59] Linking target app/scrcpy
FAILED: server/scrcpy-server 
/home/deck/build/scrcpy/server/./scripts/build-wrapper.sh /home/deck/build/scrcpy/server server/scrcpy-server release
[59/59] Linking target app/scrcpy
ninja: build stopped: subcommand failed.

[parkerlreed]

Aha java is too new (jdk 19). Trying an older version

[parkerlreed]

It's alive!

Note 10 Plus on Android 12. Working well!

[parkerlreed]

test.mp4

[reitowo]

Looking forward to this PR!

[megapro17]

Yeah I patched an exe just for fun
Android has c2.android.flac.encoder. You just need to add another encoder and enter it everywhere?

[rom1v]

Yes, it is possible to add flac support.

On Windows, you need to build an FFmpeg with flac decoding support: https://github.com/rom1v/scrcpy-deps/blob/231e4e819127dc55f1bf394943ba9887c1cb6224/build_ffmpeg_windows.sh#L40-L41

And in scrcpy, you add flac everywhere in addition to opus and aac. Here is the commit for aac: 4601735

PR welcome 😉