Audio support

[samhed]

Would require implementing, for example, a PulseAudio server in javascript.

[monkey-jsun]

Samhed, can you elaborate your thoughts on this?

I see two general approaches:

1. streaming PCM or encoded audio over internet (perhaps extending VNC protocol, or simply open another websocket) and then use web audio API to playback on client machine.
2. extend the webserving capability on the remote machine to provide some audio URL source, e.g., :5901/system-audio.mp3. And on client side we simply present a tag

What do you think?

[samhed]

My thoughts were option 1, I actually have a prototype that works for ESound which uses Web Audio API, but that's quite an outdated protocol..

[kanaka]

@samhed did you mean "Audio Data API". Web Audio API is the modern API. The old deprecated one that Mozilla prototype was the Audio Data API (https://wiki.mozilla.org/Audio_Data_API)

[samhed]

Nope, i meant Web Audio API. I made it last year. I used ESound since it is a very simple protocol and I was mostly after a PoC.

[monkey-jsun]

BTW, just curious about the choice of ESound.

I thought we might be able to use a regular compression codec and
piggyback on VNC (using a custom type).

Jun

On Wed, Jul 22, 2015 at 9:33 AM, Samuel notifications@github.com wrote:

Nope, i meant Web Audio API. I made it last year. I used ESound since it
is a very simple protocol and I was mostly after a PoC.

—
Reply to this email directly or view it on GitHub
#302 (comment).

[ikreymer]

Just wanted to check if anyone is currently working on this (I'm guessing not, but good to check)

@samhed Is your EsounD solution open source?

Otherwise, I was thinking perhaps https://github.com/pdeschen/pcm.js may be a start for pulseaudio

I would like to add sound to my project: http://oldweb.today

[samhed]

@ikreymer I'm afraid it's not open source, but I don't think the EsounD solution is worth much. I'm currently not working on this however.

One issue I see with implementing audio support into the noVNC project is that such a solution would require server-side modifications to function, which I don't think fit in noVNC. First, pulseaudio on the server has to be configured to transmit sound from applications to a TCP port. Furthermore a WebSocket has to be opened to send the audio data from that port to the noVNC client. This requires some sort of server-side service.

But after all noVNC could simply be able to interpret and play audio data from a WebSocket which probably is sufficient.

[samhed]

@ikreymer I had a discussion with my boss about the EsounD solution. We decided to make the client-side open. You can find the patch attached to the following bug:

https://www.cendio.com/bugzilla/show_bug.cgi?id=4822

Edit: Note however, that the patch is for an old version of noVNC so it will probably not work as a drop in without modifications.

[vans163]

For Qemu I think it should be possible, it seems QEMU uses a custom VNC protocol where sound can also be trammited. It would be a matter of customizing the protocol when interacting with a QEMU server and it would require decoding/+playing the sound in the browser.

Also interesting would be patching qemu to pipe in audio to play as if it was microphone input.

[ikreymer]

I wanted to add a quick update.. I've be able to get audio to work by using FFMPEG + PulseAudio. FFMPEG is able to convert to many formats and can even act as an HTTP server. However, main issue I found is the lag.

I've gotten best results by having FFMPEG write to TCP socket in OPUS format, then having a custom proxy that streams over a WebSocket. The custom proxy allows for optimizations, such as skipping what looks like silence.

If anyone is interested, here are some code references:

Script that runs ffmpeg:
https://github.com/oldweb-today/base-browser/blob/master/audio_stream.sh#L7

The TCP->websocket proxy:
https://github.com/oldweb-today/base-browser/blob/master/audio_proxy.py

On the client, I am using MediaSource Extensions API to playback the OPUS stream:
https://github.com/oldweb-today/browsers/blob/master/shepherd/static/browser_controller.js#L466

However, even with all this, there seems to be a 1-3 sec latency between the noVNC and the audio, even on a local machine, on either Chrome or Firefox. (The server is always a local Docker container).

My thinking was that the latency is from the browser decoding OPUS, so I also began experimenting with sending raw (PCM) data and using the more low-level WebAudio API createBuffer() directly. This definitely reduced latency to essentially 0, but had to deal with PCM and manually managing buffers, which I've started to in the code base as well. Was hoping that perhaps an 8-bit PCM stream might be usable in some cases, but haven't gotten it to work well enough yet.

For now, using the OPUS codec and accepting the latency. It may also be possible to use decodeAudioData API but I haven't been able to get that to work and its unclear what codecs are supported..

The above code is part of a larger system, but happy to factor it out if it is useful to someone or if anyone wants to collaborate on improving it.

I would be curious about the latency with Janus and WebRTC approaches.