feat(realtime): Add audio conversations

[richiejp]

Description

Add enough realtime API features to allow talking with an LLM using only audio.

Presently the realtime API only supports transcription which is a minor use-case for it. This PR should allow it to be used with a basic voice assistant.

This PR will ignore many of the options and edge-cases. Instead it'll just, for e.g., rely on server side VAD to commit conversation items.

Notes for Reviewers

• Configure a model pipeline or use a multi-modal model.
• Commit client audio to the conversation
• Generate a text response (optional)
• Generate an audio response
• Interrupt generation on voice detection?
• Implement message item retrieval so client can get the audio
• Allow the user to configure a composite model (pipeline model) (or can we use existing options for e.g. selecting voice style?)
• Test and fix bugs in new code
• Test for regressions in transcription mode

Fixes: #3714 (but we'll need follow issues)

Signed commits

• Yes, I signed my commits.

[netlify]

✅ Deploy Preview for localai ready!

Name	Link
🔨 Latest commit	37606d4
🔍 Latest deploy log	https://app.netlify.com/projects/localai/deploys/69611b60e2a0c7000827439a
😎 Deploy Preview	https://deploy-preview-6245--localai.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify project configuration.

[richiejp]

It's not clear to me if we have audio support in llama.cpp: ggml-org/llama.cpp#15194

[richiejp]

ggml-org/llama.cpp#13759

[richiejp]

ggml-org/llama.cpp#13784

[mudler]

my initial thought on this was to use the whisper backend for transcribing from VAD, and give the text to a text-to-text backend, this way we can always go back at this. There was also an interface created exactly for this so a pipeline can be kinda seen as a "drag and drop" until omni models are really capable.

However, yes audio input is actually supported by llama.cpp and our backends, try qwen2-omni, you will be able to give it an audio as input, but isn't super accurate (better transcribing for now).

[richiejp]

OK, I tried Qwen 2 omni and had issues with accuracy and context length which aren't a problem for a pipeline.

[richiejp]

#7812

[richiejp]

OpenAI made quite some changes to the API that possibly it would have been better to handle before this, but there are also changes in-flight to the Go realtime API library AFAICT. I really want to get something working, so I am just ignoring these changes for now and will have to address them afterwards.

[richiejp]

and it works. There is a long list of issues however I have the full pipeline working.

[richiejp]

To be clear probably nobody will want to use this given its current state, but we could merge it for my own experimentation and so I don't have to keep rebasing on master. Next I need to update the API to the current OpenAI GA. @mudler

[richiejp]

Build error: "E: Failed to fetch http://security.ubuntu.com/ubuntu/pool/main/libc/libcaca/libcaca0_0.99.beta20-4ubuntu0.1_amd64.deb 404 Not Found [IP: 91.189.91.83 80]". Strange, I can download this file.