Selecting audio track not working (?)

[Stewori]

When I try to play a non-default audio track (other language of a video file than what plays by default) the device always plays the default track anyway. I know this is likely not a cast-api issue, but perhaps you know some hint to solve it anyway.
What I'm doing: looked up info of the file via ffprobe -show_streams and configured audio tracks of the file's Media object accordingly:

		Media.MediaBuilder mbl = Media.builder(url, "video/mp4", Media.StreamType.BUFFERED);
		mbl = mbl.addTrack(new Track(null, // custom data
				true, //inband
				"en", "English",
				null, // roles
				null, // subtype, for text tracks only
				null, // trackContentID url for non-inband tracks
				null, // trackContentType mime type for non-inband tracks
				0, Track.TrackType.VIDEO));
		mbl = mbl.addTrack(new Track(null, // custom data
				true, //inband
				"de", "German",
				null, // roles
				null, // subtype, for text tracks only
				null, // trackContentID url for non-inband tracks
				null, // trackContentType mime type for non-inband tracks
				1, Track.TrackType.AUDIO));
		mbl = mbl.addTrack(new Track(null, // custom data
				true, //inband
				"en", "English",
				null, // roles
				null, // subtype, for text tracks only
				null, // trackContentID url for non-inband tracks
				null, // trackContentType mime type for non-inband tracks
				2, Track.TrackType.AUDIO));
		Media md = mbl.build();
		List<Integer> tracks = List.of(0, 2);
		MediaStatus mstat = se.load(md, tracks, true, 300.0, 1.0, true); 

AI told me I should remux to mp4 container, which I did with the same result. Audio is stereo aac, but the codec should not be the problem since the default track plays fine. Now the AI claims the likely issue is that track is chosen on load. It recommends to load without active tracks and wait a little (or asynchroneously for player state change), then set the wanted tracks. However I'm not sure how to do this with cast-api as is supports setting activeTracks only on load command. Perhaps the AI is suggesting nonsense anyway. I also tried to supply the mime info for each track but that did not change anything.
Did you ever succeed to select a secondary audio track in any way? How?

[Nadahar]

I think you're getting some nonsense from the "AI". There should be no reason to remux as long as the original format is supported, and I don't know what the point would be to not specify the track from the get-go.

I honestly don't remember if I've gotten this to work or not, but I have a vague memory that tells me that the default media receiver don't support selecting tracks. I think it says something in Google's cast documentation somewhere, that this is included in the protocol, but that the "receiver app" will have to implement support.

I'll do some more digging in my old code to see if I find some more info on the subject, but I suspect that it simply can't be done with the default media receiver.

[Nadahar]

I found this JavaDoc in some code I've written in the past, which seems to confirm:

..of the Media object itself and a List of "active" track IDs. The active track IDs can be used to specify which audio and subtitles tracks to use, but the 'Default Media Receiver' application doesn't support multiple audio tracks. It is thus in reality only used to specify subtitles.

So, it seems like this only works for subtitles tracks, not audio tracks, with the default media receiver, and since I wrote this years ago, it seems to Google have no intention of changing that.

If you find another "Chromecast app" that does support it, and that supports the default cast media namespace, it should be possible to get it to work. I have no idea if that exists though.

[Stewori]

The AI told me to remux because the original format was mkv. Thx for your insights anyway. Is it worth trying to prvide the audio track as out-of-band? E.g. in a separate mp3 or ogg file or something?
Edit: does not seem to work

[Nadahar]

The AI told me to remux because the original format was mkv. Thx for your insights anyway. Is it worth trying to prvide the audio track as out-of-band? E.g. in a separate mp3 or ogg file or something?

I have no idea if you can even do that.. but I guess it could be worth a try. But, if remuxing or transcoding is on the table anyway, why not just remux to a stream that only has the audio track you want?

To explain: I assumed that "out-of-band" is only really meant for subtitles, but you never know what might work until you try it. Subtitles are often provided in separate files, which I'm pretty sure is the reason why they provide the option to provide "parallel resources" in the first place. If an implementation can't handle selecting the proper embedded/inband audio track, I find it very unlikely that it supports actually playing two separate streams. There's quite a lot of synchronization to keep track of for that to work properly.

[Nadahar]

I think the default media receiver simply don't give you any options when it comes to audio tracks - it will play "the default" (usually the first). Your only real option is to remux the source so that "the default" is the audio track you want.

[Stewori]

Remuxing is not on the table generally. I just did this to pin down the cause of failure. Quite sad that the default media receiver is so limited I must say. Perhaps I can try to stream the data remuxed on the fly, i.e. not via buffered mode. But even if that would work, then seek won't.

[Nadahar]

But even if that would work, then seek won't.

This is a very well known problem for me, yes. Since it's limited to only seeking by byte (not time codes), you get in this pickle, and the only "option" is to remux to a file/cache, which you then stream. But, this has complications as well, since if you try to seek to the end of a stream early on, the remuxing might not have "gotten that far" yet, and what do you do then? Wait for it - which risks the whole thing to time out?

This all might be possible to achieve using HLS or DASH, but I've never experimented that far. HLS/DASH does allow seeking while transcoding/remuxing, but the implementation is probably very far from trivial.

It might be "easier" to make your own "cast app" - all it takes is that you pay to get a Google developer ID and that you can provide an HTTPS source for "the app" (which is guaranteed to be available 24/7). At least, that's all it used to take, but that might have changed. That said, I don't know how easy or hard it is to handle audio stream selection from a "cast app" - I assume that it relies on some standard "HTML5 playback functionality" from the built-in browser (probably Chrome), and that might not support it either - I don't know.

Here is some info on the subject: https://stackoverflow.com/a/79491255

I must admit that I suspect that you'll have problems getting Chrome to select an audio track from a regular video file as well, HLS/DASH has gotten almost "all the attention" for quite a while now, since that's what the commercial actors mostly use.

[Stewori]

For HLS/DASH, if at all, I would try to use whatever GStreamer supports. That seems to me the most promising bet for an open source crossplattform lib. I'll also give the web receiver path a look. However, as soon as the playback runs in the browser, maybe it does not even have to use google cast api for control.

[Nadahar]

I've never used GStreamer, so I don't know it. I do know that FFmpeg can do these things do, but it's a lot of Process and pipe handling required to make things work.

I don't know what your goal is, but you're quickly getting into the kind of things a media server does 😉