feat(sdk): add configurable media3 player builder

[jonathancaudill]

Okie-dokie!

First draft of some more advanced audio capabilities. I can edit the description later as needed before merge.

So as it turns out, I forgot a crucial detail in #119 . The SDK sandbox blocks most stuff regarding Context. So a straight-up handoff isn't really feasible. Below is what I think will work pretty well, and it still honors the split.

SDK's default audio behavior and simple building of LightAudioPlayer instances is unchanged.

SDK will allow more advanced specs to be given when building an ExoPlayer instance via the configure argument. SDK will also allow tools to change aspects of the media3 pipeline live--queueing, etc. Which is good, because streaming requires a lot of small changes to be made as cell service comes and goes, as the user queues up tracks, etc.

SDK will also enforce one LightMediaEnv per tool & one SimpleCache object per directory.

[dupontgu]

Excited to take a look at this! Hopefully will get a chance by EOW

[dupontgu]

I did a quick pass and I know @brunoro is doing the same since he's kinda owning the audio stuff.

I've just got a few little API suggestions that I put up here: 5d073d2

Basically just made a base LightAudioPlayerCore class that can take any arbitrary Player instance (as you added), and then made it so LightAudioPlayer extends that but explicitly takes in an ExoPlayer instance. I think it A) makes it a little clearer that LightAudioPlayer is kind of a safe default (but you can write your own extension of LightAudioPlayerCore if you wanna go nuts) and B) adds a cleaner surface to add those ExoPlayer-specific calls like skipSilenceEnabled.

[brunoro]

Hi @jonathancaudill, thanks for the contribution! I'm glad that you already did some work on Phono, since it's a great real life tool use case that will allow us to properly tune the LightAudioPlayer SDK surface.

I think that the approach your proposed is a good compromise between providing a full Light SDK-wrapped layer for general use cases and leaking Android media objects for advanced usage. Me and @dupontgu both agree it flows well with our thoughts of building an straightforward and opinionated audio library (vs building a even more opinionated audio engine). I'll expand a bit on the next comments on what I see as the two open questions we need cover before merging the PR.

[brunoro]

What it would take to support the proposed API changes on detached audio mode?

This is the trickier part since it involves many aspects of the API. The architecture we're working on for detached (background) audio is to have an SDK-owned foreground service per tool that would own the player instance. That raises some issues:

• LightPlayerConfigurator callbacks wouldn't be able to configure the ExoPlayer instance living in the service → this could be solved by configuring the player object service-side through Binder IPC. We'd need to make sure that the configuration object is serializable but it shouldn't be a huge lift. Same applies to other things that operate on player callbacks.
• State, lifecycle and focus handling might require some additional logic on tool side, since on release the tool just disconnects from the service, and on restart it needs to figure out how to rebuild it's internal state from the current player state → this might involve moving to the service side any eventual player logic that isn't represented in media3.
• Error handling that relies on exception will need to go through an IPC indirection layer → some additional serialization and communication work on tool side would solve this.

[brunoro]

Are the proposed changes enough for a Phono Tidal MVP?

Now I'm getting more on speculative territory because I only scratched the surface on the Phono codebase, so please help me expand the list. I noticed that the Phono Tidal backend uses some features that aren't supported by LightAudio atm:

• DASH playback and downloads: would adding the media3-exoplayer-dash dependency solve it? We would need to wrap it so not all projects depend on it, but it should be easy enough.
• Player error visibility: LightAudio doesn't expose any of that, and having access to the media3 player solves it, but it seems to me like other tools might need, so we might actually pull some of that behavior in the SDK.
• Queue mutation: same as player errors, it's something that wasn't covered but our API that likely other audio tools might need.
• Player state: same as error visibility and queue mutation, it's something we'd want to pull into LightAudioPlayer.
• Track caching: solved in this PR by LightMediaEnv, including any possible lifecycle issues since player and cache aren't released together and cache can survive an eventual player recreation.
• Audio focus handling: I saw that Phono does some of this and it's definitely something we'll want to have the SDK to handle. This needs some care since a tool can override it using the leaked media3 player object.
• Offline downloads: DownloadManager or DownloadService would both require a Context object to be initialized. I'm wondering if we can consider this to be out of the MVP initially, since it opens a bunch of other side quests that we can postpone.
• Reacting to connectivity, battery and power events: ConnectivityManager, BatteryManager and PowerManager all would require Context access. This is something that could also make it into the SDK, not only for audio tools. However, I wonder if the logic on Phono that reacts to these could be simplified to something like caching current track + resolving and prefetching the next one, plus some retry policy.

[jonathancaudill]

@dupontgu @brunoro thanks for all these awesome comments. I'll move in relevant commits and write out my thoughts on @brunoro 's comments when I get some time this evening!

[jonathancaudill]

• Audio focus handling: I saw that Phono does some of this and it's definitely something we'll want to have the SDK to handle. This needs some care since a tool can override it using the leaked media3 player object.

Yes. will definitely need to dedupe this when phono migrates to SDK.

• Offline downloads: DownloadManager or DownloadService would both require a Context object to be initialized. I'm wondering if we can consider this to be out of the MVP initially, since it opens a bunch of other side quests that we can postpone.

Yeah, this one is tricky. My plan was to just run in foreground when tool is open, then ship off as LightWork tasks if tool is closed. Obviously some kind of DownloadManager in the SDK would be good--can maybe mock something up this week or next.

• Reacting to connectivity, battery and power events: ConnectivityManager, BatteryManager and PowerManager all would require Context access. This is something that could also make it into the SDK, not only for audio tools. However, I wonder if the logic on Phono that reacts to these could be simplified to something like caching current track + resolving and prefetching the next one, plus some retry policy.

I would actually push back on this a little bit--Guy had mentioned that it would be good to put these in the SDK, and I definitely agree. And in the short term if we migrate to the SDK before those are implemented, your idea could definitely work.

The main issue is stream quality, since phono will bump stream quality up/down to speed up the caching process per-song. This could probably be replicated with a pretty simple "speed test" of sorts where if caching is moving unusually slow we bump stream quality down. I do need to work on making this more stable for phono as well, so I'm sure the caching will improve and change over time. I do think that the best solution will probably involve some kind of access to these events, though. This also applies to automatically-detected offline mode.

[jonathancaudill]

DASH playback and downloads: would adding the media3-exoplayer-dash dependency solve it? We would need to wrap it so not all projects depend on it, but it should be easy enough.

Yeah, I think this would be a good solution.

[jonathancaudill]

@dupontgu is there anything else you guys need from me at this time for this pr?

[brunoro]

is there anything else you guys need from me at this time for this pr?

@jonathancaudill not atm! I'll move forward with getting the audio SDK what it needs to build the Phono Tidal MVP. From our discussion:

• DASH: get a Light SDK provision for adding the media3-exoplayer-dash dependency. In lightool.toml you'll probably specify it as something like capabilities: audio-dash and our Gradle plugin will take care of the rest.
• Player error visibility, queue mutation and player state: to be exposed in Light SDK
• Track caching: solved in this PR by LightMediaEnv
• Audio focus handling: will need some reworking on Phono side, so no SDK changes needed?
• Offline downloads: the foreground workaround you proposed sounds like a decent middle ground for Tidal MVP. Should be in Light SDK in the future.
• Reacting to connectivity, battery and power events: figure out any heuristics that work well for Phono. Should be in Light SDK in the future.

I'll be putting up a PR shortly for LightAudio detached tool support (background audio) and after that's merged I'll circle back here. There shouldn't be any incompatibilities and I might even add some of the features I discussed above included in there.

[dupontgu]

Hey hey - Happy to merge this after a rebase! I'm happy to do it for ya if that's helpful

[jonathancaudill]

@dupontgu have been slammed at work lately!! will try to rebase this weekend when i get a chance :)

[jonathancaudill]

You're welcome to do it yourself too! Main concern is that the LightAudioPlayer/LightAudioPlayerCore split and the live control of the media3 pipeline survive the recent PR even for detached instances! Was spending some time parsing it late last night but ran out of steam, haha.

[brunoro]

@jonathancaudill hi again! Sorry for stalling this for so long. I was working on getting LightOS to behave nicely with the audio SDK by making the media session playback controls work, focus arbitration, metadata store, etc.

As I expected, detached mode changed a bunch of things so we might need to rethink the approach here. In detached mode the player lives in the SDK-owned foreground service that the system constructs and that outlives the handle owned by the tool, so several hooks in this PR no longer have anywhere to attach. About the four main aspects of this PR, here's a suggestion on how to deal with them:

LightAudioPlayerCore wrapping the Player interface

LightAudioPlayer on main keeps the Player interface, so we don't need LightAudioPlayerCore anymore. We dropped the ExoPlayer-only features (skipSilence and pauseAtEndOfItem) that weren't in Player to keep things simple.

Injecting ExoPlayer.Builder via LightPlayerConfigurator

LightAudioService.onCreate constructs the player, not the tool, and the system constructs the service. The tool holds a MediaController which has no builder. Also, media session can outlive the tool so ownership needs to be in the service. If we still need this feature we'd need to serialize the player constructor parameters, which is more or less what the LightMediaCache I propose below does.

Leaking the player via .media3Player()

We could return MediaController, which is a Player, but not the ExoPlayer instance that lives in the service.
Either way, anything the tool does on a leaked player bypasses the SDK's command queue, so i'd rather expose the control directly. Error handling and the state flows already landed on main, so what's left to be added on LightAudioPlayer is queue mutation (add/remove/move/replace). MediaController forwards all of those, so attached and detached stay identical.

SimpleCache / DatabaseProvider / DataSource via LightMediaEnv

The approach works and the service shares the tool's process, so the singletons are shared.
Implementation needs a pass for thread safety. For instance, in the current implementation two threads could construct a SimpleCache on one directory.

However, thinking about the way detached audio support turned out, i started to think that we can give proper SDK support to a couple of things, rather than exposing android objs:

• A LightMediaCache passed to audio.newPlayer, with SimpleCache and DataSource.Factory built internally. We could attach necessary data to DetachedConnectionHints, like LightAudioUsage does atm.
• For Tidal auth, maybe ResolvingDataSource.Factory wrapped in an SDK-owned resolver can do the job to attach bearer tokens on requests. We might need to figure out the credential rotation case since the resolver belongs to the tool's handle and detached playback can outlive it, so nothing can refresh a token once the tool is gone.

How do you want to handle this? I'll need to switch tracks away from audio for the next weeks, so I won't be able to tackle this myself. If you want to give it a try I can help with reviews and by discussing design idas. It'd be awesome to get a complex tool like Phono supported by the SDK!

[brunoro]

btw @dupontgu merged connectivity hooks this week #166

[jonathancaudill]

@brunoro all of those are great points, and I like the direction you're suggesting. I'll spend some time sketching out a plan tonight or later this weekend, and hopefully we can get something mergeable.

I'll also probably need to rewrite phono's Spotify backend to patch and use librespot-java instead, since Rust seems to be a no-go in the SDK.