Picture in Picture kept freezing, and it was the audio session
I have an app, AmbientCast, that puts a 3D printer's camera feed into a Picture in Picture window on an Apple TV, so you can keep an eye on a print in the corner of the screen while something else plays. For weeks that window kept freezing — the picture stopped, the app behind it was fine, and whatever was playing in the foreground stalled alongside it. Killing my app was the only thing that fixed either, at the time.
The symptom was entirely in video. The cause was the audio session.
Everything I replaced, and what it changed
The PiP window is fed by a local pipeline: the camera feed is decoded, a status overlay is composited onto it, the result is re-encoded and served as HLS from a small server inside the app, and a player reads that back to give the system something to put in the window. That is a lot of parts, and over several evenings on a physical Apple TV 4K (3rd generation) on tvOS 27 I replaced most of them. Each of these was a real change, installed and measured on hardware, and none of them moved the freeze:
- the
AVPlayerItem, replaced 42 times - the playhead, seeked forward past the stall
- the
AVPlayeritself, rebuilt - the
AVPlayerLayer, rebuilt - the local HLS server, restarted
- the address that server binds to
- the TCP connections between that server and the player
- the encode resolution, cut to a fraction of normal
- the number of hardware decode sessions, reduced by one
- hardware video sessions leaked at teardown, which turned out to be a real bug and not this one
- the pipeline teardown path
- a mid-stream format change
- the Picture in Picture restore handshake
Thirteen things. Every one of them on the video side. That is not an accident of what I happened to try — the symptom was a frozen video window, so every hypothesis I formed was about video, and I went on forming them for weeks.
The trigger was a transition, not a state
The first thing that actually moved was noticing that the freeze only ever followed a source switch — changing which printer the dashboard is showing. Every experiment above had tested the pipeline in steady state, and steady state was fine. Once I tested across the switch instead, it reproduced on demand:
PiP up, no switch 9.5 minutes, zero freezes
after a source switch froze at 46s
after a source switch froze at 68s
after a source switch froze at 43s
One thing there cost me several trials before I caught it: the degraded
state survives killing the app. The first launch after a reboot ran clean,
and every launch after a source switch froze in about the same 45 seconds
regardless of which process it was. So any trial run after a reproduction
is measuring a poisoned machine, not your change. Reboot between trials.
And note that devicectl device process launch
--terminate-existing is a SIGKILL that runs no cleanup at all, which
means it bypasses the very teardown path I was trying to test.
Why a silent video app holds an audio session at all
Picture in Picture is not a video feature as far as the system is
concerned. It is part of the same capability as background audio and
AirPlay: to get it you declare the audio background mode and
configure a .playback audio session, which is the setup
Apple's
Configuring
your app for media playback describes.
ELI5: what's an audio session?
A device has one set of speakers and many apps that might want them. An audio session is your app's declaration of how it intends to use audio — whether it should keep playing when the screen locks, and crucially whether it plays alongside other apps or takes over from them. The system arbitrates between everyone's declarations.
ELI5: what's a background mode?
iOS and tvOS normally freeze an app the moment it leaves the screen. A background mode is a permission you declare in the app's configuration saying you need to keep doing a specific kind of work anyway. They are narrow, App Review looks at them, and "audio" is the one that covers Picture in Picture.
Three things in my app existed for no reason other than keeping PiP alive:
- the
.playbackcategory, set once at launch; - a silent AAC track written into every HLS segment, because a
video-only stream left
AVPlayerItemstalled on status and tracks, so there was never a playable item to hang a window on; - the player running at
volume = 0rather thanisMuted = true— I'd found earlier that muting it cost me PiP eligibility and setting the volume to zero did not, and both are equally silent.
ELI5: why would a silent app ship an audio track?
The video here is generated on the fly and has no sound. But the parts of the system that hand out Picture in Picture windows expect a normal piece of media, and a normal piece of media has an audio track — without one the player never finished loading the stream, so there was nothing to put in a window. So the stream carries one containing silence — real audio data that happens to be inaudible.
Put together: as far as the rest of the system was concerned, my app was an app playing audio, through a session that interrupts everyone else's. And a source switch tears that whole stack down and builds a new one, which re-claims the session in the middle of whatever else is playing.
The one-line change I had spent weeks not making
// before
try AVAudioSession.sharedInstance()
.setCategory(.playback, mode: .moviePlayback)
// after
try AVAudioSession.sharedInstance()
.setCategory(.playback, mode: .moviePlayback, options: [.mixWithOthers])
The reason it took weeks is the reason it is worth writing down. Every one
of those three deliberate decisions exists to keep Picture in Picture
working, and PiP is the feature this app exists for.
.mixWithOthers looked exactly like the sort of change that
would quietly persuade the system I wasn't really playing audio and stop
handing me a window — and losing PiP to fix an interruption would have been
the worst available trade. I could not find that question answered
anywhere: not in the AVFoundation documentation, not in the forums, not in
anyone's write-up.
So I built the variant and looked at it. With the mixable session, on the same hardware:
audio: session category .playback + .mixWithOthers
pip: didStartPictureInPicture
pip: didStartPictureInPicture
hls: 61 segments, served 101 requests
PiP started on demand, twice, with the pipeline running at its normal rate.
That is one app on one device, and the device is an Apple TV — the
setCategory call is shared with my iOS target, but I have not
re-measured PiP on an iPhone since the change. It is not a guarantee from
Apple either — but it is one more data point than I could find anywhere, so
I will state it plainly: adding .mixWithOthers did not
cost me Picture in Picture on tvOS.
What it fixed
Before the change, the window froze within about a minute of every source switch, without exception. After it, on the same Apple TV with PiP up and a video app playing alongside: 36 minutes clean after one source switch and 4 more after a second, zero freezes, and the local server serving its usual 101 to 103 requests a minute throughout. The app playing in the foreground dropped 0 of 13,148 frames.
The documentation said so the whole time
From Apple's page on the
.playback
category:
By default, using this category implies that your app’s audio is nonmixable—activating your session will interrupt any other audio sessions which are also nonmixable.
That sentence has been there the whole time, and I had read the page. What I never did was connect it to a video app, because my app makes no sound that anyone can hear. The track in those segments is silence. The system does not care whether anyone can hear it.
What I'd tell you to check first
If your symptom crosses into another process, it is a message about something shared. Another app stalling, or recovering when you kill yours, is not a report about your video objects — none of mine ever helped. Audio is usually the only system-wide claim a media app holds, and it was the last thing I looked at rather than the first.
The visible subsystem recruits every hypothesis. A frozen
video window generated thirteen experiments about video. Ask early what
else the feature depends on, and if the answer includes a capability you
declare in UIBackgroundModes, start there.
Read back what the session actually is, at the moment of the symptom. Not what you set at launch — plenty of things reset the category behind you, including AVKit, capture sessions and third-party SDKs, and the last writer wins:
let session = AVAudioSession.sharedInstance()
print("audio: \(session.category.rawValue) opts \(session.categoryOptions.rawValue)")
And test the transition, not the steady state. A pipeline that is healthy while it runs tells you nothing about what happens when it is torn down and rebuilt. Mine was healthy for 9.5 minutes at a stretch and broken 46 seconds after I touched it.