AV Foundation Layer Instruction

Advanced Editing
with AV Foundation
Session 612
Scott G. Johnston
AV Foundation Engineer
These are confidential sessions—please refrain from streaming, blogging, or taking pictures
Agenda
• Custom video compositing

■ Existing architecture
■ New custom video compositing
■ Choosing pixel formats
■ Tweening
■ Performance
• Debugging compositions
■ Common pitfalls
Agenda

■ Tweening
■ Performance
■ Common pitfalls
Existing Architecture
AV Foundation editing today
• Available since iOS 4.0
and OS X Lion
• Used in video editing apps
from Apple and in the store
• Video editing
■ Temporal composition
■ Video composition
■ Audio mixing
Possible Today
Wipes, Dissolves, Transforms,…
New Opportunities
OpenGL and Everything Else
Custom Video Compositor
What Is a Video Compositor?
• Unit of video mixing code

• Receives multiple source frames
• Blends or transforms pixels
• Delivers single output frame
• Part of the composition architecture
Composition Model
AVComposition
Composition Model
AVComposition
A
B
Composition Model
AVComposition
AVVideoComposition
A B
A B
B A
A
Composition Model
AVComposition
AVVideoComposition
A B
A B
B A
A
Composition Model
AVComposition
AVVideoComposition
A B
A B
B A
A
Composition Model
AVComposition
AVVideoComposition
A B
A B
B A
A
Video Instructions
AVVideoComposition
A B
A B
B A
A
Instruction Inst. Inst. Inst. Instruction

Video Instructions
AVVideoComposition
A B
A B
B A
A

Video Instructions
AVVideoComposition
A B
A B
B A
A
Simple mix
1 source track
Video Instructions
AVVideoComposition
A B
A B
B A
A
Complex mix
>1 source tracks
Video Instructions
AVVideoComposition
A B
A B
B A
A
Compositor
Video Instructions
AVVideoComposition
A B
A B
B A
A
Compositor
Video Instructions
AVVideoComposition
A B
A B
B A
A
Compositor
Video Instructions
AVVideoComposition
A B
A B
B A
A
Compositor
Video Instructions
AVVideoComposition
A B
A B
B A
A
Compositor
Video Instructions
AVVideoComposition
A B
A B
B A
A
Compositor
Video Instructions
AVVideoComposition
A B
A B
B A
A
Compositor
Video Instructions
AVVideoComposition
A B
A B
B A
A
Compositor
Compositor
Compositor
Video Compositor
Compositor
Video Compositor
Instruction
Compositor
Video Compositor
Instruction
Compositor
Video Compositor
Instruction
Compositor
Video Compositor
Instruction
Instruction
Opacity Ramp 1 to 0
Compositor
Agenda

■ Tweening
■ Performance
■ Common pitfalls
Video Compositor
Instruction
Opacity Ramp 1 to 0
Built-In
Compositor
Video Compositor
Instruction
Opacity Ramp 1 to 0
Built-In
Compositor
Video Compositor
Instruction
Opacity Ramp 1 to 0
Built-in
Compositor
Instruction
Opacity Ramp 1 to 0
Your
Built-in
Code
Compositor
Here
Instruction
Opacity Ramp 1 to 0
Your
Code
Here
Instruction
Opacity
Your Ramp
Mixing 1 to 0
Parameters
Your
Code
Here
Instruction
Your Mixing Parameters

Your
Code
Here
Request
Instruction

Your
Code
Here
Request
@protocol AVVideoCompositing
Instruction

Your
Code
Here
AVAsynchronousVideoCompositionRequest
Request
Instruction

Your
Code
Here
AVAsynchronousVideoCompositionRequest
Request
@protocol AVVideoCompositionInstruction
Instruction

Your
Code
Here
Request
Instruction

Your
Code
Here
Request
startVideoCompositionRequest:
Instruction

Your
Code
Here
Request
startVideoCompositionRequest: finishWithComposedVideoFrame:
Instruction

Your
Code
Here
Request finishWithError:
finishCancelledRequest:
Instruction

Your
Code
Here
Request finishWithError:
Instruction

Your
Code
Here
finishWithError:
Request
Instruction
Mixing Parameters
Your
Code
Here
finishWithError:
Request
Instruction
Mixing Parameters
Your
Code
Here
finishWithError:
Request
Instruction
Mixing Parameters
Your
Code
Here
Choosing Pixel Formats
finishWithError:
Request
Instruction
Mixing Parameters
Your
Code
Here
finishWithError:
Request
Instruction
Mixing Parameters
Your
Code
Here
Source Pixel Format

YUV 8-bit 4:2:0
YUV 8-bit 4:4:4
YUV 10-bit 4:2:2
YUV 10-bit 4:4:4
RGB 24-bit
BGRA 32-bit
BGR 24-bit
ARGB 32-bit
ABGR 32-bit
finishWithError:
Request
Instruction
Mixing Parameters
Your
Code
Here
Source Pixel Format

YUV 8-bit 4:2:0
finishWithError:
Request
Instruction
Mixing Parameters
Your
Code
Here
Source Pixel Format

YUV 8-bit 4:2:0
(NSDictionary *) sourcePixelBufferAttributes
finishWithError:
Request
Instruction
Mixing Parameters
Your
Code
Here
Source Pixel Format

YUV 8-bit 4:2:0
kCVPixelFormatType_32BGRA,...
finishWithError:
Request
Instruction
Mixing Parameters
Your
Code
Here
Source Pixel Format

finishWithError:
Request
Instruction
Mixing Parameters
Your
Code
Here
Source Pixel Format

finishWithError:
Request
Instruction
Mixing Parameters
Your
Code
Here
Source Pixel Format

BGRA 32-bit
finishWithError:
Request
Instruction
Mixing Parameters
Your
Code
Here
Source Pixel Format

BGRA 32-bit
finishWithError:
Request
Instruction
Mixing Parameters
Your
Code
Here
Source Pixel Format

BGRA 32-bit
finishWithError:
Request
Instruction
Mixing Parameters
Your
Code
Here
Source Pixel Format Output Pixel Format

BGRA 32-bit
finishWithError:
Request
Instruction
Mixing Parameters
Your
Code
Here
Output Pixel Format
(NSDictionary *) requiredPixelBufferAttributesForRenderContext
finishWithError:
Request
Instruction
Mixing Parameters
Your
Code
Here
Output Pixel Format
finishWithError:
Request
Instruction
Mixing Parameters
Your
Code
Here
Output Pixel Format

finishWithError:
Request
Instruction
Mixing Parameters
Your
Code
Here
Output Pixel Format

finishWithError:
Request
Instruction
Mixing Parameters
Your
Code
Here
Output Pixel Format

BGRA 32-bit
finishWithError:
Request
Instruction
Mixing Parameters
Your
Code
Here
Output Pixel Format

BGRA 32-bit
[request.renderContext newPixelBuffer]
finishWithError:
Request
Instruction
Mixing Parameters
Your
Code
Here
Output Pixel Format

BGRA 32-bit
finishWithError:
Request
Instruction
Mixing Parameters
Your
Code
Here
Output Pixel Format

BGRA 32-bit
finishWithError:
Request
Instruction
Mixing Parameters
Your
Code
Here
Output Pixel Format

BGRA 32-bit
finishWithError:
Request
Instruction
Mixing Parameters
Your
Code
Here
Output Pixel Format

BGRA 32-bit
finishWithError:
Request
Instruction
Mixing Parameters
Your
Code
Here
Output Pixel Format

BGRA 32-bit
finishWithError:
Request
Instruction
Mixing Parameters
Your
Code
Here
Demo
CPU and GPU custom compositors
Agenda

■ Tweening
■ Performance
■ Common pitfalls
Tweening
AVVideoComposition
A B
A B
B A
A
Instruction Instruction Instruction Instruction Instruction
Compositor
Tweening
AVVideoComposition
A B
A B
B A
A
Compositor
Tweening
AVVideoComposition
A B
A B
B A
A
Compositor
Tweening
Tweening
AVVideoComposition
A
B
Instruction
Compositor
Tweening
AVVideoComposition
A
B
Compositor
Tweening
AVVideoComposition
A
B
Opacity Ramp 100% to 0%
Compositor
Tweening
AVVideoComposition
A
B
MyInstruction.timeRange = { start 5, duration 10/30 secs }

MyInstruction.opacityRamp = { 100% to 0% }
Compositor
Tweening
elapsed = 0 secs
AVVideoComposition
A
B
Compositor
Tweening
elapsed = 0 secs
AVVideoComposition
A
B
opacity = 100%
Compositor
Tweening
elapsed = 0 secs
AVVideoComposition
A
B
opacity = 100%
tween = elapsed / duration = 0.0 Compositor

Tweening
elapsed = 1/30 secs
AVVideoComposition
A
B
opacity = 90%
Calculate how far through the animation we are;

tween = elapsed / duration = 0.1 Compositor Subtract the start time, divide by the duration
Tweening
elapsed = .. secs
AVVideoComposition
A
B
opacity = ..%
tween = elapsed / duration = .. Compositor

Tweening
elapsed = .. secs
AVVideoComposition
A
B

opacity = ..%
Compositor tween = elapsed / duration = ..

Tweening
elapsed = 10/30 secs
AVVideoComposition
A
B

opacity = 0%
Compositor tween = elapsed / duration = 1.0

Agenda

■ Tweening
■ Performance
■ Common pitfalls
iOS Apps
Foreground Background
CPU CPU
—
Performance
Instruction properties
@protocol AVVideoCompositionInstruction<NSObject>
{
@property CMPersistentTrackID passthroughTrackID;
@property NSArray *requiredSourceTrackIDs;
@property BOOL containsTweening;
}
Performance
{
}
Performance
passthroughTrackID
AVVideoComposition
A B
A B
B A
A
1 source 1 source 1 source
2 sources 2 sources
Performance
passthroughTrackID
AVVideoComposition
A B
A B
B A
A
passthroughTrackID = A; // bypass compositor

Performance
passthroughTrackID
AVVideoComposition
A B
A B
B A
A
passthroughTrackID = A; // bypass compositor

Performance
{
}
Performance
requiredSourceTrackIDs
AVVideoComposition
A B
A B
B A
A
requiredSourceTrackIDs = @[ A ];
Performance
AVVideoComposition
A B
A B
B A
A
requiredSourceTrackIDs = @[ A ];
Performance
AVVideoComposition
A B
A B
B A
A
requiredSourceTrackIDs = nil
Performance
AVVideoComposition
A B
A B
B A
A
requiredSourceTrackIDs = nil Deliver Frames from ALL Tracks

Performance
{
}
Performance
containsTweening
Moving picture-in-picture,
same source frames every time
Performance
containsTweening
containsTweening = YES;
Moving picture-in-picture,
Performance
containsTweening
Performance
containsTweening
Static picture-in-picture,
Performance
containsTweening
Frame #0
Custom Compositor
Performance
containsTweening
Frame #0 Frame #1
Custom Compositor Custom Compositor
Performance
containsTweening
Frame #0 Frame #1 Frame #2
Custom Compositor Custom Compositor Custom Compositor
Performance
containsTweening
Performance
containsTweening
Re-render identical output!
Performance
containsTweening
Performance
containsTweening
I’m not animating
containsTweening = NO;
Performance
containsTweening
Performance
containsTweening
Frame #0
Custom Compositor
Performance
containsTweening
Frame #0 Frame #1
Custom Compositor
Performance
containsTweening
Custom Compositor
Performance
containsTweening
Reuse identical output
Custom Compositor
Performance
Pixel buffer formats
• Performance hit converting sources
■ H.264 decodes to YUV 4:2:0
■ Best performance, work in YUV 4:2:0
• Output format less critical

■ BGRA or YUV 4:2:0 out
Performance
Pixel buffer formats
• Performance hit converting sources
■ H.264 decodes to YUV 4:2:0
■ Best performance, work in YUV 4:2:0
• Output format less critical

■ BGRA or YUV 4:2:0 out
Sample Code
Custom compositor
• AVCustomEdit
• GPU compositor
• Materials available at:
https://developer.apple.com/wwdc/schedule/details.php?id=612
Agenda

■ New Custom video compositing
■ Tweening
■ Performance
■ Common pitfalls
Debugging Compositions
Common pitfalls
• Gaps between segments
• Misaligned track segments
• Misaligned layer instructions
• Misaligned opacity/audio ramps
• Bogus layer transforms
Common pitfalls
• Misaligned opacity/audio ramps Oops!
A B
B A
B A
Common pitfalls
So close :-(
• Bogus layer transforms Whale.m4v
Otters.m4v
Sea.m4v Waves.m4v
Common pitfalls
Not the alignment
• Misaligned track segments we’re looking for!
Sea.m4v
Waves.m4v
B
A
A
Common pitfalls
• Misaligned layer instructions Overshoot!
B
B
A
Common pitfalls
Common pitfalls
• Bogus layer transforms ?*#!
Demo
AVCompositionDebugViewer
Sample Code
Debugging compositions
• AVCompositionDebugViewer
• Materials available at:
https://developer.apple.com/wwdc/schedule/details.php?id=612
• Drop AVCompositionDebugView into your own app

• Extend it to draw your own video instructions
• Spot overlaps and gaps; tracks, video instructions, and audio mix
• Don’t forget the composition validation API
■ @protocol AVVideoCompositionValidationHandling
Summary

■ Tweening
■ Performance
■ Common pitfalls
Summary

■ Tweening
■ Performance
■ Common pitfalls
Summary

■ Tweening
■ Performance
• Debugging compositions @protocol AVVideoCompositing

@protocol AVVideoCompositionInstruction
■ Common pitfalls AVAsynchronousVideoCompositionRequest
Summary

■ Tweening
■ Performance
■ Common pitfalls
Summary

■ Tweening
■ Performance
■ Common pitfalls
Summary

■ Tweening
■ Performance
■ Common pitfalls
Summary

■ Tweening
■ Performance
■ Common pitfalls
Custom Compositors
Effects, Transitions, Generators
Custom Compositors
Effects, Transitions, Generators
More Information
John Geleynse
Director, Technology Evangelist
[email protected]
Documentation
AVFoundation
http://developer.apple.com/library/ios/#documentation/AudioVideo/Conceptual/AVFoundationPG
Apple Developer Forums

http://devforums.apple.com
Related Sessions
Pacific Heights
Moving to AV Kit and AV Foundation Tuesday 4:30PM
Nob Hill
Preparing and Presenting Media for Accessibility Wednesday 10:15AM
Nob Hill
What's New in Camera Capture Wednesday 11:30AM
Labs
Media Lab B
OS X and iOS Capture Lab Thursday 9:00AM
Media Lab B
AV Foundation Lab Thursday 2:00PM
Media Lab B
AV Foundation Lab Friday 9:00AM

AV Foundation Layer Instruction

Uploaded by

Copyright:

Available Formats

AV Foundation Layer Instruction

Uploaded by

Document Information

Original Description:

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

AV Foundation Layer Instruction

Uploaded by

Copyright:

Available Formats

What are some of the existing capabilities of AV Foundation?

What are some of the existing capabilities of AV Foundation?

How can custom video compositing be achieved in AV Foundation?

How can custom video compositing be achieved in AV Foundation?

Advanced Editing

• Custom video compositing

■ Choosing pixel formats

• Custom video compositing

■ Choosing pixel formats

• Unit of video mixing code

Instruction Inst. Inst. Inst. Instruction

Instruction Inst. Inst. Inst. Instruction

Instruction Inst. Inst. Inst. Instruction

Instruction Inst. Inst. Inst. Instruction

Instruction Inst. Inst. Inst. Instruction

Instruction Inst. Inst. Inst. Instruction

Instruction Inst. Inst. Inst. Instruction

Instruction Inst. Inst. Inst. Instruction

Instruction Inst. Inst. Inst. Instruction

Instruction Inst. Inst. Inst. Instruction

Instruction Inst. Inst. Inst. Instruction

Instruction Inst. Inst. Inst. Instruction

• Custom video compositing

■ Choosing pixel formats

Your Mixing Parameters

Your Mixing Parameters

Your Mixing Parameters

Your Mixing Parameters

Your Mixing Parameters

Your Mixing Parameters

Your Mixing Parameters

Your Mixing Parameters

Your Mixing Parameters

Your Mixing Parameters

Source Pixel Format

Source Pixel Format

Source Pixel Format

Source Pixel Format

Source Pixel Format

Source Pixel Format

Source Pixel Format

Source Pixel Format

Source Pixel Format

Source Pixel Format Output Pixel Format

Output Pixel Format

Output Pixel Format

Output Pixel Format

Output Pixel Format

Output Pixel Format

Output Pixel Format

Output Pixel Format

Output Pixel Format

Output Pixel Format

Output Pixel Format

Output Pixel Format

• Custom video compositing

■ Choosing pixel formats

Instruction Instruction Instruction Instruction Instruction

Instruction Instruction Instruction Instruction Instruction