AVAudioFoundation (3): audio and video editing

This article from: AVAudioFoundation (3): audio and video editing | www.samirchen.com

The main content of this paper is from AVFoundation Programming Guide.

Audio and video editing

As we have a brief idea of the following AVFoundation framework, let’s look at interfaces related to audio and video editing.

A composition can be thought of as a set of orbits (tracks) that can be from different media sources (asset). AVMutableComposition provides the interface to insert or delete tracks, and also to adjust the order of these tracks.

The following figure shows how a new composition gets the corresponding track from the existing asset and splicing to form a new asset.

AVAudioFoundation (3): audio and video editing

When dealing with audio, you can do some custom actions using the AVMutableAudioMix class interface, as shown in the following figure. Now you can specify a maximum volume or the volume gradient for an audio track.

AVAudioFoundation (3): audio and video editing

As shown in the diagram below, we can also use AVMutableVideoComposition to directly process video tracks in composition. When you’re dealing with a single video composition, you can specify its rendering size, zoom ratio, frame rate, and output the final video file. Through some instructions for video composition (AVMutableVideoCompositionInstruction, etc.), we can modify the background color of the video and apply layer instructions. The layer instructions (AVMutableVideoCompositionLayerInstruction) can be used to track the implementation of composition video graphics transformation, add graphics gradient, transparency transformation, increase the transparency of the gradient. In addition, you can apply the animation effects in the Core Animation Framework framework by setting the animationTool attribute of the video composition.

AVAudioFoundation (3): audio and video editing

As shown below, you can use AVAssetExportSession related interfaces to merge audio, mix, and video composition in your composition. You just need to initialize a AVAssetExportSession object, and then set its audioMix and videoComposition attributes to your audio, mix, and video composition, respectively.

AVAudioFoundation (3): audio and video editing

Create Composition

Here’s a brief overview of the audio and video editing scene. Now let’s give you a detailed overview of the specific interface. Start with AVMutableComposition.

When using AVMutableComposition to create your own composition, the most typical, we can use AVMutableCompositionTrack composition to add one or more composition tracks, for example, a simple example is to add an audio track and a video track to a composition:

AVMutableComposition *mutableComposition = [AVMutableComposition composition] Create the video composition track.; / / AVMutableCompositionTrack *mutableCompositionVideoTrack = [mutableComposition addMutableTrackWithMediaType:AVMediaTypeVideo preferredTrackID:kCMPersistentTrackID_Invalid] Create the audio composition track.; / / AVMutableCompositionTrack *mutableCompositionAudioTrack = [mutableComposition addMutableTrackWithMediaType:AVMediaTypeAudio preferredTrackID: kCMPersistentTrackID_Invalid];

When you add a new track for composition, you need to set its media type (media type) and track ID, the main media types include audio, video, subtitles, text, and so on.

It should be noted that each track requires a unique track ID, it is more convenient for ID: set track kCMPersistentTrackID_Invalid to obtain the corresponding track automatically generate a unique ID.

Adding audio-visual data to Composition

The media data is added to a composition track needs to access the media data in AVAsset, multiple track can use the AVMutableCompositionTrack interface with the same type of media added to the same composition track. The following example is to take a video asset track from each of the two AVAsset and add it to a new composition track:

Can retrieve AVAssets from a / You number of places, like the camera roll for example. AVAsset *videoAsset #AVAsset with at = < least one video; AVAsset track#> *anotherVideoAsset = < #another AVAsset with at least one video track#> the first; / / Get video track from each asset. AVAssetTrack *videoAssetTrack [[videoAsset tracksWithMediaType:AVMediaTypeVideo] AVAssetTrack *anotherVideoAssetTrack = objectAtIndex:0]; [[anotherVideoAsset = tracksWithMediaType:AVMediaTypeVideo] objectAtIndex:0]; them both to the composition. / Add [mutableCompositionVideoTrack insertTimeRange:CMTimeRangeMake (kCMTimeZero, videoAssetTrack.timeRange.duration) ofTrack:videoAssetTrack atTime:kCMTimeZero error:nil]; insertTime mutableCompositionVideoTrack Range:CMTimeRangeMake (kCMTimeZero, anotherVideoAssetTrack.timeRange.duration), ofTrack:anotherVideoAssetTrack, atTime:videoAssetTrack.timeRange.duration, error:nil];

Retrieving compatible Composition Tracks

If possible, each media type is best to use only one composition track, which optimizes the use of resources. When you play for media data, media data should be of the same type in the same composition track, you can like the following code from composition to find whether there are compatible with the current asset track composition track, and then use it:

AVMutableCompositionTrack *compatibleCompositionTrack = [mutableComposition mutableTrackCompatibleWithTrack:< #the AVAssetTrack you want to insert#>; if (compatibleCompositionTrack) {continues.} / / Implementation

It is important to note that when multiple video segments are added to the same composition track, frames may be lost when switching between video segments, especially on embedded devices. Based on this problem, you should reasonably select the number of video segments in a composition track.

Set volume gradient

With only one AVMutableAudioMix object, you can do audio processing alone for each audio track in composition.

The following code shows that if you use AVMutableAudioMix to set up a volume gradient for a audio track, add a fade effect to the sound. Use the audioMix class method to get the AVMutableAudioMix instance of the audioMixInputParametersWithTrack: interface; and then use the AVMutableAudioMixInputParameters class of the instance of AVMutableAudioMix and composition in a audio track relationship; then through the AVMutableAudioMix instance to handle the volume.

AVMutableAudioMix *mutableAudioMix = [AVMutableAudioMix audioMix]; Create the audio mix input parameters / object. AVMutableAudioMixInputParameters *mixParameters = [AVMutableAudioMixInputParameters audioMixInputParametersWithTrack: mutableCompositionAudioTrack] the volume ramp to; / / Set slowly fade the audio out over the duration of the composition. [mixParameters setVolumeRampFromStartVolume:1.f toEndVolume:0.f timeRange:CMTimeRangeMake (kCMTimeZero, mutableComposition.duration)]; / / Attach the input parameters to the audio mix. mutableAudioMix.inputParameters = @[mixParameters];

Custom video processing

We use AVMutableAudioMix audio processing, then processing the video, we use AVMutableVideoComposition, only one instance of AVMutableVideoComposition can be treated as video track all the composition, such as setting the rendering size, zoom, broadcast frame rate etc..

Let’s look at some scenes in turn.

Set video background color

All video composition must also correspond to a set of AVVideoCompositionInstruction instances, each containing at least one video composition instruction AVVideoCompositionInstruction. We can use AVMutableVideoCompositionInstruction to create our own video composition instruction, through these instructions, we can modify the background color in composition, layer, instruction and so on postprocessing.

The following code example shows how to create a video composition instruction and a composition of the whole length is set to the background color red:

AVMutableVideoCompositionInstruction *mutableVideoCompositionInstruction videoCompositionInstruction] = [AVMutableVideoCompositionInstruction; mutableVideoCompositionInstruction.timeRange = CMTimeRangeMake (kCMTimeZero, mutableComposition.duration); mutableVideoCompositionInstruction.backgroundColor = [[UIColor redColor] CGColor];

Set transparency gradient

We can also use video composition instructions to apply video composition layer instructions. AVMutableVideoCompositionLayerInstruction can be used to set up video track graphics transformations, graphics, gradients, transparency, transparency, gradients, and so forth. The order of the layer instructions stored in the layerInstructions attribute of a video composition instruction determines how the video frames in the tracks are placed and combined.

The following example code shows how to add a transparency gradient when switching from one video to second videos:

AVAsset *firstVideoAssetTrack #AVAssetTrack representing the = < first video segment played in the composition#> AVAsset; *secondVideoAssetTrack = < #AVAssetTrack representing the second video segment played in the composition#> the first; / / Create video composition instruction. AVMutableVideoCompositionInstruction *firstVideoCompositionInstruction = [AVMutableVideoCompositionInstruction videoCompositionInstruction] its time range to; / / Set span the duration of the first video track. firstVideoCompositionInstruction.timeRange = CMTimeRangeMake (kCMTimeZero, firstVideoAssetTrack.timeRange.duration); / / Create the layer instruction and associate it with the composition video track. AVMutableVideoCompositionLayerInstruction *firstV IdeoLayerInstruction = [AVMutableVideoCompositionLayerInstruction videoCompositionLayerInstructionWithAssetTrack:mutableCompositionVideoTrack] the opacity ramp to; / / Create fade out the first video track over its entire duration. setOpacityRampFromStartOpacity:1.f toEndOpacity:0.f timeRange:CMTimeRangeMake (kCMTimeZero [firstVideoLayerInstruction, firstVideoAssetTrack.timeRange.duration]); / / Create the second video composition instruction so that the second video track isn't transparent. AVMutableVideoCompositionInstruction *secondVideoCompositionInstruction = [AVMutableVideoCompositionInstruction videoCompositionInstruction]; Set its time range to span / the duration of the second video track. secondVideoCompositionInstruction.timeRange = CMTimeRangeMake (firstVideoAssetTrack.timeRange.duration, CMTimeAdd (firstVideoAssetTrack.timeRange.duration, secondVideoAssetTrack.timeRange.duration)); / / Create the second layer instruction and associate it with the composition video track. AVMutableVideoCompositionLayerInstruction *secondVideoLayerInstruction = [AVMutableVideoCompositionLayerInstruction videoCompositionLayerInstructionWithAssetTrack:mutableCompositionVideoTrack] the first layer instruction; / / Attach to the first video composition instruction. firstVideoCompositionInstruction.layerInstructions = @[firstVideoLayerInstruction]; / / Attach the second layer instruction to the second video composition instruction. secondVideoCompositionInstruction.layerInstructions = @[seco NdVideoLayerInstruction] both of the video; / / Attach composition instructions to the video composition. AVMutableVideoComposition *mutableVideoComposition [AVMutableVideoComposition = videoComposition]; mutableVideoComposition.instructions = @[firstVideoCompositionInstruction, secondVideoCompositionInstruction];

Animation effects

We also have the ability to use the Core Animation Framework framework by setting the animationTool attribute of video composition. For example: set up video watermark, video title, animation floating layer and so on.

There are two different ways to use Core Animation in video composition:

  • Add a Core Animation Layer as an independent composition track
  • Use Core Animation Layer directly to render animation in the video frame

The following code shows the latter method of adding watermarks to the center of the video area:

CALayer *watermarkLayer #CALayer representing your desired = < watermark = [CALayer; CALayer image#> *parentLayer layer]; CALayer *videoLayer layer] = [CALayer; parentLayer.frame = CGRectMake (0, 0, mutableVideoComposition.renderSize.width, mutableVideoComposition.renderSize.height); videoLayer.frame = CGRectMake (0, 0, mutableVideoComposition.renderSize.width, mutableVideoComposition.renderSize.height); [parentLayer addSublayer: videoLayer]; watermarkLayer.position = CGPointMake (mutableVideoComposition.renderSize.width/2, mutableVideoComposition.renderSize.height/4); [parentLayer addSublayer:watermarkLayer]; mutableVideoComposition.animationTool = AVVideoCompositionCoreAnimationTool / videoCompositionCoreAnimationToolWithPostProcessingAs VideoLayer:videoLayer inLayer:parentLayer];

A complete example

The example here will show you how to merge two video asset tracks and one audio asset track to a video file, in which the general steps are as follows:

  • Create a AVMutableComposition object and add multiple AVMutableCompositionTrack objects
  • Add the corresponding time range of AVAssetTrack in each composition tracks
  • Check the preferredTransform property of the video asset track to determine the video direction
  • Graphics transformation of video using AVMutableVideoCompositionLayerInstruction objects
  • Set the renderSize and frameDuration attributes of video composition
  • Export video file
  • Save video files to album

The following example code omits some memory management and notifications to remove the relevant code.

1, create a composition / /. Create a composition, and add a audio track and a video track. AVMutableComposition *mutableComposition = [AVMutableComposition composition]; AVMutableCompositionTrack *videoCompositionTrack = [mutableComposition addMutableTrackWithMediaType:AVMediaTypeVideo preferredTrackID:kCMPersistentTrackID_Invalid]; AVMutableCompositionTrack *audioCompositionTrack = [mutableComposition addMutableTrackWithMediaType:AVMediaTypeAudio preferredTrackID:kCMPersistentTrackID_Invalid]; / / 2, add asset. Two video track and one audio track are obtained from the source assets, and two video track are added sequentially in the video composition track above, and a video track is added to the audio composition track. AVAsset *firstVideoAsset #First AVAsset with at = < least one video; AVAsset track#> *secondVideoAsset = < #Second AVAsset with at least one video track#> AVAsset; *audioAsset = < #AVAsset with at least one audio track#> AVAssetTrack [[firstVideoAsset tracksWithMediaType:AVMediaTypeVideo]; *firstVideoAssetTrack = objectAtIndex:0]; AVAssetTrack = *secondVideoAssetTrack [[secondVideoAsset tracksWithMediaType:AVMediaTypeVideo] objectAtIndex:0]; AVAssetTrack audioAssetTrack = [[audioAsset * tracksWithMediaType:AVMediaTypeAudio] objectAtIndex:0] [videoCompositionTrack insertTimeRange:CMTimeRangeMake (kCMTimeZero, firstVideoAssetTrack.timeRange.duration) ofTrack:firstVideoAssetTrack atTime:kCMTimeZero error: nil]; [videoCompositionTr ACK insertTimeRange:CMTimeRangeMake (kCMTimeZero, secondVideoAssetTrack.timeRange.duration) ofTrack:secondVideoAssetTrack atTime:firstVideoAssetTrack.timeRange.duration error:nil]; [audioCompositionTrack insertTimeRange:CMTimeRangeMake (kCMTimeZero, CMTimeAdd (firstVideoAssetTrack.timeRange.duration, secondVideoAssetTrack.timeRange.duration)) ofTrack:audioAssetTrack atTime:kCMTimeZero error:nil]; / / 3, check the composition direction. After adding audio, track, and video track in composition, you must also ensure that all of the video track video directions are consistent. By default, video track defaults to "cross screen" mode, and if the video track is added in the vertical screen mode, then the exported video will be in the wrong direction. Similarly, a horizontal video and a vertical video are merged and exported, and export session will make a mistake. BOOL isFirstVideoPortrait CGAffineTransform = NO; firstTransform = firstVideoAssetTrack.preferredTransform; / / Check the first video track's preferred transform to determine if it was recorded in portrait mode. if & & (firstTransform.a = = 0; firstTransform.d = = 0; & & (firstTransform.b = 1 || firstTransform.b = = -1.0) & & (firstTransform.c = = 1; firstTransform.c = -1.0 ||)) {isFirstVideoPortrait = YES;} BOOL isSecondVideoPortrait CGAffineTransform = NO; secondTransform = secondVideoAssetTrack.preferredTransform; / / Check the second video track's preferred transform to determine if it was recorded in portrait mode. if & (secondTransform.a = = 0; & = = 0; secondTransform.d & & (secondTransform.b = = 1; || secondTransform.b = = -1.0) & & (secondTransform.c = = 1; || secondTransform.c = = -1.0) {isSecondVideoPortrait}) = YES (if (isFirstVideoAssetPortrait; & & isSecondVideoAssetPortrait (isFirstVideoAssetPortrait!) ||! & & isSecondVideoAssetPortrait)) {UIAlertView *incompatibleVideoOrientationAlert = [[UIAlertView alloc] initWithTitle:@ "Error! Cannot combine a video" message:@ "shot in portrait mode with a video shot in landscape mode." delegate:self cancelButtonTitle:@ "Dismiss" otherButtonTitles:nil]; [incompatibleVideoOrientationAlert show]; return;} / / 4, Video Composition Layer Instructions application. Once you know what you want to merge the video clip direction is compatible, so you can then use the essential layer fragment for each instructions, and the layer instructions added to video composition. / / all `AVAssetTrack` object has a `preferredTransform` property that contains the direction information of asset track. This transform will be applied when the asset track is displayed on the screen. In the following code, the layer instruction transform is set to the asset track transform, so that when you modify the video size, the new composition video can also be displayed correctly. AVMutableVideoCompositionInstruction *firstVideoCompositionInstruction = [AVMutableVideoCompositionInstruction videoCompositionInstruction] the time range of; / / Set the first instruction to span the duration of the first video track. firstVideoCompositionInstruction.timeRange = CMTimeRangeMake (kCMTimeZero, firstVideoAssetTrack.timeRange.duration); AVMutableVideoCompositionInstruction *secondVideoCompositionInstruction = [AVMutableVideoCompositionInstruction videoCompositionInstruction] the time range of; / / Set the second instruction to span the duration of the second video track. secondVideoCompositionInstruction.timeRange (firstVideoAssetTrack.timeRange.duration, CMTimeAdd = CMTimeRangeMake (firstVideoAssetTrack.timeRange.duration, second VideoAssetTrack.timeRange.duration)); / / create a two video layer instruction video composition, associated with the corresponding track, and set the transform to preferredTransform. AVMutableVideoCompositionLayerInstruction *firstVideoLayerInstruction = [AVMutableVideoCompositionLayerInstruction videoCompositionLayerInstructionWithAssetTrack:videoCompositionTrack] the transform of the; / / Set first layer instruction to the preferred transform of the first video track. [firstVideoLayerInstruction setTransform:firstTransform atTime:kCMTimeZero]; AVMutableVideoCompositionLayerInstruction *secondVideoLayerInstruction = [AVMutableVideoCompositionLayerInstruction videoCompositionLayerInstructionWithAssetTrack:videoCompositionTrack] the transform of the; / / Set second layer instruction to the preferred transform of the second video track. [secondVideoLayerInstruction setTransform:secondTransform atTime: firstVideoAssetTrack.t ImeRange.duration]; firstVideoCompositionInstruction.layerInstructions = @[firstVideoLayerInstruction]; secondVideoCompositionInstruction.layerInstructions = @[secondVideoLayerInstruction]; AVMutableVideoComposition = *mutableVideoComposition [AVMutableVideoComposition = @[firstVideoCompositionInstruction videoComposition]; mutableVideoComposition.instructions, secondVideoCompositionInstruction]; / / the 5, setting up the size and frame rate. To completely solve the video orientation problem, you also need to adjust the `renderSize` property of the video composition, and you also need to set a suitable `frameDuration`, such as 1/30, which represents 30 frames per second. In addition, the `renderScale` defaults to 1. CGSize naturalSizeFirst, naturalSizeSecond the first video; / / If asset was shot in portrait mode, then so was the second one if we made it here. if (isFirstVideoAssetPortrait) the width and {/ / Invert height for the video tracks to ensure that they display properly. naturalSizeFirst = CGSizeMake (firstVideoAssetTrack.naturalSize.height, firstVideoAssetTrack.naturalSize.width) = CGSizeMake (secondVideoAssetTrack.naturalSize.height, naturalSizeSecond; secondVideoAssetTrack.naturalSize.width);} else {/ / If the videos weren't shot in portrait mode, we can just use their natural sizes. naturalSizeFirst = firstVideoAssetTrack.naturalSize; naturalSizeSecond = secondVideoAssetTrack.naturalSize;} float renderWidth render Height the renderWidth and renderHeight; / / Set to the max of the two videos widths and heights. if (naturalSizeFirst.width > naturalSizeSecond.width) {renderWidth = naturalSizeFirst.width;} else {renderWidth} if (naturalSizeFirst.height = naturalSizeSecond.width; > naturalSizeSecond.height) {renderHeight = naturalSizeFirst.height;} else {renderHeight} = naturalSizeSecond.height; mutableVideoComposition.renderSize = CGSizeMake (renderWidth renderHeight); / / Set the frame duration to an appropriate value (i.e. 30 frames per second for video). MutableVideoComposition.frameDuration = CMTimeMake (1,30); / / 6, export composition and keep to the album. Creates a `AVAssetExportSession` object and sets the corresponding `outputURL` to export the video to the specified file. At the same time, we can also use the `ALAssetsLibrary` interface to store the exported video files in the album. Create a static date formatter so / we only have to initialize it once. static NSDateFormatter *kDateFormatter; if (kDateFormatter! = [[NSDateFormatter) {kDateFormatter alloc] init]; kDateFormatter.dateStyle = NSDateFormatterMediumStyle; kDateFormatter.timeStyle = NSDateFormatterShortStyle;} / / Create the export session with the composition and set the preset to the highest quality. AVAssetExportSession *exporter [[AVAssetExportSession alloc] initWithAsset:mutableComposition = presetName:AVAssetExportPresetHighestQuality]; / / Set the desired output URL for the file created by the export process. exporter.outputURL defaultManager] URLForDirectory:NSDocumentDirectory inDomain:NSUserDomainMask appropriateFor = [[[[NSFileManager URL:nil create:@YES error:nil] URLByAppendingPathComponent:[kDateFormatter stringFromDate:[NSDate date]]] URLByAppendingPathExtension:CFBridgingRelease (UTTypeCopyPreferredTagWithClass ((CFStringRef) AVFileTypeQuickTimeMovie, kUTTagClassFilenameExtension)]); / / Set the output file type to be a QuickTime movie. exporter.outputFileType = AVFileTypeQuickTimeMovie; exporter.shouldOptimizeForNetworkUse = YES; exporter.videoComposition = mutableVideoComposition; / / Asynchronously export the composition to a video file and save this file to the camera roll once export completes. [exporter exportAsynchronouslyWithCompletionHandler:^{dispatch_async (dispatch_get_main_queue), if (^{(exporter.status = = AVAssetExportSessionStatusCompleted) { ALAssetsLibrary *assetsLibrary = [[ALAssetsLibrary alloc] init]; if ([assetsLibrary videoAtPathIsCompatibleWithSavedPhotosAlbum:exporter.outputURL]) {[assetsLibrary writeVideoAtPathToSavedPhotosAlbum:exporter.outputURL}}}); completionBlock:NULL];}];

Other reference

  • A simple video editing Demo