Last night, I gave a talk at NSLondon on Apple's Core Image framework. Speaking to attendees afterwards, there were two frequent comments: "Core Image is far easier than I thought" and "Core Image is far more powerful than I thought".
One of the topics I covered was creating a custom filter to create a variable blur effect - something that may initially sound quite complex but is actually fairly simple to implement. I thought a blog post discussing the custom filter in detail may help spread the word on Core Image's ease of use and awesome power.
This post will discuss the kernel code required to create the effect and the Swift code required to wrap up the kernel in a CIFilter. It's worth noting that Core Image includes its own CIMaskedVariableBlur filter which we'll be emulating in this post.
Core Image Kernels
At the heart of every built in Core Image filter is a kernel function. This is a small program that's executed against every single pixel in the output image and is written in a dialect of OpenGL Shading Language called Core Image Kernel Language. We'll be looking at Core Image's CIKernel class which manages a kernel function and expects code that looks a little like:
kernel vec4 kernelFunction(sampler image)
{
return sample(image, samplerCoord(image));
}
In this example, the function accepts a single argument, image, which is of type sampler and contains the pixel data of the source image. The sample function returns the pixel color of the source image at the co-ordinates of the pixel currently being computed and simply returning that value gives is a destination image identical to the source image.
Variable Blur Kernel
Our filter will be based on a simple box blur. The kernel function will sample neighbouring pixels in a square centred on the pixel currently being computed and return the average color of those pixels. The size of that square is a function of the blur image - the brighter the corresponding pixel in the blur image, the bigger the square.The code is a little more involved than the previous "pass-through" filter:
kernel vec4 lumaVariableBlur(sampler image, sampler blurImage, float blurRadius)
{
vec3 blurPixel = sample(blurImage, samplerCoord(blurImage)).rgb;
float blurAmount = dot(blurPixel, vec3(0.2126, 0.7152, 0.0722));
int radius = int(blurAmount * blurRadius);
vec3 accumulator = vec3(0.0, 0.0, 0.0);
float n = 0.0;
for (int x = -radius; x <= radius; x++)
{
for (int y = -radius; y <= radius; y++)
{
vec2 workingSpaceCoordinate = destCoord() + vec2(x,y);
vec2 imageSpaceCoordinate = samplerTransform(image, workingSpaceCoordinate);
vec3 color = sample(image, imageSpaceCoordinate).rgb;
accumulator += color; +
n += 1.0; +
}
}
accumulator /= n;
return vec4(accumulator, 1.0);
}
In this kernel we:
- Calculate the blur amount based on the luminosity of the current pixel in the blur image and use that to define the blur radius.
- Iterate over the surrounding pixels in the input image, sampling and accumulating their values.
- Return the average of the accumulated pixel values with an alpha (transparency) of 1.0
To use that code in a filter, it needs to be passed in the constructor of a CIKernel:
let maskedVariableBlur = CIKernel(string:"kernel vec4 lumaVariableBlur....")
Implementing as a Core Image Filter
With the CIKernel created, we can wrap up that GLSL code in a Core Image filter. To do this, we'll subclass CIFilter:
class MaskedVariableBlur: CIFilter
{
}
var inputImage: CIImage?
var inputBlurImage: CIImage?
var inputBlurRadius: CGFloat = 5
The execution of the kernel is done inside the filter's overridden outputImage getter by invoking applyWithExtent on the Core Image kernel:
override var outputImage: CIImage!
{
guard let
inputImage = inputImage,
inputBlurImage = inputBlurImage else
{
return nil
}
let extent = inputImage.extent
let blur = maskedVariableBlur?.applyWithExtent(
inputImage.extent,
roiCallback:
{
(index, rect) in
return rect
},
arguments: [inputImage, inputBlurImage, inputBlurRadius])
return blur!.imageByCroppingToRect(extent)
}
The roiCallback is a function that answers the question, "to render the region, rect, in the destination, what region do I need from the source image?". My book, Core Image for Swift takes a detailed look at how this can affect both performance and the effect of the filter.
Note that the arguments array mirrors the kernel function's declaration.
Registering the Filter
To finish creating the filter, it needs to be registered. This is done with CIFilter's static registerFilterName method:
CIFilter.registerFilterName("MaskedVariableBlur",
constructor: FilterVendor(),
classAttributes: [kCIAttributeFilterName: "MaskedVariableBlur"])
The filter vendor, which conforms to CIFilterConstructor, returns a concrete instance of our filter class based on a string:
class FilterVendor: NSObject, CIFilterConstructor
{
func filterWithName(name: String) -> CIFilter?
{
switch name
{
case "MaskedVariableBlur":
return MaskedVariableBlur()
default:
return nil
}
}
}
The Filter in Use
Although the filter can accept any image as a blur image, it might be neat to create a radial gradient procedurally (this could even be controlled by a Core Image detector to centre itself on the face!).
let monaLisa = CIImage(image: UIImage(named: "monalisa.jpg")!)!
let gradientImage = CIFilter(
name: "CIRadialGradient",
withInputParameters: [
kCIInputCenterKey: CIVector(x: 310, y: 390),
"inputRadius0": 100,
"inputRadius1": 300,
"inputColor0": CIColor(red: 0, green: 0, blue: 0),
"inputColor1": CIColor(red: 1, green: 1, blue: 1)
])?
.outputImage?
.imageByCroppingToRect(monaLisa.extent)
Core Image's generator filters, such as this gradient, create images with infinite extent - hence a crop at the end. Our radial gradient looks like:
let final = monaLisa
.imageByApplyingFilter("MaskedVariableBlur",
withInputParameters: [
"inputBlurRadius": 15,
"inputBlurImage": gradientImage!])
Which yields this result: where the blur applied to the source is greater where the blur image is lighter:
Core Image for Swift
My book, Core Image for Swift, takes a detailed look at custom filters and covers the high performance warp and color kernels at length. Hopefully, this article has demonstrated that with very little code, even the simplest GLSL can provide some impressive results.Core Image for Swift is available from both Apple's iBooks Store or, as a PDF, from Gumroad. IMHO, the iBooks version is better, especially as it contains video assets which the PDF version doesn't.
If someone wants to try using this example, here's how you can build the full Core Image Kernel string (with a small bug corrected, removed extra "+" from code):
ReplyDeletelet maskedVariableBlur = CIKernel(string:("kernel vec4 lumaVariableBlur(sampler image, sampler blurImage, float blurRadius)" +
"{ \n" +
"vec3 blurPixel = sample(blurImage, samplerCoord(blurImage)).rgb;" +
"float blurAmount = dot(blurPixel, vec3(0.2126, 0.7152, 0.0722));" +
"\n" +
"int radius = int(blurAmount * blurRadius);" +
"\n" +
"vec3 accumulator = vec3(0.0, 0.0, 0.0);" +
"float n = 0.0;" +
"\n" +
"for (int x = -radius; x <= radius; x++)" +
"\n{" +
"for (int y = -radius; y <= radius; y++)" +
"\n{" +
"vec2 workingSpaceCoordinate = destCoord() + vec2(x,y);" +
"vec2 imageSpaceCoordinate = samplerTransform(image, workingSpaceCoordinate);" +
"vec3 color = sample(image, imageSpaceCoordinate).rgb;" +
"accumulator += color; " +
"n += 1.0;" +
"}\n" +
"}\n" +
"accumulator /= n;" +
"\n" +
"return vec4(accumulator, 1.0);" +
"\n" +
"}"))
I am getting:
ReplyDeletefailed because 'lumaVariableBlur', the first kernel in the string, does not conform to the calling convensions of a CIColorKernel.
What does this mean?
Spend several hours with no working filter with this tutorial.
ReplyDelete```
@objc dynamic var inputImage: CIImage?
...
```