Hidden away in Core Image's Geometry Adjustment category are a set of perspective related filters that change the geometry of flat images to simulate them being viewed in 3D space. If you work in architecture or out-of-home advertising, these filters, used in conjunction with Core Image's rectangle detector, are perfect for mapping images onto 3D surfaces. Alternatively, the filters can synthesise the effects of a perspective control lens.

Project Assets

This post comes with a companion Swift playground which is available here. The two assets we'll use are this picture of a billboard:

...and this picture of The Mona Lisa:

The assets are declared as:

    let monaLisa = CIImage(image: UIImage(named: "monalisa.jpg")!)!
    let backgroundImage = CIImage(image: UIImage(named: "background.jpg")!)!

Detecting the Target Rectangle

Our first task is to find the co-ordinates of the corners of the white rectangle and for that, we'll use a CIDetector. The detector needs a core image context and will return a CIRectangleFeature. In real life, there's no guarantee that it will not return nil, in the playground, with known assets, we can live life on the edge and unwrap it with a !.
    let ciContext =  CIContext()

    let detector = CIDetector(ofType: CIDetectorTypeRectangle,
        context: ciContext,
        options: [CIDetectorAccuracy: CIDetectorAccuracyHigh])

    let rect = detector.featuresInImage(backgroundImage).first as! CIRectangleFeature

Performing the Perspective Transform

Now we have the four points that define the corners of the white billboard, we can apply those, along with the background input image, to a perspective transform filter. The perspective transform moves an image's original corners to a new set of coordinates and maps the pixels of the image accordingly: 
    let perspectiveTransform = CIFilter(name: "CIPerspectiveTransform")!
    perspectiveTransform.setValue(CIVector(CGPoint:rect.topLeft),
        forKey: "inputTopLeft")
    perspectiveTransform.setValue(CIVector(CGPoint:rect.topRight),
        forKey: "inputTopRight")
    perspectiveTransform.setValue(CIVector(CGPoint:rect.bottomRight),
        forKey: "inputBottomRight")
    perspectiveTransform.setValue(CIVector(CGPoint:rect.bottomLeft),
        forKey: "inputBottomLeft")
    perspectiveTransform.setValue(monaLisa,
             forKey: kCIInputImageKey)

 The output image of the perspective transform filter now looks like this:


We can now use a source atop compositing filter to simply composite the perspective transformed Mona Lisa over the background:
    let composite = CIFilter(name: "CISourceAtopCompositing")!

    composite.setValue(backgroundImage,
        forKey: kCIInputBackgroundImageKey)
    composite.setValue(perspectiveTransform.outputImage!,

        forKey: kCIInputImageKey)

 The result is OK, but the aspect ratio of the transformed image is wrong and The Mona Lisa is stretched:


Fixing Aspect Ratio with Perspective Correction

To fix the aspect ratio, we'll use Core Image's perspective correction filter. This filter works in the opposite way to a perspective transform: it converts four points (which typically map to the corners of an image subject to perspective distortion) and converts them to a flat, two dimensional rectangle. 

 We'll pass in the corner coordinates of the white billboard to a perspective correction filter which will return a version of the Mona Lisa cropped to the aspect ration of the billboard if we were looking at it head on:
    let perspectiveCorrection = CIFilter(name: "CIPerspectiveCorrection")!

    perspectiveCorrection.setValue(CIVector(CGPoint:rect.topLeft),
        forKey: "inputTopLeft")
    perspectiveCorrection.setValue(CIVector(CGPoint:rect.topRight),
        forKey: "inputTopRight")
    perspectiveCorrection.setValue(CIVector(CGPoint:rect.bottomRight),
        forKey: "inputBottomRight")
    perspectiveCorrection.setValue(CIVector(CGPoint:rect.bottomLeft),
        forKey: "inputBottomLeft")
    perspectiveCorrection.setValue(monaLisa,
        forKey: kCIInputImageKey)

A little bit of tweaking to centre the corrected image to the centre of the billboard rectangle:
    let perspectiveCorrectionRect = perspectiveCorrection.outputImage!.extent
    let cropRect = perspectiveCorrection.outputImage!.extent.offsetBy(
        dx: monaLisa.extent.midX - perspectiveCorrectionRect.midX,
        dy: monaLisa.extent.midY - perspectiveCorrectionRect.midY)


    let croppedMonaLisa = monaLisa.imageByCroppingToRect(cropRect)

 ...and we now have an output image of a cropped Mona Lisa at the correct aspect ration:

Finally, using the original perspective transform filter, we pass in the new cropped version rather than the original version to get a composite with the correct aspect ratio:
    perspectiveTransform.setValue(croppedMonaLisa,
        forKey: kCIInputImageKey)

    composite.setValue(perspectiveTransform.outputImage!,

        forKey: kCIInputImageKey)

 Which gives the result we're probably after:


Core Image for Swift

Although my book doesn't actually cover detectors or perspective correction, Core Image for Swift, does take a detailed look at almost every aspect of still image processing with Core Image.

Core Image for Swift is available from both Apple's iBooks Store or, as a PDF, from Gumroad. IMHO, the iBooks version is better, especially as it contains video assets which the PDF version doesn't.


Posted by
Labels: CIDetector CIPerspectiveCorrection CIPerspectiveTransform Core Image Core Image for Swift perspective projection swift
8

View comments

  1. Can you go the other way around? Crop to a portion of the image defined by a 4 points?

    ReplyDelete
    Replies

    1. We've already done this at https://github.com/hfossli/AGGeometryKit/, but using our own resampler which makes it all look pretty pixelated. I didn't know CoreImage supported perspective.

      Delete

    2. Perspective correction is the "opposite" of perspective transform.

      Delete

    3. Perspective correction is the "opposite" of perspective transform. Các gói cước 3G Mobifone

      Delete

  2. Hi Simon , This is very helpful blog you have written for perspective.
    What if the area where i have to apply is not CIDetectorTypeRectangle type like polygon or irregular shape ?

    ReplyDelete

  3. I want to rotate image with respect to specific angle before merging.but frame gets change
    please help
    thanks in advance.

    ReplyDelete

  4. I want to rotate image round its center with respect to specific angle before merging.but frame gets change
    please help
    thanks in advance.

    ReplyDelete

  5. This comment has been removed by the author.

    ReplyDelete

It's been a fairly busy few months at my "proper" job, so my recreational Houdini tinkering has taken a bit of a back seat.

5

This blog post discusses a technique for rendering SideFX Houdini FLIP fluids as sparse fields of wormlike particles (hence my slightly over-the-top Sparse Vermiform moniker) with their color based on the fluid system's gas pressure field.

3

This post continues from my recent blog entry, Particle Advection by Reaction Diffusion in SideFX Houdini. In this project, I've done away with the VEX in the DOP network, and replaced it with a VDB Analysis node to create a vector field that represents the gradient in the reaction diffusion volume.

After watching this excellent tutorial that discusses advecting particles by magnetic fields to create an animation of the sun, I was inspired to use the same technique to advect particles by fields that are a function of reaction diffusion systems.

1

I played with animating mitosis in Houdini last year (see Simulating Mitosis in Houdini), but the math wasn't quite right, so I thought I'd revisit my VEX to see if it could be improved. After some tinkering, the video above shows my latest (hopefully improved) results.

4

This post describes a simple way to create a system comprising of a regularly surfaced fluid and a faux grain system.

1

Fibonacci spheres are created from a point set that follows a spiral path to form a sphere (you can see an example, with code at OpenProcessing).

This video contains five clips using SideFX Houdini's Grain Solver with an attached POP Wrangle that uses VEX to generate custom forces. Here's a quick rundown of the VEX I used for each clip (please forgive the use of a variable named oomph).

The Rayleigh-Taylor instability is the instability between two fluids of different densities. It can appear as "fingers" of a denser liquid dropping into a less dense liquid or as a mushroom cloud in an explosion.

The phenomenon "comes for free" in SideFX Houdini FLIP Fluids.

1

Following on from my recent blog post, Mixing Fluids in Houdini, I wanted to simulate a toroidal eddy effect where the incoming drip takes the form of a torus and the fluid flows around the circumference of its minor radius.

I suspect that mixing two or more disparate fluids is one of the first challenges Houdini newbies, like myself, set themselves.

This video contains three treatments of a gravitational tides project using Houdini FLIP fluids.

1

Here's an animation of a rocky planetoid with a liquid core that ejects a stream of water in an aquatic volcano. The planetoid's gravity pulls the water back which settles into streams and pools.

1

This may sound a tad geeky, but reaction diffusion is one of my favorite things. I've been tinkering with different implementations for four years - almost to the day - so, it felt like high-time to have a go at a SideFX Houdini version.

2

Here's a nice-ish looking and easy to set up project that animates a hapless planet caught in the gravitational field of a small dense star. As the planet approaches, it begins to melt and gets sucked into star's inescapable pull.

1

When I started thinking about using SideFX Houdini to simulate accretion, I was expecting to have my VEX skills (or lack of) pushed to their limits. However, after looking at Houdini's grains solver, I was able to come up with quite a nice looking solution with only a handful of lines of VEX.

Following on from my recent post about creating a double pendulum in SideFX Houdini, here's another project that roughly simulates the effect of a magnetic pendulum and visualizes the resulting attractor.

The system consists of a single pendulum and four fields visualized as small spheres.

I looked at creating strange attractors in Houdini recently and there's a more elegant solution over at Entagma. This post looks at an alternative way of modeling an attractor using SideFX Houdini with a double pendulum.

This post expands upon my previous post, Simulating Mitosis in Houdini. After watching Creating Points, Vertices & Primitives using Vex by Fifty50, I wondered what sort of structure my mitosis emulation would create if the generated points were joined to their parent by a connecting rod.

2

Here's a quick and easy way to create a procedural simulation of cell mitosis that, in my opinion, looks pretty impressive. 

The simulation is based on a point cloud which is wrangled inside a solver. My main network looks like:

The initial point cloud is crated with a Point Generate.

4

My first Houdini blog post, Experimenting with Impacts in SideFX Houdini, discussed viscous fluid impacts. This post looks at how to add a little more detail to the fluid object - specifically, I wanted to add colored stripes to the fluid.

1

Houdini VOPs, or VEX Operators, are networks that use a visual programming paradigm to create operators that can, for example, deform geometry, define material shaders or generate particle systems. There is a huge range of nodes available inside a VOP - everything from programming logic (e.g.

1

My recent post, Swarm Chemistry in SideFX Houdini, illustrates an interesting effect, but tweaking the values in the swarm chemistry solver is a pain. What would be great is a user interface to vary the simulation parameters rather than having to edit the VEX by hand.

Hiroki Sayama's Swarm Chemistry is a model I've tinkered with and blogged about for years. Swarm chemistry simulates systems of heterogeneous particles each with genomes that describe simple kinetic rules such as separation and cohesion.

1

This post looks at a super simple Houdini project that scatters cones across the surface of a sphere and uses a Perlin noise function in a VEX wrangle node to randomly transform each cone.

1

I posted recently about simulating the Belousov-Zhabotinsky reaction in Houdini. Although it worked, it wasn't particularly aesthetically pleasing, so here's a hopefully prettier version using a slightly different approach.

As I explore Houdini, I find myself continually in awe at the ease with which I can create complex and physically realistic effects. This post looks at creating geometry and applying a heat source to it to cause it to glow and melt.

2

The Belousov-Zhabotinsky reaction is an amazing looking chemical reaction that can be simulated with some pretty simple maths. I've written about it several times and recreated it in several technologies.

I recently posted Chaos in Swift! Visualizing the Lorenz Attractor with Metal, which was inspired by a Houdini tweet from Erwin Santacruz.

Now that my working days are spent working purely with Swift, it's a good time to stretch my technical and creative muscles and learn something new in my free time. I've long admired Houdini from SideFX, and mastering that is my new goal.

Inspired by this tweet from Erwin Santacruz, the owner of http://houdinitricks.com, I thought it was high time that I played with some strange attractors in Swift.

After reading Paul Hudson's excellent What's new in Swift 3.0 article, I thought I'd take an early dive into Swift 3.0 myself to see how it affects Core Image development.

4

Transverse or lateral chromatic aberration is an optical artefact caused by different wavelengths of light focussing at different positions on a camera's focal plane. It appears as blue and purple fringing which increases towards the edge of an image.

1

I added a new Scatter filter to Filterpedia yesterday. The filter mimics the effect of textured glass with a kernel that randomly offsets its sample coordinates based on a noise texture created with a CIRandomGenerator filter.

Simon Gladman - Advanced image processing with Core Image from Daniel Haight on Vimeo.

I'm pleased to announce that version 1.3 of my book, Core Image for Swift, is now available through Apple's iBooks Store and, as a PDF, through Gumroad.

About Me
About Me
Labels
Labels
Blog Archive
Blog Archive
Loading