SpeedSix.LensCorrect   (Discreet Lenz Raptor)

Corrects for (or adds) lens distortion. It uses a model of lens distortion that includes symmetric (radial) and asymmetric (x and y curvature) distortions. LensCorrect can semi-automatically remove distortions by estimating the radial distortion parameters that will best make what should be straight lines in the input image into actually straight lines in the output image. This works by having the user mark points along one or more lines in the input that should be straight (e.g. the edges of a building). Only symmetric distortions can be corrected for using this technique - there are too many degrees of freedom when asymmetric distortions are also present. In many cases, however, this "line straightening" method works very well. (Anamorphic - "squeeze" - lenses would be an example of a lens type where there might be significant asymmetric distortion, and hence an example of where "line straightening" would not be sufficient to correct for the distortions present.)

Interpolation type to use when resampling the image.

Choose an editing action for creating or modifying the lines needed for automatic parameter estimation. You should draw a line along one or more features in the input image (e.g. while having View set to Input) that should be straight.

Apply distortion, remove distortion, or automatically remove distortion. Auto Remove will be selected automatically when you use the Auto button.

Output the result image, or simply copy the input image. You need to view the input image using this control (set to Input) when editing straight lines for automatic radial distortion removal.

Semi-automatic Distortion Removal by Line Straightening

If there are straight line features in your input image - ideally with some of them near the image edges - and you don't think that there are likely to be significant asymmetric distortion components present (which will usually be true unless anamorphic lenses are involved), you can have LensCorrect estimate the radial distortion parameters that will correct for the distortion in your input image.

Aside: Note that LensCorrect is adding a distortion here that will "undo" the distortion it estimates is in your image. The fact that LensCorrect considers it is adding a distortion is irrelevant unless you subsequently wish to add back the original distortion. In that case, you need to use another LensCorrect set to Remove Mode - and with the Radial 1 and Radial 2 parameters set to those found by the LensCorrect that has done the "automatic removal". This point will be discussed further in the section on reapplying a lens distortion.

To use semi-automatic distortion removal, proceed as follows:

• 1. Start the LensCorrect plugin with the distorted image sequence as its input. An example of a distorted image is shown here:
  
  
  
  
• 2. If you have not just started or added a new LensCorrect, press Reset to re-initialize the plugin.
• 3. Select View: Input so that you are looking at the distorted input image.
• 4. Select Mode: Auto Remove
• 5. Pick out an image feature that should be a straight line (in an undistorted image).
• 6. Select LineEdit: Add Line. Move the cross hair cursor to the point at which you wish to start the line and press the left mouse button or press the pen down if you are using a digitizing tablet.
• 7. Add more points to the line by moving the cross hair along the image feature you are tracing out and pressing down the pen/mouse. You don't need to add very many points usually. A minimum of 4 and a maximum of 10 might be a good guideline.
• 8. End the line by selecting: LineEdit: No Action. Shown below is an example of a set of lines being entered:
  
  
  
  
• 9. Repeat Step 5 to 8 to create several lines. You won't need too many lines. A guideline would be 2 to 10 lines in total. Ideally, some (or all if there are only a few lines) would be near the edge of the image where the distortion will be most pronounced. With a radial distortion model, there will be no distortion in the center of the image, so lines near the middle won't contribute much information about the distortion that is present.
• 10. When you think you have enough lines, press: Auto Find. LensCorrect will then attempt to find a distortion that straightens out the lines you have digitised in. It will set the Radial 1 and Radial 2 parameters to the distortion it has calculated will straighten the lines. When it has finished, it will automatically set: View: Result so that you can see the results of its efforts. It will also show you where each point you digitised in has been moved to by the distortion (these points are shown a small red boxes):
  
  
  
  
• 11. If the straight line features in the result now actually appear to be straight, and you are generally satisfied with the result, you may be done. Assuming that the distortion is constant for the length of the input image sequence, you can simply run the sequence through LensCorrect without changing anything else. Here is an example of a corrected image:
  
  
  
  Note that if the lens is a zoom lens and the zoom setting changes over the sequence, it is very likely that the distortion will also change! In that case, you will need to follow the Auto Remove process described above at several points in time along the sequence and make notes of the calculated Radial 1 and Radial2 parameters at those points. You will then need to set LensCorrect to Add Mode and enter those radial distortion factors at the appropriate frames, creating a key framed sequence of radial distortion parameters. Gross changes in focus settings may also change the distortion characteristics of a lens.

• Delete Line: Selecting any point on a line will delete the entire line.
• Move Point: Selecting a point on a line will let you move that point by dragging the pen/mouse (press down the pen/mouse button, move whilst down, release the pen/mouse button to stop dragging the point.
• Delete Point: Selecting a point on a line will remove that point from the line.
• Insert Point: Selecting a point and holding the pen/mouse button down will add a new point to the line after the one you selected and let you position it as you wish.

Manual Distortion Addition and Removal

You can try to correct for lens distortion manually simply by adjusting the distortion parameters and watching what happens to the image (look particularly at features you think should be straight lines).

The dominant parameter is Radial1. This controls a radially symmetric distortion that gets more and more pronounced as you move away from the lens center (usually the middle of the image) in any direction. In fact, this distortion is proportional to the distance from the lens center squared. Lenses in which the magnification decreases as you move away from the lens center towards the edges are modelled by positive values of Radial1. This gives rise to barrel distortion which is sometimes seen in wide angle lenses. The image below shows this distortion added to (using Mode: Add) an image by various positive values of Radial1.



If a lens has a magnification that increases as you move away from the center, you get what is called pincushion distortion. The image below shows this distortion added to an image by various negative values of Radial1 (sometimes seen in higher focal length lenses).



Removing distortion using Mode: Remove is designed to "undo" the distortion introduced by Mode: Add. To the extent that Mode: Add accurately models the distortion of real lenses, Mode: Remove can remove the distortion introduced by real lenses. LensCorrect is designed so that using the same value for Radial1 (and the other parameters) in two applications of LensCorrect -- Mode: Add, followed by Mode: Remove applied to the result of the Mode: Add -- will reproduce the original image. (The same applies to a Mode: Remove followed by a Mode: Add). There is one caveat with this, however: adding or removing distortion may "shrink" the image so that parts of it become black (because the distortion "brings in" areas that are outside the original image). This obviously cannot be undone (although the black regions can be trimmed away using the Magnify control). Here is the effect of positive values of Radial1 in Mode: Remove:



The effect of negative values of Radial1 in Mode: Remove is:



One interesting thing to note in the above images is that for extreme distortions -- for example adding with Radial1 set to -1.0 -- single features in the input image appear in more than one place in the distorted image. In this case vertical and horizontal lines are mapped into ellipse-like shapes in the distorted picture. A consequence of this is that such extreme distortions cannot be fully undone, since there is no longer a "unique mapping" back from the distorted image to the original image. The practical consequence of this is that removing extreme distortions will give rise to a halo of points near the edges of the output image where the removal algorithm cannot find a unique location on the input image that corresponds to an output pixel. This "things appearing in more than one place" happens with real lenses (or other bits of glass) that cause very great distortion.

Aside: In fact, mathematically, the algorithm that removes distortion is much "harder" than the one that adds it. The addition process has what is called a closed-form solution. The removal process requires an iterative, approximate solution.

The Radial2 parameter is a "second order" symmetric distortion that is proportional to the distance from the lens center raised to the power four. You should try to remove as much distortion as possible with Radial1 before adding any Radial2 contribution. I.e. try to make straight lines straight with Radial1. Radial2 has the most effect near the edges of the image.

The (usually not very great) distortion of real lenses can usually be corrected for by Radial1, possibly with a little Radial2 added in. Some lenses also have an extra sort of distortion which depends on how far you move along the x-axis only (or the y-axis only). This is usually only significant with anamorphic lenses which are designed to "squeeze" the image along one axis. Below is an image which exhibits both radially symmetric and asymmetric distortion. Not that you can tell this easily just by looking at it!



We can use Radial1 and Radial2 to remove the radially symmetric components of the distortion. Shown below is the result of doing this:



This was done with Auto Remove, in fact. You can see that both the vertical and horizontal lines still show some curvature, however. We can try to correct for this by playing with the XCurvature and YCurvature controls which can independently adjust curvature in the x and y directions. Here is the result of doing that:



Finally, we can use the Magnify control to trim off the parts of the image which had been irretrievably lost by the distortion:



One final note on the asymmetric distortions: these are in addition to the radial distortions -- you cannot add or remove an asymmetric distortion in the absence of any radial distortion. The image below shows the progressive removal first of curvature in the X direction (XCurvature) which straightens the verticals while leaving the horizontals untouched, then the progressive removal of curvature in the Y direction (YCurvature) which straightens the horizontals.

Notes on Removing Distortion and Subsequently Adding Back the Same Distortion

As discussed above, a LensCorrect set to Mode: Add will put back the distortion removed by a LensCorrect set to Mode: Remove if the distortion parameters (Radial1, Radial2, etc.) are set to the same values in both LensCorrect instances. You can also do the Mode: Remove then the Mode: Add. So long as you use the same values for the distortion parameters in the Add as you use in the Remove you will end up with the original image. An example of the equivalence of Add/Remove and Remove/Add is shown below:



If the original image was distorted and the first LensCorrect instance is adjusted to remove the distortion, then the second LensCorrect instance will restore the original distortion. If you feed undistorted footage into the second LensCorrect instance, it will acquire the distortion exhibited by the original image you fed into the first LensCorrect instance. This lets you match other footage to the distortion you have in the original image.

There is one major issue to bear in mind, however. LensCorrect operates with fixed resolution input and output images -- the input and output are the same resolution. Since adding or removing distortion can require access to image data outside the input image bounds some of the output image can be "missing". Usually this will come out as black (although there may be a halo of points that in fact do map back to the input image beyond the black region when removing large distortions).

LensCorrect incorporates a naieve way of overcoming this problem - the Magnify control which can be used to "zoom in" to the result image until the black parts that come from outside the input image are trimmed off. You will often need to do something more sophisticated using other tools in your compositing system to trim away these areas.

In fact, in real applications, you may have to work with resized (larger) original images at a resolution that ensures that the "off input" regions do not appear in the final output (which you will have to "cut out" of the processed images from LensCorrect and whatever else you may be doing).

If you are matching computer generated imagery to real footage, one approach might be to use LensCorrect to remove and add back the orignal distortion, as described above. But you could then use the second LensCorrect instance (the one that adds back the original distortion) to distort the CGI footage to match the original. The advantage of distorting the CGI footage is that it is often possible to generate this relatively easily at a size that allows for the "margins" needed by the distortion process.

Another matter to bear in mind is that removing then adding back distortion involves two resampling processes applied to the original footage, whereas applying distortion to CGI imagery involves only one resampling process. Since each resampling involves some (hopefully slight!) loss in quality, the fewer resamplings, the better. It is also possible (if expensive) to generate CGI at arbitrarily high resolutions to compensate for any quality loss due to resampling (say at twice the resolution at which the live action footage was scanned, plus margins). Generally, live action footage is acquired at a fixed resolution, and resizing it to create the required "margins" for distortion correction is (yet another) resampling operation that must (slightly) reduce the image quality.

Below is an example of matching undistorted footage to distorted footage. First we see the distorted original footage:



We feed this in to an instance of LensCorrect and either manually adjust the parameters or use Mode Auto Remove to obtain a satisfactorily undistorted image:



This tells us the parameters we need for another instance of LensCorrect set to Mode: Add (or Mode: Remove if the distortion was removed using Mode:Auto Remove -- see the note below). This will add back the original distortion. It will add it to any footage that is fed in. Below we see an example of some CGI footage that we want to eventually composite with the original footage. It is undistorted, of course:



After feeding this into the instance of LensCorrect set up to add back the distortion found in the original footage, we get this image:



If we composite (or blend here) the distorted CGI and the original distorted footage, we see that they line up correctly:



One final note: As mentioned above, Mode: Auto Remove -- in which LensCorrect semi-automatically removes distortion by calculating parameters that straighten line features supplied by the user -- works effectively as Mode: Add. You need to use the parameters found by Mode: Auto Remove in a subsequent Mode: Remove instance of LensCorrect to add back the original distortion.

Lens Centre and Squeeze Factor

The Lens Center sets the position of the optical centre of the lens (it is specified in normalized image coordinates (0,0) bottom left of the image and (1,1) top right). For any good lens and camera system, the optical center should be almost exactly in the middle of the image plane (and hence image). So the Lens Centre will almost always be (0.5,0.5). You can change it to anywhere on the image, however.

There is one nasty side-effect, though. As you move the lens centre away from the image centre, more and more of the output image is likely to come from "off" the input image -- and hence be black. Below is an example of an image distorted with Lens Centre set to (0.65,0.38):



And here is the result of un-doing that distortion:



As you can see, we end up with an undistorted picture. But we also end up with a lot of black. This is inevitable.



Squeeze should be set to the "horizontal squeeze factor" of the input image (including contributions from anamorphic "squeeze" lenses and any "non-square pixel" image formats). If you are not using an anamorphic lens, it is usually satisfactory to leave it at 1.0. Adjusting this control on an otherwise undistorted image will have no effect. It is used internally by LensCorrect to compensate for non-uniform scales in the X and Y directions. Since the most important lens distortions depend on distance measured radially from the lens centre, LensCorrect needs to know about these axis scale factors so it can calculate radial distances correctly.