Practical Image Registration and Alignment with OpenCV and Python

In this post, I will demonstrate step by step approach to “feature-based” image alignment technique. Image alignment, also known as Image registration, is an image processing technique that helps us align different images of the same picture or scene. let’s say we have a form filled out by various people and those people may take photos of their filled forms from multiple camera angles. Below are some examples that show the variety of camera angles.

Now to analyze these images we need to align these images to the same angle. We can do this by aligning these images to the same angle as a reference image.

The above images are taken from different angles, now let’s see the ideal image (reference image) of the above forms. The image registration algorithm helps us align the above pictures (photos taken from different camera angles) to the same plane as below image (template image)

Why Feature-Based Image Alignment?

In my previous post, I demonstrated how to do image transformation using the Warp perspective. Below are the problems with Warp Perspective transformation:

• All 4 edges of the image must be visible
• We need to know the coordination of each 4 corners of the image
• Difficult to achieve good accuracy while implementing in large-scale application

The above problems can be solved using a feature-based image registration algorithm. This algorithm can implement in large-scale applications.

Recommanded Post:

• Warp perspective and Transform OpenCV python
• Build form parser using OpenCV and Python

How Image registration works

So now you know the advantages of a feature-based image registration algorithm, let’s now understand how it works. The backbone of image alignment technique is 3×3 or 2×2 matrix called Homography.

Applications of Image Alignment

There are various applications of image registration such as:

• Object tracking
• Document processing
• etc.

Image Alignment with Python and OpenCV

Now let’s start writing Python code for image alignment using OpenCV. The input images I will use are shown above. I will devide entire code into some steps to achieve final output of the aligned image

Step1: Read Images

In this step, we will import the required packages and read the template image (reference image) and the image to be aligned.

import cv2
import numpy as np

# Read Image to be aligned
imgTest = cv2.imread('input/test_form4.jpg')
# Reference Reference image or Ideal image
imgRef = cv2.imread('input/template_form.png')

Step2: Image preprocessing

In this image pre-processing step we will just convert both input images to greyscale.

# Convert to grayscale.
imgTest_grey = cv2.cvtColor(imgTest, cv2.COLOR_BGR2GRAY)
imgRef_grey = cv2.cvtColor(imgRef, cv2.COLOR_BGR2GRAY)
height, width = imgRef_grey.shape

Step3: Find Keypoints and Descriptors from images

To match features between two images (ideal image and the image to be aligned), we need to store extract and coordinate information of corresponding key points and descriptors.

• Keypoints are simply selected points or important points that can be used to compute the transformation
• Descriptors: For each keypoints, we extract Descriptors, which is the region surrounding each keypoints in the input image

There are various algorithms to find Keypoints and Descriptors like:

• SIFT
• SURF
• ORB
• etc.

In this post, we will use the ORB algorithm to detect keypoint and local invariant features (descriptors).

# Configure ORB feature detector Algorithm with 1000 features.
orb_detector = cv2.ORB_create(1000)

# Extract key points and descriptors for both images
keyPoint1, des1 = orb_detector.detectAndCompute(imgTest_grey, None)
keyPoint2, des2 = orb_detector.detectAndCompute(imgRef_grey, None)

# Display keypoints for reference image in green color
imgKp_Ref = cv2.drawKeypoints(imgRef, keyPoint1, 0, (0,222,0), None)
imgKp_Ref = cv2.resize(imgKp_Ref, (width//2, height//2))

cv2.imshow('Key Points', imgKp_Ref)
cv2.waitKey(0)

Green points are Keypoints for the Template image. Based on those points image alignment algorithm will work.

You can see key points are concentrated in some particular area of the image which means OBR feature-based algorithm is performing well.

Step4: Match features between two images

Once features are detected, we need to compute the distance between features of the reference image and features of the unaligned image. We will use Hamming method to compute this feature distance. We will use the Brute Force matcher to match those features based on Hamming distance. We will be selecting only the top matches, no need to use noisy matches

# Match features between two images using Brute Force matcher with Hamming distance
matcher = cv2.BFMatcher(cv2.NORM_HAMMING, crossCheck=True)

# Match the two sets of descriptors.
matches = matcher.match(des1, des2)

# Sort matches on the basis of their Hamming distance.
matches.sort(key=lambda x: x.distance)

# Take the top 90 % matches forward.
matches = matches[:int(len(matches) * 0.9)]
no_of_matches = len(matches)

# Display only 100 best matches {good[:100}
imgMatch = cv2.drawMatches(imgTest, keyPoint2, imgRef, keyPoint1, matches[:100], None, flags = 2)
imgMatch = cv2.resize(imgMatch, (width//3, height//3))

cv2.imshow('Image Match', imgMatch)
cv2.waitKey(0)

Step5: Find Homography

Homography matrix is the backbone of feature-based image alignment and registration technique. Once we extracted key points, we need to compute the homography matrix. For this tutorial, I am computing 2X2 homography matrices using keypoints and the RANSAC algorithm. You can try 3X3 matrices also.

Let’s say we are using 3×3 homography matrix as shown below:

Let (x₁,y₁) be the point in the first image and (x2,y₂) be the coordinates of the same physical point in the second image. Then Homography H joins them in the following way:

# Define 2x2 empty matrices
p1 = np.zeros((no_of_matches, 2))
p2 = np.zeros((no_of_matches, 2))

# Storing values to the matrices
for i in range(len(matches)):
    p1[i, :] = keyPoint1[matches[i].queryIdx].pt
    p2[i, :] = keyPoint2[matches[i].trainIdx].pt

# Find the homography matrix.
homography, mask = cv2.findHomography(p1, p2, cv2.RANSAC)

Step6: Warp Perspective

At the final stage, we will apply warp perspective transformation to homograph matrix and unaligned image together to get our final aligned image output.

# Use homography matrix to transform the unaligned image wrt the reference image.
aligned_img = cv2.warpPerspective(imgTest, homography, (width, height))
# Resizing the image to display in our screen (optional)
aligned_img = cv2.resize(aligned_img, (width//3, height//3))

# Copy of input image
imgTest_cp = imgTest.copy()
imgTest_cp = cv2.resize(imgTest_cp, (width//3, height//3))
# Save the align image output.
# cv2.imwrite('output.jpg', aligned_img)

cv2.imshow('Input Image', imgTest_cp)
cv2.imshow('Output Image', aligned_img)
cv2.waitKey(0)

Output for all 3 input images shown above………

Conclusion

Image alignment has many applications in computer vision, such as object tracking. In this article, we have described how to do feature-based image alignment using OpenCV. We started by explaining the basic theory behind Image registration. We then continued by demonstrating the steps on an example where we aligned a mobile phone photo of the book cover (photo taken from 3 different angles) with a template image of that book cover.

Anindya

Hi there, I’m Anindya Naskar, Data Science Engineer. I created this website to show you what I believe is the best possible way to get your start in the field of Data Science.

Also Read:  Social distancing detector using OpenCV and Python