Joseph Long

Image transformation and interpolation from scratch

2025-06-18T00:00:00-07:00

Many astronomers, particularly in imaging-focused research areas like my own, have used image alignment code written by others without struggling through writing their own.

Sometimes, you really should just use the library code so you can publish in a timely fashion. Really, though, you should understand exactly what's going on. Otherwise, you may find yourself with pointed questions from your coauthors (or your referees) that you're unable to answer!

Fast image interpolation is a cornerstone of computer graphics. We take this for granted today, but being able to zoom, scale, translate, and rotate images requires doing operations on thousands of pixels at sixty frames-per-second for applications like image editors or animation tools.

Since it is so fundamental, this topic has already been explained many ways by many people. Now I'm adding my own version to the mix.

The preliminaries¶

First, we need the usual Python tools for numerical computing and plotting:

In [1]:

%config InlineBackend.figure_formats = ['retina']
%config InlineBackend.print_figure_kwargs = {'facecolor': (1.0, 1.0, 1.0, 0.0)}
import numpy as np
import matplotlib.pyplot as plt

Make the plots a little more science-y:

In [2]:

plt.rc('image', 
    # (0, 0) is in lower-left as conventional in science,
    # not upper-left as in computer graphics
    origin='lower',
    # we don't need matplotlib to make smooth images; we'll do that ourselves
    interpolation='nearest',
    cmap='Blues'
)
plt.rc('figure', figsize=(5, 3))

The first order of business is to get an image to interpolate:

In [3]:

def f_test(npix):
    '''Create a square npix x npix array of zeros and draw a capital F
    that is upright and facing right when plotted with (0,0) at lower left
    as regions of ones'''
    f_test = np.zeros((npix, npix))
    mid = npix // 2
    stem = (slice(mid//8, npix - mid//8), slice((mid - mid // 4) - mid//8, (mid - mid//4) + mid // 8))
    f_test[stem] = 1
    bottom = (slice(mid - mid//8, mid + mid//8), slice((mid - mid // 4) - mid//8, (mid - mid//4) + 2*mid//3))
    f_test[bottom] = 1
    top = (slice(npix - mid//8 - mid // 4, npix - mid//8), slice((mid - mid // 4) - mid//8, (mid - mid//4) + mid))
    f_test[top] = 1
    return f_test

That makes an image like this.

In [4]:

n_pix = 128
src = f_test(n_pix)
plt.imshow(src)

Out[4]:

<matplotlib.image.AxesImage at 0x1772d5940>

The "F" shape will make it clear when we flip the parity of the image, as well as showing which direction we're rotating when we rotate.

The simplest transformation¶

The simplest transformation is a full-pixel (integer) translation. Let's make an empty image...

In [5]:

dest = np.zeros((n_pix, n_pix))

...and fill it with the above pattern, shifted ten pixels to the right.

In [6]:

dx = 10
for dest_y in range(dest.shape[0]):
    for dest_x in range(dest.shape[1]):
        src_y, src_x = dest_y, dest_x - dx
        dest[dest_y, dest_x] = src[src_y, src_x]
plt.imshow(dest)

Out[6]:

<matplotlib.image.AxesImage at 0x17747b610>

(We're cheating a little and just using NumPy indexing behavior to handle out-of-bounds coordinates—i.e. a negative src_x. Moving ten pixels left wouldn't work.)

There are a few things that should stand out to you about the above code.

Thing 1: loops¶

The first is that we have loops: in fact, a nested loop, which implies $O(n^2)$ scaling in the dimension of the image. There's no way around this; we have to visit every pixel to set its value. However, Python users know loops are slow-ish.

Let's compare a simple nested loop that sums all the pixels in the src image. First, the Python implementation:

In [7]:

%%timeit
accum = 0
for j in range(n_pix):
    for i in range(n_pix):
        accum += src[j, i]

1.51 ms ± 18.2 μs per loop (mean ± std. dev. of 7 runs, 1,000 loops each)

Not so bad, unless you do it a few thousand times. But it could be much faster:

In [8]:

%timeit np.sum(src)

4.74 μs ± 42.9 ns per loop (mean ± std. dev. of 7 runs, 100,000 loops each)

NumPy is still looping over all the elements of the array, but it does so in optimized C code instead of Python. That's enough to get a three orders of magnitude speedup.

So, be warned: the interpolation code developed in this post is optimized for clarity, not speed. (Maybe in another blog post!)

Thing 2: loop over `dest`, not `src`¶

It was unintuitive (at least, to me) that the interpolation function to transform from some image src to some image dest involves looping over the pixels in dest. But, of course, the destination's pixel coordinates are fixed. We can't write a value into the 2.75'th pixel on an axis, just the 2nd or the 3rd pixel. So, we start with the destination pixel coordinates dest_x and dest_y and transform the coordinate to find the relevant part of the source image (that's the src_y, src_x = dest_y, dest_x - dx in the example above).

Coordinate transformations¶

The typical representation of coordinate transformations for 2D graphics is as an affine transformation matrix. For the coordinates dest_x, dest_y we define a coordinate vector $\vec{v} = \begin{bmatrix}x \\ y \\ 1\end{bmatrix}$. The extra $1$ entry makes this an "augmented" vector, meaning the transform matrix becomes a $3 \times 3$ matrix instead of $2 \times 2$.

Without this extra entry, the transformation matrix could not represent a translation of the original coordinates. To get the 2D coordinates after transforming, simply drop the last entry of the vector.

As a quick refresher, here are the affine transformation matrices. (See Wikipedia for more details, pictures, etc.)

Translation¶

$$ \begin{bmatrix}x + \Delta x \\ y + \Delta y \\ 1\end{bmatrix} = \begin{bmatrix} 1 & 0 & \Delta x\\ 0 & 1 & \Delta y \\ 0 & 0 & 1 \end{bmatrix} \begin{bmatrix}x \\ y \\ 1\end{bmatrix} $$

Scaling¶

$$ \begin{bmatrix}s_x x\\ s_y y\\ 1\end{bmatrix} = \begin{bmatrix} s_x & 0 & 0\\ 0 & s_y & 0\\ 0 & 0 & 1 \end{bmatrix} \begin{bmatrix}x \\ y \\ 1\end{bmatrix} $$

Note that a parity flip (i.e. reversing the X or Y direction) is just a special case of scaling with a scale factor < 0.

Rotation¶

$$ \begin{bmatrix}x'\\ y'\\ 1\end{bmatrix} = \begin{bmatrix} \cos \theta & -\sin \theta & 0\\ \sin \theta & \cos \theta & 0\\ 0 & 0 & 1 \end{bmatrix} \begin{bmatrix}x \\ y \\ 1\end{bmatrix} $$

Shearing¶

$$ \begin{bmatrix}x'\\ y'\\ 1\end{bmatrix} = \begin{bmatrix} 1 & c_x & 0\\ c_y & 1 & 0\\ 0 & 0 & 1 \end{bmatrix} \begin{bmatrix}x \\ y \\ 1\end{bmatrix} $$

Who's getting transformed, anyway?¶

To find the relevant part of src given dest_x, dest_y, we transform from the destination image's pixel coordinates back to source image pixel coordinates. That means we have to think carefully about what's moving.

Suppose you have a feature at $(1, 1)$ in the source image and you want to place it at $(2, 3)$ in the destination image (i.e. a displacement of $+1$ in $x$ and $+2$ in $y$). That means when you transform $(2, 3)$ you should get $(1, 1)$, which means the $\Delta x$ and $\Delta y$ in the example above should be negative:

$$ \begin{bmatrix} 1 & 0 & -1\\ 0 & 1 & -2\\ 0 & 0 & 1 \end{bmatrix} \begin{bmatrix}2 \\ 3 \\ 1\end{bmatrix} = \begin{bmatrix}2 - 1 \\ 3 - 2 \\ 1\end{bmatrix} =\begin{bmatrix}1 \\1\\ 1\end{bmatrix} $$

So, when constructing a transformation, you must always be clear about whether you're expressing the image's motion or the coordinate transformation "back" to the source image pixels.

Nearest-neighbor interpolation¶

Technically, interpolation just means estimating values for new points at which we have no data. Using the nearest point where we do have data is not very accurate, but for images with large regions of uniform values it does okay. It is also very easy to implement.

Taking our image transformation loop from above, we change the way src_x and src_y are computed:

In [9]:

dx = 10
transform_mtx = np.array([
    # n.b. transforming the coordinate so x' = x - 10, moving image in +x by 10
    [1, 0, -dx],
    [0, 1, 0],
    [0, 0, 1]
])
for dest_y in range(dest.shape[0]):
    for dest_x in range(dest.shape[1]):
        src_x, src_y, _ = transform_mtx @ [dest_x, dest_y, 1]
        dest[dest_y, dest_x] = src[src_y, src_x]
plt.imshow(dest)

Out[9]:

<matplotlib.image.AxesImage at 0x177611310>

So far, so good. But we're moving by whole pixels still, which is easy mode. What if we wanted to rotate by $\pi / 6$?

In [10]:

theta = np.pi / 6
transform_mtx = np.array([
    [np.cos(theta), -np.sin(theta), 0],
    [np.sin(theta), np.cos(theta), 0],
    [0, 0, 1]
])
try:
    for dest_y in range(dest.shape[0]):
        for dest_x in range(dest.shape[1]):
            src_x, src_y, _ = transform_mtx @ [dest_x, dest_y, 1]
            dest[dest_y, dest_x] = src[src_y, src_x]
    plt.imshow(dest)
except Exception as e:
    print(type(e).__name__, e)

IndexError only integers, slices (`:`), ellipsis (`...`), numpy.newaxis (`None`) and integer or boolean arrays are valid indices

That's because we're no longer using only whole integers, so this is where the rounding to nearest neighboring pixels comes in. Let's just round().

In [11]:

theta = np.pi / 6
transform_mtx = np.array([
    [np.cos(theta), -np.sin(theta), 0],
    [np.sin(theta), np.cos(theta), 0],
    [0, 0, 1]
])
try:
    for dest_y in range(dest.shape[0]):
        for dest_x in range(dest.shape[1]):
            src_x, src_y, _ = transform_mtx @ [dest_x, dest_y, 1]
            src_x, src_y = round(src_x), round(src_y)
            dest[dest_y, dest_x] = src[src_y, src_x]
    plt.imshow(dest)
except Exception as e:
    print(type(e).__name__, e)

IndexError index 128 is out of bounds for axis 0 with size 128

Still not good enough. What's happening is obvious, but what to do about it is subtle. Not only do we not have values at exactly the positions in the destination pixel grid, but we also are trying to interpolate values from positions that are more than a pixel outside the region covered by the source image.

In other words, there are pixels we can interpolate, and pixels we have to extrapolate.

To keep things simple and make sure we have a valid src_x and src_y for every destination pixel, we can just clamp the source coordinates to the range [0, n_pix - 1].

In [12]:

theta = np.pi / 6
transform_mtx = np.array([
    [np.cos(theta), -np.sin(theta), 0],
    [np.sin(theta), np.cos(theta), 0],
    [0, 0, 1]
])
try:
    for dest_y in range(dest.shape[0]):
        for dest_x in range(dest.shape[1]):
            src_x, src_y, _ = transform_mtx @ [dest_x, dest_y, 1]
            # round to integer pixels
            src_x, src_y = round(src_x), round(src_y)
            # clamp coordinates to range [0, npix - 1]
            src_x, src_y = min(max(src_x, 0), n_pix - 1), min(max(src_y, 0), n_pix - 1)
            dest[dest_y, dest_x] = src[src_y, src_x]
    plt.imshow(dest)
except Exception as e:
    print(type(e).__name__, e)

Well, that got us past the exceptions. But there's the matter of whether you want the image to rotate by $\pi / 6$, and which direction should be positive. Conventionally, rotation counterclockwise from the $+x$ towards the $+y$ axis is positive. So, again, we negate the value in the transformation matrix:

In [13]:

theta = np.pi / 6
transform_mtx = np.array([
    # n.b. rotating by -theta so image rotates +theta CCW
    [np.cos(-theta), -np.sin(-theta), 0],
    [np.sin(-theta), np.cos(-theta), 0],
    [0, 0, 1]
])

for dest_y in range(dest.shape[0]):
    for dest_x in range(dest.shape[1]):
        src_x, src_y, _ = transform_mtx @ [dest_x, dest_y, 1]
        # round to integer pixels
        src_x, src_y = round(src_x), round(src_y)
        # clamp coordinates to range [0, npix - 1]
        src_x, src_y = min(max(src_x, 0), n_pix - 1), min(max(src_y, 0), n_pix - 1)
        dest[dest_y, dest_x] = src[src_y, src_x]
plt.imshow(dest)

Out[13]:

<matplotlib.image.AxesImage at 0x1776d7110>

Better! Let's wrap that into a function.

In [14]:

def simple_image_xform(src, dest_shape, transform_mtx):
    dest = np.zeros(dest_shape)
    n_pix_y, n_pix_x = src.shape
    for dest_y in range(dest.shape[0]):
        for dest_x in range(dest.shape[1]):
            src_x, src_y, _ = transform_mtx @ [dest_x, dest_y, 1]
            # round to integer pixels
            src_x, src_y = round(src_x), round(src_y)
            # clamp coordinates to range [0, npix - 1]
            src_x, src_y = min(max(src_x, 0), n_pix_x - 1), min(max(src_y, 0), n_pix_y - 1)
            dest[dest_y, dest_x] = src[src_y, src_x]
    return dest

theta = np.pi / 6
transform_mtx = np.array([
    # n.b. rotating by -theta so image rotates +theta CCW
    [np.cos(-theta), -np.sin(-theta), 0],
    [np.sin(-theta), np.cos(-theta), 0],
    [0, 0, 1]
])
plt.imshow(simple_image_xform(src, src.shape, transform_mtx))

Out[14]:

<matplotlib.image.AxesImage at 0x17777c050>

Looks the same, which we expected. But, if you expected the F to rotate nicely about the center, you're probably disappointed. The explanation is hinted at by the ticks on the coordinate axes: the origin is pixel (0, 0), so we're pivoting about that.

For my applications, it's more useful to have a rotation about the center. Let's construct one. First, some helper functions so we don't have to write out the transform matrices by hand.

In [15]:

def rotate_image(theta_rad: float) -> np.ndarray:
    '''Transform matrix taking [x, y, 1] -> [x', y', 1] where (x', y') are
    rotated CW by theta so an image gets rotated CCW'''
    return np.array([
        [np.cos(-theta_rad), -np.sin(-theta_rad), 0],
        [np.sin(-theta_rad), np.cos(-theta_rad), 0],
        [0, 0, 1]
    ])

In [16]:

def translate_image(dx: float, dy: float) -> np.ndarray:
    '''Transform matrix taking [x, y, 1] -> [x - dx, y - dy, 1]
    so that an image feature at (x, y) is translated to (x + dx, y + dy)'''
    return np.array([
        [1, 0, -dx],
        [0, 1, -dy],
        [0, 0, 1]
    ])

In [17]:

def scale_image(sx: float, sy: float) -> np.ndarray:
    '''Transform matrix taking [x, y, 1] -> [x / sx, y / sy, 1]
    so that an image feature at (x, y) is scaled to (sx * x, sy * y)'''
    return np.array([
        [1/sx, 0, 0],
        [0, 1/sy, 0],
        [0, 0, 1]
    ])

In [18]:

def center_x_y(im_shape: tuple[int, int]) -> tuple[float, float]:
    '''Pixel coordinates for the center of `im` in (x, y) order'''
    return (im_shape[1] - 1) / 2, (im_shape[0] - 1) / 2

Now we can perform a rotation about the center as a combination of three steps: move center to origin, perform rotation about origin, move origin to image center.

Note that we want to support destination images that aren't the same dimensions as the source image, so we need the shapes of both the source and destination images. (The translation amounts are different for the centers of the source and the destination images.)

In [19]:

dest_shape = (2 * n_pix, 2 * n_pix)

Step 1: move the image's center to the origin¶

Equivalently, transform the pixel coordinates of the destination such that (0, 0) maps to the source image's center.

In [20]:

cx_src, cy_src = center_x_y(src.shape)

In [21]:

plt.imshow(simple_image_xform(src, dest_shape, translate_image(-cx_src, -cy_src)))

Out[21]:

<matplotlib.image.AxesImage at 0x1777c9090>

Step 2: apply rotation¶

Here we see the power of composable transformations through matrix products. By multiplying the affine transformation matrices together we can chain together image operations. However, note that the order matters, and we're transforming coordinates and not images. That means for our destination pixels to be a rotated view of the source, we de- rotate first, then translate.

In [22]:

transform_mtx = translate_image(-cx_src, -cy_src) @ rotate_image(np.pi / 6)
plt.imshow(simple_image_xform(src, dest_shape, transform_mtx))

Out[22]:

<matplotlib.image.AxesImage at 0x310029810>

Step 3: move the origin back to the center of the image¶

Since we have a dest_shape that's double the size of the source image, we translate to its center rather than simply undoing the original translation.

In [23]:

cx_dest, cy_dest = center_x_y(dest_shape)

In [24]:

transform_mtx = (
    translate_image(-cx_src, -cy_src) @
    rotate_image(np.pi / 6) @
    translate_image(cx_dest, cy_dest)
)
plt.imshow(simple_image_xform(src, dest_shape, transform_mtx))

Out[24]:

<matplotlib.image.AxesImage at 0x310055090>

Then, we can put this all together in a helper function.

In [25]:

def rotate_about_center(src_shape, dest_shape, theta_rad):
    '''Transform matrix taking [x, y, 1] -> [x', y', 1] where (x', y') are
    rotated CW by theta so an image gets rotated CCW about
    its center (rather than the origin)'''
    cx_src, cy_src = center_x_y(src_shape)
    cx_dest, cy_dest = center_x_y(dest_shape)
    return (translate_image(-cx_src, -cy_src) @
            rotate_image(theta_rad) @
            translate_image(cx_dest, cy_dest))

In [26]:

plt.imshow(simple_image_xform(
    src, dest_shape,
    rotate_about_center(src.shape, dest_shape, np.pi / 6)
))

Out[26]:

<matplotlib.image.AxesImage at 0x310089950>

The same applies to scaling, so let's make a function that does both

In [27]:

def rotate_and_scale_about_center(src_shape, dest_shape, sx=1.0, sy=1.0, theta_rad=0.0):
    '''Transform matrix taking [x, y, 1] -> [x', y', 1]
    so that an image is scaled from its center (rather than the origin) and rotated CCW by theta'''
    cx_src, cy_src = center_x_y(src_shape)
    cx_dest, cy_dest = center_x_y(dest_shape)
    return (
        translate_image(-cx_src, -cy_src) @ 
        rotate_image(theta_rad) @ 
        scale_image(sx, sy) @ 
        translate_image(cx_dest, cy_dest)
    )
plt.imshow(simple_image_xform(
    src, dest_shape,
    rotate_and_scale_about_center(src.shape, dest_shape, sx=0.5, sy=0.5, theta_rad=np.pi/4)
))

Out[27]:

<matplotlib.image.AxesImage at 0x3100f60d0>

Using the destination shape argument we can upscale images to a higher resolution:

In [28]:

plt.imshow(simple_image_xform(
    src, dest_shape,
    rotate_and_scale_about_center(src.shape, dest_shape, sx=2, sy=2, theta_rad=np.pi/4)
))

Out[28]:

<matplotlib.image.AxesImage at 0x31014a850>

Bilinear: more sophisticated interpolation¶

I said above that nearest-neighbor interpolation works best when there are large regions of the same value, which is certainly true of the F test image. Let's see a less-cooperative example. Here I have plotted $f(r) = \sin(5 \pi r)$.

In [29]:

ex_n_pix = 16
ex = np.zeros((ex_n_pix, ex_n_pix))
xx, yy = np.meshgrid(np.linspace(-1, 1, num=ex_n_pix), np.linspace(-1, 1, num=ex_n_pix))
rr = np.hypot(yy, xx)
ex = np.sin(5 * np.pi * rr)
plt.imshow(ex)

Out[29]:

<matplotlib.image.AxesImage at 0x3101aee90>

The underlying function is clearly radially symmetric, so rotating it shouldn't change it--except that we've discretized it on a grid of pixels.

For fun, let's rotate it through $2\pi$ in angle and see if it differs from the input.

In [30]:

from matplotlib.animation import FuncAnimation
from IPython.display import HTML

# Create a figure and axis
fig, axs = plt.subplots(ncols=3, figsize=(6, 1.5), layout='constrained')
# Use three panels to show original, transformed, and the difference
axs[0].imshow(ex)
output = axs[1].imshow(ex)
diff = axs[2].imshow(np.zeros_like(ex), vmin=-1, vmax=1, cmap='RdBu_r')
fig.colorbar(diff)

frames_per_2pi = 32
interval_ms = 200

# Rotate by 1 / frames_per_2pi of a turn per frame
def update(frame):
    transform_mtx = rotate_and_scale_about_center(
        ex.shape, (ex_n_pix, ex_n_pix),
        1, 1, np.pi * 2 * frame / frames_per_2pi
    )
    out_image = simple_image_xform(ex, (ex_n_pix, ex_n_pix), transform_mtx)
    output.set_data(out_image)
    diff.set_data(out_image - ex)
    return [output, diff]

# Create the animation
ani = FuncAnimation(fig, update, frames=frames_per_2pi, interval=interval_ms, blit=True)

# Embed animation in notebook,
plt.close(fig)
HTML(ani.to_jshtml())

Out[30]:

Well, that's unsastisfactory. We need to level up our interpolation function.

The simplest nontrivial interpolation function is bilinear, which takes the values at the four nearest pixels and interpolates three times.

Let us suppose (without loss of generality) there are only four pixels in the image, and we're trying to interpolate a value in between them. Then for the hypothetical continuous function $f(x, y)$ we only have known values for $f(0, 0)$, $f(0, 1)$, $f(1, 0)$, and $f(1, 1)$. We are looking for an approximate value for $f(\Delta x, \Delta y)$, where $0 \le \Delta x \le 1$ and $0 \le \Delta y \le 1$.

In the figure below, the known values at the source image pixel locations are shown by the red circular markers. Using the pair of points $f(0, 0)$ and $f(0, 1)$ to define a line, we can interpolate an approximate value for $f(\Delta x, 0)$ by assuming it lies on this line.

Linearly interpolating gives $f(\Delta x, 0) \approx f(0, 0) (1 - \Delta x) + f(1, 0)(\Delta x)$, the value at the lower square marker.

Repeating the procedure with $f(1, 0)$ and $f(1, 1)$ gives us an approximation for $f(\Delta x, 1) \approx f(1, 0) (1 - \Delta x) + f(1, 1)(\Delta x)$, shown by the upper square marker.

Finally, we interpolate between $f(\Delta x, 0)$ and $f(\Delta x, \Delta y)$ to get the interpolated value for our destination pixel: \begin{align*} f(\Delta x, \Delta y) \approx&\ f(\Delta x, 0) (1 - \Delta y) + f(\Delta x, 1)(\Delta y)\\ =&\ (f(0, 0) (1 - \Delta x) + f(1, 0)(\Delta x)) (1 - \Delta y) + (f(1, 0) (1 - \Delta x) + f(1, 1)(\Delta x))(\Delta y)\\ =&\ f(0, 0) (1 - \Delta x) (1 - \Delta y)\\ &\ +\ f(1, 0) (\Delta x) (1 - \Delta y)\\ &\ +\ f(1, 0) (1 - \Delta x) (\Delta y)\\ &\ +\ f(1, 1)(\Delta x)(\Delta y) \end{align*}

We only work this out for the case where we're in the first pixel in the grid because we can always choose our coordinates such that we are there. Let's make a function to do this calculation, cutting out the $2 \times 2$ pixels we care about from the source image.

In [31]:

def bilinear_patch(src, x, y):
    max_y = src.shape[0] - 1
    max_x = src.shape[1] - 1
    int_x, int_y = int(x), int(y)
    # if the integer part == the max, this is an edge pixel
    # and interpolation will need pixels beyond the source image edge
    if int_x >= max_x or int_y >= max_y:
        return np.nan
    # if the integer part < 0, this is an edge pixel
    if int_x < 0 or int_y < 0:
        return np.nan
    # otherwise, dx and dy account for the fractional part of the pixel coord
    dx, dy = x - int(x), y - int(y)
    # the relevant pixels are the four corners of the box containing (x, y)
    patch = src[int_y:int_y + 2,int_x:int_x + 2]
    return (
        patch[0, 0] * (1 - dx) * (1 - dy) +
        patch[0, 1] * dx * (1 - dy) +
        patch[1, 0] * (1 - dx) * dy +
        patch[1, 1] * dx * dy
    )

The comments point out the places where we handle out-of-bounds coordinates by returning not-a-number (a.k.a. NaN or np.nan).

Updating the simple_image_xform function to use bilinear_patch to find the destination pixel values, we have bilinear_image_xform:

In [32]:

def bilinear_image_xform(src, dest_shape, transform_mtx):
    dest = np.zeros(dest_shape)
    for dest_y in range(dest.shape[0]):
        for dest_x in range(dest.shape[1]):
            src_x, src_y, _ = transform_mtx @ [dest_x, dest_y, 1]
            dest[dest_y, dest_x] = bilinear_patch(src, src_x, src_y)
    return dest

In [33]:

plt.imshow(bilinear_image_xform(src, dest_shape, rotate_and_scale_about_center(src.shape, dest_shape, 1.2, 1.2, np.pi / 4)))

Out[33]:

<matplotlib.image.AxesImage at 0x31036fed0>

In [34]:

fig, axs = plt.subplots(ncols=3, figsize=(6, 1.5), layout='constrained')
# Use three panels to show original, transformed, and the difference
axs[0].imshow(ex, vmin=-1, vmax=1)
output = axs[1].imshow(ex, vmin=-1, vmax=1)
diff = axs[2].imshow(np.zeros_like(ex), vmin=-1, vmax=1, cmap='RdBu_r')
fig.colorbar(diff)

# Rotate by 1 / frames_per_2pi of a turn per frame
def update(frame):
    transform_mtx = rotate_and_scale_about_center(
        ex.shape, (ex_n_pix, ex_n_pix),
        1, 1, np.pi * 2 * frame / frames_per_2pi
    )
    out_image = bilinear_image_xform(ex, (ex_n_pix, ex_n_pix), transform_mtx)
    output.set_data(out_image)
    diff.set_data(out_image - ex)
    return [output, diff]

# Create the animation
ani = FuncAnimation(fig, update, frames=frames_per_2pi, interval=500, blit=True)

# Embed animation in notebook
plt.close(fig)
ani.save('./anim.mp4')
HTML(ani.to_jshtml())

Out[34]:

Two things (again) to notice here:

Improved residuals¶

We can tell we're doing better because the residuals (rightmost panel) are smaller on average than in the nearest-neighbor example above. So, even though the image looks different up close, we're getting closer to the ideal (i.e. rotational symmetry).

Missing pixels¶

From the very first frame, there is one row and one column that are white (i.e. missing). The destination pixel no longer depends on just one source pixel; we need a little domain of four pixels.

On frame zero, we're rotating by 0º, which shouldn't change anything. However, for an $N \times N$ image, the corner pixel at $(N - 1, N - 1)$ will depend on values in pixels $(N - 1, N)$, $(N, N - 1)$, and $(N, N)$. Of course, those values are unknown (or unknowable), so the result of the interpolation is undefined.

There are various reasonable workarounds, like checking if the transformation maps to an integer coordinate in the source image and skipping the whole interpolation step. Another option is to replace the missing values in the bilinear_patch calculation with the nearest known values. Both are justifiable, but you should know which one you're doing so that you don't get confused by interpolation artifacts.

Bicubic interpolation or cubic convolution: best speed/accuracy tradeoff¶

Bilinear interpolation assumes the underlying continuous function sampled on our pixel grid increases or decreases only linearly in the space between pixels. Of course, we know plenty of functions without constant first derivatives. Can we capture some of this higher order behavior? The answer is yes, at the cost of some complexity and additional arithmetic operations.

With a carefully chosen kernel function, a $4 \times 4$ domain of pixels can contribute to a cubic curve that connects two of the pixels. (This is not quite spline interpolation, but a good approximation that doesn't require derivative information for the underlying function.)

To implement bicubic interpolation let's start with an example image. The mini_img below is a $4 \times 4$ image covering the region $-1 \le x \le 2$ and $-1 \le y \le 2$.

(Nearest-neighbor uses one pixel, bilinear uses $2 \times 2$ pixels, and bicubic uses $4 \times 4$. Why doesn't anyone use $3 \times 3$, for biquadratic? Well... that's kind of complicated.)

In [35]:

from matplotlib import colors
cmap = plt.cm.rainbow.copy()
norm = colors.Normalize(vmin=1, vmax=4)
mini_img = np.array([
    [2, 3, 1, 4],
    [4, 1, 3, 2],
    [3, 4, 2, 1],
    [2, 1, 1, 3],
])
pixel_colors2 = cmap(norm(mini_img.flat)).reshape(2, 2, -1)
plt.imshow(mini_img, cmap=cmap, origin='lower', extent=[-1.5, 2.5, -1.5, 2.5])
plt.colorbar()

Out[35]:

<matplotlib.colorbar.Colorbar at 0x31164a0d0>

If we use the 1D cubic convolution kernel to interpolate values between columns, we get four continuous curves representing $f(x, -1)$, $f(x, 0)$, $f(x, 1)$, $f(x, 2)$, one for each row.

As above with bilinear interpolation, we are assuming (without loss of generality) that we are interpolating coordinates in the domain $0 < \Delta x < 1$ and $0 < \Delta y < 1$. Evaluating these approximations at $x = \Delta x$ for each gives us four new values, $f(\Delta x, -1)$, $f(\Delta y, 0)$, $f(\Delta y, 1)$, $f(\Delta y, 2)$.

Plugging these into the same 1D interpolation function allows us to approximate $f(\Delta x, \Delta y)$.

This (somewhat complicated) figure shows what's going on.

First, each row is interpolated at $\Delta x$, shown as smaller circular points between 0 and 1. The curve connecting $y = 0$ to $y = 1$ is then evaluated at $y = \Delta y$ to get an approximate value for $f(\Delta x, \Delta y)$.

What's the magic kernel function, then? It was derived by one Robert G. Keys in a 1981 publication ("Cubic convolution interpolation for digital image processing", IEEE Transactions on Acoustics, Speech, and Signal Processing). It's not even that hard to follow if you have some knowledge of Taylor series and algebra.

(Unfortunately, as of this writing IEEE is still trying to charge $33.00 for access to a paper that's more than four decades old. So you'll have to find it on your own.)

The interpolation kernel is

$$ u(s) = \begin{cases} \frac{3}{2} |s|^3 - \frac{5}{2} |s|^2 + 1 & 0 < |s| < 1\\ -\frac{1}{2} |s|^3 + \frac{5}{2} |s|^2 - 4 |s| + 2 & 1 < |s| < 2\\ 0 & s < |s| \end{cases} $$

which, with some algebraic manipulation given in the paper, can produce the interpolation function $g(x) \approx f(x)$ which can be evaluated to approximate $f(x)$ between samples. The interpolation function is

\begin{align*} g(x) =&\ c_{k - 1} (- s^3 + 2 s^2 - s)/2 + c_k (3s^3 - 5s^2 + 2)/2\\ &+ c_{k + 1} (-3 s^3 + 4s^2 + s)/2 + c_{k + 2}(s^3 - s^2)/2 \end{align*}

where $s = (x - x_k) / h$, $h$ is the sampling interval, and $c_k = f(x_k)$ for $k = 0, 1, 2, ..., N$ are the known sample values at the coordinates $x_k$.

We've set up the coordinates above to run from -1 to 2, and we're using pixel coordinates so $h = 1$ and can be neglected. Therefore $s = \Delta x$ and $c_{k - 1} = f(-1)$, $c_{k} = f(0)$, and so on.

(Note: Here we're making a slight abuse of notation, since we're interpolating in 1D: $f(x)$ is really $f(x, y)$ for some sampled y coordinate.)

The simplified expression for our chosen coordinates becomes

\begin{align*} g(x) = &f(-1) \frac{-(\Delta x)^3 + 2(\Delta x)^2 - (\Delta x)}{2}\\ &+ f(0) \frac{3(\Delta x)^3 - 5(\Delta x)^2 + 2}{2}\\ &+ f(1) \frac{-3(\Delta x)^3 + 4(\Delta x)^2 + \Delta x}{2}\\ &+ f(2) \frac{(\Delta x)^3 - (\Delta x)^2}{2} \end{align*}

We then factor out the 1/2 and group terms by powers of $\Delta x$:

\begin{align*} g(x) = \frac{1}{2} [ &\left[-f(-1) + 3f(0) - 3f(1) + f(2)\right] (\Delta x)^3 \\ &+ \left[ 2f(-1) - 5f(0) + 4f(1) - f(2)\right] (\Delta x)^2 \\ &+ \left[ -f(-1) + f(1) \right] (\Delta x)\\ &+ 2f(0) ] \end{align*}

which we can translate to Python code:

In [36]:

def cubic1d(t, f_minus1, f_0, f_1, f_2):
    a = 2 * f_0
    b = -1 * f_minus1 + f_1
    c = 2 * f_minus1 - 5 * f_0 + 4 * f_1 - f_2
    d = -1 * f_minus1 + 3 * f_0 - 3 * f_1 + f_2
    return 0.5 * (a + t * b + t ** 2 * c + t ** 3 * d)

Now let's interpolate $f(\Delta x, \Delta y)$ in the image above for some values of $\Delta x$ and $\Delta y$.

In [37]:

delta_x, delta_y = 0.5, 0.5

The interpolation curve from cubic1d should pass through $f(0, y)$ and $f(1, y)$ when we evaluate it on the interval $0 < x < 1$.

In [38]:

coords = np.linspace(0, 1)

fig, ax = plt.subplots()
markers, _, _ = ax.stem(np.arange(-1, 3), mini_img[0])
markers.set_markersize(8)
markers.set_markeredgecolor('k')
# interpolate along 0 < y < 1
interp_curve = cubic1d(
    coords,
    mini_img[0, 0], mini_img[0, 1], mini_img[0, 2], mini_img[0, 3]
)
ax.plot(coords, interp_curve, c='C1')
# interpolate just at ∆x
val = cubic1d(
    delta_x,
    mini_img[0, 0], mini_img[0, 1], mini_img[0, 2], mini_img[0, 3]
)
ax.scatter(delta_x, val, s=100, marker='.', fc='C1', ec='none')
ax.set(xlabel='x', xlim=(-1.5, 2.5), ylim=(0, 5))

Out[38]:

[Text(0.5, 0, 'x'), (-1.5, 2.5), (0.0, 5.0)]

This shows the interpolated value for $f(\Delta x, 0)$ as an orange dot on the curve. (Why not draw a curve through all three pairs? Well, you need a domain of $4 \times 4$ pixels, so we'd need one more pixel on the "low" side and two more on the "high" side.)

To make this 2D we repeat for every row of the image:

In [39]:

interpolated_values = []
fig, ax = plt.subplots(figsize=(5, 3))

for i in range(4):
    offset = i / 10
    markers, _, _ = ax.stem(
        np.arange(-1, 3) + offset, mini_img[i],
        markerfmt=f'C{i}', linefmt=f'C{i}'
    )
    markers.set_markersize(8)
    markers.set_markeredgecolor('k')
    # interpolate along 0 < y < 1
    interp_curve = cubic1d(
        coords,
        mini_img[i, 0], mini_img[i, 1], mini_img[i, 2], mini_img[i, 3]
    )
    ax.plot(coords + offset, interp_curve, c=f'C{i}')
    # interpolate just at ∆x
    val = cubic1d(
        delta_x,
        mini_img[i, 0], mini_img[i, 1], mini_img[i, 2], mini_img[i, 3]
    )
    interpolated_values.append(val)
    ax.scatter(delta_x + offset, val, s=100, marker='.', ec='none', fc=f'C{i}')

ax.set(xlabel='x', xlim=(-1.5, 2.5), ylim=(0, 5))

Out[39]:

[Text(0.5, 0, 'x'), (-1.5, 2.5), (0.0, 5.0)]

Above, we've plotted the values from the different rows offset slightly for clarity.

Taking the interpolated values for $f(\Delta x, y)$ for $y = -1, 0, 1, 2$ (shown by the point markers on the interpolated curves above) as a new 1D sequence, we interpolate again:

In [40]:

fig, ax = plt.subplots()
for v, y in zip(interpolated_values, np.arange(-1, 3)):
    ax.scatter(v, y, marker='.', s=100)
    ax.plot([0, v], [y, y], ls=':')

ax.plot(cubic1d(np.linspace(0, 1), *interpolated_values), coords)

val = cubic1d(delta_y, *interpolated_values)
ax.scatter(val, delta_y, s=100, marker='.', fc='k')

ax.set(ylabel='y', ylim=(-1.5, 2.5), xlim=(0, 5))

Out[40]:

[Text(0, 0.5, 'y'), (-1.5, 2.5), (0.0, 5.0)]

This interpolation along a second direction is the "bi" in bicubic. We can write it up as a function that assumes a 4 x 4 pixel subimage to operate on.

In [41]:

def bicubic(dx, dy, region):
    # Perform 4 1D interpolations by dx along the rows of region
    b_minus1 = cubic1d(dx, region[0, 0], region[0, 1], region[0, 2], region[0, 3])
    b_0 = cubic1d(dx, region[1, 0], region[1, 1], region[1, 2], region[1, 3])
    b_1 = cubic1d(dx, region[2, 0], region[2, 1], region[2, 2], region[2, 3])
    b_2 = cubic1d(dx, region[3, 0], region[3, 1], region[3, 2], region[3, 3])
    # perform 1 interpolation by dy along the column of b values
    interpolated_value = cubic1d(dy, b_minus1, b_0, b_1, b_2)
    return interpolated_value

The interpolated $f(\Delta x, \Delta y)$ should agree, and it does:

In [42]:

fig, ax = plt.subplots(figsize=(5, 3))
# ax.stem(np.arange(-1, 3), interpolated_values, orientation='horizontal')
for v, y in zip(interpolated_values, np.arange(-1, 3)):
    ax.scatter(v, y, marker='.', s=100)
    ax.plot([0, v], [y, y], ls=":")

ax.plot(cubic1d(np.linspace(0, 1), *interpolated_values), coords)

ax.scatter(bicubic(delta_x, delta_y, mini_img), delta_y, s=100, marker='.', fc='k')

ax.set(ylabel='y', ylim=(-1.5, 2.5), xlim=(0, 5))

Out[42]:

[Text(0, 0.5, 'y'), (-1.5, 2.5), (0.0, 5.0)]

Now, we just have to do this for every pixel in the destination image. Easy!

Since it requires a $4 \times 4$ pixel domain to interpolate from, I've restructured the code to construct a cutout image with a fill value for missing pixels (i.e. on the edges).

In [43]:

def get_or_fill(image, x, y, fill_value):
    # get image[y, x] replacing out-of-bounds pixels with `fill_value`
    if y < 0 or y >= image.shape[0]:
        return fill_value
    if x < 0 or x >= image.shape[1]:
        return fill_value
    return image[y, x]

def cutout_for_bicubic(image, src_x, src_y, fill_value):
    # construct a subframe of 4 x 4 pixels
    cutout = np.zeros((4, 4))
    for y, row in enumerate(range(-1, 3)):
        for x, col in enumerate(range(-1, 3)):
            cutout[y, x] = get_or_fill(image, src_x + col, src_y + row, fill_value)
    return cutout

def bicubic_image_xform(src, dest_shape, transform_mtx, fill_value=0.0):
    dest = np.zeros(dest_shape)
    for dest_y in range(dest.shape[0]):
        for dest_x in range(dest.shape[1]):
            src_x, src_y, _ = transform_mtx @ [dest_x, dest_y, 1]
            int_x, int_y = int(np.floor(src_x)), int(np.floor(src_y))
            region = cutout_for_bicubic(src, int_x, int_y, fill_value)
            dx, dy = src_x - int_x, src_y - int_y
            dest[dest_y, dest_x] = bicubic(dx, dy, region)
    return dest

Let's see the effect of different fill values on the resulting image. Here I'm rotating by 45º with zeros and with np.nans for out-of-bounds points.

In [44]:

xform = rotate_and_scale_about_center(
    ex.shape, (ex_n_pix, ex_n_pix),
    1, 1, np.deg2rad(45)
)
fig, axs = plt.subplots(ncols=2)
axs[0].set_title('zeros')
out_image = bicubic_image_xform(ex, (ex_n_pix, ex_n_pix), xform, fill_value=0.0)
axs[0].imshow(out_image, vmin=-1, vmax=1)

axs[1].set_title('NaNs')
out_image = bicubic_image_xform(ex, (ex_n_pix, ex_n_pix), xform, fill_value=np.nan)
axs[1].imshow(out_image, vmin=-1, vmax=1)

Out[44]:

<matplotlib.image.AxesImage at 0x3121b65d0>

Arguably, the interpolated values at the edges on the left are just nonsense and there for aesthetic reasons. The missing value treatment least likely to mislead you is to treat every computation that takes an undefined value as producing an undefined value, which is exactly what a fill value of np.nan does.

Let's see how it does with the radially symmetric signal.

In [45]:

fig, axs = plt.subplots(ncols=3, figsize=(6, 1.5), layout='constrained')
# Use three panels to show original, transformed, and the difference
axs[0].imshow(ex, vmin=-1, vmax=1)
output = axs[1].imshow(ex, vmin=-1, vmax=1)
diff = axs[2].imshow(np.zeros_like(ex), vmin=-1, vmax=1, cmap='RdBu_r')
fig.colorbar(diff)

# Rotate by 1 / frames_per_2pi of a turn per frame
def update(frame):
    transform_mtx = rotate_and_scale_about_center(
        ex.shape, (ex_n_pix, ex_n_pix),
        1, 1, np.pi * 2 * frame / frames_per_2pi
    )
    out_image = bicubic_image_xform(ex, (ex_n_pix, ex_n_pix), transform_mtx, np.nan)
    output.set_data(out_image)
    diff.set_data(out_image - ex)
    return [output, diff]

# Create the animation
ani = FuncAnimation(fig, update, frames=frames_per_2pi, interval=interval_ms, blit=True)

# Embed animation in notebook
plt.close(fig)
HTML(ani.to_jshtml())

Out[45]:

Not bad! Returning to the "F" test image for a moment, let's investigte the bicubic interpolator's behavior around sharp edges. Here we translate the test pattern by a noninteger number of pixels (to make the sub-pixel behavior of the interpolator visible).

Let's zoom in on the base of the F and plot a slice through the image.

In [46]:

result = bicubic_image_xform(src, src.shape, translate_image(10.25, 12.5))

# define detail region
row_slice = 40
col_center = 58
detail_size = 25

fig, (ax1, ax2, ax3) = plt.subplots(1, 3, figsize=(8,2))

# Show translation result
ax1.imshow(result)
# Show detail
ax2.imshow(result)
ax2.axhline(row_slice)
ax2.set(
    xlim=(col_center - detail_size/2, col_center + detail_size/2),
    ylim=(row_slice - detail_size/2, row_slice + detail_size/2)
)
# Show stem plot of slice through image
ax3.stem(np.arange(result.shape[1]), result[row_slice])
ax3.set_xlim(col_center - detail_size/2, col_center + detail_size/2)
ax3.axhline(1.0, linestyle=':')
ax3.axhline(0.0, linestyle=':')

Out[46]:

<matplotlib.lines.Line2D at 0x3125442d0>

We put in ones and zeros in the test pattern, so why have we got values below zero and above one now? In the test pattern, the the edge of the "F" in the test image is a discontinuity; from one pixel to the next the value jumps from 0 to 1. Intuitively, we think that the edge should be "infinitely sharp", so the result of translation should still transition from zero to one in the space of a single pixel.

Why is this so different from our intuition? Is this "right"?

Oversampling, undersampling, and band limits¶

This is "ringing", basically the same thing as Gibbs phenomenon. It all boils down to what you're capturing in your pixels. If your image contains infinitely sharp edges, like our "F" example, you need an infinite number of pixels to express that in a pixelated / sampled format. So, no, the image above is not "right"; this is why graphic designers use a completely different image representation to work in. In practice, we have given the computer a number of samples, representing a signal up to some band-limit that corresponds to how finely sampled (i.e. sharp) it is.

Fortunately, if you're taking pictures of natural phenomena, there are no infinitely sharp edges. When considering the detection of exoplanet signals, we are dealing with a more constrained version of the image interpolation problem than, e.g., Photoshop does. In high-contrast imaging, our science images are always oversampled relative to the band-limit of the telescope system. That means that we can be confident that our interpolation and resampling is preserving information (at least, to a good approximation).

Consider a counter-example: an undersampled image of a star might look like this:

In [47]:

star = np.array([
    [0, 0, 0, 0, 0],
    [0, 0, 0, 0, 0],
    [0, 0, 1, 0, 0],
    [0, 0, 0, 0, 0],
    [0, 0, 0, 0, 0],
])
plt.imshow(star, vmin=0, vmax=1)
plt.scatter(2, 2, marker='*', c='w')
plt.colorbar()

Out[47]:

<matplotlib.colorbar.Colorbar at 0x3125f6fd0>

Here I've marked the star location at $(2, 2)$. It might really be at $(2.25, 2)$, but it's such a tiny feature it gets summed up in the pixel ranging from $1.5 < x < 2.5$ and $1.5 < y < 2.5$.

What if we translate the image 0.5 pixels right, using our interpolator from above?

In [48]:

plt.imshow(
    bicubic_image_xform(star, star.shape, translate_image(0.5, 0)),
    vmin=0, vmax=1
)
plt.scatter(2.5, 2, marker='*', c='w')
plt.colorbar()

Out[48]:

<matplotlib.colorbar.Colorbar at 0x3126add10>

If your model for the "star" in pixel $(2, 2)$ is that it's illuminating one pixel-sized square uniformly from its center, shifting the square by half a pixel puts the star on the border between pixels at $(2.5, 2)$, so it is quite reasonable to put half the light in the adjacent pixel. On the other hand, if your star is really at $(2.25, 2.25)$, the star shifts to $(2.75, 2.25)$ so more than half of the light should be there.

Given this transformed image, you would also be unable to tell if it is one star of brightness 1.0 on a pixel boundary, or two stars of brightness 0.5 that happen to be next to each other.

In an undersampled image, there's no way to know. In other words, there is no way to rotate or translate the undersampled image without introducing potentially major changes in it.

Though it's not an isolated point like a star, the hard edge of the "F" pattern is an example of one of these undersampled features.

Oversampled scene¶

The most extreme case for an undersampled scene is a single pixel, but what model should we use for an oversampled scene? You won't go far wrong with a 2D Gaussian; even extreme adaptive optics PSFs are nearly Gaussian in their core.

Let's construct a 16 x 16 pixel scene with a Gaussian at the center. To compare with the interpolated image we will also generate one where the Gaussian has been displaced by dx = 2.5 pixels. Here we're generating two "truth" images based on the true analytical expression for $f(x, y)$ (implemented as gauss2d).

In [49]:

samples = 17
xs = np.arange(samples)
xx, yy = np.meshgrid(xs, xs)
rr = np.hypot(yy, xx)
sigma = 1.5

def gauss2d(x, y, x0, y0, amp, sigma_x, sigma_y):
    return amp * np.exp(-(
            ((x - x0)**2 / (2 * sigma_x**2))
            +
            ((y - y0)**2 / (2 * sigma_y**2))

    ))

# place Gaussian at array center
xc = yc = (samples - 1) / 2
sigma = 1.5
dx = 2.5

fig, axs = plt.subplots(ncols=2, figsize=(6, 2))
gaussian_before = gauss2d(xx, yy, xc, yc, 1, sigma, sigma)
fig.colorbar(axs[0].imshow(gaussian_before, vmax=1))
axs[0].scatter(xc, yc, marker='*', c='w')
axs[0].set(title='before')

# show scene translated by dx
gaussian_translated = gauss2d(xx, yy, xc + dx, yc, 1, sigma, sigma)
fig.colorbar(axs[1].imshow(gaussian_translated, vmax=1))
axs[1].scatter(xc + dx, yc, marker='*', c='w')
axs[1].set(title=f'translated by {dx=}')

Out[49]:

[Text(0.5, 1.0, 'translated by dx=2.5')]

Now we attempt to perform the translation by interpolation, using the "before" image.

In [50]:

np.sum(gaussian_before)

Out[50]:

np.float64(14.137166708707573)

In [51]:

gaussian_translated_interp = bicubic_image_xform(
    gaussian_before, gaussian_before.shape,
    translate_image(dx, 0)
)
plt.imshow(gaussian_translated_interp, vmin=0, vmax=1)
plt.colorbar()

Out[51]:

<matplotlib.colorbar.Colorbar at 0x3128ae210>

That is close enough to the "truth" image that it's probably easier to make sense of a plot showing their difference.

In [52]:

plt.imshow(
    (gaussian_translated_interp - gaussian_translated),
    cmap='RdBu_r', vmin=-0.05, vmax=0.05
)
plt.colorbar()

Out[52]:

<matplotlib.colorbar.Colorbar at 0x312969450>

Here you can see that the bicubic interpolation is not perfect. It is not possible in general to perfectly describe a Gaussian curve with a 3rd order polynomial, so you can see deviations where the cubic approximation is insufficient.

The right way to interpolate¶

Hopefully, the above has demonstrated that there is no one "right way" to resample images. Every method comes with trade-offs in speed and accuracy, and there are good ones I didn't cover (because this post is long enough). For a true band-limited signal, for example, using a zero-padded Fourier transform upsamples without loss of information (except for floating-point roundoff error). In practice, features like hot pixels and cosmic rays will always be "undersampled", and masking bad pixels will introduce ringing at the mask edges, so I don't use that method very often. Maybe I'll write that one up later.

The implementations above are intended to illuminate the way the algorithm works, but are not optimized for speed. Using someone else's optimized implementation is preferable if you don't have the time to build your own, but you should be sure to understand the accuracy and tradeoffs made by the implementation you use. For examples of the possible pitfalls, see the Bicubic bake-off post.

Bicubic bake-off: comparing scikit-image, OpenCV, scipy.ndimage, and scipy.interpolate

2025-01-07T00:00:00-07:00

While working on a post about implementing interpolation from scratch, I wanted to check my 2D bicubic interpolation against some existing libraries. Surprisingly, they don't agree!

I haven't gotten to the bottom of why, but I thought it worthwhile to document with an example. Let's import the tools we need:

In [1]:

%config InlineBackend.figure_formats = ['retina']
%config InlineBackend.print_figure_kwargs = {'facecolor': (1.0, 1.0, 1.0, 0.0)}
import numpy as np
import matplotlib.pyplot as plt

plt.rc('image', cmap='PuOr', origin='lower')
plt.rc('figure', figsize=(5, 2))

import doodads as dd  # my personal utility library, used only for plotting

Let's construct a signal that should interpolate fairly well. We're going to use sinusoids well within the band-limit of the image, so it's theoretically possible to interpolate and upsample without introducing aliasing*.

* Under some assumptions like infinite periodicity and using sinc interpolation... we're not doing that. But we should get close!

In [2]:

def f(xx, yy):
    phase = np.pi/6
    return 2 * np.cos(xx + phase) * np.cos(yy / 2) * np.cos(yy)

samples = 10
xmin, xmax = 0, samples - 1
dx = 1
img_extent = [xmin - dx/2, xmax + dx/2, xmin - dx/2, xmax + dx/2]
coords = np.arange(samples)
xx, yy = np.meshgrid(coords, coords)

img = f(xx, yy)
plt.imshow(img, extent=img_extent)

Out[2]:

<matplotlib.image.AxesImage at 0x3226efd30>

Here we've evaluated $f(x, y) = \cos(x / 2) \cos (y / 2) \cos(y)$ at the centers of pixels $0 \le x \le 9$ and $0 \le y \le 9$. (Note the extent= argument to imshow making the tick marks land in the right spot, taking into account the pixel extent from $x - 0.5$ to $x + 0.5$.)

Since it's analytic, we can make a "truth" signal that's sampled 100x more finely in $x$ and $y$.

In [3]:

upsample = 100
upsample_coords = np.linspace(-dx/2, samples - dx / 2, upsample * samples)
upsample_coords = np.arange(upsample * samples) / upsample - 0.5  # subtract half a pixel to account for finite size

Here we plot the coarsely-sampled signal on the left and the more finely sampled "truth" signal on the right.

In [4]:

xx, yy = np.meshgrid(upsample_coords, upsample_coords)
upsampled_truth = f(xx, yy)
plt.subplot(121)
plt.imshow(img)
plt.subplot(122)
plt.imshow(upsampled_truth, extent=img_extent)

Out[4]:

<matplotlib.image.AxesImage at 0x3229f4040>

We can also pull out one row of the coarsely-sampled signal and plot it signal-processing style:

In [5]:

plt.stem(coords, img[0])

Out[5]:

<StemContainer object of 3 artists>

The first row of the image (i.e. $y=0$) should be row 50 in our upsampled image indexing:

In [6]:

np.argwhere(upsample_coords == 0)

Out[6]:

array([[50]])

We can overplot the "truth" signal and make sure it passes through the original points, as a check:

In [7]:

plt.stem(coords, img[0])
plt.plot(upsample_coords, upsampled_truth[50])

Out[7]:

[<matplotlib.lines.Line2D at 0x323c998a0>]

Now, let's upsample the original coarsely sampled image using interpolation and see if it agrees with the truth! The new image size is just the original size multiplied by upsample.

In [8]:

dsize = (img.shape[0] * upsample, img.shape[1] * upsample)

scikit-image interpolation¶

Scikit-image requires some additional options get it to behave as (I) expected. Unlike linear interpolation, cubic interpolation can produce values outside the range of the input—that is well-known. However, the scikit-image interpolation functions require you to pass clip=False to get the expected behavior.

First let's look at the default behavior (clip=True):

In [9]:

# scikit-image
from skimage import transform

upsampled_skimage_bad = transform.resize(img, dsize, order=3)
dd.show_diff(upsampled_skimage_bad, upsampled_truth,
             as_percent=False, extent=img_extent, vmax=0.25, colorbar=True)
print(f"{np.min(img)=:1.3}\n{np.max(img)=:1.3}")
print(f"{np.min(upsampled_truth)=:1.3}\n{np.max(upsampled_truth)=:1.3}")
print(f"{np.min(upsampled_skimage_bad)=:1.3}\n{np.max(upsampled_skimage_bad)=:1.3}")

np.min(img)=-1.99
np.max(img)=1.94
np.min(upsampled_truth)=-2.0
np.max(upsampled_truth)=2.0
np.min(upsampled_skimage_bad)=-1.99
np.max(upsampled_skimage_bad)=1.94

Above, you can see that the upsampled_skimage_bad array has identical minimum and maximum pixel values to the original image. The $f(x, y)$ we defined has extrema that don't perfectly align with a sample location, so imposing this clip=True constraint means we'll definitely disagree with the "truth" image.

The plot is a difference image, subtracting the "ground truth" from the interpolation. You can see a couple of little "pimples" where the clipping has moved a predicted value and made it a worse approximation. For our purposes, we want clip=False.

With that pre-requisite dealt with, we can resize the image and compare the interpolation to the ground truth.

In [10]:

upsampled_skimage = transform.resize(img, dsize, order=3, clip=False)

Now, we plot the interpolated image and truth image on identical color scales, as well as the difference image. Below, values from row 0 of the interpolated image (orange) are overlaid on the original and finely-sampled truth image.

In [11]:

def comparison_plot(title, orig, truth, upsampled, row=0):
    fig, axs = plt.subplot_mosaic('abc\nddd', figsize=(8, 4))
    axs['d'].stem(coords, orig[row])
    upsampled_idx = upsample // 2 + row * upsample
    axs['d'].plot(upsample_coords, truth[upsampled_idx])
    axs['d'].plot(upsample_coords, upsampled[upsampled_idx])
    axs['d'].set(
        xlabel='original pixel column index'
    )
    dd.three_panel_diff_plot(
        upsampled, truth,
        as_percent=False,
        extent=img_extent,
        colorbar=True,
        ax_a=axs['a'], title_a=title,
        ax_b=axs['b'], title_b='ground truth',
        ax_aminusb=axs['c'],
        diff_kwargs=dict(vmax=0.25),
    )

comparison_plot('scikit-image', img, upsampled_truth, upsampled_skimage)

Performance at the edges is worse, as may be expected. The bicubic interpolation domain needs $4 \times 4$ pixel values, and the edge pixels only have that on one or two sides.

Of course, choosing the mode= differently will give you different behavior at the hairy edges. None of these options is more "right" than the others but you should know that argument exists in case you need it.

In [12]:

upsampled_skimage_symm = transform.resize(img, dsize, order=3, clip=False, mode='symmetric')
upsampled_skimage_const = transform.resize(img, dsize, order=3, clip=False, mode='constant')

fig, axs = plt.subplots(ncols=3, figsize=(10, 2))
dd.show_diff(upsampled_truth, upsampled_skimage, as_percent=False, extent=img_extent, vmax=0.25, colorbar=True, ax=axs[0])
dd.show_diff(upsampled_truth, upsampled_skimage_symm, as_percent=False, extent=img_extent, vmax=0.25, colorbar=True, ax=axs[1])
dd.show_diff(upsampled_truth, upsampled_skimage_const, as_percent=False, extent=img_extent, vmax=0.25, colorbar=True, ax=axs[2])
axs[0].set_title("reflect")
axs[1].set_title("symmetric")
axs[2].set_title("constant")

Out[12]:

Text(0.5, 1.0, 'constant')

Aside: scikit-image and data types¶

There's another footgun to be aware of in scikit-image when working with camera images, which are not floating-point by default. Since our sample signal is floating-point, illustrating this require defining a new sample image mini_img with dtype=np.uint16 (like a 10- or 12-bit camera might produce):

In [13]:

mini_img = np.array([
    [2, 3, 4, 1],
    [4, 1, 3, 2],
    [3, 4, 2, 1],
    [2, 1, 1, 3],
], dtype=np.uint16)
plt.colorbar(plt.imshow(mini_img, vmin=0))

Out[13]:

<matplotlib.colorbar.Colorbar at 0x32593b610>

In [14]:

mini_img_upsample = transform.resize(mini_img, (32, 32), order=3, clip=False)
mini_img_upsample_preserve = transform.resize(mini_img, (32, 32), order=3, preserve_range=True, clip=False)

fig, axs = plt.subplots(ncols=2, figsize=(7, 2))
axs[0].set(title='preserve_range=False')
cb1 = fig.colorbar(axs[0].imshow(mini_img_upsample))
cb1.formatter.set_scientific(False)
axs[1].set(title='preserve_range=True')
fig.colorbar(axs[1].imshow(mini_img_upsample_preserve))

Out[14]:

<matplotlib.colorbar.Colorbar at 0x325a63280>

The image on the left uses the default behavior and the image on the right uses the preserve_range=True option. The one on the left has pretty dramatically different values because it has been automatically rescaled from the dynamic range of a 16-bit integer into the interval [0, 1] of a floating point representation. In astronomical image processing this is not usually what you want.

So, we said above we don't want to blindly preserve the range of the inputs by clipping, but now we must supply preserve_range=True to actually get the desired behavior.

This is somewhat confusing.

In any case, official scikit-image documentation even says not to use floating point images with values beyond the $[-1, 1]$ interval, so maybe this library is best left for the computer vision people.

OpenCV¶

Speaking of computer vision people, that's what the CV in "OpenCV" stands for. They implement the core algorithms of computer vision as a compiled C++ library with Python bindings, so they're quite fast.

My friend Dr. Logan Pearce relies on their implementation of interpolation, so let's see how it stacks up.

In [15]:

# OpenCV
import cv2

upsampled_cv = cv2.resize(img, dsize, interpolation=cv2.INTER_CUBIC)

In [16]:

comparison_plot('OpenCV', img, upsampled_truth, upsampled_cv)

As expected, the values near the edges are a bit off where interpolation became extrapolation. Otherwise, pretty similar to scikit-image (though with fewer extra arguments required to get normal behavior).

There's one place where OpenCV really shines though:

In [17]:

%timeit cv2.resize(img, dsize, interpolation=cv2.INTER_CUBIC)

392 µs ± 7.25 µs per loop (mean ± std. dev. of 7 runs, 1,000 loops each)

In [18]:

%timeit transform.resize(img, dsize, order=3, clip=False)

58.1 ms ± 678 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)

It's a nice couple orders of magnitude faster to run, compared to the scikit-image function we used above.

scipy.ndimage¶

The default behavior of scipy.ndimage transformation functions is surprising.

In [19]:

# scipy
from scipy.ndimage import zoom

upsampled_scipy_bad = zoom(img, upsample, order=3)

In [20]:

comparison_plot('SciPy ndimage (defaults)', img, upsampled_truth, upsampled_scipy_bad)

Well, that's clearly... not what we wanted.

It turns out that SciPy's implementation does not match our intuition about finitely-sized pixels. In astronomical images, a camera pixel is a physical thing that has a width and a height, so we take the pixel coordinates as the value of a pixel-sized patch centered on the pixel coordinates. That's how we can have a value for -0.49 in the examples above. To the SciPy interpolator, a sample is an infinitely small point with a known value (a "knot") to constrain the path of a spline.

So, to SciPy, the image we're asking to zoom() actually starts at $(0, 0)$ and not $(-0.5, -0.5)$. That puts $(0, 0)$ in the first element of the row we plotted as a lollipop plot. If we started our upsampled coordinates at 0 instead of -0.5, we'd see what they intended.

In [21]:

plt.stem(coords, img[0])
plt.plot(upsample_coords, upsampled_truth[50])
plt.plot(np.linspace(0, samples - 1, upsample * samples), upsampled_scipy_bad[50])

Out[21]:

[<matplotlib.lines.Line2D at 0x32d9cfe80>]

This is... still not what we wanted.

It's not clear to me what's going on with the missing amplitude. It's also counter-intuitive for e.g. integrating. If all your pixels represent an area $1 \times 1$ but the edges have area $0.5 \times 1$ and the corners $0.5 \times 0.5$, you may confuse yourself. (Of course, you shouldn't zoom before integrating...)

SciPy supports the more intuitive interpretation for signal processing with the grid_mode=True argument to zoom(), which produces something more like the expected output. (Ignore the warning for the moment.)

In [22]:

upsampled_scipy_bad2 = zoom(img, upsample, order=3, grid_mode=True)

/Users/josephlong/mambaforge/envs/py310/lib/python3.10/site-packages/scipy/ndimage/_interpolation.py:801: UserWarning: It is recommended to use mode = grid-constant instead of constant when grid_mode is True.
  warnings.warn(

In [23]:

plt.stem(coords, img[0])
plt.plot(upsample_coords, upsampled_truth[50])
plt.plot(upsample_coords, upsampled_scipy_bad2[50])

Out[23]:

[<matplotlib.lines.Line2D at 0x324f17070>]

Here, the difference in indexing convention rears up again in the form of clamping all values outside $0 \le x \le 9$ to zero. You must also supply a mode argument to interpolate those pixel values. (That's what the warning above is about, though mode='grid-constant' is not the only valid choice.)

In [24]:

upsampled_scipy = zoom(img, upsample, order=3, grid_mode=True, mode='grid-mirror')

In [25]:

comparison_plot('SciPy ndimage (grid_mode)', img, upsampled_truth, upsampled_scipy)

Runtime-wise, performance is comparable to scikit-image.

In [26]:

%timeit zoom(img, upsample, order=3, grid_mode=True, mode='grid-mirror')

58 ms ± 717 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)

Of course, zoom() just happened to be a convenient test operation to compare. In practice, you'll want to be able to apply arbitrary affine transformations or even distortion solutions in your interpolation step.

Since the default options apply assumptions that conflict with our intuition for finite-size pixels, I wouldn't recommend the SciPy ndimage functions for that. There may be other implications of the assumed indexing convention that crop up later on.

scipy.interpolate¶

Fortunately, scipy.interpolate's default behavior is a lot more intuitive. Unfortunately, depending on whether you start out searching for "rotate image scipy" or "interpolate data scipy", you could end up in either ndimage or interpolate docs without any idea that there's two options.

The RectBivariateSpline class supports spline interpolation of 2D data with regular spacing.

In [27]:

from scipy.interpolate import RectBivariateSpline

It takes in 1D arrays for the $x$ and $y$ coordinates of a 2D array of pixels/samples. There's a pitfall here that we'll explore in a moment.

In [28]:

rbvs_interp = RectBivariateSpline(coords, coords, img)

The resulting object can be called with 1D arrays of new $x$ and $y$ coordinates, like upsample_coords from above.

In [29]:

upsampled_rbvs = rbvs_interp(upsample_coords, upsample_coords)

In [30]:

comparison_plot('SciPy RectBivariateSpline', img, upsampled_truth, upsampled_rbvs)

Possibly noteworthy: the computation time is almost all at interpolation time, not instantiation time. Unfortunately, that means pre-constructing and reusing RectBivariateSpline instances doesn't add much in terms of efficiency.

In [31]:

%timeit RectBivariateSpline(coords, coords, img)

16.5 µs ± 46.5 ns per loop (mean ± std. dev. of 7 runs, 100,000 loops each)

In [32]:

%timeit rbvs_interp(upsample_coords, upsample_coords)

8.97 ms ± 111 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

The good news is that it's about six times faster than scikit-image for this example case, albeit still an order of magnitude slower than OpenCV. Possibly more accurate than OpenCV, though, since it's using spline interpolation instead of (I think) cubic convolution.

Now, what's the catch?

A confusion of `x` and `y`¶

The scipy.interpolate functions for 2D interpolation adopt a naming convention at odds with how matplotlib shows images. For an array of shape (4, 5), RectBivariateSpline would call 4 the number of x samples and 5 the number of y samples. Plotting the array, however, will show four rows of five columns.

The documentation for the callable rbvs_interp points this out:

        x, y : array_like
            Input coordinates.

            If `grid` is False, evaluate the spline at points ``(x[i],
            y[i]), i=0, ..., len(x)-1``.  Standard Numpy broadcasting
            is obeyed.

            If `grid` is True: evaluate spline at the grid points
            defined by the coordinate arrays x, y. The arrays must be
            sorted to increasing order.

            The ordering of axes is consistent with
            ``np.meshgrid(..., indexing="ij")`` and inconsistent with the
            default ordering ``np.meshgrid(..., indexing="xy")``.

The last sentence tells us that where RectBivariateSpline is concerned x is the first axis, y is the second.

Of course, interpreting any block of memory as an image is a matter of establishing conventions. For example, the default behavior of matplotlib is to place the origin (i.e. pixel (0, 0)) at upper left, as you would for printing out a matrix (left to right, top to bottom). (This causes consternation for astronomers, who place the origin at lower left, but the default can be changed. This customization was made up there in the very first cell.)

The risk is that non-square images and transformations that do different things to x and y coordinates will not behave as expected. For example, we can translate the interpolated grid by subtracting an offset from upsample_coords.

In [33]:

offset_x = 2.5
plt.imshow(rbvs_interp(upsample_coords - offset_x, upsample_coords))

Out[33]:

<matplotlib.image.AxesImage at 0x32bd5d540>

That's not what we wanted! The resulting image has been translated by offset_x in the direction represented by the first axis of img: i.e., the $y$ direction. So, to translate horizontally, we adjust the second coordinate argument (which it calls y).

In [34]:

plt.imshow(rbvs_interp(upsample_coords, upsample_coords - offset_x))

Out[34]:

<matplotlib.image.AxesImage at 0x32bdb19f0>

This API only allows translation and scaling, because the 1D coordinate vectors give a constant x across a row / constant y along a column. Fortunately, since SciPy version 0.14.0, the RectBivariateSpline interpolator accepts a grid=False argument. Supplying all the xs and ys allows arbitrary transformations and handily sidesteps confusion about axis naming/ordering (since you are responsible for reshaping from a 1D vector of interpolated values back into a 2D array).

There would be an example of that here, but this post is long enough already.

Which interpolation function should I use for image processing in Python?¶

For speed, ergonomics of API, and (approximate) correctness, I have to agree with Dr. Pearce: the OpenCV geometric transformations functions are the way to go. If you watch out for the x/y switcharoo, SciPy's RectBivariateSpline may provide better accuracy at the cost of slower calculations.

(Or, of course, you could implement it yourself! That way you know exactly what it's doing with your precious pixels of data.)

Thanks to Philip Mocz and Louis Desdoigts for help tracking down what SciPy ndimage is doing.

A small collection of exoplanet detection illustrations

2024-07-25T00:00:00-07:00

For the introduction chapter of my Ph.D. thesis, I wanted to make some attractive minimalist figures. I knew I had a lot of specialist jargon to explain in the public portion of my defense, and wanted to minimize distractions. Someone suggested that I make these figures available for others to use, so I'm taking that suggestion (albeit somewhat late).

Feel free to use these figures for any non-commercial application, provided you preserve the attribution in them! See details.

Since I already wrote captions for these figures in my thesis, I'm reusing them here.

Adaptive optics system

Diagram of an adaptive optics system. Starlight (orange) arrives at Earth as planar wavefronts above the atmosphere (upper left), then propagates through atmospheric inhomogeneities (blue circles), picking up aberrations. This aberrated light is collected by a telescope, and the beam is bounced off of a deformable mirror (DM) to cancel the aberrations. Residual aberrations remain due to lag and the limitations of the hardware. The light passes through a beamsplitter that sends part of the light to the wavefront sensor (WFS), where the residual aberrations are measured. The measurement is used by the controller to compute an updated deformable mirror position to cancel the residual wavefront error, which is then applied to the DM, closing the control loop.

Download: SVG PDF

Transit method

Illustration of the systematic bias inherent in transit observations using three configurations of a notional planet in orbit around a distant star. The observer views the scene in the square panel (albeit without resolving planet from star) as the planet moves from left to right between the observer and the star. They record the brightness of the host star over time, resulting in the light curve below.

Configuration A

In configuration A, the planet is on a short orbit in a plane aligned with the observer’s line of sight.

Download: SVG PDF

Configuration B

In configuration B, the planet’s orbit is inclined relative to the line of sight. However, the planet still transits the star, providing a very similar light curve to configuration A.

Download: SVG PDF

Configuration C

In configuration C, the angle of inclination is the same, but the planet is on a longer orbit. Consequently, that the planet proceeds on its orbit without ever occulting the star, producing no transit signal to detect.

Download: SVG PDF

Alternative non-transiting configuration

In this configuration, the planet follows an orbit that is face-on from the observer's perspective and therefore does not transit its host star and cannot be detected by the transit method.

Download: SVG PDF

Radial velocity method

Download: SVG PDF

Illustration of the radial velocity exoplanet detection method. A planet is depicted at three points (A, B, and C) in its orbit about its host star. The gray inset shows a view from above the plane of the orbit, indicating the common center of mass both bodies orbit with ×. At point A, the planet’s velocity towards the observer is maximized, and therefore so is the star’s velocity away from the observer. A hypothetical feature in the star’s spectrum would then be red-shifted by some amount ∆λ that can be measured with a spectrograph. At point B, the planet’s velocity is perpendicular to the line of sight, and there is no detectable red- or blue-shift. At point C, the planet is moving away from the observer, and the star towards them, blue-shifting the spectrum.

Flat wavefronts approximation

Download: SVG PDF

Illustration of how a spherical wavefront can approach an idealized flat one when seen at Earth.

Off-axis source wavefronts

Download: SVG PDF

Illustration of light from two sources arriving as a combination of an on-axis and an off-axis plane wave, brought to focus to create an image of a star and a companion. Right square panel shows the image at the detector.

Sky rotation

Download: SVG PDF

Illustration of sky rotation from target rise to transit to set time. When a ground-based telescope tracks a target from horizon to horizon, the apparent orientation of the scene changes (unless a field rotator is used to compensate).

Alternative figure

This figure has been adapted to fill a widescreen presentation slide.

Download: SVG PDF

Angular differential imaging

Download: SVG PDF

Illustration of angular differential imaging (ADI). A: Capture a sequence of exposures as the scene appears to rotate during the target’s passage across the sky—from rise to set—as the parallactic angle changes. (Or, for space-based imaging, as the roll about the telescope pointing vector changes.) B: Combine (e.g. by median stacking) the images in the sequence to provide a better estimate of the PSF of the host star while ignoring planet light. C: Subtract this estimate from each frame separately. D: Rotate the resulting images by their parallactic angle (and any offset) to place north up and east to the left in all the images. E: Stack the subtracted images in the rotated frame to combine signal from the planet while averaging out noise from the residual starlight.

Licensing and reuse

This work is licensed under CC BY-NC 4.0.

Non-commercial distribution and adaptation are permitted within the terms of the license, as long as attribution is preserved.

Launching CLI apps from the macOS GUI

2021-11-20T00:00:00-07:00

I spend most of my time working from the command-line, so being able to inspect an astronomical FITS format image in DS9 with ds9 -multiframe -zoom to fit ./thing.fits as fast as I can type it is useful to me. (Except, obviously, I have an alias for those options. I'm not an animal.)

SAO distributes X11 builds of DS9 for macOS, which I can launch from the command line, so I use those. Now, their "Aqua" builds of ds9 for macOS use the Tcl/Tk framework's implementation of the native macOS widgets (prettier!) and Finder double-click-ability (convenient!) at the cost of convenient command-line launching and, most recently, incompatibility with Dark Mode™ on macOS Monterey (disaster!).

So, I stick with the command line version. Sometimes, though, I want to open a FITS file by double-clicking it in the Finder. This is the hack I came up with to enable that.

Step 0. Get the command-line (X11) version of ds9

Go to the ds9 download page and get the appropriate X11-based release for your OS. (Confused about x86 vs. ARM64? Check the Apple menu at top left, and choose "About This Mac". If it says "Apple" next to "Chip", you have ARM64. If it says "Intel", you have x86. So, if you have an Apple chip, you want to click the link that says something like "X11 Big Sur 11.0 ARM64"--unless there's a newer version for a newer macOS by the time you're reading this.)

Extract the file, and put the contents (both ds9 and ds9.zip, not any folder they extracted into) into /usr/local/bin. (The /usr/local/bin folder is hidden in the Finder, but you can type open /usr/local/bin in the terminal to open a window there.)

Verify this works first! That means opening a terminal window and typing ds9 and waiting to make sure XQuartz starts, and the ds9 window opens, and no major errors appear in your terminal window.

Step 1. Open Script Editor

Only "applications" can open files in macOS, not dinky little "executables". So, we have to make an "application."

In your /Applications/Utilities folder, there's an app called "Script Editor". Open it up and paste this in:

on run
    process({})
end run

on open dropped_files
    process(dropped_files)
end open

on process(dropped_files)
    set file_list to ""
    if length of dropped_files > 0 then
        set parent_dir to do shell script "dirname " & quoted form of POSIX path of item 1 of dropped_files
    else
        set parent_dir to do shell script "echo $HOME"
    end if
    repeat with file_path in dropped_files
        set file_list to file_list & " " & quoted form of POSIX path of file_path
    end repeat
    do shell script "cd " & parent_dir & "; /usr/local/bin/ds9 -multiframe -zoom to fit " & file_list & " &> /dev/null &"
end process

You can tweak the options following the path to ds9 in the above do shell script line to be what you prefer. The ampersand (&) is the concatenation operator, so what we're doing is constructing a command just like the one we'd type in the terminal to launch ds9 with one or more files as arguments. The final &> /dev/null & backgrounds the process and indicates we don't care about its outputs at all, so that the script can exit without waiting for ds9. (Otherwise, you'd get an extra icon in the dock every time you launched it this way.)

The set parent_dir... line makes the working directory of the resulting ds9 process the same as the parent directory of the first file in the list to open, instead of the default (/).

Save as "File Format: Application" to create a .app bundle (I called mine ds9.app). Put it in /Applications, or where-ever you prefer.

Step 2. Edit Info.plist

App bundles are just folders, so for the next trick open ds9.app/Contents/Info.plist in a text editor. You can control-click (or right-click, or two-finger-click) the ds9 app bundle icon and choose "Show contents" to see inside and get to Contents/Info.plist. Open it in your favorite text editor for code.

Right before the final </dict>, add:

    <key>CFBundleDocumentTypes</key>
    <array>
        <dict>
            <key>CFBundleTypeExtensions</key>
            <array>
                <string>fits</string>
                <string>fit</string>
            </array>
            <key>CFBundleTypeName</key>
            <string>Flexible Image Transport System image</string>
            <key>CFBundleTypeRole</key>
            <string>Editor</string>
            <key>LSHandlerRank</key>
            <string>Owner</string>
        </dict>
    </array>

Next, save the .plist file. You can check if everything's right using plutil, e.g.

$ plutil ds9.app/Contents/Info.plist
ds9.app/Contents/Info.plist: OK

(Note: This edit should be sufficient to make the app a drop target for FITS files, but it seems like LaunchServices can get confused. You might need a magic incantation like /System/Library/Frameworks/CoreServices.framework/Frameworks/LaunchServices.framework/Support/lsregister -kill -domain local -domain system -domain user to regenerate the list of file associations.)

Double-click your app bundle to launch an XQuartz version of ds9. It might take a second. (If you just installed XQuartz for the first time at the beginning of this guide, you may even need to reboot first.)

Step 3. Update file associations (if needed)

There's probably a command-line way to do this, but right-clicking on a one .fit and one .fits file and choosing Get Info gets you the window shown below. Select your new .app under the "Open with:" section and click "Change All..." to get this prompt

Confirm by clicking "Continue," and we're done. Now, double-clicking any .fits or .fit file should open an XQuartz ds9 window. Dragging multiple files onto the icon will open them within the same window as separate frames.

Thanks

I cobbled this together from a couple places: First, https://github.com/RichardBronosky/AppleScript-droplet, which has a more flexible solution for launching arbitrary scripts. Second, the bundle keys and values were taken from this StackOverflow answer, though it omits the fact you must include <key>CFBundleDocumentTypes</key> before the array.

Thanks to Meghan O'Brien and Avalon McLeod for feedback on an earlier version of this blog post!

Updates

2022-08-13 -- added parent_dir to script example, made path in script consistent with putting ds9 in /usr/local/bin

Finding a planet by subtracting starlight

2020-09-28T00:00:00-07:00

When I started graduate school, I had to get up to speed on high contrast imaging. As is typical, I had a bunch of different papers to read that—together—represented the state of the art. Turning that into a complete conceptual picture that I could implement myself took some time. (And, of course, once you 'get it' you wonder why it took you so long...)

This is not quite an explorable explanation, but it's something I wish I had when I started graduate school. If you'd like to, you should be able to follow along with nothing more than this notebook, an internet connection, Python, NumPy, matplotlib, and scikit-image.

Table of contents

Introduction
Get the data
The preliminaries
The algorithm
Apply KLIP and ADI
Optimizing $K_\text{klip}$ and measuring signal-to-noise
The Case of the Missing Precision
Conclusion

Introduction¶

To image planets around other stars, we are usually impeded by the brightness of the star relative to the planet. Therefore, clever schemes to remove starlight have been the cornerstone of exoplanet direct imaging. The state of the art in starlight subtraction advanced dramatically in 2012 with the publication in the Astrophysical Journal Letters of "Detection and Characterization of Exoplanets and Disks Using Projections on Karhunen-Loève Eigenimages" by Rémi Soummer, Laurent Pueyo, and James Larkin. (Rémi and Laurent I actually worked with at STScI, though not on high-contrast imaging!)

In their paper, they explain how the eigenvectors of the covariance matrix of a reference set of star images may be used to estimate (and subtract) starlight in a new observation. Almost simultaneously, Adam Amara and Sascha P. Quanz published their paper "PYNPOINT: an image processing package for finding exoplanets" in the Monthly Notices of the Royal Astronomical Society, describing a closely related technique.

Though the specific notation and algorithm proposed differed, the two papers applied the method of principal component analysis to starlight subtraction. Principal component analysis goes by many names, as the following quote from the Wikipedia page demonstrates:

Depending on the field of application, it is also named the discrete Karhunen–Loève transform (KLT) in signal processing, the Hotelling transform in multivariate quality control, proper orthogonal decomposition (POD) in mechanical engineering, singular value decomposition (SVD) of X (Golub and Van Loan, 1983), eigenvalue decomposition (EVD) of XTX in linear algebra, factor analysis (for a discussion of the differences between PCA and factor analysis see Ch. 7 of Jolliffe's Principal Component Analysis), Eckart–Young theorem (Harman, 1960), or empirical orthogonal functions (EOF) in meteorological science, empirical eigenfunction decomposition (Sirovich, 1987), empirical component analysis (Lorenz, 1956), quasiharmonic modes (Brooks et al., 1988), spectral decomposition in noise and vibration, and empirical modal analysis in structural dynamics.

These different names generally come with minor mathematical differences, such as whether means are subtracted along one dimension or another, or eigenimages are obtained from the eigenvectors of the image-to-image covariance matrix instead of from the singular vectors of the data matrix, but we will not attempt to cover all possible variants—just the description in Soummer et al.

There are two key aspects to these algorithms:

identifying the eigenimages, and
truncation as noise suppression.

As we will see, a set of images in an array of $K \times n_\text{pix} \times n_\text{pix}$ elements can be transformed into an array of $K \times n_\text{pix} \times n_\text{pix}$ eigenimages. These eigenimages are ordered such that the first image captures most of the variance in the dataset (i.e. the average image), with the successive components capturing variance in the residuals after previous components have been removed.

The truncation step relies on the assumption (which can be checked empirically) that, in a set of $K$ observations contaminated with noise transformed into $K$ corresponding eigenimages, only the first $K_\text{klip} < K$ of those eigenimages will capture useful information for subtracting starlight from a new image outside the reference set, while the rest capture only noise.

The following code sample follows the algorithm as laid out in Soummer et al., following their notation (within the constraints of Python).

In the following, $I_\psi$ means a PSF intensity pattern realized for a telescope state $\psi$. We're trying to compute $\hat{I}_\psi$, an approximation of the PSF intensity pattern, in hopes that subtracting it will reveal a faint astronomical signal (which they call $A$). Our observed frames $T$ are combinations of the PSF and an astronomical signal: $T = I_\psi + A$. If our approximation is good, we subtract it off like so: $T - \hat{I}_\psi = A$, recovering the astronomical signal of interest. (This is a simplification of what is stated in Soummer et al. since, as we will see, $A$ will not be recovered perfectly in general.)

To begin, we import our libraries in the usual fashion, and make some adjustments for attractive plots:

In [1]:

# %config InlineBackend.figure_formats = ['retina']
%config InlineBackend.figure_formats = ['svg']
%config InlineBackend.print_figure_kwargs = {'facecolor': (1.0, 1.0, 1.0, 0.0)}
import numpy as np
import matplotlib
import matplotlib.pyplot as plt

matplotlib.rcParams.update({
    'image.origin': 'lower',
    'image.interpolation': 'nearest',
    'image.cmap': 'magma',
    'font.family': 'serif',
})

Get the data¶

These data of $\beta$ Pictoris from NACO were made available by Carlos Gomez Gonzalez, author of VIP (an image processing package implementing this and other algorithms). Unlike most exoplanet direct imaging data, they have a planet in them, which makes them great for a demo.

Download naco_betapic_preproc.npz and save it to the current directory, or just run this cell:

In [2]:

!curl -OL https://github.com/carlgogo/VIP_extras/raw/master/datasets/naco_betapic_preproc.npz

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100 2177k  100 2177k    0     0  10.6M      0 --:--:-- --:--:-- --:--:-- 17.1M

Load the NumPy archive and assign the data cube to frames_full.

In [3]:

naco_betapic_preproc = np.load('./naco_betapic_preproc.npz')
frames_full = naco_betapic_preproc['cube']

Peek at the mean frame. These data were taken with a vortex coronagraph, so don't be surprised that the star looks more like a donut.

In [4]:

plt.imshow(np.average(frames_full, axis=0))
_ = plt.colorbar()

The preliminaries¶

To save a lot of extra typing, we will rearrange our images into image vectors first. Instead of a 3D cube where the first index is frame number, second is pixel Y index, and third is pixel X index, we now have a 2D matrix where the first (row) index is frame number and the second (column) index is an unwrapped pixel index between 0 and ((X pixels) * (Y pixels) - 1).

In [5]:

image_shape = frames_full.shape[1], frames_full.shape[2]
frame_vecs_full = frames_full.reshape(frames_full.shape[0], image_shape[0] * image_shape[1])

In section 2.1 of the paper, an $N$ pixel PSF intensity pattern (star image) is written $I_\psi(n)$, where $n$ denotes the $n$th pixel in the image and $\psi$ is a symbolic placeholder for any properties of the imaging system that could change. (In practice, we can't—and don't need to—compute $\psi$, but it lets us refer to different realizations of the PSF $\{ I_{\psi_0}, I_{\psi_1}, ..., I_{\psi_k} \}$.)

The authors use $n \in [1, N]$ as the unwrapped pixel index (eq. 1). We count from 0 in Python / NumPy, making it $n \in [0, N)$ when used in the code. Unless explicitly stated, 1-based indices in equations in the text correspond to 0-based indices in the code. This makes both read more naturally, and the infrequency of explicit indexing in well-written NumPy code should (I hope) prevent this from being a source of confusion.

We will need this N later, so pull it out of the shape and put it in a variable.

In [6]:

N = frame_vecs_full.shape[1]

The algorithm¶

Karhunen-Loève Image Projection (KLIP) consists of the following steps.

Step 1: subtract image means and select target and reference images¶

We subtract the means from all images before separating them into a target image and set of reference images. By averaging along the column axis of our matrix of unwrapped images, we get the mean value in each image across all its pixels. (There's some ambiguity in the original paper as to which direction this mean should go—along an image or down a pixel—but one of the directions produces the correct answer.)

In [7]:

mean_vec = np.average(frame_vecs_full, axis=1)
mean_vec.shape

Out[7]:

(61,)

In [8]:

frame_vecs_meansub = frame_vecs_full - mean_vec[:,np.newaxis]

For no particular reason, we select frame 30 (0-indexed) as the target image $T(n)$ and the rest of the frames form the set of references $R_k(n)$.

In [9]:

target = frame_vecs_meansub[30]
reference = np.delete(frame_vecs_meansub, 30, axis=0)

target.shape, reference.shape

Out[9]:

((10000,), (60, 10000))

To index into the reference set of PSFs $R_k$ for $k \in [0, K)$ we'll need the number of references, $K$, so stash it in a variable.

In [10]:

K = reference.shape[0]

Step 2: compute the Karhunen-Loève transform¶

Each image vector is 10,000 elements, but we have only $K = 60$ reference images. To have a hope of accurately subtracting starlight from a target image that our algorithm hasn't seen, we must find a low-dimensional representation (≤60) that can reproduce a good approximation of the target $T$ from our 60 reference images $R_k$.

We could try finding the dot product of our target image vector with each reference image vector, but our images are not orthogonal (indeed, we expect them to be largely the same).

The Karhunen-Loève transform creates a basis set of orthogonal eigenimages which is "optimal". The sense in which it is optimal is in how it may be truncated: it is an "energy compacting" basis. The first KL basis vector explains the greatest part of the variance in the references, the second explains the next-greatest, and so on. Thus, removing (truncating) basis vectors from the end of the transform basis still leaves a basis set that reproduces a particular realization of the PSF $I_\psi$ with the least mean squared error (proof on Wikipedia).

This is the core of KLIP, and is really a multi-step process. Soummer et al. define the $k$th member of the KL basis for the set of reference PSFs $\{ R_k(n) \}_{k=1...K}$ in their equation 5:

Eq. 5: $$Z_k^\text{KL} (n) = \frac{1}{\sqrt{\Lambda_k}} \sum_{p = 1}^{K} c_k(\psi_p) R_p(n) $$ where the vectors $C_k = [c_k(\psi_1) ... c_k(\psi_K)]$ are the eigenvectors of the $K$-dimensional covariance matrix $E_{RR}$ of the $R_k(n)$ references over S, and ${\Lambda}_{k=1...K}$ are its eigenvalues.

So, before we can compute the transform in eq. 5, we must construct the covariance matrix $E_{RR}$ of the references. They define it in their equation 6:

Eq. 6: $$ E_{RR}[i, j] = \sum_{n=1}^{N_S} R_i(n) R_j(n)$$

In equation 6, $N_S$ is the number of pixels in the search region $S$. Since our search region is the entire image, $N_S = N$. NumPy includes a built-in covariance function, which we can call on our matrix of reference image vectors.

Note: NumPy divides the result by $N - 1$, so to match the definition of $E_{RR}$ in the paper we must in turn multiply the covariance matrix by $N - 1$.

In [11]:

E = np.cov(reference) * (N - 1)
plt.figure(figsize=(6,6))
plt.imshow(E, origin='upper')

Out[11]:

<matplotlib.image.AxesImage at 0x30551c310>

We could have just as easily computed it ourselves by multiplying $R$ and $R^T$. (In fact, this should be ever-so-slightly faster.)

In [12]:

np.allclose(E, reference @ reference.T)

Out[12]:

True

The next prerequisite for evaluating eq. 5 is the vectors $C_k = [c_k(\psi_1) ... c_k(\psi_K)]$. These are the eigenvectors of the covariance matrix we just computed, which by virtue of it being symmetric, can be solved for with numpy.linalg.eigh.

We also plot the resulting eigenvalues in the order returned to reveal an important detail:

In [13]:

lambda_values_out, C_out = np.linalg.eigh(E)
plt.plot(lambda_values_out)
plt.yscale('log')
plt.xlabel('Index')
plt.ylabel('Eigenvalue')
C_out.shape

Out[13]:

(60, 60)

The authors' equation 7 restates the definition of eigenvalue and eigenvector...

$$ E_{RR} \cdot C_k = \Lambda_k C_k$$

... with a hint about indexing:

$$ \Lambda_1 > \Lambda_2 > ... > \Lambda_K.$$

To match this convention, reverse both the eigenvectors (arranged in columns, therefore axis=1) and the eigenvalues:

In [14]:

lambda_values = np.flip(lambda_values_out)
C = np.flip(C_out, axis=1)

Finally, we are able to evaluate equation 5 to compute the Karhunen-Loève transform of the reference PSFs.

Eq. 5: $$Z_k^\text{KL} (n) = \frac{1}{\sqrt{\Lambda_k}} \sum_{p = 1}^{K} c_k(\psi_p) R_p(n) $$

where $c_k(\psi_p)$ is (confusingly, for me) the $p$-th entry in the $k$-th column of $C$. And note that $(n)$ indexes a particular pixel value in the unwrapped image (or eigenimage) on both sides of the equation, so we can use NumPy broadcasting to treat a whole image at a time.

The following code calculates $Z^\text{KL} = [Z_1^\text{KL} ... Z_K^\text{KL}]$. For clarity, aside from the pixel index $n$, I'm just using (slow) Python loops for now instead of NumPy operations. Again, our indices will range from $p = 0$ to $p = K - 1$ and from $k = 0$ to $k = K - 1$ to match Python indexing.

In [15]:

Z_KL_slow = np.zeros((N, K))
for k in range(K):
    for p in range(K):
        Z_KL_slow[:,k] += (C[p,k] * reference[p]) / np.sqrt(lambda_values[k])
Z_KL_slow.shape

Out[15]:

(10000, 60)

Now we can see what the KL transform basis thinks are the dominant modes in this dataset:

In [16]:

Z_KL_slow_images = Z_KL_slow.T.reshape((reference.shape[0],) + image_shape)
plt.figure(figsize=(12, 4))
for i in range(5):
    plt.subplot(1, 5, i + 1)
    plt.imshow(Z_KL_slow_images[i])

As expected, the stellar PSF is not stable across the frames in our reference set, with the dominant contributor to the variation being atmospheric seeing-induced aberration and speckles. These eigenimages capture the most common kinds of aberration found in our reference set.

The observant reader will notice that the summation in equation 5 looks a lot like a dot-product. We're summing along an index $p$, which indexes into the row of a particular eigenvector and the column of a particular reference PSF.

In other words, the vector of the $n$th pixels $n \in [1,N]$ of eigenimage $Z_k^\text{KL}$ is given by

$$ \begin{align*} \begin{bmatrix} Z_k^\text{KL}(1) \\ \vdots \\ Z_k^\text{KL}(N) \end{bmatrix}_{1 \times N} &= \begin{bmatrix} \frac{1}{\sqrt{\Lambda_k}} (C_{p=1,k} R_{p=1,n=1} + ... + C_{p=K,k} R_{p=K,n=1}) \\ \vdots \\ \frac{1}{\sqrt{\Lambda_k}} (C_{p=1,k} R_{p=1,n=K} + ... + C_{p=K,k} R_{p=K,n=K}) \end{bmatrix} \end{align*} $$

where I've written indices into the $C$ and $R$ matrices as row, column subscripts to emphasize that the $k$th KL mode uses the values of the $k$th column of $C$ and the entire $R$ matrix.

If you're like me, this is easier to explain if we pull it apart and show $C$ and $R$ as block matrices. We normally think of eigenvectors indexed by column. Our reference image-vectors are arranged in rows. (The matrix shape is noted in row $\times$ column subscripts.)

$$ \begin{align*} C = \begin{bmatrix} | & ... & | \\ C_1 & ... & C_{K} \\ | & ... & | \\ \end{bmatrix}_{K \times K} & & R = \begin{bmatrix} — & R_1 & — \\ & \vdots & \\ — & R_K & — \\ \end{bmatrix}_{K \times N} \end{align*} $$

$Z_1^\text{KL}$, the first KL basis vector, is then given by $$ Z_1^\text{KL} = \frac{1}{\sqrt{\Lambda_1}} \begin{bmatrix} — & C_1 & — \\ \end{bmatrix} \begin{bmatrix} — & R_1 & — \\ & \vdots & \\ — & R_K & — \\ \end{bmatrix} = \frac{1}{\sqrt{\Lambda_k}} \begin{bmatrix} — & C_1 R & — \\ \end{bmatrix}_{1 \times N} $$

To construct the basis, then, we want $$ Z^\text{KL} = \begin{bmatrix} | & ... & | \\ C_1 R \Lambda_1^{-1/2} & ... & C_K R \Lambda_K^{-1/2}\\ | & ... & | \\ \end{bmatrix}_{K \times N} $$

to apply the scaling we need to rescale every element in $C_K$ by the corresponding $\Lambda_k^{-1/2}$, so call $X$ the diagonal matrix whose entries are $\Lambda_1^{-1/2}, ..., \Lambda_K^{-1/2}$.

$$ X = \begin{bmatrix} \Lambda_1^{-1/2} & & 0\\ & \ddots & \\ 0 & & \Lambda_K^{-1/2} \end{bmatrix}_{K \times K} $$

Then we use $(C X)^T$ to construct $(Z_\text{KL})^T$

$$ \begin{align*} (Z^\text{KL})^T = (C X)^T R = \begin{bmatrix} — & C_1 R \Lambda_1^{-1/2} & — \\ & \vdots & \\ — & C_K R \Lambda_K^{-1/2} & — \\ \end{bmatrix}_{N \times K} \end{align*} $$

so that $$ \begin{align*} Z^\text{KL} = ((C X)^T R)^T = \begin{bmatrix} | & ... & | \\ C_1 R \Lambda_1^{-1/2} & ... & C_K R \Lambda_K^{-1/2}\\ | & ... & | \\ \end{bmatrix}. \end{align*} $$

After sorting out the repeated transpositions and reordering, we have

$$ Z^\text{KL} = R^T ((C X)^T)^T = R^T (C X) $$

and can express the above doubly-nested loop in a single line of NumPy matrix algebra:

In [17]:

Z_KL = reference.T @ (C * np.power(lambda_values, -1/2))

Compare to the other version (using atol and rtol to account for numerical noise):

In [18]:

np.allclose(Z_KL_slow, Z_KL, atol=1e-6, rtol=1e-4)

Out[18]:

True

Step 3: truncate the basis set¶

All Soummer et al. say in this step is to "choose a number of modes $K_\text{klip}$ to keep in the estimate of $\hat{I}_{\psi_0}(n)$".

While simple in principle, in practice this is chosen by adjusting $K_\text{klip}$ such that the signal-to-noise ratio of a source is optimized. (See Optimizing $K_\text{klip}$, later on.)

Why truncate instead of keeping all 60 KL basis vectors?

When your references are images of totally different targets and you know ahead of time that they don't contain anything but $I_\psi$, the intensity pattern for a particular combination of telescope and conditions, it's less important! There is a risk of overfitting, but the same property that allows us to explain most of the signal with only a few modes means that modes representing pure noise won't influence the reconstruction much.

Ground-based observations employing angular differential imaging (like this example) are another matter, as the target and reference are generally images of the same star (potentially combined with other astrophysical signal, like an exoplanet or disk), under the same conditions, varying only the rotation of the field. After subtracting the stellar PSF, the images are then rotated to a common orientation before combining, hopefully revealing an astrophysical signal that was mostly uncorrelated with the stellar PSF.

Under those circumstances, even when separating your target from your reference image, faint astrophysical sources near the star in one reference image may overlap their position in the target image—causing them to be mostly or entirely removed by KLIP.

While we will see the effects of varying $K_\text{klip}$ on the end product later, for now let's set it at 30.

In [19]:

K_klip = 30

Step 4: estimate the PSF from its projection onto eigenimages¶

To find $\hat{I}_{\psi_0}$, the best estimate of $I_{\psi_0}$ (the stellar PSF contribution to the target), we project our target $T$ onto each of the eigenimages in the KL basis to find that eigenimage's contribution to $T$, then multiply that by the eigenimage to obtain the pixel values and sum them.

The authors express $\hat{I}_{\psi_0}$ as follows in equation 8:

$$ \hat{I}_{\psi_0}(n) = \sum_{k=1}^{K_\text{klip}} \langle T, Z_k^\text{KL} \rangle_S Z_k^\text{KL}(n) $$

where $\langle T, Z_k^\text{KL} \rangle_S$ signifies the inner product over the search zone $S$ (in our case, the whole image).

Let us first construct Z_KL_truncated, which contains only the first K_klip eigenimages.

In [20]:

Z_KL_truncated = Z_KL[:,:K_klip]

Each column of $Z_k^\text{KL}$ is an eigenimage, and the projection of $T$ along that eigenimage is just a simple dot product:

$$ \langle T, Z_k^\text{KL} \rangle = Z_k^\text{KL} \cdot T $$

which we then multiply by the eigenimage to get its contribution to $\hat{I}_{\psi_0}$. Writing the above summation with a slow Python loop, we get:

In [21]:

Ihat_psi0_slow = np.zeros(N)
for k in range(K_klip):
    projection = np.dot(Z_KL_truncated[:,k], target)
    Ihat_psi0_slow += projection * Z_KL_truncated[:,k]

Let's compare $\hat{I}_{\psi_0}$ to $T$:

In [22]:

plt.figure(figsize=(12, 4))
plt.subplot(121)
plt.imshow(target.reshape(image_shape))
plt.colorbar()
plt.subplot(122)
plt.imshow(Ihat_psi0_slow.reshape(image_shape))
plt.colorbar()

Out[22]:

<matplotlib.colorbar.Colorbar at 0x307c1e290>

Looks reasonable.

The astute reader may once again observe that projections and summations can be readily expressed with matrix operations. The dot products of $T$ with all $K_\text{klip}$ eigenvectors can be expressed as a $K_\text{klip}$-dimensional vector $v = (Z^\text{KL})^T T$.

In [23]:

(Z_KL_truncated.T @ target).shape

Out[23]:

(30,)

We then form a linear combination of the columns of $Z^\text{KL}$ with the projection vector, which is just $Z^\text{KL} v$.

So, the full PSF reconstruction can be written: $\hat{I}_{\psi_0} = Z^\text{KL} ((Z^\text{KL})^T T)$

Note: if you don't use parentheses to force NumPy to evaluate $Z^\text{KL} ((Z^\text{KL})^T T)$ rather than $(Z^\text{KL} (Z^\text{KL})^T) T$, it will create an $N \times N$ intermediate matrix rather than a $K_\text{klip}$ dimensional intermediate vector. This can make a big difference in runtime and memory use!

In [24]:

%%timeit
(Z_KL_truncated @ Z_KL_truncated.T) @ target

332 ms ± 13 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

In [25]:

%%timeit
Z_KL_truncated @ (Z_KL_truncated.T @ target)

223 μs ± 621 ns per loop (mean ± std. dev. of 7 runs, 1,000 loops each)

This gives us $\hat{I}_{\psi_0}$ in one line of Python:

In [26]:

Ihat_psi0 = Z_KL_truncated @ (Z_KL_truncated.T @ target)

Step 5: calculate the final image¶

Easy: subtract our Ihat_psi0 from our target and reshape it for display:

In [27]:

plt.imshow((target - Ihat_psi0).reshape(image_shape))
plt.colorbar()

Out[27]:

<matplotlib.colorbar.Colorbar at 0x307ca6750>

It's not totally obvious what we're seeing here. Clearly there is still residual starlight, and we didn't expect to see the planet in a single frame from ground based data. How do we know we did it right?

These data depend on angular differential imaging (ADI) to decorrelate planet signal from speckles, so co-adding the starlight subtracted frames naively still would not help, as we'd be smearing our planet signal over a whole arc of pixels.

Fear not, it is quite easy to apply the last step to combine KLIP with ADI, revealing an exoplanet!

Apply KLIP and ADI¶

Since this is an angular differential imaging dataset, we starlight-subtract our frames individually before rotating them to a common orientation and combining them.

Properly applying KLIP and ADI would require recomputing the $Z^\text{KL}$ basis 61 times for the 61 different images in the dataset that each take a turn as $T$, the target. The construction of the reference set $R$ leaving out $T$ would produce a different $E_{RR}$ and a different set of eigenvectors each time. (For fans of big-$O$ notation, yes, this increases the exponent by one, making a nominally $O(N^3)$ algorithm into $O(N^4)$!)

As we're just trying to demonstrate the application of KLIP, we can compute a single Karhunen-Loève basis from all the observations without that additional complication and still see something interesting.

The dataset from NACO includes an array naco_betapic_preproc['angles'] representing the parallactic angle of the observation.

In [28]:

plt.plot(naco_betapic_preproc['angles'])
plt.xlabel('Image #')
plt.ylabel('Parallactic angle [deg]')

Out[28]:

Text(0, 0.5, 'Parallactic angle [deg]')

We can bring in rotate() from scikit-image to (de-)rotate our starlight-subtracted images before combination.

In [29]:

from skimage.transform import rotate

Now, putting it all together:

In [30]:

# wrapped in a function to not clobber existing variables with these names...
def klip_and_derotate(K_klip):
    # Step 1a, already mean-subtracted
    reference = frame_vecs_meansub
    # Step 2: compute KL transform
    E = reference @ reference.T
    lambda_values_out, C_out = np.linalg.eigh(E)
    lambda_values = np.flip(lambda_values_out)
    C = np.flip(C_out, axis=1)
    Z_KL = reference.T @ (C * np.power(lambda_values, -1/2))
    # Step 3: passed in as K_klip
    Z_KL_truncated = Z_KL[:,:K_klip]
    # Steps 4 and 5 are computed once per frame in the sequence
    derotated_image = np.zeros(image_shape)
    for i in range(frame_vecs_meansub.shape[0]):
        # Step 4: compute the estimate of I
        target = frame_vecs_meansub[i]
        Ihat_psi0 = Z_KL_truncated @ (Z_KL_truncated.T @ target)
        # Step 5: subtract the estimate from the target image
        residual = target - Ihat_psi0
        residual_image = residual.reshape(image_shape)
        # Derotate and add to final KLIP+ADI result
        derotated_image += rotate(residual_image, -naco_betapic_preproc['angles'][i])
    return derotated_image

In [31]:

plt.imshow(klip_and_derotate(30))
plt.colorbar()

Out[31]:

<matplotlib.colorbar.Colorbar at 0x307dc6750>

Look at that! We found our planet! $\beta$ Pictoris b!

As promised, we can now explore the effects of varying $K_\text{klip}$ on ADI data.

In [32]:

plt.figure(figsize=(18, 2.5))
for idx, current_K_klip in enumerate([5, 15, 30, 45, 55]):
    plt.subplot(1, 5, idx + 1)
    plt.imshow(klip_and_derotate(current_K_klip))
    plt.colorbar()
    plt.title(current_K_klip)

Optimizing $K_\text{klip}$ and measuring signal-to-noise¶

As we can see, more modes are not necessarily better. If you watch the color bars, the peak of the planet decreases dramatically as $K_\text{klip}$ increases and retaining 55 modes from our 61 reference PSFs subtracts almost all of the exoplanet signal. The quantity we want to optimize is not the peak planet brightness, nor the noise level in counts, but the signal-to-noise ratio. Clearly, some $5 < K_\text{klip} < 55$ produces the most signal from this dataset.

Regrettably, this computation involves even more tedious bookkeeping than the starlight subtraction itself, but it's not terribly difficult. We just repeat our starlight subtraction at each possible $K_\text{klip}$, measure our signal-to-noise ratio, and choose the best one.

First, we need a rough location for our planet in pixels. We express it in radius (in pixels) and position angle (degrees East of North, a.k.a. counter-clockwise from +Y).

In [33]:

plt.imshow(klip_and_derotate(5))

# show crosshairs on planet
planet_x, planet_y = 62, 63
plt.axhline(planet_y, c='w', ls=':')
plt.axvline(planet_x, c='w', ls=':')

# convert to r and degrees E of N (CCW from +Y)
ctr = (image_shape[0] - 1) / 2
planet_dx, planet_dy = planet_x - ctr, planet_y - ctr
r_px, pa_deg = (
    np.hypot(planet_dx, planet_dy),
    np.rad2deg(np.arctan2(planet_dy, planet_dx)) - 90
)
r_px, pa_deg

Out[33]:

(np.float64(18.398369492974098), np.float64(-42.797401838234194))

The next step is to define a ring of apertures around the star (image center) to measure signal and noise. This is just geometry, $(x, y) = (r \cos \theta, r \sin \theta)$ stuff.

In [34]:

def simple_aperture_locations(r_px, pa_deg, resolution_element_px, 
                              exclude_nearest=0, exclude_planet=False):
    '''Generator yielding aperture centers (x, y) in a ring of radius `r_px`
    and starting at angle `pa_deg` E of N. Unless `exclude_planet` is True,
    the first (x, y) pair gives the planet location (signal aperture).
    
    Specifying `exclude_nearest` > 0 will skip that many apertures 
    from either side of the signal aperture's location'''
    circumference = 2 * r_px * np.pi
    aperture_pixel_diameter = resolution_element_px
    n_apertures = int(circumference / aperture_pixel_diameter)
    start_theta = np.deg2rad(pa_deg + 90)
    delta_theta = np.deg2rad(360 / n_apertures)
    if not exclude_planet:
        yield (r_px * np.cos(start_theta),
               r_px * np.sin(start_theta))
    for i in range(1 + exclude_nearest, n_apertures - exclude_nearest):
        offset_x, offset_y = (
            r_px * np.cos(start_theta + i * delta_theta),
            r_px * np.sin(start_theta + i * delta_theta)
        )
        yield offset_x, offset_y

The size of the aperture should be approximately a resolution element, which for this dataset I just happen to know is about 4 pixels.

In [35]:

fwhm_naco = 4

It'll help to see them:

In [36]:

def show_simple_aperture_locations(image, resolution_element_px, r_px, pa_deg, 
                                   exclude_nearest=0, exclude_planet=False, ax=None):
    '''Plot `image` and overplot the circular apertures of diameter 
    `resolution_element_px` in a ring at radius `r_px`
    starting at `pa_deg` E of N.
    '''
    if ax is None:
        ax = plt.gca()
    ctr = (image.shape[0] - 1) / 2
    im = ax.imshow(image)
    ax.axhline(ctr, color='w', linestyle=':')
    ax.axvline(ctr, color='w', linestyle=':')
    planet_dx, planet_dy = r_px * np.cos(np.deg2rad(90 + pa_deg)), r_px * np.sin(np.deg2rad(90 + pa_deg))
    ax.arrow(ctr, ctr, planet_dx, planet_dy, color='w', lw=2)
    for offset_x, offset_y in simple_aperture_locations(r_px, pa_deg, resolution_element_px, exclude_nearest=exclude_nearest, exclude_planet=exclude_planet):
        ax.add_artist(plt.Circle(
            (ctr + offset_x, ctr + offset_y),
            radius=resolution_element_px/2,
            edgecolor='orange',
            facecolor='none',
        ))

show_simple_aperture_locations(
    klip_and_derotate(5), 
    fwhm_naco, 
    r_px, 
    pa_deg, 
    exclude_planet=True, 
    exclude_nearest=1
)

To do our simple aperture photometry, we'll need a helper for the cartesian coordinates at each pixel.

In [37]:

def cartesian_coords(center, data_shape):
    '''center in x,y order; returns coord arrays xx, yy of data_shape'''
    yy, xx = np.indices(data_shape, dtype=float)
    center_x, center_y = center
    yy -= center_y
    xx -= center_x
    return xx, yy

Now, for every one of our apertures, we want to do something to the pixels inside it.

In [38]:

def reduce_apertures(image, r_px, starting_pa_deg, resolution_element_px, operation, 
                     exclude_nearest=0, exclude_planet=False):
    '''apply `operation` to the pixels within radius `resolution_element_px`/2 of the centers
    of the simple aperture locations for a planet at `r_px` and `starting_pa_deg`, returning
    the locations and the results as a tuple with the first location and result corresponding
    to the planet aperture'''
    center = (image.shape[0] - 1) / 2, (image.shape[0] - 1) / 2
    xx, yy = cartesian_coords(center, image.shape)
    locations = list(simple_aperture_locations(r_px, starting_pa_deg, resolution_element_px, exclude_nearest=exclude_nearest, exclude_planet=exclude_planet))
    simple_aperture_radius = resolution_element_px / 2
    results = []
    for offset_x, offset_y in locations:
        dist = np.sqrt((xx - offset_x)**2 + (yy - offset_y)**2)
        mask = dist <= simple_aperture_radius
        results.append(operation(image[mask]))
    return locations, results

Compute the total signal and the set of noise measurements:

In [39]:

locations, results = reduce_apertures(
    klip_and_derotate(5),
    r_px,
    pa_deg,
    fwhm_naco,
    np.average
)

Note that we've passed np.average as our reduction operation. That means our estimate of the signal is independent of the number of pixels in the circular aperture, which can vary as we go around the circle because of the imprecision in drawing a circle with a discrete pixel grid.

In [40]:

planet_signal, noise_measurements = results[0], results[1:]

The signal-to-noise ratio defined in Mawet et al. (2014) is based on the two-sample Student $t$ test to account for the fact that at small separations, very few apertures will fit in the ring. Using their equation 9 we have

$$ \text{SNR} = \frac{ \bar{x}_1 - \bar{x}_2 }{ s_2 \sqrt{1 + \frac{1}{n_2}} } $$

where $\bar{x}_1$ is the mean of the signal measurements (i.e. the value of the signal aperture sum), $\bar{x}_2$ the mean of the noise measurements (the mean of the noise aperture sums), $s_2$ is the empirical standard deviation of the noise measurements, and $n_2$ is the number of noise measurements in the set.

In [41]:

def calc_snr_mawet(signal, noises):
    '''Calculate signal to noise following the 
    two-sample t test as defined in Mawet 2014'''
    return (
        signal - np.average(noises)
    ) / (
        np.std(noises) * np.sqrt(1 + 1/len(noises))
    )

In [42]:

calc_snr_mawet(planet_signal, noise_measurements)

Out[42]:

np.float64(6.87388019623193)

Now we loop over values for $K_\text{klip} \in [5, 55)$ and see how the signal-to-noise behaves. (I'm also excluding one aperture on either side of the planet location.)

In [43]:

K_klip_vals = np.arange(5, 55)
snrs = np.zeros_like(K_klip_vals)

for idx, K_klip in enumerate(K_klip_vals):
    detection_image = klip_and_derotate(K_klip)
    locations, results = reduce_apertures(
        detection_image, 
        r_px, 
        pa_deg, 
        fwhm_naco, 
        np.sum, 
        exclude_nearest=1
    )
    planet_signal, noise_measurements = results[0], results[1:]
    snrs[idx] = calc_snr_mawet(planet_signal, noise_measurements)

plt.plot(K_klip_vals, snrs)
plt.xlabel(r'$K_{klip}$')
plt.ylabel('SNR')

Out[43]:

Text(0, 0.5, 'SNR')

Selecting the optimal $K_\text{klip}$ is then:

In [44]:

optimal_K_klip = K_klip_vals[np.argmax(snrs)]
optimal_K_klip

Out[44]:

np.int64(9)

And then we can visualize the result:

In [45]:

plt.imshow(klip_and_derotate(optimal_K_klip))
plt.colorbar()

Out[45]:

<matplotlib.colorbar.Colorbar at 0x31417e750>

The Case of the Missing Precision¶

We have 61 references in $R$, meaning $Z^\text{KL}$ contains 61 eigenimages. What happens if we try to subtract all 61 modes?

In [46]:

plt.imshow(klip_and_derotate(61))
plt.colorbar()

Out[46]:

<matplotlib.colorbar.Colorbar at 0x31422e750>

That's weird. Why is it not identically zero across the image? We established above that transforming an observation to the KL basis and back is possible without loss of information.

In the world of math, that would be true, but it turns out that we've lost numerical precision along the way. Forming the covariance matrix $E$ loses precision when small values are added to large ones. Computing the eigenimages from the covariance eigenvectors compounds this error. Then, of course, we're rotating each frame by a different amount, creating the nice circular patterns.

While it's not useful to subtract all possible modes from a dataset when the reference set contains the target image, it's an interesting diagnostic of the numerical error we're accumulating. Can we do better?

Without going into the math (maybe in another blog post), we can obtain the same eigenimages from a singular value decomposition of the data.

In [47]:

u, s, vt = np.linalg.svd(frame_vecs_meansub, full_matrices=False)
Z_KL_svd = vt.T

Repeating the above procedure, we use the right singular vectors of the data as our KL basis:

In [48]:

derotated_image = np.zeros(image_shape)
for i in range(frame_vecs_meansub.shape[0]):
    # Step 4: compute the estimate of I
    target = frame_vecs_meansub[i]
    Ihat_psi0 = Z_KL_svd @ (Z_KL_svd.T @ target)
    # Step 5: once for each target image
    residual = target - Ihat_psi0
    residual_image = residual.reshape(image_shape)
    # Bonus: combine derotated image to generate final KLIP+ADI result
    derotated_image += rotate(residual_image, -naco_betapic_preproc['angles'][i])

plt.imshow(derotated_image)
plt.colorbar()

Out[48]:

<matplotlib.colorbar.Colorbar at 0x314ccb110>

There's still structure in the residuals, but if we compare them on the same color scale it becomes obvious just how much smaller they are:

In [49]:

plt.figure(figsize=(12, 4))
plt.subplot(121)
plt.imshow(klip_and_derotate(61), vmin=-1.5, vmax=1.5)
plt.colorbar()
plt.subplot(122)
plt.imshow(derotated_image, vmin=-1.5, vmax=1.5)
plt.colorbar()

Out[49]:

<matplotlib.colorbar.Colorbar at 0x314e08190>

Four orders of magnitude is not a bad improvement! Why don't we do this all the time?

It's all about runtime:

In [50]:

E = reference @ reference.T
%timeit np.linalg.eigh(E)

306 μs ± 12.3 μs per loop (mean ± std. dev. of 7 runs, 1,000 loops each)

In [51]:

%timeit np.linalg.svd(reference, full_matrices=False)

44.6 ms ± 771 μs per loop (mean ± std. dev. of 7 runs, 10 loops each)

Okay, so the SVD is slower. How much slower?

Computing the eigenvectors of an $n \times n$ matrix costs roughly $O(n^3)$ in time, and the SVD of an $m \times n$ matrix (where $m > n$) costs $O(mn^2)$.

Our covariance matrix is $K \times K$, where $K$ the number of observations we have. (Cost: $O(K^3)$.) The long dimension of our PSF references matrix is $N$, the number of pixels in the image. (Cost: $\min (O(NK^2), O(KN^2))$, since the solver transparently transposes and swaps the left and right singular vectors when it would be faster.) So long as $K \ll N$, it will be advantageous to compute the image-to-image covariance first.

When might that not be true?

In this notebook, we've used all 10,000 pixels from the example NACO data, so $N = N_S = 10000$. In practice, it's possible to optimize the starlight subtraction of particular regions of the image using a matrix of image vectors containing just the pixels in that region, potentially reducing $N$ dramatically.

At some sufficiently large $K$ it will be faster to compute the SVD of $R$ than the eigenvectorization of $E_{RR}$. Where that cutoff is exactly will vary based on hardware and implementation, but the SVD will always win in the limit as $K \rightarrow \infty$ at constant $N$.

Conclusion¶

Hopefully this has demystified starlight subtraction. For the working astronomer's day-to-day use, I recommend PyKLIP, VIP, or PYNPOINT. However, it's always a good idea to know how your tools work!

It's not that hard to discover $\beta$ Pic b, especially when you already know it's there.

In [52]:

plt.imshow(klip_and_derotate(optimal_K_klip))
plt.title('$\\beta$ Pic b')
plt.xticks([])
_ = plt.yticks([])

Joseph Long

Image transformation and interpolation from scratch

The preliminaries¶

The simplest transformation¶

Thing 1: loops¶

Thing 2: loop over dest, not src¶

Coordinate transformations¶

Translation¶

Scaling¶

Rotation¶

Shearing¶

Who's getting transformed, anyway?¶

Nearest-neighbor interpolation¶

Step 1: move the image's center to the origin¶

Step 2: apply rotation¶

Step 3: move the origin back to the center of the image¶

Bilinear: more sophisticated interpolation¶

Improved residuals¶

Missing pixels¶

Bicubic interpolation or cubic convolution: best speed/accuracy tradeoff¶

Oversampling, undersampling, and band limits¶

Oversampled scene¶

The right way to interpolate¶

Bicubic bake-off: comparing scikit-image, OpenCV, scipy.ndimage, and scipy.interpolate

scikit-image interpolation¶

Aside: scikit-image and data types¶

OpenCV¶

scipy.ndimage¶

scipy.interpolate¶

A confusion of x and y¶

Which interpolation function should I use for image processing in Python?¶

A small collection of exoplanet detection illustrations

Adaptive optics system

Transit method

Configuration A

Configuration B

Configuration C

Alternative non-transiting configuration

Radial velocity method

Flat wavefronts approximation

Off-axis source wavefronts

Sky rotation

Alternative figure

Angular differential imaging

Licensing and reuse

Launching CLI apps from the macOS GUI

Step 0. Get the command-line (X11) version of ds9

Step 1. Open Script Editor

Step 2. Edit Info.plist

Step 3. Update file associations (if needed)

Thanks

Updates

Finding a planet by subtracting starlight

Introduction¶

Get the data¶

The preliminaries¶

The algorithm¶

Step 1: subtract image means and select target and reference images¶

Step 2: compute the Karhunen-Loève transform¶

Step 3: truncate the basis set¶

Step 4: estimate the PSF from its projection onto eigenimages¶

Step 5: calculate the final image¶

Apply KLIP and ADI¶

Optimizing $K_\text{klip}$ and measuring signal-to-noise¶

The Case of the Missing Precision¶

Conclusion¶

Thing 2: loop over `dest`, not `src`¶

A confusion of `x` and `y`¶