TimelapseThing - A Toolset for Timeseries JPGs


By: Shawn O'Neil



Preface

I'd like to preface this page by saying that there isn't much here that is very novel; for the most part this is my own foray into creative timelapse videography, and the creative bits that I have come up with are mostly small modifications of known techniques. In particular, a1ex at magiclantern.fm deserves credit for pioneering the use of curve-fitting to exposure values over time as a method of deflickering, and Matt Molloy's work on image stacking inspired my simple use of this technique in timelapses.

Introduction

TimelapseThing is a Python script that includes a number of tools for working with timelapse image series (in JPG format). It can be used to deflicker images (if some images in the series are mildy under- or over-exposed), merge a series of JPGs into a video, batch crop and resize, expand a series into a longer series, and lighten/darken/median merge using a sliding window approach (see below) for artistic effect and artifact removal (such as someone walking across your timelapse and getting caught in a frame). The heavy lifting is done using ImageMagick's convert tool, and ffmpeg for video creation, so these tools are required.

A note on usage: I usually use Magic Lantern firmware for Canon cameras for an intervalometer; the sunset experiments below use my camera's auto-exposure setting (aperture priority with fixed ISO, though I'm currently exploring shutter-speed priority with auto ISO and aperture which I think will be better). I will soon start playing with HDR timelapses with preprocessing using Photomatix, and I also occasionally do some post-processing in Lightroom (and there are also some good tutorials and plugins available for creating timelapses from within Lightroom).

Finally, it's a good idea to crop and resize the images to 1280x720 (for 720p) or 1920x1080 (for 1080p) first to minimize filesizes and computational time while working!


Figure 1: The basic window, and an example of the end result.

The sections below cover the various functions, starting with the coolest: deflickering and window merging.

Deflickering

The deflickering algorithm first analyzes each picture, estimating its exposure. This is done by either considering the mean exposure value of each pixel in the image, or the median. While the latter may in some situations be more robust, in my tests the choice makes very little difference, but the mean method is much faster and so this is the recommended default. Next, the algorithm adjusts the exposure of each image in an effort to produce a smooth exposure series over the timeframe. The exposures call be adjusted to match the first few frames (Flat), adjusted to match a linear fit of the original exposure series (Linear), or to match the original exposure series smoothed using a sliding window based approach (Original). Here I'd like to acknowledge a1ex at magiclantern.fm who developed this idea using the median/Linear combination.

Figure 2: The Deflicker window, and exposure curves before and after deflickering.

This window also allows the user to modify an exposure adjustment smoothly over time in exposure stops. For example, the Original.Plus0to1EV curve in figure 2 is the Original smoothed curve with +0EV added to the first frame through +1EV added to the last. There is also a "None" deflickering option so that this adjustment can be applied independently of deflickering.

Figure 3: Before and after deflickering (Original.Plus0to1EV). Hit play on both quickly, or see these in sync here.

If you'd like to see the original video side-by-side with the other methods, here are some videodoubler links:

One thing I've unfortunately noticed is that some residual flicker is very hard to get rid of. I believe this is because although I can adjust the overall exposure, exposure flicker changes not only the overall exposure of an image but also its distribution. Ideally I could redistribute the entire exposure range somehow (perhaps something like a quantile normalization method), but I haven't figured out how to implement that yet.

Window Merging

This idea is drawn from Matt Molloy's work on image stacking. If a timelapse looks so good when the images are stacked, why not stack subsets of the images in a sliding window and then animate the results (e.g. images 1-10 produce a frame, 2-11 produce a frame, and so on)?

As a bonus, we have the option for various kinds of stacking: we can merge images and select the mean of the pixel values, the median (very slow computationally), the lighter of the pixel values, or the darker of the pixel values, for various interesting effects. The size of the window selected will also have an effect.

Figure 4: Lightening (left) and darkening (right) sliding-window based merging.

Notice that in the above, the darkened version removes car lights whereas the lightened version extends them. This approach can also be used for moving star trails, though here the the star trails wouldn't 'grow' but would be a fixed length (determined by the window size) and move.

The median option can also be used to, in theory at least, remove single-frame artifacts such as people walking across the frame. By considering sets of windows and selecting the median pixel for each set, artifacts should be removed with minimal changes to the image. The following figure provides a crude example (where the shutter angle is too small for the timelapse) where the median pixel is selected with a small window size of 5 frames.

Figure 5: Removing an artifact (mid-way through left video) using a sliding-window median pattern. The stuttered motion is caused by the shutter speed being too fast relative to the frame rate.

Expansion

I was interested in expanding the length of my timelapses by interpolating frames (though in reality this is best done by selecting an appropriate frame rate interval and shutter speed, using neutral density filters (or more creative solutions) if necessary to allow for long exposure times). After some experimentation, I settled on interpolating images by using a sliding gaussian distribution, where we interpolate as a weighted average of frames given a standard deviation (the mean moves over the frames in steps that determine the number of new frames). To save on computation, this is only done over frames within a certain window of the mean:


Figure 6: Gaussian interpolation of image timeseries.

To see the effect of this, compare these videos: the right video contains twice as many frames as the former, with a 1-image standard deviation interpolation. If taken to extremes, this expansion can create an interesting effect, though it's fairly reminiscent of cheap tv slow-mo shots:

Figure 6: Video expansion. On the left, a video shot at 60fps played back at 24fps. On the right, an expansion of 2x in length with standard deviation of 4 images.

Crop and Resize

The only other feature worth discussing in detail at this time is the batch crop/resize tool. This tool allows you to select a new size for your images (say, 1280x720, for 720p) even if the new aspect ratio isn't equal to the old aspect ratio. If the aspect ratios aren't equal, the new aspect ratio will be calculated and the largest fitting crop with that ratio will be selected from the original frames. If the gravity option is set to "Center" it will be left alone, if set to "North" the new frame will be pushed to the top (if it isn't already there), if "West" it will pushed to the left (if not already there), etc. After that, the selected areas are cropped and resized to produce the output.


Figure 7: Crop and resize configuration.

In the future, I may include a sharpening option and/or a Ken Burns-style zoom and pan effect.

Miscellaneous

In general, the tool will ask the user to select a series of images to operate on, and a directory for the output images to be written to. The names of written images will always be of the form 000001.jpg, 000002.jpg, and so on; if it looks like images will overwrite existing images, the user will be prompted. For usability, the default input/output directory is the most recently selected directory. When inputting a list of images, the order is usually important (for example, in window merging, they need to ordered correctly) -- I believe the order used is the order they are shown by the selection window. Usually this is the correct numeric order. I may add an option to sort them internally, but this would assume a particular naming convention (such as 000001.jpg, 000002.jpg, etc.) which may change for different camera output (IMG_3134.JPG, IMG_3135.JPG, etc.).

The "Lighten/Darken Merge" button on the main interface can be used to layer merge a series of JPG images into a single composed JPGs in the style of Matt Molloy's work on image stacking.

When creating a video, the input files must be named like 000001.jpg, 000002.jpg, etc. For this reason there is also a tool which copies a series of files and gives them names in this style.

Download

If you make something cool with TimelapseThing, how about including a link back here? Download

Also, there are some new "beta" features that you'll see that aren't documented here. They might be quite slow and/or require lots of temporary data space, so I suggest testing them with small timelapses first.

TimelapseThing requires the ImageMagick suite, and ffmpeg if you want to use it for video creation (note that I'm still working on getting the settings correct for video creation, currently the command run is ffmpeg -y -r FRAMERATE -i INDIRECTORY/%06d.jpg -b:v 12000k -filter:v scale='trunc(oh*a*2)/2:HEIGHT' OUTNAME.mp4).

If you are working on a mac with Mountain Lion, you might find these links handy: ImageMagick and ffmpeg. In particular, the former provides an installer for ImageMagic, and the latter provides ffmpeg, which needs to be copied into the appropriate system location. After installing ImageMagick, you can get ffmpeg and TimelapseThing by opening a Terminal window and typing the following (note that I make no gaurantees as to the security of the software you are downloading from other sources, nor do I vouch that the links referenced below will continue to work).

wget 'http://ffmpegmac.net/resources/SnowLeopard_Lion_Mountain_Lion_17.03.2013.zip' -O ffmpeg.zip
unzip ffmpeg.zip
sudo mv ffmpeg /usr/local/bin
sudo chown root:wheel /usr/local/bin/ffmpeg
sudo chmod 755 /usr/local/bin/ffmpeg
rm -f ffmpeg.zip

wget http://shawntoneil.com/data/code/timelapsething/timelapsething.py     
chmod +x timelapsething.py
				

Then you should be able to open the finder and find timelapsething.py in your home directory. Being a Mac/Linux user, this is all the help I can really provide, sorry Windows folks ;)