Style Transfer for Time Lapse Photography

Updated on Saturday, February 19, 2022

Style Transfer for Time Lapse Photography

TensorFlow Hub has a great sample for transferring the style of one image to another. You might have seen Munch's The Scream applied to a turtle, or Hokusai's The Great Wave off Kanagawa to the Golden Gate Bridge. It's fun to play with and I wondered how it would work for a timelapse video. I just posted my first attempt, four different shots of San Francisco and I think it turned out pretty well.

The four sequences were all shot on an A7C, one second exposure, ND3 filter and aperture/ISO as needed to hit one thousandth of a second before fitting the filter. Here's an example shot:

Style Transfer for Time Lapse Photography

I didn't want The Scream or The Wave, so I asked my kids to pick two random pieces of art each so I could have a different style for each sequence:

Style Transfer for Time Lapse Photography

The style transfer network wants a 256x256 style image so I cropped and resized the art as needed.

The sample code pulls images from URLs. I modified it to connect to Google Drive, iterate through a source folder of images and write the transformed images to a different folder. I'm running this in Google Colab which has the advantage that you get to use Google's GPUs and the disadvantage that it will disconnect, timeout, run out of memory etc. To work around this the modified code can be run as many times as needed to get through all the files and will only process input images that don't already exist in the output folder. Here's a gist of my adapted colab notebook:

One final problem is that the style transfer example produces square output images. I just set the output to 1920x1920 and then cropped a HD frame out the middle of each image to get around this.

Here's a more detailed workflow for the project:

I usually shoot timelapse with a neutral density filter to get some nice motion blur. When I shot this sequence it was the first time I'd used my filter on a new camera/lens and screwing in the filter threw off the focus enough to ruin the shots. Lesson learned - on this camera I need to nail the focus after attaching the filter. As I've been meaning to try style transfer for timelapse I decided to use this slightly bad sequence as the input. Generally for timelapse I shoot manual / manual focus, fairly wide aperture and ISO 100 unless I need to bump this up a bit to get to a 1 second exposure with the filter.

After shooting I use LRTimelapse and Lightroom 6 to edit the RAW photos. LRTimelapse reduces flicker and works well for applying a pan and/or zoom during processing as well. For this project I edited before applying the style transfer. The style transfer network preserves detail very well and then goes crazy in areas like the sky. Rather than zooming into those artifacts I wanted to keep them constant which I think gives a greater sense of depth as you zoom in or out.

Once the sequence is exported from Lightroom I cancel out of the LRTimelapse render window and switch to Google Colab. Copy the rendered sequence to the input folder and the desired style image and then run the notebook to process. If it misbehaves then Runtime -> Restart and run all is your friend.

To get to video I use ffmpeg to render each sequence. For this project at 24 frames per second and cropping a 1920x1080 frame from each of the 1920x1920 style transfer images.

Then DaVinci Resolve to edit the sequences together. I added a 2 second cross dissolve between each sequence and a small fade at the very beginning and end.

Finally, music. I use Filmstro Pro and for this video I used the track Durian.

Here's the result:

Add your comment...

Related Posts

You Might Also Like

(All Code Posts)

San Francisco New Year's Eve Timelapse 2020

Updated on Saturday, February 19, 2022

San Francisco New Year's Eve Timelapse 2020

A timelapse of San Francisco on New Year's Eve 2020:

Shot from Treasure Island, Corona Heights Park. Fort Baker and near Battery 129 in the Marin Headlands. Each sequence was transformed with a style transfer neural network (full details).

(Previously: San Francisco New Year's Eve Timelapse)

Add your comment...

Related Posts

You Might Also Like

(More Timelapses)

Book reviews for December 2020

The Kingdom: A novel by Jo Nesbø

The Kingdom: A novel by Jo Nesbø

5/5

 

The Redeemer (Harry Hole, #6) by Jo Nesbø

The Redeemer (Harry Hole, #6) by Jo Nesbø

3/5

 

Measure What Matters: How Google, Bono, and the Gates Foundation Rock the World with OKRs by John E. Doerr

Measure What Matters: How Google, Bono, and the Gates Foundation Rock the World with OKRs by John E. Doerr

4/5

 

Police (Harry Hole, #10) by Jo Nesbø

Police (Harry Hole, #10) by Jo Nesbø

4/5

 

Knife (Harry Hole, #12) by Jo Nesbø

Knife (Harry Hole, #12) by Jo Nesbø

5/5

 

Breaking the Two-Party Doom Loop: The Case for Multiparty Democracy in America by Lee Drutman

Breaking the Two-Party Doom Loop: The Case for Multiparty Democracy in America by Lee Drutman

4/5

 

Ready Player Two (Ready Player One, #2) by Ernest Cline

Ready Player Two (Ready Player One, #2) by Ernest Cline

3/5

 

Add your comment...

Related Posts

You Might Also Like

(All Reviews)

Abrigo Valley

Updated on Sunday, May 23, 2021

Abrigo Valley

Abrigo Valley

Abrigo Valley Trail to Santos Trail returning on Briones Crest Trail, a pleasant three and a half mile hike in Briones Regional Park.

(Previously: Mariposa Loop)

Hike starts at: 37.927261, -122.158768. View in Google Earth.

Add your comment...

Related Posts

You Might Also Like

(Hike Map)

Merced

Updated on Saturday, May 8, 2021

Merced

Merced

Merced

Merced

Lake Merced in San Francisco (Previously: Lake Merced).

Add your comment...

Related Posts

You Might Also Like

(Recent Photos)

Grabtown Gulch and Borden Hatch Mill in Purisima Creek

Updated on Saturday, May 8, 2021

Grabtown Gulch and Borden Hatch Mill in Purisima Creek

Grabtown Gulch and Borden Hatch Mill in Purisima Creek

Grabtown Gulch and Borden Hatch Mill in Purisima Creek

A five mile loop in Purisima Creek Redwoods Open Space Preserve. Super limited parking so a good one for an early start. Descend Grabtown Gulch Trail for a little over two miles (and it doesn't really stop going down). Then after about five seconds along Purisima Creek ascend Borden Hatch Mill Trial all the way back to the top.

Hike starts at: 37.422216, -122.338163. View in Google Earth.

Add your comment...

Related Posts

You Might Also Like

(Hike Map)

Great Conjunction

Updated on Wednesday, May 12, 2021

Great Conjunction

Great Conjunction

2020 Great Conjunction of Jupiter and Saturn, including the Galilean moons.

Add your comment...

Related Posts

You Might Also Like

(Recent Photos)

Winter Solstice 2020

Winter Solstice 2020

Winter starts now in the Northern Hemisphere, Summer to the south. 10:03 UTC Dec 21 2020 rendered in Catfood Earth.

(Previously: Winter Solstice 2019)

Add your comment...

Related Posts

You Might Also Like

(All Code Posts)

Accessing Printer Press ESC to cancel

Updated on Wednesday, March 1, 2023

ESC

Minutes spent so far this year waiting for Excel to talk to a printer: about 290.

Lifetime Excel print jobs: maybe 2?

How to fix Accessing Printer Press ESC to cancel:

  1. Press Start, search for Printer and run the Printers & scanners setting item.
  2. Find and click Microsoft Print to PDF and click Manage.
  3. Click Set as default.

The only real disadvantage is you'll need to select a different printer when you actually need to print anything (which you probably don't).

Add your comment...

Related Posts

You Might Also Like

(All Etc Posts)

Book reviews for November 2020

The Saints of Salvation by Peter F. Hamilton

The Saints of Salvation by Peter F. Hamilton

4/5

 

Dead Lies Dreaming (The Laundry Files, #10) by Charles Stross

Dead Lies Dreaming (The Laundry Files, #10) by Charles Stross

4/5

 

Poseidon's Wake (Poseidon's Children) by Alastair Reynolds

Poseidon's Wake (Poseidon's Children) by Alastair Reynolds

3/5

 

The Sentinel (Jack Reacher, #25) by Lee Child

The Sentinel (Jack Reacher, #25) by Lee Child

4/5

 

Add your comment...

Related Posts

You Might Also Like

(All Reviews)