Style Transfer for Time Lapse Photography

By Robert Ellison. Updated on Saturday, February 19, 2022.

TensorFlow Hub has a great sample for transferring the style of one image to another. You might have seen Munch's The Scream applied to a turtle, or Hokusai's The Great Wave off Kanagawa to the Golden Gate Bridge. It's fun to play with and I wondered how it would work for a timelapse video. I just posted my first attempt, four different shots of San Francisco and I think it turned out pretty well.

The four sequences were all shot on an A7C, one second exposure, ND3 filter and aperture/ISO as needed to hit one thousandth of a second before fitting the filter. Here's an example shot:

I didn't want The Scream or The Wave, so I asked my kids to pick two random pieces of art each so I could have a different style for each sequence:

The style transfer network wants a 256x256 style image so I cropped and resized the art as needed.

The sample code pulls images from URLs. I modified it to connect to Google Drive, iterate through a source folder of images and write the transformed images to a different folder. I'm running this in Google Colab which has the advantage that you get to use Google's GPUs and the disadvantage that it will disconnect, timeout, run out of memory etc. To work around this the modified code can be run as many times as needed to get through all the files and will only process input images that don't already exist in the output folder. Here's a gist of my adapted colab notebook:

{ "nbformat": 4, "nbformat_minor": 0, "metadata": { "colab": { "name": "Style Transfer for Timelapse.ipynb", "provenance": [], "authorship_tag": "ABX9TyNSFwNMHz6Mtpw3zLReIFU+", "include_colab_link": true }, "kernelspec": { "name": "python3", "display_name": "Python 3" }, "accelerator": "GPU" }, "cells": [ { "cell_type": "markdown", "metadata": { "id": "view-in-github", "colab_type": "text" }, "source": [ "<a href=\"https://colab.research.google.com/gist/abfo/c2fe8af9976ae09a324f2cc410a86008/style-transfer-for-timelapse.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>" ] }, { "cell_type": "markdown", "metadata": { "id": "HEHPFmEEV7Yk" }, "source": [ "Apply style transfer to a set of images in Google Drive. Based on the style transfer guide [here](https://www.tensorflow.org/hub/tutorials/tf2_arbitrary_image_stylization) and the drive example [here](https://colab.research.google.com/notebooks/io.ipynb#scrollTo=RWSJpsyKqHjH). This code expects a folder in the root of your drive called stylegan. The folder should contain two subfolders, input (images to style), output and a style image called style.jpg. This should be 256x256 pixels. The sample code works with square images and will take the center of the source image if in a different aspect ratio. For my purposes I'm using 4k frames and then setting the output size to 1920x1920 in order to crop a 1920x1080 HD frame from the processed impages.:" ] }, { "cell_type": "code", "metadata": { "id": "CUGH5d1o5jPh" }, "source": [ "content_img_size = (1920, 1920)\r\n", "style_img_size = (256, 256) " ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": { "id": "rcj_QQd75oMH" }, "source": [ "Set up tensorflow and some support functions:" ] }, { "cell_type": "code", "metadata": { "id": "4Qtbh0rXFcCd" }, "source": [ "# Copyright 2019 The TensorFlow Hub Authors. All Rights Reserved.\r\n", "#\r\n", "# Licensed under the Apache License, Version 2.0 (the \"License\");\r\n", "# you may not use this file except in compliance with the License.\r\n", "# You may obtain a copy of the License at\r\n", "#\r\n", "# http://www.apache.org/licenses/LICENSE-2.0\r\n", "#\r\n", "# Unless required by applicable law or agreed to in writing, software\r\n", "# distributed under the License is distributed on an \"AS IS\" BASIS,\r\n", "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\r\n", "# See the License for the specific language governing permissions and\r\n", "# limitations under the License.\r\n", "# ==============================================================================\r\n", "\r\n", "from google.colab import drive\r\n", "from google.colab import files\r\n", "import functools\r\n", "import os\r\n", "from matplotlib import gridspec\r\n", "import matplotlib.pylab as plt\r\n", "import numpy as np\r\n", "import tensorflow as tf\r\n", "import tensorflow_hub as hub\r\n", "import os\r\n", "import PIL.Image\r\n", "\r\n", "print(\"TF Version: \", tf.__version__)\r\n", "print(\"TF-Hub version: \", hub.__version__)\r\n", "print(\"Eager mode enabled: \", tf.executing_eagerly())\r\n", "print(\"GPU available: \", tf.test.is_gpu_available())\r\n", "\r\n", "def crop_center(image):\r\n", " \"\"\"Returns a cropped square image.\"\"\"\r\n", " shape = image.shape\r\n", " new_shape = min(shape[1], shape[2])\r\n", " offset_y = max(shape[1] - shape[2], 0) // 2\r\n", " offset_x = max(shape[2] - shape[1], 0) // 2\r\n", " image = tf.image.crop_to_bounding_box(\r\n", " image, offset_y, offset_x, new_shape, new_shape)\r\n", " return image\r\n", "\r\n", "@functools.lru_cache(maxsize=None)\r\n", "def load_image(image_url, image_size=(256, 256), preserve_aspect_ratio=True):\r\n", " \"\"\"Loads and preprocesses images.\"\"\"\r\n", " # Cache image file locally.\r\n", " #image_path = tf.keras.utils.get_file(os.path.basename(image_url)[-128:], image_url)\r\n", " image_path = image_url\r\n", " # Load and convert to float32 numpy array, add batch dimension, and normalize to range [0, 1].\r\n", " img = plt.imread(image_path).astype(np.float32)[np.newaxis, ...]\r\n", " if img.max() > 1.0:\r\n", " img = img / 255.\r\n", " if len(img.shape) == 3:\r\n", " img = tf.stack([img, img, img], axis=-1)\r\n", " img = crop_center(img)\r\n", " img = tf.image.resize(img, image_size, preserve_aspect_ratio=True)\r\n", " return img\r\n", "\r\n", "def show_n(images, titles=('',)):\r\n", " n = len(images)\r\n", " image_sizes = [image.shape[1] for image in images]\r\n", " w = (image_sizes[0] * 6) // 320\r\n", " plt.figure(figsize=(w * n, w))\r\n", " gs = gridspec.GridSpec(1, n, width_ratios=image_sizes)\r\n", " for i in range(n):\r\n", " plt.subplot(gs[i])\r\n", " plt.imshow(images[i][0], aspect='equal')\r\n", " plt.axis('off')\r\n", " plt.title(titles[i] if len(titles) > i else '')\r\n", " plt.show()\r\n", "\r\n", "def tensor_to_image(tensor):\r\n", " tensor = tensor*255\r\n", " tensor = np.array(tensor, dtype=np.uint8)\r\n", " if np.ndim(tensor)>3:\r\n", " assert tensor.shape[0] == 1\r\n", " tensor = tensor[0]\r\n", " return PIL.Image.fromarray(tensor)" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": { "id": "bZwqFmPM43tQ" }, "source": [ "Load the hub module, this will take a few minutes" ] }, { "cell_type": "code", "metadata": { "id": "V5KZbbG-47hk" }, "source": [ "hub_handle = 'https://tfhub.dev/google/magenta/arbitrary-image-stylization-v1-256/2'\r\n", "hub_module = hub.load(hub_handle)" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": { "id": "NM-z5lKXGBQf" }, "source": [ "Open drive (you'll need to allow access each session, and load the style image)" ] }, { "cell_type": "code", "metadata": { "id": "T9759bbNIp8l" }, "source": [ "drive.mount('/content/drive', force_remount=True)" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "metadata": { "id": "7QWqAgulWd3Z" }, "source": [ "style_image = load_image('/content/drive/MyDrive/stylegan/style.jpg', style_img_size)\r\n", "style_image = tf.nn.avg_pool(style_image, ksize=[3,3], strides=[1,1], padding='SAME')\r\n", "show_n([style_image], ['Style image'])" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": { "id": "GAyE0dNiD44E" }, "source": [ "Process all the files... This might take a while and sometimes fails so it won't process if the output file alredy exists. You can run this cell as many times as needed to process all the input files (if you get out of memory errors try restarting the runtime and running the cells above again). This will run much faster if you can grab a GPU instance. If you change anything (like the style image) you'll want to clear the output folder and run again. " ] }, { "cell_type": "code", "metadata": { "id": "GK9ifHT4D_s5" }, "source": [ "directory = '/content/drive/MyDrive/stylegan/input'\r\n", "outdirectory = '/content/drive/MyDrive/stylegan/output'\r\n", "for file in os.listdir(os.fsencode(directory)):\r\n", " filename = os.fsdecode(file)\r\n", " outputfile = os.path.join(outdirectory, filename)\r\n", " if (os.path.isfile(outputfile)):\r\n", " continue\r\n", " print(filename)\r\n", " content_image = load_image(os.path.join(directory, filename), content_img_size, preserve_aspect_ratio=True)\r\n", " outputs = hub_module(tf.constant(content_image), tf.constant(style_image))\r\n", " stylized_image = outputs[0]\r\n", " tensor_to_image(stylized_image).save(outputfile)\r\n", "\r\n", "#make sure files are saved when we're done\r\n", "print('Done, disconnecting from drive...')\r\n", "drive.flush_and_unmount()" ], "execution_count": null, "outputs": [] } ] }

One final problem is that the style transfer example produces square output images. I just set the output to 1920x1920 and then cropped a HD frame out the middle of each image to get around this.

Here's a more detailed workflow for the project:

I usually shoot timelapse with a neutral density filter to get some nice motion blur. When I shot this sequence it was the first time I'd used my filter on a new camera/lens and screwing in the filter threw off the focus enough to ruin the shots. Lesson learned - on this camera I need to nail the focus after attaching the filter. As I've been meaning to try style transfer for timelapse I decided to use this slightly bad sequence as the input. Generally for timelapse I shoot manual / manual focus, fairly wide aperture and ISO 100 unless I need to bump this up a bit to get to a 1 second exposure with the filter.

After shooting I use LRTimelapse and Lightroom 6 to edit the RAW photos. LRTimelapse reduces flicker and works well for applying a pan and/or zoom during processing as well. For this project I edited before applying the style transfer. The style transfer network preserves detail very well and then goes crazy in areas like the sky. Rather than zooming into those artifacts I wanted to keep them constant which I think gives a greater sense of depth as you zoom in or out.

Once the sequence is exported from Lightroom I cancel out of the LRTimelapse render window and switch to Google Colab. Copy the rendered sequence to the input folder and the desired style image and then run the notebook to process. If it misbehaves then Runtime -> Restart and run all is your friend.

To get to video I use ffmpeg to render each sequence. For this project at 24 frames per second and cropping a 1920x1080 frame from each of the 1920x1920 style transfer images.

Then DaVinci Resolve to edit the sequences together. I added a 2 second cross dissolve between each sequence and a small fade at the very beginning and end.

Finally, music. I use Filmstro Pro and for this video I used the track Durian.

Here's the result:

Add your comment...

(All Code Posts)

(Published to the Fediverse as: Style Transfer for Time Lapse Photography #code #ml #tensorflow #drive #python #video How to apply the TensorFlow Hub style transfer to every frame in a timelapse video using Python and Google Drive. )

code, ml, tensorflow, drive, python, video

San Francisco New Year's Eve Timelapse 2020

By Robert Ellison. Updated on Saturday, February 19, 2022.

A timelapse of San Francisco on New Year's Eve 2020:

Shot from Treasure Island, Corona Heights Park. Fort Baker and near Battery 129 in the Marin Headlands. Each sequence was transformed with a style transfer neural network (full details).

(Previously: San Francisco New Year's Eve Timelapse)

Add your comment...

(More Timelapses)

(Published to the Fediverse as: San Francisco New Year's Eve Timelapse 2020 #timelapse #sanfrancisco #video San Francisco New Year's Eve Timelapse 2020 (style transfer neural network version). )

timelapse, san francisco, video

Book reviews for December 2020

By Robert Ellison.

The Kingdom: A novel by Jo Nesbø

5/5

The Redeemer (Harry Hole, #6) by Jo Nesbø

3/5

Measure What Matters: How Google, Bono, and the Gates Foundation Rock the World with OKRs by John E. Doerr

4/5

Police (Harry Hole, #10) by Jo Nesbø

4/5

Knife (Harry Hole, #12) by Jo Nesbø

5/5

Breaking the Two-Party Doom Loop: The Case for Multiparty Democracy in America by Lee Drutman

4/5

Ready Player Two (Ready Player One, #2) by Ernest Cline

3/5

Add your comment...

(All Reviews)

reviews, lee drutman, ernest cline

Abrigo Valley

By Robert Ellison. Updated on Sunday, May 23, 2021.

Abrigo Valley Trail to Santos Trail returning on Briones Crest Trail, a pleasant three and a half mile hike in Briones Regional Park.

(Previously: Mariposa Loop)

Hike starts at: 37.927261, -122.158768. View in Google Earth.

Add your comment...

(Hike Map)

(Published to the Fediverse as: Abrigo Valley #hike #map Briones Regional Park hike taking in Abrigo Valley Trail, Santos Trail and Briones Crest Trail. 3.5 miles. )

hike, map

Merced

By Robert Ellison. Updated on Saturday, January 18, 2025.

Lake Merced in San Francisco (Previously: Lake Merced).

The photograph titled "Merced" captures a serene and tranquil scene along the edge of a lake or river. A group of seagulls is perched on a weathered railing, their feathers a muted blend of grays and browns, perfectly complementing the overcast sky and gentle colors of the water and foliage. The scene unfurls into the distance, with tall grasses lining the banks, merging elegantly with a dense woodland of dark green trees. The seagulls are a focal point, poised with an air of quiet contemplation, while others are caught mid-flight, adding a sense of motion to the still composition. The setting is imbued with an understated beauty, portraying a moment of calm in the lap of nature.

The photograph employs a naturalistic style with an emphasis on harmony and balance. The composition adheres to the rule of thirds, with the railing and seagulls occupying a prominent horizontal line within the frame. This alignment immediately draws the viewer's eye towards the avian subjects, and the soft diagonal of the riverbank adds depth and direction. I appreciate the gentle color palette and the way the details are captured with clarity yet without overwhelming contrast. However, the lighting could have been slightly more dynamic to bring out variations in the texture of the feathers and landscape. Overall, the image succeeds in its quiet elegance, capturing a moment of peace with simplicity and grace.

Add your comment...

(Recent Photos)

(Published to the Fediverse as: Merced #photo #lakemerced #sanfrancisco Photos from Lake Merced in San Francisco, California. )

photo, lake merced, san francisco

Grabtown Gulch and Borden Hatch Mill in Purisima Creek

By Robert Ellison. Updated on Saturday, May 8, 2021.

A five mile loop in Purisima Creek Redwoods Open Space Preserve. Super limited parking so a good one for an early start. Descend Grabtown Gulch Trail for a little over two miles (and it doesn't really stop going down). Then after about five seconds along Purisima Creek ascend Borden Hatch Mill Trial all the way back to the top.

Hike starts at: 37.422216, -122.338163. View in Google Earth.

Add your comment...

(Hike Map)

(Published to the Fediverse as: Grabtown Gulch and Borden Hatch Mill in Purisima Creek #hike #purisimacreek #map Awesome 5 mile hike in Purisima Creek, down Grabtown Gulch Trail and then back up Borden Hatch Mill Trial. Lots of down and then lots of up. )

hike, purisima creek, map

Great Conjunction

By Robert Ellison. Updated on Saturday, January 18, 2025.

2020 Great Conjunction of Jupiter and Saturn, including the Galilean moons.

Great Conjunction

The photograph titled "Great Conjunction" captures a serene evening sky, with a mesmerizing blend of colors that transition smoothly from a deep indigo at the top to soft oranges near the horizon. In the center of this celestial pasture, two bright celestial bodies—a fascinating conjunction—hang delicately. The horizon is lined with the silhouette of trees and gentle hills, offering a dark contrast to the vibrant sky. Wispy clouds streak subtly across the frame, adding texture and depth to the expansive sky, mirroring the quiet presence of the earth below.

The composition of the photograph adheres to the principles of simplicity and symmetry, with the focal point being the conjunction of the two planets, which draws the eye upwards. The horizon is positioned in the lower third, following the rule of thirds, which adds balance and allows the viewer to absorb the vastness of the sky. The soft gradients of color create a soothing atmosphere, inviting contemplation of the wonders above. What is particularly commendable is the subtlety of the light's transition, which enhances the tranquil mood. However, there could be more interplay with the foreground to anchor the scene a bit more, offering a counterbalance to the celestial focus. This photograph succeeds in evoking a sense of awe, capturing not just a moment, but an experience of cosmic beauty.

Add your comment...

(Recent Photos)

(Published to the Fediverse as: Great Conjunction #photo #saturn #jupiter #conjunction Photo of the 2020 Great Conjunction including Jupiter, Saturn and the Galilean moons. Shot from San Francisco, California. )

photo, saturn, jupiter, conjunction

Winter Solstice 2020

By Robert Ellison.

Winter starts now in the Northern Hemisphere, Summer to the south. 10:03 UTC Dec 21 2020 rendered in Catfood Earth.

(Previously: Winter Solstice 2019)

Add your comment...

(All Code Posts)

code, winter, solstice, catfood, earth

Accessing Printer Press ESC to cancel

By Robert Ellison. Updated on Monday, October 28, 2024.

ESC

Minutes spent so far this year waiting for Excel to talk to a printer: about 290.

Lifetime Excel print jobs: maybe 2?

How to fix Accessing Printer Press ESC to cancel:

Press Start, search for Printer and run the Printers & scanners setting item.
Find and click Microsoft Print to PDF and click Manage.
Click Set as default.

The only real disadvantage is you'll need to select a different printer when you actually need to print anything (which you probably don't).

Add your comment...

(All Etc Posts)

(Published to the Fediverse as: Accessing Printer Press ESC to cancel #etc #excel #microsoft How to fix the annoying Accessing Printer Press ESC to cancel wait in Excel by setting a different default printer in Windows 10. )

etc, excel, microsoft