FRA BUD
Google Pixel 8 Pro 2mm f2.0 1/800s ISO41
LH1336
Related Posts
(Published to the Fediverse as: FRA BUD #photo #plane #fra #bud FRA BUD )
Vernal (Spring) Equinox 2025
Spring has sprung (09:02 UTC, March 20, 2025) for the Northern Hemisphere and Autumn is here if you are south of the Equator. Rendered in Catfood Earth.
Related Posts
- Autumnal Equinox 2024
- Winter Solstice 2024
- Summer Solstice 2024
- Catfood Earth
- Animation of a year of Global Cloud Cover
(Published to the Fediverse as: Vernal (Spring) Equinox 2025 #code #earth #catfood #equinox #spring #vernal #autumn The exact moment (09:02 UTC, March 20, 2025) of the Spring Equinox in Catfood Earth. )
Building a Digital Photo Frame
Google recently bricked my digital photo frame, so I set out to build a new one. I did this in two parts - a Raspberry Pi to display the photos, and a C# script to preprocess them. The second part is optional but worth it.
The display side of the project turned out to be way easier than I thought. There is a utility called fbi that will run a slideshow to a HDMI monitor. Create a boot SD card from the lite version of Raspberry PI OS, copy over your photos and then run:
sudo apt-get update sudo apt-get -y install fbi
You can then connect a monitor and test that images are displaying as expected.
Create a file called launch.sh with the following:
#!/bin/sh # launch.sh # run photos sleep 2m fbi -T 1 -noverbose -mode 1366x768-30 -t 60 -u -blend 1500 -a /home/rob/photos/*.jpg
-T 1 uses the first display, -noverbose just shows the photos, -mode depends on your monitor and is likely safe to omit, -t 60 changes the image ever sixty seconds, -u displays in a random order, -blend 1500 cross fades for 1.5 seconds between images, -a is auto zoom and the path at the end is where ever you copied your photos to.
The sleep 2m command allows the system to complete booting and bring up the login prompt. Without this the photos might start first and then the login shell ends up on top, which is pretty boring.
Make the script executable (chmod 755 launch.sh) and then edit your crontab:
sudo crontab -e
Add the following:
@reboot sh /home/rob/launch.sh >/home/rob/logs/cronlog 2>&1 30 22 * * * /sbin/shutdown -h now
The first line runs the launch.sh script at startup and sends any output to a log file (adjust paths to your system). The second line will shut down the Pi at 10:30pm every day. I use this so I can control when it's running with a smart plug - the plug turns everything on at 7am and off at 10:45pm, and the shutdown prevents the Pi from getting in a bad state from having power removed unceremoniously. If you want it running all the time just omit this line.
Reboot and you should have a working digital photo frame. Fbi will do a reasonable job with your photos, but I wanted something better.
The C# script below preprocesses all my photos to make the best use of the frame. I'm using an old 1366x768 TV so I want all the photos at that resolution and 16x9 aspect ratio. Many of my photos are 4x3, or 3x4, or 9x16, or something else entirely. I don't want any black borders so cropping will be involved.
Cropping is handled by detecting faces and then trying to keep as many in frame as possible. This uses FaceAiSharp.
Photos are processed in two stages. The first stage just crops horizontal photos to fit the screen. Any vertical photos are saved in a list, and then paired using image embeddings (CLIP using this code). My implementation pairs the most similar photos - it would be easy to do the most different as well.
Here's the code. I run this with C# 9 in VSCode on Windows. You'll probably want to change the input and output folders, and possibly the output resolution as well:
| using SixLabors.ImageSharp; | |
| using SixLabors.ImageSharp.Processing; | |
| using SixLabors.ImageSharp.PixelFormats; | |
| using SixLabors.ImageSharp.Formats.Jpeg; | |
| using FaceAiSharp; | |
| using System.Numerics.Tensors; | |
| using Microsoft.ML.OnnxRuntime; | |
| using Microsoft.ML.OnnxRuntime.Tensors; | |
| const string InputFolder = @"D:\photo-frame\input"; | |
| const string OutputFolder = @"D:\photo-frame\output"; | |
| const int OutputWidth = 1366; | |
| const int OutputHeight = 768; | |
| const float OutputAspectRatio = (float) OutputHeight / OutputWidth; | |
| int fileNumber = 0; | |
| Console.WriteLine("photo-frame-process"); | |
| ImageEmbedder imageEmbedder = new ImageEmbedder(); | |
| var faceDetector = FaceAiSharpBundleFactory.CreateFaceDetectorWithLandmarks(); | |
| Random rng = new Random(); | |
| List<MetaImage> metaImages = new List<MetaImage>(); | |
| DirectoryInfo di = new DirectoryInfo(InputFolder); | |
| string[] extensions = { "*.jpg", "*.jpeg", "*.png", "*.gif", "*.webp" }; | |
| List<FileInfo> files = new List<FileInfo>(); | |
| foreach(string ext in extensions) | |
| { | |
| files.AddRange(di.GetFiles(ext, SearchOption.AllDirectories)); | |
| } | |
| foreach(FileInfo file in files) | |
| { | |
| // debug single image | |
| //if (file.Name != "IMG_1023.JPG") { continue; } | |
| Console.Write(file.Name); | |
| using (Image<Rgb24> image = Image.Load<Rgb24>(file.FullName)) | |
| { | |
| image.Mutate(i => i.AutoOrient()); | |
| float aspect = (float) image.Height / image.Width; | |
| // happiest path - same aspect ratio | |
| if (aspect == OutputAspectRatio) | |
| { | |
| Console.WriteLine(" - same aspect ratio"); | |
| image.Mutate(x => x.Resize(OutputWidth, OutputHeight)); | |
| image.SaveAsJpeg(GetNextFileName(), new JpegEncoder {Quality = 90}); | |
| continue; | |
| } | |
| // vertical image | |
| if (image.Width < image.Height) | |
| { | |
| Console.WriteLine(" - vertical image"); | |
| metaImages.Add(new MetaImage(file.FullName, imageEmbedder.Embed(file.FullName))); | |
| continue; | |
| } | |
| // horizontal image | |
| Console.WriteLine(" - horizontal image"); | |
| try | |
| { | |
| FaceAwareResize(image, OutputWidth, OutputHeight); | |
| image.SaveAsJpeg(GetNextFileName(), new JpegEncoder {Quality = 90}); | |
| } | |
| catch (Exception ex) | |
| { | |
| Console.WriteLine($"Error processing {file.FullName}: {ex.Message}"); | |
| } | |
| } | |
| } | |
| while(metaImages.Count > 0) | |
| { | |
| if (metaImages.Count == 1) | |
| { | |
| // one left over image, just resize as best as possible... | |
| MetaImage metaImage = metaImages[0]; | |
| metaImages.Remove(metaImage); | |
| Console.WriteLine($"{Path.GetFileName(metaImage.ImagePath)} - single vertical image"); | |
| using (Image<Rgb24> image = Image.Load<Rgb24>(metaImage.ImagePath)) | |
| { | |
| try | |
| { | |
| image.Mutate(i => i.AutoOrient()); | |
| FaceAwareResize(image, OutputWidth, OutputHeight); | |
| image.SaveAsJpeg(GetNextFileName(), new JpegEncoder {Quality = 90}); | |
| } | |
| catch (Exception ex) | |
| { | |
| Console.WriteLine($"Error processing {metaImage.ImagePath}: {ex.Message}"); | |
| } | |
| } | |
| break; | |
| } | |
| // find two vertical images to combin | |
| MetaImage meta1 = metaImages[rng.Next(metaImages.Count)]; | |
| MetaImage? meta2 = null; | |
| float bestSimilarity = float.MinValue; | |
| // find the second image that is closest to the first image based on cosine similarity | |
| foreach(MetaImage candidate in metaImages) | |
| { | |
| if (candidate == meta1) | |
| { | |
| continue; | |
| } | |
| float similarity = TensorPrimitives.CosineSimilarity(meta1.Embedding, candidate.Embedding); | |
| if (similarity > bestSimilarity) | |
| { | |
| bestSimilarity = similarity; | |
| meta2 = candidate; | |
| } | |
| } | |
| if (meta2 == null) | |
| { | |
| throw new Exception("No second image found"); | |
| } | |
| metaImages.Remove(meta1); | |
| metaImages.Remove(meta2); | |
| Console.WriteLine($"{Path.GetFileName(meta1.ImagePath)} - vertial image paired with {Path.GetFileName(meta2.ImagePath)}"); | |
| try | |
| { | |
| using (Image<Rgb24> image1 = Image.Load<Rgb24>(meta1.ImagePath)) | |
| { | |
| image1.Mutate(i => i.AutoOrient()); | |
| using (Image<Rgb24> image2 = Image.Load<Rgb24>(meta2.ImagePath)) | |
| { | |
| image2.Mutate(i => i.AutoOrient()); | |
| FaceAwareResize(image1, OutputWidth / 2, OutputHeight); | |
| FaceAwareResize(image2, OutputWidth / 2, OutputHeight); | |
| // create a new image with the two images combined | |
| using (Image<Rgb24> combinedImage = new Image<Rgb24>(OutputWidth, OutputHeight)) | |
| { | |
| combinedImage.Mutate(x => x.DrawImage(image1, new Point(0, 0), 1f)); | |
| combinedImage.Mutate(x => x.DrawImage(image2, new Point(OutputWidth / 2, 0), 1f)); | |
| combinedImage.SaveAsJpeg(GetNextFileName(), new JpegEncoder {Quality = 90}); | |
| } | |
| } | |
| } | |
| } | |
| catch (Exception ex) | |
| { | |
| Console.WriteLine($"Error processing {meta1.ImagePath} and {meta2.ImagePath}: {ex.Message}"); | |
| } | |
| } | |
| Console.WriteLine("photo-frame-process done!"); | |
| // utility below | |
| void FaceAwareResize(Image<Rgb24> image, int width, int height) | |
| { | |
| RectangleF? detectRect = null; | |
| var faces = faceDetector.DetectFaces(image); | |
| foreach(var face in faces) | |
| { | |
| if (detectRect.HasValue) | |
| { | |
| detectRect = RectangleF.Union(detectRect.Value, face.Box); | |
| } | |
| else | |
| { | |
| detectRect = face.Box; | |
| } | |
| } | |
| RectangleF coreRect = detectRect ?? new RectangleF(image.Width / 2.0f, image.Height / 2.0f, 0.1f, 0.1f); | |
| // get the center of coreRect as PointF | |
| PointF center = new PointF(coreRect.X + coreRect.Width / 2.0f, coreRect.Y + coreRect.Height / 2.0f); | |
| float targetAspectRatio = (float) width / height; | |
| float imageAspectRatio = (float) image.Width / image.Height; | |
| if (targetAspectRatio >= imageAspectRatio) | |
| { | |
| // figure out the best Y position | |
| float targetHeight = image.Width * ((float)height / width); | |
| float y = center.Y - targetHeight / 2.0f; | |
| if (y < 0) | |
| { | |
| y = 0; | |
| } | |
| else if (y + targetHeight > image.Height) | |
| { | |
| y = image.Height - targetHeight; | |
| } | |
| int intY = (int)y; | |
| int intTargetHeight = (int)targetHeight; | |
| int extra = image.Height - (intY + intTargetHeight); | |
| if (extra < 0) | |
| { | |
| intTargetHeight += extra; | |
| } | |
| Rectangle targetRect = new Rectangle(0, intY, image.Width, intTargetHeight); | |
| // crop to targetRect | |
| image.Mutate(x => x.Crop(targetRect)); | |
| } | |
| else | |
| { | |
| // figure out the best X position | |
| float targetWidth = image.Height * ((float)width / height); | |
| float x = center.X - targetWidth / 2.0f; | |
| if (x < 0) | |
| { | |
| x = 0; | |
| } | |
| else if (x + targetWidth > image.Width) | |
| { | |
| x = image.Width - targetWidth; | |
| } | |
| int intX = (int)x; | |
| int intTargetWidth = (int)targetWidth; | |
| int extra = image.Width - (intX + intTargetWidth); | |
| if (extra < 0) | |
| { | |
| intTargetWidth += extra; | |
| } | |
| Rectangle targetRect = new Rectangle(intX, 0, intTargetWidth, image.Height); | |
| // crop to targetRect | |
| image.Mutate(x => x.Crop(targetRect)); | |
| } | |
| // Resize the image to the target dimensions while keeping the face in the center | |
| image.Mutate(x => x.Resize(new ResizeOptions | |
| { | |
| Size = new Size(width, height) | |
| })); | |
| } | |
| string GetNextFileName() | |
| { | |
| fileNumber++; | |
| return Path.Combine(OutputFolder, $"{fileNumber:00000000}.jpg"); | |
| } | |
| internal class MetaImage | |
| { | |
| public string ImagePath { get; set; } | |
| public float[] Embedding { get; set; } | |
| public MetaImage(string imagePath, float[] embedding) | |
| { | |
| ImagePath = imagePath; | |
| Embedding = embedding; | |
| } | |
| } | |
| /// <summary> | |
| /// Embeds an image using the OpenAI CLIP model. | |
| /// See https://github.com/bartbroere/clip.dll/blob/master/Program.cs | |
| /// </summary> | |
| internal class ImageEmbedder | |
| { | |
| private InferenceSession _model; | |
| public ImageEmbedder() | |
| { | |
| if (!File.Exists("clip-image-vit-32-float32.onnx")) | |
| { | |
| using (HttpClient httpClient = new HttpClient()) | |
| { | |
| var response = httpClient.GetAsync("https://huggingface.co/rocca/openai-clip-js/resolve/main/clip-image-vit-32-float32.onnx").Result; | |
| using (var fs = new FileStream("clip-image-vit-32-float32.onnx", FileMode.CreateNew)) | |
| { | |
| response.Content.CopyToAsync(fs).Wait(); | |
| } | |
| } | |
| } | |
| _model = new InferenceSession("clip-image-vit-32-float32.onnx"); | |
| } | |
| public float[] Embed(string imagePath) | |
| { | |
| return Embed(Image.Load<Rgb24>(File.ReadAllBytes(imagePath))); | |
| } | |
| public float[] Embed(Image<Rgb24> image) | |
| { | |
| var smallestSide = Math.Min(image.Width, image.Height); | |
| image.Mutate(x => x.Crop( | |
| new Rectangle( | |
| (image.Width - smallestSide) / 2, | |
| (image.Height - smallestSide) / 2, | |
| smallestSide, | |
| smallestSide | |
| ))); | |
| image.Mutate(x => x.Resize(224, 224)); | |
| var inputTensor = new DenseTensor<float>(new[] {1, 3, 224, 224}); | |
| for (var x = 0; x < 224; x++) | |
| { | |
| for (var y = 0; y < 224; y++) | |
| { | |
| // Normalize from bytes (0-255) to floats (constants borrowed from CLIP repository) | |
| inputTensor[0, 0, y, x] = Convert.ToSingle((((float) image[x, y].R / 255) - 0.48145466) / 0.26862954); | |
| inputTensor[0, 1, y, x] = Convert.ToSingle((((float) image[x, y].G / 255) - 0.4578275 ) / 0.26130258); | |
| inputTensor[0, 2, y, x] = Convert.ToSingle((((float) image[x, y].B / 255) - 0.40821073) / 0.27577711); | |
| } | |
| } | |
| var inputs = new List<NamedOnnxValue> {NamedOnnxValue.CreateFromTensor("input", inputTensor)}; | |
| var outputData = _model.Run(inputs).ToList().Last().AsTensor<float>().ToArray(); | |
| return outputData; | |
| } | |
| } |
Related Posts
- Style Transfer for Time Lapse Photography
- Fastest image merge (alpha blend) in GDI+
- Predicting when fog will flow through the Golden Gate using ML.NET
- Long term solar powered time lapse camera using Arduino
- Capture DropCam (Nest Cam) frames to Google Drive
(Published to the Fediverse as: Building a Digital Photo Frame #code #c# #raspberrypi #ai #ml How to create a Raspberry Pi based digital photo frame with face aware cropping and AI image pairing. )
Bye Skype
Microsoft is killing Skype in May 2025. I haven't used it for a few years, but I'm going to shed a tear or two.
It's hard to overestimate how important Skype was a little over 20 years ago. I had recently moved to San Francisco and made a lot of phone calls to friends and family back in the UK. There was this thing called a landline, and as well as local service you had to choose a long distance provider with all sorts of complicated tariffs and fees. Cell phones were their own nightmare and even for SMS you paid per text message. Incomprehensibly, ring tones were on their way to being a multi-billion dollar business. My current phone has been on vibrate for its entire life and I travel the world with free high speed Internet included in my plan. Skype marked the beginning of this transition using P2P to open VoIP and video calls to the masses. I even had a cordless phone that made local calls over the phone line but seamlessly switched to Skype for long distance.
Skype's P2P stack could be used for more than phone calls. I built a business to backup your computer over Skype. It's one of the best things I've ever worked on. The Internet was slow for most people. and so you could do a local backup on LAN and then send incrementals P2P via Skype whenever your computer was idle. We also did superior local backups with full history, encryption and locked file support when most competitors choked on PST files. Skype were kind enough to include us as a premium extra in the client. We did some paid search and PR, but the lion's share of our customer acquisition came from this placement.
My current job involves many Teams calls, but before that we operated on hundreds of Skype channels. Before that work was driven by email and conference calls. I'm ambivalent about this shift. Email has some advantages, and hours of video conferences are way more draining than the voice bridge alternative. Teams brilliantly combines channels and chats that look very similar but work completely differently adding an entirely new mental load to the workday. Skype used to display a special cat emoji if you held down three keys at once. I know which philosophy I prefer.
Related Posts
(Published to the Fediverse as: Bye Skype #etc #skype #microsoft Memories of Skype, including building a backup business on it, that mysterious cat emoji, and a landline with Skype built in. We'll miss you! )
Google Photos killed my Aura Frame
I want to do two things with my digital photos. First, keep them safe, especially all those precious memories of random parking meters and unfathomable HVAC mechanisms. Second, enjoy looking at the small subset that are precious family moments. I'm not a special snowflake, these basic requirements should represent a large and competitive market. Unfortunately I'm not holding my breath for much in the way of consumer friendly regulation for the next four years in the US.
Google is making some changes to their Photos API next month, which amount to "Get the fuck out of our Photos API". They're a polite organization so they phrase it a little differently: "We're excited to see the creative solutions developers will build using the new Picker API and the updated Library API.". The developer documentation is a little more pointed: "If your app relies on accessing the user's entire library, you may need to re-evaluate your app or consider alternative approaches."
I have an Aura Carver Mat, a nice digital photo frame that I synced to a shared Google Photos album. Easy for me to add photos, way too easy for the kids to add photos - a fantastic device. As of next month it's ewaste though. I'm not going to upload photos slowly through some picker API like an animal. I'm going to end up building something complicated out of a Raspberry Pi (adds to actuarially unrealistic to do list).
This change doesn't impact backup, because that was already broken. For a while Google Photos nicely integrated with Google Drive and I ended up with a local copy of everything that I could then backup through other means. I'm never going to trust any one company to look after important files and so my philosophy is to backup twice online and once to an external hard drive that lives in a fire safe. (At one point I even built a backup company based on this model).
Google killed the Drive integration and so I MacGyvered together an apps script based solution that used the Google Photos API. This revealed to me that the Photos API would not return location information. Even worse it was impossible to get the full resolution version of a video to download. So it's not like I was in love with the API before this most recent change.
My current approach is a mix of sad and awesome. The sad part is that I use Google Takeout once a month to get an archive of all my online photos. Thankfully this still works. The awesome - I wrote this photo sorter tool that takes the messy download and organizes it by year and month. And I also wrote a volume shadow copy tool that lets you backup a drive without getting hung up on locked files. Those pieces get my photos safely to an external drive, and I upload to Amazon Photos too (the third leg of my backup stool).
Update - fixed with a Raspberry Pi!
Related Posts
- Folder Insights
- Photo Sorter
- How to backup Google Photos to Google Drive automatically after July 2019 with Apps Script
- Capture DropCam (Nest Cam) frames to Google Drive
- Leaving the Nest
(Published to the Fediverse as: Google Photos killed my Aura Frame #etc #google #photos #aura #backup Google is changing their Photos API so my Aura frame is ewaste. Thoughts on using and backing up digital photos. )
Simple Perceptron
Related Posts
- San Francisco Budget - The Missing Manual
- Does closing the Great Highway cause an increase in traffic accidents?
- Life, Non-locality and the Simulation Hypothesis
- Which PG&E rate plan works best for EV charging?
- Building a Digital Photo Frame
(Published to the Fediverse as: Simple Perceptron #code #ml #perceptron #wml Python notebook illustrating a scratch perceptron implementation as well as an sklearn example. )
DEN SFO
Google Pixel 8 Pro 2mm f2.0 1/4,900s ISO42
UA304
Related Posts
(Published to the Fediverse as: DEN SFO #photo #plane #den #sfo DEN SFO )
BNA DEN
Google Pixel 8 Pro 7mm f1.7 1/250s ISO21
UA1048
Related Posts
(Published to the Fediverse as: BNA DEN #photo #plane #bna #den BNA DEN )
SFO BNA
Google Pixel 8 Pro 7mm f1.7 1/950s ISO21
UA499
Related Posts
(Published to the Fediverse as: SFO BNA #photo #plane #sfo #bna SFO BNA )
Adding AI to Todoist with Google Apps Script and OpenAI
This simple script adds useful AI to Todoist and runs in the cloud on Google Apps Script.
I use Todoist to run my life, and my dream is for it to be able to complete certain types of tasks for me. With OpenAI Operator and Anthropic Computer Use this is getting closer and closer to reality. Yes, there is a risk that Todoist spends all of my money on paper clips. But there is also the upside that it will eventually do my weekly shopping, pay my bills and call back the dentist's office. Google's new Ask for Me is promising too, even if right now it's just going to bother nail salons out of existence.
I already put together an Alexa replacement using a Raspberry Pi and the OpenAI realtime API. It switches lights on and off, adds things to my to do list, figures out when the next L is coming and more (I'll blog more about this soon). One thing I learned is that this kind of thing can get pretty expensive. I can see why Amazon is procrastinating on an LLM Alexa. But costs keep going down, and the future will get more evenly distributed over time.
The first version of this script has two objectives. Respond to tasks, and create calendar events. Here's the script:
| // config - ok to leave PERPLEXITY_API_TOKEN blank | |
| const OPENAI_API_TOKEN = ''; | |
| const PERPLEXITY_API_TOKEN = '' | |
| const TODOIST_API_TOKEN = ''; | |
| const AI_TASK_LABEL = 'ai'; | |
| const AI_MESSAGE_PREFIX = 'AI:'; | |
| const MAX_GENERATION_ATTEMPTS = 10; | |
| const OPENAI_MODEL = 'gpt-4o' | |
| const PERPLEXITY_MODEL = 'sonar-pro' | |
| const IMAGE_MIME_TYPES = [ | |
| 'image/jpeg', | |
| 'image/jpg', | |
| 'image/png', | |
| 'image/webp', | |
| 'image/gif' | |
| ]; | |
| function trigger() { | |
| runAssistant(); | |
| } | |
| function runAssistant() { | |
| // process all open tasks with the AI_TASK_LABEL label | |
| tasks = getAiTasks(); | |
| tasks.forEach(task => { | |
| checkTask(task); | |
| }); | |
| } | |
| function checkTask(task) { | |
| // if the user was the last message then we generate a response | |
| var responseNeeded = true; | |
| const comments = getComments(task.id); | |
| comments.forEach(comment => { | |
| if (comment?.content) { | |
| responseNeeded = !comment.content.startsWith(AI_MESSAGE_PREFIX); | |
| } | |
| }); | |
| if(responseNeeded){ | |
| processTask(task, comments); | |
| } | |
| } | |
| function processTask(task, comments) { | |
| // build the conversation from the task and comments... | |
| // the message array is in openai chat completion format | |
| const messages = []; | |
| const now = new Date(); | |
| const timeZone = Session.getScriptTimeZone(); | |
| const timeZoneName = Utilities.formatDate(now, timeZone, "z"); | |
| // Format the date. This pattern outputs: "August 10, 2025 20:00:00 PST" | |
| const formattedDate = Utilities.formatDate(now, timeZone, "MMMM dd, yyyy HH:mm:ss z"); | |
| messages.push({ | |
| role: 'developer', | |
| content: `You are a helpful assistant that works with Todoist tasks. You are given the current task and any comments and try to help as best as you can. If the user is researching you respond with the information they're looking for. If you have a tool that can help with the task you call it. If you believe that you have fully completed the task then you call the complete_task function to close it. The current task ID is ${task.id}. The current date and time is ${formattedDate}.` | |
| }); | |
| if (task?.content) { | |
| messages.push({ | |
| role: 'user', | |
| content: task.content | |
| }); | |
| } | |
| if (task?.description) { | |
| messages.push({ | |
| role: 'user', | |
| content: task.description | |
| }); | |
| } | |
| comments.forEach(comment => { | |
| if (comment?.content) { | |
| if (comment.content.startsWith(AI_MESSAGE_PREFIX)) { | |
| // AI message | |
| messages.push({ | |
| role: 'assistant', | |
| content: comment.content.substring(AI_MESSAGE_PREFIX.length).trim() | |
| }); | |
| } else { | |
| // User message - might include an image | |
| content = [] | |
| // the text part of the comment | |
| content.push({type: 'text', text: comment.content}); | |
| // if there is an immage attachment add it | |
| if (comment.attachment) { | |
| if (IMAGE_MIME_TYPES.includes(comment.attachment.file_type)) { | |
| var url = ''; | |
| if (comment.attachment.tn_l) { | |
| // use large thumbnail if it exists | |
| url = comment.attachment.tn_l[0]; | |
| } else { | |
| // full attachment - may fail if too large | |
| url = comment.attachment.file_url; | |
| } | |
| // download to base 64 - openai can't access Todoist attachments from the URL | |
| // see https://platform.openai.com/docs/guides/vision#uploading-base64-encoded-images | |
| base64 = getAttachment(url); | |
| content.push({ | |
| type: 'image_url', | |
| image_url: { | |
| url: `data:${comment.attachment.file_type};base64,${base64}` | |
| } | |
| }); | |
| } | |
| } | |
| messages.push({ | |
| role: 'user', | |
| content: content | |
| }); | |
| } | |
| } | |
| }); | |
| if (messages[messages.length - 1].role = 'user') { | |
| // if the user is the last in the thread generate an AI response | |
| generateAiResponse(messages, task.id, timeZoneName); | |
| } | |
| } | |
| function generateAiResponse(messages, taskId, timeZoneName) { | |
| const payload = { | |
| model: OPENAI_MODEL, | |
| messages: messages, | |
| tools: getTools(timeZoneName) | |
| }; | |
| // loop to allow for tool use | |
| for(var retry = 0; retry < MAX_GENERATION_ATTEMPTS; retry++) { | |
| const options = { | |
| method: 'post', | |
| contentType: 'application/json', | |
| headers: { | |
| 'Authorization': 'Bearer ' + OPENAI_API_TOKEN | |
| }, | |
| payload: JSON.stringify(payload) | |
| }; | |
| const response = UrlFetchApp.fetch('https://api.openai.com/v1/chat/completions', options); | |
| const result = JSON.parse(response.getContentText()); | |
| if (result.choices && result.choices.length > 0) { | |
| if (result.choices[0].message.tool_calls && result.choices[0].message.tool_calls.length > 0) { | |
| // we have at least one tool request, add the requst message to the conversation | |
| payload.messages.push(result.choices[0].message); | |
| // run all the tools... | |
| result.choices[0].message.tool_calls.forEach(tool_call => { | |
| const args = JSON.parse(tool_call['function'].arguments); | |
| var result = ''; | |
| Logger.log(`Calling ${tool_call['function'].name} (call ID ${tool_call.id})`) | |
| switch(tool_call['function'].name) { | |
| case 'create_event': | |
| try { | |
| result = createCalendarAppointment(args.title, args.start, args.end, args?.description, args?.location, args?.guests); | |
| } catch (error) { | |
| result = error.message; | |
| } | |
| break; | |
| case 'complete_task': | |
| try { | |
| result = closeTask(args.task_id); | |
| } catch (error) { | |
| result = error.message; | |
| } | |
| break; | |
| case 'answer_question': | |
| try { | |
| result = generatePerplexityResponse(args.question); | |
| } catch (error) { | |
| result = error.message; | |
| } | |
| break; | |
| default: | |
| result = 'Unknown function?!'; | |
| break; | |
| } | |
| Logger.log(`Tool response: ${result.substring(0, 50)}...`) | |
| payload.messages.push({ | |
| "role": "tool", | |
| "tool_call_id": tool_call.id, | |
| "content": result | |
| }); | |
| }) | |
| continue; | |
| } else { | |
| // message back from AI, post it as a comment to the task | |
| const aiMessage = result.choices[0].message.content; | |
| Logger.log(`AI Response: ${aiMessage.substring(0, 50)}...`) | |
| addComment(taskId, AI_MESSAGE_PREFIX + ' ' + aiMessage) | |
| break; | |
| } | |
| } | |
| } | |
| } | |
| function generatePerplexityResponse(question) { | |
| const messages = []; | |
| messages.push({ | |
| role: 'system', | |
| content: 'You are an artificial intelligence assistant and you answer questions for your user. Your answer will be interpreted by another AI so do not inclue any formating or special text in your answer. Be brief and answer the question in a single concise paragraph. You never ask any clarifying questions or suggest any follow up, just respond as best as you can.' | |
| }); | |
| messages.push({ | |
| role: 'user', | |
| content: question | |
| }); | |
| const payload = { | |
| model: PERPLEXITY_MODEL, | |
| messages: messages | |
| }; | |
| const options = { | |
| method: 'post', | |
| contentType: 'application/json', | |
| headers: { | |
| 'Authorization': 'Bearer ' + PERPLEXITY_API_TOKEN | |
| }, | |
| payload: JSON.stringify(payload) | |
| }; | |
| const response = UrlFetchApp.fetch('https://api.perplexity.ai/chat/completions', options); | |
| const result = JSON.parse(response.getContentText()); | |
| return result.choices[0].message.content; | |
| } | |
| function getTools(timeZoneName) { | |
| tools = []; | |
| // complete a todoist task by ID, call closeTask() | |
| tools.push({ | |
| "type": "function", | |
| "function": { | |
| "name": "complete_task", | |
| "description": "Closes or completes a task", | |
| "parameters": { | |
| "type": "object", | |
| "properties": { | |
| "task_id": { | |
| "type": "string", | |
| "description": "The ID of the task to complete" | |
| } | |
| }, | |
| "required": ["task_id"] | |
| } | |
| } | |
| }); | |
| // create a meeting on the user's calendar, call createCalendarAppointment() | |
| tools.push({ | |
| "type": "function", | |
| "function": { | |
| "name": "create_event", | |
| "description": "Adds an event to the user's calendar. The start and end timestamps must be in 'MMMM D, YYYY HH:mm:ss z' javascript format", | |
| "parameters": { | |
| "type": "object", | |
| "properties": { | |
| "title": { | |
| "type": "string", | |
| "description": "Name of the calendar event, i.e. 'Lunch with Bob'." | |
| }, | |
| "start": { | |
| "type": "string", | |
| "description": `Start time for the event, i.e. 'August 10, 2025 20:00:00 ${timeZoneName}', assume ${timeZoneName} if the user does not specify` | |
| }, | |
| "end": { | |
| "type": "string", | |
| "description": `End time for the event, i.e. 'August 10, 2025 21:00:00 ${timeZoneName}', assume ${timeZoneName} if the user does not specify, assume 1 hour duration if no end time or length is given` | |
| }, | |
| "description": { | |
| "type": "string", | |
| "description": "Optional description for the event" | |
| }, | |
| "location": { | |
| "type": "string", | |
| "description": "Optional location for the event" | |
| }, | |
| "guests": { | |
| "type": "string", | |
| "description": "Optional comma separated list of email addresses to invite to the event. Never provide an email address unless the user specifically provided it" | |
| }, | |
| }, | |
| "required": ["title", "start", "end"] | |
| } | |
| } | |
| }); | |
| if (PERPLEXITY_API_TOKEN){ | |
| tools.push({ | |
| "type": "function", | |
| "function": { | |
| "name": "answer_question", | |
| "description": "Answers a question using Internet search via the Perplexity Sonar API. Use this for information after your knowlege cutoff date, to reserch questions you do not know the answer to, or for local search.", | |
| "parameters": { | |
| "type": "object", | |
| "properties": { | |
| "question": { | |
| "type": "string", | |
| "description": "The question to answer, i.e. 'How many stars are there in the galaxy?'" | |
| } | |
| }, | |
| "required": ["question"] | |
| } | |
| } | |
| }); | |
| } | |
| return tools; | |
| } | |
| function createCalendarAppointment(title, start, end, description, location, guests) { | |
| options = {} | |
| if (description?.length > 0) { | |
| options.description = description; | |
| } | |
| if (location?.length > 0) { | |
| options.location = location; | |
| } | |
| if (guests?.length > 0) { | |
| options.guests = guests; | |
| options.sendInvites = true; | |
| } | |
| CalendarApp.getDefaultCalendar().createEvent(title, | |
| new Date(start), | |
| new Date(end), | |
| options); | |
| return `Calendar event ${title} has been created.`; | |
| } | |
| function closeTask(taskId) { | |
| const options = { | |
| method: 'post', | |
| headers: { | |
| 'Authorization': 'Bearer ' + TODOIST_API_TOKEN | |
| } | |
| }; | |
| UrlFetchApp.fetch(`https://api.todoist.com/rest/v2/tasks/${taskId}/close`, options); | |
| return `Task ${taskId} has been closed. You don't need to do anything else and can move to your next step.`; | |
| } | |
| function addComment(taskId, comment) { | |
| const payload = { | |
| task_id: taskId, | |
| content: comment | |
| }; | |
| const options = { | |
| method: 'post', | |
| contentType: 'application/json', | |
| headers: { | |
| 'Authorization': 'Bearer ' + TODOIST_API_TOKEN | |
| }, | |
| payload: JSON.stringify(payload) | |
| }; | |
| UrlFetchApp.fetch('https://api.todoist.com/rest/v2/comments', options); | |
| } | |
| function getComments(taskId) { | |
| var response = UrlFetchApp.fetch(`https://api.todoist.com/rest/v2/comments?task_id=${encodeURIComponent(taskId)}`, { | |
| headers: { | |
| Authorization: 'Bearer ' + TODOIST_API_TOKEN | |
| } | |
| }); | |
| return JSON.parse(response.getContentText()); | |
| } | |
| function getAiTasks() { | |
| var response = UrlFetchApp.fetch(`https://api.todoist.com/rest/v2/tasks?label=${encodeURIComponent(AI_TASK_LABEL)}`, { | |
| headers: { | |
| Authorization: 'Bearer ' + TODOIST_API_TOKEN | |
| } | |
| }); | |
| return JSON.parse(response.getContentText()); | |
| } | |
| function getAttachment(url) { | |
| var options = { | |
| method: "get", | |
| headers: { | |
| "Authorization": "Bearer " + TODOIST_API_TOKEN | |
| } | |
| }; | |
| var response = UrlFetchApp.fetch(url, options); | |
| var fileBlob = response.getBlob(); // Get the file as a Blob | |
| var base64String = Utilities.base64Encode(fileBlob.getBytes()); // Convert Blob to Base64 | |
| return base64String; | |
| } |
To get this working you need API keys from OpenAi and Todoist. Perplexity is optional, if you have a key add it at the top. It only works with tasks that have the right label, ai is the default - you can change this with AI_TASK_LABEL. I initially developed this with o1, but it looks like the tool use was rushed out too quickly and it calls the same tool repeatedly. GPT-4o works well enough and you can test switching out the model by changing OPENAI_MODEL.
Quick configuration guide - create a Google Apps Script project in Google Drive. Paste in the code above and add your API keys and any other desired configuration changes. Go to Project Settings in the right hand navigation and make sure your time zone is correct. Then go to Triggers and schedule the function trigger to run periodically (I use every 5 minutes). You should be done.
Back in Todoist add the ai label to a task (or whatever label you set in the script) and the AI will respond. With the current script there are two use cases - ask it to create an event (it can invite people, add details to the description, etc.), or ask it to research some aspect of the task you're working on. I think this is helpful because it has the full context of the task, and while you're working in Todoist it's useful to store the history there as well.
The point here is to extend the number of tasks that the script can take on. Add new tools for the AI to consider in the getTools() function, and then wire that tool into an implementation in generateAIResponse(). createCalendarAppointment() is a good example of using built in Google services - as well as the calendar it's pretty easy to interact with email, Google docs and many more. I'm planning to add file uploads as well, and will update this post with iterations of the script that add helpful functionality.
OpenAI recommends around 20 tools as the maximum. At that point it might be necessary to break this up into multiple assistants with different tool collections.
Let me know in the comments if you manage to use this to start crossing anything off your list.
Updated 2025-02-17 01:18:
Updated the script to support images and Perplexity.
Image support takes advantage of multimodal input. Any image attachments will be sent as part of the conversation. This uses the large thumbnail in Todoist by default. It supports JPEG, GIF, PNG and WEBP. If a thumbnail is not available it will send the full size image without resizing, depending on how large this might not be accepted by OpenAI.
Perplexity is implemented as a tool (so OpenAI is always called, and it may optionally call out to Perplexity to get more information). This is useful for web search, local search and knowledge past the training cutoff for the OpenAI model. It's optional, if you don't include a Perplexity API key then it just won't be used.
Here's a simple use case - add a photo of a book and ask Todoist to find the URL where you can order it on Kindle.
More Google Apps Script Projects
- Get an email when your security camera sees something new (Apps Script + Cloud Vision)
- Get an email if your site stops being mobile friendly (no longer available)
- Export Google Fit Daily Steps, Weight and Distance to a Google Sheet
- Email Alerts for new Referers in Google Analytics using Apps Script
- Animation of a year of Global Cloud Cover
- Control LIFX WiFi light bulbs from Google Apps Script
- How to backup Google Photos to Google Drive automatically after July 2019 with Apps Script
- Using the Todoist API to set a due date on the Alexa integration to-do list (with Apps Script)
- Automate Google PageSpeed Insights and Core Web Vitals Logging with Apps Script
- Using the Azure Monitor REST API from Google Apps Script
- Monitor page index status with Google Sheets, Apps Script and the Google Search Console API
(Published to the Fediverse as: Adding AI to Todoist with Google Apps Script and OpenAI #code #ai #openai #todoist #ml #appsscript #gas #google #perplexity How to add an AI assistant to Todoist, includes code to respond to tasks and create calendar appointments with gpt-4o. )
