ITHCWY: Robert Ellison's Blog

Stars over Lake Tahoe

Timelapse of stars over Lake Tahoe, California

4K timelapse of stars and the Milky Way over Lake Tahoe, California.

Email Alerts for new Referers in Google Analytics using Apps Script

Referral Traffic in Google Analytics

It's useful to know when you have a new website referrer. Google Analytics is plagued with spam referral and you want to filter this out of reporting as quickly as possible to stop it from skewing your data. It's also helpful to be able to respond quickly to new referral traffic - maybe leave a comment or promote the new link on social media.

The script below will send you a daily email with links to any new referrers.

var TableId = 'ga:your-view-id';
var SendEmailTo = 'your-email-address';

function main() {
  var scriptProperties = PropertiesService.getScriptProperties();
  var currentProps = scriptProperties.getProperties();
  var anythingNew = false;
  var newText = '';
  
  var yesterday = Utilities.formatDate(new Date(new Date().getTime() - 24 * 60 * 60 * 1000), Session.getTimeZone(), 'yyyy-MM-dd');
  var options = {
    'dimensions': 'ga:fullReferrer',
    'filters': 'ga:medium==referral',
    'max-results': 20000
  };
  var report = Analytics.Data.Ga.get(TableId, yesterday, yesterday, 'ga:sessions', options);
  if (report.rows) {
    for (var i = 0; i < report.totalResults; i++) {
      if (!(report.rows[i][0] in currentProps)) {
        Logger.log('Found new referrer: ' + report.rows[i][0]);
        scriptProperties.setProperty(report.rows[i][0], report.rows[i][1]);
        anythingNew = true;
        newText += 'New referrer: ' + report.rows[i][0] + '\r\n';
      }
    }
  } else {
    Logger.log('GA report is empty');
  }
  
  if (anythingNew) {
    MailApp.sendEmail(SendEmailTo, 'Found new referrers for ' + TableId + ' on ' + new Date(), newText);
  }
}

Start a new apps script project in Google Drive and paste in the code. At the top enter the view ID that you want to monitor and the email address that should receive reports.

Choose Advanced Google Services from the Resources menu and switch on the Google Analytics API. Then click the Google API Console link and enable the Google Analytics API there as well.

Finally pick Current project's triggers from the Edit menu and trigger the main function daily at a convenient time.

This script saves known referrers in script properties. For a site with lots of traffic this may run out of space in which case you might need to switch this out and write known referrers to a sheet instead.

Reading and Writing Office 365 Excel from a Console app using the Microsoft.Graph C# Client API

I needed a console app that reads some inputs from an online Excel workbook, does some processing and then writes back the results to a different worksheet. Because I enjoy pain I decided to use the thinly documented new Microsoft.Graph client library. The sample code below assumes that you have a work or education Office 365 subscription.

using System;
using System.Collections.Generic;
using System.Globalization;
using System.IO;
using System.Linq;
using System.Net.Http.Headers;
using System.Threading.Tasks;
using Microsoft.Graph;
using Microsoft.IdentityModel.Clients.ActiveDirectory;
using Newtonsoft.Json.Linq;

namespace Excel365Test
{
    /// <summary>
    /// 1) Install Microsoft.Graph NuGet Package
    /// 2) Install Microsoft.IdentityModel.Clients.ActiveDirectory NuGet Package
    /// 3) Register app at https://portal.azure.com/ - need app ID and redirct URL below
    /// </summary>
    class Program
    {
        static void Main(string[] args)
        {
            TokenCache tokenCache = new TokenCache();

            // load tokens from file 
            string tokenPath = Path.Combine(Environment.GetFolderPath(Environment.SpecialFolder.LocalApplicationData), "Excel365Test");
            if (!Directory.Exists(tokenPath)) { Directory.CreateDirectory(tokenPath); }
            tokenPath = Path.Combine(tokenPath, "tokens.dat");
            if (System.IO.File.Exists(tokenPath))
            {
                tokenCache.Deserialize(System.IO.File.ReadAllBytes(tokenPath));
            }

            // this is the OAUTH 2.0 TOKEN ENDPOINT from https://portal.azure.com/ -> Azure Active Directory -> App Registratuons -> End Points
            var authenticationContext = new AuthenticationContext("https://login.windows.net/your-url-here/", tokenCache);

            // only prompt when needed, you'll get a UI the first time you run
            var platformParametes = new PlatformParameters(PromptBehavior.Auto);

            var authenticationResult = authenticationContext.AcquireTokenAsync("https://graph.microsoft.com/",
                "your-app-id",     // Application ID from https://portal.azure.com/
                new Uri("http://some.redirect.thing/"),         // Made up redirect URL, also from https://portal.azure.com/
                platformParametes).Result;
            string token = authenticationResult.AccessToken;

            // save token so we don't need to re-authorize
            System.IO.File.WriteAllBytes(tokenPath, tokenCache.Serialize());
            
            // use the token with Microsoft.Graph calls
            GraphServiceClient client = new GraphServiceClient(new DelegateAuthenticationProvider(
            (requestMessage) =>
            {
                requestMessage.Headers.Authorization = new AuthenticationHeaderValue("bearer", token);

                return Task.FromResult(0);
            }));

            // test reading from a sheet - in this case I have a test worksheet with a two column table for name/value pairs
            var readSheet = client.Me.Drive.Items["your-workbook-id"].Workbook.Worksheets["test"];
            var readTables = readSheet.Tables.Request().GetAsync().Result;
            string readTableId = readTables[0].Name;
            var table = readSheet.Tables[readTableId].Rows.Request().GetAsync().Result;
            
            // convert page to a dictionary... this doesn't handle pagination
            Dictionary<stringdecimal> tableValues = table.CurrentPage.ToDictionary(r => r.Values.First.First.ToString(), 
                r => Convert.ToDecimal(r.Values.First.Last, CultureInfo.InvariantCulture));

            // test adding a row to a table with four columns
            // sadly it seems you need this exact format, a regular JArray or JObject fails

            WorkbookTableRow newRow = new WorkbookTableRow
            {
                Values = JArray.Parse("[[\"1\",\"2\",\"3\",\"4\"]]")
            };
            
            var outputSheet = client.Me.Drive.Items["your-workbook-id"].Workbook.Worksheets["data"];
            var outputTables = outputSheet.Tables.Request().GetAsync().Result;
            string outputTableId = outputTables[0].Name;
            var outputResult = outputSheet.Tables[outputTableId].Rows.Request().AddAsync(newRow).Result;

            // the excel unit tests seem to be the most useful documentation right now:
            // https://github.com/microsoftgraph/msgraph-sdk-dotnet/blob/dev/tests/Microsoft.Graph.Test/Requests/Functional/ExcelTests.cs
        }
    }
}

Paste the code into a new console project and then follow the instructions at the top to add the necessary NuGet packages. You'll also need to register an application at https://portal.azure.com/. You want a Native application and you'll need the Application ID and the redirect URL (just make up some non-routable URL for this). Under Required Permissions for the app you should add read and write files delegated permissions for the Microsoft Graph API.

Hope this saves you a few hours. Comment below if you need a more detailed explanation for any of the above.

Point Reyes Deer

A deer at Point Reyes

A deer on a treacherous cliff-top path at Point Reyes.

Host change

I'm switching hosts so there will be various DNS changes and some downtime today.

Point Reyes Lighthouse

Point Reyes Lighthouse

Summer Solstice 2017

Summer Solstice 2017

Summer starts now (or Winter for the Southern Hemisphere). Rendered in Catfood Earth.

(Previously, Previously, Previously, Previously)

Spherical Sunset

Spherical sunset timelapse 360 degree video

An experimental spherical timelapse of a sunset looking west over the Pacific from West Portal, San Francisco.

In your browser use w, a, s, d to tilt and pan. Best viewed in a VR headset.

Shot on the Richo Theta S, post processed with Lightroom, LRTimelapse and FFmpeg.

Book reviews for May 2017

Change Agent by Daniel Suarez

Change Agent by Daniel Suarez

5/5

Stonking near future bio thriller.

 

As a courtesy

As a courtesy to the next passenger

I'm on a recently built A340-600. This sign is about as useful as the ashtrays. This must be a weird tradition that gets handed down from airplane to airplane from one sign author who got grossed out by the thought of a moist sink but has never squelched around in piss on the lower deck stink fest that is installed on this particularly strange airbus.

It's like someone loved the whole elegant spiral staircase up to a bar motif of the 747 and thought wouldn't it be a giggle to do the exact opposite.