Widgets and Scraping

Goodreads does still offer embeddable shelf widgets, which are served as a JavaScript file containing an HTML string assigned to a variable.

It’s not the prettiest thing in the world, but it’s something to fetch and work with:

curl -sS "$GOODREADS_WIDGET" -o "$tmpdir/widget.js"

The widget URL encodes your shelf preferences as query parameters (number of books, sort order, which fields to show, &c).

This script uses a default URL pointed at my read shelf, but it can be overridden via the first argument or a GOODREADS_WIDGET environment variable, which means you could generate posts for other shelves like to-be-read, or the like.

Extracting the HTML

The JS file looks roughly like:

var widget_code = '<div>...escaped HTML...</div>'

First, we need to strip the JS wrapper and “unescape” the four escape patterns Goodreads uses (\n, \/, \", \'). I had never used sed all that much, but it’s pretty powerful with a touch of regex:

head -1 "$tmpdir/widget.js" \
  | sed "s/^[[:space:]]*var widget_code = '//; s/'[[:space:]]*$//" \
  | sed 's/\\n/\n/g' \
  | sed "s/\\\\\"/\"/g" \
  | sed "s/\\\\'/'/g" \
  | sed 's|\\\/|/|g' \
  > "$tmpdir/widget.html"

The whole widget is on one line, so the shorthand head -1 pulls the entire thing. Once that line is in stdout, it’s fairly straightforward to pipe sed substitutions until we have readable HTML.

Parsing the Books

Each book in the widget HTML is wrapped in a gr_custom_each_container div. csplit splits the HTML into individual files on that pattern:

csplit -sz -f "$tmpdir/book_" "$tmpdir/widget.html" '/gr_custom_each_container/' '{*}'

Then we loop over those files and extract the pieces we need with grep and fancy regex:

full_title=$(grep -oP '<a title="\K[^"]+' "$block" | head -1)
title=$(grep -A2 'gr_custom_title' "$block" | grep -oP '>\K[^<]+(?=</a>)' | head -1)
author=$(grep -A2 'gr_custom_author' "$block" | grep -oP '>\K[^<]+(?=</a>)' | head -1)
img=$(grep 'gr_custom_book_container' -A5 "$block" | grep -oP 'src="\K[^"]+' | head -1)
rating=$(grep -o 'gr_red_star_active' "$block" | wc -l)

The rating is derived by counting active star images rather than reading a numeric value; granted, it’s a little roundabout and brittle, but it works.

Each book gets written out as a line of NDJSON with jq -nc.

Finding What’s New

The comparison logic here is slightly different from the Discogs plugin. There is no date field explicitly, but the data is returned by date descending, so we can rely on the order of the keys to determine what’s new. We first build a composite key for each book in both the fetched collection and the local DB, which is just the author and title joined with ||.

for f in "$l_keys" "$f_keys"; do
  sed 's/\r$//; s/[[:space:]]\+$//' "$f" > "$f.tmp" && mv "$f.tmp" "$f"
done
grep -Fvxf "$l_keys" "$f_keys" > "$tmpdir/new.keys"

Then we trim any trailing whitespace and standardize line endings to avoid false mismatches, which allows us to do a simple grep to find the new keys based on line mismatches:

Building the Post

One piece the Goodreads plugin has that Discogs doesn’t is a small helper to render a numeric rating as a Unicode star string:

stars() {
  local r=$1 s=""
  for ((i=1; i<=5; i++)); do
    [ "$i" -le "$r" ] && s+="★" || s+="☆"
  done
  echo "$s"
}

The rest of the post-building follows the same pattern as my Discogs plugin (frontmatter, then a block per book with cover image, title, author, and stars):

echo "![]($img){width=150px}"
echo "## $title"
echo "**${author}**"
echo "Rating: $star_display"

New books get prepended to the local NDJSON database, and off we go!

Scraping a widget is obviously less reliable than a real API; however, the alternative is a manual solution which isn’t nearly as fun.

-CH

Setup

The whole process lives in plugins/discogs.sh, a Bash script invoked by my top-level (and very creatively named) plugin manager publish.sh, which loads environment variables and fires each plugin in sequence:

./discogs.sh syrchanan db

The script accepts two positional arguments:

1. A Discogs username (in this case, syrchanan).
2. An output directory (in this case, db).

It also looks for two optional environment variables:

• DISCOGS_API_TOKEN: A personal API token for the Discogs API, which allows for higher rate limits than the free usage allotment.
• PAGELIM: A limit on how many pages of collection data to fetch (each page contains 100 records, the default of 1 should be plenty for sporadic updates).

Fetching the Collection

The Discogs API exposes a user’s collection at /users/{username}/collection/folders/0/releases, sorted by date added descending.

The script iterates through the pages (up to $PAGELIM), writing each release as a compact JSON object to a temporary NDJSON¹ file:

jq -c '.releases[]? as $r | $r.basic_information as $b | {
  added:    $r.date_added,
  title:    $b.title,
  year:     (try ($b.year|tonumber) catch null),
  formats:  $b.formats[0],
  labels:   $b.labels[0],
  artists:  $b.artists[0],
  genres:   $b.genres,
  styles:   $b.styles,
  img:      $b.cover_image
}' "$resp" >> "$outfile"

This results in a file with one line per release, such that each line is a cleaned up JSON object with the fields I care about. Since the API returns your collection sorted by date descending, the newest releases are at the top of the file – a feature that becomes important for the next step.

Looking for New Records

The local “database” is also just an NDJSON file on disk, which contains every release that has already been posted². Also, rather than comparing full objects, I build a composite key for each release, comprised of the date added, artist name, and title — joined with ||:

jq -r '[(.added // ""), ((.artists.name) // ""), (.title // "")]
  | map(gsub("\\n"; " ") | gsub("\\r"; ""))
  | join("||")' "$fetched" | sort -ur > "$f_keys"

Both the fetched collection and the local DB get this same format. Next, the script walks through the fetched keys from newest to oldest, writing any that don’t match the most recent local key into a new.keys file. It stops at the first match, which is a little fragile, but still works because everything is sorted by date descending.

If new.keys is empty, the script exits cleanly with no output.

Building the Automated Post

For each new record in our newly minted new.keys file, we pull the full object back out of the fetched data, extracts the fields, and append a markdown block to a new Quarto document:

filename="posts/auto/$(date -I)-discogs.qmd"

cat > "$filename" << QMD_HEAD
---
title: "New Records"
date: $(date -I)
categories:
  - Automated Updates
  - Music
draft: false
---
QMD_HEAD

Each record then gets a cover image, an H2 markdown (##) heading, and a markdown table with the label, year, catalog number, format, and genre:

echo "![]($img){width=300px}"
echo "## $title"
echo "**$artists**"
echo "| Label  | $label |"
echo "| Year   | $year  |"
# ...&c &c

Updating the Local DB

Once the post is written, the new releases get prepended to the local NDJSON database so we don’t double count any records the next time the script runs:

cat "$tmpdir/updated_db.ndjson" "$local" > "$tmpdir/combined.ndjson"
mv "$tmpdir/combined.ndjson" "$local"

The whole thing runs pretty quick, and produces a clean .qmd file; it requires nothing beyond bash, curl, and jq. From there, Quarto renders it into the site like any other post (with a special tag so you all can ignore them if you want).

I have a similar plugin for Goodreads that follows the same pattern — a post for another day.

-CH

Dollars and Democracy

Dollars and Democracy (Classic Reprint)

Philip Burne-Jones

Rating: ★★★★★

The war in the United States: Report to the Swiss Military Department : Preceded by a Discourse to the Federal Military Society Assembled at Berne, Aug. 18, 1862

Ferdinand Lecomte

Rating: ★★★★★

This post was automatically generated from my Goodreads shelf.

-CH

Moral Aspects of City Life: A Series of Lectures

E.H. Chapin

Rating: ★★★★☆

This post was automatically generated from my Goodreads shelf.

-CH

Deck Of Cards / Sweet Thing

T. Texas Tyler
“T” Texas Tyler

Just Before The Battle Mother / Old Folks At Home

Monroe Quartette

Night Train To Memphis / Low And Lonely

Roy Acuff And His Smoky Mountain Boys

Handsome Molly / Train Forty-Five

Grayson & Whitter

Train Of Love / There You Go

Johnny Cash & The Tennessee Two
Johnny Cash And Tennessee Two

Guess Things Happen That Way

Johnny Cash & The Tennessee Two
Johnny Cash And The Tennessee Two

I Forgot To Remember To Forget

Johnny Cash & The Tennessee Two
Johnny Cash And The Tennessee Two

What’d I Say

Jerry Lee Lewis
Jerry Lee Lewis And His Pumping Piano

This post was automatically generated from my Discogs collection.

-CH

El Capote De Paseo / Amor De Madre

Banda Original Columbia
Banda Columbia

Tip Top-Medley Fox Trot / If A Wish Could Make It So-Medley Fox Trot

Six Brown Brothers (2)

Detour / Someone Won Your Heart Little Darlin’

Foy Willing & The Riders Of The Purple Sage

Mississippi / Flaming Youth

Duke Ellington And His Orchestra

Open The Door Richard! / Lonesome Blues

Jack McVea’s All Stars
Jack McVea And His All Stars

How ’Bout That Mess / Drum Boogie

Gene Krupa And His Orchestra

The Original Fox Trot / That Moaning Saxophone Rag

Fred Van Eps Trio
Van Eps Trio

My Lonely Lola Lo (In Hawaii) / My Own Iona (Moi One-Ionae)

Sterling Trio

Stars And Stripes Forever March / Fairest Of The Fair March

Sousa’s Band

Crazy Man, Crazy / Whatcha Gonna Do

Bill Haley And His Comets
Bill Haley With Haley’s Comets

The Wang Wang Blues / The Way You Look Tonight

Benny Goodman Sextet

This post was automatically generated from my Discogs collection.

-CH

The Hobo From The T & P Line

Almoth Hodges

Cotillion

National Barn Dance Orchestra

Mobile County Blues / Just Because

Nelstone’s Hawaiians

A Corn Licker Still In Georgia - Part 3 / A Corn Licker Still In Georgia - Part 4

Clayton McMichen

Leroy / My True Love

Jack Scott

Mansions For Me / Mother’s Only Sleeping

Bill Monroe & His Blue Grass Boys

Ubangi Stomp / Black Jack David

Warren Smith (3)

Last Night / Night Before

The Mar-Keys
Mar-Keys

In Sunny Tennessee / Columbus Stockade Blues

Hal Lone Pine And His Mountaineers

This post was automatically generated from my Discogs collection.

-CH

The Quintessential American

Characters in the media often adhere to certain predefined roles, allowing audiences to understand the character much quicker. These “stock” characteristics do not have to only be limited to the character’s actions; rather, they can also speak to matters of identity. The perfect example of this is James Bond, who is quintessentially British¹. He is suave, upper-class, has command of every situation – essentially, he is everything that the stereotypical Briton has come to represent.

However, this phenomenon extends beyond the overly British Bond – there are counterparts for many other cultures all over the world. “The Man With No Name”, from the Dollars Trilogy, can be considered stereotypically American as a character, specifically in The Good, The Bad, and The Ugly.

The Man With No Name, or “Blondie” (as Tuco, the Ugly, calls him), is defined almost entirely by core tenets of the American stereotype². Popularly, among the world’s various forms of media, Americans have commonly been depicted as arrogant and over-zealous.

These stereotypes are activated by a few different aspects of Blondie’s environment and character³. First and foremost, the film is set in New Mexico around the middle of the American Civil War, a time at which this area was still very much a frontier.

The harsh lands and unforgiving climates set the stage perfectly for Blondie to eventually triumph over them, giving the impression that he will triumph over anything that comes his way. This goes hand in hand with Blondie’s quiet, self-assured nature – they both perpetuate an arrogance that goes unmatched amongst the other characters of the film. These traits of Blondie’s act as primers, triggering the audience to recall the well-established schema⁴ of the “arrogant American” identity.

Another quintessentially American trait is to believe in one’s own goodness, no matter the methods employed. Historically, the United States has employed, well, less than honest methods to gain and keep power on the world stage.

As such, it has become a part of the national identity – so long as America comes out on top, most of the steps in between can be overlooked. Accordingly, Blondie’s actions adhere to this mantra of doing good with less than scrupulous means.

Throughout the film, he is an anti-hero, willing to lie, cheat, gamble, kill and whatever else is necessary to get ahead. However, in the enigma code⁵ that is the film’s title, he is the titular “Good” character, which implies that he has morals to some extent, and at the very least, is the best of the rest of the characters. The audience, watching The Man With No Name act contrary to his given descriptor, is exposed to example after example of the ends justifying the means in stereotypical American fashion.

Though the previously mentioned schema tends to support the interpretation of negative traits in Blondie, there are some good ones as well. Pax Americana is the idea that as soon as the United States gained hemispheric economic and militaristic dominance after World War II, an era of peace followed in its trail.

In the early parts of the film, Blondie struggles to gain the upper hand against Tuco, right up until he learns where the gold is. This knowledge, combined with Blondie being constantly armed after this inflection point, is very reminiscent of Pax Americana. Whether the audience can define it or not, they have been primed through repeated exposure to these ideas⁶, so even the subtlest surfacing of them will bring to mind American identity.

Perhaps, this is indeed the most critical part of implicit association: the viewer does not have to explicitly recall precise definitions and try to apply them to the situation. All that is needed is enough of a pattern for the brain to recognize it, and the rest falls in to place⁷.

The Man With No Name’s arrogance, antithetical nature and similarities to Pax Americana all bring forth views of American identity in the minds of the audience. As media consumers, we have developed a common schema of what the American identity consists of; however, realizing the connection from a character on screen to implicit associations is another story.

I had a rather fun time writing this one, as it meant I got to rewatch a classic!

-CH

Foreword

As you can probably tell by now, folk music is my favorite genre of music because every part of it is a historical artifact. The recorders, the lyrics, the performers – each have their own rich background which makes the combination of them even more special. However, American folk specifically holds a special place in my heart, since it’s a unique blend of worldly traditions with specifically American experiences.

“John Henry” and “John Hardy” from the last post in this series are perfect examples: both men¹ exemplify the common man being a hero (as opposed to kings, knights, etc. from other Western traditions), the frontier experience of lawlessness and overcoming the odds, as well as the historical context of Post-Reconstruction meets the Industrial Revolution.

So without any further ado, let’s look at “Bury Me Not on the Lone Prarie”, a popular cowboy folk song, as another unique example of American folk music.

19th Century

20th Century

Thoughts

As of 1927, our ballad, which we have so carefully traced over 100+ years, has certainly entered the public sphere, and is enjoyed by many from coast to coast. At this time, the ballad is available for purchase in both print and vinyl, and as such, our journey with the song’s evolution will end here.

Evolving from a sailor’s plea not to be lost at sea, all the way to a cowboy’s dying wish on the frontier, “Bury Me Not on the Lone Prairie” grew alongside the spirit of the American West. Each modification, re-telling, floating verse, or publication only adds to the storied past of this iconically American ballad. It’s no longer just a song, as its story of creation is just another piece of what makes it so special. Stories of the common man, of overcoming the odds, of the rough-and-tumble - all pieces which come together in this ballad to form a unique blend of American experiences.

I hope you’ve enjoyed the accompanying playlist; I wanted to include a few notes on the performances and recordings of the ballad as a whole:

The first recording I can find is Bentley Ball (on the Columbia Label) in 1919/1920ish, which is actually our writer of the Song-a-Logue of America.

Below, find a list of some of the other recordings I found mention of¹⁶:

Carl T. Sprague (1926); Vernon Dalhart (1927); Pickard Family (1927); Bill Childers (1928); Jules Verne Allen (1928); Asa Martin (1932); Carter Family (1939); Cisco Houston (1952); Tex Ritter (1959); Burl Ives (1961); Johnny Cash (1966).

I really love the Carter Family, but I would say Sprague’s version is the best of them all – he truly sings it as a lament. The emotion comes through, and you can really imagine the hardship faced by the character of the song. The rest of the recordings are also great, but are played a little faster, so to me at least, they don’t convey the same feeling.

-CH

Part 1

To begin, we need a function that will let us partition our disk based on the rules of the problem. In the input, numbers alternate between being a file or being free space, so we need to split up which are which with an odd/even split:

function partition(disk_map) {
  const disk = []
  // for all numbers
  for (let i = 0; i < disk_map.length; i++) {
    // even case
    if (i % 2 === 0) {
      // file, repeat number
      for (let j = 0; j < disk_map[i]; j++) disk.push( String(i / 2) )
    //odd case
    } else {
      // spacer, print "." instead
      for (let k = 0; k < disk_map[i]; k++) disk.push('.')
    }
  }
  return disk
}

Now that we are actually able to build out what the disk looks like, it gets a whole lot easier to solve, especially if we work backwards. Working forwards, we would keep expanding the string, and our indices would be off; however, by working backwards, we can insert whatever we want and still keep track of where we are in the string.

With that in mind, it’s really not difficult – we just map through the empty spaces, and fill with whatever needs to go there!

part_1 = {

  // partition disk into array of pieces
  const disk = partition(raw)

  // work backwards
  for (let i = disk.length - 1; i >= 0; i--) {

    // get index of next spacer in seq
    const next_space = disk.indexOf('.')
    
    // if next digit is a '.' or if index is below last '.' in array
    if (disk[i] === '.') {
      continue // skip
    }

    // if no more spacers before our current position
    if (next_space > i) {
      break // exit early to avoid cycling through a bunch of continue reps
    }

    // replace that position with the digit we are on
    disk[next_space] = disk[i]

    // replace with '.' to mark it is now empty
    disk[i] = '.'
    
  }
  
  return disk.reduce((acc, nxt, i) => {
    return nxt === '.' ?
      acc :
      acc + Number(nxt)*i
  }, 0)
}

part_1

⭐

Part 2

In part 2, we can no longer move one file (character in our string) at a time – we have to move whole blocks at once. While this may seem daunting, it’s fairly similar to what we did before, but instead of iterating through characters, we are now going to iterate through blocks.

To make this easier, let’s first parse the input a little more than before, and create an object with the blocks, and empty arrays as our spacers. That way, as we iterate through, we can dump the blocks into available spacers all by manipulating our arrays.

part_2 = {
  const disk_map = raw.split("").map(Number)

  // split into blocks and empty array spacers
  const disk = {
    blocks: disk_map.filter( (d, i) => i % 2 === 0 ).map( (d, i) => [d, i]),
    spacers: disk_map.filter( (d, i) => i % 2 !== 0 ).map( (d, i) => [d, []])
  }
  
  // going backwards (without doing the last one, as that will lead our array)
  for (let i = disk.blocks.length - 1; i > 0; i--) {
    
    // go through gaps up to the backwards run
    for (let j = 0; j < i; j++) {

      // check if block space is <= available spacer size 
      if (disk.blocks[i][0] <= disk.spacers[j][0]) {
        
        // decrement the space counter
        disk.spacers[j][0] -= disk.blocks[i][0]

        // fill space with block
        disk.spacers[j][1].push( 
            ...new Array(disk.blocks[i][0]).fill(disk.blocks[i][1]) 
        )

        // replace block empty char
        disk.blocks[i][1] = "."

        // exit because slot found for block
        break
      } // no else statement since we want to leave as-is if no space
    }
  }

  // output arr
  const defrag = []

  // build output arr
  for (let i = 0; i < disk.spacers.length; i++) {

    // push block
    defrag.push(...new Array(disk.blocks[i][0])
               .fill(disk.blocks[i][1]) )

    // push spacer fill
    defrag.push(...disk.spacers[i][1])

    // push remaining unfilled spacers
    defrag.push(...new Array(disk.spacers[i][0]).fill(".") )
    
  }

  return defrag.reduce((acc, nxt, i) => nxt === '.' ? acc : acc + nxt*i, 0)
}

part_2

⭐

The hardest part here was deciding to work backwards through the disk. Initially, I took a recursive-heavy approach to work forwards, and that became a pain fast. The good news is, other days benefit from this reversed approach, so it was a good reminder to keep it as an option.

-CH

Part 1

In part 1, we need to find all the possible antinode locations, which are derived from pairs of matching antennae sets. To do so, we’ll need to create a little bit of infrastructure below:

function pairwise(arrays, dupes = false) {
  // recursive so we need a short circuit
  if (arrays.length === 0) return []
  // extract matching item
  const [first, ...rest] = arrays
  // get all other pairs recursively
  const end = pairwise(rest)
  // if we allow dupes, nest them together
  const other = end.map(([first2, rest2]) => {
    return dupes
      ? [first2, [first, ...rest2]]
      : [first2, [...rest2]]
  }) 
  return [[first, rest], ...other]
}

Above, we created a function to create all pairwise matches from an array. Using this, we can find all the locations of antinodes, since the new ones must be on the same plane as the existing antennae.

Once we have our pairs, then we actually have to find the possible locations. Luckily for us, this is quite simple, since we can use the good ol’ slope formula to calculate the possible positions.

function find_antinodes(a, b, n) {
    // y2 - y1 / x2 - x1
    const slope = [ a[0] - b[0], a[1] - b[1] ]
    // return the new positions in both directions
    return [
      [ a[0] + n*slope[0], a[1] + n*slope[1] ],
      [ b[0] - n*slope[0], b[1] - n*slope[1] ]
    ]
  }

With these in hand, the solution becomes fairly trivial; all we need to do now is iterate through our input Map to:

1. Get all the pairwise matches of antennae in each group
2. Calculate the slope of each sub-pair
3. Count all unique antinode locations

part_1 = {

  const antinodes = []

  // for each set
  input.antennae.forEach(antenna_set => {
    antinodes.push(
      // get all pairwise matches
      pairwise(antenna_set, false).flatMap(([ref, pair]) => {
        return pair
            // get the possible locations for each pair
            .flatMap(d => find_antinodes( 
                [ref.row, ref.col], 
                [d.row, d.col], 
                1 )
            )
          // remove any outside grid boundaries
      }).filter(([row, col]) => 
                row >= 0 &&
                row < input.nrow &&
                col >= 0 &&
                col < input.ncol            
      )
    )
  })

  return new Set(
    antinodes
        .flat()
        // combine coords with left shift and bitwise OR
        .map(d => d[0] << 8 | d[1])
    ).size

}

part_1

⭐

Part 2

In part 2, the difference is that antinodes may be more than 1-step away from existing antennae. This makes it fairly easy, since we can just extend our functions from before to look at a longer range!

With a quick tweak to n in our find_antinodes() function (by using number of rows we can be sure that we will exceed the bounds for each antennae set in both directions), we can easily cross this day off our list!

part_2 = {
  const antinodes = []
  // get array of row count to iterate across as N
  const n_mults = Array(input.nrow).fill(0).map((_, i) => i+1)
  
  input.antennae.forEach(antenna_set => {
    antinodes.push(
      // same as before
      pairwise(antenna_set, false).flatMap(([ref, pair]) => {
        return n_mults.flatMap(n => {
          // map across all N distances
          return pair.flatMap(d => find_antinodes( [ref.row, ref.col], [d.row, d.col], n ))
        })
          // same filter as before to check boundaries
      }).filter(([row, col]) => 
                  row >= 0 &&
                  row < input.nrow &&
                  col >= 0 &&
                  col < input.ncol            
      )
    )
  })

  // left shift and bitwise OR
  const all_antinode = antinodes.flat().map(d => d[0] << 8 | d[1])
  // do the same for antennae positions, since they count in this part
  const all_antennae = [...input.antennae.entries()]
      .map(d => d[1])
      .flat()
      .map(d => d.row << 8 | d.col)

  // count unique antinodes AND antennae
  return new Set( [ ...all_antinode, ...all_antennae ] ).size
}

part_2

⭐

Pretty fun day! I always enjoy a bit of regex, and some classic math to boot.

I’m looking forward to catching up on a bunch of these AoC days, so stay tuned for more soon!

-CH

Appendix

// Read in data and parse
relationships = {
  let raw = await FileAttachment("seinfeld_relationships@7.json").json(), 
      counts = new Object()
  raw = raw.filter(d => (characters.includes(d.To) || characters.includes(d.From)) && (types.includes(d.Type)))
  raw.map(d => d.From).forEach(d => {
    return counts[d] ? counts[d]++ : counts[d] = 1
  })
  return {
    nodes: Array.from(new Set(raw.map(d => d.From))).map(d => ({id: d, count: counts[d]})),
    links: raw
  }
}

// Build lookup for values
lookups = ({
  legend: [
    {type: "Friend", color: "#2ca02c"},
    {type: "Family", color: "#d62728"},
    {type: "Antagonistic", color: "#ff7f0e"},
    {type: "Romantic", color: "#1f77b4"},
    {type: "Professional", color: "#9467bd"}
  ],
  characters: ["Jerry Seinfeld", "George Costanza", "Elaine Benes", "Cosmo Kramer"]
})

// Create character selection
viewof characters = Inputs.checkbox(
  lookups.characters, {
  value: lookups.characters,
  label: html`<b>Main Characters</b>`
  }
)

// Create relationship type selection
viewof types = Inputs.checkbox(
  lookups.legend.map(d => d.type).reverse(), {
  value: lookups.legend.map(d => d.type).reverse(), 
  label: html`<b>Relationship Types</b>`,
  format: x => html`<span style="text-transform: capitalize; border-bottom: solid 2px ${lookups.legend.filter(d => d.type === x)[0].color}; margin-bottom: -2px;">${x}`
})

// Build D3 chart
relationship_chart = {
  const height = 800
  const width = 1200

  const color = d3.scaleOrdinal()
    .domain(lookups.legend.map(d => d.type))
    .range(lookups.legend.map(d => d.color))

  const links = relationships.links.map(d => ({source: d.From, target: d.To, type: d.Type}))
  const nodes = relationships.nodes.map(d => ({...d}))

  const sim = d3.forceSimulation(nodes)
    .force("link", d3.forceLink(links).id(d => d.id).distance(100))
    .force("charge", d3.forceManyBody().strength(-50))
    .force("collide", d3.forceCollide().radius(20))
    .force("center", d3.forceCenter( width / 2, height / 2 ).strength(1.1))
    .on("tick", ticked)

  const svg = d3.create("svg")
    .attr("width", width)
    .attr("height", height)
    .attr("viewBox", [0, 0, width, height])
    .attr("style", "max-width: 100%; height: auto;")

  const link = svg.append("g")
      .attr("stroke", "#999")
      .attr("stroke-opacity", 0.6)
    .selectAll()
    .data(links)
    .join("line")
      .attr("stroke-width", 1.1)
      .attr("stroke", d => color(d.type))

  const globs = svg.append("g")
      .attr("stroke", "#fff")
      .attr("stroke-width", 1.1)
    .selectAll("g")
    .data(nodes)
    .enter()
    .append("g")

  const node = globs.append("circle")
      .attr("r", d => Math.max(3, Math.sqrt(d.count)+3))

  node.append("title")
    .text(d => `${d.id} (${d.count})`)

  globs.append("text")
    .text(d => d.id)
    .attr("dy", "-1em")
    .attr("text-anchor", "middle")
    .attr("stroke", "black")
    .attr("stroke-width", 0.1)
    .style("font-size", "12px")
  
  globs.call(d3.drag()
           .on("start", dragstarted)
           .on("drag", dragged)
           .on("end", dragended))

  const legendDots = svg.append("g")
    .selectAll("legendDots")
    .data(lookups.legend.map(d => d.type))
    .join("circle")
      .attr("cx", width - 170)
      .attr("cy", (d, i) => height - 50 - i*20)
      .attr("r", 4)
      .attr("fill", d => color(d))

  const legendText = svg.append("g")
    .selectAll("legendText")
    .data(lookups.legend.map(d => d.type))
    .join("text")
      .attr("x", width - 160)
      .attr("y", (d, i) => height - 50 - i*20)
      .attr("text-anchor", "left")
      .style("alignment-baseline", "middle")
      .text(d => d)

  const legendTitle = svg.append("g")
    .selectAll("legendTitle")
    .data(["Relationship Types"])
    .join("text")
      .attr("x", width - 190)
      .attr("y", height - 155)
      .text("Relationship Types")
      .attr("text-anchor", "left")
      .style("text-decoration", "underline")
      .style("alignment-baseline", "middle")

  function ticked() {
    globs
      .attr("transform", d => `translate(${d.x}, ${d.y})`)
    
    link
      .attr("x1", d => d.source.x)
      .attr("y1", d => d.source.y)
      .attr("x2", d => d.target.x)
      .attr("y2", d => d.target.y)

  }

  function dragstarted(event) {
    if (!event.active) sim.alphaTarget(0.3).restart()
    event.subject.fx = event.x
    event.subject.fy = event.y
  }

  function dragged(event) {
    event.subject.fx = event.x
    event.subject.fy = event.y
  }

  function dragended(event) {
    if (!event.active) sim.alphaTarget(0)
    event.subject.fx = null
    event.subject.fy = null
  }

  invalidation.then(() => sim.stop())
  
  return svg.node()

}

Recap

Last time, we left off with our second truth table exploring Church Numerals and some incrementing mathematical operations. If you missed it, catch up here before moving on to this post’s topic: data structures and some new mathematical operations. ¹

Data Structures

Now that we can represent numbers as functions, and critically, operate on them doing basic arithmetic, we need to find a way to store values for reference later. For example, the most basic of data structures is the n-tuple, which is an ordered list of n values.

Storing values gets pretty tricky in lambda calculus – remember, everything is a function, and functions naturally want to execute their operations on their arguments. However, we can get around this by using a form of encapsulation.

Our outer function will wrap the arguments, doing nothing with them. Then, when we need a value, we pass an successor function to the stored value, finally executing on it, and retrieving the real value. Straightforward? Not in the least.

Let’s make some functions, then walk through them on the other side:

BUILD_3TUPLE = a => b => c => f => f(a)(b)(c)

GET_A = t => t(a => b => c => a)

GET_B = t => t(a => b => c => b)

GET_C = t => t(a => b => c => c)

In the above BUILD_3TUPLE, can you decipher which part is our “tuple”? Indeed, it is the portion f => f(a)(b)(c)! This is because the function f doesn’t actually do anything, it’s just going to hold on to the tuple (a,b,c) until we need them.

Once we do need them, it’s time to call our accessor functions! These GET_* functions all do the same thing: retrieve the positional value by using t to unwrap the tuple, and only return the position we need.

Let’s try an example, where we want to extract the middle value from the tuple (0, 1, 2).

{
  const tuple = BUILD_3TUPLE(N0)(N1)(N2)
  const b_val = GET_B(tuple)
  return lambda_to_int( b_val )
}

Using this idea, we can now implement a decrementing function – a procedure that seems fairly trivial, considering we have it’s counterpart already defined. However, it’s not so straight forward as it appears.

Incrementing something is easy with functions; if you recall, we just wrap another layer around the outside. However, to get the predecessor, you would have to unwrap a layer of the nested functions, which we have no great way of doing.

Instead, we’ll have to keep values “in memory”. Let’s start by defining a 2-tuple (PAIR), and the needed accessors:

PAIR = a => b => f => f(a)(b)

FIRST = t => t(a => b => a)

SECOND = t => t(a => b => b)

With this in hand, we can create a function known as shift-and-increment (or phi), which takes a pair of numbers and returns a new pair of the second number and its increment: (a, b) => (b, b+1).

There can be many uses for this (looking at you, linked lists), but for now, we will use the concept to achieve our decrement function. The phi of a pair will always contain it’s predecessor, so we can back our way into coming up with the previous number:

PHI = p => PAIR( SECOND(p) )( SUCC( SECOND(p) ) )

Above, we are forming a PAIR based on p (prior pair). This means that we want the second element of p to become the first of PHI: (SECOND(p)), and the increment/successor of the second element of p as the second element of PHI: (SUCC(SECOND(p))).

PRED = n => FIRST( n(PHI)( PAIR(N0)(N0) ) )

SUB = m => n => n(PRED)(m)

In the above, we are implementing PRED and then SUB, which utilizes PRED in its underlying logic.

PRED is our decrement function, which takes a 2-tuple and starts counting up to it from (0,0). Once we hit our target number in the second position, we can then return the number in the first position, since we know that should be one less.

SUB takes this farther by adding framework around PRED. Specifically, we are applying PRED n times to m. In other words, starting at m, we are then subtracting 1 from it n times. I will note this is not an efficient implementation whatsoever – for each step, we have to count up from 0 to get the prior number. It works fine here since our numbers are low, but if you imagine doing 1000-999, or something even more drastic, you can imagine how long it would take.

Using this idea, we can use a neat trick of the number functions themselves to check whether the number is 0 or not. Remember, a number is just the number of nested functions are in a stack - so 0 is represented by no functions in the stack. This is helpful because we can now check how many functions are called: any number of times > 1 means it is not equal to 0, no calls means it is:

ISZERO = n => n(x => FALSE)(TRUE)

Similarly, we can take advantage of a feature of subtraction to find if a value is less than or equal to another. Technically, since none of our numbers are signed (how would you even add a sign based on nested functions???), we can just check if the result of subtracting the terms is 0. It’s ambiguous, yes (0 could mean the value is negative or 0), but in this case, we can use that “feature” to determine our inequality:

LEQ = m => n => ISZERO( SUB(m)(n) )

With these brand new operations in hand, let’s update our truth table from last post with our newest functions:

part_3_math_table = {
  const output = [];
  
  [N0, N1, N2, N3, N4].forEach(a => {    
    output.push({
      n: lambda_to_int(a),
      succ: lambda_to_int( SUCC(a) ),
      plus: lambda_to_int( PLUS(a)(a) ),
      mult: lambda_to_int( MULT(a)(a) ),
      pow: lambda_to_int( POW(a)(a) ),
      pred: lambda_to_int( PRED(a) ),
      "sub(2)": lambda_to_int( SUB(a)(N2) ),
      iszero: lambda_to_js( ISZERO(a) ),
      "leq(2)": lambda_to_js( LEQ(a)(N2) )
    })
  })

  return output
}

Inputs.table(part_3_math_table)

In the immortal words of Bill and Ted, excellent! We’ve got some basic data structures and even more operations built out.

I’m not yet sure if I’ll continue this series past this point, as I feel like I have a solid feel for lambda calculus at this point.

-CH

Church Numerals

Now that we have basic operations and logic gates, we need to advance past 1’s and 0’s to be able to represent more complex numerals. The difficulty here is going to be representing numerals with functions, for a few reasons:

1. There are an infinite number of numerals to represent
2. How can we combine the numerals using different functions, without defining the numerals themselves?

We are going to explore Church Numerals, which define a numeral N by encapsulating a function N times. This is pretty cool, as we are then no longer representing the numeral N as a “count”; instead, it becomes more of a “timer”. Let’s take a look:

N0 = f => a => a // apply f 0 times

N1 = f => a => f(a) // apply f 1 time

N2 = f => a => f(f(a)) // apply f 2 times

N3 = f => a => f(f(f(a))) // apply f 3 times

N4 = f => a => f(f(f(f(a)))) // apply f 4 times

… and so on. However, the absolute last thing I want to do is sit here and write more functions to define every number ever. So, let’s call it good here, and use this (admittedly) short list to test our operations with.

Mathematical Operations

Similar to the idea of a “wrapper” function from last post (see NOT()), we need to define functions that will wrap around numbers to perform operations on them.

The first of these is the most simple, the successor function SUCC(). A fancy name for a basic operation, all this does is increment the number by 1. All it really is going to do is add another layer of wrapping around the number – that way, when we count the layers by “time”, our count, and therefore value, will have increased by 1.

SUCC = n => f => x => f(n(f)(x))

In the above, f => x => f(n(f)(x)) creates our new number. To break it down further, n is the amount of times we are wrapping f around x, which means on its own, n(f)(x) is equal to n, the input number.

The crucial part of this function is the wrapper f() – as that is the piece which adds another “layer” to our number function, thereby incrementing its “count” by 1. If we substitute out the n discussed above for N (our numeral), this simplifies down to N => f(N), where the incrementation is far easier to spot.

Now, we can move towards addition using the same principles. Instead of wrapping the “number” with one additional function call (successor), we need to wrap it with a variable number of functions:

PLUS = m => n => f => x => m(f)(n(f)(x))

In the above, we achieve this by using the same “inner” variable for both functions, x. Accordingly, we want to apply f to x for each nested “number” function (m and n). Thus, we are applying f to x for m + n times, and following the same pattern of counting nested levels, we end up with our sum.

This principle is very extensible to multiplication – rather than operating on the same variable x, we want the “number” functions to operate on each other:

MULT = m => n => f => m(n(f))

Rather than increasing the variable x first m times, then n times, we need to (increase the variable f n times) m times.

A more verbose way to write MULT() would be to add the x variable back in:

MULT = m => n => f => x => m(n(f))(x)

In this form, it is easier to see that instead of both m and n operating on f(x), we are instead having m operate on n, which is in turn operating on f(x).

Exponentiation takes this idea farther, and instead of using f(x) as the base we are incrementing, we want to increase the base, b, itself by e (exponent) amount of times.

POW = b => e => e(b)

Just like in MULT(), there is a more verbose way to show this:

POW = b => e => f => x => e(b)(f)(x)

Here, it’s less clean, but much clearer that e and b operate independently of f(x), and only once calculated, adds e(b) wrappers around f(x). Also, note that we are using f(x) here as the base, which is equal to 1 – if we had only x instead, we would end up with an incorrect amount of wrappers.

With these basic arithmetic functions, let’s test our shiny new operation functions on a (drumroll please…) new truth table with our short list of numbers!

As we did last time, we’ll need to create a helper function to convert the lambda functions to actual numerals…

lambda_to_int = (int) => {
  return int(x => x + 1)(0)
}

… letting us build our table!

part_2_math_table = {
  const output = [];
  
  [N0, N1, N2, N3, N4].forEach(a => {    
    output.push({
      n: lambda_to_int(a),
      succ: lambda_to_int( SUCC(a) ),
      plus: lambda_to_int( PLUS(a)(a) ),
      mult: lambda_to_int( MULT(a)(a) ),
      pow: lambda_to_int( POW(a)(a) )
    })
  })

  return output
}

Inputs.table(part_2_math_table)

Woohoo! We’ve got some basic operations working now. However, you might have noticed that these operations are strictly incrementing our value.

Next time, we’ll take a look at how to reverse the direction with subtraction, and the underlying data structures that that will require.

-CH

Basics

IDENTITY = x => x

Our first function is very simple, but will show the foundation of what we are after.

This is the identity function, which returns exactly what the input is. Notice we are following all rules above: 1 argument & statement, &c

Let’s extend that a little farther:

RETURN_FIRST = a => b => a

RETURN_FIRST(1)(2)

RETURN_SECOND = a => b => b

RETURN_SECOND(1)(2)

In the above functions, we are technically nesting a function within another, such that the argument of the first may or may not be called by the second function. RETURN_FIRST() and RETURN_SECOND() do exactly as they say, and this ‘switch’ ability is going to be key to start adding logic.

Booleans and Logic Gates

TRUE = a => b => a

FALSE = a => b => b

This may not seem like a lot, or more like a regurgitation of the sample switch return functions above, but this is critical to the base of our logic. The above are our replication of boolean states, which are identical to RETURN_FIRST and RETURN_SECOND – in fact, as we go on, it will be helpful to keep in mind that TRUE will always return the first argument, and FALSE the second.

Naturally, these booleans can be combined together, creating AND/OR logic gates:

AND = c => d => c(d)(c)

OR = c => d => c(c)(d)

As you can see, the arguments to the functions above are all functions, and will return functions. For example (below), calling AND(TRUE)(FALSE) and OR(TRUE)(FALSE) will return the appropriate functions, either FALSE(), or TRUE().

As an example, let’s examine the AND() & OR() step by step:

1. First, they accept two arguments, which we’ve seen before. In this function, the shorthand is c and d, but we’ll refer to them now as first and second arguments.
2. Once the arguments are collected, the bulk of the function is c(d)(c) or c(c)(d).
3. Within these blocks, the order is critical, as the positions of the arguments of TRUE() and FALSE() dictate what the end result will be.

We’re able to use creative permutations of returning the first (TRUE) and second (FALSE) values, giving us our basic AND/OR logic gates. As a thought experiment, let’s walk through all the possible permutations:

• AND(TRUE)(TRUE) will call TRUE(TRUE)(TRUE), returning: TRUE()
• AND(TRUE)(FALSE) will call TRUE(FALSE)(TRUE), returning: FALSE()
• AND(FALSE)(FALSE) will call FALSE(FALSE)(FALSE), returning: FALSE()
• AND(FALSE)(TRUE) will call FALSE(TRUE)(FALSE), returning: FALSE()

Likewise, for OR():

• OR(TRUE)(TRUE) will call TRUE(TRUE)(TRUE), returning: TRUE()
• OR(TRUE)(FALSE) will call TRUE(TRUE)(FALSE), returning: TRUE()
• OR(FALSE)(FALSE) will call FALSE(FALSE)(FALSE), returning: FALSE()
• OR(FALSE)(TRUE) will call FALSE(FALSE)(TRUE), returning: TRUE()

And just to confirm that this is working as expected:

AND(TRUE)(FALSE) === FALSE

OR(TRUE)(FALSE) === TRUE

As opposed to the above, NOT() looks like it will take 3 arguments rather than the 2 we’ve seen so far. However, the best way to think about it is that it takes only 1 argument: one of the other logic functions: AND(), OR(), &c.

In the function, consider c as the outer function (AND, OR, &c.), so instead of calling AND(a)(b), it acts as a wrapper around the logic gate and will flip the inputs, translating it into AND(b)(a). If you refer back to the AND()/OR() tables above, you can pretty clearly see how this would negate all the results.

Let’s take a look:

NOT = c => a => b => c(b)(a)

NOT( AND(TRUE)(TRUE) ) === FALSE

To extend this idea of a wrapper, we can leave the input order the same, and get a ternary operator (IF) out of it (just as a shorthand), of the form IF(condition)(value if true)(value if false) like so:

IF = c => a => b => c(a)(b)

IF( AND(TRUE)(FALSE) )(TRUE)(FALSE) === FALSE