Code And Cocktails

Two Things I Learned About GitHub Actions

| Comments

Background

TLDR: I have GitHub Action Workflows that run Common Lisp code; the setup of the Workflow containers was duplicated and the duplication was going to be a problem. Learned: 1) the lines of the jobs.steps.run will be placed in a file; jobs.steps.shell is a command that will take that file as an argument ({0} can be used as a placeholder for that file) 2) actions are not hard to make, and docker actions have some particular ways of working that can be surprising.

In my work as the Common Lisp Exercism Track maintainer I try to do as much as I can in Common Lisp just to use it more and expand my knowledge of it. This post is not about Common Lisp however.

The track has two GitHub Action Workflows which need to run Common Lisp code for its CI (one to run all the tests and the other to do some checking of the validity of the track’s configuration). Originally the workflows used a standard ubuntu-latest container and then installed SBCL and QuickLisp (both downloading the QuickLisp script and running it), setting those both up to make sure they could find the code, then loading the system (via QuickLisp to download any dependencies it may have) then finally calling the ASDF:TEST-SYSTEM function on the system ("TEST-EXERCISES" for example). (See the original test-exercises workflow code for an example.)

If it were only a single workflow this would not be too bad. But there were two workflows and I might want to make more. And there are other repositories for the Common Lisp track which I also maintain which I did not yet have CI set up and knew I’d need to do this same sort of thing. So this needed to be cleaned up.

First thing I learned

I knew that the clfoundation/sbcl docker image had SBCL installed and could easily be set up for QuickLisp so I wanted to use that somehow as the base of my workflows.

You can do this per workflow job by setting the container.image setting. This was a good step, my workflow no longer needed to download and install SBCL or download the QuickLisp script.

So now my job.steps.run could easily access SBCL and the QuickLisp script to set things up and run my code. But then I learned something to make the running of my code better.

steps can have a shell setting which is the name of a command which is given a file. The file contains all the lines of the run in the same step. Better yet shell can be of the form cmd arg1 arg2... {0} other-args...where {0} will be replaced with that filename. So that means I could run lisp code as my run:

      - name: Run Tests
        run: |
          (ql:quickload "test-exercises")
          (asdf:test-system "test-exercises")
        shell: sbcl --noinform --non-interactive --load {0}

(See the workflow using an published image as base for this example.)

The second thing I learned

NOTE: This learning is mostly a trial-and-error learning what didn’t work.

While the above was definitely good there was still duplicated setup and commands. What I wanted was my own action for performing an operation on a system.

I created an action in the same repository which could then be used by my workflows. That is instead of a run my workflow would have a uses: ./.github/actions/perform-system. Note that the path to the action is relative to the root of the repository.

An action is effectively a directory with an action.yml file in it. That file has similar syntax to the workflow. There are three choices for actions: Javascript based, Docker based or “Composite”. Actions also can take inputs which are provided by with settings in a workflow that uses the action.

Since a composite action is just one that has several run commands I thought that would be the one to use, but as there did not seem to be a way to set an image to use for my action and it did not seem to be using the image I specified in my workflow this did not work out.

So I then went down the route of a docker action. I initially created a Dockerfile which used the clfoundation/sbcl and installed QuickLisp. I also created a entry point script which would take the inputs to my action (the system name, location of the system and operation to perform).

But… that didn’t work. After much trial and error I finally realized (and then found the documentation that explains what was happening) that my docker image being run wasn’t quite the same one I was building with my Dockerfile. Specifically things like the $HOME setting was different; and there was still the perplexing question of where my code was going to be in the image.

Because $HOME was different between building the image and running the image my plan of installing QuickLisp in the image was not going to work (QuickLisp’s installation adds things to your home directory). So that got pushed into my entry point script.

Even with these annoyances the resulting action is compact and easy to understand and the workflow was greatly simplified.

Next things to learn

I want to learn more about the docker images to see if I can improve how I set it up and if there is a reasonable way I can create a test-script for it so I can run it ‘locally’ for experimenting with it.

I also want to learn how to use this new action in other repositories. I think it may just be a matter of moving it into another public repository.

Notes on Zettelkasten, GTD and the Memex

| Comments

(Recently I started hearing about this note taking system called Zettelkasten. (Seems, as usual, I’m a bit late to the game as the Interwebs was talking about it more in 2020 and prior…) What follows is some notes and thoughts about it and how it connects with other thoughts.)

Zettelkasten

Zettelkasten (or ‘Slip Box’ / ‘Card Box’ in English) is a note taking system used (and perhaps invented) by Niklas Luhmann. He said it was the way he was able to so prolific in publishing academic papers. It involves writing notes/thoughts on small cards and organizing them in a tree structure, physically in a card box. Each card gets an index number (based upon where it is the tree) and references to other cards if applicable. Index numbers were made of letters and numbers alternating thus allowing for the branching. Cards “1” and “2” are not particularly related, but “1a1” is a note that continues from “1a”, while “1b” is a note continuing from “1” but not necessarily related to “1a”. What Luhmann thought was very important was that there was not any organization a priori. He felt that “accident” was very important and having an organization ahead of time would limit the possible thoughts one could have.

More details on this system and how to use it can be found in Luhmann’s paper and also in Sönke’s How to Take Smart Notes.

GTD

How does this connect with GTD? Luhmann’s process of reading and taking notes reminds me a little of the GTD’s separation of capture and processing.

Luhmann’s system had three types of notes: “fleeting”, “literature” and “permanent”. The only differences as far as I can really see between “fleeting” and “literature” notes is where they came from and how much supporting bibliographic material they may have. Both are written down at any time and collected together. Later (but not too much later) they are reviewed and either discarded or turned into permanent notes. Permanent notes are stored in the Zettelkasten. As notes from the inbox are processed one decides what one needs to do with them, where do they fit into the system.

Memex

In 1945 Vannevar Bush published an article in The Atlantic Monthly entitled “As We May Think”. In it he describes the “Memex” a sort of work desk / computer which could be used to search for and view documents (on microfilm) stored in the device. More data could be added to it - either personal notes / documents which would be scanned in; annotations on the documents; or even new documents bought or given to the user by others (you could make copies of documents to share with others…)

Basically if you squint at it just right he described a sort of World Wide Web. Documents available at your fingertips and documents were linked together. This availability and linking would allow people to do research on any topic quite easily, synthesizing existing information into new information.

One thing he mentioned that the Memex would have tat we do not have in the WWW is his idea of “trails”. With the Memex you would be able to annotate and save the “trail” of our research, that is you could save the fact that looking at document A led you to document B which lead you to a footnote of document E etc. etc.. These trails were another piece of data one could store and share with the Memex. A trail could be shared with a colleague thus not only sharing the documents you looked at, but how you looked at them.

I think it could be interesting if this idea of trails could be developed.

Synthesis

So how do these thoughts connect? For me the idea of the Zettelkasten and of the Memex are similar, how to store and access information; how to synthesize information into new information. That alone is a good thing - but how to deal with all the information you may be putting into the system? Via GTD’s separation of capture & process, Zettelkasten’s fleeting vs. permanent notes.

A new piece of information or thought can be captured quickly - now your brain can go back to the job at hand. Later (hopefully not too much later) one goes through all that captured information and decides what to do with it. Some of it may, in retrospect be not important - you discard. Some may be important - you store it. Not only that but during storage one synthesizes it with existing knowledge which may immediately create new information.

REPL Driven Development

| Comments

(As I was preparing to write this blog I see that 1) a few other people seem to be discussing around this topic (e.g. David Vujic), Erik Engheim, et al.) and 2) that I already discussed this some 10 years ago TDD and REPL Analogies?! Just adding this note to make it clear that this is nothing new, just my thoughts and experiences.)

Background

I am the author/maintainer of an Exericism track for Common Lisp. Currently we are doing a big push to prepare and release v3 of the platform. During this push I’ve been doing more Common Lisp coding than usual1 which has brought me back to REPL driven development.

I personally find it very useful to be able to “play around” or explore a problem before sitting down to actual produce a software solution. Sometimes the code, as I explore and play, becomes the “real” code I will keep. Thus this is not a prototype or spike where the output would be thrown away, or at least not shipped to production2.

At the time of my earlier writing I was working in a language which did not have a strong REPL offering (or any at all) and found TDD to be a way for me to do this exploration in a reasonable fashion. I found, lacking a REPL, TDD provided me a fast feedback cycle which was necessary for my exploratory approach. Having a REPL shortens that feedback cycle for exploration.

What is REPL Driven development

A REPL with full editing features (such as SLIME or SLY in GNU Emacs) combines the power of the language, the power of the editor/IDE, with fast feedback that allows me to explore and play with ideas.

Like TDD, REPL Driven Development/Design lets me experiment with different interface designs of how my modules3 will fit together. It also allows me to quickly check that my most recent changes haven’ t caused any problem.

I can also use it when I’m unclear on what the solution might even be. For example when consuming a JSON formatted configuration file for my exercism track, and I want to better understand how the different parts of the data related to each other, where there may be inconsistencies or lacunae in the data.

The general workflow is to work with some code in the REPL, testing and designing it and when it seems ‘right’, saving it to a source file. The workflow can also include working in source files and evaluating parts of the code to load them into the REPL. In a well integrated REPL there is little difference in the editing experience between the REPL and a source file.

In Common Lisp the REPL includes a debugger with a sophisticated condition and restart system which lets me write code I wish I had, then when I run it I am asked what to do about the undefined functions or variables. I can choose to quickly define them or stub them out and then continue, or I can quit the debugger to start the feedback loop again after making further changes.

What it is not

One thing that this cycle does not provide is a a set of tests which can be run repeatedly as you get with TDD. However the interactions at the REPL involve many of the same test cases you’d use in TDD - typical data, empty data, aberrant data, etc. So those can be collected, mixed with some assertions and used as the tests to save for later regression testing / documentation.

Next steps

To be honest, adding bindings when stopped in the debugger is not part of my workflow because it is still not ‘natural’ for me to do, my fingers all too quickly dismiss the debugger from lots of habit. This workflow is my next experiment.

  1. I am a hobbyist programmer of the language at best… 

  2. That never happens. 

  3. By module I mean any ‘building block’ of the software I am building. These might be functions, classes, packages, modules, sub-modules, all of the above. (cf. Parnas). 

How to Write a Heroku Buildpack for Fun and / or Profit

| Comments

Heroku buildpacks are used to prepare (/e.g./ build/compile) an application to allow it to be run by Heroku on their dynos. There are many that exist so it is not likely that you will need to write your own. And that is quite true for the one I wrote. But I wrote one partly to find out how to do it, so let me share what I learned.

The API for Buildpacks is pretty simple and it can be easy to write one. The API is three scripts that will be run by Heroku when it runs the buildpack: detect, compile, release. Buildpacks are git repositories so it is common to host them on GitHub (but that is not required).

While I say that the API is simple, I hope it is useful to summarize details of the API here. Perhaps restating it with my own words will help others (or just me in the future).

Environment that the buildpack is run in

Buildpacks are run on the same stack as the application is.

Buildpacks are run in their own directory. This is not the directory where the application code is. Also this buildpack directory is not available to later buildpacks nor the application. So if the buildpack creates build artifacts those must be put somewhere else.

Besides the buildpack directory there are three other directories available to the buildpack:

  • BUILD_DIR: this is the directory where the application code is.
  • CACHE_DIR: this is a directory which can be used to cache build artifacts (see below).
  • ENV_DIR: this is a directory which contains information about the [Heroku Config Vars][heroku-confi-vars] (see blow).

bin/detect

This script’s job is to decide if this buildpack should be run. It is passed a single argument, the BUILD_DIR. This script typically will work by checking for a particular file in BUILD_DIR. If the buildpack should not run the script should return a non-zero value. If the buildpack should run then besides a zero return value it must also print to standard output a “human readable” name. This will show up in the build logs and shows that the buildpack is chosen.

bin/compile

This script is a huge part of a build pack. It has the job of compiling/preparing the application. It is passed three arguments: BUILD_DIR, CACHE_DIR and ENV_DIR.

There are two environment variables available to the compile script:

  • STACK: which stack the buildpack is running on
  • SOURCE_VERSION: the “version”. For applications deployed via git pushes that will be a commit SHA.

Heroku Config Vars are not available as environment variables to the buildpack. But they are available via the ENV_DIR (see below).

Any output to standard output will appear in the build log. It is recommended that buildpacks output messages in ways that fit the style of the other parts of the build logs (see below for some shell script functions to help with this).

If a buildpack needs to communicate environment to next buildpack it can write a script export in the buildpack’s directory which will be executed before the next buildpack is run.

If a buildpack needs to communicate environment to the application (e.g. PATH) it can write shell scripts to BUILD_DIR/.profile.d. These will be run during dyno startup. They will be run after the Heroku Config Vars are set in the environment. Each script should be named with a .sh suffix. There is no guarantee to the ordering which these files will be executed.

bin/release

The release script is run after compilation and returns meta data back to the build process. It is passed a single argument, the BUILD_DIR.

It must return, on standard output, a YAML formatted hash with upt to two keys: addons and default_process_types.

Any addons specified will be added to the application the first time it is deployed. The default_process_types key defines default Procfile entries to use.

A good start is with the following script which creates an empty YAML hash. It can be expanded with the required keys when needed:

#!/bin/sh

cat << EOF
---
EOF

CACHE_DIR

The CACHE_DIR is a directory that can be used to store build artifacts between builds. For example if a buildpack downloads and installs some software, it can store that in CACHE_DIR and then skip the download and install the next time.

Note, the API specifies that the CACHE_DIR may not exist, so if bin/compile wants to use it it should ensure it exists before doing so.

ENV_DIR

This directory contains files for each Heroku config var. For a config var FOO=bar there will be a file FOO with the contents bar.

bin/compile logging helpers

Since the compile script should use output format that is similar to the rest of the build process the following shell functions may be useful

log_header() {
  echo "-----> " $*
}

indent() {
  sed -u 's/^/       /'
}

log() {
  echo $* | indent
}

They can be used like this:

log_header "Staring Build..."

cat output.txt | indent

log "Build complete."

which will produce:

-----> Starting Build...
       ...contents of output.txt...
       Build complete.

Standard Joke

| Comments

Arabic-to-Roman number conversion is a common programming exercise, often used as a TDD exercise. It is an exercise on many Exercism.IO1 language tracks including my own track for Common Lisp2.

So how do you solve this in some arbitrary programming language?

Typically people write some code to loop through a mapping of numeric values to roman numeral digits and do repeated decrements while accumulating the roman numeral digits. That is to say if you start with 1970 and your mapping contains 1000 => "M"; 900 => "CM"; 500 = "D" you decrement by 1000 and accumulate "M", then since the resulting number is under 1000 you try the next mapping, decrement by 900 and accumulate "CM" etc.

How do you do this in Common Lisp?

(format t "~@r" 1970)

Yes: Built. In. To. The. Language.3 (In fact using ~@:r will provide output as using the alternate roman numeral format which does not have special cases for numbers like 4. i.e. it will output “IIII” rather than “IV”.)

Common Lisp4 is a language with an ANSI specification5. The specification process went from the early-to-mid 1980’s to the mid-1990s. Its goal was to create a single “common” Lisp collecting the good bits from all the splintered Lisp languages used at the time. I’ve heard that it involved a good amount of “horse-trading” as each interested party tried to get the resulting language to contain their special or idiosyncratic feature. So how did it come to have this odd feature?

With a little digging I have found the real story: It was a joke.

In “The Evolution of Lisp”6 Richard Gabriel explains that in 1974 Guy Steele Jr put roman numeral handling into MacLisp7 as a hack. BASE and IBASE were variables which specified the output and input bases for numbers. Before his hack they were limited to vales between 2 and 36. Steele added the ability for them to be set to ROMAN. Thus if set code like (+ i 1) would evaluate to ii since the expression would have been read in as the equivalent of (+ 1 1) and upon output the value 2 would be represented in roman numerals. (It seems in the original announcement of this ‘feature’ that you could also set BASE and IBASE to CUNEIFORM which would cause your Lisp to become wedged. Oh so witty.)

From that time this feature became a bit of a traditional hack - being ported to ZetaLisp8 as part of FORMAT.

Thus when Common Lisp was being created this feature was included in FORMAT9 as it was a normal feature of that function in the Lisps that Common Lisp was being aggregated from.

Since then this feature also found its way into Clojure10 as part of its cl-format11 function.


Writing Mix Tasks for Fun and Profit.

| Comments

I’m beginning to learn a little Elixir & Phoenix and I ran into a case where I wish I had a Mix task for something. Specifically I wanted to run npm scripts with mix so I’d only have one command to run instead of both mix and npm for my toy Phoenix project.

Writing a Mix task is reasonably straightforward with only a few steps:

  1. If you want to create a mix task called “echo” then create a module called Mix.Tasks.Echo. The task name seen in mix help is based upon the name of the module minus Mix.Tasks. (The module needs to be called Mix.Tasks.Something otherwise mix will not see it.)
  2. Add use Mix.Task to this module.
  3. Write a public method called run. It has the type signature: run([binary]) :: any. That means that it will get a list of strings (the command line arguments) and can return anything.
  4. Add a @shortdoc. This will be used as the text in mix help. Without this your task will not appear in mix help but will still be usable.
  5. Optionally add a @moduledoc. This will be used if you run mix help YOURTASK

You can put this module where ever you want in lib but typically you would put it into lib/mix/tasks.

That’s it.

An interesting thing I found was that step #2 was not actually needed. That behaviour defines @shortdoc so without it you cannot use @shortdoc to add the task to mix help output.

Since I was creating a mix task for use in a build I needed to make sure that if the task was not successful that mix would return an error code so that the shell could see the error and fail a build. At first I assumed that the return value of the task was how that would be done; however I didn’t find much documentation about this. I experimented with some likely return values like :error or {:error, "something"} but that had no effect, it always returned a zero exit status to the shell. Ultimately I choose to raise an error when the task didn’t work and that definitely caused a non-zero exit status.

If you want to see the end of result of this experimentation you can check out my first ever hex package: mix-npm. The source can be found in the GitHub repository: verdammelt/mix_npm.

Learning From My Apprentice

| Comments

My company, Cyrus Innovation, has started an Apprenticeship program. This program involves bringing on people who are brand new to programming and pair them with a Senior developer to train them in good practices. About a month ago I ‘acquired’ an Apprentice: Paige.

I’ve never done this sort of thing before and I quickly came to realize that the relationship wasn’t just Master & Apprentice but really Apprentice-Master & Apprentice. I am not only learning as I go what it means to have an Apprentice but also learning from my Apprentice.

Even more than normally during pair-programming I feel I need to explain myself. Not only explaining how to do something in the programming language, but how to do it with our tools, and how we’ll do something in the context of our team. Add to that explaining the why of it all as well. This exercise helps me remember and reconsider why I do things the way I do, why they are important.

This also brings to light good practices which I have chosen not to do in this current situation. Perhaps there are reasons, perhaps those reasons are good, sometimes they are not. By having to explain the trade-offs and reconsider the bad reasons I am learning why I make these decisions.

Also by embodying some good behaviors she is teaching me some tricks about how to learn. She is a careful attentive listener, always asking questions to get clarification. She also tests her knowledge by sharing what she does know (even if not sure) which either adds to the collective knowledge or is corrected.

So I hope she is learning something by pair-programming with me; but I know I’m learning something from her.

Quickie: Less Columns on Your Board

| Comments

So you have a “Agile Board”(TM) congratulations! If you are like any of the people I’ve worked with recently you’ve got lots of columns to keep track of all the possible states a story card could be in.

Back away from the board…

You should start with very few columns. I think four should be plenty:

  • Backlog
  • Current
  • Doing
  • Done

Backlog is the place to keep all your new ideas. Make sure the next priorities are at the top and ready to go, as you go down the list don’t worry about the rest too much, they can stay vague.

Current is for the things you are planning on doing this iteration. Use the Goldilocks principle. Don’t agree to do too much or too little. Just the Right(TM) amount.

Doing is for things that are being worked on RIGHT NOW. A good number of items here is a function of the number of people on your team and how they collaborate (pairing, mobbing, solo if you must). If there are too few or too many items there is a problem - discuss it. If something stays here for days that is the sign of a problem - discuss it.

Done is for keeping track of the valuable things you are delivering as a team. Rejoice. Throw them away after sufficient rejoicing (sometime after the next iteration starts is a good time).

What about Ready for QA? or Ready for Deploy? you ask. I’d ask why isn’t QA and/or Deploy part of your definition of done?

What about Blocked. OK this one might be useful. But a red sticky (or equivalent for non-physical boards) on the card is probably enough. Moving it to another column makes it less visible, and a card being blocked is a problem and we want problems to be visible.

Of course it is your process not mine. Use the columns you need. But know why you need them, and feel free to add and remove them if after discussion you realize the board is not serving you anymore.

The Craftsperson and Their Tools

| Comments

When I think of how “Craftspeople” do their work I think of tools. Good tools. They pick good tools because they know that it is easier to get the job done when you have the right tool, and it is well made. They don’t use a tool that is the wrong size for them, the tool fits their hand. They may also modify their tools to better fit their hand.

Sometimes a craftsperson will make entirely new tools for their work. It might be a special jig for cutting the same type of cut in a lot of lumber, or for drilling the same sort of hole. Sometimes these special jigs even come into common use, such as a door lock installation jig, or a mitre box.

Because a craftsperson uses their own tools, and special tools, they might seem to have a handicap when they don’t have those tools. But a unless the job cannot be done except with a particular tool (it is very difficult to cut a piece of wood with a hammer), they can still get the job done (albeit perhaps a bit slower).

This is because they know how to do the job; the tools are just how they go about it.

Some people think that the idiom: “A poor craftsman blames his tools.” implies that tools are not important. This does not mean that tools are not important! This means that a craftsperson knows that the failure to do a good job is not the fault of the tool; but the fault of the craftsperson: their skill, or their choice of tool.

Do you pick the right tool for the job? Do you change it to fit your hand? Do you make special tools for the work?

Gotcha of Using YouTubePlayerFragment and ViewPager for Android

| Comments

(Cross posted to the Cyrus Innovation Blog)

Recently I learned some Android programming by writing a simple app for a client. It was a great opportunity to learn the platform and how “easy” it is to write an app. I ran into one ‘gotcha’ that I thought might be valuable to others.

One feature that was needed was a swipeable carousel of YouTube videos. Google provides some widgets for showing YouTube videos on an Android device and YouTubePlayerFragment was a (almost) perfect fit for my needs1. Also ViewPager was just the thing for creating the swipeable list of items. It was easy enough to create a subclass of FragmentPageAdapter which knew the list of videos and created YouTubePlayerFragments as needed (actually a subclass whose job was to handle the initialization of the YouTubePlayerFragment).

While this was easy to code - it was not so easy to make it actually work.

Trying to play videos resulted in a cryptic message about the player not playing because it was not visible on the screen. The coordinates in the error message made it seem like the object was way to the left of the visible screen. That was the first clue. It was perplexing though since the player was quite obviously right there on the screen.

Some debugging gave me the second clue I needed. When I pressed play on the player on the screen, multiple players were firing events saying that they were playing. Multiple players?

Reading2 into the documentation of ViewPager some more told me that it will request multiple views from the ViewPageAdapter, so that other views are “ready to go”. But why did they all respond when I clicked on one of them?

More debugging did not solve the mystery but solidified my hypothesis: The YouTubePlayer and/or YouTubePlayerFragment has state shared between all their instances. That is the only explanation that would fit the observed behavior.

So I needed a way to ensure that only one YouTubePlayer was in play at a time. The ViewPager documentation says you can change the number of other pages that will be created. Changing that did not work for me - at least one other view was always created. That left me with ensuring that only one player was initialized.

I tried various event listeners but found that none of them fit the need. Sometimes I would get an event firing both on the active and the inactive viewer and it was not possible to tell the difference. Finally I found one thing that did seem consistent and usable: setUserVisibleHint. It was called on the fragment with a true value when that fragment was the one shown to the user and was called with false when it was not. So I made sure my fragment was not initialized until it got told that it was visible; and then released it when it was no longer visible.


  1. Except for the supremely annoying fact that the YouTube player widgets DO NOT WORK on emulators. So I had to do all this work with a physical device tethered to my machine. Like a savage. 

  2. Reading is Fundamental