(Recently I started hearing about this note taking system called Zettelkasten. (Seems, as usual, I’m a bit late to the game as the Interwebs was talking about it more in 2020 and prior…) What follows is some notes and thoughts about it and how it connects with other thoughts.)

Zettelkasten

Zettelkasten (or ‘Slip Box’ / ‘Card Box’ in English) is a note taking system used (and perhaps invented) by Niklas Luhmann. He said it was the way he was able to so prolific in publishing academic papers. It involves writing notes/thoughts on small cards and organizing them in a tree structure, physically in a card box. Each card gets an index number (based upon where it is the tree) and references to other cards if applicable. Index numbers were made of letters and numbers alternating thus allowing for the branching. Cards “1” and “2” are not particularly related, but “1a1” is a note that continues from “1a”, while “1b” is a note continuing from “1” but not necessarily related to “1a”. What Luhmann thought was very important was that there was not any organization a priori. He felt that “accident” was very important and having an organization ahead of time would limit the possible thoughts one could have.

More details on this system and how to use it can be found in Luhmann’s paper and also in Sönke’s How to Take Smart Notes.

GTD

How does this connect with GTD? Luhmann’s process of reading and taking notes reminds me a little of the GTD’s separation of capture and processing.

Luhmann’s system had three types of notes: “fleeting”, “literature” and “permanent”. The only differences as far as I can really see between “fleeting” and “literature” notes is where they came from and how much supporting bibliographic material they may have. Both are written down at any time and collected together. Later (but not too much later) they are reviewed and either discarded or turned into permanent notes. Permanent notes are stored in the Zettelkasten. As notes from the inbox are processed one decides what one needs to do with them, where do they fit into the system.

Memex

In 1945 Vannevar Bush published an article in The Atlantic Monthly entitled “As We May Think”. In it he describes the “Memex” a sort of work desk / computer which could be used to search for and view documents (on microfilm) stored in the device. More data could be added to it - either personal notes / documents which would be scanned in; annotations on the documents; or even new documents bought or given to the user by others (you could make copies of documents to share with others…)

Basically if you squint at it just right he described a sort of World Wide Web. Documents available at your fingertips and documents were linked together. This availability and linking would allow people to do research on any topic quite easily, synthesizing existing information into new information.

One thing he mentioned that the Memex would have tat we do not have in the WWW is his idea of “trails”. With the Memex you would be able to annotate and save the “trail” of our research, that is you could save the fact that looking at document A led you to document B which lead you to a footnote of document E etc. etc.. These trails were another piece of data one could store and share with the Memex. A trail could be shared with a colleague thus not only sharing the documents you looked at, but how you looked at them.

I think it could be interesting if this idea of trails could be developed.

Synthesis

So how do these thoughts connect? For me the idea of the Zettelkasten and of the Memex are similar, how to store and access information; how to synthesize information into new information. That alone is a good thing - but how to deal with all the information you may be putting into the system? Via GTD’s separation of capture & process, Zettelkasten’s fleeting vs. permanent notes.

A new piece of information or thought can be captured quickly - now your brain can go back to the job at hand. Later (hopefully not too much later) one goes through all that captured information and decides what to do with it. Some of it may, in retrospect be not important - you discard. Some may be important - you store it. Not only that but during storage one synthesizes it with existing knowledge which may immediately create new information.

(As I was preparing to write this blog I see that 1) a few other people seem to be discussing around this topic (e.g. David Vujic), Erik Engheim, et al.) and 2) that I already discussed this some 10 years ago TDD and REPL Analogies?! Just adding this note to make it clear that this is nothing new, just my thoughts and experiences.)

Background

I am the author/maintainer of an Exericism track for Common Lisp. Currently we are doing a big push to prepare and release v3 of the platform. During this push I’ve been doing more Common Lisp coding than usual¹ which has brought me back to REPL driven development.

I personally find it very useful to be able to “play around” or explore a problem before sitting down to actual produce a software solution. Sometimes the code, as I explore and play, becomes the “real” code I will keep. Thus this is not a prototype or spike where the output would be thrown away, or at least not shipped to production².

At the time of my earlier writing I was working in a language which did not have a strong REPL offering (or any at all) and found TDD to be a way for me to do this exploration in a reasonable fashion. I found, lacking a REPL, TDD provided me a fast feedback cycle which was necessary for my exploratory approach. Having a REPL shortens that feedback cycle for exploration.

What is REPL Driven development

A REPL with full editing features (such as SLIME or SLY in GNU Emacs) combines the power of the language, the power of the editor/IDE, with fast feedback that allows me to explore and play with ideas.

Like TDD, REPL Driven Development/Design lets me experiment with different interface designs of how my modules³ will fit together. It also allows me to quickly check that my most recent changes haven’ t caused any problem.

I can also use it when I’m unclear on what the solution might even be. For example when consuming a JSON formatted configuration file for my exercism track, and I want to better understand how the different parts of the data related to each other, where there may be inconsistencies or lacunae in the data.

The general workflow is to work with some code in the REPL, testing and designing it and when it seems ‘right’, saving it to a source file. The workflow can also include working in source files and evaluating parts of the code to load them into the REPL. In a well integrated REPL there is little difference in the editing experience between the REPL and a source file.

In Common Lisp the REPL includes a debugger with a sophisticated condition and restart system which lets me write code I wish I had, then when I run it I am asked what to do about the undefined functions or variables. I can choose to quickly define them or stub them out and then continue, or I can quit the debugger to start the feedback loop again after making further changes.

What it is not

One thing that this cycle does not provide is a a set of tests which can be run repeatedly as you get with TDD. However the interactions at the REPL involve many of the same test cases you’d use in TDD - typical data, empty data, aberrant data, etc. So those can be collected, mixed with some assertions and used as the tests to save for later regression testing / documentation.

Next steps

To be honest, adding bindings when stopped in the debugger is not part of my workflow because it is still not ‘natural’ for me to do, my fingers all too quickly dismiss the debugger from lots of habit. This workflow is my next experiment.

Heroku buildpacks are used to prepare (/e.g./ build/compile) an application to allow it to be run by Heroku on their dynos. There are many that exist so it is not likely that you will need to write your own. And that is quite true for the one I wrote. But I wrote one partly to find out how to do it, so let me share what I learned.

The API for Buildpacks is pretty simple and it can be easy to write one. The API is three scripts that will be run by Heroku when it runs the buildpack: detect, compile, release. Buildpacks are git repositories so it is common to host them on GitHub (but that is not required).

While I say that the API is simple, I hope it is useful to summarize details of the API here. Perhaps restating it with my own words will help others (or just me in the future).

Environment that the buildpack is run in

Buildpacks are run on the same stack as the application is.

Buildpacks are run in their own directory. This is not the directory where the application code is. Also this buildpack directory is not available to later buildpacks nor the application. So if the buildpack creates build artifacts those must be put somewhere else.

Besides the buildpack directory there are three other directories available to the buildpack:

• BUILD_DIR: this is the directory where the application code is.
• CACHE_DIR: this is a directory which can be used to cache build artifacts (see below).
• ENV_DIR: this is a directory which contains information about the [Heroku Config Vars][heroku-confi-vars] (see blow).

bin/detect

This script’s job is to decide if this buildpack should be run. It is passed a single argument, the BUILD_DIR. This script typically will work by checking for a particular file in BUILD_DIR. If the buildpack should not run the script should return a non-zero value. If the buildpack should run then besides a zero return value it must also print to standard output a “human readable” name. This will show up in the build logs and shows that the buildpack is chosen.

bin/compile

This script is a huge part of a build pack. It has the job of compiling/preparing the application. It is passed three arguments: BUILD_DIR, CACHE_DIR and ENV_DIR.

There are two environment variables available to the compile script:

• STACK: which stack the buildpack is running on
• SOURCE_VERSION: the “version”. For applications deployed via git pushes that will be a commit SHA.

Heroku Config Vars are not available as environment variables to the buildpack. But they are available via the ENV_DIR (see below).

Any output to standard output will appear in the build log. It is recommended that buildpacks output messages in ways that fit the style of the other parts of the build logs (see below for some shell script functions to help with this).

If a buildpack needs to communicate environment to next buildpack it can write a script export in the buildpack’s directory which will be executed before the next buildpack is run.

If a buildpack needs to communicate environment to the application (e.g. PATH) it can write shell scripts to BUILD_DIR/.profile.d. These will be run during dyno startup. They will be run after the Heroku Config Vars are set in the environment. Each script should be named with a .sh suffix. There is no guarantee to the ordering which these files will be executed.

bin/release

The release script is run after compilation and returns meta data back to the build process. It is passed a single argument, the BUILD_DIR.

It must return, on standard output, a YAML formatted hash with upt to two keys: addons and default_process_types.

Any addons specified will be added to the application the first time it is deployed. The default_process_types key defines default Procfile entries to use.

A good start is with the following script which creates an empty YAML hash. It can be expanded with the required keys when needed:

#!/bin/sh

cat << EOF
---
EOF

CACHE_DIR

The CACHE_DIR is a directory that can be used to store build artifacts between builds. For example if a buildpack downloads and installs some software, it can store that in CACHE_DIR and then skip the download and install the next time.

Note, the API specifies that the CACHE_DIR may not exist, so if bin/compile wants to use it it should ensure it exists before doing so.

ENV_DIR

This directory contains files for each Heroku config var. For a config var FOO=bar there will be a file FOO with the contents bar.

bin/compile logging helpers

Since the compile script should use output format that is similar to the rest of the build process the following shell functions may be useful

log_header() {
  echo "-----> " $*
}

indent() {
  sed -u 's/^/       /'
}

log() {
  echo $* | indent
}

They can be used like this:

log_header "Staring Build..."

cat output.txt | indent

log "Build complete."

which will produce:

-----> Starting Build...
       ...contents of output.txt...
       Build complete.

Arabic-to-Roman number conversion is a common programming exercise, often used as a TDD exercise. It is an exercise on many Exercism.IO¹ language tracks including my own track for Common Lisp².

So how do you solve this in some arbitrary programming language?

Typically people write some code to loop through a mapping of numeric values to roman numeral digits and do repeated decrements while accumulating the roman numeral digits. That is to say if you start with 1970 and your mapping contains 1000 => "M"; 900 => "CM"; 500 = "D" you decrement by 1000 and accumulate "M", then since the resulting number is under 1000 you try the next mapping, decrement by 900 and accumulate "CM" etc.

How do you do this in Common Lisp?

(format t "~@r" 1970)

Yes: Built. In. To. The. Language.³ (In fact using ~@:r will provide output as using the alternate roman numeral format which does not have special cases for numbers like 4. i.e. it will output “IIII” rather than “IV”.)

Common Lisp⁴ is a language with an ANSI specification⁵. The specification process went from the early-to-mid 1980’s to the mid-1990s. Its goal was to create a single “common” Lisp collecting the good bits from all the splintered Lisp languages used at the time. I’ve heard that it involved a good amount of “horse-trading” as each interested party tried to get the resulting language to contain their special or idiosyncratic feature. So how did it come to have this odd feature?

With a little digging I have found the real story: It was a joke.

In “The Evolution of Lisp”⁶ Richard Gabriel explains that in 1974 Guy Steele Jr put roman numeral handling into MacLisp⁷ as a hack. BASE and IBASE were variables which specified the output and input bases for numbers. Before his hack they were limited to vales between 2 and 36. Steele added the ability for them to be set to ROMAN. Thus if set code like (+ i 1) would evaluate to ii since the expression would have been read in as the equivalent of (+ 1 1) and upon output the value 2 would be represented in roman numerals. (It seems in the original announcement of this ‘feature’ that you could also set BASE and IBASE to CUNEIFORM which would cause your Lisp to become wedged. Oh so witty.)

From that time this feature became a bit of a traditional hack - being ported to ZetaLisp⁸ as part of FORMAT.

Thus when Common Lisp was being created this feature was included in FORMAT⁹ as it was a normal feature of that function in the Lisps that Common Lisp was being aggregated from.

Since then this feature also found its way into Clojure¹⁰ as part of its cl-format¹¹ function.

I’m beginning to learn a little Elixir & Phoenix and I ran into a case where I wish I had a Mix task for something. Specifically I wanted to run npm scripts with mix so I’d only have one command to run instead of both mix and npm for my toy Phoenix project.

Writing a Mix task is reasonably straightforward with only a few steps:

1. If you want to create a mix task called “echo” then create a module called Mix.Tasks.Echo. The task name seen in mix help is based upon the name of the module minus Mix.Tasks. (The module needs to be called Mix.Tasks.Something otherwise mix will not see it.)
2. Add use Mix.Task to this module.
3. Write a public method called run. It has the type signature: run([binary]) :: any. That means that it will get a list of strings (the command line arguments) and can return anything.
4. Add a @shortdoc. This will be used as the text in mix help. Without this your task will not appear in mix help but will still be usable.
5. Optionally add a @moduledoc. This will be used if you run mix help YOURTASK

You can put this module where ever you want in lib but typically you would put it into lib/mix/tasks.

That’s it.

An interesting thing I found was that step #2 was not actually needed. That behaviour defines @shortdoc so without it you cannot use @shortdoc to add the task to mix help output.

Since I was creating a mix task for use in a build I needed to make sure that if the task was not successful that mix would return an error code so that the shell could see the error and fail a build. At first I assumed that the return value of the task was how that would be done; however I didn’t find much documentation about this. I experimented with some likely return values like :error or {:error, "something"} but that had no effect, it always returned a zero exit status to the shell. Ultimately I choose to raise an error when the task didn’t work and that definitely caused a non-zero exit status.

If you want to see the end of result of this experimentation you can check out my first ever hex package: mix-npm. The source can be found in the GitHub repository: verdammelt/mix_npm.

My company, Cyrus Innovation, has started an Apprenticeship program. This program involves bringing on people who are brand new to programming and pair them with a Senior developer to train them in good practices. About a month ago I ‘acquired’ an Apprentice: Paige.

I’ve never done this sort of thing before and I quickly came to realize that the relationship wasn’t just Master & Apprentice but really Apprentice-Master & Apprentice. I am not only learning as I go what it means to have an Apprentice but also learning from my Apprentice.

Even more than normally during pair-programming I feel I need to explain myself. Not only explaining how to do something in the programming language, but how to do it with our tools, and how we’ll do something in the context of our team. Add to that explaining the why of it all as well. This exercise helps me remember and reconsider why I do things the way I do, why they are important.

This also brings to light good practices which I have chosen not to do in this current situation. Perhaps there are reasons, perhaps those reasons are good, sometimes they are not. By having to explain the trade-offs and reconsider the bad reasons I am learning why I make these decisions.

Also by embodying some good behaviors she is teaching me some tricks about how to learn. She is a careful attentive listener, always asking questions to get clarification. She also tests her knowledge by sharing what she does know (even if not sure) which either adds to the collective knowledge or is corrected.

So I hope she is learning something by pair-programming with me; but I know I’m learning something from her.

So you have a “Agile Board”(TM) congratulations! If you are like any of the people I’ve worked with recently you’ve got lots of columns to keep track of all the possible states a story card could be in.

Back away from the board…

You should start with very few columns. I think four should be plenty:

• Backlog
• Current
• Doing
• Done

Backlog is the place to keep all your new ideas. Make sure the next priorities are at the top and ready to go, as you go down the list don’t worry about the rest too much, they can stay vague.

Current is for the things you are planning on doing this iteration. Use the Goldilocks principle. Don’t agree to do too much or too little. Just the Right(TM) amount.

Doing is for things that are being worked on RIGHT NOW. A good number of items here is a function of the number of people on your team and how they collaborate (pairing, mobbing, solo if you must). If there are too few or too many items there is a problem - discuss it. If something stays here for days that is the sign of a problem - discuss it.

Done is for keeping track of the valuable things you are delivering as a team. Rejoice. Throw them away after sufficient rejoicing (sometime after the next iteration starts is a good time).

What about Ready for QA? or Ready for Deploy? you ask. I’d ask why isn’t QA and/or Deploy part of your definition of done?

What about Blocked. OK this one might be useful. But a red sticky (or equivalent for non-physical boards) on the card is probably enough. Moving it to another column makes it less visible, and a card being blocked is a problem and we want problems to be visible.

Of course it is your process not mine. Use the columns you need. But know why you need them, and feel free to add and remove them if after discussion you realize the board is not serving you anymore.

When I think of how “Craftspeople” do their work I think of tools. Good tools. They pick good tools because they know that it is easier to get the job done when you have the right tool, and it is well made. They don’t use a tool that is the wrong size for them, the tool fits their hand. They may also modify their tools to better fit their hand.

Sometimes a craftsperson will make entirely new tools for their work. It might be a special jig for cutting the same type of cut in a lot of lumber, or for drilling the same sort of hole. Sometimes these special jigs even come into common use, such as a door lock installation jig, or a mitre box.

Because a craftsperson uses their own tools, and special tools, they might seem to have a handicap when they don’t have those tools. But a unless the job cannot be done except with a particular tool (it is very difficult to cut a piece of wood with a hammer), they can still get the job done (albeit perhaps a bit slower).

This is because they know how to do the job; the tools are just how they go about it.

Some people think that the idiom: “A poor craftsman blames his tools.” implies that tools are not important. This does not mean that tools are not important! This means that a craftsperson knows that the failure to do a good job is not the fault of the tool; but the fault of the craftsperson: their skill, or their choice of tool.

Do you pick the right tool for the job? Do you change it to fit your hand? Do you make special tools for the work?

(Cross posted to the Cyrus Innovation Blog)

Recently I learned some Android programming by writing a simple app for a client. It was a great opportunity to learn the platform and how “easy” it is to write an app. I ran into one ‘gotcha’ that I thought might be valuable to others.

One feature that was needed was a swipeable carousel of YouTube videos. Google provides some widgets for showing YouTube videos on an Android device and YouTubePlayerFragment was a (almost) perfect fit for my needs¹. Also ViewPager was just the thing for creating the swipeable list of items. It was easy enough to create a subclass of FragmentPageAdapter which knew the list of videos and created YouTubePlayerFragments as needed (actually a subclass whose job was to handle the initialization of the YouTubePlayerFragment).

While this was easy to code - it was not so easy to make it actually work.

Trying to play videos resulted in a cryptic message about the player not playing because it was not visible on the screen. The coordinates in the error message made it seem like the object was way to the left of the visible screen. That was the first clue. It was perplexing though since the player was quite obviously right there on the screen.

Some debugging gave me the second clue I needed. When I pressed play on the player on the screen, multiple players were firing events saying that they were playing. Multiple players?

Reading² into the documentation of ViewPager some more told me that it will request multiple views from the ViewPageAdapter, so that other views are “ready to go”. But why did they all respond when I clicked on one of them?

More debugging did not solve the mystery but solidified my hypothesis: The YouTubePlayer and/or YouTubePlayerFragment has state shared between all their instances. That is the only explanation that would fit the observed behavior.

So I needed a way to ensure that only one YouTubePlayer was in play at a time. The ViewPager documentation says you can change the number of other pages that will be created. Changing that did not work for me - at least one other view was always created. That left me with ensuring that only one player was initialized.

I tried various event listeners but found that none of them fit the need. Sometimes I would get an event firing both on the active and the inactive viewer and it was not possible to tell the difference. Finally I found one thing that did seem consistent and usable: setUserVisibleHint. It was called on the fragment with a true value when that fragment was the one shown to the user and was called with false when it was not. So I made sure my fragment was not initialized until it got told that it was visible; and then released it when it was no longer visible.

(Cross posted to the Cyrus Innovation Blog)

It looks likely that I’ll be doing some ReactNative work soon so I took some time to start setting up my Emacs environment. All my relevant setup can be found in the init-react.el file in my GitHub dotfiles repository. This is likely to change so the previous link may not match the code below. The code below matches specifically the initial version (59e7728).

Hereinbelow I will add more annotation to those already found in that file along with snippets of code.

The first thing that needed to be set up was a mode for React code. React code files can mix Javascript with HTML markup and it does not appear that js-mode (the built-in JavaScript mode) handles that. After a short Googling it looks like web-mode is a mode that can handle it. After some brief testing it does appear to work reasonably. It appears that the React/ReactNative community has not decided to use *.js or *.jsx as the extension for the code files and since web-mode appears to handle JavaScript just fine I chose to use it in all cases.

(add-to-list 'auto-mode-alist '("\\.jsx?$" . web-mode))

The next thing I wanted to get set up was a linter. I thought this especially important as React uses an ES6 dialect of JavaScript which I am not entirely familiar with yet and a linter can help me “do the right thing”. With a suggestion from a coworker (who has done some React work) I chose ESLint with the AirBnB configuration settings. These defaults prompted me to standardize on two spaces for indentation. (Setting js-indent-level as js-mode is still in use for JSON files.)

  (setq web-mode-markup-indent-offset 2
        web-mode-css-indent-offset 2
        web-mode-code-indent-offset 2)
  (setq js-indent-level 2)

Setting up the linter to run via flycheck took a small amount of work since I don’t like to install project specific tools globally. (I know that this is contrary to current mores, but I have been tripped up by global vs. local installations before so I shy away from them when I can.)

First I needed to integrate NVM with Emacs so that Emacs could run ESLint at all.

(require 'nvm)
(nvm-use (caar (last (nvm--installed-versions))))

(The choice to use the last version found is totally arbitrary. If and when I get more versions of Node.js on my machine I’ll have to make a more careful choice.)

Next I hooked into projectile to look for a locally installed ESLint and use it if found. The projectile-after-switch-project-hook functions are called after Projectile has switched directories to the project so one can simply check the project for the desired file.

(add-hook 'projectile-after-switch-project-hook 'mjs/setup-local-eslint)

(defun mjs/setup-local-eslint ()
    "If ESLint found in node_modules directory - use that for flycheck.
Intended for use in PROJECTILE-AFTER-SWITCH-PROJECT-HOOK."
    (interactive)
    (let ((local-eslint (expand-file-name "./node_modules/.bin/eslint")))
      (setq flycheck-javascript-eslint-executable
            (and (file-exists-p local-eslint) local-eslint))))

(Note: the function is interactive because I found at least a few times I was looking at a JavaScript file which I had come to not via projectile. By making it interactive lets me use it manually in the rare case I need to.)

Flycheck’s ESLint integration is limited to only certain modes and web-mode is not one of them so I needed to add that to the white-list of modes

  (with-eval-after-load 'flycheck
    (push 'web-mode (flycheck-checker-get 'javascript-eslint 'modes))))

With that I can easily write React/ReactNative code with all the bells and whistles I like.

Later I will add support for building and testing I’m sure. But first I need to determine what building & testing in a ReactNative environment will even look like.