REPL-Driven Development and Learning Velocity

The last Lisp advantage

Our next Apropos will feature Nathan Marz on May 20. Be sure to subscribe!

---

REPL-Driven Development and Learning Velocity

The main advantage of Lisps (including Clojure) over other languages is the REPL (Read-Eval-Print Loop). Lisp used to have a bunch of advantages (if statements, garbage collection, built-in data structures, first-class closures, etc.), but these are common now. The last holdout is the REPL.

The term REPL has diluted, so I should define it: A REPL is a way to interactively inspect and modify running, partially correct software. My typical workflow is to open my editor, start the REPL, and start the application server from within it. I can make requests from my browser (to the running server), recompile functions, run functions, add new libraries, and inspect variables.

The REPL accelerates learning by increasing the speed and information richness of feedback. While programming, you learn about:

• Your problem domain

• The languages and libraries you’re using

• The existing codebase and its behavior

The REPL improves the latency and bandwidth of feedback. Faster and richer feedback helps you learn. It lets you ask more questions, check your assumptions, and learn from and correct your mistakes. Fast, rich feedback is essential to achieving a flow state.

The obvious contrast with REPLs is with the mainstream edit-compile-run (ECR) loop that most languages enable. You edit your source code, run the compiler, and run the code. Let’s look at the main differences between REPL and ECR:

• In ECR, your state starts from scratch. In REPL, your state is maintained throughout. All of the variables you set up are still there. Web sessions are still open.

• In ECR, your compiler may reject your program, forcing you back into the Edit phase. Nothing is running, so you must fix it before continuing. In REPL, when the compiler rejects your change, the system is still running with the old code, so you can use runtime information.

• In ECR, if you want to try something out, you have to write an entire program to compile and run. In REPL, trying something out means typing the expression and hitting a keystroke.

The end result is that the ECR loop is much slower than the REPL. One of the benefits of modern incremental testing practices (like TDD) is that it approximates the fast feedback you get from the REPL:

• With testing, you do not maintain your state. However, you write the code to initialize the state for each test.

• With testing, you make small changes to the code before rerunning the tests so you are usually not far from a running program.

• With testing, it is easy to add a new test and run just that.

The advantage of testing is that you have a regression suite, which you don’t get from the REPL. But feedback in testing is poorer. I haven’t heard of anyone writing a test just to see what the result of an expression is. And, by the way, doing incremental testing with the REPL is a breeze. It’s like the best of both worlds.

The REPL gives you fast, rich feedback in three main ways:

• Maintaining state — Your running system is still running, with all in-memory stuff still loaded. This means that after editing a function and hitting a keystroke, you can inspect the result of your change with a delay that is below human perception.

• Running small expressions — Within your running system, you can understand the return value from any expression you can write, including expressions calling your code or libraries you use. The cost of running these expressions is below a psychological threshold, so they feel almost free compared to having to scaffold a test or a public static void main(String[] argv).

• Ad hoc inspection of the running system — This is a big one you gain skill with over time. You can do anything you can imagine, from running your partially completed function (just to make sure it does what you expect) to printing out the value of a global variable (that you saved values to from your last web request). The flexibility makes tools like debuggers feel rigid.

However, other languages have chipped away at the advantages of REPL-Driven Development. I already talked about incremental testing approaches (like TDD) and how they approximate the feedback of the REPL. But there are more technologies that provide good feedback in the mainstream ECR (Edit-Compile-Run) paradigm:

• Static analysis — You can get feedback on problems without leaving the edit phase with tools like LSP and squiggles under your code.

• Static types — If you’ve got good types and you know how to use them, the Edit-Compile cycle can also give you rich feedback. The question is whether your compiler is fast enough to keep up.

• IDEs with run buttons — Many IDEs for compiled languages use their own, incremental compiler. The code is constantly being compiled as you edit. When you hit the Run button, you’re essentially cutting out the Compile phase (which can often be very lengthy). If you can set it up to run a small expression at a keystroke, you’re very close.

• Autocomplete — Autocomplete speeds up the Edit phase. Autocomplete with the REPL is a cinch. You can inspect what variables are available in the environment dynamically. However, modern IDEs can use static analysis to aid autocomplete.

• Incremental testing — Incremental testing (like TDD) speeds up the Edit and Run phases. Added here just for completeness.

But don’t tell anyone: we can use these in addition to the REPL. In fact, Clojure has an excellent LSP and a great testing story. The only thing we don’t have is a great story about static typing.

Many languages claim that they have a REPL. But what they really have is an interactive prompt where you can type in expressions and get the result. This is great! But it doesn’t fully capture the full possibility of the REPL.

People often ask what’s missing, especially from a very dynamic language like JavaScript. I’ve tried to set up a REPL-Driven Development workflow in JavaScript, but I encountered these roadblocks. There are probably more:

• Redefining const: In Clojure, we pride ourselves on immutability. However, global variables are mutable so that we can redefine them during development. Unfortunately, JavaScript engines are strict about global variables defined with const. I found no way to redefine them once they were defined.

• Reloading code: It’s not clear how to reload code after it has changed. Let’s say I have a module called catfarm.js and I modify one of the functions in it. My other module old-macdonald.js imports catfarm. How do I get old-macdonald to run the new code? The engines I tried did not re-read the imported modules, instead opting for a cached version. In addition, even if they did, the old-macdonald code needs to be recompiled. Clojure’s global definitions have a mechanism to allow them to be redefined, and any accesses to them after redefinition are immediately available. If I recompile a function a in Clojure, the next time I call b (which calls a), it calls the new version of a.

• Calling functions from other modules: When you import a module, it’s basically a compiler directive where you say what you’re importing. But how do you call things from that aren’t imported? Why would you do that? Because you want to try them out! This, combined with not being able to re-import them, makes it really hard to incrementally work on your code. In Clojure, require is a function that loads new modules. And in-ns is a function that lets you navigate through the modules like in a directory structure from your terminal. require() in JavaScript (the old way of doing modules) worked more like that.

Hot module reloading is an attempt to address these limitations. It also seems like a major project with a lot of pitfalls. And it still doesn’t maintain state. Maybe it will get good enough one day to be close to REPL-Driven Development. Or maybe Node should have a “REPL” mode where const variables can be redefined and modules can be reloaded and navigated.

The biggest problem with REPLs is that they require an enormous amount of skill to operate effectively. To know what you need to recompile, you must understand how Clojure loads code and how Vars work. You need to navigate namespaces. Most importantly, you need to develop the habit of using the REPL, which is not built-in. It’s common for someone in a beginner’s chat to ask “what happens when I call foo on a nil?” My first reaction is: “Why are you asking me? Instead of typing it here, type it in the REPL!” People need to be indoctrinated.

Here are some ways to improve your use of the Clojure REPL:

• Next time you’re writing a function, use a rich comment block ( (comment …) ) to call that function. After the tiniest of modifications, call the function to inspect the return value. This is useful when writing long chains of functions.

• The next time you’re tempted to look up documentation for a function, do a little experiment to see what happens at the REPL. Some common questions to answer:

  • What type does a function return?

  • Can I call that with an empty value?

  • What does the zero-argument version do?

• Learn the keystrokes in your editor to evaluate:

  • The whole file

  • The current top-level form (often referred to as defn)

  • The expression just before the cursor

• Learn the keystrokes to run the tests from your editor. Run your tests. Edit and compile functions (see previous bullet). Run the tests again. This combines the best of incremental testing and RDD.

I love the REPL. I miss it when I go to other languages. The REPL is also forgiving. It can make up for a lot of missing tooling (like autocomplete and debuggers). The REPL also requires a lot of skill. Squiggles and LSP can immediately give you hints for making your code better. But the REPL requires a deep understanding of the language and lots of practice. This is ultimately the biggest barrier to its adoption. People who haven’t learned how to use the REPL don’t even know what they are missing.

PS: You can learn the art of REPL-Driven Development in my Beginner Clojure signature course.