About Articles Projects Links Apps Feed

Modern, functional Common Lisp: Myths and tips

Common Lisp is a great language. Of course, it cannot be the best at everything, it comes with its load of historical idiosyncrasies and it might not be the language of the future.

But there is one aspect of programming in which Common Lisp keeps it first position after so many years: “interactive programming.” Only few languages come close (maybe Racket is one of them). Interactive programming boosts your productivity so much that I believe it should always be part of the decision process when it comes to choosing a language to work with. Hopefully this advance in the practice of programming will be remembered when we design the language of the future.

I believe that for this reason alone Common Lisp is worth learning (and using).

In the following article, I’m not going to proselytize the language: it’s not perfect and it won’t suit everyone. I simply want to give a depiction more accurate than the common misconceptions about the language, and hopefully help the curious as well as the experienced user enhance their experience of programming.

I believe that many negative misconceptions about the language stem from myths of forgotten times which bear little relevance today.

Some common myths include:

Some of these myths subsist today even among Common Lisp developers, simply because the news of the recent developments in the ecosystem haven’t reached them. Indeed, Common Lisp, as an extensible language, has modernized tremendously over the last decades.

Common Lisp is a gigantic programming language and ecosystem. As such, this article is in no way exhaustive! I invite the community to reach out and share their recommendations, corrections and other discussions!

If you enjoy this article and would like to help me keep writing, consider chipping in, every little bit helps to keep me going :)

Thank you!

Getting started with the right tools and learning resources

Tools

Common Lisp has earned the reputation of being difficult to set up. Today, some good solutions exist to help newcomers:

  • Portacle is an all-in-one bundle that includes SBCL, Emacs, SLIME and more. While the user must learn the basics of Emacs, Portacle is known to be relatively friction-less.
  • The Guix package manager can set up a Common Lisp development environment in one go:

    guix install sbcl git emacs emacs-slime emacs-helm-slime emacs-company emacs-magit emacs-paredit emacs-rainbow-delimiters
    

    Replace guix install with guix environment --ad-hoc if you don’t want to persist a profile.

    If you prefer VI-style bindings:

    guix install sbcl git emacs emacs-slime emacs-helm-slime emacs-company emacs-magit emacs-paredit emacs-rainbow-delimiters emacs-evil emacs-evil-collection
    

    If you prefer Lispy over paredit to automate s-expression manipulation (such as automatically balancing parentheses):

    guix install sbcl git emacs emacs-slime emacs-helm-slime emacs-company emacs-magit emacs-lispy emacs-rainbow-delimiters
    

    Should you need Common Lisp libraries, you can also install many of them with Guix. For the rest, you’ll have to resort to Quicklisp which is not provided by Guix, so you’ll have to install it manually: https://www.quicklisp.org/beta/

Learning resources

The community-maintained Common Lisp cookbook is probably the best starting point with excellent references to everything from string manipulation to continuous integration.

It’s also a good portal to other resources such as:

That said, the introductory documentation can be rather overwhelming in my opinion and I’ve invited the community to work on a gentle, modern primer for the complete beginner:

https://github.com/LispCookbook/cl-cookbook/issues/270

Feedback welcome!

Editor and offline documentation

Emacs setup

Many Common Lisp developers use Emacs. You don’t have to, many other editors support Common Lisp as well, such as Atom and SLIMA.

SLIME and SLY are famous for providing a stellar interactive development experience.

On top of this, there are Helm packages to enhance your SLIME / SLY experience with live, narrowing fuzzy-search and completion for about everything:

Local Common Lisp HyperSpec

The HyperSpec is the Common Lisp reference available here: http://www.lispworks.com/documentation/HyperSpec/Front/index.htm.

While the website look is rather dated, SLY and SLIME will interface with it so that you can browse the specifications directly from Emacs without having to open a browser.

Moreover, the HyperSpec can be installed locally, to be consulted offline. See http://quickdocs.org/clhs/.

There is also a Guix package for it available from the Nonguix channel, and it’s also available in Zeal. (Thanks to /u/dzecniv for sharing this one.)

SLIME vs. SLY

SLY is a fork of SLIME. SLIME tends to be more conservative while SLY is slightly more featureful.

A few benefits of SLY over SLIME:

  • It has more consistent, cleaner code and reuses more of Emacs facilities (like comint-mode for the REPL).
  • It has a better color theme in my opinion.
  • It offers better fuzzy completion.
  • A feature unique to SLY: stickers allow the user to “stick” non-persistent code to help with debugging by annotating it. See https://github.com/joaotavora/sly#stickers.
  • The setup is essentially zero-config with sane defaults, with all the features like mrepl, stickers, etc. enabled by default. Of course you can disable the features you don’t like.
  • It compiles with ASDF which makes it easier to package, bundle or deploy.

SLIME does include some features that are only available in SLY as plugins, such as ASDF management. I recommend the following plugins for SLY:

SLIME / SLY tips and tricks

Check out the following not-so-well-known commands, they could dramatically improve your workflow! (Most of them have a SLY equivalent.)

  • slime-who-*
  • slime-eval-last-expression-in-repl (C-c C-j)
  • slime-list-compiler-notes
  • slime-export-symbol-at-point (C-c x)
  • slime-export-class, slime-export-structure
  • slime-trace-dialog-toggle-trace (C-c M-t)
  • slime-inspect-definition
  • slime-delete-system-fasls (Useful when .fasls are out-of-sync)
  • slime-repl-clear-buffer (C-c M-o: useful when lispy or paredit goes berserk)
  • slime-profile-package, then run the desired functions, then slime-profile-report.
  • hyperspec-lookup-format and hyperspec-lookup-reader-macro.

In particular, note that slime-who-specializes lists the methods of a given class, which answers a common complaint coming from people used to languages from the Algol family: the ability to complete the methods of the foo class by typing foo.<TAB>.

ASDF

ASDF is the de-facto standard build system for Common Lisp.

To use a FOO system, place it in in ~/common-lisp and call (asdf:load-system :FOO).

Better: from the SLIME or SLY REPL, press ,load-system and you’ll be provided with a list of systems that can be loaded.

Inferred systems

Historically ASDF required you to explicitly list the files to compile. This was cumbersome and error-prone, but thankfully it has been fixed some time ago with the package-inferred-system extension which I highly recommend:

https://common-lisp.net/project/asdf/asdf/The-package_002dinferred_002dsystem-extension.html#The-package_002dinferred_002dsystem-extension

Benefits:

  • No need to list files.
  • No need to list dependencies. With non-inferred-systems, the dependency list can be wrong, e.g. include too many of them or miss some. In that sense inferred-systems are more robust.
  • Enforces that every file must be a package.

Trick:

  • If your system has many files, and thus that many packages, it can get quickly cumbersome for the user to load all those packages one by one. Instead, you can create a “meta package” which imports all the other packages of your system, so that the user only has to load this one to access the whole system.
  • How can we get the list of dependencies since we don’t list them explicitly in the .asd anymore? Thankfully the snippet below will retrieve them programmatically for us:

    (declaim (ftype (function (string) list) package-dependencies))
    (defun package-dependencies (pkg-name)
      (let (depends)
        (labels ((iter (openlist)
    	       (if (null openlist) depends
    		   ;; is this a subsystem of foo?
    		   (let ((find (search  pkg-name (first openlist))))
    		     (if (and find (zerop find))
    			 (iter (append (asdf:system-depends-on (asdf:find-system (first openlist))) (rest openlist)))
    			 ;; if not, it's a direct dependency: collect it
    			 (progn
    			  (pushnew (first openlist) depends :test 'equalp)
    			  (iter (rest openlist))))))))
          (iter (list pkg-name)))))
    

    Then:

    (package-dependencies "dbus")
    ; => ("ironclad cl-xmlspam split-sequence flexi-streams ieee-floats iolib babel trivial-garbage alexandria
    

Modernizing the language with packages

Myth: Common Lisp is old and does not enjoy the features of modern languages from recent years.

Truth: While the Common Lisp standard is from the 1990s, many modern libraries have continuously updated the language.

One of the biggest selling points of Common Lisp (and Lisp languages in general) is that the language itself is extensible. This means that if new paradigms become fashionable, they can easily be added to the language.

In particular, Common Lisp has the reputation among functional programmers of being very imperative and object-oriented. While these styles were popular in the past, nothing prevents us from programming functionally in Common Lisp.

As with many languages, Common Lisp packages can be use-d, which means all symbols will be merged in the current name space, or import-ed, which means all symbols will be accessible only via the package prefix.

The benefit of use is that importing a language-enhancing package gives us a feel that the enhancements are first class.

Use-ing packages has a few drawbacks however, so be mindful of them. If you prefer the safe way, stick to package imports.

  • The drawback however is that if you change your mind and decide to import the package instead, you’ll have to add the package prefix to all the related symbols manually.

    This issue could be eliminated with sufficient editor support to refactor symbols by their packages. See https://github.com/slime/slime/issues/532 for a discussion.

  • It’s harder to see from just a glance which symbols comes from which package.
  • Use-ing a package without pinning its version (or even commit) is exposes your project to breakage if the package upstream adds a new symbols that would conflict with one of your symbols. As a minimum requirement to use-ing packages, make sure you pin their exact version. (Thanks to flamingbird for the detailed explanation.)

Local package nicknames

Since you should (almost always) import packages, you’ll have to prefix their symbols with the package name. Sometimes this package name can be lengthy to write: in this case, you should know that you can assign a local nickname to any package.

(defpackage my-package
  (:use #:cl)
  (:local-nicknames (#:a #:alexandria.dev.0)))

So now you can call the much shorter a:compose from your package!

See package-local-nicknames.

This approach is preferred over using the rename-package procedure which is global.

Recommended language-enhancing packages

The following packages are widespread and useful enough that can be import-ed without hesitation:

  • Alexandria: various language and functional programming procedures.
  • UIOP: Macros and operating-system-level procedures. Included with ASDF in most Common Lisp compilers.
  • Trivia: Pattern matching.

The following libraries are also extremely useful, although not to everyone’s taste:

  • Trivial-types: More common type definitions.
  • Serapeum: a huge collection of utilities (named let, local definitions, hooks, file “human” size, string manipulation, etc.), built as a complement to Alexandria and UIOP. It greatly extends the language.
  • Series: Lazy sequences and supporting higher-order functions.
  • defclass-std: Saner and more robust defaults for class definitions. In particular, this prevents accidental typos in the accessor or the initarg, and it sets a default value (the initform) automatically by default.
  • Sycamore: Functional data structure library

Other language-enhancing libraries:

  • Rove: The (only?) testing framework with coverage support.
  • ppcre: Advanced string manipulation with unicode support.
  • named-readtables: First class name space for readtables.
  • generic-cl: Generic function wrapper over various functions in the Common Lisp standard, such as equality predicates and sequence operations. Includes a few lazy sequence utilities.

Functional programming

Myth: Common Lisp is a procedural, object-oriented language. Truth: Many functional programming paradigms are available in Common Lisp.

Scheme developers in particular will be happy to find:

  • compose, curry in Alexandria.
  • named let, local definitions, flip and more in Serapeum.
  • match in Trivia.
  • Lazy sequences in Series.
  • Functional, immutable data structures in FSet, cl-hamt, Sycamore (thanks to Jach for pointing it out), folio2 (thanks to mikelevins for sharing this one), modf (thanks to dzecniv for sharing).

The availability of the Series package means in particular that you don’t have to use the (in)famous LOOP macros if you don’t like them.

Static, parametric and algebraic typing

Myth: Common Lisp does not support static typing.

Truth: ECL, CCL and SBCL at least support compile-time type-checking.

While not exactly static typing since it’s optional, the technical term for it is “gradual typing.”

Note that types defined with a satisfies predicate are not checked at compile-time unless at the top-level (this is true for SBCL 2.0 at least).

Classes are a whole different beast and there type checking can currently be done with a separate library: https://github.com/fisxoj/sanity-clause. Also see https://lisp-journey.gitlab.io/blog/how-to-check-slots-types-at-make-instance/.

Serapeum provides helpers for type manipulation: https://github.com/ruricolist/serapeum/blob/master/REFERENCE.md#types. defstar also provides syntactic sugar for type declaration (it also comes with a named let).

Algebraic types are supported out of the box, but notations can be improved with https://github.com/stylewarning/cl-algebraic-data-type.

There is even an ML-style library for Common Lisp: Coalton.

More on the topic:

Type examples

From the HyperSpec:

(defun equidimensional (a)
  (or (< (array-rank a) 2)
      (apply #'= (array-dimensions a)))) ; =>  EQUIDIMENSIONAL
(deftype square-matrix (&optional type size)
  `(and (array ,type (,size ,size))
	(satisfies equidimensional))) ; =>  SQUARE-MATRIX

Some useful types from the trivial-types package:

(deftype pathname-designator ()
  '(or string
       file-associated-stream
       pathname))

(deftype function-designator ()
  '(or symbol
       function))

(deftype string-designator ()
  '(or character
       symbol
       string))

In practice:

(declaim (ftype (function (trivial-types:pathname-designator) pathname) my-basename))
(defun my-basename (path)
  (uiop:pathname-parent-directory-pathname path))

(compile nil '(lambda () (my-basename 17)))
; in: LAMBDA ()
;     (MY-BASENAME 17)
;
; note: deleting unreachable code
;
; caught WARNING:
;   Constant
;     17 conflicts with its asserted type
;     (OR (VECTOR CHARACTER) (VECTOR NIL) BASE-STRING
;         (AND STREAM (SATISFIES TRIVIAL-TYPES:FILE-ASSOCIATED-STREAM-P)) PATHNAME).
;   See also:
;     The SBCL Manual, Node "Handling of Types"
;
; compilation unit finished
;   caught 1 WARNING condition
;   printed 1 note

Export at definition site

One of the main pain points in Common Lisp development is exporting public symbols.

Traditionally, symbols were exported from a dedicated “package” file that’s loaded first. This is cumbersome and error-prone since the symbols and their exports may run out of sync.

There is an export function which can be called at definition site. Sadly, since it’s a function, it won’t be called at compilation time and thus will yield an error if you attempt to access the symbol from a foreign package at this point.

The trick is to force the evaluation of export at compilation time with:

(eval-when (:compile-toplevel :load-toplevel :execute)
  (export FOO))

Since this is rather lengthy, I recommend you use Serapeum’s export-* macros: https://github.com/ruricolist/serapeum/blob/master/REFERENCE.md#export-always-symbols-optional-package-nil-package-supplied.

Hopefully the syntax will get even more practical once issue 38 is resolved.

Documentation

Contextual documentation is part of the Common Lisp standard, the so-called “docstrings” which can be placed at the beginning of variable and function definitions.

A detail that is perhaps not so well known is that classes, slots and packages also support docstrings via the :documentation keyword. It’s underused in the community and I believe we should encourage its use.

Another perk of Common Lisp is the ability to declare the types of functions, variables, class slots, etc.

So instead of adding a comment about the accepted types of a function arguments and the type of the return value, simply declaim it:

(declaim (ftype (function (number) string) number->string))
(defun number->string (n)
  ...)

This has the double benefit of documenting the argument types and the return types of your function, as well as enforcing compile-time type-checking.

Same goes with class slots:

(defclass my-compiler ()
  ((name :accessor name
	 :initarg :name
	 :initform ""
	 :type string
	 :documentation "Name of your favourite Common Lisp compiler.")
   ...))

Compilers

Common Lisp is a standard and Common Lisp compilers are legions.

Some target specific use cases.

  • ECL targets embedded development, with good interop with C.
  • ABCL is build on top of the Java virtual machine and thus has good interop with Java.

For general purpose development, the go-to compiler these days is SBCL. It is actively developed and has many benefits:

  • Probably the best gradual type checking of all Common Lisp compilers.
  • Fast compilation.
  • Fast executables.
  • Statistical profiler.
  • Coverage analysis.

Debugging and stepper

Many Common Lisp compilers offer great support for debbuging. In particular, the step macro is part of the standard.

With SBCL, you must first compile the code you want to debug with

(declaim (optimize (speed 0) (space 0) (debug 3)))

or evaluate the target expressions with C-u C-c C-c.

Then you can run

(step (foo bar))

to step through the execution of (foo bar). With SLIME / SLY, the Emacs point will follow the currently-executed expression!

Compiler warnings

To increase the level of warning detection, call

(declaim (optimize (speed 0) (space 0) (debug 3)))

before loading a system.

Also reload-ing a system might raise more warnings. You can use ,reload-system in SLIME / SLY.

More debugging tips

See this great tutorial in four parts which covers breakpoints, stepping, inspection, class redefinitions and restarts: https://malisper.me/debugging-lisp-part-1-recompilation/.

(Thanks to Jach for sharing.)

Deployment and executable size

The myth: Common Lisp scripts and binaries are hard to distribute; binaries are too big.

The truth: Common Lisp compilers offer a lot of flexibility.

  • If portability across compilers is not an issue, it’s trivial to distribute scripts, e.g. by using the #!sbcl --script shebang.
  • It’s even possible to distribute the compiled files (e.g. the .fasls) since they are executable. The exact same compiler must be present on the target machine however.
  • Binaries produced by some compilers like SBCL or CCL are rather large for a reason: they include a self-standing complete Lisp image, that is to say more or less the whole compiler plus the required libraries. The benefit: the binary does not depend on the compiler and thus the binary is extremely portable; to top it all, the user can connect a REPL to the stand-alone binary and hack the application while it’s running!
  • Some compilers like ECL or CLISP can produce dynamic binaries that link against a runtime library (just like C). Those binaries tend to be very small.
  • SBCL can produce compressed executables with no noticeable impact on startup time. With compression, a hello-world binary with all of SBCL fits in some 15¬†MiB.

Tip: Add this to your .asd to automatically create compressed binaries with SBCL:

#+sb-core-compression
(defmethod asdf:perform ((o asdf:image-op) (c asdf:system))
  (uiop:dump-image (asdf:output-file o c) :executable t :compression t))

Guix vs. Roswell

Roswell is a Common Lisp environment manager that can install various Common Lisp compilers, deploy applications, etc.

While less popular among Common Lisp developers, I believe that Guix advantageously supersedes Roswell:

  • Guix does not distribute opaque binaries, it allows us to produce reproducible packages (which are also increasingly more bootstrappable). Thus Guix is much more trustable.
  • Guix environments can mix in any other packages, such as libraries or other programming languages. It is thus possible to create environments for complex projects dealing with multiple programming languages at the same time.
  • Guix can run environments in containers.
  • Guix can automatically “pack” a Common Lisp application with all its dependencies (including non-Lisp ones) as a tarball or a Docker image.

Take-away

I believe that Common Lisp deserves its reputation of a robust, practical, general purpose programming language. It’s high time we moved on from the old myths and we fixed the few pain points that still remain.

The setup and the learning curve is still a bit steep today, but it does not have to be this way: with the right web sites, documentation and tools, I’m convinced we could boost Common Lisp accessibility and reach out to many more developers out there.

Fashions are changing and many third-party libraries allows the language to keep up with the latest trendy paradigms.

Finally, Common Lisp is a gigantic ecosystem that we never stop exploring. I believe it’s important for the community to be dynamic and communicative about the evolution of the language. So don’t hesitate to reach out to me, share you critics and tell me what you think about this article!

Comments

Date: 2020-02-25 (Last update: 2020-07-12)

Made with Emacs 26.1 (Org mode 9.1.9)

Creative Commons License