# Wilfred Hughes programming, language design, and human factors This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles. ## Difftastic: A Structural Diff Tool for Programming Languages DevFeed: [Difftastic: A Structural Diff Tool for Programming Languages]() Original publisher: [Read original article]() Author: Wilfred Hughes Published: 2022-09-06T00:00:00Z Content type: tutorial Language: en Sources: [Wilfred Hughes]() Topics: [Parsing](), [Tree-sitter](), [Parser](), [Code](), [Lisp](), [JavaScript]() Tags: [code](), [emacs](), [javascript](), [lisp](), [parsing](), [programming](), [time]() ### AI overview The article explains how difftastic implements structural diffs for programming languages. It describes parsing source code with tree-sitter, converting parse trees into a uniform s-expression representation, and calculating diffs as a shortest-path problem on a directed acyclic graph. ### Source excerpt I've always wanted a structural diff tool, so I built difftastic. This has been the most fascinating, most frustrating, and most challenging program I've ever written. How Hard Could It Be? If you write Lisp code for a while, you start to see code like JSON. Everything is basically a list. json-diff example json-diff already exists, and it's pretty good. I wanted something similar for programming languages. After a huge amount of experimentation, I have something that works. In this post, I'll show you how it works. I won't show the many, many dead ends and failed designs along the way. We can pretend that I got it right first time. Parsing The Code If I want to compare two programs, I first need a parse tree for each program. I need an accurate lexer, a basic parser, and I need to preserve comments. tree-sitter was a great fit here. You define a grammar in JSON or JS, and it generates a C library that anyone can use. It's not 100% accurate (e.g. the C++ parser doesn't have preprocessor data) but it's more than good enough. list: ($) => seq("(", repeat($._sexp), ")"), vector: ($) => seq("[", repeat($._sexp), "]"), Here's an excerpt from my Emacs Lisp grammar. There's a ton of tree-sitter parsers available too. Difftastic now supports 44 different syntaxes, and adding new ones is so straightforward that my manual includes a worked example. Using difftastic with Emacs Lisp After parsing, difftastic converts the tree-sitter parse tree to an s-expression. Everything is a list or an atom. This uniform representation enables the diffing logic to work on any language that I can parse. For example, given a JavaScript program like this: foo(1, 2) tree-sitter parses it to this parse tree: expression_statement call_expression identifier "foo" arguments ( number "1" , number "2" ) difftastic then converts the tree to this s-expression representation: List { open_content: "", children: [ Atom "foo", List { open_content: "(", children: [ Atom "1", Atom ",", Atom "2", ], close_con ## Why Small Programming Languages Can Encourage Implementations Instead of Use DevFeed: [Why Small Programming Languages Can Encourage Implementations Instead of Use]() Original publisher: [Read original article]() Author: Wilfred Hughes Published: 2019-03-24T00:00:00Z Content type: opinion Language: en Sources: [Wilfred Hughes]() Topics: [Programming](), [Esolang](), [Lisp](), [Clojure](), [Racket]() Tags: [clojure](), [languages](), [lisp](), [programming](), [programming-languages]() ### AI overview This commentary examines how small, elegant programming-language specifications can encourage developers to build implementations rather than use the language. It discusses BF, Scheme, Shen, Forth, Clojure, and Racket, and argues that language designers should consider the risk while recognizing that multiple implementations can also indicate language health. ### Source excerpt Some programming languages languish due to obscurity. They lack breathless blog posts exclaiming how much nicer they are to use. Other languages are too ambitious. They aspire to support so many features that the original implementers struggle to get a first version working. For example, the type system in Fortress required constraint solving which took exponential time. Sometimes a usable language struggles simply because it's too much fun to write your own. Developers end up building their own implementation rather than actually using the language. The most obvious implementation-focused language is BF. Despite having many implementations, BF programmers have to encourage the implementers to actually try using the language! Scheme is also susceptible to this. Wikipedia lists 31 different Scheme implementations, not to mention the many toy implementations. Writing a Scheme is a great introduction to interpreters, especially once you get beyond the minimal lisp featureset. I've certainly written more implementation code than Scheme code. The problem seems to be languages with a small, well written specification. Shen is a multiparadigm lisp defined in terms of an elegant base language with only 46 system functions. This has resulted in a remarkable 15 third-party implementations, but only a small number of libraries implemented in the language. This phenomenon is not limited to lisps. Forth is also a language that developers often prefer to implement rather than use. Jones Forth is both a Forth tutorial and a discussion of how to build a Forth compiler. There are even stories of people spending years working on implementations without learning much of the language. Designing a language with a straightforward implementation is not a bad thing. It's just a pitfall that language designers need to be aware of. Some crypto systems have this problem too. It seems that we need languages to be big enough that new users write hello world in the language, not write a tool for ## How High Are Your Tests? DevFeed: [How High Are Your Tests?]() Original publisher: [Read original article]() Author: Wilfred Hughes Published: 2019-03-04T00:00:00Z Content type: article Language: en Sources: [Wilfred Hughes]() Topics: [Testing](), [test]() Tags: [database](), [http](), [testing](), [unit-test](), [unit-tests](), [web-browser]() ### AI overview The article proposes describing tests by a numerical "test height" from 0 to 100 instead of treating unit and integration tests as discrete categories. It contrasts isolated, fast, reliable unit tests with higher-level tests that use databases, full web stacks, HTTP, and browsers, which provide broader coverage but introduce slower execution, maintenance, variability, and less useful failures. ### Source excerpt How big can a unit test be? How small can an integration test be? It's easy to argue about whether a test is a 'true' unit test or not. If we test several classes together, is it still a unit test? If we use a small external API, must we call it an integration test? The problem is that testing is a spectrum, but our terminology only allows discrete levels. Putting A Number On It I propose we treat testing as a numerical scale instead. Let's introduce a concept of test height, from 0 to 100. At 0, we have low-level, isolated unit tests. They run quickly, they run in process, and they're extremely reliable. No setup or teardown is required. At 100, we have high-level tests of full computer systems. We spin up an elaborate infrastructure of databases, external services, and exercise real protocols. Many processes run, runtime can be very variable, and flakiness is a continuing challenge. Let's look at some examples! We'll look at some code for a wiki website, and discuss different tests we could write. 20: An Isolated Test At the lowest level, we have code that only depends on its inputs. def slugify(value): if not value: return value # we don't want /, # or ? in our URL value = value.replace('/', '') .replace('#', '') .replace('?', '') # replace whitespace with underscores return re.sub('[-\\s]+', '_', value) We can easily write a small unit test for a function like this. There are no dependencies on external resources, or even external libraries. def test_slugify(): assert slugify('foo/bar baz?') == 'foobar_baz' This test only requires a single process, and our test assertion is simply looking at runtime values. 40: Running With A Scratch Database Our wiki stores its data in a database. Let's look at some view code that ultimately creates database rows. def create_user(request): if request.POST: form = UserForm(request.POST) if form.is_valid(): form.save() return HttpResponseRedirect(reverse('all_users')) else: form = UserForm() template_vars = {'form': form} return ## Helpful: One Year On DevFeed: [Helpful: One Year On]() Original publisher: [Read original article]() Author: Wilfred Hughes Published: 2018-06-22T00:00:00Z Content type: release Language: en Sources: [Wilfred Hughes]() Topics: [Emacs](), [debugging](), [Tooling](), [tracing](), [Code]() Tags: [debugging](), [debugging-tools](), [emacs](), [features](), [new-features](), [tracing]() ### AI overview The article reviews Helpful one year after its first release, highlighting new Emacs debugging and tracing integration, buffer summaries, alias information, variable modification, expanded docstring handling, automatic links, and Info manual links. It also notes availability on MELPA and MELPA stable. ### Source excerpt It's been a year since the first release of Helpful! It's gained a ton of new features, and I'd love to share the highlights with you. Tool Integration Emacs has some excellent built-in debugging tools that I wanted to expose within Helpful. These tools missed the first release, but I've now had the chance to build them. From a Helpful buffer, you can now toggle edebug on a function. This allows you to easily step through code. You can also toggle tracing. Tracing is an underrated built-in feature of Emacs. You can use it to confirm functions have the inputs and outputs you're expecting. It's really useful when exploring unfamiliar code. In this example, I've enabled tracing on projectile-project-root to see when it's called, and what values it's returning. Summaries After some great user feedback, Helpful buffers now start with a summary of what you're looking at. Users often want a direct link to the source code, so this is included in the summary. Helpful also mentions if a function is interactive or autoloaded, just like describe-function. If a user doesn't know what that means, those words now link to the relevant part of the Emacs manual! Aliases Helpful tries to show all relevant information for the current thing. I've overhauled aliases with this in mind. For example, if you view make-hash-table, you can now see that there is another alias of this function, but it's now deprecated. Modifying Variables It's now possible to set variables directly from Helpful buffers. This was inspired by counsel-set-variable, which has an excellent similar feature. If a variable is a defcustom, then Helpful also includes a link to the relevant part of Customize. Even Better Docstrings Helpful now handles all Emacs docstring syntax. It handles references to keybindings, keymaps, and even supports the obscure features like \ and \='. fortran-mode is a great example of a docstring that uses a lot of Emacs docstring features. Recent versions of Helpful try even harder to ## The Emacs Guru Guide to Key Bindings DevFeed: [The Emacs Guru Guide to Key Bindings]() Original publisher: [Read original article]() Author: Wilfred Hughes Published: 2018-01-06T00:00:00Z Content type: tutorial Language: en Sources: [Wilfred Hughes]() Topics: [Emacs](), [Tutorial]() Tags: [beginners](), [commands](), [emacs](), [guide](), [mnemonic](), [shortcuts]() ### AI overview This tutorial explains the logic behind Emacs key bindings. It covers mnemonic bindings, consistent patterns for movement and text operations, and ways to discover which commands and shortcuts are active. ### Source excerpt Imagine that you hold Control and type your name into Emacs. Can you describe what will happen? - The 'Emacs Guru Test' Emacs shortcuts (known as 'key bindings') can seem ridiculous to beginners. Some Emacs users even argue you should change them as soon as you start using Emacs. They are wrong. In this post, I'll describe the logic behind the Emacs key bindings. Not only will you be closer to passing the guru test, but you might even find you like some of the defaults! There Are How Many? Emacs has a ton of key bindings. ELISP> (length global-map) 143 Emacs is a modal editor, so most key bindings are mode-specific. However, my current Emacs instance has well over a hundred global shortcuts that work everywhere. (Keymaps are nested data structures, so this actually undercounts! For example, C-h C-h and C-h f are not counted separately.) Even that is a drop in the bucket compared with how many commands we could define key bindings for. ELISP> (let ((total 0)) (mapatoms (lambda (sym) (when (commandp sym) (setq total (1+ total))))) total) 8612 How can we possibly organise all these commands? Mnemonic Key Bindings Basic commands are often given key bindings based on their name. You'll encounter all of these important commands in the Emacs tutorial. Command Key Binding eXecute-extended-command M-x Next-line C-n Previous-line C-p Forward-char C-f Backward-car C-b iSearch-forward C-s Mnemonics are a really effective way of memorising things. If you can remember the name of the command, you can probably remember the key binding too. Organised Key Bindings Many Emacs movement commands are laid out in a consistent pattern. For example, movement by certain amount: Command Key Binding forward-char C-f forward-word M-f forward-sexp C-M-f Moving to the end of something: Command Key Binding move-end-of-line C-e forward-sentence M-e end-of-defun C-M-e Transposing, which swaps text either side of the cursor: Command Key Binding transpose-chars C-t transpose-words M-t transpose-sexps ## These Weeks in Remacs III DevFeed: [These Weeks in Remacs III]() Original publisher: [Read original article]() Author: Wilfred Hughes Published: 2017-10-16T00:00:00Z Content type: article Language: en Sources: [Wilfred Hughes]() Topics: [Emacs](), [Rust](), [pull-requests](), [Data structures](), [Randomizer]() Tags: [contributions](), [emacs](), [floating-point](), [pull-requests](), [rust]() ### AI overview This Remacs update reports contributions and feature progress, including Emacs functionality ported to Rust, new Rust APIs for elisp data structures, and evolving conventions for representing elisp types. It also discusses Rust-specific implementation techniques and an unsuccessful attempt to use a Rust hash map inside Remacs. ### Source excerpt Time for another Remacs update: lots of contributions, a wide range of features, and even a logo! Contributing Since the last update, we've seen contributions from lots of new people. We've added @brotzeit and @shanavas786, bringing us to seven wonderful people who can approve your PRs. Speaking of PRs, we've merged an amazing 64 pull requests since the last update! If you're looking for a good feature for your first contribution, @brotzeit has been regularly adding new suggestions under the 'good first issue' label. Features Many Emacs features have now been ported to Rust, with new Rust APIs for accessing elisp datastructures. Here's an overview of the features that have landed. Arithmetic: arithmetic, floating point, random number generation (using a Rust RNG!), and comparisons. Symbols: symbol properties, interning, obarrays unbinding, keywords and indirect symbols. Checksums: MD5sum (using a Rust MD5 crate!). Windows: liveness check, type check, overlays and minibuffer, minibuffer check positions and margins. Processes: accessing, type check, data structures and names. Buffers: for the current thread, accessing, file names, size and modification. Point: bobp, bolp, eolp, markers, point-min, point-max forward-point and goto-char. Hash tables: copying and accessing. Characters: multibyte conversions, character tables, category tables Fonts: type checks. Miscellaneous: prefix arguments and identity. We're also periodically pulling GNU Emacs features into Remacs, so all the features available GNU Emacs trunk are included in Remacs. Idiomatic Rust in Remacs Remacs has gradually developed a set of conventions for elisp data types. For each type Foo, we define a LispObject::as_foo, LispObject::as_foo_or_error and a FooRef when you know your elisp datatype is actually a Foo. For example, here's how overlay-start was implemented in C: DEFUN ("overlay-start", Foverlay_start, Soverlay_start, 1, 1, 0, doc: /* Return the position at which OVERLAY starts. */) (Lisp_Object ov ## Helpful: Adding Contextual Help to Emacs DevFeed: [Helpful: Adding Contextual Help to Emacs]() Original publisher: [Read original article]() Author: Wilfred Hughes Published: 2017-08-30T00:00:00Z Content type: release Language: en Sources: [Wilfred Hughes]() Topics: [Emacs](), [Code](), [debug]() Tags: [closure](), [code](), [debug](), [emacs](), [files](), [function](), [source]() ### AI overview The article announces Helpful v0.1, an Emacs package that adds contextual help for functions and symbols. It reports keybindings across keymaps, discoverable debugging actions, focused and fontified docstrings, symbol references through elisp-refs, and source recovery or formatting for interactively defined, closure, and byte-compiled functions. ### Source excerpt I've just released Helpful, a new way of getting help in Emacs! The *Help* built-in to Emacs is already pretty good. Helpful goes a step further and includes lots of contextual info. Let's take a look. Have you ever wondered which major modes have a keybinding for a function? Helpful reports keybindings in all keymaps! When you're hacking on some new code, you might end up with old function aliases after renaming a function. Helpful provides discoverable debug buttons, so you don't need to remember fmakunbound. Helpful also has strong opinions on viewing docstrings. Summaries are given focus, and text is fontified. We solve the text-quoting-style debate by removing superfluous puncuation entirely. Helpful will even show all the references to the symbol you're looking at, using elisp-refs. This is great for understanding how and where a function is used. Finally, Helpful will rifle through your Emacs instance to find source code to functions: If you've defined a function interactively, Helpful will use edebug properties to find the source code. If Emacs can only find the raw closure, helpful will convert it back to an equivalent defun. If Emacs can only find the byte-compiled files, helpful will just pretty-print that. I've just released v0.1, so there will be bugs. Please give it a try, and let me know what you think, or how we can make it even more, well, helpful! ## Suggest.el: Synthesising Constants DevFeed: [Suggest.el: Synthesising Constants]() Original publisher: [Read original article]() Author: Wilfred Hughes Published: 2017-08-06T00:00:00Z Content type: release Language: en Sources: [Wilfred Hughes]() Topics: [Code](), [bug]() Tags: [bug](), [code]() ### AI overview Suggest.el v0.4 adds constant synthesis, improves suggestion ranking, searches intermediate values more efficiently, and documents related projects. It also expands the operations the tool can discover. ### Source excerpt Suggest.el v0.4 is now out, and it offers some really interesting new ways of making suggestions. Supplying Constants Suppose the user gives us the input '(a b c d) and desired output 'a. We would already suggest car, but that only gets the first element of the list. They may have wanted elt or nth, which get elements at a specific position. We now try adding constants to the user's inputs, specifically nil, t, -1, 0, 1 and 2. This makes suggest.el much more effective. Here's the example we mentioned: ;; Inputs (one per line): '(a b c d) ;; Desired output: 'a ;; Suggestions: (car '(a b c d)) (elt '(a b c d) 0) ; <- new (nth 0 '(a b c d)) ; <- new We can now suggest grouping items in a list pairwise: ;; Inputs (one per line): '(a b c d e f) ;; Desired output: '((a b) (c d) (e f)) ;; Suggestions: (-partition 2 '(a b c d e f)) ; <- new Converting a vector to a list: ;; Inputs (one per line): (vector 1 2 3) ;; Desired output: (list 1 2 3) ;; Suggestions: (string-to-list (vector 1 2 3)) (append (vector 1 2 3) nil) ; <- new (-rotate 0 (vector 1 2 3)) ; <- new (-concat (vector 1 2 3) nil) ; <- new Truncating lists: ;; Inputs (one per line): '(a b c d e) ;; Desired output: '(c d e) ;; Suggestions: (-drop 2 '(a b c d e)) ; <- new (-slice '(a b c d e) 2) ; <- new (cdr (cdr '(a b c d e))) Choosing good values for constants is difficult, but the current set seems to be a good tradeoff between performance, the likelihood of finding a result, and the number of useful results. Ranking Suggestions Now we have more possibilities, ordering our suggestions is more complex. The first prototype didn't always get the ordering correct: ;; Inputs (one per line): 0 ;; Desired output: 1 ;; Suggestions: (+ 0 1) ; <- new (- 1 0) ; <- new (1+ 0) The user is probably looking for the increment function, 1+. (+ 0 1) feels like stating the obvious. Suggest.el prefers function calls that don't require extra arguments, giving us a better order: ;; Inputs (one per line): 0 ;; Desired output: 1 ;; Sugg ## Optimising Dash.el DevFeed: [Optimising Dash.el]() Original publisher: [Read original article]() Author: Wilfred Hughes Published: 2017-07-29T00:00:00Z Content type: tutorial Language: en Sources: [Wilfred Hughes]() Topics: [Emacs](), [Library](), [Compiler](), [Code](), [Open Source]() Tags: [code](), [compiler](), [emacs](), [library](), [open-source](), [optimisation]() ### AI overview This tutorial examines performance optimization techniques in Dash.el and Emacs Lisp. It benchmarks iterative code against mapcar, compares wrapper functions, aliases, and primitives, and explains how Emacs byte compilation can improve or eliminate function calls. The article notes that the latest Dash.el version includes these improvements. ### Source excerpt Dash.el is a lovely library, and one of the most popular on MELPA. If we can squeeze every last drop of performance out of it, everyone benefits. Let's take a look at the black art of making elisp faster. Measure First! Chris Wellons has a great optimisation blog post that discusses the performance overhead of creating lambdas with mapcar. If we look at --map, it does indeed create anonymous functions: (defmacro --map (form list) "Anaphoric form of `-map'." `(mapcar (lambda (it) ,form) ,list)) Creating anonymous functions instantiates a closure, which isn't free. Let's write an iterative equivalent: (defmacro --map-loop (form list) (declare (debug (form form))) (let ((result-sym (make-symbol "result"))) `(let (,result-sym) (dolist (it ,list) (push ,form ,result-sym)) (nreverse ,result-sym)))) List Length mapcar (seconds) dolist (seconds) 1 0.000010 0.000028 1,000 0.0027 0.0079 100,000 0.74 1.24 (Full benchmark code here.) Surprisingly, mapcar is consistently faster in this particular benchmark! Other Emacsers have observed dolist outperforming mapcar for short lists. mapcar is primitive, and primitives tend to be fast. dolist clearly isn't a speedup in all situations. Let's try something else. Matching Primitive Performance Some dash.el functions are equivalent to primitive functions. For example, -first-item is equivalent to car, -drop is equivalent to nthcdr. We could write -first-item like this: (defun -first-item (lst) (car lst)) However, this adds the overhead of an extra function call compared with calling car directly. Instead, dash.el does this: (defalias '-first-item 'car) Let's do a small benchmark, to ensure that defalias giving us the peformance we want: Approach time (seconds) wrapper function 0.1399 alias 0.0055 use car directly 0.0050 (Full benchmark code here.) For shame! Our alias still isn't as fast as using the primitive. Let's compare the disassembly using M-x disassemble. (defalias 'car-alias 'car) (defun use-car-alias (x) (car-alias x)) ;; byte ## Remacs II: New Rust Features, GNU Emacs Compatibility Updates, and Build Improvements DevFeed: [Remacs II: New Rust Features, GNU Emacs Compatibility Updates, and Build Improvements]() Original publisher: [Read original article]() Author: Wilfred Hughes Published: 2017-07-15T00:00:00Z Content type: release Language: en Sources: [Wilfred Hughes]() Topics: [Rust](), [Emacs](), [Lisp](), [Docker Compose](), [GitHub](), [Documentation](), [Linux](), [macOS]() Tags: [docker](), [documentation](), [emacs](), [github](), [linux](), [lisp](), [macos](), [pull-requests](), [rust]() ### AI overview This Remacs project update describes new Elisp primitive functions, Rust support, GNU Emacs compatibility documentation, platform fixes, a restructured codebase, procedural macros, Rust crates, and Docker Compose support for building without a local development toolchain. ### Source excerpt It's been six months since the last Remacs update, and many new features have landed! Community We now have a Gitter chat room! Do drop by if you have any questions or wish to discuss Remacs. There's a low traffic Remacs Subreddit too. We've added @jeandudey and @birkenfeld to the GitHub collaborators, bringing us to five fine people who can approve your pull requests. Elisp Features We're still tracking upstream GNU Emacs master, so new features there are landing in Remacs (1, 2). We've added a lot new elisp primitive functions: Strings: characterp, multibyte conversions (1, 2, 3), and comparisons Vectors: type definitions, functions Buffers: type definitions, functions Symbols: various functions A much requested feature, adding Rust support to find-function, has been added. This was an unusual PR as it includes some elisp changes in Remacs. We now have documentation on our compatibility with GNU Emacs. This covers all known implementation differences, platform support differences, and describes how to detect Remacs in elisp code. Cleanup Platforms: We've dropped MS-DOS support. The Remacs build has been fixed on 32-bit Linux and 32-bit macOS. The codebase has been split out: remacs-lib (Rust equivalents of gnulib) remacs-sys (type definitions of Emacs types and C functions) remacs-macros (procedural macros supporting elisp primitive functions in Rust) src (Rust implementation code of elisp) Signal name mapping is pure Rust code. We now run rustfmt on every PR. If you fancy building Remacs without installing a dev toolchain (compilers, C libraries etc), there's now a docker-compose.yml to make your life easy. Macros It wouldn't be a proper lisp project without some macro magic. After several PRs and discussions, Remacs now includes a procedural macro to simplify defining elisp functions in Rust. For example, here's vectorp: /// Return t if OBJECT is a vector. #[lisp_fn] fn vectorp(object: LispObject) -> LispObject { LispObject::from_bool(object.is_vector()) } Lever