Bystroushaak's blog / English section / Programming / Moldable tools; a book and a movement

Moldable tools; a book and a movement

๐Ÿ’ก
There is also Czech version; Moldable tools; kniha a hnutรญ

The dissertation in the form of a book called Moldable Tools concretizes and gives voice to the ideas of developers, who are historically and typically focused around Smalltalk. In retrospect, and with a little exaggeration, the people around these ideas can be called the Moldable movement. Let's see what those ideas are.

The book Moldable Tools is aimed at programmers, a reader is understood to be a programmer who is using a debugger, an object inspector or a system search. These tools are typically written in such a way, that it's not possible to change and extend their functionality.

A motive, that is constantly repeated in the book, is an understanding of development tools as lenses through which developers perceive and reason about their software systems. This analogy is followed by contemplation on the topic of how to ensure the move towards maximal tool abstraction, in the direction from low-level technical realizations of a computer towards domain abstractions of the modeled system.

Generic development tools do not allow developers to directly reason in terms of domain abstractions. This is usually enabled by various tools tailored to particular application domains. These tools, however, are not general, and usually, are hard to extend.

Moldable movement is trying to intertwine these two approaches and offer some kind of framework and a way of thinking. This should make it easier to create software with the effectivity of classical development tools and allow easy extensibility, including a transition to a higher level of abstraction.

Terminological detour: domain abstractions

What is actually meant by the term "domain abstraction"? Programming can be perceived as modeling. Creation of a program that solves a problem can be looked at as the creation of a model of domain abstraction. Some kind of simulation which, if accurate enough, will in its essence solve the given problem.

You can imagine it as, for example, modeling in a physically accurate 3D program. It can be used to model a certain situation, for example, a junction, and then see how well traffic will flow through it. Or a wind turbine where you'll be using simulation to assess its efficiency and see how it will behave in a storm.

(Image comes from the video https://vimeo.com/174539015)

Programming can be viewed from a certain point of view as similar to modeling of the real world. For example, we often experience this when working with databases, where the tables literally model a customer, administrators, various customer records, and also define who has access to what. In essence, many tasks can be understood as modeling of some desired state, and the program as a (physical, mathematical, logical, ..) engine that define and solve correct and incorrect changes in the state of this model.

Object-oriented programming is directly inspired by this approach, hence it uses "objects" for modeling. However, this does not mean that in other paradigms there is no modeling of domain abstraction, only that it occurs at a different language level. Just as you can model abstraction using objects, you can use nested functions, recursive logical inference engines, pattern matching, or pure mathematics.

Demonstration of domain abstraction tools

Domain abstraction in general programming languages is practically always expressed by more than one construct of language. For example, an object requires a fairly extensive description and definition of methods before it accurately models, for example, a relationship between a customer, an e-shop, purchased goods, e-shop workers, and shipments.

From a programmer's point of view, it almost always makes sense to create a tool for working with domain abstraction. In almost every e-shop, you don't work with the domain abstraction model using REPL shell of a programming language, but through a graphical interface. For example, it is possible to cancel an order by clicking on a button or to monitor its status.

I found one of the more successful examples in my previous work. I worked on a system that can analyze images from security cameras in real-time, identify individual objects in them and perform high-level operations over them. Among other things, there was a subsystem for monitoring of underground tunnels, where people frequently pass trough and tell the operator how many people went inside and out. The system had to be able to distinguish all kind of things that were moving in the tunnels, whether they were trains, or various animals, like rats, cats, or crows.

To train the system the senior developer created a domain abstraction tool for cases when it was necessary to retrain the system because it evaluated something in the wrong way. It was a program that allowed you to easily and intuitively browse through video frames. Simply by using a mouse cursor, you could mark with different colors various object in these frames, such as people or aforementioned crows or trains. The system then used this data to train the classifier.

There were more programs like this, but this one impressed me especially as it allowed the transition from a low-level domain model description to a high-level domain abstraction. The standard low-level model we worked with as developers consisted of snapshots, bitmaps, color blobs, and lots of code and raw vector data of trained classifiers. One had to understand exactly how the system works at the low level to be able to understand the data with which the system works. On the other hand, the program used for classification was so simple and intuitive, that a small child would've been able to use it.

Problem of a wrong level of abstraction

Debugging can be a laborious activity requiring much manual and repetitive work. On the one hand, debuggers support language-level operations, while developers think in terms of domain abstractions. As a consequence, developers need to mentally construct high-level abstractions on top of language constructs, which can be time-consuming. On the other hand, debuggers rarely provide support for identifying and navigating through those high-level abstractions. This leads to repetitive tasks that increase debugging time.

Debugger is a typical example of a program that only works at the level of the debugged language. This in many cases can be a wrong level of abstraction.

If you have code of an e-shop and a customer causes an error there, it is often a problem just to find out which customer it was, and then you dive into low-level representation, where all of a sudden you are faced with customer_id and object representations and ORM database mapping, or plain SQL queries.

๐Ÿ“‚tinySelf, my programming language interpreter, is written in a restrictive compiled version of python. Theoretically, I have the option to use the python tools to debug, namely the pdb debugger, or the PyCharm debugger. When I find an error in my language, I usually run into a problem which is caused by the fact that the debugger I used works at too low a level of the modeled problem. I model a lexer, parser, compiler, and a bytecode interpreter. Debugger however only offers me debugging of the programming language "matter" from which it is modeled.

For example, once I had a problem finding correct parent slots, which replace inheritance in Self-like languages. If you have experience with class-based-programming-languages, imagine that I had an object whose class inherited from another class with some attribute. An error occurred while trying to access this attribute.

When I tried to debug this error, I was constantly coming across the fact that when I use python tools, I'm actually debugging at too low a level of a model. For one bytecode instruction to execute in my interpreter, it was necessary to go through hundreds of lines of python code. Typically, it was necessary to go through dozens of instructions before I understood the nature of the problem. By the time I reached the correct point, I spent half an hour by stepping through the code of the interpreter.

The feeling that I was getting from it, was as if I had been trying to fix grammatical errors by looking at the letters on a paper under a microscope. At the moment when I try to fix the errors in the text, I am not interested in their representation and exact distribution of particles of ink on the paper. Although it is entirely possible to use this level of abstraction, it is too low for everyday life. Likewise, I needed to move to my tinySelf objects, instead of their representations. Only then did I find the problem.

Moldable movement and raising the domain abstraction

In the book, the author explores various possibilities of how extensibility is typically achieved. For example, he focuses on various architectures of applications, and how plugins are created in them. Among other things, he discusses and analyzes the integration of plugins within the standard API, use of connectors or a message bus. He also examines assembly of applications in the Unix style. In various studies with real users, he shows how difficult it is to create a plugin and extend the functionality of a component.

He also focuses on the ways to raise the level of domain abstraction of development tools, where he mentions two approaches:

General programming languages lack language constructs for specific domains. Developers must use other language elements to model domain abstractions. In Python, these elements often include objects, a context manager, or decorators.

In contrast, there are languages โ€‹โ€‹that allow you to define different DSLs and use them dynamically. For example, Helvetia, or REBOL, which allows you to directly access the BNF parser and define new grammars. This makes it possible, for example, to use the English parser for business logic in the middle of a language. Also, the basic installation comes with data types and grammar for URL, email, or various currencies. Instead of having to create an object manually, as would be done in Python (EmailAddress("some@adress.cz")), the language directly understands the email address and directly creates a target native object. Thus, the rising of domain abstraction is achieved by modifying the language to match the domain. For example, it is possible to define native objects and a parser for customers and various transactions between them and the e-shop. Debugger and other tools then do not distinguish between the extended and non-extended parts and support greater abstraction somehow โ€œnativelyโ€.

Moldable tools, on the other hand, focuses on adapting existing tools instead of language. Modifications are not applied to a language, but to a debugger, inspector and other instruments.

The following principles are proposed for this:

Developers can then adapt and extend individual tools to fit the relevant development context by:

In the book, of course, all of this is discussed and explained in great detail, and everything contained in this blog is just a quick overview.

The author created three different tools to verify his thesis:

Moldable tools

These tools allow you to explore and think about runtime objects, their behavior, and abstractions.

Inspector

Inspector allows objects to define how they are displayed by adding special methods to the object. Something like .__ repr __() or .__ str __() (or .toString()), but for the user interface of an inspector, debugger, or spotter.

For example, an object containing an image can display a specific bitmap, a graphical interface element its widget, an SVG map outline of a particular country, and a color object can show square with the color. In addition, each object can define a "display predicate", that specifies when and how to display it, as well as controls. These, for example, allow a color object to change color settings graphically by using a color palette instead of entering hexadecimal values. There doesn't have to be only one "display predicate", depending on the context it is possible to display an SVG image as a bitmap, opened in a vector editor, listed as a point table, or as an XML representation.

(CHIลž, Andrei. Moldable Tools. Lulu.com, 2016, s. 51. ISBN 978-1-326-74717-6.)

This way, the inspector allows displaying objects in their contexts. It also offers the ability to explore connected objects and browse through them, as well as in the history of displayed objects and their contexts, like when a user clicks on hyperlinks on a web page.

Spotter

Spotter offers something like a search engine for individual objects, again based on activation predicates defined directly in the object. For example, you can search for objects by their values โ€‹โ€‹or by tags. In general, it offers objects a simple way to offer their search based on domain abstraction in a few lines, but also tries to offer them to be automatically discovered depending on the context. In the words of the authors:

[.. spotter offers ..] inexpensive creation of search processors, support for multiple data sources and context-aware searches.

Improved search results are achieved by:

(i) group and rank results, (ii) support rapid skimming through result sets, and (iii) enable in-depth exploration of result sets

Debugger

Debuggers are comprehension tools.

โ€” page 114

Probably the biggest amount of work is focused in a debugger, which integrates both previous approaches and tries to move abstraction as high as possible.

Moldable debugger should allow developers to create and automate high-level abstractions. The user can define their own graphical representation, grouping relevant debugging operations for a specific domain and context. The debugger also offers automatic discovery of suitable debuggers and allows them to be switched dynamically.

(CHIลž, Andrei. Moldable Tools. Lulu.com, 2016, s. 126. ISBN 978-1-326-74717-6.)

Here, for example, you can see a debugger for a parser:

(CHIลž, Andrei. Moldable Tools. Lulu.com, 2016, s. 124. ISBN 978-1-326-74717-6)

A typical extension consists of domain-specific debugging operations and domain-specific view, both based on a debugging session.

Developers can create domain-specific extensions by:

  1. Extending the debugging session with additional functionality;
  1. Creating domain-specific debugging predicates and operations;
  1. Specifying a domain-specific debugging view;
  1. Linking debugging operations to graphical widgets.

Highlights

What I really liked is the fact that the book is backed by numbers. The author has tested his theories on several dozens of developers. He checked statistics, was interested in how much his tools were being used, what suits people and what doesn't, what is the average length of written extensions and so on.

(CHIลž, Andrei. Moldable Tools. Lulu.com, 2016, s. 105. ISBN 978-1-326-74717-6.)

The last point deserves to be further elaborated, as one of the project's objectives is to simplify the creation of new extensions as much as possible. This means that the adjustment or creation of new functionality should have taken place quickly, but also should have been relatively short and undemanding. As can be seen from the table above, but also from several other tables in the book, this objective has indeed succeeded. Success can also be seen in the adoption and acceptance of these instruments as an official part of Pharo Smalltalk.

In the book, the author does a truly remarkable amount of meta-thinking, moreover, presented in an easy-to-understand way. He helped me a lot to understand what Alan Kay meant when he talked about using computers for simulations. I used to take it too literally, in the sense of physical simulation somewhere in CAD. In retrospect, it is clear to me that he didn't mean the physical level, but the level of modeling of domain abstraction.

If you're wondering what the development tools will look like in 20 years, and perhaps if you want to participate in this effort now, I strongly recommend reading the book.

Interesting quotes

There are two ways of constructing a software design: One way is to make it so simple that there are obviously no deficiencies, and the other way is to make it so complicated that there are no obvious deficiencies. The first method is far more difficult. It demands the same skill, devotion, insight, and even inspiration as the discovery of the simple physical laws which underlie the complex phenomena of nature.
โ€” C.A.R. Hoare

Software is expressed using programming languages and crafted with the aid of tools. Programming languages are frameworks of communication for transferring abstract models of the real world from the human mind to computers [Ingalls 1981]. They evolved to provide developers with a vocabulary that hides the details of the computer, and strive to enable developers to create and to express software applications in terms of domain abstractions. This occurred since expressing software in terms of domain abstractions rather than generic programming language constructs has a positive impact on program comprehension.

Nevertheless, software has no physical shape. Developers craft software exclusively by interacting with development tools. Development tools provide the means for transforming an abstract model of the real world from a humanโ€™s mind into a program ( i.e. , executable model), within the design space of a programming language. Hence, development tools have a direct impact on the thinking habits of developers, affecting how they perceive, craft and reason about software [Dijkstra 1972]. Even if developers can design a software application in terms of domain abstractions, to take advantage of domain abstractions when creating and evolving that application their development tools need to incorporate those domain abstractions.

โ€” (CHIลž, Andrei. Moldable Tools. Lulu.com, 2016, s. 1. ISBN 978-1-326-74717-6.)


Hence, a generic and disconnected approach of integrating searching into IDEs leads to information foraging loops where significant effort is wasted recovering concepts instead of directly reasoning in terms of those concepts.

โ€” (CHIลž, Andrei. Moldable Tools. Lulu.com, 2016, s. 78. ISBN 978-1-326-74717-6.)


Software is contextual by design. Stakeholders, developers, technology and the randomness of everyday life make each software system unique. Yet, when crafting software our tools act as if all software systems are the same.

This entire work is about showing that this does not have to be the case. Tools are means to an end. We should shape them, they should not shape us.

โ€” (CHIลž, Andrei. Moldable Tools. Lulu.com, 2016, s. 158. ISBN 978-1-326-74717-6.)

Note on the quality of the release from lulu.com

lulu.com is a web-based service that lets you print custom books on demand. It also allows authors to easily start selling books with this on demand printing, where nothing is printed in advance. As a result, there is no risk that you may be left with a hundred pieces of an unmarketable book somewhere in a warehouse, as it happens with classical publishers and novice authors.

Although it is a great service that I often use to print, for example, manuals or various papers longer than fifty pages, the quality of glued books is quite poor, and it's not the first book from them, that has disintegrated in my hands while I was reading it:

Personally, I take books to some degree as consumer goods, but this seems to me to be a drop too much.

Download link

The book is available under the Creative Commons license at the University of Bern:

Become a Patron