Literate Programming, as it was meant by Knuth (cweb)

The package documentation, blog and present abilities of the go ecosystem are already great, but as a former user of knuth’s cweb I experienced the power of the real literate approach, when writing algorithms.

“The main idea is to regard a program as a communication to human beings rather than as a set of instructions to a computer.” (found as a Knuth quote on Literate programming - HaskellWiki)

There are several discussions about this topic that quickly lead into the idea, that having a way to generate web content from comments in the code, would make a program literate. But having some package documentation in the code compares to a literate program, as a pile of notes compares to a leather coated book, containing a collection of elaborated novels.

One of the main powers of literate programming is to focus on the main algorithms, while hiding the more technical sections of the code, from the human reader.

Literate programs split the code into sections that might not have to do anything with the text structure of the code for the go compiler. A literate programming tool basically has to restructure code for the human readable output (as cweave does in cweb) and reassemble the code for the compiler (see ctangle).

Let’s look at an example:

@* Say Hello.

When you learn a programming language it's a good manner to first say hello to
the language, by making the computer say hello to the world. Of course this can
be done in several ways, but the most common go way is to use the `fmt` package.
@<say hello@>=
fmt.Printf("Hello, world!\n")

@* Appendix A:Technical frame of the program.
In the appendix we can assemble the sections.
@go 
package main

import "fmt"

func main() {
    @<say hello@>
}

I used a syntax similar to cweb, to give some hints about how this is done. What both parser basically have to do is to search the code for @ characters. The precompiler for the human readable output will format the sections and pretty print the code, while the precompiler for the go compiler will reassemble the code sections in a way the go compiler wants it, beginning with the first @go section found in the source text.

From my experience, once you learned to write code this way, you don’t want to miss it, because it also restuctures the way you think about the problem you want to solve with the code.

It is a pain that the only language I have the full set of literate tools at hand is plain C, though you can massage the system a bit, to do C++ or Objective-C.

I really would appreciate something similar for Go! If I find some time, next months, I will start such a tool, but I first want to check what the go community thinks about this approach and how we may need to modernize the syntactical structure, to better match the look-and-feel of go code.

Frankly, I would not recommend using a cweb style of programming in Go. Go’s philosophy includes readability. This means that everyone agrees on a common style of coding. Tools like gofmt enforce this, as well as conventions like those that help generating godoc documentation right from your code.

A cweb style of coding will inevitably make your code rather unreadable to other Go programmers, and most probably also unsuitable to gofmt or godoc.

Yes, I think this is a serious difference between real literate programs and any other code in any language, plus go comes with the gofmt tool, that suggests a stable format for all go code.

On the other hand a literate program is not just the code, but a general approach to describe a problem and its solution.

I once implemented a whole community backend system with user properties, messages and friendships in a literate style that became one of my best reference projects when I need to implement in C code. I miss that approach to solve a problem everyday and in every language.

I wish it wasn’t just a few programmers out there, who made similar experiences, but at least I want to be able to write such code. The method itself is quite old and not very common, but I don’t think that it ever reached the popularity it really deserves.

Have you seen Go codewalks?

There are also whole file examples which could potentially be used to achieve a result close to what you are describing.

A couple of possible reasons for the lack of popularity that come to my mind (disclaimer: I never have written LP-style programs):

  • The problem that LP solves is not a real pain point in software development. LP might be helpful for beginners but not so for the professional developer.
  • LP appears to encourage - if not enforce - a linear style of programming along the author’s “train of thought”, similar to writing down a novel. (Disclaimer: I never wrote a novel yet. Apologies to all novelists if writing a novel is all but linear.) This linear, narrative style might not match with, or even contradict, the need for structuring code along function points, API’s, packages, architectural entities, etc. Or worse, this linear “writing along one’s train of thought” style might even lead to poorly structured programs.
  • One central idea of LP seems to be that it should be possible to use (or refer to) code before the place it is defined; something that Pascal and C do not support. Modern languages, however, do allow that, so there is no need for LP with regard to this aspect.
  • Good skills in writing self-documenting code, plus some well-balanced amount of comments, further reduce the need for LP.

Just my twopence though.

I’m sorry, but I don’t think that any of your points but part of the first, does hold for people who do and really love LP. From my point of view, literate programming is able to change the more technical work of programming into a real creative work of writing for programs and computer systems.

I would never tell a beginner to try literate programming. Of course due to the low acceptance rate over all developers, the professional developer will have collaboration problems, when he writes code that is changed and extended by many others.

And yes, the reader should follow the authors “train of thought”, that does not need to be linear. That depends on the style, the author prefers. Actually the author is central for literate programming, as well as it is for writing literature. By the way, an author who is experienced in literate programming, might be reviewed by a “lecturer”, who does not need to have much pre-knowledge about what the author wants to achieve with his code, as the intention should communicate through reading the produced document anyway.

Focussing on the “train of thought” and reducing the need to follow a global technical form does not lead to poorly structured programs, but to a structure that stresses the intention of the author. If that is poor, the abilities of the authors are poor. But the ability of the authors will raise quickly with the experience to communicate literal.

It is not a central idea of LP to refer to code before it is defined. That’s just a marginal outcome of the feature that the flow of the document can be freed of the structure of the programming language. The main language becomes the mother language of the author (or english ;)) and the main structure of the document is ruled by the idea, not by the programming language.

Think of a big switch that handles command line arguments. Some arguments are standard and does not express anything on their own, such as --version, --help, --foreground or --daemon are quite self expressive and may only need little internal description. Other arguments might completely change the behaviour of the program, such as --client and --server or --do-something-strange.

You don’t have much freedom to stress such parts of the code. You can stress the meaning of such flags in the function documentation for such a switch function, but that will not include the details of the implementation for the interesting switches.

That way, even a well documented code will technically force to split usage documentation and implementation details.

With literate programming again, the mix of documentation and code is completely free.

Actually literate programming frees the author to be creative and though there is a long way to go, to free professional programmers from corporate and language structures, I think we are already on that way.

Again: Don’t make the mistake, literate programming would be for beginners. A programming language noob should first learn his/her language syntax and grammar and follow some other author’s structure, before he/she is able to plan the solution of an idea and balance the focus between usage description and intrinsic details, within the story he/she tells.

… I will stop my input to the discussion here. I just mentioned that here to promote the idea of literate programming.

My next post will announce a project to handle literate programming in go and will hopefully start as a literate project. But that might take me some two to four weeks…

Kind Regards.

1 Like

Knuth wrote a paper called Axioms and Hulls where he included a LP version of the Delaunay Triangulation algorithm. It was my first exposure to CWeb. This was back in 1995 or so. I had to go through it line by line and convert the algorithm into FTN77. The experience was not pleasant. On the other hand, it might not have been pleasant had it been in C either. I appreciated the documentation explaining each function, but I kept thinking - I would not want to write my programs in this way, way too complicated, and the syntax… But then, this came straight from his TeX system too, which I also found very dense.

I wish you well in your project to create a GoWeb, but I doubt it will do any better to win over converts as did Donald Knuth. There’s been a lot of success in writing API documentation from the code comments, and in writing code from UML style graphical structures and class flow charts. Having also been involved in a large-ish C++ project, by comparison I think that Go code is already pretty literate when combined with a sparing amount of comments and gofmt.

It’s worth putting Knuth’s CWeb into perspective. He was also the author of TeX way before we had PDF, Word, markup, and IDE, PyDoc, Doxygen, etc. He was the pioneer of many great ideas that had no competition at the time. Even simple MarkDown and Wikidocs allow for better docs. Now, documentation is not fully what CWeb does best. We also have to include dependency management, which I believe is a breath of fresh air in Go. It can only get better.

1 Like

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.