Webbit

There are many technologies prevalent in Web infrastructure today. Some of the most well-known ones include PHP, Flask, Django, Ruby on Rails, Hugo, etc. We can understand that most of these solutions operate within one of three modalities:

1. Templating engines — These rely on some script written in, say, Python ^[1] to act as a HTTP daemon ^[2], processing and routing requests, eventually presenting pages to the user through a "template document," a static HTML page containing blanks that are filled in by the script for each request.
2. Dynamic hypertext preprocessors — There are much fewer of these, but PHP is the obvious example here: One continues to write HTML as normal, but there are some special tokens that allow for the execution of an interpreted language that can generate HTML programmatically as well. The code is re-run every time the page loads, much like with most template framework solutions. This interpreted language provides the facilities you would expect from any other, including the ability to "include" other files, one of PHP's most basic use cases.
3. Static site generators — With the canonical example being Hugo, these are programs that typically take a collection of Markdown files (or something else) representative of, say, the posts on a blog, and produce an entire web tree ^[3] of static files that can be served efficiently. Some of these also use a template engine without the associated framework for handling requests in order to generate pages as well.

Webbit does not really fit in any of these categories. In fact, it fits—or can be extended to fit—into just about all of them, as you will come to see.

How does it work?

The gist is that Webbit applies techniques learned from compiling software to compiling web-sites. This is not exactly a terminological leap—static site generators often refer to "compiling" a web-site as well—but it is a functional leap. Much as NetBSD's pkgsrc relies on a tree of BSD Makefiles to perform the higher-level operations of package management, Webbit uses an extensible collection of GNU Makefiles to transform a source tree of web documents into a compiled tree.

The experienced reader will already be struck by where this is going, but to clarify: Makefiles provide a domain-specific language commonly used in developing C and C++ programs to specify a set of rules for producing output binaries from source code, which is then used to permit incremental builds ^[1] by automatically keeping track of the dependencies between source/binary objects, when the source file corresponding to each output was last modified, and the sequence of rules to apply to get from each source file to its output ^[2]. If you're still green on how exactly that works, don't worry! A gentle introduction for those unfamiliar with such tools can be found here.

So, applied to the Web, this already provides some compelling features. For example, Webbit already contains some demonstrative rules for producing optimised versions of all of the PNG-format images contained within your web tree. It also has rules to produce compressed versions for all compressible output files, which is a significant boon for performance if you use xthttpd's static gzip compression feature (or something similar). It also includes a C utility and associated rules for minifying ^[3] your CSS files. These are all fairly common tasks a webmaster would previously have to either perform manually, or write their own ad-hoc batch processing commands for.

Most significantly, though, are two other capabilities Webbit provides. One such rule module allows you to produce web pages using the Commonmark language ^[4]as well as the m4 macro pre-processor, not unlike how C is produced alongside the C pre-processor. Remember how you used to "include" your navigation bars, page footers, comments sections, etc. with PHP, like they were separate pages you could just tack onto existing ones? Webbit can do that too! In fact, it can do even more: using m4's system command facilities, you can also execute, say, Perl code directly during the generation of your HTML pages, for example. You'll see that later!

Another rule module allows you to produce CGI programs written in C, Perl, TinyScheme, or something else (if you decide to extend it) directly within your web tree. This allows you to mix static and dynamic pages! A collection of C headers for accelerating the development of fast CGI programs (<1msec start-up time) has also been included.

This yields a powerful combination; elaborate trees of static pages and heavily optimised resources can be crafted, and along-side them one can sprinkle in little CGI programs to handle the more dynamic tasks as well. For example, you could write macros in m4 to produce a comment form and to display individual comments, and then use a small C or Perl program to handle the form submission, clean up any dangerous parts of the input ^[5], dump the user's comment as a macro call into a file that is included into the main page, and then execute the make command on the server to re-build all modified pages. Yes, that's right: you can make a working comments section on a static page, and you don't even have to resort to executing client-side JavaScript.

Note that this power must be used responsibly. I have outlined many possibilities here to demonstrate theoretical potential, but you almost certainly will not and should not need the vast majority of it. While Webbit can greatly ease the tedium of many productive endeavours, it also carries the potential to become an almost bottomless timesink. One would do well to heed the warning placed in the introduction to the GNU m4 manual:

Some people find m4 to be fairly addictive. They first use m4 for simple problems, then take bigger and bigger challenges, learning how to write complex sets of m4 macros along the way. Once really addicted, users pursue writing of sophisticated m4 applications even to solve simple problems, devoting more time debugging their m4 scripts than doing real work. Beware that m4 may be dangerous for the health of compulsive programmers.

Why is it needed?

Generally, the underlying hypertext behind a web-site tends to contain much tedious repetition. Writing raw HTML, we can imagine having to type out the necessary links and containing elements for navigation buttons on every single page. Even omitting this, HTML is tedious to work with for most cases; far more keystrokes are required than necessary to express the most common structures we usually want to express, say emphasising some text, or starting a new paragraph. Of course plenty of people have already noticed this, otherwise we wouldn't be in the situation we find ourselves in today; we have amassed many different ways to approach the underlying problem of "How can we produce a program that mechanically derives standard HTML from user-generated information laid out in a language that is more convenient for us to write?"

Webbit is consequent from the understanding that many of the most economical solutions to this problem are also the most inflexible. Not wanting to be stuck within a particular web-site structure (say, a blog), Webbit instead tries to follow the Larry Wall quote, "Easy things should be easy, and hard things should be possible." To that end, you can even use Webbit to continue writing raw HTML if you want to, just benefiting from only the image compression functionality for example. Another way to think about it is that all existing web trees are also Webbit trees ^[1], and Webbit exists to translate the superset of features it provides over regular HTML into the subset supported by conventional web browsers.

It may also be observed that Webbit's implementation is also quite similar to the ikiwiki project. Indeed, Webbit can be used to create a wiki that functions very similarly. However, if a wiki is what you want to make, then you should probably use ikiwiki, as it was designed to do that out of the box. Webbit is a much more general tool, and started primarily out of trying to generalise the construction of personal web-sites, not the wiki format. While it may be interesting to explore implementing something like ikiwiki on top of Webbit, and I don't wish to discourage anyone from doing so, as of right now it may be preferable to use them in a complementary way.

How do I use it?

First, you need a web tree initialised with the Webbit system. Here we consider each level of the web tree as zero or more source files you want to be compiled into the final tree. At the root of the web tree, you need a directory—or a symbolic link to a directory—containing the Webbit system, let's say proc, and then a Makefile adjacent to it, containing nothing more than the following line:

    1 include proc/mk/collect.mk

That's it. Run make or make all, and the compiled tree will be placed in the out directory (by default). You won't get anything yet, because you don't have any actual files to compile. (make clean removes all output files).

Once you are ready, you should have a directory tree that looks like this:

$ ls -l
[...]
-rw-r--r-- 1 amelia amelia  340 Jun  8 16:04 Makefile
drwxr-xr-x 1 amelia amelia  188 Jul 11 00:27 proc
[...]

Before I get to writing any actual documents, it will be helpful to know a few other things: how to extend Webbit, and how to use sub-directories in your web tree.

Sub-directories

Let's say you want to compile some documents in the sub-directory software. Webbit won't know about them by default, but there exists a set of rules for recursing into sub-directories listed in the variable subdir, which you should specify before including Webbit (otherwise your changes will be made after Webbit has already "run"):

    1 subdir += software
    2 
    3 include proc/mk/collect.mk

Of course, the sub-directory needs its own Makefile. proc/mk is added to the GNU Make include path at the point of the root Makefile's execution, so in the sub-directory we can simply create the following Makefile:

    1 include collect.mk

Extending Webbit

For every directory in proc, Webbit goes to great lengths to search the corresponding directory at the root of your web tree as well. For example, it was mentioned earlier that proc/mk is added to GNU Make's include path. Well, so is mk, so you can also create files containing custom rules in there. If you read proc/mk/collect.mk, you will also see that you can insert extra code in-between stages of Webbit's execution, or even disable existing modules, with the utility_block, utility_block_extra, plugin_block_extra, plugin_block, filter_block, and filter_block_extra variables, but this is an advanced use-case.

You can also create m4 libraries like this (in the directory m4), header files for CGI programs written in C (cgi), and so on. Generally, Webbit duplicates (via the dup_dir function in proc/mk/utility.mk) include directories for each program it uses in both the Webbit directory and the web tree. For example, m4_include will search both m4 and proc/m4.

You can take a look at the existing modules in proc/mk to get an idea of how to write your own. Some trivial cases would be png.mk, css.mk, and copy.mk.

The First Page

Webbit can be extended to use any preprocessor to generate static pages, even PHP (and this isn't a bad idea--you could use Webbit to have a mix of static and dynamic PHP pages!) but by default extensive support for using m4 is provided. To learn more about that, you will want to take a look at the GNU m4 manual.

Once you're ready, it's time to create a demonstration page. Typically all m4 pages actually use the .md extension, to represent Markdown/Commonmark, but this is only because Commonmark permits freely mixing with HTML.

So, index.md would look something like this:

m4_include(html.m4)
__auto_page_md

# Hello world!

__inline_perl

print "Hello world from perl!\n";

^D

(Where ^D is the ASCII EOF character, used to signal that the perl process should stop reading, which you can type in GNU Emacs with C-q C-d — Consult the manual for your text editor of choice to see how to type such control sequences.)

__auto_page_md is defined in html.m4, and by default wraps all the following content in a typical HTML document. You can modify the functionality in a few ways:

• Define __author to change the content of the meta Author header.
• Define __title to change the page title.
• Define __my_header to add more meta tags to the <head></head> section.
• Define __my_body_header to add additional text at the start of the/each page.
• Define __my_footer to add text at the end.

You can of course discern this from reading html.m4. A more important thing to know is that __auto_page_md calls m4_disengage_all, defined in standard.m4. This changes m4's quote characters [ and ] such that they require preceding backslashes (like an escape sequence), so as to not conflict with the syntax for specifying anchors in Markdown. You can temporarily change this functionality with m4_engage and m4_disengage. Many other useful functions are defined and commented in standard.m4, and it is loaded by default in all m4 documents. You should take a look!

Where you go from here is your choice. Webbit gives you the power to create just about anything you want, but narrows the method of implementation so as to keep things as fast as possible. Using tools like this you can create something that adheres to Landau's idea of the mildly dynamic web-site without the compromise of having to execute PHP for every request. As indicated by the comment section example above, you can come up with interesting ways to achieve PHP-like things without violating this principle.

You will also find that Webbit already includes modules to perform things you probably wanted to do, like highlighting source code. The source code is terse but kept to a minimum, so as to allow you to flick through features that already exist and quickly ascertain how to use them.

What about CGI?

CGI programs are written how they've always been written; the web-server executes some file.cgi, feeds it environment variables representing properties of the user's request, and the program writes a HTTP response back through its standard output channel, which is then passed back to the client. In cgi.mk, some rules are already specified to produce CGI programs:

• hello.pl is translated to hello.cgi, and Webbit locates the Perl interpreter on the running system and inserts an appropriate shebang directly into the file. This eliminates the overhead of searching the environment for the Perl interpreter on each request, which on my system shaves off about a third of the startup time per request of a simple "Hello world!" program.
• hello.ss is translated to hello.cgi, and Webbit does the same as with Perl except for the TinyScheme interpreter, or some other Scheme interpreter of your choice.
• hello.c is compiled to hello.cgi, using linker flags produced by the shell script at hello.ld.

While Webbit normally tries to pick up files automatically, CGI programs must be specified manually by adding them to the cgi_perl, cgi_scheme, and cgi_c variables respectively.

The Git client is an interesting case of a CGI C program. The actual Git client is stored in one big static library, and is linked against your hello.c. You do this by creating a hello.ld containing:

    1 #!/bin/sh
    2 . ${establish_proc}git/git.ld

This just defers to the linker script at proc/git/git.ld, which handles the actual process of linking against the Git client code for you ^[1]. Within hello.c you only need to define the following variables:

    1 const char *extra_head; /* Extra text to add to <head></head> */
    2 const char *my_header;  /* Extra text at the top of each page */
    3 const char *my_footer;  /* Ditto, for the bottom */
    4 const char *repo;       /* The repository name, NULL if any */
    5 const char *repo_dir;   /* The directory to search for repositories */
    6 const char *clone_url;  /* The clone URL to prepend to the current repo */

And that's it--the Git client library even already provides a main function.

How do I get it?

You might find it convenient to keep track of your web tree using a version control system like Git. In that case, you can insert Webbit under a sub-directory of the web tree as a Git submodule.

Otherwise, you can just download the latest version using the links at the top of this page and extract it manually. It's simple!

Just make sure your system has the following dependencies:

• GNU Make, for Webbit itself.
• GNU m4, if you want to preprocess pages with it.
• Perl, if you want to execute the included Perl utilities.
• The GNU Compiler Collection, namely the GNU C Compiler, in order to compile the Git client, CSS minifier, and CGI programs. Other compilers should work too, and if you can't do that, then you can specify the disable_git=1 variable to make.
• zopfli and brotli, for static page compression.
• libgit2 for the Git client.
• xxHash for the CGI cache and the Git client (which uses it).
• Andre Simon's highlight, for syntax highlighting (if you use it).

If you run Microsoft Windows, you can create a UNIX-like environment suitable for running Webbit (and actually install the dependencies above) by using the popular Cygwin system.