awk-doxygen, Doxygen documentation for AWK

A few days ago, I wrote about bash-doxygen, a Doxygen filter I built for documenting Bash functions and variables. There is a mildly amusing detail buried inside that project: the core of the filter is itself an AWK script.

That is hardly unusual.

I have encountered AWK repeatedly over the years in places where Bash alone stopped being the right tool for a particular part of the job. A shell script might orchestrate commands, files, and processes, then reach a point where it needs to parse records, transform structured text, maintain state, or perform more substantial pattern matching. AWK fits that space well.

It is also one of those tools that seems to exist almost everywhere while receiving surprisingly little attention.

That combination creates an interesting maintenance problem.

AWK programs are often compact, capable, and stable enough that somebody writes one, gets it working, and then leaves it alone for a long time. Months or years later, another person opens the file and encounters an execution model that differs from the surrounding shell code, along with variables that spring into existence through use, pattern/action rules that behave unlike ordinary functions, and function parameters that may conventionally serve as local variables.

That is exactly the sort of code for which documentation earns its keep.

So I built awk-doxygen.

Like bash-doxygen, awk-doxygen is a documentation-led Doxygen filter. It takes explicitly documented AWK constructs and translates them into a small Doxygen-friendly pseudo-C++ representation. It does not attempt to turn AWK into C++, and it does not claim to be a complete AWK parser.

The maintained source remains AWK. The generated representation exists only so Doxygen has something it already knows how to index.

AWK needs AWK-shaped documentation

It would have been possible to take the bash-doxygen model, change a few regular expressions, and declare victory.

I do not believe that would have been useful.

AWK and Bash have different execution models. Bash functions communicate success and failure largely through exit statuses. AWK functions have actual return values. Bash has recognizable variable declarations such as local, readonly, and declare. AWK variables generally come into existence when they are used.

Some of AWK’s most important behavior may not live in functions at all. It can live in BEGIN, END, or ordinary pattern/action rules executed as records flow through the program.

Even AWK’s conventional approach to local variables deserves explanation.

Consider this function:

function normalize(value,    result) {
    result = value
    return result
}

Syntactically, both value and result are parameters. By convention, callers supply value and omit result, allowing result to behave as function-local storage.

The spacing makes that convention readable to a human, though the spaces themselves have no semantic meaning to AWK.

awk-doxygen therefore distinguishes those two roles explicitly:

## @fn normalize(value)
## @brief Normalizes a supplied value.
## @details
## Converts the value into the canonical representation expected by callers.
##
## @param value Value to normalize.
## @local result Scratch value used while normalizing.
##
## @par STDIN
## Nothing is read directly from STDIN.
## @par STDOUT
## Nothing is written to STDOUT.
## @par STDERR
## Nothing is written to STDERR.
##
## @returns The normalized value.
function normalize(value,    result) {
    result = value
    return result
}

@param describes the caller-visible interface. @local describes a formal parameter that normal callers intentionally omit.

That distinction becomes important when somebody has to modify the function years later without knowing which arguments form part of its actual contract.

Global state is different, too

AWK does not have a general source-level declaration for global variables.

That means there may be no declaration for a documentation tool to discover and attach commentary to.

awk-doxygen treats an explicit @var block as the documentation declaration:

## @var record_count
## @brief Number of accepted input records.
## @details
## Initialized during BEGIN processing and incremented after validation succeeds.

There does not need to be an assignment immediately afterward.

That is intentional. The documentation states that record_count is significant global state. The filter does not guess whether it is numeric or string-like, scalar or array, mutable or effectively constant, or where its first assignment occurs.

Those semantics belong in the maintained documentation.

For Doxygen, awk-doxygen synthesizes a generic AwkValue declaration. That type is an indexing convenience. AWK did not suddenly acquire a static type system while nobody was looking.

Then there are AWK rules

Functions and variables still do not describe an entire AWK program.

Consider:

BEGIN {
    FS = ":"
}

There is meaningful behavior here, even though there is no function name to which documentation can naturally attach.

awk-doxygen adds @rule for that purpose:

## @rule initialize
## @brief Initializes parsing state.
## @par Trigger
## Runs once during BEGIN processing before the first input record is read.
BEGIN {
    FS = ":"
}

Ordinary pattern/action rules can be documented the same way:

## @rule accepted_record
## @brief Processes records that passed the initial validity check.
## @par Trigger
## Runs for records whose first field contains a supported record type.
$1 ~ /^(user|group)$/ {
    ++record_count
    print $0
}

For Doxygen indexing, the filter creates synthetic file-local functions for these rules. A BEGIN rule named initialize, for example, becomes something similar to:

static void awk_doxygen_begin_initialize();

That generated function does not exist in the AWK program. It gives Doxygen a stable entity to index while allowing the maintained source to describe the thing that actually exists: an AWK rule.

This was one of the places where copying bash-doxygen mechanically would have produced the wrong abstraction. AWK deserves documentation that reflects AWK.

Documentation remains intentional

awk-doxygen follows the same principle as bash-doxygen in another important respect: undocumented source remains undocumented.

The filter is not intended to inspect an AWK file and manufacture an API from everything that looks interesting. Documentation is an explicit maintenance decision.

That matters particularly for AWK because a program may contain tiny helper functions, incidental global values, and transient rules whose existence does not make them useful public documentation subjects.

The author decides which parts of the program carry a maintained documentation contract.

The filter’s job is to preserve and validate that decision.

Documentation drift can fail the build

Once documentation starts describing interfaces, state, and behavior, stale documentation can be worse than missing documentation. It can confidently tell the next maintainer the wrong thing.

awk-doxygen therefore validates the structural claims that it can verify.

For functions, it can detect conditions such as an @fn name that disagrees with the following function, a documented parameter that does not exist, an undocumented formal, duplicate parameter documentation, parameters documented in the wrong order, or a caller-visible @param appearing after a conventional @local.

It also validates @var and @rule identities and checks that documented rules are associated with rule forms the filter actually recognizes.

Run normally, those conditions produce diagnostics.

Run with --strict, they also produce a non-zero exit status:

awk -f ./doxygen-awk.awk -- --strict ./program.awk > ./program.dox.cpp

That makes documentation drift enforceable in CI.

A function signature can change without leaving behind documentation that still describes the old interface. A rule can move or change shape without quietly retaining documentation attached to something the filter can no longer identify.

The tool cannot prove that prose is correct. It can verify some of the structural promises surrounding that prose, which is a useful boundary.

Using it with Doxygen

The filter can be run directly:

awk -f ./doxygen-awk.awk ./program.awk > ./program.dox.cpp

The resulting .cpp file is an intermediate representation for Doxygen. It is not intended to compile or execute.

A Doxyfile can invoke the filter automatically:

PROJECT_NAME = "AWK Project"
INPUT = .
FILE_PATTERNS = *.awk
RECURSIVE = YES
FILTER_PATTERNS = *.awk=./doxygen-awk.awk
EXTENSION_MAPPING = awk=C++
EXTRACT_ALL = NO
EXTRACT_STATIC = YES

EXTRACT_STATIC = YES matters because documented AWK rules are represented as synthetic file-local functions. That lets two different AWK files use the same natural rule identity without Doxygen treating them as one global pseudo-function.

Like bash-doxygen, awk-doxygen also has a --compact mode. The default output preserves source-line correspondence wherever practical so Doxygen diagnostics and generated declarations remain close to the original AWK locations. --compact removes those placeholder blank lines when that correspondence is unnecessary.

A conservative parser is still the point

AWK is a small language. That does not make implementing a complete, portable-AWK parser a sensible requirement for a documentation filter.

awk-doxygen deliberately recognizes the structures it needs.

Named functions are supported when their complete formal list appears on one physical line. The opening brace may appear on the same line or on a later line after blank or comment-only lines.

Documented BEGIN and END rules are supported, along with ordinary pattern/action rules and action-only rules whose opening action brace gives the filter a reliable structural anchor.

Pattern-only rules remain valid AWK and may deserve prose documentation, though the current filter does not manufacture Doxygen entities for them. Doing so reliably would require a broader association rule than I currently believe the project needs.

That boundary is deliberate.

The goal is to connect intentional documentation with a conservative, well-defined subset of AWK source and to reject claims the filter cannot support. Comprehensive parsing of every program an AWK implementation might accept is outside that boundary.

The regression suite reinforces that boundary across both mawk and GNU awk.

The code that people forget about

This is ultimately why I wanted awk-doxygen to exist.

AWK does not receive the attention of newer languages, and many developers may go years without writing much of it. Yet it continues to sit quietly inside build systems, administration tools, data-processing pipelines, shell utilities, and mature projects because it remains well suited to the work for which it was designed.

bash-doxygen itself is an example.

Its core transformation problem was a natural fit for AWK: read text, recognize patterns, maintain modest state, and emit transformed text. Once that code worked, there was little reason to replace it merely because something newer exists.

There is good reason to make its intent maintainable.

That is the category of code I worry about most: code important enough to keep running and stable enough to go long periods without anyone needing to understand it. Eventually, somebody will.

Documentation gives that future maintainer something better than archaeology.

awk-doxygen is available on GitHub, including the filter, AWK documentation standard, regression tests, Architecture Decision Records, release tooling, and generated Doxygen reference documentation.

Next, I’ll return to the surrounding Bash toolchain and look at bashdeps, the dependency manager I use to pin, verify, and materialize exact external artifacts before a build consumes them.