python-doxygen, when Python docstrings meet Doxygen

After writing Doxygen filters for Bash and AWK, Python looked as though it ought to be the straightforward one. Python already has documentation syntax, and Doxygen already understands Python, so the space between them appeared smaller than the problems I had solved for Bash and AWK. The interesting part turned out to be that both systems already had established ideas about what Python documentation should look like and how it should be interpreted.

With Bash and AWK, I was introducing documentation structures into languages where Doxygen needed substantial help understanding what I wanted documented. Python was different because it already had docstrings, PEP 257 conventions, type annotations, and established Sphinx/reStructuredText fields. I wanted to preserve that model rather than replace it with Doxygen-specific Python merely because Doxygen happened to be one of the publication targets. That distinction became the organizing principle for python-doxygen: the maintained source should remain ordinary Python, while the translation needed by Doxygen should happen at the Doxygen boundary.

A function could therefore remain documented in a form familiar to Python developers and Python-oriented tooling:

def load(path: str) -> str:
    """Load a value from ``path``.

    :param path: Path to load.
    :returns: The loaded value.
    :raises ValueError: The path is invalid.
    """

I did not want developers to maintain a second documentation block beside that docstring or learn a private documentation dialect for this one output format. The source documentation should remain useful to Python linters, IDEs, documentation tools, and anything inspecting __doc__. The problem was therefore less about inventing documentation syntax and more about reconciling the syntax Python already used with the structure Doxygen expected.

Three documentation models in one pipeline

The first thing I had to account for was that the same documentation passes through three slightly different models before the final reference material is produced. Python sees a docstring. Python documentation tooling sees PEP 257 structure and Sphinx/reStructuredText fields such as these:

:param path: Path to load.
:returns: The loaded value.
:raises ValueError: The path is invalid.

Doxygen has its own structured commands for the same general concepts:

@param path Path to load.
@return The loaded value.
@exception ValueError The path is invalid.

Each representation is reasonable within its own tooling, although they are not interchangeable. I could have made the maintained Python use Doxygen syntax, and Doxygen supports Python-specific documentation forms that would have allowed that approach. I rejected it because the maintained documentation belongs to the Python source first. Doxygen is one consumer of that source, so I did not want its authoring conventions to become the language in which the source had to be documented.

python-doxygen therefore sits between the Python-oriented source contract and Doxygen’s structured documentation model:

The filter leaves the Python declarations alone and translates the structured fields inside recognized docstrings. The earlier example becomes approximately:

def load(path: str) -> str:
    """Load a value from ``path``.

    @param path Path to load.
    @return The loaded value.
    @exception ValueError The path is invalid.
    """

At that point, the translation itself looked correct. The input had been recognized, the Sphinx-style fields had become Doxygen commands, and Doxygen could see the resulting text. The complete pipeline still had another problem, and it came from a Doxygen feature that would ordinarily seem helpful in exactly this situation.

Doxygen was preserving the text instead of interpreting it

Doxygen already has special handling for Python docstrings through the PYTHON_DOCSTRING configuration setting. A Python documentation project might reasonably leave that setting at its default:

PYTHON_DOCSTRING = YES

With that setting enabled, Doxygen treats Python docstrings as preformatted text. For ordinary docstrings, that behavior can be useful because Doxygen preserves the body instead of interpreting the contents as Doxygen markup. For python-doxygen, the same behavior created a subtle failure. The filter could successfully translate:

:param path: Path to load.

into:

@param path Path to load.

and Doxygen would receive that translated line, preserve it, and place it into the generated documentation. The presence of @param path in the output looked like evidence that the pipeline was working, although it proved only that the text had survived. Doxygen was not necessarily interpreting @param as a parameter command, which meant I had evidence for preservation when what I needed was evidence for interpretation.

That distinction changed the problem for me. A documentation filter is not successful merely because the expected string appears somewhere downstream. If the purpose of the translation is to produce structured documentation, the final consumer has to assign the translated text the intended structure.

The counterintuitive fix

The configuration change that made the integration work sounds backwards:

PYTHON_DOCSTRING = NO

To make translated Python docstrings useful to Doxygen, I had to disable Doxygen’s special preformatted-docstring treatment. The docstrings themselves do not disappear; Doxygen still parses the Python source and associates the resulting documentation with the corresponding Python entity. What changes is how Doxygen interprets the contents. With the native preformatted handling disabled, the commands emitted by python-doxygen can be parsed as Doxygen commands rather than displayed as literal text.

That difference means:

@param path Path to load.

can become an actual parameter description,

@return The loaded value.

can become structured return documentation, and

@exception ValueError The path is invalid.

can become exception documentation rather than a line containing an at-sign and some familiar-looking words. The PYTHON_DOCSTRING = NO setting is therefore part of python-doxygen’s integration contract, not an incidental configuration preference.

The tests had to cover the complete pipeline

That discovery also changed what I considered adequate testing. Golden-output tests remain useful because they can prove that a given Python documentation field is translated into the expected Doxygen command. Given this input:

:param path: Path to load.

I can verify that python-doxygen emits:

@param path Path to load.

That test tells me whether the translator behaved as intended, although it says nothing about whether Doxygen interprets the result the same way. The PYTHON_DOCSTRING problem demonstrated that the transformation itself could be correct while the complete documentation pipeline remained semantically wrong. I therefore added an integration path that runs the translated Python through Doxygen and examines the generated documentation so the stronger question can be answered: did Doxygen represent the parameter as a parameter, the return text as return documentation, and the exception as exception documentation?

I care about that distinction because a documentation pipeline can produce convincing output while still losing structure along the way. Seeing the right words is weaker evidence than seeing the right meaning reflected in the generated reference material. The integration tests are there to establish the latter.

Translation also requires knowing when not to translate

Once the basic pipeline worked, the next temptation was to recognize more Python syntax. That path has an obvious attraction because every additional form the filter understands appears to make it more capable, although Python has enough syntax that a documentation filter written in portable AWK can wander quickly toward becoming an accidental Python parser. I did not want python-doxygen to reimplement Python’s tokenizer, parser, type checker, or linter; its job is to recognize a governed documentation surface and translate the documentation that belongs to it.

Triple-quoted strings are a good example of why that boundary matters. They are not always docstrings:

def example():
    value = """This is a runtime string."""

Rewriting that string merely because it contains triple quotes would alter ordinary Python source rather than documentation. Python also permits string prefixes, multi-line declarations, decorators, async functions, generators, properties, and enough lexical edge cases to make “find triple quotes with a regular expression” an increasingly fragile design for this filter. The filter therefore recognizes the forms required by the adopted Python documentation standard, including ordinary triple-double-quoted docstrings, raw r"""...""" docstrings, one-line prose docstrings, and conventional multi-line class and function declarations, while leaving source alone when it cannot safely establish that the text belongs to the documentation surface.

I prefer that conservative failure mode. A missed translation remains visible and can be corrected. A mistranslation can quietly produce documentation that says something the source never said, which is a more damaging failure for a tool whose purpose is to preserve technical meaning.

Some fields do not have a one-word answer

Even after Doxygen was interpreting the translated commands correctly, not every Python documentation field had an exact Doxygen equivalent. Parameters, returns, and exceptions have direct enough mappings:

:param path: description

becomes:

@param path description

while:

:returns: description

becomes:

@return description

and:

:raises ValueError: description

becomes:

@exception ValueError description

Generators exposed a different case. Python’s:

:yields: A validated record.

does not mean the same thing as an ordinary function return, so mapping it mechanically to @return would create attractive documentation with the wrong semantics. python-doxygen instead gives yielded values their own Doxygen paragraph titled Yields. The same design problem appears with :type: and :rtype: fields when they are legitimately present on intentionally unannotated interfaces; those become dedicated titled paragraphs rather than pretending Doxygen has exactly the same type-documentation model as the maintained Python. The job is translation rather than vocabulary substitution, and preserving the meaning of an awkward case matters more than forcing every field into a familiar Doxygen tag.

Python remains the authority

The work eventually led me back to the question of authority. The generated Doxygen output should not become a second source of truth, and python-doxygen should not become the place where Python semantics are decided. The authoritative documentation remains in the Python source, while Python-native tools remain responsible for questions such as whether parameter names agree with signatures, whether type annotations are correct, whether a documented exception reflects the actual contract, and whether a return description has drifted from the implementation.

python-doxygen translates what the maintained source says into a representation Doxygen can use. That leaves the responsibilities divided along boundaries I can reason about:

Python source
    owns the maintained documentation contract

Python linters and tests
    validate Python semantics and source conventions

python-doxygen
    translates the governed documentation surface

Doxygen
    builds and indexes the reference documentation

I expected Python syntax to be the difficult part of this project, and some of it did require careful handling: raw docstrings, one-line docstrings, multi-line declarations, runtime strings, and continuation text all needed explicit decisions and regression tests. The more consequential problem lived between the two documentation systems. Doxygen’s native Python support was useful, while its native docstring behavior prevented the translated documentation from being interpreted the way I needed. The eventual architecture lets Python remain Python-native, lets Doxygen consume structured commands, and gives the tests a way to verify the meaning that survives the boundary rather than only the text.

That is what python-doxygen does. It is a portable AWK filter that keeps Python-native documentation in the source, translates the structured portions Doxygen needs, preserves source it cannot safely interpret, and verifies the resulting representation through Doxygen itself. The project includes the filter, documentation standards, focused regression fixtures, Architecture Decision Records, Doxygen integration tests, generated reference documentation, and versioned release artifacts.

Next, I’ll close this Doxygen run by looking at PHP, where PHPDoc-style DocBlocks and Doxygen overlap closely enough that I can adopt a common subset rather than maintaining another translation filter.