feat: support Lua and JavaScript extensions

[gennaroprota]

This mirrors the existing JavaScript helpers for Lua, and adds support for corpus-mutation extensions written either in Lua or Javascript.

---

Adds two related script-extension capabilities and the reflection plumbing they rely on:

1. Lua scripts as Handlebars helpers, mirroring the existing JavaScript helper surface, so generators can call into Lua from templates the same way they already call into JS. Helpers compose across addon directories (layering), so a supplemental addon can override a same-named helper in the base addon.
2. Corpus-mutation extensions in Lua and JavaScript. Scripts under <addons>/extensions/*.{lua,js} run between corpus finalisation and the first generator invocation. Each script may expose a global transform_corpus(corpus) function; the host calls it once with a read view of the corpus, and the script applies mutations through a mrdocs.set(id, field, value) API. The script side is intentionally a narrow, allowlist-gated write surface (name, extraction, isCopyFromInherited, doc, loc, returnType). Reads piggy-back on the same DOM the generators already see, so scripts and templates have one shape to learn.

The follow-up design for richer entry points (multiple capabilities per file, paired helpers, generator registration, enable/disable, auto-generated mrdocs.* reference) lives in issue #1210; this PR ships the simplest rung of that ladder.

To make the script-facing types stay in sync with the C++ model without a hand-maintained binding table, the extension layer is driven by reflection:

• A new MRDOCS_DESCRIBE_KINDS macro registers the closed set of concrete derived classes of a polymorphic base. The macro lives in include/mrdocs/Support/DescribeKinds.hpp. Each per-base list lives privately under src/lib/Metadata/ and is fed by the existing *Nodes.inc X-macro files, so the registered set has one source of truth.
• mrdocs.set builds a Polymorphic<Base> from a DOM object whose kind: field picks a concrete derived class registered with MRDOCS_DESCRIBE_KINDS.

A dom::Object field-lookup bug surfaced during the work (lookup returned the key instead of the value) and is fixed in the same PR.

Changes

• Source: Extension stack split across four small TUs for navigability:
  • src/lib/Extensions/AddonDiscovery.{hpp,cpp} -- collect <root>/extensions/*.{lua,js} across addon roots (sorted by full path).
  • src/lib/Extensions/SetMember.{hpp,cpp} -- the entire write surface: allowlist, assignFromDom overload set, trySetMember, buildPolymorphic, ExtensionState, buildCorpusDom, setMemberImpl. This is the file that a maintainer touching the allowlist or the type machinery edits.
  • src/lib/Extensions/LuaBinding.{hpp,cpp} -- Lua-side glue (state plumbing, mrdocs.set registration, luaValueToDom).
  • src/lib/Extensions/JsBinding.{hpp,cpp} -- JavaScript-side glue (dom::Function-backed mrdocs.set).
  • src/lib/Extensions/RunExtensions.{hpp,cpp} -- orchestrator: collect scripts then dispatch to the right per-script entry point. CorpusImpl.cpp invokes runExtensions() after finalisation.
  • The shared addonRoots() helper used by both the Handlebars layer and the extension layer was promoted to src/lib/Support/AddonRoots.hpp (a previous copy in src/lib/Gen/hbs/AddonPaths.hpp is gone).
  • src/lib/Support/Lua.cpp's dom::Array metatable is now a Lua-convention 1-indexed sequence (arr[1] is first, #arr is the length, ipairs/pairs work).
  • src/lib/Gen/hbs/Builder.cpp updated to load Lua helpers alongside JS ones; new include/mrdocs/Support/DescribeKinds.hpp defines the MRDOCS_DESCRIBE_KINDS macro and its trait/query types; per-base kind lists under src/lib/Metadata/; small additions to include/mrdocs/Support/MergeReflectedType.hpp, Expected.hpp, and TypeTraits.hpp; dom::Object field-lookup fix.
• Tests:
  • New unit test src/test/Extensions/SetMember.cpp covers the mrdocs.set error paths: argument-shape errors, unknown id, off-allowlist field, allowlisted-but-missing-on-this-symbol-kind, type mismatch for string/bool/enum fields, unknown enum name, polymorphic-write errors (not-an-object, missing kind, unknown kind, unknown sub-field), unknown sub-field on a struct field, and the happy path for Optional<T> null-reset.
  • New unit test src/test/Support/DescribeKinds.cpp covers the has_describe_kinds trait and the for_each(describe_kinds<Base>{}, ...) iteration in both the variadic and the BEGIN/END forms of the registration macro.
• Golden tests: New extension test trees extensions/{js-set-name,lua-set-name,lua-set-return-type}/ exercise rename, doc rewrite, and a Polymorphic<Type> write. Three more trees lock in claims the docs make: extensions/lua-extension-ordering/ (alphabetical-by-full-path ordering across two addon roots), extensions/lua-clear-doc/ (mrdocs.set(id, "doc", nil) clears the optional), and extensions/lua-empty-script/ (an extension file with no transform_corpus is silently skipped). New generator/hbs/{lua-helper,lua-helper-layering}/ exercise Lua Handlebars helpers, including helper layering across multiple addon directories.
• Docs: New docs/modules/ROOT/pages/addons.adoc documents addon discovery (lookup paths, addons, addons-supplemental, override vs. layering semantics). docs/modules/ROOT/pages/extensions.adoc documents the corpus-mutation extension model end to end, including a worked example, the writable-fields table, an "Invariants and operations" section spelling out which corpus properties the rest of Mr.Docs depends on and which structural operations are/aren't reachable from scripts, a Stability section (the extensibility contract scripts can rely on), and a Design rationale section comparing the four alternatives considered for the write surface and explaining why (D) was picked. docs/modules/ROOT/pages/generators.adoc gains a [#custom-helpers] section covering JS and Lua helpers and the layering model. docs/mrdocs.yml excludes the reflection helpers from the generated reference.
• Third-party: third-party/patches/lua/CMakeLists.txt updated. The Lua source lives in-tree under third-party/lua/ and is built as part of MrDocs; this file is MrDocs's CMake override for that in-tree build (upstream Lua does not ship a CMakeLists.txt). The diff is to MrDocs's override, not to upstream Lua. The change drops two files from the Lua build: onelua.c (a unity-build entry point that defines its own main, which would conflict with MrDocs's main when statically linked) and ltests.c (Lua's internal test harness, only valid when LUA_USER_H enables it -- not meant to ship in a library build).
• Breaking changes: None on the user-facing CLI. The reflection support headers gain new entries; this is additive.

Testing

• Eight golden-test trees under test-files/golden-tests/extensions/ and test-files/golden-tests/generator/hbs/lua-helper*/ execute real Lua and JS scripts against fixed fixtures and assert the .xml, .html, and .adoc output byte for byte. Any future regression in the helper or extension surface fails these tests.
• Two new unit-test suites (clang.mrdocs.Extensions.SetMember, clang.mrdocs.Support.DescribeKinds) cover, respectively, the mrdocs.set error paths and the MRDOCS_DESCRIBE_KINDS trait and iteration.
• No CI workflow changes needed; the existing golden-test job runs all of test-files/golden-tests/ on every build, so the new trees stay covered automatically.

Documentation

• extensions.adoc documents the corpus-mutation extension model: a worked example (synthesising briefs from a naming convention), file layout, the transform_corpus hook, the corpus argument, the mrdocs.set API and its hand-maintained allowlist (with a note pointing at the future auto-generated table on issue Design the extension entry-point ladder beyond transform_corpus #1210), the lifecycle (where extensions run relative to extraction and rendering), an "Invariants and operations" section, a "Stability" section spelling out the extensibility contract, and a "Design rationale" section comparing four shapes (A-D) considered for the write surface.
• generators.adoc gains the [#custom-helpers] section covering JS and Lua Handlebars helpers, including the underscore-prefix utility convention, the options-object stripping rule, and resolution order.
• addons.adoc is a new shared page covering lookup paths and the override-vs-layering semantics shared by both helpers and extensions.

[github-actions]

✨ Highlights

• 🧪 New golden tests added

🧾 Changes by Scope

Scope	Lines Δ%	Lines Δ	Lines +	Lines -	Files Δ	Files +	Files ~	Files ↔	Files -
🛠️ Source	50%	2616	2518	98	30	20	10	-	-
🥇 Golden Tests	25%	1313	1313	-	69	69	-	-	-
📄 Docs	12%	603	595	8	6	3	3	-	-
🧪 Unit Tests	10%	524	524	-	2	2	-	-	-
🧰 Tooling	2%	123	123	-	1	1	-	-	-
🤝 Third-party	1%	31	18	13	1	-	1	-	-
🏗️ Build	<1%	26	26	-	1	-	1	-	-
Total	100%	5236	5117	119	110	95	15	-	-

Legend: Files + (added), Files ~ (modified), Files ↔ (renamed), Files - (removed)

🔝 Top Files

• src/lib/Extensions/SetMember.cpp (Source): 727 lines Δ (+727 / -0)
• docs/modules/ROOT/pages/extensions.adoc (Docs): 449 lines Δ (+449 / -0)
• src/lib/Support/Lua.cpp (Source): 329 lines Δ (+310 / -19)

Generated by 🚫 dangerJS against b00938b

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 82.12%. Comparing base (74bf1a3) to head (b00938b).

Additional details and impacted files

@@           Coverage Diff            @@
##           develop    #1196   +/-   ##
========================================
  Coverage    82.12%   82.12%           
========================================
  Files           33       33           
  Lines         3149     3149           
  Branches       734      734           
========================================
  Hits          2586     2586           
  Misses         387      387           
  Partials       176      176           

Flag	Coverage Δ
bootstrap	82.12% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[cppalliance-bot]

An automated preview of the documentation is available at https://1196.mrdocs.prtest2.cppalliance.org/index.html

If more commits are pushed to the pull request, the docs will rebuild at the same URL.

2026-05-27 08:44:05 UTC

[alandefreitas]

The PR is great. It's really nice to see we already have examples that transform the corpus via the scripts.

I don't see much value in putting Lua scripts as Handlebars helpers (a cheap improvement win in an extension type that already exists) with Corpus-mutation extensions in Lua and JavaScript (a completely new feature that's huge).

a new Describe.hpp

Doesn't it already exist? And wasn't the distinction between Describe.hpp and other headers related to reflection that Describe.hpp would only be about porting Boost.Describe?

MRDOCS_DESCRIBE_HIERARCHY

I believe this needs better in-source documentation. We even support that now (or are about to).

I also don't understand these *Hierarchy.hpp files that much. The PR describes that they exist, but not why they exist. All I know is they're using this pattern of including everything, which defeats the purpose of public headers. I also don't see why the user ever needs that information to be public.

They also seem to defeat the purpose of things in a sense. Because we're trying to avoid manually listing things with reflection, but we are still doing it there. The typical pattern we use for that is a single source of truth as inc files.

I also don't understand how a hierarchy can be described as a list unless there's some special convention about this list.

Tests

There's enough new functionality (even public functionality) here that seems to deserve unit tests.

Golden tests

Do they really cover everything that's possible? What are the typical use cases for this? Do we get errors on invalid transformations? How do we merge symbols in the corpus? How do you add symbols to the corpus?

Third-party: third-party/patches/lua/CMakeLists.txt updated for the vendored Lua build.

What do you mean by vendored Lua build here?

• No CI workflow changes needed — the existing golden-test job runs all of test-files/golden-tests/ on every build, so the new trees stay covered automatically going forward.

Documentation

I think the documentation should make the same distinction you made here. It could maybe be described as corpus-mutation extensions or something. Because we have this concept of extensions. Then, things people understand as extensions are these customizations of the template system and the corpus-mutation extensions. They are very different layers and share a common logic of knowing where to put addons, etc. They also have to know about supplemental addons, which they're probably going to use the most. The documentation concerns the user journey.

Also, any reference should be generated automatically. I don’t understand the API very well. Is mrdocs.set the only function we have?

I also don’t see the pattern we had discussed about taking inspiration from Lua/Darktable. It seems the new extension system uses the same pattern as the helpers extension system, but that pattern is not appropriate there. What if an extension wants to provide different functionalities? At a minimum, it's important to include a comparison and an explanation of why the PR chose a different strategy.

Is that the best Python and Lua API we could have on the Lua/JS side? What are the alternatives? They look very different from what things look like on the C++ side. Are they extendable, or will we need to break the Lua/JS API every time there’s a new feature?

The "Corpus argument" section describes a subset of a symbol's API (I don't really understand why, since the full reference is elsewhere), but the API of the corpus object itself isn't described.

[gennaroprota]

Thanks for the thorough review. Great stuff in there :-). Here's a point by point reply:

• Splitting Lua scripts as Handlebars helpers: yes, I will.
• A new Describe.hpp: there's no new Describe.hpp (AI hallucinated :-)). You are right that the file started as the Boost.Describe port, but the hierarchy machinery looked like a natural extension. I can move that part to a separate file, if you like.
• Documentation of MRDOCS_DESCRIBE_HIERARCHY: yes, will expand it.
• Why the various *Hierarchy.hpp exist: the macro static-asserts is_base_of<Base, D> for each derived class, so they must be complete at the macro's expansion point. We can't put it in the base's header (because derived classes are not visible there), and can't put it in each derived's header (because the macro needs them all at once).
• Why are they public: good point. I'll move them to src/.
• Driving lists from .inc files: yes, I will refactor.
• "Hierarchy" as a flat list: fair, the term "hierarchy" is too broad. The macro actually describes a flat set of leaf derived classes sharing a base. Deeper inheritance isn't modeled (and we don't need it). I can rename it to "MRDOCS_DESCRIBE_DERIVED_LEAVES" or "MRDOCS_DESCRIBE_MOST_DERIVED", if you like.
• Unit tests: agreed. Will add coverage.
• Golden coverage:
  • Errors on invalid transformations: not golden tested (a golden of a failing build is awkward). I'll unit-test them.
  • Merging symbols: intentionally not supported, as structural changes can break corpus invariants.
  • Adding symbols: likewise.
• "Vendored Lua build": I just meant that the Lua source is in-tree under third-party/lua/ and built as a part of MrDocs. third-party/patches/lua/CMakeLists.txt is MrDocs's CMake override for that in-tree build (upstream Lua doesn't ship a CMakeLists) and the PR diff is to that override, not to upstream Lua.
• Documentation: agreed. The current single extensions.adoc conflates two layers. Will restructure into (a) template helpers (both JS and Lua), (b) corpus-mutation extensions (both languages), (c) a shared "Addons" section covering lookup paths, addons vs. addons-supplemental, and how both file types are discovered under the same roots.
• Auto-generated reference: Yes.
• Is mrdocs.set the only function? Yes (but see the next item).
• Darktable comparison: Yes, you are right that I didn't follow through. Just wanted the simplest thing that worked first. But the simplest thing has obvious drawbacks. I'll switch, and add a comparison/rationale to the docs.
• Is this the best API? Honest framing: it's defensible but not necessarily optimal. I considered the following alternatives:
  A. Domain-specific helpers (mrdocs.rename, mrdocs.deprecate, ...). Stable and readable, but each one hand-written and must be kept in sync with C++.
  B. Stable script-side schema independent of C++ names. A separate schema for the script API where field names aren't C++ identifiers. Scripts don't break when C++ fields are renamed. Cost: a schema to maintain alongside the C++ types.
  C. Direct DOM manipulation. Mutable corpus; scripts assign fields directly. Most "natural" shape, but no invariants protection.
  D. Reflection-driven generic setter (this PR): one function, dispatch by field name, allowlist gate, reflective sub-dispatch for nested and polymorphic types. Stays in sync with C++ for free; the contract is "the normalized C++ member name is the script API name."
  I went with (D) because it auto-tracks C++ changes and keeps the surface narrow. The weakness is what you'd expect: renaming an allowlisted member would break scripts (though aliases would mitigate).
• Why it looks different from the C++ side: different consumers and constraints: dynamic-typed and sandboxed vs. strongly typed and refactor-safe. The read shape isn't bespoke: scripts see the same DOM the generators already use for rendering. Only the write surface is new, deliberately narrow, and it goes through one gated function rather than direct field assignment.
• Extensibility and breakage:
  • New described C++ fields -> Automatically appear in the script DOM. No script API change.
  • New writable fields -> Need an allowlist addition.
  • New mrdocs.* functions -> Additive.
    Breakage only on rename or removal of an allowlisted field (aliases mitigate if needed).
• "Corpus argument" section: fair point. The current section doesn't describe corpus itself. I'll rewrite.

Open decisions for you:

• Describe.hpp split.
• MRDOCS_DESCRIBE_DERIVED rename.

[alandefreitas]

Open decisions for you:

Up to you. I'll let you come up with the best design you can and we can iterate on it.

[gennaroprota]

Open decisions for you:

Up to you. I'll let you come up with the best design you can and we can iterate on it.

I moved the new macro out of Describe.hpp (which now only contains the Boost.Describe port) and renamed it to MRDOCS_DESCRIBE_KINDS, as "kind" is the term used throughout the code for the concrete derived classes of a polymorphic base.

[alandefreitas]

Thanks for the iteration. Mostly looking good.

In the branch: the Describe.hpp split, the MRDOCS_DESCRIBE_KINDS rename, the in-source doc, the per-base files under src/, the .inc-driven lists, the addons / extensions / generators page restructure, the Corpus-argument rewrite, and Polymorphic support in mrdocs.set. All good.

Still open from our back-and-forth:

• Lua-helpers vs. mutation split is half-done. We agreed internally on a commit-level split rather than separate PRs.
• Unit tests for the error paths in mrdocs.set (unknown id, off-allowlist field, type mismatch, unknown enum, unknown kind, unknown sub-field). Goldens cover the happy paths only.
• Auto-generated reference for the mrdocs.* API; the current table is hand-maintained.

A few things from the original review that didn't get picked up in the back-and-forth:

• A use-cases section in extensions.adoc (the opening paragraph names three examples but doesn't walk one end-to-end).
• The extensibility contract you described in the comment (new described fields appear automatically, writable fields need allowlist additions, mrdocs.* growth is additive) belongs in the docs; right now it lives only in the GitHub comment.
• "vendored Lua build" now was explained but that means the original description of the motivation for the change in cmakelists.txt is circular.

One thing I'd like us to revisit before we close: "intentionally not supported, structural changes break invariants" on merging and adding symbols is technically true but too compact to act on. I understand there's a trade-off here between not breaking invariants and being able to perform anything useful. But I don't know what exactly is the trade-offs and what operations can help us escape the trade-off by ensuring invariants aren't broken. Because a transform extension that only receives constants as arguments is not useful for anything. To pick the right boundary we'd need to formalize in docs:

• What "corpus invariants" and symbol invariants cover here (which generator, lookup, and finalizer properties depend on what).
• What useful things can we do right now while still not breaking the invariants? Filtering symbols? Including symbols? Appending data? What useful transformations can the transformation extension perform?
• The within-symbol guarantees we already have (kind stays kind, no off-shape writes), since that's the easy yes and it isn't framed that way today.
• How can symbols carry extra data? The custom-data-bag idea we'd considered in slack? Is it a strict subset of add-symbol with no invariant risk and covers a lot of real use cases.
• Per-operation breakdown of add / remove / merge: what breaks, what can be re-validated post-extension, what's outright impossible.
• A pick on the trade-off curve between "no structural changes" and "anything goes, post-hoc validate."
• How can we create new data types? For instance, we used to have the proposal that support for niebloids would come from extensions. How would that be performed?
• How to enable/disable extensions?

It's OK to leave things for the future. It's not OK to not be clear about what we have right now and where we're going with this. And without that written down in the docs, any group discussion in the channel will derail once more people join, since nobody's reasoning from the same data. Maybe some of that could become its own section in #1210, or a sibling design issue.

Thanks for what we got here so far 😁 I know this is a much more complex feature than anything else we've been doing so far.

[alandefreitas]

Here's another review by claude:https://gist.github.com/alandefreitas/837bfc0f10e70fce33feffff5527b35a

[gennaroprota]

I addressed all your and Claude's points. In particular, the new "Invariants and operations" section of extensions.adoc should answer your eight questions above.

[alandefreitas]

Thanks, @gennaroprota! Reviewing right away.

[alandefreitas]

Great! 😊🤩

Small things still worth confirming before merge:

• The JSON-driven partial reflection looks good. Congrats. access is no longer in the allowlist (it was in an earlier round). Is it deliberate? I've seen cases where people might want that.
• The commit squash into two commits is still missing for merging. This one can't be done in the GitHub UI. Feel free to make more independent commits if you think they're separate changes worth listing separately in the history.

That's it on my end.

[gennaroprota]

Thanks, Alan :-).

• The removal of access was deliberate, and not really specific to access. The reflection-driven setter turns a kebab-case enumerator string ("public", ...) into an enum value through MRDOCS_DESCRIBE_ENUM, and none of the Metadata/Specifiers enums (AccessKind, ConstexprKind, StorageClassKind, ...) are described on this branch. That batch is described on feat/schema_generation, which needs every enum described for schema generation. The enums that are described here are the ones this PR's machinery needed - which is exactly why extraction is settable (ExtractionMode is described) while access isn't (AccessKind isn't, yet). If access were allowlisted today, mrdocs.set(id, "access", "public") would pass the allowlist gate and then fail at the generic "cannot yet write" path, which is worse than a clean "not settable" error because it looks supported until you try it. This will be a one-line JSON edit once the Specifiers enums are described.
• I'll rebase locally. I think reducing this to two commits would conflate things. Off-hand, I think we need 5 commits.