Skip to content

schema: add notification schemas and generated bindings#8938

Open
nepet wants to merge 4 commits intoElementsProject:masterfrom
nepet:schema/add-notifications
Open

schema: add notification schemas and generated bindings#8938
nepet wants to merge 4 commits intoElementsProject:masterfrom
nepet:schema/add-notifications

Conversation

@nepet
Copy link
Copy Markdown
Member

@nepet nepet commented Mar 12, 2026

Important

26.04 FREEZE March 11th: Non-bugfix PRs not ready by this date will wait for 26.06.

RC1 is scheduled on March 23rd

The final release is scheduled for April 15th.

Checklist

Before submitting the PR, ensure the following tasks are completed. If an item is not applicable to your PR, please mark it as checked:

  • The changelog has been updated in the relevant commit(s) according to the guidelines.
  • Tests have been added or modified to reflect the changes.
  • Documentation has been reviewed and updated as needed.
  • Related issues have been listed and linked, including any that this PR closes.
  • Important All PRs must consider how to reverse any persistent changes for tools/lightning-downgrade

Add JSON schemas for 18 notification topics (native and xpay), generate Rust and gRPC bindings from them via msggen, and include round-trip deserialization tests for the most complex types.

  • balance_snapshot
  • coin_movement
  • channel_state_changed
  • connect / disconnect
  • deprecated_oneshot
  • forward_event
  • invoice_creation / invoice_payment
  • log
  • onionmessage_forward_fail
  • openchannel_peer_sigs
  • plugin_started / plugin_stopped
  • sendpay_success / sendpay_failure
  • shutdown
  • warning
  • pay_part_start / pay_part_end

@nepet nepet requested a review from cdecker as a code owner March 12, 2026 10:40
@madelinevibes madelinevibes added this to the v26.04 milestone Mar 13, 2026
@daywalker90
Copy link
Copy Markdown
Collaborator

I'm somewhat of a notification schemer myself, so i compared this PR with mine and hopefully combined we got it correct (surely !?).

I think we should not bump versions in Cargo.toml's in this PR since it will trigger an automatic release to crates.io and i'd like to get some other changes in first. Unless you like to be last or nobody minds multiple version bumps in a short time period.

Also CI will complain (check the diff in ‎cln-rpc/src/model.rs for example) since you didn't run msggen in the same way. Safest bet is uv make gen. If that doesn't do anything because it didn't notice any changes run these:

PYTHONPATH=contrib/msggen/ uv run contrib/msggen/msggen/__main__.py bundle doc/schemas
PYTHONPATH=contrib/msggen/ uv run contrib/msggen/msggen/__main__.py -s generate

Also i think you should add the notification subscriptions in ‎plugins/grpc-plugin/src/main.rs in this PR aswell.

@daywalker90
Copy link
Copy Markdown
Collaborator

Another thing i just tested to confirm: The documentation and the code of pay_part_start and pay_part_end does not match. In the docs there are origin and payload fields which the code does not create. Instead it's just the content directly from the payload.

@daywalker90
Copy link
Copy Markdown
Collaborator

I also noticed that our ChannelState primitive was missing the CLOSED variant that is used in the channel_state_changed notification and it has to be added to our ChannelState primitive in ‎cln-rpc/src/primitives.rs: https://github.com/ElementsProject/lightning/pull/8874/changes#diff-3028641b95db0d609127ca8852d32491eb363b07d754baf936f9c9568c51b109

(ignore the assert diff)

@cdecker
Copy link
Copy Markdown
Member

cdecker commented Mar 13, 2026

Very nice changes, just some minor issues that @daywalker90 pointed out ^^

@nepet
Copy link
Copy Markdown
Member Author

nepet commented Mar 18, 2026

@daywalker90 I hope I got all your findings fixed 😅! Thanks a lot for reviewing!

@nepet nepet force-pushed the schema/add-notifications branch from 0fc831d to 6dd88f8 Compare March 18, 2026 15:36
@daywalker90
Copy link
Copy Markdown
Collaborator

cln-grpc/proto/primitives.proto's ChannelState also needs Closed = 14; added. I forgot that one myself too.

@nepet nepet force-pushed the schema/add-notifications branch 6 times, most recently from 38e8801 to f81c863 Compare March 22, 2026 10:06
@madelinevibes madelinevibes modified the milestones: v26.04, 26.06 Mar 22, 2026
@madelinevibes madelinevibes added the 26.04 RC This PR needs to go into the next release candidate for 26.04 label Mar 30, 2026
@daywalker90 daywalker90 force-pushed the schema/add-notifications branch from f81c863 to f06384c Compare March 30, 2026 20:29
Add schema definitions for the documented native notifications which were
still only described in prose.

This makes the notification payloads available in doc/schemas so msggen can
consume them in later commits, while keeping this commit schema-only.

The balance_snapshot schema follows the current implementation shape:
the notification wrapper contains a single balance_snapshot object whose
accounts field is an array of account snapshots.

doc: add flat xpay notification schemas

Add schema files for the documented xpay notifications using the temporary
flat naming convention.

These topics are plugin-emitted rather than native lightningd notifications,
so keep them separate from msggen generation for now while still recording
their subscriber-visible payload shape in doc/schemas.
@nepet nepet force-pushed the schema/add-notifications branch from 8dfa430 to fb6b86a Compare March 31, 2026 08:12
@nepet
Copy link
Copy Markdown
Member Author

nepet commented Mar 31, 2026

So do I get it correctly, that primitives for both modules, rpc and grpc, aren't generated or am I just blind ^^? In this case I'd also need to add Closed to the enum ChannelState in cln-rpc/src/primitives.rs

@daywalker90
Copy link
Copy Markdown
Collaborator

So do I get it correctly, that primitives for both modules, rpc and grpc, aren't generated or am I just blind ^^? In this case I'd also need to add Closed to the enum ChannelState in cln-rpc/src/primitives.rs

Correct, and you already did :)

@nepet nepet force-pushed the schema/add-notifications branch from fb6b86a to 4196075 Compare March 31, 2026 08:29
@nepet
Copy link
Copy Markdown
Member Author

nepet commented Mar 31, 2026

Amended "0ddd055f08e7" to also include ChannelsState::Closed in cln-rpc/src/primitives.rs

@nepet
Copy link
Copy Markdown
Member Author

nepet commented Mar 31, 2026

@daywalker90 I missed the TryFrom implementation :D It has a catch all in the match so it's easy to miss it 😅 but I added it now.

@daywalker90
Copy link
Copy Markdown
Collaborator

@daywalker90 I missed the TryFrom implementation :D It has a catch all in the match so it's easy to miss it 😅 but I added it now.

Good catch!

@daywalker90
Copy link
Copy Markdown
Collaborator

We have an actual regression in performance here.
I ran test_gossip.py 2x each locally with -n5:

master pr
~7m ~22m

nepet added 3 commits April 1, 2026 00:37
Expand the native notification list consumed by msggen and regenerate the Rust and gRPC outputs for the documented notification schemas.

This wires the new schema-backed native notifications through cln-rpc, cln-grpc, and the bundled msggen schema metadata. Keep the temporary xpay-prefixed plugin notifications out of generation for now.

coin_movement.extra_tags remains a repeated string in the generated gRPC surface for now because msggen does not currently emit enums nested under repeated array items correctly.

Changelog-Changed: cln-rpc and cln-grpc now expose notification bindings for balance_snapshot, coin_movement, deprecated_oneshot, disconnect, forward_event, invoice_creation, invoice_payment, log, onionmessage_forward_fail, openchannel_peer_sigs, plugin_started, plugin_stopped, sendpay_failure, sendpay_success, shutdown, and warning.

msggen: generate xpay notification bindings

Treat xpay as a built-in plugin for notification generation and include its documented notification schemas in the msggen output.

This extends the generated cln-rpc and cln-grpc notification surfaces with the flat xpay schema files, while exposing the actual notification names `pay_part_start` and `pay_part_end` in generated APIs.

Changelog-Changed: cln-rpc and cln-grpc now expose xpay notification bindings for `pay_part_start` and `pay_part_end`.
Signed-off-by: Peter Neuroth <pet.v.ne@gmail.com>
@daywalker90 daywalker90 force-pushed the schema/add-notifications branch from 4196075 to 8e51c94 Compare March 31, 2026 22:38
@daywalker90
Copy link
Copy Markdown
Collaborator

It was because we subscribed to the "shutdown" notification and didn't handle it properly which makes the plugin hang for 30 seconds for every test teardown.

I removed the subscription to "shutdown" entirely, since i think it would just be for the cln-grpc plugin itself (where we don't need it if we don't do something special on shutdown) and not to be broadcasted via grpc.

Copy link
Copy Markdown
Contributor

@rustyrussell rustyrussell left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There's a bit of back-and-forth which would be helped by combining some commits, and the commit messages are a bit messy from previous combinations, but the result is great.

@daywalker90 daywalker90 modified the milestones: 26.06, v26.04 Apr 1, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

26.04 RC This PR needs to go into the next release candidate for 26.04 PLEASE clear CI 🫠

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants