Add on-demand full-state checkpointing for OpenShift AI / KubeFlow preemption#686

Open

RobotSail wants to merge 18 commits intomainfrom

claude/on-demand-checkpointing-dCsmo

Commits on Apr 14, 2026

Add on-demand full-state checkpointing for OpenShift AI / KubeFlow preemption

claude
authored and
RobotSail
committed
Address review feedback for on-demand checkpointing
RobotSail
committed
Check for on-demand checkpoint after each minibatch backward
RobotSail
committed
Add diagnostic note when on-demand checkpoint fails to save in time
RobotSail
committed
Fix help text: checkpoint check happens after each minibatch backward
RobotSail
committed
Add checkpoint checks at 5 synchronization points per training step

RobotSail
and
claude
committed
test: add unit tests for on-demand checkpointing
RobotSail
committed
fix: correct help text for on_demand_checkpointing sync point count
RobotSail
committed

Commits on Apr 16, 2026