Skip to content

incremental collaboration sync with targeted bootstrap#1081

Open
hweihwang wants to merge 2 commits intomainfrom
feat/incremetal-sync-improvements
Open

incremental collaboration sync with targeted bootstrap#1081
hweihwang wants to merge 2 commits intomainfrom
feat/incremetal-sync-improvements

Conversation

@hweihwang
Copy link
Copy Markdown
Contributor

@hweihwang hweihwang commented Mar 9, 2026

Main Code Changes

This PR changes the collaboration sync path so we stop treating every edit like a full-scene replacement.

  • src/utils/syncSceneData.ts
    This is the diff/merge layer. It selects only the elements whose versions changed for outbound websocket updates, and it merges incoming remote element updates incrementally into the current scene.
  • src/hooks/useSync.ts
    This is the persistence and broadcast scheduler. It lets the designated syncer own persisted scene sync, and it runs server/websocket sync from the latest settled scene snapshot instead of an earlier intermediate state.
  • src/hooks/useCollaboration.ts
    This is the room bootstrap path. A newly joined client requests a targeted full-scene init (SCENE_INIT_REQUEST -> SCENE_INIT) instead of relying on room-wide full-scene rebroadcasts. Full-scene replacement is still kept for bootstrap, recovery, and explicit replacement paths.

In Plain Terms

  • normal collaboration sends small scene deltas instead of the whole board on every edit
  • a newly joined collaborator still gets the full board, but only that socket gets it
  • one active socket is responsible for persisted scene sync, which reduces duplicate saves and conflicting full-scene pushes

Proof / Safety

  • Incremental selection is covered in tests/integration/syncSceneData.spec.mjs: changed-element-only selection, new-element incremental send, remote version merge without downgrade, remote-only sync-marker advance, and "unsent local edits only" after remote merges.
  • Scheduling behavior is covered in tests/integration/useSyncScheduling.spec.tsx: trailing-edge sync uses the settled scene snapshot, and readiness transitions reschedule pending work instead of flushing stale intermediate state.
  • Targeted bootstrap delivery is covered in tests/integration/socket.spec.mjs and tests/integration/multinode-redis.spec.mjs: direct scene bootstrap reaches only the targeted socket, including across nodes.
  • End-to-end collaboration coverage in playwright/e2e/collaboration.spec.ts asserts direct bootstrap on join and one-element SCENE_UPDATE delivery during a real two-session collaboration flow.
  • Invariant: only unsynced local edits are incrementally broadcast; remote-applied versions advance sync markers without causing echo rebroadcasts.

Benchmark

This branch now includes a rerunnable full-stack benchmark instead of a helper-only microbench:

  • npm run bench:incremental-sync
  • npm run bench:incremental-sync -- --scenario planning-flow --mutation selection-drag
  • npm run bench:incremental-sync -- --output-json tools/benchmarks/incremental-sync-results.json
  • runner: tools/benchmarks/runIncrementalSyncBenchmark.mjs
  • benchmark spec: playwright/bench/incremental-sync-benchmark.spec.ts
  • fixtures/hooks: playwright/support/incrementalSyncBenchmarkFixtures.ts, playwright/support/incrementalSyncBenchmarkHooks.ts

What it does:

  • boots the Playwright Nextcloud test stack plus the real websocket server
  • creates a real whiteboard in Nextcloud
  • seeds the board through the real PUT /apps/whiteboard/{fileId} save path
  • opens two real whiteboard sessions and measures targeted bootstrap plus emitted/received SCENE_UPDATE payloads

Local output on April 2, 2026 (default 3x3 matrix, scale=1, runs=1):

Scenario Mutation Elements Changed Bootstrap Full update Incremental Reduction Late join ready Emit Remote apply
retro board single text edit 214 1 120.3 KB 115.5 KB 690 B 99.40% 1233 ms 549 ms 552 ms
planning flow selection drag 364 18 203.7 KB 198.7 KB 9.8 KB 95.07% 1006 ms 567 ms 569 ms
media review mixed session burst 174 27 98.1 KB 97.1 KB 15.3 KB 84.27% 1429 ms 537 ms 541 ms

Across the full 3x3 matrix:

  • payload reduction ranged from 84.27% to 99.66%
  • actual emitted element count matched the intended mutation size in every case
  • late-join targeted bootstrap emit time ranged from 898 ms to 1429 ms
  • incremental emit time ranged from 535 ms to 578 ms
  • remote apply time on the other session ranged from 537 ms to 579 ms

Dependency and prerequisite CI updates for this work are tracked separately in #1116.

@hweihwang hweihwang force-pushed the feat/incremetal-sync-improvements branch 2 times, most recently from b95f4aa to fff5e76 Compare March 30, 2026 11:15
@hweihwang hweihwang changed the title incremental scene sync with socket-scoped leader incremental scene sync Mar 30, 2026
@hweihwang hweihwang force-pushed the feat/incremetal-sync-improvements branch 2 times, most recently from 495dbf5 to 359fa23 Compare March 30, 2026 12:55
@hweihwang hweihwang marked this pull request as ready for review March 30, 2026 12:56
@hweihwang hweihwang force-pushed the feat/incremetal-sync-improvements branch 2 times, most recently from 4d12410 to b1c380a Compare March 30, 2026 13:51
@hweihwang hweihwang force-pushed the feat/incremetal-sync-improvements branch 2 times, most recently from d665eb6 to abe90f6 Compare March 31, 2026 14:10
@hweihwang hweihwang changed the title incremental scene sync incremental collaboration sync with targeted bootstrap Apr 1, 2026
@hweihwang hweihwang force-pushed the feat/incremetal-sync-improvements branch 5 times, most recently from 5b07050 to d819a0c Compare April 2, 2026 08:00
@blizzz blizzz added enhancement New feature or request 3. to review labels Apr 2, 2026
@hweihwang hweihwang force-pushed the feat/incremetal-sync-improvements branch 2 times, most recently from aa9ffb8 to 5e63350 Compare April 2, 2026 09:52
Signed-off-by: Hoang Pham <hoangmaths96@gmail.com>
Signed-off-by: Hoang Pham <hoangmaths96@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

3. to review enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

High network bandwidth usage when using websocket server Switch whiteboard collab to Excalidraw-style incremental scene sync (stop full-scene spam)

2 participants