Skip to content

NOTES-INTERNALS.md

Latest commit

 

History

History
161 lines (115 loc) · 6.67 KB

NOTES-INTERNALS.md

File metadata and controls

161 lines (115 loc) · 6.67 KB

Apple Notes Internals

Research notes from reverse-engineering the Apple Notes storage format.

Database

Notes live in a Core Data-backed SQLite database:

~/Library/Group Containers/group.com.apple.notes/NoteStore.sqlite

Key tables:

Table Purpose
ZICCLOUDSYNCINGOBJECT Polymorphic table for all entities (ICNote, ICFolder, ICAttachment, etc.)
ZICNOTEDATA Note content blobs. ZNOTE FK to the note's Z_PK
ZICCLOUDSTATE Per-object sync version tracking
ATRANSACTION Core Data persistent history transactions
ACHANGE Per-entity change records within a transaction
Z_PRIMARYKEY Entity metadata + auto-increment max values

Note Content Format

Each note's content is stored as a gzip-compressed protobuf blob in ZICNOTEDATA.ZDATA. The protobuf contains:

  • text — full plain text of the note (Unicode, \ufffc for attachment placeholders)
  • crdtEntries — character-level CRDT edit history
  • metadata — UUID + per-replica counters
  • attributeRuns — styling spans (bold, headings, checklists, links, etc.)

See notestore.proto for the full schema and proto.ts for the codec.

CRDT Entries

Each note contains a list of CRDT entries tracking character insertions and deletions:

CRDTEntry {
  start: { replicaId, counter }   // allocation position
  length: uint32                   // number of characters
  end: { replicaId, counter }     // parent/predecessor reference
  flag: int32                      // 1 = tombstone (deleted text)
  sequence: uint32                 // 1-based ordering
}

The last entry is always a sentinel: start={0, 4294967295}, length=0, end={0, 4294967295}.

Key findings

  • The text field is the source of truth for display, but the CRDT entries are the source of truth for sync. If you change the text without updating CRDT entries, the next real edit through Notes.app will revert your change.
  • Apple compacts the CRDT log on every save. Entries get reordered by document position (not creation time), and can be split when text is inserted in the middle. This is NOT a pure append-only log.
  • Metadata counters are [next_start_counter, next_end_counter] for replica 1. When adding entries, bump the start counter by the number of characters inserted.
  • The end field's exact semantics are not fully decoded. For batch inserts (multiple chars typed at once), end is {1, 0}. For single-char sequential typing, end counters increment.
  • Entries with flag=1 are tombstones (deleted characters). They still count toward counter allocation but not toward the visible text length.

Attribute Runs

Styling is stored as a list of attribute runs that tile the text. Each run specifies:

  • length — number of characters covered
  • paragraphStyle — style type (0=Title, 1=Heading, 2=Subheading, 4=Mono, 100=Bullet, 101=Dash, 102=Number, 103=Checkbox), paragraph UUID, indent, block quote
  • fontWeight — 1=bold, 3=bold+italic(?)
  • underlined, strikethrough, superscript — integer flags
  • link — URL string (https://... or applenotes:note/...)
  • attachmentInfo — { id, type } for inline attachments
  • timestamp — seconds since Unix epoch, present on body text runs

The sum of all attribute run lengths must equal the text length.

How to Edit a Note and Have It Sync

Tested and confirmed working with iCloud sync. The full procedure:

1. Modify the protobuf

  • Update the text field
  • Add/modify CRDT entries (new entry before sentinel, bump sequence)
  • Update metadata counters (next_start_counter += chars_added)
  • Add/update attribute runs to cover the new text length

2. Write to ZICNOTEDATA

UPDATE ZICNOTEDATA SET ZDATA = <gzipped_protobuf>, Z_OPT = Z_OPT + 1
WHERE Z_PK = <notedata_pk>;

3. Update the note row

UPDATE ZICCLOUDSYNCINGOBJECT
SET ZMODIFICATIONDATE1 = <apple_epoch_now>, Z_OPT = Z_OPT + 1
WHERE Z_PK = <note_pk>;

Apple epoch = seconds since 2001-01-01T00:00:00Z.

4. Bump the cloud state version

The note's ZCLOUDSTATE column points to a ZICCLOUDSTATE row. Increment ZCURRENTLOCALVERSION so it exceeds ZLATESTVERSIONSYNCEDTOCLOUD:

UPDATE ZICCLOUDSTATE
SET ZCURRENTLOCALVERSION = ZCURRENTLOCALVERSION + 1,
    ZLOCALVERSIONDATE = <apple_epoch_now>,
    Z_OPT = Z_OPT + 1
WHERE Z_PK = <cloudstate_pk>;

This is what tells the sync daemon "this record is dirty."

5. Create Core Data persistent history entries

Create a transaction:

INSERT INTO ATRANSACTION (Z_PK, Z_ENT, Z_OPT, ZBUNDLEIDTS, ZPROCESSIDTS, ZTIMESTAMP)
VALUES (<next_pk>, 16002, NULL, 6, 7, <apple_epoch_now>);
-- ZBUNDLEIDTS=6 → "com.apple.Notes", ZPROCESSIDTS=7 → "Notes"

Create change records (one per modified entity):

-- ICNoteData (entity 19): ZCOLUMNS = 0x10 (ZDATA column)
INSERT INTO ACHANGE (Z_PK, Z_ENT, Z_OPT, ZCHANGETYPE, ZENTITY, ZENTITYPK, ZTRANSACTIONID, ZCOLUMNS)
VALUES (<next_pk>, 16001, NULL, 1, 19, <notedata_pk>, <txn_pk>, X'10');

-- ICNote (entity 12): ZCOLUMNS = modification date bitmask
INSERT INTO ACHANGE (Z_PK, Z_ENT, Z_OPT, ZCHANGETYPE, ZENTITY, ZENTITYPK, ZTRANSACTIONID, ZCOLUMNS)
VALUES (<next_pk>, 16001, NULL, 1, 12, <note_pk>, <txn_pk>, X'00000000000000200000');

-- ICCloudState (entity 2): ZCOLUMNS = 0x24 (version + date columns)
INSERT INTO ACHANGE (Z_PK, Z_ENT, Z_OPT, ZCHANGETYPE, ZENTITY, ZENTITYPK, ZTRANSACTIONID, ZCOLUMNS)
VALUES (<next_pk>, 16001, NULL, 1, 2, <cloudstate_pk>, <txn_pk>, X'24');

6. Update Z_PRIMARYKEY

UPDATE Z_PRIMARYKEY SET Z_MAX = <new_max> WHERE Z_ENT = 16002; -- TRANSACTION
UPDATE Z_PRIMARYKEY SET Z_MAX = <new_max> WHERE Z_ENT = 16001; -- CHANGE

Notes

  • Notes.app has a file/database watcher and will pick up changes live (no restart needed).
  • The sync daemon pushes the change to iCloud automatically once the cloud state version is bumped.
  • ZBUNDLEIDTS and ZPROCESSIDTS reference rows in ATRANSACTIONSTRING (6 = "com.apple.Notes", 7 = "Notes").
  • ZCHANGETYPE 1 = update, 0 = insert, 2 = delete.
  • The ZCOLUMNS blob is a bitmask of which Core Data model properties changed.
  • All of this should be done inside a single SQLite transaction for atomicity.

What Doesn't Work

  • Editing text without updating CRDT entries: The change displays locally but gets reverted on the next real edit through Notes.app.
  • Writing to SQLite without Core Data tracking: The change displays locally (after app restart) but never syncs to iCloud. The sync daemon doesn't know the record is dirty.
  • Direct SQLite writes while Notes.app is running: Notes.app may overwrite your changes on its next save. Close it first, or accept that it might detect the change via its database watcher.