From f22b6093f8a97b70d4f78d6e444af02ee32bf8b6 Mon Sep 17 00:00:00 2001 From: Lev Kokotov Date: Wed, 8 Apr 2026 13:38:25 -0700 Subject: [PATCH 1/4] feat: document sharded sequences and new mirroring features --- docs/configuration/pgdog.toml/mirroring.md | 8 + docs/features/mirroring.md | 51 +++++- .../sharding/schema_management/migrations.md | 1 + docs/features/sharding/sequences.md | 147 ++++++++++++++++++ docs/features/sharding/unique-ids.md | 35 +++-- 5 files changed, 229 insertions(+), 13 deletions(-) create mode 100644 docs/features/sharding/sequences.md diff --git a/docs/configuration/pgdog.toml/mirroring.md b/docs/configuration/pgdog.toml/mirroring.md index 3731ea1..813fd34 100644 --- a/docs/configuration/pgdog.toml/mirroring.md +++ b/docs/configuration/pgdog.toml/mirroring.md @@ -41,3 +41,11 @@ Default: **none** (optional) The percentage of transactions to mirror, specified as a floating point number between 0.0 and 1.0. See [mirroring](../../features/mirroring.md) for more details. This overrides the [`mirror_exposure`](./general.md#mirror_exposure) setting. Default: **none** (optional) + +### `level` + +The type of statements to mirror. Available options are: + +- `ddl` +- `dml` +- `all` (default) diff --git a/docs/features/mirroring.md b/docs/features/mirroring.md index 7ffcb18..2efe119 100644 --- a/docs/features/mirroring.md +++ b/docs/features/mirroring.md @@ -14,7 +14,7 @@ Mirroring in PgDog is asynchronous and should have minimal impact on production Mirroring -## Configuration +### Configuration To use mirroring, first configure both the mirror and the production database in [`pgdog.toml`](../configuration/pgdog.toml/databases.md). Once both databases are running, add a `[[mirroring]]` section: @@ -43,7 +43,7 @@ Each client connected to the main database has its own queue, so concurrency sca You can have as many mirror databases as you like. Queries will be sent to each one of them, in parallel. More mirrors will require more CPU and network resources, so make sure to allocate enough compute to PgDog in production. -## Mirror queue +### Mirror queue If the mirror database(s) can't keep up with production traffic, queries will back up in the queue. To make sure it doesn't overflow and cause out-of-memory errors, the size of the queue is limited: @@ -69,7 +69,7 @@ If the queue gets full, all subsequent mirrored transactions will be dropped unt !!! note "Replication" Since mirror queues can drop queries, it is not a replacement for Postgres replication and should be used for testing & benchmarking purposes only. -## Exposure +### Exposure It's possible to limit how much traffic mirror databases receive. This is useful when warming up databases from a snapshot or if the mirror databases are smaller than production and can't handle as many transactions. @@ -96,8 +96,51 @@ Acceptable values are between **0.0** (0%) and **1.0** (100%). This is changeable at runtime, without restarting PgDog. When adding a mirror, it's a good idea to start slow, e.g., with only 0.1% exposure (`mirror_exposure = 0.01`), and gradually increase it over time. -## Realism +### Realism We try to make mirrored traffic as realistic as possible. For each statement inside a transaction, we record the timing between that statement and the next one. When replaying traffic against a mirror, we pause between statements for the same amount of time. This helps reproduce lock contention experienced by production databases, on the mirrors. + +### Filtering + +It's possible to filter what kind of statements mirrors receive using configuration, for example: + +=== "pgdog.toml" + ```toml + [[mirroring]] + source_db = "source" + destination_db = "dest" + level = "ddl" + ``` +=== "Helm chart" + ```yaml + mirroring: + - sourceDb: source + destinationDb: dest + level: ddl + ``` + +The `level` setting supports the following arguments: + +| Argument | Description | +|-|-| +| `ddl` | Mirror only DDL statements like `CREATE`, `DROP`, etc. | +| `dml` | Mirror all statements except DDL, e.g. `INSERT`, `UPDATE`, etc. | +| `all` | Mirror all statements. This is the default. | + +DDL-only mirroring is useful when maintaining long-running logical replicas, since the logical replication protocol doesn't support synchronizing schema changes. + +#### Query parser + +Filtering specific statements requires parsing queries. If your database setup doesn't have replicas or sharding, the query parser is typically disabled. Before using this feature, make sure to enable it in [`pgdog.toml`](../configuration/pgdog.toml/general.md#query_parser): + +=== "pgdog.toml" + ```toml + [general] + query_parser = "on" + ``` +=== "Helm chart" + ```yaml + queryParser: on + ``` diff --git a/docs/features/sharding/schema_management/migrations.md b/docs/features/sharding/schema_management/migrations.md index 9eca2ad..d48e303 100644 --- a/docs/features/sharding/schema_management/migrations.md +++ b/docs/features/sharding/schema_management/migrations.md @@ -1,6 +1,7 @@ --- icon: material/arrow-u-left-bottom --- + # Schema migrations PgDog expects that all shards have, roughly, the same tables. A notable exception to this rule is partitioned tables, diff --git a/docs/features/sharding/sequences.md b/docs/features/sharding/sequences.md new file mode 100644 index 0000000..3913f87 --- /dev/null +++ b/docs/features/sharding/sequences.md @@ -0,0 +1,147 @@ +--- +icon: material/numeric +--- + +# Sharded sequences + +!!! note "Unique IDs" + Sharded sequences require a bit more configuration to get working. If you're looking + for an easy way to generate cross-shard unique 64-bit integers, consider [Unique IDs](unique-ids.md). + +!!! note "Experimental feature" + This feature is new and experimental. Please report any issues you may run into and test it + before deploying to production. + +Sharded sequences are a way to generate monotonically increasing, globally unique 64-bit integers, without large gaps between numbers +or using a timestamp-based approach that produces very large numbers. + +They can be used for producing cross-shard unique primary keys in [sharded](query-routing.md#sharding-configuration) tables, directly inside the database. + +## How it works + +Sharded sequences combine two Postgres primitives: + +1. A normal sequence (created with `CREATE SEQUENCE`) +2. A hashing function, `satisfies_hash_partition`, used for number selection + +The two are called inside a PL/pgSQL function that fetches numbers from a sequence until `satisfies_hash_partition` returns `true`, for the total number of shards in the cluster and the shard number it's being executed on: + +```postgresql +LOOP + SELECT nextval('normal_seq'::regclass) INTO val; + + IF satisfies_hash_partition(/* ... */, val) THEN + RETURN val; + END IF; + +END; +``` + +Since fetching values from a sequence is very quick, we are able to find the correct number without introducing significant latency to row creation. The Postgres hash function is also good at producing uniform outputs, so all shards will have similar, small gaps between generated numbers. + +### Configuration + +Sharded sequences can only be used to generate primary keys for _sharded_ tables. [Omnisharded](omnishards.md) tables cannot use database sequences since they aren't guaranteed to produce the same number on all shards. + +To make sure this constraint is enforced, PgDog can inject [unique IDs](unique-ids.md) into omnisharded-targeted `INSERT` queries only: + +=== "pgdog.toml" + ```toml + [rewrite] + primary_key = "rewrite_omni" + ``` +=== "Helm chart" + ```yaml + rewrite: + primaryKey: rewrite_omni + ``` + +This configuration setting is required to use sharded sequences, so make sure to set it before proceeding. + +### Installation + +To install and use sharded sequences, configure [rewrites](#configuration) to target omnisharded tables only, add all the shards to [`pgdog.toml`](../../configuration/pgdog.toml/databases.md) `[[databases]]` section, and run the following [admin database](../../administration/index.md) command: + +=== "Admin database" + ``` + SETUP SCHEMA; + ``` +=== "CLI" + Since PgDog is also a CLI application, you can run the same command as follows: + + ``` + $ pgdog setup --database + ``` + + | Option | Description | + |-|-| + | `database` | Database `name` in `pgdog.toml`. | + +This command will perform the following steps: + +1. Install the [schema manager](schema_management/index.md) into all database shards along with the necessary PL/pgSQL functions +2. Find all tables that contain `BIGINT PRIMARY KEY` columns (incl. `BIGSERIAL`) and change their default values to call the sharded sequence function + +Once done, all subsequent `INSERT` statements that don't specify the primary key will automatically use the sharded sequence for their respective tables, for example: + +=== "Queries" + ```postgresql + -- Using DEFAULT explicitly. + INSERT INTO users + (id, email, tenant_id) + VALUES + (DEFAULT, 'admin@example.com', 5) RETURNING id; + + -- Omitting the primary key. + INSERT INTO users + (email, tenant_id) + VALUES + ('user@example.com', 5) RETURNING id; + ``` +=== "Output" + ``` + id + ---- + 1 + (1 row) + + id + ---- + 5 + (1 row) + ``` + +The returned `id` will be globally unique and monotonically increasing. + +### Migrations + +The schema manager will only install the sharded sequence in tables currently present in the database. When adding new tables or primary keys, make sure to execute the following PL/pgSQL function +as well: + +```postgresql +SELECT pgdog.install_sharded_sequence('schema_name', 'table_name', 'column_name'); +``` + +| Argument | Description | +|-|-| +| Schema name | The name of the schema where the table is being created. This is commonly the `public` schema, but can be any other as well. | +| Table name | The name of the new or existing table with the primary key. | +| Column name | The name of the primary key column. | + +##### Example + +The entire migration can be executed inside the same transaction: + +```postgresql +BEGIN; + +CREATE TABLE public.users ( + id BIGINT PRIMARY KEY, + email VARCHAR NOT NULL, + created_at TIMESTAMPTZ +); + +SELECT pgdog.install_sharded_sequence('public', 'users', 'id'); + +COMMIT; +``` diff --git a/docs/features/sharding/unique-ids.md b/docs/features/sharding/unique-ids.md index 77e5aa1..f88bdb4 100644 --- a/docs/features/sharding/unique-ids.md +++ b/docs/features/sharding/unique-ids.md @@ -3,15 +3,9 @@ icon: material/identifier --- # Unique IDs -To generate unique identifiers, regular PostgreSQL databases use [sequences](https://www.postgresql.org/docs/current/sql-createsequence.html). For example, `BIGSERIAL` and `SERIAL` columns get their values by calling: +To generate unique identifiers, regular PostgreSQL databases use [sequences](https://www.postgresql.org/docs/current/sql-createsequence.html). For example, `BIGSERIAL` and `SERIAL` columns get their values by calling `SELECT nextval('users_id_seq')`. -```postgresql -SELECT nextval('sequence_name'); -``` - -This guarantees that these columns contain unique and monotonically increasing integers. - -If your database is sharded, however, using sequences will create identical IDs for different rows on different shards. To address this, PgDog can generate unique 64-bit signed identifiers internally, based on the system clock. +This guarantees that these columns contain unique and monotonically increasing integers. If your database is sharded, however, using regular sequences will create identical IDs for different rows on different shards. To address this, PgDog can generate unique 64-bit signed identifiers internally, based on the system clock. ## How it works @@ -95,7 +89,7 @@ If you're migrating data from an existing database, you can ensure that all IDs unique_id_min = 5_000_000 ``` -When set, all generated IDs are guaranteed to be larger than this value. +When set, all generated IDs are guaranteed to be larger than this value. This feature however is normally not needed, since IDs generated by this function are very large. ## Limitations @@ -117,3 +111,26 @@ ID range is **69.73 years**, set to overflow on **August 3, 2095**. We expect da Since the identifiers are time-based, to ensure uniqueness, PgDog limits how many IDs can be generated per unit of time. This limit is currently **4,096** IDs per millisecond. When it's reached, PgDog will pause ID generation until the clock ticks to the next millisecond. This gives it an effective ID generation rate of _4,096,000 / second / node_, which should be sufficient for most deployments. + +## Compact IDs + +The unique ID algorithm generates very large 64-bit integers. This is because the timestamp portion is located 22 bits off to the left (little-endian). Some applications pass those IDs directly to the JavaScript-written frontends, which cannot display those numbers accurately: JS doesn't support any numbers larger than 2^53 - 1. + +For this reason, we added a more "compact", 53-bit unique ID generator function. It can be used by enabling it in [`pgdog.toml`](../../configuration/pgdog.toml/general.md): + +```toml +[general] +unique_id_function = "compact" +``` + +!!! warning "Switching to the compact generator" + If you're currently using the `"standard"` unique ID generator (default), be careful switching because the IDs it will generate will be considerably smaller, breaking the monotonic guarantee + and possibly causing unique index constraint errors. + +### Limitations + +Since the bit space in this function is smaller, and the timestamp granularity had to remain the same (ms), the space allocated to the node identifier +and the internal sequence has been reduced accordingly. + +For this reason, the compact function only has a generation rate of _64,000 / second / node_ and supports up to 64 total nodes +in the same deployment. From 4e719278a2330bf4a7cc71eb49a6c8dcf27164ce Mon Sep 17 00:00:00 2001 From: Lev Kokotov Date: Wed, 8 Apr 2026 13:43:31 -0700 Subject: [PATCH 2/4] fix ci --- tests/test_code_blocks.py | 66 ++++++++++++++++++++++++--------------- 1 file changed, 40 insertions(+), 26 deletions(-) diff --git a/tests/test_code_blocks.py b/tests/test_code_blocks.py index a2c7912..3065e66 100644 --- a/tests/test_code_blocks.py +++ b/tests/test_code_blocks.py @@ -3,14 +3,17 @@ import glob import re import subprocess -from markdown_it import MarkdownIt import sys import pglast -mdp = MarkdownIt() - +# Match fenced code blocks, including those nested inside mkdocs-material +# `===` content tabs (which indent the fence by 4+ spaces). The leading +# indentation is captured so it can be stripped from each code line, and the +# closing fence must match both the indent and the opening fence marker. pattern = re.compile( - r'(?msi)^(?P[`~]{3,})[^\n]*\r?\n(?P.*?)^(?P=fence)[ \t]*\r?$' + r'(?ms)^(?P[ \t]*)(?P`{3,}|~{3,})(?P[^\n]*)\r?\n' + r'(?P.*?)' + r'^(?P=indent)(?P=fence)[ \t]*\r?$' ) replication = [ @@ -19,31 +22,42 @@ ] def verify(binary): - for file in glob.glob("docs/**/*.md", - recursive=True): + for file in glob.glob("docs/**/*.md", recursive=True): with open(file, "r") as f: content = f.read() - print(f"Checking {file}") - tokens = mdp.parse(content) - for token in tokens: - if token.type == "fence" and token.info == "toml": - if "[[users]]" in token.content: - check_file(binary, "users", token.content) - elif "[lib]" in token.content: - pass + print(f"Checking {file}") + for m in pattern.finditer(content): + info = m.group("info").strip().lower() + indent = m.group("indent") + code = m.group("code") + if indent: + # Dedent the code body so configcheck/pglast see clean text. + stripped_lines = [] + for line in code.splitlines(keepends=True): + if line.startswith(indent): + stripped_lines.append(line[len(indent):]) else: - check_file(binary, "pgdog", token.content) - elif token.type == "fence" and token.info == "postgresql": - try: - pglast.parser.parse_sql(token.content) - except Exception as e: - found = False - for cmd in replication: - if cmd in token.content: - found = True - if not found: - print(token.content) - raise e + stripped_lines.append(line) + code = "".join(stripped_lines) + + if info == "toml": + if "[[users]]" in code: + check_file(binary, "users", code) + elif "[lib]" in code: + pass + else: + check_file(binary, "pgdog", code) + elif info == "postgresql": + try: + pglast.parser.parse_sql(code) + except Exception as e: + found = False + for cmd in replication: + if cmd in code: + found = True + if not found: + print(code) + raise e def check_file(binary, kind, content): tmp = f"/tmp/pgdog_config_test.toml" From b0b082c5b9ed5f8c5ea76f530bccfc1dbfeda51e Mon Sep 17 00:00:00 2001 From: Lev Kokotov Date: Wed, 8 Apr 2026 14:45:14 -0700 Subject: [PATCH 3/4] Eyy! --- docs/features/mirroring.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/features/mirroring.md b/docs/features/mirroring.md index 2efe119..eb6bd2c 100644 --- a/docs/features/mirroring.md +++ b/docs/features/mirroring.md @@ -61,7 +61,7 @@ If the mirror database(s) can't keep up with production traffic, queries will ba [[mirroring]] source_db = "source" destination_db = "dest" - queue_depth = 500 + queue_length = 500 ``` If the queue gets full, all subsequent mirrored transactions will be dropped until there is space in the queue again. From 60d976f6ea6e6dfbcdf6fd7ef5f1dc6981e42e0f Mon Sep 17 00:00:00 2001 From: Lev Kokotov Date: Wed, 8 Apr 2026 15:00:30 -0700 Subject: [PATCH 4/4] look at all that --- docs/features/sharding/manual-routing.md | 4 ++-- docs/features/sharding/sequences.md | 14 ++++++++------ docs/features/sharding/sharding-functions.md | 9 +++++++++ docs/migrating-to-pgdog/from-pgbouncer.md | 4 ++-- tests/test_code_blocks.py | 4 +++- 5 files changed, 24 insertions(+), 11 deletions(-) diff --git a/docs/features/sharding/manual-routing.md b/docs/features/sharding/manual-routing.md index d0fa5f4..d675f78 100644 --- a/docs/features/sharding/manual-routing.md +++ b/docs/features/sharding/manual-routing.md @@ -30,7 +30,7 @@ The PostgreSQL query language supports adding inline comments to queries. They a The following query will be sent to shard number zero: ```postgresql - /* pgdog_shard: 0 */ CREATE INDEX CONCURRENTLY users_id_idx USING btree(id); + /* pgdog_shard: 0 */ CREATE INDEX CONCURRENTLY users_id_idx ON users USING btree(id); ``` === "Sharding key" This query will be sent to whichever shard maps to the key `"us-east-1"`: @@ -70,7 +70,7 @@ The `SET` command comes from the PostgreSQL query language and is used to change ```postgresql BEGIN; SET LOCAL pgdog.shard TO 0; - CREATE INDEX users_id_idx USING btree(id); + CREATE INDEX users_id_idx ON users USING btree(id); COMMIT; ``` === "Sharding key" diff --git a/docs/features/sharding/sequences.md b/docs/features/sharding/sequences.md index 3913f87..f559faa 100644 --- a/docs/features/sharding/sequences.md +++ b/docs/features/sharding/sequences.md @@ -27,14 +27,16 @@ Sharded sequences combine two Postgres primitives: The two are called inside a PL/pgSQL function that fetches numbers from a sequence until `satisfies_hash_partition` returns `true`, for the total number of shards in the cluster and the shard number it's being executed on: ```postgresql -LOOP - SELECT nextval('normal_seq'::regclass) INTO val; +DO $$ +BEGIN + LOOP + SELECT nextval('normal_seq'::regclass) INTO val; - IF satisfies_hash_partition(/* ... */, val) THEN + IF satisfies_hash_partition(/* ... */, val) THEN RETURN val; - END IF; - -END; + END IF; + END LOOP; +END $$; ``` Since fetching values from a sequence is very quick, we are able to find the correct number without introducing significant latency to row creation. The Postgres hash function is also good at producing uniform outputs, so all shards will have similar, small gaps between generated numbers. diff --git a/docs/features/sharding/sharding-functions.md b/docs/features/sharding/sharding-functions.md index 6fd34a0..115f99e 100644 --- a/docs/features/sharding/sharding-functions.md +++ b/docs/features/sharding/sharding-functions.md @@ -51,18 +51,27 @@ All queries referencing the `user_id` column will be automatically sent to the m Different integer types are treated the same by the query router. If you're using `BIGINT`, `INTEGER` or `SMALLINT` as your sharding key, you can specify `bigint` in the configuration: ```toml + [[sharded_tables]] + database = "prod" + column = "user_id" data_type = "bigint" ``` === "Text" !!! note "Text types" `VARCHAR`, `VARCHAR(n)`, and `TEXT` use the same encoding and are treated the same by the query router. For either one, you can specify `varchar` in the configuration: ```toml + [[sharded_tables]] + database = "prod" + column = "serial_number" data_type = "varchar" ``` === "UUID" !!! note "UUID types" Only UUIDv4 is currently supported for sharding in the query router. ```toml + [[sharded_tables]] + database = "prod" + column = "unique_id" data_type = "uuid" ``` diff --git a/docs/migrating-to-pgdog/from-pgbouncer.md b/docs/migrating-to-pgdog/from-pgbouncer.md index c980da6..e87b93e 100644 --- a/docs/migrating-to-pgdog/from-pgbouncer.md +++ b/docs/migrating-to-pgdog/from-pgbouncer.md @@ -58,8 +58,8 @@ Both PgBouncer and PgDog can override the user's password used to connect to Pos name = "prod" host = "10.0.0.1" port = 5432 - server_user = "postgres" - server_password = "hunter2" + user = "postgres" + password = "hunter2" pooler_mode = "transaction" ``` diff --git a/tests/test_code_blocks.py b/tests/test_code_blocks.py index 3065e66..1b715ef 100644 --- a/tests/test_code_blocks.py +++ b/tests/test_code_blocks.py @@ -30,6 +30,7 @@ def verify(binary): info = m.group("info").strip().lower() indent = m.group("indent") code = m.group("code") + original = code if indent: # Dedent the code body so configcheck/pglast see clean text. stripped_lines = [] @@ -56,7 +57,8 @@ def verify(binary): if cmd in code: found = True if not found: - print(code) + print(f"Error in {file}:") + print(original) raise e def check_file(binary, kind, content):