Skip to content

ci: add swap during build, use tpchgen-cli #1443

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
timsaucer merged 12 commits into from
Mar 26, 2026
Merged

ci: add swap during build, use tpchgen-cli #1443

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
f106c2b
restrict number of rustc jobs during build stage
timsaucer Mar 25, 2026
14f6b0f
temporarily run release build on PR
timsaucer Mar 25, 2026
3ee76a8
Change optimization setting for substrait
timsaucer Mar 25, 2026
f36b1e6
Add swap during release build
timsaucer Mar 26, 2026
cdfff1e
Remove temporary checks to build in PR
timsaucer Mar 26, 2026
ea02fba
Try using tpchgen-cli for test files. commit answers
timsaucer Mar 26, 2026
0904924
taplo fmt
timsaucer Mar 26, 2026
f9a3a70
do not run rat on data files
timsaucer Mar 26, 2026
63e998e
ci needs ./ in path
timsaucer Mar 26, 2026
f933819
add no-project to uv run
timsaucer Mar 26, 2026
3f1d9a3
Temporary debug lines to figure out what is happening in CI
timsaucer Mar 26, 2026
7f9c17a
filter null value during aggregation instead now that https://github.…
timsaucer Mar 26, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 13 additions & 0 deletions .github/workflows/build.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -226,6 +226,19 @@ jobs:
with:
enable-cache: true

- name: Add extra swap for release build
if: inputs.build_mode == 'release'
run: |
set -euxo pipefail
sudo swapoff -a || true
sudo rm -f /swapfile
sudo fallocate -l 16G /swapfile || sudo dd if=/dev/zero of=/swapfile bs=1M count=16384
sudo chmod 600 /swapfile
sudo mkswap /swapfile
sudo swapon /swapfile
free -h
swapon --show
- name: Build (release mode)
uses: PyO3/maturin-action@v1
if: inputs.build_mode == 'release'
Expand Down
17 changes: 5 additions & 12 deletions .github/workflows/test.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -101,21 +101,14 @@ jobs:
cd examples/datafusion-ffi-example
uv run --no-project pytest python/tests/_test*.py

- name: Cache the generated dataset
id: cache-tpch-dataset
uses: actions/cache@v5
with:
path: benchmarks/tpch/data
key: tpch-data-2.18.0

- name: Run dbgen to create 1 Gb dataset
if: ${{ steps.cache-tpch-dataset.outputs.cache-hit != 'true' }}
- name: Run tpchgen-cli to create 1 Gb dataset
run: |
cd benchmarks/tpch
RUN_IN_CI=TRUE ./tpch-gen.sh 1
mkdir examples/tpch/data
cd examples/tpch/data
uv pip install tpchgen-cli
uv run --no-project tpchgen-cli -s 1 --format=parquet

- name: Run TPC-H examples
run: |
cd examples/tpch
uv run --no-project python convert_data_to_parquet.py
uv run --no-project pytest _tests.py
3 changes: 2 additions & 1 deletion dev/release/rat_exclude_files.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -47,4 +47,5 @@ benchmarks/tpch/queries/q*.sql
benchmarks/tpch/create_tables.sql
.cargo/config.toml
**/.cargo/config.toml
uv.lock
uv.lock
examples/tpch/answers_sf1/*.tbl
2 changes: 1 addition & 1 deletion examples/tpch/_tests.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -115,7 +115,7 @@ def test_tpch_query_vs_answer_file(query_code: str, answer_file: str) -> None:
get_answer_file(answer_file),
schema=read_schema,
delimiter="|",
file_extension=".out",
file_extension=".tbl",
)

df_expected = df_expected.select(*expected_selections)
Expand Down
5 changes: 5 additions & 0 deletions examples/tpch/answers_sf1/q1.tbl
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
l|l|sum_qty |sum_base_price |sum_disc_price |sum_charge |avg_qty |avg_price |avg_disc |count_order
A|F|37734107.00|56586554400.73|53758257134.87|55909065222.83|25.52|38273.13|0.05| 1478493
N|F|991417.00|1487504710.38|1413082168.05|1469649223.19|25.52|38284.47|0.05| 38854
N|O|74476040.00|111701729697.74|106118230307.61|110367043872.50|25.50|38249.12|0.05| 2920374
R|F|37719753.00|56568041380.90|53741292684.60|55889619119.83|25.51|38250.85|0.05| 1478870
21 changes: 21 additions & 0 deletions examples/tpch/answers_sf1/q10.tbl
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
c_custkey |c_name |revenue |c_acctbal |n_name |c_address |c_phone |c_comment
57040|Customer#000057040 |734235.25|632.87|JAPAN |Eioyzjf4pp |22-895-641-3466|sits. slyly regular requests sleep alongside of the regular inst
143347|Customer#000143347 |721002.69|2557.47|EGYPT |1aReFYv,Kw4 |14-742-935-3718|ggle carefully enticing requests. final deposits use bold, bold pinto beans. ironic, idle re
60838|Customer#000060838 |679127.31|2454.77|BRAZIL |64EaJ5vMAHWJlBOxJklpNc2RJiWE |12-913-494-9813| need to boost against the slyly regular account
101998|Customer#000101998 |637029.57|3790.89|UNITED KINGDOM |01c9CILnNtfOQYmZj |33-593-865-6378|ress foxes wake slyly after the bold excuses. ironic platelets are furiously carefully bold theodolites
125341|Customer#000125341 |633508.09|4983.51|GERMANY |S29ODD6bceU8QSuuEJznkNaK |17-582-695-5962|arefully even depths. blithely even excuses sleep furiously. foxes use except the dependencies. ca
25501|Customer#000025501 |620269.78|7725.04|ETHIOPIA | W556MXuoiaYCCZamJI,Rn0B4ACUGdkQ8DZ |15-874-808-6793|he pending instructions wake carefully at the pinto beans. regular, final instructions along the slyly fina
115831|Customer#000115831 |596423.87|5098.10|FRANCE |rFeBbEEyk dl ne7zV5fDrmiq1oK09wV7pxqCgIc|16-715-386-3788|l somas sleep. furiously final deposits wake blithely regular pinto b
84223|Customer#000084223 |594998.02|528.65|UNITED KINGDOM |nAVZCs6BaWap rrM27N 2qBnzc5WBauxbA |33-442-824-8191| slyly final deposits haggle regular, pending dependencies. pending escapades wake
54289|Customer#000054289 |585603.39|5583.02|IRAN |vXCxoCsU0Bad5JQI ,oobkZ |20-834-292-4707|ely special foxes are quickly finally ironic p
39922|Customer#000039922 |584878.11|7321.11|GERMANY |Zgy4s50l2GKN4pLDPBU8m342gIw6R |17-147-757-8036|y final requests. furiously final foxes cajole blithely special platelets. f
6226|Customer#000006226 |576783.76|2230.09|UNITED KINGDOM |8gPu8,NPGkfyQQ0hcIYUGPIBWc,ybP5g, |33-657-701-3391|ending platelets along the express deposits cajole carefully final
922|Customer#000000922 |576767.53|3869.25|GERMANY |Az9RFaut7NkPnc5zSD2PwHgVwr4jRzq |17-945-916-9648|luffily fluffy deposits. packages c
147946|Customer#000147946 |576455.13|2030.13|ALGERIA |iANyZHjqhyy7Ajah0pTrYyhJ |10-886-956-3143|ithely ironic deposits haggle blithely ironic requests. quickly regu
115640|Customer#000115640 |569341.19|6436.10|ARGENTINA |Vtgfia9qI 7EpHgecU1X |11-411-543-4901|ost slyly along the patterns; pinto be
73606|Customer#000073606 |568656.86|1785.67|JAPAN |xuR0Tro5yChDfOCrjkd2ol |22-437-653-6966|he furiously regular ideas. slowly
110246|Customer#000110246 |566842.98|7763.35|VIETNAM |7KzflgX MDOq7sOkI |31-943-426-9837|egular deposits serve blithely above the fl
142549|Customer#000142549 |563537.24|5085.99|INDONESIA |ChqEoK43OysjdHbtKCp6dKqjNyvvi9 |19-955-562-2398|sleep pending courts. ironic deposits against the carefully unusual platelets cajole carefully express accounts.
146149|Customer#000146149 |557254.99|1791.55|ROMANIA |s87fvzFQpU |29-744-164-6487| of the slyly silent accounts. quickly final accounts across the
52528|Customer#000052528 |556397.35|551.79|ARGENTINA |NFztyTOR10UOJ |11-208-192-3205| deposits hinder. blithely pending asymptotes breach slyly regular re
23431|Customer#000023431 |554269.54|3381.86|ROMANIA |HgiV0phqhaIa9aydNoIlb |29-915-458-2654|nusual, even instructions: furiously stealthy n
Loading
Loading