Skip to content

Reduce peak memory usage during release builds to fix OOM on manylinux runners #1445

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
kevinjqliu wants to merge 2 commits into
base: main
Choose a base branch
from
+16 −3
Open

Reduce peak memory usage during release builds to fix OOM on manylinux runners #1445

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
83138ef
adjust swap to 8gb
kevinjqliu Mar 27, 2026
634e32d
modify profile.release
kevinjqliu Mar 27, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 14 additions & 1 deletion .github/workflows/build.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -158,6 +158,19 @@ jobs:
with:
enable-cache: true

- name: Add extra swap for release build
if: inputs.build_mode == 'release'
run: |
set -euxo pipefail
sudo swapoff -a || true
sudo rm -f /swapfile
sudo fallocate -l 8G /swapfile || sudo dd if=/dev/zero of=/swapfile bs=1M count=8192
sudo chmod 600 /swapfile
sudo mkswap /swapfile
sudo swapon /swapfile
free -h
swapon --show

- name: Build (release mode)
uses: PyO3/maturin-action@v1
if: inputs.build_mode == 'release'
Expand Down Expand Up @@ -232,7 +245,7 @@ jobs:
set -euxo pipefail
sudo swapoff -a || true
sudo rm -f /swapfile
sudo fallocate -l 16G /swapfile || sudo dd if=/dev/zero of=/swapfile bs=1M count=16384
sudo fallocate -l 8G /swapfile || sudo dd if=/dev/zero of=/swapfile bs=1M count=8192
Copy link
Copy Markdown
Contributor Author

@kevinjqliu kevinjqliu Mar 27, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we dont need all 16GB, take less disk space

sudo chmod 600 /swapfile
sudo mkswap /swapfile
sudo swapon /swapfile
Expand Down
4 changes: 2 additions & 2 deletions Cargo.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -64,8 +64,8 @@ pyo3-build-config = "0.28"
datafusion-python-util = { path = "crates/util" }

[profile.release]
lto = true
codegen-units = 1
lto = "thin"
codegen-units = 2

# We cannot publish to crates.io with any patches in the below section. Developers
# must remove any entries in this section before creating a release candidate.
Expand Down
Loading