ci: add swap during build, use tpchgen-cli

[timsaucer]

Which issue does this PR close?

Related to #1429 but we need to verify if it resolves the issue.

Rationale for this change

We are getting OOM errors during build. This adds a swap to the build stage to prevent them.

What changes are included in this PR?

Add swap during build stage.
Added tpchgen-cli to generate TPC-H data, so also committed the answer files.

Are there any user-facing changes?

No

[kevinjqliu]

I was looking into fixing this and testing on my fork,

adding swap helped
https://github.com/kevinjqliu/datafusion-python/pull/4/files#diff-5c3fa597431eda03ac3339ae6bf7f05e1a50d6fc7333679ec38e21b337cb6721R244-R255

we can add it to give us more buffer

[kevinjqliu]

this should help, we can even do CARGO_BUILD_JOBS=1 and also add it to the "debug" mode below on L241

[kevinjqliu]

Lets see if scoping it to substrait will help here.

I think we might need to bite the bullet and do this:

[profile.release]
lto = "thin"
codegen-units = 4

or override the options using env var just for that one job.

but note that this affects the final artifact

[kevinjqliu]

Claude on why profile.release.package.substrait might not work:
"""
That's a reasonable idea but it won't help with your specific OOM. The problem is that with lto = true (fat LTO), all crates get merged into a single IR blob inside rustc for the final optimization pass — the per-package opt-level and codegen-units overrides apply during the initial compilation of each crate, but by the time the LTO link step runs, rustc is holding everything together regardless.
"""

The priority order that minimizes artifact impact:

• More swap — zero artifact impact, just survival insurance (this worked when i tested on fork)
• mold linker — zero artifact impact, just more memory-efficient linking (trying this out https://github.com/kevinjqliu/datafusion-python/actions/runs/23552615526) EDIT this failed
• Set release option (lto = "thin", codegen-units = 4)

[timsaucer]

Thanks @kevinjqliu ! If the swap is good enough to get us around this OOM then I think that's the best option.

[kevinjqliu]

22 unapproved licences. Check rat report: rat.txt

maybe need to add the new examples/tpch/answers_sf1/* files to the dev/release/rat_exclude_files.txt file

[kevinjqliu]

LGTM!

Looks like adding the swap worked.

[kevinjqliu]

        Error: Custom { kind: NotFound, error: "Could not find `protoc`.
      If `protoc` is installed, try setting the `PROTOC` environment
      variable to the path of the `protoc` binary. To install it on Debian,
      run `apt-get install protobuf-compiler`. It is also available at

probably needs

      - name: Install protoc
        uses: arduino/setup-protoc@v3
        with:
          repo-token: ${{ secrets.GITHUB_TOKEN }}

[timsaucer]

Super frustrating because these same tests pass locally for me every time

[kevinjqliu]

i was able to repro the issue locally. claude says its a datafusion regression, im running df 52 locally to check

[timsaucer]

I take it back. Now I'm getting a failure. I must not have built recently. Investigating.

[timsaucer]

Found it: apache/datafusion#21011 and that tpch code depended on it.

[kevinjqliu]

on Mac M4
Tried with 52.3 on this commit (7f484cd) and it worked.

Looks like its due to apache/datafusion#21011
and

datafusion-python/examples/tpch/q21_suppliers_kept_orders_waiting.py

Line 91 in 1397c5d

"failed_suppliers", F.array_remove(col("failed_suppliers"), lit(None))

[kevinjqliu]

heres a potential fix kevinjqliu@1fb80a3

according to claude
"""
Note: the upstream fix also changes the semantics — per Trino/Spark, array_remove(array, NULL) should return NULL (not remove nulls from the array). So even after the upstream fix lands, the q21 example would still need the workaround since it relies on null-removal behavior.
"""

[timsaucer]

looks like claude code and tim code came to the same conclusion

[timsaucer]

here goes nothing...

[kevinjqliu]

Error: The operation was canceled.

https://github.com/apache/datafusion-python/actions/runs/23614615854/job/68778937649

@timsaucer did you cancel it?

[timsaucer]

Nope, I don't think so. I'm going to let the rest finish and rerun that work flow

[kevinjqliu]

nice! its all green now https://github.com/apache/datafusion-python/actions/runs/23614615854