0.65.0

Changes

⚠️ Breaks

• Remove BitBuffer::into_mut and require callers to handle failure to acquire unique ownership (#7136) @robert3005
• Update DeltaArray to always pad values to 1024 elements and use bit transpose functions (#7007) @robert3005

✨ Features

• Add web UI (#7151) @gatesn
• feat[vortex-datafusion]: use PartitionedFile instead of path in create_reader (#7156) @asubiotto
• Part 2 of making Variant a canonical array (#7143) @AdamGS
• feat[array]: use transmute for up/down cast of Arc (#7140) @joseph-isaacs
• feat[array]: change VTable execution to pass owned arrays (#6867) @joseph-isaacs
• feat(cuda): hybrid GPU dispatch - fuse dyn + standalone kernels (#7127) @0ax1
• default mimalloc for vortex-python wheels (#7096) @onursatici
• feat(cuda): add SEQUENCE source op to dyn dispatch (#7078) @0ax1
• Vector Extension Type (#6964) @connortsui20
• Canonical variant array (#7026) @AdamGS
• IsSorted AggregateFn (#7029) @gatesn
• IsConstant Aggregate Function (#7028) @gatesn
• MinMax AggregateFn (#7006) @gatesn
• Variant DType (#6912) @AdamGS
• feat[back-compat]: Add a script to upload fixtures to a repo (#6993) @joseph-isaacs
• NanCount AggregateFn (#7003) @gatesn
• Cut over sum aggregate function (#6910) @gatesn
• feat[compat-gen]: add expected_encodings to validate fixture construction (#6978) @joseph-isaacs

🚀 Performance

• do not validate already valid arrow varbin arrays (#7089) @onursatici
• perf[fsst]: like pushdown using a dfa (#6935) @joseph-isaacs
• Decimal sum uses smallest physical type that supports output dtype (#7008) @robert3005
• perf: skip allocation for prim cast if possible (#6997) @0ax1
• Faster true count using AVX2 and AVX512 instructions (#6931) @robert3005

🐛 Bug Fixes

23 changes

• fix(python): cap substrait-python before 0.29.0 (#7161) @RealXuChe
• fix(python): cap substrait below 0.85.0 (#7153) @RealXuChe
• fill_forward_nulls resets at 1024 chunk boundary (#7149) @robert3005
• Support reading SparseArray with non zero offset (#7150) @robert3005
• fix: remove nested block_on call in cuda_execute (#7147) @0ax1
• Revert "fix: disambiguate array ops files" (#7138) @joseph-isaacs
• fix: disambiguate array ops files (#7137) @0ax1
• fix[delta]: compress with nulls undef values use last valid value (or 0) (#7124) @joseph-isaacs
• fix fuzzer issue #7074 (#7077) @joseph-isaacs
• fix[back-compat]: use superset not subset in the check mod (#7075) @joseph-isaacs
• fix[scalar_fns]: add correct pre-condition checks for all custom pushdown (#6977) @joseph-isaacs
• Sum aggregate doesn't include nan values (#7009) @robert3005
• Fix scalar partial ordering comparison (#6999) @connortsui20
• fix: compat script (#7024) @joseph-isaacs
• fix[compat]: Update the action to use the correct check command (#7016) @joseph-isaacs
• Add backwards compatibly handling of sum stats of extension types (#7014) @AdamGS
• fix[vortex-array]: handle lazily sliced REE to_arrow (#7011) @asubiotto
• Check for overflow in decimal sum (#7005) @gatesn
• fix[vortex-array]: fix rebuild_trim_elements overflow (#6995) @asubiotto
• fix: Zstd canonicalize with >2GB decompressed buffers (#6989) @SumedhArani
• Sparse followup (#6980) @connortsui20
• fix[sparse]: correctly handle cast for all-patched (#6979) @joseph-isaacs
• fix[duckdb]: build cpp.rs only if changed (#6951) @joseph-isaacs

📖 Documentation

4 changes

• Update vtables docs regarding forwarding from erased to typed (#7134) @gatesn
• docs: clarify GPU dispatch flow (#7132) @0ax1
• More detailed docs on ScalarFnVTable (#7027) @connortsui20
• Add high-level docs to vortex-btrblocks (#6984) @connortsui20

🧰 Maintenance

55 changes

• chore[array]: iterative execute method p1 (#6860) @joseph-isaacs
• fix fuzz_report script (#7141) @a10y
• Cleanup fastlanes encodings crate (#7118) @robert3005
• Testing FFI from C side (#7126) @myrrc
• chore[release]: add compat-gen-upload to release (#7067) @joseph-isaacs
• chore(deps): lock file maintenance (#7121) @renovate[bot]
• chore(deps): update datafusion monorepo to v52.4.0 (#7120) @renovate[bot]
• chore(deps): update all patch updates (#7119) @renovate[bot]
• chore(deps): update taiki-e/cache-cargo-install-action digest to 1ce845d (#7117) @renovate[bot]
• chore(deps): update codspeedhq/action digest to 1c8ae48 (#7116) @renovate[bot]
• Only pass vtable once via ExtDType (#7105) @gatesn
• Function to return vtable from an array (#7085) @gatesn
• chore(deps): lock file maintenance (#7088) @renovate[bot]
• Remove vortex-test-e2e (#7102) @connortsui20
• bump aws-lc-rs to clear advisory (#7100) @onursatici
• chore(deps): update rust crate tar to v0.4.45 [security] (#7099) @renovate[bot]
• chore[mask]: intersect_by_rank benchmarks (#7083) @joseph-isaacs
• chore: cache cpu feat check for micro-benchmarks (#7079) @0ax1
• chore: AggregateFnRef [de]serialize (#7070) @blaginin
• chore[fuzz]: (temp) remove duplicate struct field names (#7073) @joseph-isaacs
• Update CLAUDE.md (#7069) @gatesn
• Print to stdout in compress-bench and random-access-bench (#7065) @connortsui20
• chore(deps): update rust crate oneshot to 0.2.0 (#7042) @renovate[bot]
• chore(deps): update rust crate termtree to v1 (#7058) @renovate[bot]
• chore(deps): update rust crate rand_distr to 0.6 (#7046) @renovate[bot]
• chore(deps): update rust crate pyo3 to 0.28.0 (#7043) @renovate[bot]
• chore(deps): update rust crate zip to v8 (#7059) @renovate[bot]
• chore(deps): update ssciwr/doxygen-install action to v2 (#7060) @renovate[bot]
• chore(deps): update actions/download-artifact action to v8 (#7056) @renovate[bot]
• chore(deps): update aws-actions/configure-aws-credentials action to v6 (#7057) @renovate[bot]
• chore(deps): update actions/checkout action to v6 (#7055) @renovate[bot]
• chore(deps): update rust crate test-with to 0.15 (#7050) @renovate[bot]
• chore(deps): update rust crate strum to 0.28 (#7048) @renovate[bot]
• chore(deps): update rust crate sysinfo to 0.38.0 (#7049) @renovate[bot]
• chore(deps): update rust crate sketches-ddsketch to 0.4.0 (#7047) @renovate[bot]
• chore(deps): update rust crate noodles-vcf to 0.86.0 (#7041) @renovate[bot]
• chore(deps): update plugin com.palantir.consistent-versions to v3.15.0 (#7035) @renovate[bot]
• chore(deps): update rust crate noodles-bgzf to 0.46.0 (#7040) @renovate[bot]
• chore(deps): update rust crate criterion to 0.8 (#7037) @renovate[bot]
• chore(deps): update plugin com.vanniktech.maven.publish to v0.36.0 (#7036) @renovate[bot]
• chore(deps): update plugin com.gradleup.shadow to v9.4.0 (#7034) @renovate[bot]
• chore(deps): update plugin com.diffplug.spotless to v8.4.0 (#7033) @renovate[bot]
• chore(deps): update gradle to v9.4.0 (#7032) @renovate[bot]
• chore(deps): update rust crate fsst-rs to v0.5.8 (#7031) @renovate[bot]
• chore(deps): update rust crate fsst-rs to v0.5.7 (#7030) @renovate[bot]
• chore[ci]: use pre-build image and strip workflow installs (#7013) @joseph-isaacs
• chore: clean up #[inline] annotations (#7000) @0ax1
• chore[compat]: more fixtures compressed (#6986) @joseph-isaacs
• chore(deps): update plugin com.palantir.git-version to v5 (#6949) @renovate[bot]
• add more encoding fixtures (#6976) @joseph-isaacs
• runtime: Instrument with tracing (#6981) @brancz
• Remove git hooks example scripts (#6982) @connortsui20
• Ignore justfile (#6941) @connortsui20
• chore: only ignore c++ diagnostics for ddb headers (#6970) @0ax1
• Fix fuzzer issue template (#6963) @connortsui20