Skip to content

Tag

Polars

4 posts

Polars 2.0 Is Faster. With ArrowMetal on the Apple Silicon GPU, 52 of 107 Queries Go 1.35x to 7.94x Faster Still
Data Engineering10 min read

Polars 2.0 Is Faster. With ArrowMetal on the Apple Silicon GPU, 52 of 107 Queries Go 1.35x to 7.94x Faster Still

Polars 2.0.0 shipped on 6 October and runs its group-bys, joins and sorts faster than 1.44. Two days later ArrowMetal's MetalEngine was re-measured against it on an Apple M4 Max: the GPU default takes 52 of 107 queries at 50 million rows, each 1.35x to 7.94x faster than the faster Polars 2.0 engine, and hands 12 back that it used to take. What Polars 2.0 changed, and how a crossover table answers it.

79 views
Read
Your Apple Silicon GPU Loses to One CPU Core Until a Million Rows. I Measured 111 Operations, Then Rebuilt ArrowMetal 0.2.0 Around the Answer
Data Engineering19 min read

Your Apple Silicon GPU Loses to One CPU Core Until a Million Rows. I Measured 111 Operations, Then Rebuilt ArrowMetal 0.2.0 Around the Answer

Every GPU data library benchmarks itself at 50 million rows. Your dataframe has 80,000. On an Apple M4 Max, summing 1,000 integers takes the GPU 112 microseconds and Polars less than one: the GPU is more than 100 times behind. I built one of these libraries, so I measured the row count where the GPU overtakes the fastest CPU code for 111 operations: the median needs 10,000,000 rows against a multi-core library, about a million against one core, and sixteen never get there. So ArrowMetal 0.2.0 refuses the GPU below the line, byte-identical, and around that router it grew GPU readers for CSV, JSON, nested Parquet, Delta Lake and Iceberg, a Polars engine and a DuckDB optimizer extension.

182 views
Read
Polars vs DuckDB vs ArrowMetal GPU on Apple Silicon: Sort and Group-By Benchmarks
Data Engineering12 min read

Polars vs DuckDB vs ArrowMetal GPU on Apple Silicon: Sort and Group-By Benchmarks

Polars, DuckDB and ArrowMetal on an Apple M4 Max: sort and group-by benchmarks at 10M and 50M rows, wall time next to CPU time, and the rows where the CPU is still ahead.

279 views
Read
Apache Arrow Compute on the Apple Silicon GPU: The First Arrow Project I Created That Does It, With 173 Operations Measured Against Polars, pyarrow and pandas
Data Engineering11 min read

Apache Arrow Compute on the Apple Silicon GPU: The First Arrow Project I Created That Does It, With 173 Operations Measured Against Polars, pyarrow and pandas

I built ArrowMetal, the first Apache Arrow project I could find that runs compute on the Apple silicon GPU. Apple silicon has one memory for CPU and GPU, and an Arrow buffer in shared Metal memory is already a GPU buffer; no Arrow project used that. ArrowMetal does: 307 of Arrow's 307 compute functions, seven languages, take at 24.2x pyarrow on an M4 Max, and 339 benchmark rows against the fastest CPU idiom of Polars, pyarrow, pandas and numpy, including the 77 where the CPU is still ahead.

316 views
Read