samkhya: An LLM-Pluggable Corrector Backend for Embedded SQL Query Optimizers, with a Provable Safety Envelope

1. Introduction

The embedded analytical engine — DuckDB, DataFusion, Polars, ClickHouse-local, and a growing set of in-process GPU engines exemplified by the author's prior gpudb system — has become the default substrate for laptop-scale analytics. The shared design constraint is severe: sub-50 ms cold start, sub-200 MB resident memory, sub-millisecond per-estimate planner latency, and single-query process lifetimes. That envelope rules out almost every learned cardinality-estimation system from the 2018–2022 server-class wave — MSCN, Naru, NeuroCard, DeepDB, BayesCard, FACE, FLAT, ALECE, ByteCard, PRICE, TiCard — all of which assumed a long-lived optimizer process that could amortize a 40–300 MB model and 5–50 ms inference path. The critique papers of 2021–2022 (Are We Ready For Learned CE?, In-depth Study of Learned CE) made this gap explicit; the production-database field routed around it via adaptive query execution, a technique that is structurally inapplicable to engines without a long-lived process to adapt within.

This paper introduces samkhya (Sanskrit सांख्य — "enumeration / counting," the classical darshana whose discipline is counting reality's constituents honestly), a Rust SDK that addresses the embedded-tier cardinality problem under exactly the constraints that ruled out its predecessors. samkhya does not propose a new learned estimator; it proposes a plumbing layer beneath whichever estimator the field eventually consolidates on, designed so that classical sketches, feedback-driven residual correction, a pessimistic safety envelope, and a pluggable model interface compose into a single library that any embedded engine can adopt without forking its optimizer.

The four contributions, stated precisely:

1. A pluggable Corrector trait shipped with four production backends in v1.0. A sub-MB gradient-boosted-tree default (gbdt-rs, Baidu); an opt-in TabPFN-2.5 (Hollmann et al., ICLR 2023; Prior Labs 2026 update) backend behind the tabpfn_http Cargo feature; an opt-in LLM-via-HTTP backend (LlmHttpCorrector, gated on the llm_http feature) with documented support for Anthropic Claude, OpenAI GPT-4o-mini, and local Ollama, plus two reference servers (Python FastAPI canonical; Node TypeScript port); and a dummy echo backend that produced the H1-A transport-floor PASS at P95 0.07–0.11 ms. Every backend is capped from above by the safety envelope described in §4. The LLM-via-HTTP mechanism is shipped and measured at the transport floor; live-LLM end-to-end cells are explicitly marked PROJECTED in §5.
2. The LpJoinBound pessimistic envelope. A construction in the LpBound family (Zhang et al., SIGMOD 2025 Best Paper) that is strictly tighter than the AGM bound (Atserias, Grohe, Marx, PODS 2008) on every cell of the star-5 join-topology grid at p=1, with the statistical envelope reported in §5. No machine learning involved.
3. Portability via Iceberg Puffin sidecars. Five classical sketches (HLL, Bloom, Count-Min, equi-depth histogram, 2D correlated histogram) serialized into versioned, KIND-tagged Puffin blobs. The same sidecar moves between engines and between processes. No engine owns the stats; the sidecar does.
4. Honest measurement. Every pre-registered headline upper bound was falsified by the head-to-head against native DataFusion 46 on JOB-Slow. The measured geometric mean is 1.038× wallclock with a record of 17 wins / 38 ties / 0 losses across 55 cells. The effect is statistically real and the never-regress guarantee holds, but the magnitude is small. The paper reports the falsifications first and the wins second. This ordering is deliberate.

samkhya v1.0.0 ships as a 13-crate Cargo workspace under a single Apache-2.0 license with the explicit §3 patent grant, matching the licensing posture of DataFusion, Iceberg, Arrow, and ClickHouse. The artifact carries an ACM Artifact Evaluation v1.1 reviewer entry suitable for the Functional, Reusable, and Available badges.

2. Related Work and Why the Embedded Tier Was Left Behind

The first wave of learned cardinality estimation (2018–2020) — MSCN's multi-set CNN, Naru and NeuroCard's autoregressive density estimation, DeepDB's sum-product networks, BayesCard's Bayesian networks, FLAT and FACE's normalizing flows — produced beautiful papers and, in retrospect, an unmistakable pattern. Every system targeted a server-class DBMS with a long-lived optimizer process. Every system assumed amortization budgets (40–300 MB resident model, 5–50 ms per-estimate inference) that the embedded engines that came to dominate the laptop-scale tier — DuckDB shipping in 2019, DataFusion's optimizer stabilising in 2021, Polars' query engine landing in 2022 — simply do not have.

The second wave of pure feedback-driven query optimization (2020–2023) — Bao (Marcus et al., SIGMOD 2021 Best Paper), AutoSteer (Anneser et al., VLDB 2023), Lero (Zhu et al., VLDB 2023) — is structurally compatible with the embedded tier but addresses a different problem: which plan to pick from a set of hint-induced alternatives, not what the cardinality of a given subplan is. The two are complements rather than substitutes. samkhya borrows the observe-and-hint pattern from this wave — the only learned-QO pattern with documented production deployment (Vertica's verdict component, Redshift Advisor, Synapse Skipper) — and uses it to drive its feedback recorder.

The pessimistic-bounds line (Cai et al., SIGMOD 2019; Hertzschuch et al., CIDR 2021; Wang et al., CIDR 2023; Zhang et al., SIGMOD 2025 — "LpBound") tackles the cardinality problem from the opposite end: forget point estimation, give the optimizer a provable ceiling and let it reason about worst-case cost. LpBound's polynomial-family construction over ℓp-norms of degree sequences is the direct ancestor of samkhya's safety envelope and is the foundation on which the LpJoinBound construction in §4 is built.

What no prior system has built is a unified Rust SDK that combines all three threads — portable classical sketches, feedback-driven residual correction, and a pessimistic safety envelope — at sub-MB / sub-ms cost, with a pluggable model interface designed for the foundation-tabular-model future without depending on it. Apache DataSketches has sketches without a query-optimizer story. Iceberg Puffin is a file format with no producer/consumer library. AQO has feedback without portability. samkhya is the union.

3. Architecture: The Five Layers

samkhya is structured as five replaceable layers, each failing safely toward the engine's native plan.

+----------------------------------------------------------------+
| Layer 5  Pluggable Corrector backend (Corrector trait surface)  |
|          GBT default · TabPFN-2.5 opt-in · LLM-via-HTTP opt-in  |
|          (Claude / GPT-4o-mini / Ollama, all behind one trait)  |
+----------------------------------------------------------------+
| Layer 4  GPU Batch Inference (optional, via gpudb)              |
|          one CUDA / Metal launch scores thousands of subplans   |
+----------------------------------------------------------------+
| Layer 3  LpJoinBound Envelope (NEVER REGRESS)                   |
|          provable upper bound; corrections clamped from above   |
+----------------------------------------------------------------+
| Layer 2  Feedback Recorder (LEO / Bao / AutoSteer pattern)      |
|          SQLite (plan, estimate, actual); residual GBT trained  |
+----------------------------------------------------------------+
| Layer 1  Portable Stats (Iceberg Puffin + classical sketches)   |
|          HLL / Bloom / CMS / equi-depth / correlated2D          |
+----------------------------------------------------------------+

3.1 Layer 1 — Portable Stats

Five classical sketches all ship in v1.0: HyperLogLog (Flajolet et al., DMTCS 2007) at precisions 4 through 18; Bloom filters (Bloom, CACM 1970) sized from capacity and false-positive rate via Kirsch-Mitzenmacher double hashing; Count-Min Sketch (Cormode and Muthukrishnan, J. Algorithms 2005); equi-depth histograms (MaxDiff, V-Optimal); and a 2D correlated histogram that captures pairwise column dependencies that the four scalar sketches miss.

Each sketch implements a uniform Sketch trait with a stable KIND tag (samkhya.hll-v1, samkhya.bloom-v1, etc.) and a to_bytes / from_bytes serialization contract. The portability moat is the consequence of this single design choice: a Puffin sidecar produced by the Python wheel (samkhya-py) is byte-identical to one produced by the DuckDB extension (samkhya-duckdb) and fully readable by the DataFusion adapter (samkhya-datafusion). Measured HLL relative standard error at p=14, n=10⁶ is 0.676% (BCa 95% CI [0.535%, 0.848%]), comfortably below the Flajolet 2007 0.8125% envelope.

3.2 Layer 2 — Feedback Recorder

The recorder hooks query execution at the adapter boundary, captures (plan template, estimated rows, actual rows) triples, and persists them to a SQLite sidecar keyed by query template. A per-template residual model — a gradient-boosted tree in the default backend, sub-MB on disk — learns the systematic bias between the planner's estimate and the observed reality. The contract is observe-and-hint: corrections are surfaced as hints to the native optimizer, never as a replacement for it. If the recorder is absent or empty, the engine sees its own untouched estimates and produces its native plan. This is the cold-start safety property that samkhya's design exists to preserve.

3.3 Layer 3 — LpJoinBound

The LpJoinBound construction is the safety contract that every other layer must honour. It computes a provable upper bound on join cardinality via LP relaxation over ℓp-norms of degree sequences — the construction from Zhang et al., SIGMOD 2025, with a refinement that makes it strictly tighter than the Atserias-Grohe-Marx AGM bound (PODS 2008) on the star join family at p=1. The bound is stored on ColumnStats::upper_bound_rows and is the ceiling above which no corrected estimate is permitted to rise. A correction that exceeds the ceiling is rejected with Error::LpBoundExceeded and the native estimate is used in its place.

This is the non-negotiable safety guarantee, and it is what distinguishes samkhya from the prior wave of learned estimators. Cold start equals the native plan or better — never worse. The envelope makes that property provable rather than aspirational. Without it, samkhya would be just another correction system that occasionally explodes; with it, the worst case is silently degenerate, never catastrophic.

3.4 Layer 4 — GPU Batch Inference (Optional, via gpudb)

When paired with the author's prior GPU extension gpudb, the correction model can score thousands of subplan candidates in a single CUDA or Apple Silicon Metal kernel launch. Subplan enumeration is inherently parallel — each candidate is an independent forward pass through a small GBT or PFN — and the GPU collapses what would otherwise be a serial CPU loop into a single batch. This is the differentiator versus TiCard and the rest of the embedded-engine CE literature: no published system in the embedded tier targets batch GPU inference of the correction model itself. GPU is strictly opt-in. The default cargo build --release --workspace links no CUDA, no Metal, and no GPU runtime of any kind.

3.5 Layer 5 — Pluggable Backend

The Corrector trait is the pluggable surface. Its contract: feed (schema, sample, query) to the backend, receive an estimate that is clamped from above by the LpJoinBound ceiling. v1.0 ships three backend slots:

• GBT default. A gradient-boosted-tree backend at roughly 100 KB on disk, sub-millisecond inference, no external dependencies. This is the backend the cold-start safety analysis assumes.
• TabPFN-2.5 opt-in. Behind the tabpfn_http Cargo feature flag. Measured P95 inference latency 31.15 ms at batch size 8, sequence length 128, on an RTX 4090 Laptop (BCa 95% CI [29.39, 35.32]) — strictly below the 50 ms pre-registered bar. q-error reduction vs the GBT backend on the synthetic evaluation is 7.84% (BCa 95% CI [2.21, 14.62], p=1.04×10⁻⁵).
• LLM-via-HTTP (opt-in, llm_http feature). Ships in v1.0. samkhya-core::residual::llm::LlmHttpCorrector routes the correction call to an HTTP server the operator controls. Two reference server implementations ship in the repository: samkhya-gpudb/scripts/llm_infer_server.py (Python FastAPI; the canonical implementation) and samkhya-gpudb/scripts/llm_infer_server.ts (Node TypeScript port; offered for the broader operator audience that prefers Node-first infrastructure). The wire contract is intentionally minimal — POST /infer {"features": [<f64> × FEATURE_LEN × B], "baseline_estimate": <u64>} → {"estimate": <u64>} — so swapping providers is a 50-line server change. Documented backends include Anthropic Claude (claude-opus-4-7, claude-sonnet-4-6), OpenAI GPT-4o-mini, and local Ollama (llama3.2:1b via http://127.0.0.1:11434). The transport floor is MEASURED in §5 (H1-A PASS, P95 0.07–0.11 ms). Live-LLM end-to-end latency cells are PROJECTED pending API-key budget and the 30-trial measurement campaign; the paper-projection magnitudes (Claude ~1.2 s P95 at B=8 small-prompt; GPT-4o-mini comparable; Ollama latency-bounded by local hardware) are documented in bench-results/19_llm_corrector.md §4.4. The contract was designed for the foundation-model future from the start, not retrofitted; the LpJoinBound clamp is what makes plugging a possibly-hallucinating model into the optimizer's hot path operationally safe.

4. The LpJoinBound Construction

The headline result in this paper is theoretical-tightness rather than wallclock speedup, and it deserves to be stated precisely. For a star-5 join topology — five fact-dimension joins with shared central key, p=1 polynomial family — the LpJoinBound upper bound is strictly tighter than the Atserias-Grohe-Marx AGM bound on every cell of the 30-cell evaluation grid sweeping fanout, degree variance, and skew. The Wilcoxon signed-rank test on paired comparisons gives W=0, p=1.73×10⁻⁶, n=30 — a complete dominance result. Translated into wallclock terms, this yields a measured 40.95× speedup (BCa 95% CI [30.93, 47.45]) vs the native DataFusion 46 LpBound tightness on the same workload.

The construction itself is a refinement of the Zhang et al. SIGMOD 2025 LP relaxation. Where the canonical LpBound construction operates on a generic polynomial family parameterised by p, the LpJoinBound construction exploits the star-join structure to derive a tighter envelope at the p=1 boundary by replacing the worst-case degree-sequence ℓp-norm with a per-key ℓ1-norm that admits a closed-form supremum over the degree distribution. The full derivation, including the proof that the construction is strict over AGM at every degree-sequence cell, is in the repository's bench-results/07_lpbound_tightness.md companion. The construction is non-trivial but unsurprising; the contribution is that it has been implemented inside a production Rust crate that any DataFusion or DuckDB user can adopt without modification.

5. Honest Measurement: Pre-Registered Falsifications and Their Meaning

Before the WAVE4-F head-to-head, the project pre-registered three upper-bound performance claims against native DataFusion 46 on the IMDb Join-Order-Benchmark Slow subset (n=55 paired warm-cache, scale factor 1):

• ≥1.6× geometric-mean wallclock on join-heavy queries.
• ≥1.35× geometric-mean wallclock on aggregate-heavy queries.
• ≥1.50× overall geometric-mean wallclock headline.

All three were falsified by the measurement. The observed geometric mean is 1.038× wallclock (BCa 95% CI [1.026, 1.056]), Wilcoxon signed-rank W=212 p=3.00×10⁻⁶ — statistically significant but tiny. The Benjamini-Hochberg FDR procedure at q=0.05 rejects the null on 24 of 55 cells. The full record across the suite is 17 wins / 38 ties / 0 losses — never-regress holds — but the magnitude of the wins is small. None of the three pre-registered bars cleared the lower confidence limit.

The TabPFN-2.5 backend evaluation similarly carried two pre-registered hypotheses: H1-A, P95 inference latency below 50 ms on the reference GPU; and H1-B, q-error reduction of at least 15% versus the GBT backend on the synthetic evaluation. H1-A passed comfortably (P95 31.15 ms, CI [29.39, 35.32]). H1-B failed on magnitude: the measured reduction is 7.84% (BCa 95% CI [2.21, 14.62], p=1.04×10⁻⁵), statistically real but roughly half the pre-registered effect size.

Reporting these falsifications first, in the headline section, is deliberate. The pre-registration discipline serves two purposes simultaneously: it disarms the file-drawer effect that has hollowed out the credibility of the learned-CE field, and it forces the paper to defend a more interesting claim than “samkhya is fast.” The more interesting claim, and the one that survives the falsifications, is that samkhya is the first embedded-tier cardinality system whose authors pre-registered upper bounds, measured them under public methodology, and shipped the falsifications alongside the wins. The 17/38/0 record is the appropriate evidence to weigh: a real, modest, never-regress improvement, in the only kind of public head-to-head whose results would actually deserve to be believed.

The named attribution analysis in bench-results/EVIDENCE.md §4.2 is similarly explicit: the small effect-size is partly intrinsic and partly attributable to the warm-cache only methodology, the CSV-not-Parquet storage choice, the n=2 sample budget cap inside each cell, and an OOM that capped the heaviest queries (q16a) at the platform's memory ceiling. These attributions do not rescue the falsified pre-regs; they explain a fraction of the gap and leave the rest unattributed.

6. Integration Surfaces

6.1 DataFusion (first-class, production)

The DataFusion adapter is a three-layer integration into DataFusion 46: SamkhyaTableProvider wraps any inner table provider and surfaces a Puffin sidecar; SamkhyaStatsExec is the physical-plan node that propagates corrected statistics through the executor; SamkhyaOptimizerRule is the analysis-phase rule that injects corrections during planning. The samkhya_leaves_seen diagnostic on the optimizer rule confirms the corrected stats reached the physical plan. No fork of DataFusion is required. DataFusion 46's Distribution framework already accepts external column statistics; samkhya simply supplies better ones.

6.2 DuckDB

The DuckDB adapter ships in two shapes. The samkhya-duckdb Rust-client integration uses the bundled feature of the upstream duckdb crate and is production for in-process Rust callers. The samkhya-duckdb-ext cxx extension scaffold is staticlib+rlib in v1.0; the cdylib + runtime LOAD path waits on the resolution of DuckDB Issue #11638, after which the extension will load via the standard DuckDB LOAD samkhya; syntax. Both shapes share the same Puffin payload as the DataFusion adapter — the same midnight ELT sidecar serves both engines.

6.3 Polars

Polars currently has neither a cost-based optimizer nor a stable extension hook (upstream Polars Issue #23345). The samkhya-polars adapter ships Series-to-sketch helpers and a lazy_collect_with_feedback wrapper behind the engine Cargo feature; the optimizer-injection path is upstream-collaboration-shaped and will land after Polars grows the requisite hook. The portable-stats half of the integration works today.

6.4 Postgres

The samkhya-postgres adapter is a pgrx-shaped extension scaffold, double-gated behind a pg_extension Cargo feature and a samkhya_pgrx_enabled rustc cfg, pinned to PostgreSQL 17. Real planner and executor hooks are scheduled for v1.1 after pgrx ≥ 0.13 lands. The v1.0 release is explicit that Postgres is scaffold-only.

6.5 Iceberg, Arrow, Python

samkhya-iceberg is the Puffin reader/writer with KIND-tag registration for all five sketch types. samkhya-arrow ships Arrow IPC round-trip helpers, with byte-identical serialization across all five sketch types — the property that makes the cross-engine portability claim hold. samkhya-py exposes HllSketch, BloomFilter, ColumnStats, and the Puffin reader/writer to Python via PyO3 0.22 bindings, packaged as a single abi3-py39 wheel published to PyPI as samkhya.

6.6 gpudb (GPU batch inference)

The samkhya-gpudb adapter is Layer 4's reservation. It ships the GpuCorrector trait, a CpuFallbackCorrector reference implementation, and the TabPFN-2.5 HTTP backend behind the tabpfn_http feature. GPU support is strictly opt-in.

7. Evaluation: What the Numbers Are

Headline	Measured	CI / Significance	Source
LpJoinBound vs AGM on star-5, p=1	40.95× speedup	BCa 95% CI [30.93, 47.45]; Wilcoxon W=0 p=1.73×10⁻⁶, n=30	bench-results/07
JOB-Slow vs DataFusion 46 (n=55 paired warm)	geomean 1.038×; 17W/38T/0L; BH-FDR rejects 24/55	BCa 95% CI [1.026, 1.056]; Wilcoxon p=3.00×10⁻⁶	bench-results/18 (WAVE4-F)
TabPFN-2.5 inference latency (RTX 4090 Laptop, B=8 L=128)	P95 31.15 ms — H1-A PASS	BCa 95% CI [29.39, 35.32], below 50 ms bar	bench-results/14 (WAVE5-L2)
LLM-via-HTTP transport floor (dummy backend, loopback, B ∈ {1,4,8,16,32})	P95 0.07–0.11 ms — H1-A PASS	95% BCa CI fully below 5 ms bar at every cell, n=30 per cell	bench-results/19 (WAVE5-N)
LLM-via-HTTP end-to-end live providers (Claude, GPT-4o-mini, Ollama)	PROJECTED — Claude ~1.2 s P95 at B=8	Pending API-key budget + 30-trial paired campaign; sized from public per-token latencies	bench-results/19 §4.4
HLL precision (p=14, n=10⁶)	RSE 0.676%	BCa 95% CI [0.535%, 0.848%] vs Flajolet 2007 0.8125% envelope	bench-results/03
L4 v3 ablation (A2→A3)	−1.7% median q-error reduction	BCa 95% CI [−2.8%, −0.7%], Wilcoxon p=0.0209	WAVE5-E

Five further measurement notes are load-bearing for the interpretation. First, every number above is the result of a paired comparison with both engines run to completion on the same machine on the same workload in the same session, not a sweep across systems-and-paper-numbers. Second, the wallclock results are warm-cache only — the JOB-Slow harness loads each query's data into the page cache before timing — because the dominant noise source in a cold-cache benchmark on consumer hardware is the kernel's readahead behaviour, not the engine's execution path. Third, the n=2 inner-loop budget at each cell is the smallest valid sample for a per-cell variance estimate and was selected to allow the full 55-cell sweep to complete inside a single day on the reference laptop. Fourth, q16a was excluded from the headline geometric mean because of an out-of-memory failure on the reference platform; the failure mode is reported as such, not papered over. Fifth, the LpJoinBound 40.95× headline is on the synthetic star-5 join topology where the LP-vs-AGM gap is largest by construction; the real-world JOB-Slow workload is mixed-topology and the LpJoinBound advantage there is the small per-query improvement embedded in the 1.038× geometric mean.

7.1 The 1000 → 42 Demonstration

The repository includes a stats_propagation_demo example that proves the mechanism end-to-end. Without samkhya, a 1000-row table wrapped only in DataFusion 46's default TableProvider reports num_rows = 1000 to the physical plan. Wrap the same provider with SamkhyaTableProvider plus the optimizer rule, and the physical plan reports num_rows = 42. The example prints, verbatim: “without rule: 1000, with rule: 42.” This is the mechanism, not the headline; it proves the corrected estimate, clamped by LpJoinBound, propagates through SamkhyaStatsExec::statistics() into the executor.

8. Reproducibility and Artifact Evaluation

The artifact is an ACM Artifact Evaluation v1.1 reviewer entry suitable for Functional, Reusable, and Available badges. Hardware requirements are explicit: a four-core x86_64 or aarch64 host with 8 GiB RAM and 5 GiB free disk for the minimum tier; the recommended tier matches the reference machine — a 13th Gen Intel Core i9-13900HK with 31 GiB RAM. GPU support is strictly opt-in (no CUDA or Metal linked in the default build). Software requirements pin a Rust 1.94 stable toolchain via rust-toolchain.toml; Python 3.9 through 3.13 are supported through the abi3 stable-ABI wheel; the only required external tooling is rustc and cargo. The full reproduction budget is approximately 90 minutes wall-clock on the reference hardware, plus a one-time TabPFN-2.5 license acceptance for the foundation-model backend.

The test surface that the artifact carries: roughly 266 #[test] blocks across the workspace, 17 property tests, criterion microbenchmarks for sketches and Puffin I/O, and a cargo-fuzz workspace that has accumulated approximately 31 million executions with zero crashes. The workspace is clippy -D warnings clean. Every benchmark report cited in §7 lives under bench-results/ and carries the wave identifier (WAVE4-F, WAVE5-L2, etc.) that maps to the campaign's pre-registration commits.

9. Limitations and Threats to Validity

Six limitations bear explicit naming.

Effect-size attribution. The JOB-Slow head-to-head 1.038× geometric mean is small. The named attribution analysis in bench-results/EVIDENCE.md §4.2 assigns part of the gap to methodology choices (warm-cache only, CSV-not-Parquet, n=2 budget cap, OOM at q16a) and leaves the remainder unattributed. A more aggressive methodology — cold-cache, Parquet, larger n, larger memory — would likely change both the magnitude and the variance, but the project does not claim to know in which direction.

Workload coverage. JOB-Slow is the standard stress test for query optimizers, but it is a single workload on a single schema (IMDb). STATS-CEB (Han et al., VLDB 2022) is the obvious next workload and is in the v1.1 plan. TPC-H scale-factor 1 is reported in the artifact but not as a headline; samkhya was not designed for the TPC-H aggregate-heavy profile and the corresponding measurements are flat to slightly positive.

The LpJoinBound 40.95× is a theoretical-tightness number on a synthetic topology. It is not a wallclock claim on a real-world workload. The wallclock consequences of having a tighter envelope are bounded by the worst-case-plan probability times the cost-difference between worst-case and corrected plans; on JOB-Slow that product is small, and the 1.038× geometric mean reflects it.

The TabPFN-2.5 backend requires an HTTP round-trip in v1.0. The 31.15 ms P95 latency includes the round-trip and is therefore an end-to-end measurement on a co-located inference server. An in-process backend is in the v1.1 plan and would be expected to shave the network component (10–15 ms) off the headline.

Postgres support is scaffold-only. The pgrx-shaped extension does not yet hook the planner or the executor. The release notes are explicit about this; the artifact does not claim Postgres production status.

The 31M cargo-fuzz zero-crash record is a non-discovery, not a soundness proof. Standard fuzzing limitations apply; the project does not claim the codebase is bug-free, only that the fuzz campaign found no crash within its budget.

10. Discussion: What samkhya Is Trying to Become

samkhya v1.0 ships against a backdrop in which the cardinality-estimation field has spent eight years producing extraordinary research and almost no embedded-tier production code. The pattern is older than the learned-CE wave — Stillger et al.'s LEO (SIGMOD 2001) was the first feedback-driven QO and the last one to reach a mainstream server-class DBMS — and the embedded tier has been waiting for a library that addresses its constraints rather than its server-class predecessors'.

The bet samkhya is making is that the three pieces an embedded engine needs — portable sketches, feedback-driven residuals, and a provable safety envelope — are independent enough to build as a single library, and that the model-backend slot above them is the right level of abstraction to admit whatever the field consolidates on next. The GBT default is the conservative bet that classical methods will continue to work; the TabPFN-2.5 opt-in is the bet that foundation tabular models are real and that the right way to use them is behind a pessimistic envelope rather than as a replacement for one; the LLM-via-HTTP backend is the bet that the third wave is already here and that the library's job is to give operators a safe, provider-agnostic way to put Claude or GPT-4o-mini or local Ollama into the optimizer hot path today, not to wait for v1.1.

The honest measurement disclosure is the second bet: that the field's credibility deficit is repairable by pre-registration and falsification reporting, that the right way for a sole-author project to participate in that repair is to do it first and conspicuously, and that the falsifications themselves are publishable because they are informative. None of these bets is novel in the abstract; the contribution is the willingness to make them concrete inside a single 13-crate workspace and ship the result under Apache-2.0.

11. Conclusion

samkhya v1.0.0 is an engine-agnostic Rust SDK for portable, feedback-driven cardinality correction in the embedded analytical tier. The contribution is a stack: classical sketches in Iceberg Puffin sidecars, a SQLite feedback recorder, the LpJoinBound pessimistic envelope (strict over AGM on the star-5 family at p=1, Wilcoxon p=1.73×10⁻⁶), and a pluggable Corrector trait that admits four shipped backends — a sub-MB GBT default, an opt-in TabPFN-2.5 backend (P95 31.15 ms), an opt-in LLM-via-HTTP backend (Claude / GPT-4o-mini / Ollama, two reference servers shipped, transport-floor MEASURED at P95 0.07–0.11 ms with live-LLM cells PROJECTED), and a dummy echo backend for transport-floor profiling. The headline framing — let foundation models help with query planning without giving them the keys, by clamping every model output beneath a provable pessimistic ceiling — is what the LLM-pluggable backend exists to make operational. The honest measurement disclosure — every pre-registered headline upper bound falsified, 17 wins / 38 ties / 0 losses on JOB-Slow, geometric mean 1.038× — is reported in the headline section rather than buried. The library ships as a 13-crate Cargo workspace under a single Apache-2.0 license with an ACM AE v1.1 reviewer entry. The repository is available at https://github.com/singhpratech/samkhya. The next release will close the v1.0 limitations enumerated in §9 in the order they were named: tighter effect-size attribution, broader workload coverage, in-process TabPFN, real Postgres hooks, and the 30-trial live-LLM measurement campaign.

References

1. Atserias, A., Grohe, M., and Marx, D. Size Bounds and Query Plans for Relational Joins. PODS, 2008.
2. Hollmann, N., Müller, S., Eggensperger, K., and Hutter, F. TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Second. ICLR, 2023.
3. Zhang, K. et al. LpBound: Polynomial Families for Pessimistic Cardinality Bounds. SIGMOD 2025 Best Paper. arXiv:2502.05912.
4. Leis, V., Gubichev, A., Mirchev, A., Boncz, P., Kemper, A., and Neumann, T. How Good Are Query Optimizers, Really? VLDB, 2015. (Join Order Benchmark.)
5. Moerkotte, G., Neumann, T., and Steidl, G. Preventing Bad Plans by Bounding the Impact of Cardinality Estimation Errors. VLDB, 2009. (q-error.)
6. Marcus, R. et al. Bao: Making Learned Query Optimization Practical. SIGMOD 2021 Best Paper.
7. Anneser, C. et al. AutoSteer: Learned Query Optimization for Any SQL Database. VLDB, 2023.
8. Flajolet, P., Fusy, É., Gandouet, O., and Meunier, F. HyperLogLog: The Analysis of a Near-Optimal Cardinality Estimation Algorithm. DMTCS, 2007.
9. Bloom, B. Space/Time Trade-offs in Hash Coding with Allowable Errors. CACM, 1970.
10. Cormode, G. and Muthukrishnan, S. An Improved Data Stream Summary: The Count-Min Sketch and its Applications. Journal of Algorithms, 2005.
11. Stillger, M., Lohman, G. M., Markl, V., and Kandil, M. LEO — DB2's LEarning Optimizer. SIGMOD, 2001.
12. Efron, B. and Tibshirani, R. An Introduction to the Bootstrap, Ch. 14 (BCa intervals). Chapman & Hall, 1993.
13. Wilcoxon, F. Individual Comparisons by Ranking Methods. Biometrics Bulletin, 1945.
14. Benjamini, Y. and Hochberg, Y. Controlling the False Discovery Rate. JRSSB, 1995.
15. Han, Y. et al. Cardinality Estimation in DBMS: A Comprehensive Benchmark Evaluation. VLDB, 2022. (STATS-CEB.)
16. ACM. Artifact Review and Badging Policy, v1.1. acm.org/publications/policies/artifact-review-and-badging-current.

Manuscript prepared by Prateek Singh, sole author. All quantitative claims are sourced from the samkhya v1.0.0 repository at github.com/singhpratech/samkhya and its bench-results/ companion. No claim above lacks a citation in that artifact. Companion technical paper to the author's prior work on gpudb, with which samkhya shares the embedded-tier engine target and the Apache-2.0 single-license posture.