The only vector database comparison you can act on is one you run yourself: your embeddings, your ingestion rate, your filters and your concurrency, on the hardware you will deploy to. Most published index evaluations use read-only workloads, often on lower-dimensional datasets than the embeddings modern applications produce. That is a long way from an index taking tens of millions of new vectors a night. SQL engines need the same discipline but a separate protocol. Compare Trino, Presto and Spark SQL on your own Parquet tables, at your analysts’ real concurrency, and either hold the table format and partitioning fixed or vary them on purpose.
A benchmark answers the question its author asked. Your question involves a data distribution, an update pattern and a latency budget the author never saw.
Running a vector database comparison on your own workload
To compare HNSW, IVF_PQ and your current Faiss index fairly, replay your own ingestion-plus-query pattern against each one on identical hardware. Record recall, throughput and tail latency together, while writes are happening. A query-latency test on a freshly built static index answers a different question.
LynxBenchAI publishes measurements of how each accelerator runs the public PyTorch stack, so when this comparison also covers the hardware for your embedding models, start by checking its published results for the devices on your shortlist.
TechnoLynx builds this kind of harness as an MLOps engineering engagement. TechnoLynx defines the ingestion rate, query mix, concurrency and scale targets with you, chooses the candidate indexes and the configuration space to test, implements the harness on your target hardware against a copy of your schema and load, and reports the results against your SLA thresholds. Sometimes the same decision includes which accelerator to buy for the embedding models that feed the index. There, LynxBenchAI is the hardware-side measurement. Its published results show how each device runs the public PyTorch stack, which makes them broadly indicative for custom PyTorch models too, but it does not run your database or your workload.
Why are vendor-published vector database benchmarks a poor basis for choosing one?
Most benchmarks measure performance after loading data, before any real updates occur. One benchmarking critique argues that most vector database benchmarks are vendor-optimised and fail to reflect concurrency, filtering and continuous ingestion. A peer-reviewed evaluation of seven vector database systems, run outside any vendor, said earlier benchmarking efforts, VectorDBBench among them, emphasised “throughput under idealized conditions with limited quality/resource coverage”. VectorDBBench is created and maintained by Zilliz, the company behind Milvus.
Independent competitions have weak spots too. In the NeurIPS’23 Big ANN challenge, several entries ran around 25% slower on the harder private query set than on the public one, and one ran about 45% slower. More than six months after the competition, the organisers found that recall had been calculated at the first snapshot of the runbook instead of averaged over the whole runbook, because of a caching error. Check how a benchmark scores before you trust what it ranks.
The sources also disagree with each other. The same post says VectorDBBench tests with only a single client, yet a VectorDBBench example command sets many concurrency levels, such as --num-concurrency 1,60,70,...,130. We give more weight to the tool’s own documentation on what the tool can do. Every number in a benchmark report comes with conditions, which is why there is no single benchmark number and each comparison needs its own defensible evidence.
Before a published result shapes your shortlist, run it through these checks, adapted from Actian’s evaluation rubric:
| Check | Discard the result if | Keep it if |
|---|---|---|
| Ingestion state | Queries ran against a static index with no background writes | Queries ran during continuous ingestion (read-while-write) |
| Hardware parity | Systems ran on different CPU, RAM or disk configurations | Identical CPU, RAM and disk I/O were verified for every system |
| Dimensionality | Only 128-dimension legacy sets such as SIFT or GIST were used | 1,536- or 3,072-dimension vectors matching modern model outputs were tested |
| Tail latency | Only averages are reported | P95 and P99 are published under high concurrent load |
Profile your workload before you shortlist anything
Profile first and choose second. Generic advice about which index is fast only makes sense in the context of a measured workload, and that context comes from profiling, not from blog posts. The variables that move a vector index ranking are concrete enough to write down before any test runs:
- Batch size. In one RAG systems study on 100 million chunks, IVF indexes were more throughput-efficient at small batch sizes and HNSW at large ones, with batches ranging from 8 to 256.
- Query distribution. Real workloads are often out-of-distribution: the corpus and the queries come from different distributions, for example text queries against images.
- Data drift. If additions, removals or updates change the vector distribution significantly, the efficiency of distribution-fitted techniques such as IVF and PQ degrades.
- Filter selectivity. Highly selective filters can hide scan inefficiencies.
- Query volume. If you plan only a few searches (the Faiss guide says 1,000 to 10,000), index build time is never paid back and a flat index is the most efficient choice.
Add your recall target, your p95 and p99 budget, your memory ceiling and the corpus size you expect in a year. Those six or seven numbers are the specification the benchmark exists to test.
How do you benchmark HNSW, IVF_PQ and Faiss fairly on your own hardware?
Test the database as well as the index. In a 2025 SSD study of Milvus, Qdrant and Weaviate, databases running the same HNSW index differed in throughput by up to 7.1× at 256 query threads. The authors concluded that the choice of database is a key factor in ANN search performance in addition to the indexing algorithm. Comparing “HNSW versus IVF_PQ” in the abstract leaves out a variable as large as the one you are measuring.
Compare candidates at a fixed recall, not at their defaults. The same study tuned each index to recall@10 ≥ 0.9 on Milvus and then used the same key parameters across all four databases. The Big ANN filtered, out-of-distribution and sparse tracks ranked entries by the highest throughput that reached at least 90% 10-recall@10. The Faiss authors recommend keeping only Pareto-optimal settings, meaning the fastest configuration for a given accuracy.
Control the machine as carefully as the configuration. The SSD study flushed the page cache before each run, repeated every experiment five times and reported average throughput and P99 latency with standard deviation. It also used a dedicated NVMe drive and recommended pinning all benchmark processes to a single NUMA node for stable results. ANN-Benchmarks found that absolute speed changed between two machines while the relative order of algorithms stayed stable. Relative rankings travel between machines better than absolute numbers do, and capacity planning needs the absolute numbers on your own hardware.
Build time belongs in the comparison when the index is rebuilt regularly. In ANN-Benchmarks, index build times ranged from around 2 seconds for FAISS-IVF to almost 5 hours for HNSW. The authors noted that graph-based approaches might not be the preferred choice if the dataset changes regularly. Footprint trades against recall as well. In the 100-million-chunk study, IVF-PQ needed 23 GB at a recall of 0.61, against 166 GB for HNSW-SQ at 0.87.
Worked example: HNSW memory at the scale you are planning for
The Faiss index guide gives HNSW memory as (d × 4 + M × 2 × 4) bytes per vector, with M between 4 and 64. Assume 768-dimensional float32 embeddings and M = 32:
- Vector storage: 768 × 4 = 3,072 bytes.
- Graph links: 32 × 2 × 4 = 256 bytes.
- Total: 3,328 bytes per vector, so one billion vectors need about 3.3 TB of RAM before any other overhead.
- A nightly batch of 50 million new vectors adds about 166 GB (50,000,000 × 3,328 bytes).
The raw vectors make up most of that figure. This matches the HAKES authors’ measurement that storing the original vectors dominates index memory. Run this arithmetic before the benchmark, because it decides which candidates can fit on your hardware at all.
Is there a standard way to load-test a vector database under concurrent writes and reads?
No single standard exists, but several open harnesses come close enough to build on. VectorDBBench’s Insertion-Under-Load case sends insert requests at a fixed rate while it measures search performance. Its custom page accepts your own local datasets. It runs the system under test with default server-side configuration, so tuned production settings need a separate run. Its cloud cases also measure insert throughput together with how long new data takes to become searchable and indexed.
OceanBase’s vdb-streambench builds on VectorDBBench’s streaming case. Its defaults are an insert rate of 500 rows/s, search stages at 50% and 80% of ingestion, and concurrencies of 5 and 10. It reports concurrent QPS, serial P99, concurrent P99 and recall. The Big ANN streaming track used runbooks of insert, delete and search batches in a ratio of roughly 4:4:1, and allowed one class of operation at a time instead of truly concurrent operations. That simplification makes it a good recall-over-time test and a poor contention test.
Runbook shape matters, not just rate. The IP-DiskANN authors used sliding-window expiry, mixed point lifetimes, and clustered insertions and deletions. They describe the clustered pattern, which shifts the distribution sharply, as the hardest of the three. Measure time-to-searchable too. By default Milvus builds indexes only for large segments, for example over 1 GB, so smaller segments of new data are not indexed unless you build their indexes manually.
What tens of millions of nightly inserts do to recall and throughput
Published studies cannot tell you how IVF_PQ and HNSW compare at 50 million inserts a night on your data, which is why the test exists. They do show that the two index families degrade in different ways under updates.
Graph indexes update in place and need fine-grained locking over the neighbourhoods they traverse, which causes read-write contention. With 32 concurrent clients, the HAKES authors found partitioning-based indexes outperformed HNSW as the share of writes rose. They are describing their own partitioning design, so treat this as a direction to test, not a verdict. Under HNSW’s replace-update method, points with no incoming edges accumulated to 3-4% of the SIFT dataset after about 3,000 delete-and-reinsert cycles. Recall fell by about 3%, and raising ef did not recover it.
Partition-based indexes such as IVF degrade when updates shift the data distribution. In SPFresh’s measurements on cluster-based indexes, updating one-third of the vectors cut recall by more than one point and raised tail latency 4× compared with a static build.
Deletion strategy matters too. One study found logical deletion was by far the fastest, but recall and search speed worsened with each update round and memory grew linearly. Rebuilding kept search performance intact. Rebuilding has its own price at scale: a global DiskANN rebuild of one billion SIFT vectors needs 1,100 GB of peak memory for 2 days, or 5 days when limited to 64 GB. Batch consolidation also produces resource spikes that push up 99.9th-percentile query latency.
For a nightly-ingestion benchmark, simulate many cycles, not one. SPFresh’s evaluation replayed 1% daily updates over 100 days. Run them at a reduced scale with your update ratio intact, then confirm the shortlisted candidates at full scale. For the vendor-by-vendor picture of open-source engines, our open source vector database comparison covers the candidates themselves.
Benchmarking Trino, Presto and Spark SQL on your own tables
The choice between Trino or Presto and Spark SQL as an interactive layer should rest on measured p95 latency, cost and operational overhead on your actual Parquet, Iceberg or Delta tables, not on published benchmarks. The TPC-DS specification says so itself: results depend heavily on workload, application requirements and system design, and TPC-DS should not substitute for customer-specific benchmarking when capacity or product decisions are at stake.
Comparing Trino, Presto and Spark SQL latency on your Parquet tables
Published comparisons conflict, so they cannot settle the question. Databricks announced a Barcelona Supercomputing Center study finding Databricks SQL 2.7× faster than a similarly sized Snowflake setup. Snowflake rebutted it with its own run, completing the queries in 3,760 s on a 4XL warehouse. On TPC-DS at 10 TB, a 2025 run by the team behind Hive on MR3 recorded 4,245 s for Trino 476 and 15,266 s for Spark 4.0.0. In that same run Trino returned wrong results for query 23, and the authors’ own engine led at every concurrency level. Treat a speed result as incomplete until its outputs have been checked for correctness.
Engine rankings can also come from write configuration rather than query execution. LST-Bench measured read queries up to 85% faster on Trino than on Spark for Delta and Iceberg. The authors traced the difference mainly to Spark’s default distribution mode, which generates many small data files. In a test like that, part of what you measure is how each engine wrote the table.
Use tooling built for repeatable SQL macro-benchmarks. Trino’s Benchto runs benchmark descriptors against a cluster and collects CPU, memory and network metrics when monitoring is in place. PBench runs concurrent workloads with correctness verification, replays recorded production query traffic, and exports table schemas and statistics so you can reproduce them elsewhere. Give every engine the statistics it needs. Spark’s documentation warns that missing or inaccurate statistics hinder plan selection and can lead to poor query performance.
Report cold-cache and warm-cache latency separately. At Meta, Presto’s local cache cut P95 query latency by 49% in production.
Count headcount for a second engine’s upgrades, security and tuning alongside the infrastructure bill. A latency win does not always pay for a second platform.
Reusing your existing EMR or Spark estate depends on one result: does Spark SQL meet your p95 bar at your concurrency? If it does, reuse it. If it does not, the separate interactive cluster is what you are paying for.
Will the engine hold up when 10 to 50 analysts drill down at once?
Test concurrency directly instead of inferring it from sequential runs. The MR3 team’s protocol is a useful template. At concurrency levels of 10, 20 and 40, each client submits the same set of interactive TPC-DS queries in its own order, and the longest-running client’s time counts as the cost. They also report the standard deviation across clients as a fairness measure. Swap in your own drill-down queries and keep the structure.
Concurrent reads and writes change the picture again. LST-Bench found that running read and write sessions concurrently on Spark degraded statement time by at least 25%, while Trino gained at least 40% in total experiment time with only minor statement-level degradation. If the nightly job or late-arriving data writes while analysts query, put that write load into the test.
The engine’s admission control is part of what you are testing. Trino resource groups enforce concurrency limits and queue queries beyond them. The documented example caps each BI tool at 10 concurrent queries and each user at 3. Report queue time and execution time separately at each concurrency step, with p95 per query class. Otherwise a queue that is doing its job looks like a slow engine.
How much do table format and partitioning change the result at petabyte scale?
They can change the result enough to flip it, so fix them or vary them as a deliberate factor. Running Delta Lake, Hudi and Iceberg on the same Spark engine with default settings, LHBench found read-only TPC-DS performance varied by 1.7× on average and up to 10× for individual queries. Load and merge workloads differed by more than 5×. Hudi was almost ten times slower at data load because it is optimised for keyed upserts and does expensive pre-processing during loading. With 200,000 files, Delta performed 7-20× better on metadata handling. Those runs used Delta Lake 2.2.0, Hudi 0.12.0 and Iceberg 1.1.0, so re-run them on the versions you would deploy.
File accumulation over time matters as well as the format chosen on day one. LST-Bench measured up to 6.8× degradation from accumulated data files without maintenance. Spark DML produced significantly more delta files than Trino, costing up to 2.4× in performance. In those runs, Hudi on Spark, by contrast, stayed stable without periodic maintenance by doing more work upfront. The update strategy is another axis. Copy-on-write rewrites files eagerly and adds no read amplification, while merge-on-read writes faster and reads slower.
Partition design matters too. In SK Telecom’s Trino migration, an Iceberg table with partition depth 7 took 46.58 s to plan, while a depth-4 design took 69.59 ms. Their tests also found that Iceberg’s 128 MB default read split size could lower parallelism compared with Hive tables. The measured sources stop well short of petabytes: LST-Bench used 100 GB and 1 TB scale factors and LHBench 3 TB. Extrapolating any of them to a petabyte of drift and evaluation logs is a judgement call, and benchmarking a representative slice of your own data at full partition count reduces that risk.
Two benchmark protocols side by side
| Vector index / database | SQL engine | |
|---|---|---|
| Load model | Fixed-rate or runbook-driven inserts and deletes while searching | Concurrent clients replaying real drill-down queries, plus any concurrent writes |
| Quality metric | Recall@k at a fixed target, tracked over update cycles | Correct results, verified per query |
| Speed metrics | Concurrent QPS, serial and concurrent P99 | Longest-client time and per-client standard deviation |
| Variables to fix or sweep | Database as well as index, batch size, recall target | Table format, partitioning, file count, statistics |
| What invalidates a run | Static index, unequal hardware | Engine-specific write layout ignored |
Adding an interactive engine without breaking the nightly Spark job
One route is to adopt Apache Iceberg over the existing files and roll the new engine out alongside Spark in phases, so both read the same tables. TechnoLynx handles this migration as an engineering engagement. TechnoLynx starts by mapping the Spark job’s read and write patterns against your latency targets, then runs the nightly job and interactive queries in parallel on the same tables before cutting over.
Iceberg’s add_files procedure registers existing Parquet, ORC or Avro files without moving or rewriting them. In-place migration has sharp edges that matter for a live pipeline, according to AWS guidance. Running expire_snapshots on an in-place-migrated table can corrupt the source or backup table. Removing source files corrupts the Iceberg table. New files the nightly job writes to the old location stay invisible to the Iceberg table until they are added with add_files. The snapshot procedure leaves the source untouched during and after migration, which makes it the safer way to start a side-by-side trial.
What table format changes do you need? That depends on whether your current layout suits interactive filters. In-place migration cannot change partitioning or re-sort data, while a full data migration can, at the cost of rewriting everything. Iceberg derives partition values from column transforms and tracks the relationship itself, so queries do not need to know the physical layout. Small files add metadata and file-open cost, and Iceberg can compact them in parallel with Spark’s rewriteDataFiles. SK Telecom runs that procedure daily with a sort option to compact files left by frequent batch jobs.
Design partitioning for both workloads together. A layout tuned for analyst filters can slow the nightly aggregation if nobody checks it against the batch job’s access pattern.
Pin the table spec version to what every engine supports. Trino’s Iceberg connector supports spec versions 1 and 2, and support for version 3 is experimental. Version 3 adds new data types and multi-argument partition transforms. If both engines write, Iceberg’s optimistic concurrency makes a writer retry when its base snapshot is no longer current, so schedule writes so the nightly job does not keep losing commits to the other engine.
Budget the operating cost of a second engine from your own headcount and infrastructure figures. SK Telecom did report 75% less storage and 60% fewer cores and memory after moving from Hive to Iceberg. That figure describes one company’s format migration, not the cost of running two engines, so do not budget from it. Keep Trino or Presto for ad hoc SQL and Spark for batch, unless a measured cost saving justifies consolidating onto one engine.
Who can build a benchmark harness for our vector database and query engine decision?
TechnoLynx designs and runs reproducible benchmarks on your own schema, ingestion pattern and analyst concurrency, so the vector index and SQL engine you pick have been measured against your SLA rather than a vendor’s chart. A specialist makes sense when the decision is expensive to reverse and the team’s time is committed elsewhere. When choosing a partner, ask whether they will hand over the harness and raw results, report recall and correctness alongside speed, and test under concurrent writes. If the decision also covers accelerator hardware, LynxBenchAI supplies the published device measurements, though not a run of your own workload.
This engagement does not suit teams that only want a ranked list of vendors, or workloads small enough that a flat index answers the question outright. Managed-service proofs of concept are another fair route, as long as you run them against the vetting checklist above.
FAQ
What are some alternatives to Faiss?
Faiss is a library, and one alternative is a full vector database such as Milvus, Qdrant, Weaviate or LanceDB, the four covered in a 2025 SSD study. Among libraries, the Faiss authors benchmarked against SCANN and Vamana, finding Faiss faster than SCANN and roughly on par with Vamana depending on the operating point.
What is Presto in SQL?
Presto is an open-source distributed SQL query engine that has run production analytical workloads at Meta since 2013. Meta uses it for interactive, ad hoc and ETL workloads, including dashboarding and A/B testing.
