# LAION-100M benchmark artifacts

This directory contains VectorDBBench's LAION-100M dataset: 100,000,000 base vectors with 768-dimensional floating-point embeddings and L2 distance. The dataset definition is in [VectorDBBench](https://github.com/zilliztech/VectorDBBench/blob/main/vectordb_bench/backend/dataset.py#L145-L155).

## Files

| File or pattern | Purpose |
|---|---|
| `train-00-of-100.parquet` … `train-99-of-100.parquet` | Base dataset, one million rows per shard; schema: `id: int64`, `emb: list<float>` (768 values). |
| `test.parquet` | Original 1,000 query vectors; schema: `id: int64`, `emb: list<float>`. |
| `neighbors.parquet` | Original top-1,000 ground-truth neighbor IDs for every query in `test.parquet`. |
| `test_nq200.parquet` | First 200 rows of `test.parquet`, used for the large-top-K benchmark. |
| `neighbors_top100k_nq200.parquet` | Exact top-100,000 ground-truth neighbor IDs for each query in `test_nq200.parquet`. |
| `neighbors_top1m_nq200.parquet` | Exact top-1,000,000 ground-truth neighbor IDs for each query in `test_nq200.parquet`. |
| `generate_laion100m_gt.py` | Reproducible four-GPU streaming generator for the unfiltered and integer-filtered large-top-K ground truth; supports checkpointed resume and independent verification. |
| `scalar_labels.parquet` | One scalar string label per base-vector ID for label-equality filter benchmarks. |
| `scalar_labels_2kb.parquet` | The same label assignments padded to roughly 2 KiB per string, for large scalar-payload benchmarks. |
| `neighbors_int_{50p,60p,70p,80p,90p,95p,98p,99p,99.5p,99.8p,99.9p}.parquet` | Exact filtered ground truth for the original queries under `id >= dataset_size * percentage`. |
| `neighbors_int_<rate>_top<width>_nq200.parquet` | Exact wide integer-filter ground truth for `test_nq200.parquet`; the rate-to-width inventory is below. |
| `neighbors_labels_label_{0.1p,0.2p,0.5p,1p,2p,5p,10p,20p,50p}.parquet` | Exact filtered ground truth for the original queries under equality with the named label. |
| `README.md` | This inventory and integrity information. |

Filter naming and expressions follow [VectorDBBench's filter definitions](https://github.com/zilliztech/VectorDBBench/blob/main/vectordb_bench/backend/filter.py#L34-L93). The wide ground truth supports [issue #826](https://github.com/zilliztech/VectorDBBench/issues/826).

## Wide integer-filter ground truth

Each filter retains base vectors satisfying `id >= threshold`. A top-100K artifact is available for every rate; the widest artifact is capped by the eligible population.

| Filter rate | Threshold | Eligible vectors | Published widths |
|---:|---:|---:|---:|
| 50% | 50,000,000 | 50,000,000 | 100K, 1M |
| 60% | 60,000,000 | 40,000,000 | 100K, 1M |
| 70% | 70,000,000 | 30,000,000 | 100K, 1M |
| 80% | 80,000,000 | 20,000,000 | 100K, 1M |
| 90% | 90,000,000 | 10,000,000 | 100K, 1M |
| 95% | 95,000,000 | 5,000,000 | 100K, 1M |
| 98% | 98,000,000 | 2,000,000 | 100K, 1M |
| 99% | 99,000,000 | 1,000,000 | 100K, 1M |
| 99.5% | 99,500,000 | 500,000 | 100K, 500K |
| 99.8% | 99,800,000 | 200,000 | 100K, 200K |
| 99.9% | 99,900,000 | 100,000 | 100K |

## Large-top-K artifact integrity

| File | Bytes | SHA-256 |
|---|---:|---|
| `test_nq200.parquet` | 337,550 | `80376447e111f0e304fb67ecb233603996ba0583c1c6279a994523db4d4a17c7` |
| `neighbors_top100k_nq200.parquet` | 76,960,793 | `55ac377d2997295e144efd255c0c0cc587171abd54f7cbfc57efef1e2e99684e` |
| `neighbors_top1m_nq200.parquet` | 765,447,965 | `d2e8448c09f01b0324ae0577dedcae4f04294819ecb84ca78e8276ed7ba166c1` |
| `generate_laion100m_gt.py` | 49,229 | `bd2364ddb05a06bb07c8833527d85cf57c2dfbf06e6317b22ace97878c2c93d9` |
| `neighbors_int_50p_top100k_nq200.parquet` | 75,469,155 | `38d3f0a294fb0acf1520eb599cac1ee583a2dea1c6f38cf5e2e45ebf3761e930` |
| `neighbors_int_50p_top1m_nq200.parquet` | 747,381,739 | `4de6e8fc828d4aebc576dc4ddb2007d069dff325a06784c867555a571f8d745a` |
| `neighbors_int_60p_top100k_nq200.parquet` | 74,990,277 | `ffeec6c551be023f0b8ba2ef8aa1ba1c107527cb6087ba23d7f44584c2794772` |
| `neighbors_int_60p_top1m_nq200.parquet` | 744,323,151 | `c4b86264bfc4170d4067f1888fb18408d6b3a17e023a38ed219811d94221fff8` |
| `neighbors_int_70p_top100k_nq200.parquet` | 73,324,957 | `f71c40121ed5a427eda26426012aeeb87b5a4612b4d56c27b1f685d7bd0d885e` |
| `neighbors_int_70p_top1m_nq200.parquet` | 727,757,414 | `e654b918b9546ce3cdebecb61c022522d267de031da1db42421da48f98f94447` |
| `neighbors_int_80p_top100k_nq200.parquet` | 71,153,697 | `224a388c8bbd98eee0b343dd99bc7824192eb0f7d4d5d3df3b88cb2ce7fd6e57` |
| `neighbors_int_80p_top1m_nq200.parquet` | 707,379,150 | `ad0b0221549052074159cc66029204415510b0f0f9737f63ff7a07a9ff10f4d2` |
| `neighbors_int_90p_top100k_nq200.parquet` | 63,613,219 | `b5ef8e8c2b95450fff4226821b71a7af53350b10d349a76ef9f804df541ab623` |
| `neighbors_int_90p_top1m_nq200.parquet` | 632,247,723 | `e0b889b0943832b91742ace18ebeb1d1c959bbded1f17d5d0a5721d565e40202` |
| `neighbors_int_95p_top100k_nq200.parquet` | 65,052,631 | `144a7c9a92f2e06eb49fbc0741f67ee484fec11c5664eab29ac338b57ad12e03` |
| `neighbors_int_95p_top1m_nq200.parquet` | 647,952,653 | `6446d1e2c95d41b01030ad84277fab25199565a73667f4c4a3982c4a64f2e752` |
| `neighbors_int_98p_top100k_nq200.parquet` | 65,908,598 | `4241359d44027a1c141636c864fca877b16f067406261b56a59622a694ebd720` |
| `neighbors_int_98p_top1m_nq200.parquet` | 656,992,613 | `1d353ca85b65f5a4127d9ec7b4b8bf5c86323c81c17067634e385814e27df328` |
| `neighbors_int_99p_top100k_nq200.parquet` | 65,935,155 | `0a9c163e6638a88c9361fd25c38ea913f0000f22b6e700afa2a224841241e678` |
| `neighbors_int_99p_top1m_nq200.parquet` | 658,440,148 | `f9c1abfb7b6280e3eb07b2abbd77c19f1618e5199caeeaf30c816542c5615e2c` |
| `neighbors_int_99.5p_top100k_nq200.parquet` | 64,291,567 | `125b4bcafdce88079f3c57b57a33686a3e16069f5b97ba43caf36a852fad2b2c` |
| `neighbors_int_99.5p_top500k_nq200.parquet` | 321,137,007 | `bc395cb74db1b5f26453939953a1864a52cbc959b2a74762dc5d0233b9e22cf1` |
| `neighbors_int_99.8p_top100k_nq200.parquet` | 59,983,265 | `f4a4e928f7b1979da47d39fc655ffad5991d8c2d03cd0f2cf01ab5cdbeea5bef` |
| `neighbors_int_99.8p_top200k_nq200.parquet` | 120,000,002 | `1cd4881d5aff25177b49d741e0520451f6cc1da5156781d04ef3a213d75eb020` |
| `neighbors_int_99.9p_top100k_nq200.parquet` | 42,815,447 | `1050fc0f82c2f8b3a0de7aec9a4d22759628d6fd7dd29d9815667c02b633037e` |