Working with Sparse Vectors
1. Understanding Sparse Vector Format
| Field | Type | Description |
|---|---|---|
indices | List[int] | Non-zero dimension positions |
values | List[float] | Corresponding non-zero values |
| Length match | — | len(indices) == len(values) |
2. Creating Sparse Vector Objects
3. Upserting Sparse Vectors
Example: Hybrid Upsert
index.upsert(vectors=[{
"id": "doc1",
"values": dense, # required, even for sparse-heavy
"sparse_values": sparse,
"metadata": {"source": "wiki"},
}])Warning: Sparse vectors require a
dotproduct index. Cosine/Euclidean indexes reject sparse values.4. Querying with Sparse Vectors
Example: Hybrid Query
res = index.query(
vector=dense_q,
sparse_vector={"indices": [...], "values": [...]},
top_k=10,
)5. Understanding Sparse Vector Use Cases
| Use Case | Why Sparse |
|---|---|
| Keyword/lexical search | Exact term match |
| BM25 retrieval | Classic IR scoring |
| SPLADE | Learned sparse embeddings |
| Domain jargon | Out-of-vocab for dense models |
6. Validating Sparse Vector Format
Example: Validate
def valid_sparse(s):
return (
isinstance(s.get("indices"), list)
and isinstance(s.get("values"), list)
and len(s["indices"]) == len(s["values"])
and all(isinstance(i, int) and i >= 0 for i in s["indices"])
and len(set(s["indices"])) == len(s["indices"]) # no dupes
)| Rule | Detail |
|---|---|
| Indices unique | Duplicates rejected |
| Non-negative ints | 0-based positions |
| Same length | indices and values |
7. Optimizing Sparse Vector Size
| Tip | Effect |
|---|---|
| Top-K term pruning | Keep most-informative terms only |
| IDF threshold | Drop low-information terms |
| Avoid stopwords | Reduce noise + size |
8. Handling Sparse Vector Indexing
| Requirement | Detail |
|---|---|
| Metric | dotproduct only |
| Pod type | p1, p2, s1 supported |
| Serverless | Supported with dotproduct |
9. Understanding Sparse Performance Considerations
| Factor | Impact |
|---|---|
| High term count | Slower upsert/query |
| Hybrid (dense+sparse) | ~1.5–2× query latency |
| Storage | Adds bytes per vector |
10. Debugging Sparse Vector Errors
| Error | Cause |
|---|---|
| Length mismatch | indices != values |
| Duplicate indices | Same position twice |
| Wrong metric | Index not dotproduct |
| Empty sparse | Empty indices/values rejected |