Upserting Vectors
1. Generating Vector IDs
| Strategy | Pros / Cons |
|---|---|
| UUID v4 | Random, no collisions; not human-readable |
| Content hash | Deduplication built-in; expensive to compute |
| Source ID | e.g., doc_42_chunk_3; traceable, requires source schema |
| Auto-increment | Sequential; needs central counter |
Note: IDs are strings up to 512 chars. Avoid spaces and special chars for portability.
2. Validating Vector Dimensions
Example: Validate Before Upsert
EXPECTED_DIM = 1536
def validate(vec):
if len(vec) != EXPECTED_DIM:
raise ValueError(f"dim {len(vec)} != {EXPECTED_DIM}")
if not all(isinstance(x, (int, float)) for x in vec):
raise TypeError("non-numeric value")| Error | Cause |
|---|---|
| 400 Dimension mismatch | Vector length != index dim |
| 400 Invalid type | NaN, Inf, or non-numeric value |
3. Performing Basic Upsert
Example: Single Upsert
index = pc.Index("products")
index.upsert(vectors=[
{"id": "p1", "values": [0.1, 0.2, ...]},
{"id": "p2", "values": [0.3, 0.4, ...]},
])Example: Node.js
const index = pc.index("products");
await index.upsert([
{ id: "p1", values: [0.1, 0.2] },
{ id: "p2", values: [0.3, 0.4] },
]);4. Upserting with Metadata
Example: Metadata
index.upsert(vectors=[{
"id": "doc1",
"values": [...],
"metadata": {
"source": "wiki",
"page": 12,
"tags": ["ml", "vectors"],
"published": True,
},
}])| Metadata Type | Supported |
|---|---|
| string | Yes |
| number (int/float) | Yes |
| boolean | Yes |
| list of strings | Yes |
| nested object | No (flatten with dot notation) |
5. Batch Upserting Vectors
Example: Batched Upsert
def chunks(lst, size):
for i in range(0, len(lst), size):
yield lst[i:i+size]
for batch in chunks(all_vectors, 100):
index.upsert(vectors=batch)| Limit | Value |
|---|---|
| Max batch size | 2 MB or 1000 vectors |
| Recommended | 100 vectors @ 1536d |
6. Upserting to Namespace
| Param | Behavior |
|---|---|
namespace="name" | Creates if missing |
namespace="" | Default namespace |
7. Optimizing Batch Size
| Dim | Recommended Batch |
|---|---|
| 384 | 200–500 |
| 768 | 200 |
| 1536 | 100 |
| 3072 | 50 |
Note: Stay under 2 MB payload. Larger batches reduce overhead but risk timeouts.
8. Handling Upsert Failures
Example: Retry with Backoff
import time
for attempt in range(5):
try:
index.upsert(vectors=batch)
break
except Exception as e:
if attempt == 4: raise
time.sleep(2 ** attempt)| Error | Strategy |
|---|---|
| 429 Rate limit | Exponential backoff |
| 413 Payload too large | Reduce batch size |
| 500/503 | Retry with backoff |
| 400 Invalid input | Fix data; don't retry |
9. Using Async Upsert Operations
Example: Async Python
from concurrent.futures import ThreadPoolExecutor
with ThreadPoolExecutor(max_workers=10) as pool:
futures = [pool.submit(index.upsert, vectors=b) for b in batches]
for f in futures: f.result()10. Monitoring Upsert Throughput
| Metric | Source |
|---|---|
| Vectors/sec | Client-side timer |
| Index growth | describe_index_stats() |
| Errors | Application log / Pinecone console |
| Latency p95 | Pinecone metrics dashboard |