Using Workspace Indexing and Embeddings

1. Understanding Codebase Indexing

Concept	Description	Key Features
Indexing Scope	Process of analyzing and structuring codebase content for semantic search and retrieval	Supports multiple languages, handles large-scale codebases
Semantic Parsing	Extracts meaning from code structure, comments, and metadata	Enables intelligent search beyond keyword matching
Incremental Updates	Efficiently updates index when code changes occur	Reduces resource consumption and improves responsiveness

Example: Indexing Configuration

{
  "indexing": {
    "enabled": true,
    "languages": ["javascript", "typescript", "python"],
    "exclude": ["node_modules", ".git"],
    "updateFrequency": "onSave"
  }
}

Note: Workspace indexing is critical for AI-assisted development tools to understand and reason about your codebase.

2. Configuring Workspace Search Scope

Option	Values	Description
search.scope	`"workspace"` \| `"folder"` \| `"file"`	Determines the scope of search indexing. Default is `workspace`.
search.exclude	`string[]`	Patterns to exclude from indexing. E.g., `["/node_modules", "/.git"]`.
search.include	`string[]`	Patterns to explicitly include in indexing. E.g., `["*/.js", "*/.ts"]`.

Example: Configure search scope in settings

{
  "search.scope": "workspace",
  "search.exclude": [
    "**/node_modules",
    "**/.git"
  ],
  "search.include": [
    "**/*.js",
    "**/*.ts"
  ]
}

Note: Changes to search scope require a workspace re-index to take effect.

3. Excluding Files from Indexing

File Pattern	Description	Exclusion Method
node_modules/**	Node.js modules directory	`.gitignore` or `.indexignore`
*/.log	Log files	`.indexignore`
build/**	Build artifacts	`.indexignore`
*.tmp	Temporary files	`.indexignore`
/dist/	Compiled distribution files	`.indexignore`

Example: .indexignore configuration

# Exclude large directories
node_modules/
build/

# Exclude specific file types
*.log
*.tmp

# Exclude individual files
config/secrets.json

Note: The .indexignore file follows the same syntax as .gitignore, making it easy to maintain consistent exclusions across tools.

4. Using Semantic Code Search

Semantic code search enables developers to find relevant code snippets, functions, or classes using natural language queries instead of traditional keyword-based search. It leverages embeddings and vector databases to understand code semantics.

Feature	Description	Implementation
Query Input	Natural language query describing desired code	Text input field with autocomplete support
Code Embedding	Vector representation of code elements	Generated using models like CodeBERT or BERT
Vector Store	Database for storing and querying embeddings	Uses Pinecone, Weaviate, or FAISS

Example: Semantic Search Query

const query = "find all functions that handle user authentication";
const results = await semanticSearch(query, { topK: 5 });
console.log(results);

Note: The accuracy of semantic search depends heavily on the quality and coverage of code embeddings in the vector store.

Key Benefits

Improved code discovery in large codebases
Reduces time spent searching for relevant code
Enables understanding of code structure through natural language

5. Refreshing Stale Indexes

Stale indexes can lead to outdated search results and reduced accuracy in AI-assisted development environments. Refreshing these indexes ensures that the latest code changes are reflected in the workspace index.

Index Type	Refresh Trigger	Manual Refresh Command	Notes
Workspace Index	Code changes, file additions/deletions	`refresh-index`	Triggers a full re-index of the workspace
Embedding Index	Model updates, codebase modifications	`rebuild-embeddings`	Rebuilds embeddings for all files in the workspace

Example: Refreshing Workspace Index

refresh-index --workspace /home/user/project

Example: Rebuilding Embedding Index

rebuild-embeddings --model gpt-4 --workspace /home/user/project

Note: Refreshing indexes can be resource-intensive and may take several minutes depending on the size of the workspace.

Warning: Manual refreshes should be performed during low-activity periods to avoid impacting ongoing development workflows.

6. Understanding Embedding-Based Retrieval

Concept	Description	Use Case
Semantic Similarity	Measures how closely two pieces of text align in meaning, not just keyword matches	Search for code snippets or documentation with similar intent
Embedding Vector	N-dimension numerical representation of text or code	Used for efficient similarity computation in vector databases
Vector Store	Database optimized for storing and querying embedding vectors	Fast retrieval of semantically similar content in large codebases

Example: Retrieving Similar Code Snippets

const queryEmbedding = await embedder.embed("function to calculate total price");
const similarSnippets = await vectorStore.query(
  queryEmbedding,
  { topK: 5 }
);

Note: Embedding-based retrieval is particularly powerful in AI-assisted development for finding relevant code examples or documentation that match a developer's semantic intent.

7. Troubleshooting Indexing Performance

Issue	Possible Cause	Solution
Slow indexing	Large codebase or inefficient configuration	Use `.indexignore` to exclude non-essential files. Optimize indexing settings like `maxFilesToIndex`.
High memory usage	Too many files indexed simultaneously	Reduce `maxConcurrentIndexing`. Set `indexingStrategy` to `incremental`.
Indexing fails or crashes	Memory limits exceeded or corrupted cache	Clear the workspace index cache. Restart the AI assistant with `indexing.reset`.

Example: Adjusting Indexing Settings

{
  "indexing": {
    "maxFilesToIndex": 5000,
    "maxConcurrentIndexing": 4,
    "indexingStrategy": "incremental"
  }
}

Note: Indexing performance can be affected by system resources and network latency when using remote codebases.

8. Indexing Large Monorepos

Configuration	Values	Description
indexing.maxFiles	`10000`	Maximum number of files to index in a monorepo
indexing.maxSizeMB	`500`	Maximum total size of files to index (MB)
indexing.excludePatterns	`["node_modules/", "dist/", "*.log"]`	Patterns to exclude from indexing

Example: Configure monorepo indexing

{
  "indexing": {
    "maxFiles": 15000,
    "maxSizeMB": 1000,
    "excludePatterns": [
      "node_modules/",
      "dist/",
      "*.log",
      ".git/"
    ]
  }
}

Note: For very large monorepos, consider using incremental indexing to avoid full re-indexing on every change.

Indexing Strategy for Monorepos

Use maxFiles and maxSizeMB to prevent memory issues
Exclude build artifacts and logs to reduce index size
Implement incremental indexing for faster updates