Managing Agent Context Windows

1. Understanding Token Limits per Model

Model	Max Context Length	Token Limit	Notes
GPT-4	`128K`	`128,000`	Supports up to 128K tokens in context window
GPT-4 Turbo	`128K`	`128,000`	Improved performance and cost efficiency
GPT-3.5 Turbo	`16K`	`16,385`	Fast and cost-effective for smaller tasks
Claude 3 Opus	`200K`	`200,000`	Best for long and complex content
Claude 3 Sonnet	`200K`	`200,000`	Balance of speed and capability
Gemini Pro	`32K`	`32,768`	Google's multimodal model with strong reasoning

Note: Token limits vary by model and can change over time. Always verify current limits when integrating.

Example: Calculating Token Usage

// Example of estimating tokens in a prompt
const prompt = "Explain the concept of token limits in AI models.";
const tokens = prompt.split(/\s+/).length;
console.log(`Estimated tokens: ${tokens}`); // Output: Estimated tokens: 10

2. Managing Context Window Overflow

Strategy	Description	Implementation	Notes
Truncation	Remove older content from context window	`// Example: Keep only last 2000 tokens const context = fullContext.slice(-2000);`	Simple but may lose important prior context
Summarization	Generate summaries of older content	`// Example: Summarize past exchanges const summary = await summarize(context.slice(0, 500));`	Preserves key information but adds latency
Re-ranking	Rank and prioritize most relevant content	`// Example: Use embeddings to rank const ranked = context.sort((a, b) => relevance(a) - relevance(b));`	Requires additional computation, more accurate
Caching	Cache frequently referenced content	`// Example: Cache API responses const cached = cache.get(key) \|\| fetchAndCache(key);`	Improves performance for repeated queries

Example: Context Truncation Strategy

function truncateContext(context, maxTokens = 2048) {
  const truncated = context.slice(-maxTokens);
  return truncated;
}

Note: Truncation is the fastest but can lead to loss of context needed for accurate responses.

Warning: Summarization strategies may introduce inaccuracies if summaries are not well-formed.

3. Summarizing Long Conversations

Method	Description	Usage
auto-summarize	Automatically summarizes long conversation history	`{ enabled: true, maxTokens: 1000 }`
manual-summary	Manually trigger summarization with custom prompt	`{ prompt: "Summarize the key points..." }`

Example: Auto-summarize configuration

{
  "context": {
    "autoSummarize": {
      "enabled": true,
      "maxTokens": 1000
    }
  }
}

Example: Manual summary trigger

const summary = await agent.summarize({
  prompt: "Summarize the key decisions and code changes from this session."
});

Note: Summarization helps maintain context window limits in long development sessions.

4. Pruning Irrelevant Context

Strategy	Description	Implementation	Notes
Least Recently Used (LRU)	Removes the least recently accessed context items	`const lruCache = new LRUCache(100);`	Efficient for time-based pruning
Least Frequently Used (LFU)	Removes items with the lowest access frequency	`const lfuCache = new LFUCache(100);`	Good for identifying rarely used context
Content-Based Filtering	Filters based on content relevance or similarity	`const relevanceScore = calculateRelevance(context, query);`	Requires NLP or similarity algorithms
Time-Based Expiration	Automatically removes context older than a threshold	`const maxAge = 3600000; // 1 hour`	Simple but may remove useful context

Example: Pruning with LRU

class ContextManager {
  constructor(maxSize) {
    this.context = new Map();
    this.maxSize = maxSize;
  }

  addContext(key, value) {
    if (this.context.size >= this.maxSize) {
      const firstKey = this.context.keys().next().value;
      this.context.delete(firstKey);
    }
    this.context.set(key, value);
  }

  getContext(key) {
    return this.context.get(key);
  }
}

Note: Pruning strategies should be tuned based on the AI agent's task and context size constraints.

5. Using File Attachments Efficiently

File Type	Supported	Notes
.js	`true`	JavaScript files for code analysis and context
.ts	`true`	TypeScript files for type-aware context
.json	`true`	Configuration and data files for context
.md	`true`	Documentation and READMEs for semantic context
.txt	`true`	Plain text files for raw content context
.css	`true`	CSS files for styling context
.html	`true`	HTML files for frontend context
.py	`true`	Python files for backend context
.java	`true`	Java files for enterprise context
.go	`true`	Go files for systems context

Example: Attaching Files to Context

const contextManager = new AgentContextManager();
contextManager.attachFile("src/main.js");
contextManager.attachFile("config/app.json");
contextManager.attachFile("docs/api.md");

Note: Attachments are cached and reused to avoid redundant processing.

Warning: Large files may impact performance; consider using .gitignore patterns to exclude unnecessary files.

Attachment Strategy	Description	Best Use Case
Full File	Entire file content is attached	Small files or when full context is needed
Partial File	Only relevant sections or lines are attached	Large files where only specific parts matter
Metadata Only	File metadata (name, size, type) without content	When file existence is sufficient for context

Efficient File Attachment Tips

Attach only files relevant to the current task or agent scope
Use glob patterns like **/*.js to batch multiple files
Exclude large binary or generated files (e.g., node_modules/)
Preprocess files to extract key code blocks or comments before attachment

6. Understanding Context Caching

Cache Type	Description	Usage	Notes
Memory Cache	Stores recent interactions and responses in memory	For fast access to recent context during active sessions	Automatically cleared when session ends
Persistent Cache	Stores context in a database or file system	For maintaining context across multiple sessions or user sessions	Requires manual cleanup to prevent bloat
Hybrid Cache	Combines in-memory and persistent storage	Optimizes performance with quick access and long-term retention	Best for high-traffic or multi-user environments

Example: Memory Cache Implementation

const contextCache = new Map();

function addToCache(key, value) {
  contextCache.set(key, value);
}

function getFromCache(key) {
  return contextCache.get(key);
}

Example: Persistent Cache with File Storage

const fs = require('fs');
const path = require('path');

function saveContext(key, context) {
  const cachePath = path.join(__dirname, 'cache', `${key}.json`);
  fs.writeFileSync(cachePath, JSON.stringify(context));
}

function loadContext(key) {
  const cachePath = path.join(__dirname, 'cache', `${key}.json`);
  if (fs.existsSync(cachePath)) {
    return JSON.parse(fs.readFileSync(cachePath, 'utf-8'));
  }
  return null;
}

Note: Context caching strategies should be tailored to the specific AI agent's usage patterns and performance requirements.

7. Optimizing Context for Cost

Strategy	Description	Impact	Notes
Token Reduction	Reduce input/output tokens by summarizing or truncating content before sending to LLMs.	Reduces cost and latency	Use `truncate` or `summarize` functions before processing.
Context Window Management	Implement sliding window or chunking strategies to keep relevant context within limits.	Improves cost efficiency and performance	Use `context.window` or `chunking` logic to maintain relevance.
Caching Strategy	Cache responses for repeated queries to avoid reprocessing same context.	Reduces cost and improves response time	Implement `cache.key` logic to store and reuse results.
Prompt Engineering	Optimize prompts to reduce token count while maintaining clarity and intent.	Decreases cost and increases accuracy	Use `prompt.optimization` techniques to minimize redundancy.

Example: Reducing Context Size

const reduceContext = (context) => {
  if (context.length > maxTokens) {
    return context.slice(0, maxTokens);
  }
  return context;
};

const optimizedContext = reduceContext(inputContext);

Note: Token count varies by model; ensure your token limits align with the specific LLM being used.

8. Handling Context Truncation Errors

Error Type Description Resolution Strategy Example

ContextOverflowError

Occurs when the agent's context window exceeds the model's maximum token limit.

Implement chunking or summarization logic to reduce input size.

Error Type	Description	Resolution Strategy	Example
ContextOverflowError	Occurs when the agent's context window exceeds the model's maximum token limit.	Implement chunking or summarization logic to reduce input size.	`try { const response = await agent.process(context); } catch (error) { if (error.name === 'ContextOverflowError') { const reducedContext = chunkAndSummarize(context); const response = await agent.process(reducedContext); } }`
TokenLimitExceeded	Triggered when input tokens surpass the model's defined limit (e.g., 8192 for GPT-4).	Use a sliding window or LRU cache to maintain relevant context.	`const maxTokens = 8192; if (tokenCount(context) > maxTokens) { const windowedContext = slidingWindow(context, maxTokens); const response = await agent.process(windowedContext); }`

try {
  const response = await agent.process(context);
} catch (error) {
  if (error.name === 'ContextOverflowError') {
    const reducedContext = chunkAndSummarize(context);
    const response = await agent.process(reducedContext);
  }
}

TokenLimitExceeded

Triggered when input tokens surpass the model's defined limit (e.g., 8192 for GPT-4).

Use a sliding window or LRU cache to maintain relevant context.

const maxTokens = 8192;
if (tokenCount(context) > maxTokens) {
  const windowedContext = slidingWindow(context, maxTokens);
  const response = await agent.process(windowedContext);
}

Example: Sliding Window Implementation

function slidingWindow(context, maxTokens) {
  const tokens = tokenize(context);
  if (tokens.length <= maxTokens) return context;

  const windowStart = tokens.length - maxTokens;
  return detokenize(tokens.slice(windowStart));
}

Note: Context truncation errors are common in long conversations or when processing large codebases. Implementing a context-aware caching strategy can significantly reduce these errors.