Evaluating AI-Generated Code Quality
1. Reviewing Generated Code for Bugs
| Code Pattern | Description | Common Issues | Review Checklist |
|---|---|---|---|
| Logic Flow | Review control structures and conditional logic for correctness |
|
|
| Error Handling | Check for proper exception handling and graceful degradation |
|
|
| Data Validation | Ensure inputs and outputs are validated before processing |
|
|
| Security Considerations | Verify that generated code follows security best practices |
|
|
Example: Bug Detection in AI-Generated Code
// AI-generated code with potential bug
function calculateDiscount(price, discountPercent) {
const discountedPrice = price - (price * discountPercent / 100);
return discountedPrice;
}
// Bug: No validation for negative inputs or invalid discount percent
const result = calculateDiscount(-50, 20); // Returns -40, which is incorrect
Note: AI-generated code often lacks proper validation and error handling, which can lead to runtime issues or unexpected behavior in production environments.
Key Areas to Inspect
- Input validation for all user-provided data
- Edge case handling in loops and conditionals
- Exception safety with proper try/catch blocks
- Security vulnerabilities such as injection flaws
2. Checking for Hallucinated APIs
| API | Description | Status | Notes |
|---|---|---|---|
| /api/v1/validate | Validates if an API endpoint exists and is accessible | NEW | Used to detect hallucinated endpoints in AI-generated code |
| /api/v1/docs | Retrieves documentation for a given API | DEPRECATED | Deprecated in favor of OpenAPI spec validation |
| /api/v1/mock | Generates mock responses for API endpoints | EXPERIMENTAL | Used for testing AI-generated API integrations |
Example: Validating API Endpoints
const validateEndpoint = async (url) => {
try {
const response = await fetch(url, { method: 'GET' });
if (response.ok) {
return { valid: true, status: response.status };
} else {
return { valid: false, status: response.status };
}
} catch (error) {
return { valid: false, error: error.message };
}
};
// Example usage:
const result = await validateEndpoint('https://api.example.com/users');
console.log(result); // { valid: true, status: 200 }
Note: Hallucinated APIs often return 404 or 500 errors when accessed.
3. Validating License Compliance
Example: Checking License Compliance for Open Source Dependencies
const licenseChecker = require("license-checker");
licenseChecker.init({
start: './',
excludePrivatePackages: true
}, (err, packages) => {
if (err) {
console.error("License check failed:", err);
} else {
Object.keys(packages).forEach(pkg => {
const license = packages[pkg].license;
if (license && !isPermissive(license)) {
console.warn(`Non-permissive license found: ${pkg} - ${license}`);
}
});
}
});
function isPermissive(license) {
const permissiveLicenses = [
"MIT", "ISC", "BSD-2-Clause", "BSD-3-Clause",
"Apache-2.0", "JSON"
];
return permissiveLicenses.includes(license);
}
| License Type | Description | Usage Restrictions | Compliance Status |
|---|---|---|---|
| MIT | Permissive license allowing reuse with minimal restrictions | Include copyright notice and license text | COMPLIANT |
| Apache-2.0 | Permissive with patent grant and attribution requirement | Include copyright notice, license text, and patent grant | COMPLIANT |
| GPL v3 | Copyleft license requiring derivative works to also be GPL | Any derivative work must also be GPL licensed | NON-COMPLIANT |
| BSD-3-Clause | Permissive with attribution requirement | Include copyright notice, license text, and disclaimer | COMPLIANT |
| Mozilla Public License (MPL) 2.0 | Copyleft license with weak copyleft clause | Any modifications to licensed files must be open-sourced | COMPLIANT |
Note: When using AI-assisted development tools, ensure that any generated code or dependencies comply with the project's chosen license model to avoid legal issues.
4. Measuring Suggestion Acceptance Rates
| Metric | Description | Formula | Interpretation |
|---|---|---|---|
| Acceptance Rate | Percentage of AI suggestions that are accepted by developers | (Accepted Suggestions / Total Suggestions) * 100 |
Higher rate indicates better alignment with developer intent and code quality |
| Rejection Rate | Percentage of AI suggestions that are rejected by developers | (Rejected Suggestions / Total Suggestions) * 100 |
High rejection rate may indicate poor suggestion relevance or quality |
| Re-acceptance Rate | Percentage of previously rejected suggestions that are later accepted | (Re-accepted Suggestions / Rejected Suggestions) * 100 |
Indicates potential improvement in AI understanding over time |
Example: Calculating Acceptance Rate
// Sample data from a development team's AI-assisted workflow
const totalSuggestions = 500;
const acceptedSuggestions = 375;
const acceptanceRate = (acceptedSuggestions / totalSuggestions) * 100;
console.log(`Acceptance Rate: ${acceptanceRate}%`); // Output: Acceptance Rate: 75%
Note: Acceptance rate should be measured over a sufficient time window to account for context changes and evolving codebases.
Key Considerations for Acceptance Rate Metrics
- Context Dependency: Acceptance rates can vary significantly based on project context and developer familiarity with AI tools.
- Time Factor: Suggestions may be accepted or rejected at different stages of development.
- Quality vs. Quantity: Focus on the quality of accepted suggestions rather than just the volume.
5. Benchmarking Agent Performance
| Metric | Description | Measurement | Notes |
|---|---|---|---|
| Code Accuracy | Percentage of generated code that compiles and runs without errors | Automated testing suite | Measures syntactic and semantic correctness |
| Task Completion Rate | Percentage of tasks completed successfully by the AI agent | Manual evaluation + automated tracking | Includes partial completions and error handling |
| Response Time | Average time to generate a code snippet or solution | Latency measurement tools | Includes model inference and code generation time |
| Code Maintainability | Readability and modularity of generated code | Static analysis tools (e.g., ESLint, SonarQube) | Assesses adherence to best practices |
| Adaptability | Ability to adjust code based on feedback or new requirements | User studies and task reiterations | Measures flexibility in iterative development |
Example: Measuring Code Accuracy
const testSuite = [
{
input: "Write a function to calculate factorial",
expectedOutput: "function factorial(n) { ... }"
},
{
input: "Implement a binary search tree",
expectedOutput: "class TreeNode { ... }"
}
];
const results = testSuite.map(test => {
const generatedCode = aiAgent.generate(test.input);
return {
input: test.input,
code: generatedCode,
passesCompilation: compileAndRun(generatedCode),
matchesExpected: generatedCode.includes(test.expectedOutput)
};
});
Note: Benchmarking should include both synthetic and real-world use cases to ensure comprehensive evaluation.
Key Metrics for AI Agent Benchmarking
- Code Accuracy: Critical for production use
- Task Completion Rate: Indicates reliability in complex tasks
- Response Time: Affects developer productivity
- Code Maintainability: Long-term sustainability factor
- Adaptability: Essential for evolving project requirements
6. Comparing Outputs Across Agents
| Agent | Output Type | Code Quality Score | Notes |
|---|---|---|---|
| GitHub Copilot | JavaScript |
8.2/10 | Highly accurate for common patterns |
| Tabnine | Python |
7.5/10 | Good for code completion, less so for complex logic |
| Amazon CodeWhisperer | Java |
7.8/10 | Strong in enterprise contexts, moderate suggestions |
| Codeium | TypeScript |
7.0/10 | Decent performance, but less mature than others |
Example: Comparing Code Suggestions
// GitHub Copilot suggestion
function calculateTotal(items) {
return items.reduce((sum, item) => sum + item.price, 0);
}
// Tabnine suggestion
function calculateTotal(items) {
let total = 0;
for (let i = 0; i < items.length; i++) {
total += items[i].price;
}
return total;
}
Comparison Grid: Agent Output Consistency
- GitHub Copilot: More consistent with standard patterns
- Tabnine: Slightly more verbose, but reliable
- Amazon CodeWhisperer: Focused on enterprise-grade code
- Codeium: Less consistent, more experimental
7. Tracking Regression from AI Changes
| Aspect | Description | Tools/Methods |
|---|---|---|
| Code Change Impact | Measures how AI-generated code affects existing functionality | git diff, pytest --tb=short |
| Test Coverage | Ensures AI changes don't reduce test coverage | coverage.py, istanbul |
| Performance Regression | Monitors for slowdowns or resource usage increases | benchmark.js, pyperf |
Example: Regression Test Setup
const { performance } = require('node:perf_hooks');
const oldCode = require('./old-ai-module');
const newCode = require('./new-ai-module');
// Performance check
const startOld = performance.now();
oldCode.run();
const endOld = performance.now();
const startNew = performance.now();
newCode.run();
const endNew = performance.now();
console.log(`Old: ${endOld - startOld}ms`);
console.log(`New: ${endNew - startNew}ms`);
Note: Use
git bisect to isolate AI-generated commits that introduced regressions.
Best Practices for Regression Tracking
- Automate regression tests in CI pipeline
- Use version control to track AI-generated code changes
- Implement performance benchmarks for each AI change
8. Building an Internal Evaluation Suite
Example: Basic evaluation suite structure
// Define test cases for code quality checks
const testCases = [
{
name: "Code Readability",
description: "Evaluate code clarity and structure",
criteria: [
{
id: "no-long-functions",
description: "Functions should not exceed 20 lines",
severity: "warning"
},
{
id: "consistent-naming",
description: "Use camelCase for variables and functions",
severity: "error"
}
]
},
{
name: "Security Checks",
description: "Ensure no vulnerable code patterns",
criteria: [
{
id: "no-unsafe-eval",
description: "Avoid usage of eval()",
severity: "error"
}
]
}
];
// Run evaluation on generated code
function evaluateCode(code, testCases) {
const results = [];
testCases.forEach(testCase => {
const testCaseResult = {
name: testCase.name,
checks: []
};
testCase.criteria.forEach(criterion => {
const pass = checkCriterion(code, criterion);
testCaseResult.checks.push({
id: criterion.id,
passed: pass,
severity: criterion.severity
});
});
results.push(testCaseResult);
});
return results;
}
function checkCriterion(code, criterion) {
// Implementation depends on specific checks
return true; // Placeholder
}
| Component | Description | Implementation |
|---|---|---|
| Test Case Definition | Structured format for defining code quality criteria | name, description, criteria |
| Evaluation Engine | Core logic to run checks against code | Iterates through test cases and criteria |
| Check Criteria | Specific rules to validate code quality | id, description, severity |
Key Features of Evaluation Suites
- Modular Design - Each test case can be independently developed and maintained
- Configurable Severity - Different levels of checks (warning, error, info)
- Extensible - Easy to add new criteria or test cases
- Automated Reporting - Generates detailed reports on code quality issues
Note: The evaluation suite should be integrated into CI/CD pipelines to ensure code quality standards are maintained.