Skip to main content

Evaluating AI-Generated Code Quality

1. Reviewing Generated Code for Bugs

Code Pattern Description Common Issues Review Checklist
Logic Flow Review control structures and conditional logic for correctness
  • Missing edge cases
  • Infinite loops
  • Incorrect branching
  • Do all paths lead to a result?
  • Are all conditions properly handled?
Error Handling Check for proper exception handling and graceful degradation
  • Uncaught exceptions
  • Missing fallbacks
  • Overly broad try-catch blocks
  • Are errors logged appropriately?
  • Does the code recover from failures?
Data Validation Ensure inputs and outputs are validated before processing
  • Unvalidated user input
  • Improper type checking
  • Missing null checks
  • Are inputs sanitized?
  • Do outputs match expected types?
Security Considerations Verify that generated code follows security best practices
  • SQL injection risks
  • Buffer overflows
  • Hardcoded secrets
  • Are sensitive values parameterized?
  • Is input sanitized for injection attacks?

Example: Bug Detection in AI-Generated Code

// AI-generated code with potential bug
function calculateDiscount(price, discountPercent) {
    const discountedPrice = price - (price * discountPercent / 100);
    return discountedPrice;
}

// Bug: No validation for negative inputs or invalid discount percent
const result = calculateDiscount(-50, 20); // Returns -40, which is incorrect
Note: AI-generated code often lacks proper validation and error handling, which can lead to runtime issues or unexpected behavior in production environments.

Key Areas to Inspect

  • Input validation for all user-provided data
  • Edge case handling in loops and conditionals
  • Exception safety with proper try/catch blocks
  • Security vulnerabilities such as injection flaws

2. Checking for Hallucinated APIs

API Description Status Notes
/api/v1/validate Validates if an API endpoint exists and is accessible NEW Used to detect hallucinated endpoints in AI-generated code
/api/v1/docs Retrieves documentation for a given API DEPRECATED Deprecated in favor of OpenAPI spec validation
/api/v1/mock Generates mock responses for API endpoints EXPERIMENTAL Used for testing AI-generated API integrations

Example: Validating API Endpoints

const validateEndpoint = async (url) => {
  try {
    const response = await fetch(url, { method: 'GET' });
    if (response.ok) {
      return { valid: true, status: response.status };
    } else {
      return { valid: false, status: response.status };
    }
  } catch (error) {
    return { valid: false, error: error.message };
  }
};

// Example usage:
const result = await validateEndpoint('https://api.example.com/users');
console.log(result); // { valid: true, status: 200 }
Note: Hallucinated APIs often return 404 or 500 errors when accessed.

3. Validating License Compliance

Example: Checking License Compliance for Open Source Dependencies

const licenseChecker = require("license-checker");

licenseChecker.init({
  start: './',
  excludePrivatePackages: true
}, (err, packages) => {
  if (err) {
    console.error("License check failed:", err);
  } else {
    Object.keys(packages).forEach(pkg => {
      const license = packages[pkg].license;
      if (license && !isPermissive(license)) {
        console.warn(`Non-permissive license found: ${pkg} - ${license}`);
      }
    });
  }
});

function isPermissive(license) {
  const permissiveLicenses = [
    "MIT", "ISC", "BSD-2-Clause", "BSD-3-Clause",
    "Apache-2.0", "JSON"
  ];
  return permissiveLicenses.includes(license);
}
License Type Description Usage Restrictions Compliance Status
MIT Permissive license allowing reuse with minimal restrictions Include copyright notice and license text COMPLIANT
Apache-2.0 Permissive with patent grant and attribution requirement Include copyright notice, license text, and patent grant COMPLIANT
GPL v3 Copyleft license requiring derivative works to also be GPL Any derivative work must also be GPL licensed NON-COMPLIANT
BSD-3-Clause Permissive with attribution requirement Include copyright notice, license text, and disclaimer COMPLIANT
Mozilla Public License (MPL) 2.0 Copyleft license with weak copyleft clause Any modifications to licensed files must be open-sourced COMPLIANT
Note: When using AI-assisted development tools, ensure that any generated code or dependencies comply with the project's chosen license model to avoid legal issues.

4. Measuring Suggestion Acceptance Rates

Metric Description Formula Interpretation
Acceptance Rate Percentage of AI suggestions that are accepted by developers (Accepted Suggestions / Total Suggestions) * 100 Higher rate indicates better alignment with developer intent and code quality
Rejection Rate Percentage of AI suggestions that are rejected by developers (Rejected Suggestions / Total Suggestions) * 100 High rejection rate may indicate poor suggestion relevance or quality
Re-acceptance Rate Percentage of previously rejected suggestions that are later accepted (Re-accepted Suggestions / Rejected Suggestions) * 100 Indicates potential improvement in AI understanding over time

Example: Calculating Acceptance Rate

// Sample data from a development team's AI-assisted workflow
const totalSuggestions = 500;
const acceptedSuggestions = 375;

const acceptanceRate = (acceptedSuggestions / totalSuggestions) * 100;
console.log(`Acceptance Rate: ${acceptanceRate}%`); // Output: Acceptance Rate: 75%
Note: Acceptance rate should be measured over a sufficient time window to account for context changes and evolving codebases.

Key Considerations for Acceptance Rate Metrics

  • Context Dependency: Acceptance rates can vary significantly based on project context and developer familiarity with AI tools.
  • Time Factor: Suggestions may be accepted or rejected at different stages of development.
  • Quality vs. Quantity: Focus on the quality of accepted suggestions rather than just the volume.

5. Benchmarking Agent Performance

Metric Description Measurement Notes
Code Accuracy Percentage of generated code that compiles and runs without errors Automated testing suite Measures syntactic and semantic correctness
Task Completion Rate Percentage of tasks completed successfully by the AI agent Manual evaluation + automated tracking Includes partial completions and error handling
Response Time Average time to generate a code snippet or solution Latency measurement tools Includes model inference and code generation time
Code Maintainability Readability and modularity of generated code Static analysis tools (e.g., ESLint, SonarQube) Assesses adherence to best practices
Adaptability Ability to adjust code based on feedback or new requirements User studies and task reiterations Measures flexibility in iterative development

Example: Measuring Code Accuracy

const testSuite = [
  {
    input: "Write a function to calculate factorial",
    expectedOutput: "function factorial(n) { ... }"
  },
  {
    input: "Implement a binary search tree",
    expectedOutput: "class TreeNode { ... }"
  }
];

const results = testSuite.map(test => {
  const generatedCode = aiAgent.generate(test.input);
  return {
    input: test.input,
    code: generatedCode,
    passesCompilation: compileAndRun(generatedCode),
    matchesExpected: generatedCode.includes(test.expectedOutput)
  };
});
Note: Benchmarking should include both synthetic and real-world use cases to ensure comprehensive evaluation.

Key Metrics for AI Agent Benchmarking

  • Code Accuracy: Critical for production use
  • Task Completion Rate: Indicates reliability in complex tasks
  • Response Time: Affects developer productivity
  • Code Maintainability: Long-term sustainability factor
  • Adaptability: Essential for evolving project requirements

6. Comparing Outputs Across Agents

Agent Output Type Code Quality Score Notes
GitHub Copilot JavaScript 8.2/10 Highly accurate for common patterns
Tabnine Python 7.5/10 Good for code completion, less so for complex logic
Amazon CodeWhisperer Java 7.8/10 Strong in enterprise contexts, moderate suggestions
Codeium TypeScript 7.0/10 Decent performance, but less mature than others

Example: Comparing Code Suggestions

// GitHub Copilot suggestion
function calculateTotal(items) {
  return items.reduce((sum, item) => sum + item.price, 0);
}

// Tabnine suggestion
function calculateTotal(items) {
  let total = 0;
  for (let i = 0; i < items.length; i++) {
    total += items[i].price;
  }
  return total;
}

Comparison Grid: Agent Output Consistency

  • GitHub Copilot: More consistent with standard patterns
  • Tabnine: Slightly more verbose, but reliable
  • Amazon CodeWhisperer: Focused on enterprise-grade code
  • Codeium: Less consistent, more experimental

7. Tracking Regression from AI Changes

Aspect Description Tools/Methods
Code Change Impact Measures how AI-generated code affects existing functionality git diff, pytest --tb=short
Test Coverage Ensures AI changes don't reduce test coverage coverage.py, istanbul
Performance Regression Monitors for slowdowns or resource usage increases benchmark.js, pyperf

Example: Regression Test Setup

const { performance } = require('node:perf_hooks');
const oldCode = require('./old-ai-module');
const newCode = require('./new-ai-module');

// Performance check
const startOld = performance.now();
oldCode.run();
const endOld = performance.now();

const startNew = performance.now();
newCode.run();
const endNew = performance.now();

console.log(`Old: ${endOld - startOld}ms`);
console.log(`New: ${endNew - startNew}ms`);
Note: Use git bisect to isolate AI-generated commits that introduced regressions.

Best Practices for Regression Tracking

  • Automate regression tests in CI pipeline
  • Use version control to track AI-generated code changes
  • Implement performance benchmarks for each AI change

8. Building an Internal Evaluation Suite

Example: Basic evaluation suite structure

// Define test cases for code quality checks
const testCases = [
  {
    name: "Code Readability",
    description: "Evaluate code clarity and structure",
    criteria: [
      {
        id: "no-long-functions",
        description: "Functions should not exceed 20 lines",
        severity: "warning"
      },
      {
        id: "consistent-naming",
        description: "Use camelCase for variables and functions",
        severity: "error"
      }
    ]
  },
  {
    name: "Security Checks",
    description: "Ensure no vulnerable code patterns",
    criteria: [
      {
        id: "no-unsafe-eval",
        description: "Avoid usage of eval()",
        severity: "error"
      }
    ]
  }
];

// Run evaluation on generated code
function evaluateCode(code, testCases) {
  const results = [];
  testCases.forEach(testCase => {
    const testCaseResult = {
      name: testCase.name,
      checks: []
    };
    testCase.criteria.forEach(criterion => {
      const pass = checkCriterion(code, criterion);
      testCaseResult.checks.push({
        id: criterion.id,
        passed: pass,
        severity: criterion.severity
      });
    });
    results.push(testCaseResult);
  });
  return results;
}

function checkCriterion(code, criterion) {
  // Implementation depends on specific checks
  return true; // Placeholder
}
Component Description Implementation
Test Case Definition Structured format for defining code quality criteria name, description, criteria
Evaluation Engine Core logic to run checks against code Iterates through test cases and criteria
Check Criteria Specific rules to validate code quality id, description, severity

Key Features of Evaluation Suites

  • Modular Design - Each test case can be independently developed and maintained
  • Configurable Severity - Different levels of checks (warning, error, info)
  • Extensible - Easy to add new criteria or test cases
  • Automated Reporting - Generates detailed reports on code quality issues
Note: The evaluation suite should be integrated into CI/CD pipelines to ensure code quality standards are maintained.

Was this article helpful?