FIXES_APPLIED.md

AutoDev Fixes Applied

Date: 2026-02-15 Status: Complete

---

Summary

Applied 4 critical fixes to AutoDev based on run analysis:

1. ✅ Verification now runs actual tests - No more false positives
2. ✅ Patches saved to logs - Can debug what LLM generated
3. ✅ Fuzzy diff matching - More resilient to line number offsets
4. ✅ Code change validation - Rejects empty/no-op patches
5. ✅ Improved prompt - More explicit requirements

---

Fix 1: Real Test Verification

File: ooda/act.py

Before: Only checked syntax and compilation

def verify_all(self, file_path: str):
    # 1. Syntax check
    # 2. Import check
    return all_passed, messages

After: Runs actual Mal tests via Docker

def verify_all(self, file_path: str):
    # 1. Syntax check
    # 2. Import check
    # 3. RUN ACTUAL TESTS via Docker
    observer = self._get_observer()
    test_success, test_output = observer.run_tests()

    if test_success:
        messages.append(f"Tests: [OK] All tests passing!")
    else:
        parsed = observer.parse_mal_test_output(test_output)
        missing = parsed.get("missing_functions", [])
        messages.append(f"  Still missing: {', '.join(missing[:5])}")

    return test_success, messages

Impact: Patches that don't add working code will now fail verification.

---

Fix 2: Patch Logging

File: ooda/act.py

Added: Save patches to separate files for debugging

def log(self, iteration, observe_data, orient_data, decide_data, act_result):
    # ... existing code ...

    # Save patch to separate file for easier debugging
    patch_result = decide_data.get("patch_result")
    if patch_result:
        patch_file = self.logs_dir / f"patch_iter_{iteration}.diff"
        patch_file.write_text(patch_result.patch)

Impact: Can now inspect what the LLM actually generated.

---

Fix 3: Fuzzy Diff Matching

File: ooda/act.py

Added: Search nearby lines when exact match fails

def _apply_hunk(self, lines, hunk, line_offset):
    # Try exact match first
    result = self._try_apply_hunk_at(lines, hunk, actual_start)
    if result[0]:
        return result

    # Fuzzy matching: search for context lines nearby
    for offset in range(-20, 21):
        fuzzy_start = actual_start + offset
        result = self._try_apply_hunk_at(lines, hunk, fuzzy_start)
        if result[0]:
            return True, None

Impact: Patches are more likely to apply even with line number offsets.

---

Fix 4: Code Change Validation

File: ooda/decide.py

Added: Check that patch actually adds code

# Check if patch adds actual code (not just comments/whitespace)
code_added = False
for line in patch.split('\n'):
    if line.startswith('+') and not line.startswith('+++'):
        content = line[1:].strip()
        if content and not content.startswith('#'):
            code_added = True
            break

if not code_added:
    print(f"    [FAIL] Patch doesn't add any actual code")
    is_valid = False

Impact: Empty patches or comment-only patches are rejected.

---

Fix 5: Improved Prompt

File: prompts/mal_patch_generation.txt

Added: Critical requirements section

## CRITICAL REQUIREMENTS
- Generate COMPLETE, WORKING CODE - not placeholders or TODOs
- The diff must reference the ACTUAL line numbers in the current code
- Each function must be FULLY IMPLEMENTED with proper error handling
- NO stubs, NO "pass" statements, NO placeholder comments

Added: How-to section for creating diffs

## How to Create the Diff
1. Find the location in the Current Code where your new code should go
2. Note the EXACT line number (count from line 1)
3. Include context lines (lines starting with space) before and after

Impact: LLM should generate better patches with proper line numbers.

---

Expected Behavior Changes

Before This Fix

Iteration 1: SUCCESS (patch_applied=True, verified=True)
  → But: No functions actually added!
  → Problem: Verification only checked syntax

After This Fix

Iteration 1: FAIL (patch_applied=True, verified=False)
  → Reason: Tests still failing
  → Verification runs actual tests
  → Loop continues until tests pass

---

Testing

Run AutoDev again to verify fixes:

$env:LLM_PROVIDER = "glm"
python autodev.py -f step4.py --max-iterations 10 --verbose

Expected improvements:

1. Fewer false positive "success" results
2. Patch files saved to logs/patch_iter_N.diff
3. More patches apply successfully (fuzzy matching)
4. Better quality patches (improved prompt)

---

Next Iterations

If issues persist:

1. Check patch files - ls logs/patch_iter_*.diff
2. Analyze failures - Look for patterns in rejection reasons
3. Refine prompt - Add more examples based on actual failures
4. Consider alternative approach - Direct code insertion instead of patches