Support for generating test cases #41

Activity

1. cweill commented on Apr 8, 2017

   

   cweill

   on Apr 8, 2017

   Owner

   More actions

   @emou - I like your idea. Generating test cases fits nicely with the goal of gotests: to make writing tests easier. If you could outline a design of how this would work, I would be happy to critique it and provide guidance.

   One of the core challenges is figuring out how to seamlessly get testing/quick to generate test cases and then pass them to gotests. It's not immediately clear how we can get gotests to execute test code, since it's a binary separate from your code which just performs static analysis right now.
2. cweill

   added

   enhancement

   on Apr 8, 2017
3. cweill

   added

   help wanted

   on Mar 16, 2018
4. cweill commented on Oct 21, 2025

   

   cweill

   on Oct 21, 2025

   Last edited by cweill

   Owner

   More actions

   Proposal: LLM-Based Test Case Generation

   I'd like to propose an approach to address this issue using local LLM models to generate actual test cases, complementing gotests' existing scaffolding.

   The Problem

   Currently, gotests generates excellent table-driven test scaffolding but leaves test case creation to developers:

   tests := []struct {
       name string
       args args
       want int
   }{
       // TODO: Add test cases.  ← Developer must fill this manually
   }

   Proposed Solution

   Add an optional -ai flag that generates test cases using local LLM models:

   $ gotests -ai -w math.go

   Generated output:

   tests := []struct {
       name string
       args args
       want int
   }{
       {
           name: "positive_numbers",
           args: args{a: 2, b: 3},
           want: 5,
       },
       {
           name: "zero_values",
           args: args{a: 0, b: 0},
           want: 0,
       },
       {
           name: "negative_numbers",
           args: args{a: -1, b: -2},
           want: -3,
       },
   }

   Why This Approach?

   Unlike property-based testing (original issue proposal), this:

   • ✅ Stays within gotests' static analysis model - No runtime execution required
   • ✅ Completely free - Uses local open-source models via Ollama
   • ✅ Privacy-first - Code never leaves your machine
   • ✅ Opt-in - Doesn't change existing behavior without the flag
   • ✅ Fast - Tiny models generate cases in 2-3 seconds
   • ✅ Works offline - No internet required after model download

   ---

   MVP Design

   Core Features

   Single Provider: Ollama (local LLM runtime)
   Default Model: qwen2.5-coder:0.5b (500MB, runs on any laptop)
   Test Cases: 3 per function (normal, edge, error cases)
   Validation: In-memory compilation check with retry logic

   End-User Workflow

   First-time setup (one-time):

   # Install Ollama
   $ curl -fsSL https://ollama.com/install.sh | sh
   
   # Pull tiny code model (~400MB)
   $ ollama pull qwen2.5-coder:0.5b

   Ongoing usage:

   # Same as before, just add -ai flag
   $ gotests -ai -w myfile.go

   Example: Error Handling

   Source code:

   func Divide(n, d int) (int, error) {
       if d == 0 {
           return 0, errors.New("division by zero")
       }
       return n / d, nil
   }

   Generated test:

   func TestDivide(t *testing.T) {
       tests := []struct {
           name    string
           args    args
           want    int
           wantErr bool
       }{
           {
               name: "normal_division",
               args: args{n: 10, d: 2},
               want: 5,
               wantErr: false,
           },
           {
               name: "division_by_zero",
               args: args{n: 10, d: 0},
               want: 0,
               wantErr: true,  // AI correctly expects error
           },
           {
               name: "negative_divisor",
               args: args{n: 10, d: -2},
               want: -5,
               wantErr: false,
           },
       }
       for _, tt := range tests {
           t.Run(tt.name, func(t *testing.T) {
               got, err := Divide(tt.args.n, tt.args.d)
               if (err != nil) != tt.wantErr {
                   t.Errorf("Divide() error = %v, wantErr %v", err, tt.wantErr)
                   return
               }
               if got != tt.want {
                   t.Errorf("Divide() = %v, want %v", got, tt.want)
               }
           })
       }
   }

   ---

   Architecture

   Provider Interface (Extensible)

   // internal/ai/provider.go
   type Provider interface {
       GenerateTestCases(ctx context.Context, fn *models.Function) ([]TestCase, error)
       IsAvailable() bool
   }
   
   type TestCase struct {
       Name    string
       Args    map[string]string  // param name -> value
       Want    map[string]string  // return name -> expected value  
       WantErr bool
   }

   Compilation Validation

   Uses Go's built-in go/parser and go/types packages for in-memory validation:

   func ValidateGeneratedTest(testCode, pkgName string) error {
       fset := token.NewFileSet()
       
       // Parse
       file, err := parser.ParseFile(fset, "test.go", testCode, parser.AllErrors)
       if err != nil {
           return fmt.Errorf("parse error: %w", err)
       }
       
       // Type-check
       conf := types.Config{Importer: importer.Default()}
       _, err = conf.Check(pkgName, fset, []*ast.File{file}, nil)
       return err
   }

   Retry logic with feedback:

   Attempt 1: LLM generates code
            → Validator: "undefined variable 'x'"
            → Retry with error feedback
   Attempt 2: LLM fixes issue
            → Validation passes ✓
   

   File Structure

   internal/ai/
   ├── provider.go      # Provider interface (30 lines)
   ├── ollama.go        # Ollama implementation (200 lines)
   ├── prompt.go        # Prompt templates (50 lines)
   ├── parser.go        # Parse JSON responses (20 lines)
   └── validator.go     # Compilation checking (50 lines)
   
   Total new code: ~375 lines
   Modified code: ~45 lines (flags, template injection)
   

   Template Changes

   Minimal modification to internal/render/templates/function.tmpl:

   tests := []struct {
       // ... fields ...
   }{
       {{if .AIGeneratedCases}}
       {{range .AIGeneratedCases}}
       {
           name: "{{.Name}}",
           args: args{ {{range .Args}}{{.Key}}: {{.Value}}, {{end}} },
           want: {{.Want}},
           wantErr: {{.WantErr}},
       },
       {{end}}
       {{else}}
       // TODO: Add test cases.
       {{end}}
   }

   ---

   Recommended Models

   All run locally via Ollama:

   Model	Size	Speed	Quality	Use Case
   qwen2.5-coder:0.5b	400MB	Very Fast	Good	Default (runs anywhere)
   qwen2.5-coder:1.5b	1.5GB	Fast	Better	Balance
   qwen2.5-coder:7b	4.4GB	Medium	Best	High quality (needs GPU)
   phi3:mini	2.2GB	Fast	Good	Alternative (Microsoft)
   deepseek-coder:6.7b	3.8GB	Medium	Best	Code-specialized

   Users can switch models with -ai-model=qwen2.5-coder:7b

   ---

   What It Handles (90% Use Cases)

   ✅ Simple functions (math, string manipulation)
   ✅ Functions returning errors
   ✅ Basic types (int, string, bool, []byte)
   ✅ Edge cases (zero, nil, negative values)
   ✅ Boundary conditions
   ✅ Error scenarios

   Future enhancements (remaining 10%):

   • Complex custom types
   • Functions with 10+ parameters
   • Mocking/dependency injection
   • Integration test scenarios

   ---

   Implementation Phases

   Phase 1: MVP (This Proposal)

   • Provider interface
   • Ollama implementation
   • -ai flag
   • In-memory validation
   • Basic prompt engineering

   Deliverable: Working -ai flag with qwen2.5-coder:0.5b

   Phase 2: Multi-Provider

   • Generic OpenAI-compatible provider
   • OpenAI API support
   • Claude API support
   • Provider auto-detection

   Phase 3: Quality & Polish

   • Enhanced prompts for complex types
   • Better error handling
   • Unit tests for AI integration
   • Documentation and examples

   Phase 4: Advanced Features

   • Interactive review mode
   • Caching generated cases
   • Fine-tuned prompts per construct
   • Existing test enhancement

   ---

   Why Ollama + Tiny Models?

   Ollama is the Docker of LLMs:

   • Easy installation (one curl command)
   • Manages models locally
   • Simple API
   • Active development (Go-based project)
   • Large model library

   Tiny models (0.5B-7B parameters):

   • Run on CPU (no GPU required for 0.5B)
   • Fast inference (2-3 seconds)
   • Good enough for basic test cases
   • Completely free and open source
   • Recent (2024-2025), trained on modern code

   👍React with 👍1Reacted by Sergey Kostyaev
5. cweill

   added a commit that references this issue on Oct 21, 2025

   feat: add AI test generation foundation (WIP)

   ...

   d0873d4
6. s-kostyaev commented on Oct 23, 2025

   

   s-kostyaev

   on Oct 23, 2025

   Contributor

   More actions

   Why only 3 cases? It's not enough for complex functions with multiple branches. I think it should be customizable. Or 3 cases only for MVP?
7. cweill

   closed this as completedin #194on Oct 23, 2025
8. cweill

   added a commit that references this issue on Oct 23, 2025

   feat: AI-powered test case generation (#194)

   ...

   5e8b5b0
9. cweill commented on Oct 23, 2025

   

   cweill

   on Oct 23, 2025

   Owner

   More actions

   @s-kostyaev Great question. The AI feature supports customizable test case counts. We provide two
   flags:

   • -ai-min-cases (default: 3) - minimum number of test cases to generate
   • -ai-max-cases (default: 10) - maximum number of test cases to generate

   The LLM determines how many test cases to generate within this range based on the function's
   complexity. For complex functions with multiple branches, you can increase these limits:

   gotests -ai -ai-min-cases=5 -ai-max-cases=20 -all path/to/your/file.go

   This allows the AI to generate anywhere from 5-20 test cases depending on what it determines is
   appropriate for the function. The validation limits are:

   • Minimum: at least 1 case
   • Maximum: up to 100 cases

   So yes, it's fully customizable! The default 3-10 range was chosen as a reasonable starting point
   that balances coverage with generation time, but you can adjust it based on your needs.

   👍React with 👍1Reacted by Sergey Kostyaev