refactor: cleanup pass — sqlc, slog, scraper registry, drop go-jet

.github/workflows/build.yml

@@ -75,7 +75,7 @@ jobs:
       - name: Set up Go
         uses: actions/setup-go@v4
         with:
-          go-version: "1.21"
+          go-version: "1.24"
           check-latest: true
           cache: false
 
@@ -91,7 +91,7 @@ jobs:
           GOARCH: ${{ matrix.arch }}
           CGO_ENABLED: 0
         run: |
-          go build -ldflags="-s -w" -o ${{ matrix.binary_name }} ./cmd/aproxy
+          go build -ldflags="-s -w -X main.Version=${{ github.ref_name }}" -o ${{ matrix.binary_name }} ./cmd/aproxy
 
       - name: Upload artifact
         uses: actions/upload-artifact@v4

CLAUDE.md

@@ -65,10 +65,10 @@ docker-compose up
 
 ### Core Components
 
-- **Scraper** (`pkg/scraper/`): Fetches proxy lists from multiple sources (ProxyScrape, FreeProxyList, Geonode, ProxyListOrg, GitHub)
+- **Scraper** (`pkg/scraper/`): Fetches proxy lists from multiple sources (ProxyScrape, FreeProxyList, ProxyListOrg, GitHub)
 - **Checker** (`pkg/checker/`): Validates proxy health with SQLite caching, intelligent check intervals, and unified logging
 - **Manager** (`pkg/manager/`): Manages proxy pool with database persistence, in-memory cache, and auto-refresh
-- **Database** (`internal/database/`): SQLite-based persistent storage with Jet ORM for type-safe queries
+- **Database** (`internal/database/`): SQLite-based persistent storage with sqlc-generated type-safe queries
 - **Proxy Server** (`pkg/proxy/`): HTTP/HTTPS proxy server with privacy features
 - **Config** (`internal/config/`): Advanced configuration management with Viper and validation support
 
@@ -121,10 +121,9 @@ AProxy uses **Viper** for advanced configuration management with validation:
 3. **Config files**: YAML, JSON, TOML supported (searches `./`, `./config/`, `/etc/aproxy/`)
 4. **Defaults**: Sensible defaults for all settings
 
-**Supported Scraper Sources:**
+**Supported Scraper Sources:** (all are plain `host:port` / `proto://host:port` text lists)
 - `proxyscrape`: ProxyScrape API
 - `freeproxylist`: FreeProxyList scraper
-- `geonode`: Geonode API scraper  
 - `proxylistorg`: ProxyListOrg scraper
 - `github`: GitHub proxy list scraper (proxifly/free-proxy-list)
 
@@ -180,12 +179,13 @@ curl -x http://localhost:8080 \
 - `checker.test_url`: URL used to test proxy health (default: `http://icanhazip.com`)
 
 **Scraper Configuration Options:**
-- `scraper.sources`: List of proxy sources to use (default: `["proxyscrape", "freeproxylist", "geonode", "github"]`)
+- `scraper.sources`: List of proxy sources to use (default: `["proxyscrape", "freeproxylist", "github"]`)
 - `scraper.timeout`: Request timeout for scraping (default: `30s`)
 - `scraper.user_agent`: User agent string for scraper requests
 
 **Logging Configuration:**
-- Currently logs to stdout only in JSON format
+- Logs to stdout as JSON via `log/slog` (each line carries `component`, and `id` for correlated operations)
+- Level via `LOG_LEVEL` env var (`debug`/`info`/`warn`/`error`, default `info`)
 - Log level can be controlled via command line or environment variables
 - File-based logging is not yet implemented
 
@@ -215,45 +215,39 @@ The SQLite database includes:
 - **Indexes**: Optimized for fast lookups by host:port, status, and timestamps
 - **Automatic cleanup**: Removes old unhealthy proxies based on configuration
 
-## Recent Improvements
-
-### GitHub Proxy Scraper (v1.2)
-- **New GitHub source**: Added scraper for proxifly/free-proxy-list GitHub repository
-- **Enhanced source variety**: Now supports 5 different proxy sources for better diversity
-- **Configuration validation**: Added `github` to allowed scraper sources in config validation
-
-### Logging System Improvements (v1.2)
-- **Unified logging**: Standardized all checker logging to use internal logger package
-- **Reduced verbosity**: Removed verbose individual proxy failure debugging output
-- **Consistent log levels**: Proper use of InfoBg/WarnBg throughout checker components
-- **Better performance**: Less logging overhead during proxy health checks
-
-### Configuration System (v1.1)
-- **Migrated to Viper**: Replaced manual config parsing with Viper library
-- **Added validation**: All config values validated using `go-playground/validator`
-- **YAML support**: Config files now use YAML format (JSON/TOML also supported)
-- **Better error messages**: Detailed validation errors with field names and constraints
-
-### Docker Support
-- **Production-ready**: Multi-stage Docker build with Alpine Linux
-- **Security**: Non-root user, minimal attack surface
-- **Health checks**: Proper health endpoint with curl-based Docker healthchecks
-- **Persistent volumes**: Database and logs stored in `./data/` for volume mounting
-- **Resource limits**: CPU and memory constraints in docker-compose
-
-### Performance Optimizations
-- **Non-blocking architecture**: Server starts immediately, proxy checking happens in background
-- **Progressive checking**: Proxies checked in small batches with delays to reduce system load
-- **Intelligent caching**: Only checks proxies older than 10 minutes, persists all results including failures
-- **Fixed race conditions**: HTTPS CONNECT bidirectional copying now uses channel coordination
-- **Batch database updates**: Replaced concurrent individual updates with single transaction batches
-- **Removed GitHub sources**: Eliminated high-volume proxy sources to reduce database load
-- **Improved caching**: Better SQLite connection pooling and prepared statements
+## Extending the Codebase
+
+### Adding a new proxy source
+Most sources are plain text lists, so you don't write a new file — add a row to the `sources` registry in `pkg/scraper/list.go`:
+1. Append a `source{name, urls, defaultType}` to the `sources` slice in `pkg/scraper/list.go`. `parseLine` already handles both `proto://host:port` and bare `host:port` lines.
+2. Add `<name>` to the `oneof=...` validator tag on `Scraper.Sources` in `internal/config/config.go` (otherwise config validation rejects it).
+3. Optionally add it to the `scraper.sources` default list in `setDefaults` (`internal/config/config.go`).
+
+For a non-text source (custom JSON API, etc.), implement the `Scraper` interface (`pkg/scraper/types.go`) in its own file and append an instance in `NewMultiScraperWithConfig`. `MultiScraper.ScrapeAll` runs all configured sources, dedups, and aggregates; the manager hands the result to the checker.
+
+### Database queries are sqlc-generated
+- The data layer uses **sqlc** (`sqlc.yaml`). SQL lives in `internal/database/schema.sql` (table defs) and `internal/database/query.sql` (named queries). Run `sqlc generate` to regenerate `internal/database/db/` — do not hand-edit those generated files.
+- `internal/database/service.go` wraps the generated `db.Queries` with the methods the app calls. The one hand-written query is `GetProxiesByAddresses` (sqlc's sqlite engine can't do `sqlc.slice()` IN-lists).
+- ⚠️ The schema lives in **two** places that must stay in sync: `schema.sql` (read by sqlc) and the inline string in `db.go`'s `initSchema()` (run at startup). Change both.
+
+### Logging conventions
+Use the internal `logger` package (`internal/logger/logger.go`), not the standard `log`, for component logging:
+- `logger.New("<component>")` creates a component-scoped logger.
+- ID-tagged methods (`Info`/`Warn`/`Error`/`Debug`) take a request/operation ID from `logger.GenerateID()` to correlate a multi-step operation's log lines.
+- `*Bg` variants (`InfoBg`/`WarnBg`/`ErrorBg`/`DebugBg`) are for background/non-correlated operations — used throughout the checker.
+
+## Request Flow
+
+1. `main.go` loads config (Viper) → opens SQLite DB → builds `scraper.ScraperConfig` and `checker.CheckerConfig`.
+2. `manager.NewDBManagerWithConfig` wires together the multi-scraper, the DB-backed checker, and the in-memory cache; `mgr.Start(updateInterval)` launches background refresh.
+3. `RefreshProxies` (manager) calls `scraper.ScrapeAll` → `dbChecker.CheckProxiesWithCaching` (skips proxies checked within `check_interval`) → persists results to SQLite → reloads the healthy in-memory cache.
+4. `proxy.NewServer(mgr, cfg)` serves requests; `GetNextProxy`/`GetRandomProxy` pull from the cache, `ReportProxyFailure` feeds failures back.
+5. Server starts immediately on cached proxies — the pool fills progressively in the background (non-blocking startup).
 
 ### Key Dependencies
 - `github.com/spf13/viper`: Configuration management
 - `github.com/go-playground/validator/v10`: Configuration validation
-- `github.com/go-jet/jet/v2`: Type-safe SQL queries
+- `sqlc` (build-time, not a runtime dep): generates the type-safe query layer in `internal/database/db/`
 - `modernc.org/sqlite`: Pure Go SQLite driver
 
 ## Development Notes

cmd/aproxy/main.go

@@ -4,18 +4,16 @@ import (
 	"context"
 	"flag"
 	"fmt"
-	"log"
 	"os"
 	"os/signal"
 	"syscall"
 	"time"
 
 	"aproxy/internal/config"
 	"aproxy/internal/database"
-	"aproxy/pkg/checker"
+	"aproxy/internal/logger"
 	"aproxy/pkg/manager"
 	"aproxy/pkg/proxy"
-	"aproxy/pkg/scraper"
 )
 
 var (
@@ -24,9 +22,8 @@ var (
 	version    = flag.Bool("version", false, "Show version")
 )
 
-var (
-	Version = "1.0.0"
-)
+// Version is set at build time via -ldflags "-X main.Version=...".
+var Version = "dev"
 
 const (
 	Banner = `
@@ -44,7 +41,7 @@ ______ ______ ______ ______ ______ ______ ______ ______
                                                ░ ░     
 ______ ______ ______ ______ ______ ______ ______ ______
 
-AProxy - Anonymous Proxy Server v%s
+AProxy - Anonymous Proxy Server %s
 https://github.com/ArnabXD/aproxy
 
 ______ ______ ______ ______ ______ ______ ______ ______
@@ -56,84 +53,58 @@ func main() {
 	flag.Parse()
 
 	if *version {
-		fmt.Printf("AProxy v%s\n", Version)
+		fmt.Printf("AProxy %s\n", Version)
 		return
 	}
 
 	fmt.Printf(Banner, Version)
 
+	log := logger.New("main")
+
 	if *genConfig {
 		if err := config.SaveConfigTemplate("config.yaml"); err != nil {
-			log.Fatalf("Failed to generate config: %v", err)
+			log.Fatal("Failed to generate config: %v", err)
 		}
 		fmt.Println("Default config generated: config.yaml")
 		return
 	}
 
 	cfg, err := config.LoadConfig(*configPath)
 	if err != nil {
-		log.Fatalf("Failed to load config: %v", err)
+		log.Fatal("Failed to load config: %v", err)
 	}
 
-	log.Printf("Starting AProxy v%s", Version)
+	log.InfoBg("Starting AProxy %s", Version)
 	config.PrintConfig(cfg)
 
 	// Initialize database
 	db, err := database.NewDB(cfg.Database.Path)
 	if err != nil {
-		log.Fatalf("Failed to initialize database: %v", err)
+		log.Fatal("Failed to initialize database: %v", err)
 	}
 	defer db.Close()
 
-	// Create configuration objects for checker and scraper
-	scraperConfig := scraper.ScraperConfig{
-		Timeout:   cfg.Scraper.Timeout,
-		UserAgent: cfg.Scraper.UserAgent,
-		Sources:   cfg.Scraper.Sources,
-	}
-
-	checkerConfig := checker.CheckerConfig{
-		TestURL:    cfg.Checker.TestURL,
-		Timeout:    cfg.Checker.Timeout,
-		MaxWorkers: cfg.Checker.MaxWorkers,
-		UserAgent:  cfg.Checker.UserAgent,
-	}
-
-	// Use database manager with configuration
-	mgr := manager.NewDBManagerWithConfig(db, scraperConfig, checkerConfig, cfg.Checker.CheckInterval, cfg.Checker.BackgroundEnabled, cfg.Checker.BatchSize, cfg.Checker.BatchDelay)
+	mgr := manager.NewDBManager(db, cfg)
 	if err := mgr.Start(cfg.Proxy.UpdateInterval); err != nil {
-		log.Fatalf("Failed to start proxy manager: %v", err)
-	}
-
-	proxyConfig := &proxy.Config{
-		ListenAddr:     cfg.Server.ListenAddr,
-		ReadTimeout:    cfg.Server.ReadTimeout,
-		WriteTimeout:   cfg.Server.WriteTimeout,
-		IdleTimeout:    cfg.Server.IdleTimeout,
-		MaxConnections: cfg.Server.MaxConnections,
-		EnableHTTPS:    cfg.Server.EnableHTTPS,
-		MaxRetries:     cfg.Server.MaxRetries,
-		StripHeaders:   cfg.Server.StripHeaders,
-		AddHeaders:     cfg.Server.AddHeaders,
-		AuthToken:      cfg.Server.AuthToken,
+		log.Fatal("Failed to start proxy manager: %v", err)
 	}
 
-	server := proxy.NewServer(mgr, proxyConfig)
+	server := proxy.NewServer(mgr, cfg.Server)
 
 	go func() {
 		if err := server.Start(); err != nil {
-			log.Printf("Server error: %v", err)
+			log.ErrorBg("Server error: %v", err)
 		}
 	}()
 
-	log.Printf("Proxy server started on %s", cfg.Server.ListenAddr)
-	log.Println("Press Ctrl+C to stop")
+	log.InfoBg("Proxy server started on %s", cfg.Server.ListenAddr)
+	log.InfoBg("Press Ctrl+C to stop")
 
 	c := make(chan os.Signal, 1)
 	signal.Notify(c, os.Interrupt, syscall.SIGTERM)
 
 	<-c
-	log.Println("Shutting down...")
+	log.InfoBg("Shutting down...")
 
 	// Stop the manager first to cancel background operations
 	mgr.Stop()
@@ -143,8 +114,8 @@ func main() {
 	defer cancel()
 
 	if err := server.Stop(ctx); err != nil {
-		log.Printf("Server shutdown error: %v", err)
+		log.ErrorBg("Server shutdown error: %v", err)
 	}
 
-	log.Println("Shutdown complete")
+	log.InfoBg("Shutdown complete")
 }

config.example.yaml

@@ -31,7 +31,7 @@ scraper:
   sources:
     - "proxyscrape"
     - "freeproxylist"
-    - "geonode"
+    - "github"
 
 checker:
   test_url: "http://icanhazip.com"

go.mod

@@ -3,34 +3,31 @@ module aproxy
 go 1.24.1
 
 require (
-	github.com/go-jet/jet/v2 v2.13.0
 	github.com/go-playground/validator/v10 v10.26.0
 	github.com/spf13/viper v1.20.1
 	golang.org/x/net v0.38.0
 	modernc.org/sqlite v1.38.0
 )
 
 require (
-	github.com/davecgh/go-spew v1.1.1 // indirect
 	github.com/dustin/go-humanize v1.0.1 // indirect
 	github.com/fsnotify/fsnotify v1.8.0 // indirect
 	github.com/gabriel-vasile/mimetype v1.4.8 // indirect
 	github.com/go-playground/locales v0.14.1 // indirect
 	github.com/go-playground/universal-translator v0.18.1 // indirect
 	github.com/go-viper/mapstructure/v2 v2.3.0 // indirect
+	github.com/google/go-cmp v0.7.0 // indirect
 	github.com/google/uuid v1.6.0 // indirect
 	github.com/leodido/go-urn v1.4.0 // indirect
 	github.com/mattn/go-isatty v0.0.20 // indirect
 	github.com/ncruces/go-strftime v0.1.9 // indirect
 	github.com/pelletier/go-toml/v2 v2.2.3 // indirect
-	github.com/pmezard/go-difflib v1.0.0 // indirect
 	github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec // indirect
 	github.com/sagikazarmark/locafero v0.7.0 // indirect
 	github.com/sourcegraph/conc v0.3.0 // indirect
 	github.com/spf13/afero v1.12.0 // indirect
 	github.com/spf13/cast v1.7.1 // indirect
 	github.com/spf13/pflag v1.0.6 // indirect
-	github.com/stretchr/testify v1.10.0 // indirect
 	github.com/subosito/gotenv v1.6.0 // indirect
 	go.uber.org/atomic v1.9.0 // indirect
 	go.uber.org/multierr v1.9.0 // indirect

go.sum

@@ -9,8 +9,6 @@ github.com/fsnotify/fsnotify v1.8.0 h1:dAwr6QBTBZIkG8roQaJjGof0pp0EeF+tNV7YBP3F/
 github.com/fsnotify/fsnotify v1.8.0/go.mod h1:8jBTzvmWwFyi3Pb8djgCCO5IBqzKJ/Jwo8TRcHyHii0=
 github.com/gabriel-vasile/mimetype v1.4.8 h1:FfZ3gj38NjllZIeJAmMhr+qKL8Wu+nOoI3GqacKw1NM=
 github.com/gabriel-vasile/mimetype v1.4.8/go.mod h1:ByKUIKGjh1ODkGM1asKUbQZOLGrPjydw3hYPU2YU9t8=
-github.com/go-jet/jet/v2 v2.13.0 h1:DcD2IJRGos+4X40IQRV6S6q9onoOfZY/GPdvU6ImZcQ=
-github.com/go-jet/jet/v2 v2.13.0/go.mod h1:YhT75U1FoYAxFOObbQliHmXVYQeffkBKWT7ZilZ3zPc=
 github.com/go-playground/assert/v2 v2.2.0 h1:JvknZsQTYeFEAhQwI4qEt9cyV5ONwRHC+lYKSsYSR8s=
 github.com/go-playground/assert/v2 v2.2.0/go.mod h1:VDjEfimB/XKnb+ZQfWdccd7VUvScMdVu0Titje2rxJ4=
 github.com/go-playground/locales v0.14.1 h1:EWaQ/wswjilfKLTECiXz7Rh+3BjFhfDFKv/oXslEjJA=