Feat: improve db efficiency - #305
Merged
Merged
Conversation
…hared memory management
The rationale for shared_buffers, work_mem and the /dev/shm sizing belongs in helm/README.md, which already carries it. A values file is read to find a knob, not to learn why the knob exists, and thirty lines of prose between two settings makes it harder to do that. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The seen-paths distinctness set had reached 2.9M members and ~94MB in Redis -- essentially the whole instance -- to produce a single dashboard integer. It could never stop growing: the set is keyed on request paths, and Krawl generates random link paths, so the key space has no ceiling. The standalone backend already capped itself at 100k entries for exactly this reason; scalable mode inherited the risk without the mitigation. PFADD/PFCOUNT hold that in a fixed ~12KB with 0.81% standard error. Measured at 50k members: 2.58MB -> 14.1KB, 188x smaller. unique_paths is no longer event-incremented in scalable mode. PFADD reports whether the registers changed, not whether the member was new, and it does return 1 for members already present -- incrementing on that drifts the counter upward with no bound between reconciles. The counter is instead realigned wholesale from PFCOUNT on the metrics flush, which is O(1) since Redis caches the cardinality in the HLL header. migrate_legacy_sets() folds any existing set into the estimator and UNLINKs it, so deployments reclaim the memory on their next start. It runs outside the needs_seed() gate deliberately: installs old enough to have the legacy set already hold the seed marker, so a gated migration would never fire on the ones that need it. get_all() had to learn the new prefix as well -- it scans krawl:counter:* and calls int() on every value, which a HyperLogLog blob would have broken, taking the /metrics scrape down with it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Redis shipped with maxmemory unset and the default noeviction policy, so it grew unbounded and then failed writes rather than shedding load -- the same OOM shape the bundled PostgreSQL had. A live instance was sitting at 94MB with a 129MB peak against a 128Mi container limit. volatile-lru is the policy that matches how the keyspace is used: every krawl:cache:* entry is written with an expiry and is recomputable, while the krawl:counter:* metrics are written without one and are not. Evicting only keys that carry a TTL sheds cache under pressure and can never drop a counter. allkeys-lru would silently corrupt the metrics. The command block was previously emitted only when a password was set, so password-less deployments had no way to receive server flags at all. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The aggregate caches were stored as one JSON blob and sliced in Python, so serving any ten-row page pulled the entire aggregate over the wire and parsed it: agg:map_ips holds up to 50k IP rows, agg:attackers and agg:top_paths are sized at limit=100_000. Storing them as Redis LISTs lets LRANGE fetch just the requested window. Measured on a realistic 50k-row agg:map_ips (7.3MB): 391ms -> 12.4ms per request, 32x faster. Writes stage under a temporary key and RENAME into place, so a reader during a warmup refresh sees the previous list rather than a partially populated one. Standalone keeps the plain in-memory list -- there is no serialization boundary to cross there, so slicing it directly was already optimal. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
BlessedRebuS
approved these changes
Sep 1, 2026
Contributor
Ruff — lint & security issuesNo issues found. |
Contributor
Bandit — security issuesNo HIGH severity issues found. View full report |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This pull request introduces several improvements and optimizations to Krawl's database maintenance, metrics tracking, and Helm chart configuration. The most significant changes include the introduction of a PostgreSQL statistics maintenance module, migration of unique-path tracking to HyperLogLog for efficiency, and enhanced configurability and resource tuning for PostgreSQL and Redis in Kubernetes deployments.
Database maintenance and metrics improvements:
database/maintenance.pymodule that ensures PostgreSQL planner statistics are collected at critical times (on first boot and after retention purges), preventing poor query plans and performance regressions after restarts or bulk deletes. This includes thebootstrap_analyzeandanalyze_tablesfunctions. [1] [2]DatabaseManager.engineproperty for direct access to the SQLAlchemy engine, enabling operations likeANALYZEthat require connection-level control.SETfor unique paths with a fixed-size HyperLogLog estimator, drastically reducing memory usage while maintaining approximate counts. Includes migration logic for legacy sets. [1] [2] [3] [4] [5]Helm chart and Kubernetes resource/configuration enhancements:
postgres.config, increased default memory limits and requests, introducedpostgres.shmSizefor tuning/dev/shm(critical for parallel vacuum), and documented recommended settings for busy sensors. [1] [2] [3] [4] [5] [6]maxmemoryandmaxmemoryPolicyto prevent OOM kills and control eviction behavior, and increased default memory limits. [1] [2] [3]Dashboard cache optimization:
These changes improve Krawl's performance, scalability, and maintainability, especially in large-scale or Kubernetes environments.