Repository navigation
feat(infra): send failures to a dead-letter queue and alert by email - #23
Merged
Merged
Conversation
A match check Scheduler cannot deliver to fetch-results is sent to the queue named in MATCH_CHECK_DEAD_LETTER_QUEUE_ARN instead of being dropped. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
One SQS queue (14 days, SQS-managed encryption, HTTPS only) receives what the planner, fetch-results and notify still fail after Lambda's retries, through an on-failure destination that keeps the event and the error, and what the morning schedule, the match check schedules and the MatchFinished rule cannot deliver to their Lambda. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Three CloudWatch alarms notify the final-whistle-alerts SNS topic when they go to ALARM: a message sent to the failure queue, a match given up while still not found on the page, and no planner run for 26 hours. The topic accepts HTTPS only and CloudWatch alarms of this account only; the email subscription is added by hand, so the address never enters the repository. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
D37 (one failure queue for every asynchronous step), D38 (alerts by email through SNS) and D39 (alarms on failures of the system, not on missing scores). The README and the diagram follow, the diagram also catching up with earlier decisions and marking what is not built yet. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
planner,fetch-resultsandnotifystill fail after Lambda's retries (on-failure destination, which keeps the event and the error), and what the morning schedule, the match check schedules and theMatchFinishedrule cannot deliver to their Lambda (D37).plannerrun for 26 hours (D39).final-whistle-alertsSNS topic, a channel that keeps working when Telegram does not (D38).Notes for review:
cdk diff. Until it is confirmed, the alarms notify nobody.$LATEST, the version invoked today; they move to theprodalias with D15.Closes #7