Skip to content

feat(infra): send failures to a dead-letter queue and alert by email - #23

Merged
TomSoulage merged 5 commits into
mainfrom
feat/dlq-and-alarms
Nov 1, 2026
Merged

TomSoulage merged 5 commits into
mainfrom
feat/dlq-and-alarms

Conversation

@TomSoulage

@TomSoulage TomSoulage commented Nov 1, 2026 •

Copy link
Copy Markdown
Owner
  • One SQS dead-letter queue for every asynchronous step: what planner, fetch-results and notify still fail after Lambda's retries (on-failure destination, which keeps the event and the error), and what the morning schedule, the match check schedules and the MatchFinished rule cannot deliver to their Lambda (D37).
  • Three CloudWatch alarms on failures of the system, not on missing scores: a message sent to the queue, a match given up while still not found on the page, and no planner run for 26 hours (D39).
  • Alerts go by email through the final-whistle-alerts SNS topic, a channel that keeps working when Telegram does not (D38).
  • D37–D39, the README and the architecture diagram, which now also catches up with earlier decisions and marks what is not built yet.

Notes for review:

  • The email subscription is not in the stack: it is added by hand once the topic is deployed, so the address never appears in the repository or in a cdk diff. Until it is confirmed, the alarms notify nobody.
  • The on-failure destinations are set on $LATEST, the version invoked today; they move to the prod alias with D15.
  • Two points are verified only after deployment: that the failure queue alarm counts the messages sent by Lambda, EventBridge and Scheduler (a real failed async invocation), and that the alarms can publish on the topic.
  • D39 assumes a postponed match stays on the Treize Mondial page without a score; no saved page shows one yet.

Closes #7

TomSoulage and others added 5 commits November 1, 2026 16:38
A match check Scheduler cannot deliver to fetch-results is sent to the
queue named in MATCH_CHECK_DEAD_LETTER_QUEUE_ARN instead of being dropped.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
One SQS queue (14 days, SQS-managed encryption, HTTPS only) receives what
the planner, fetch-results and notify still fail after Lambda's retries,
through an on-failure destination that keeps the event and the error, and
what the morning schedule, the match check schedules and the MatchFinished
rule cannot deliver to their Lambda.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Three CloudWatch alarms notify the final-whistle-alerts SNS topic when they
go to ALARM: a message sent to the failure queue, a match given up while
still not found on the page, and no planner run for 26 hours. The topic
accepts HTTPS only and CloudWatch alarms of this account only; the email
subscription is added by hand, so the address never enters the repository.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
D37 (one failure queue for every asynchronous step), D38 (alerts by email
through SNS) and D39 (alarms on failures of the system, not on missing
scores). The README and the diagram follow, the diagram also catching up
with earlier decisions and marking what is not built yet.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@TomSoulage TomSoulage added documentation Improvements or additions to documentation feature New feature or improvement infra CDK, IAM and AWS resources app Lambda code: adapters, planner, fetch-results, notify labels Nov 1, 2026
@TomSoulage
TomSoulage merged commit 03d8e0d into main Nov 1, 2026
11 checks passed
@TomSoulage
TomSoulage deleted the feat/dlq-and-alarms branch November 1, 2026 15:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

app Lambda code: adapters, planner, fetch-results, notify documentation Improvements or additions to documentation feature New feature or improvement infra CDK, IAM and AWS resources

Projects

None yet

Development

Successfully merging this pull request may close these issues.

DLQ and alarms

1 participant