feat(planner): schedule a check per match every morning

README.md

@@ -16,7 +16,7 @@ It is also a side project to practise serverless and event-driven design on a re
 ## How it works
 
 1. **Every morning**, a schedule runs the `planner` Lambda. It reads the day's kick-off times on [Treize Mondial](https://treizemondial.fr/matchs-du-jour-rugby-a-xiii/).
-2. For each match, the planner creates a **one-time schedule** about 2 hours after kick-off. No match, nothing is scheduled.
+2. For each match, the planner creates a **one-time schedule** about 2 hours after kick-off, never at night (23:00 to 08:00). No match, nothing is scheduled.
 3. When a schedule fires, `fetch-results` reads the result through a **source adapter** and publishes a `MatchFinished` event on an EventBridge bus. If the score is not published yet, it reschedules itself one hour later.
 4. `notify` is subscribed to `MatchFinished`. For each event it:
    1. **checks** in DynamoDB that the score was not already sent,
@@ -57,8 +57,9 @@ Work in progress: the diagram shows the target architecture, built one step at a
 
 - Done: TypeScript project, required pull request checks (lint, types, unit tests, commitlint, Gitleaks, CodeQL), Dependabot updates, an empty CDK app synthesized on each pull request with cdk-nag rules, GitHub OIDC roles, a `cdk diff` of each pull request and a `cdk deploy` on each merge to `main`.
 - Done: the Treize Mondial adapter, the event bus and the `fetch-results` Lambda, which publishes `MatchFinished` once a match is finished.
-- Done: the `notify` Lambda, subscribed to `MatchFinished`: it checks DynamoDB, sends the score to Telegram, then records it. Nothing invokes `fetch-results` yet.
-- Next: the planner and its one-time schedules, then the DLQ, the alarms, the smoke test and the `prod` alias rollback.
+- Done: the `notify` Lambda, subscribed to `MatchFinished`: it checks DynamoDB, sends the score to Telegram, then records it.
+- Done: the `planner` Lambda, run every day at 10:00 (Paris time): it schedules one `fetch-results` check per match, about 2 hours after kick-off and never at night. A score not published yet at that check is not looked up again.
+- Next: rescheduling a check when the score is late, then the DLQ, the alarms, the smoke test and the `prod` alias rollback.
 
 Progress is tracked in the [v1 milestone](https://github.com/TomSoulage/final-whistle/milestone/1).
 

docs/decisions.md

@@ -19,6 +19,10 @@ Log of the project's structural choices. Each decision has a stable identifier (
 
 - **D4. A morning planner instead of hourly polling.** One run each morning reads the day's kick-off times and creates one **one-time schedule** per match, about 2 hours after kick-off. Compared with polling every hour: far fewer runs, the source website is read only when a result is expected, and nothing runs on days without matches. Trade-off: a match added to the page after the planner has run is not scheduled.
 - **D5. Reschedule when the score is late.** If the score is not published when the schedule fires, `fetch-results` creates a new schedule one hour later instead of failing.
+- **D31. The planner runs at 10:00, Paris time.** A recurring EventBridge Scheduler schedule (`cron(0 10 * * ? *)`, time zone `Europe/Paris`, so daylight saving is handled by AWS) runs `planner`. By 10:00 the page of the day has had time to fill in, which narrows the trade-off of D4 and the missing kick-off times of D34. A match whose check is already due by then (a night match abroad) is checked a minute after the planner runs. The planner reads the day through the adapter (`listDayMatches`), which fails if the main page is not for the Paris date: the run fails loudly instead of scheduling another day's matches. A match whose schedule cannot be created does not stop the others: each failure is logged with its match id, the run fails at the end, and Lambda's retry finds the other matches already scheduled.
+- **D32. One-time match checks.** Each check is a one-time schedule in the `final-whistle-match-checks` group, set with an `at()` expression in Paris wall-clock time and the `Europe/Paris` time zone, never in the past (at least a minute ahead). It invokes `fetch-results` with the match key (D28) and deletes itself once it has run. Schedule names only allow letters, digits, `-`, `_` and `.`, so the name is `match-` plus 32 hex characters of the SHA-256 of the match id: the same on every run, so running the planner again creates nothing new while the checks are pending (`ConflictException` counts as already scheduled). A check that has already run has deleted itself, so it is created again and the match checked once more; a score already sent is then dropped by `notify` (D8). Scheduler invokes `fetch-results` with its own role, which may do nothing else; `planner` may only create schedules in that group and pass that role to Scheduler. Both Scheduler roles (match checks and the morning schedule) only trust schedules of this account (`aws:SourceAccount`). Two cdk-nag `IAM5` wildcard findings are acknowledged on their resources (D22): schedule names are computed at run time, and the CDK Lambda target also grants the planner's versions and aliases. The morning schedule stays in the default group, out of reach of the planner's rights. Scheduler is in the Always Free quota (D17).
+- **D33. Quiet hours: no check from 23:00 to 08:00, Paris time.** A check due in that window is moved to 08:00, so no score is published, and no notification sent, at night. The rule lives where checks are scheduled, so `notify` stays unaware of it. Limit: a `notify` invocation that fails just before 23:00 is retried by Lambda a few minutes later, so a message can arrive shortly after 23:00.
+- **D34. A missing kick-off time is checked at 17:00** (was P2). When the page gives no kick-off time, the first check is at 17:00: 15:00, the most common kick-off on the saved pages, plus 2 hours. The planner logs a warning, so the frequency of the case is measured. It was absent from the 29 matches of the saved pages.
 
 ### Events and delivery guarantees
 
@@ -70,5 +74,4 @@ Log of the project's structural choices. Each decision has a stable identifier (
 ## To decide
 
 - **P1. Several recipients.** Sending to more than one person (a list of chat ids), with one failure not blocking the others.
-- **P2. Kick-off times missing in the morning.** What to do when a match has no kick-off time yet when the planner runs.
 - **P3. Retry limit.** When to stop rescheduling a match whose score never comes (postponed or abandoned match).

infra/final-whistle-stack.ts

@@ -2,7 +2,9 @@ import { Stack, type StackProps } from 'aws-cdk-lib';
 import { EventBus } from 'aws-cdk-lib/aws-events';
 import type { Construct } from 'constructs';
 import { FetchResultsFunction } from './fetch-results-function.js';
+import { MatchCheckSchedules } from './match-check-schedules.js';
 import { NotifyFunction } from './notify-function.js';
+import { PlannerFunction } from './planner-function.js';
 import { createSentMatchesTable } from './sent-matches-table.js';
 
 export const EVENT_BUS_NAME = 'final-whistle';
@@ -13,7 +15,9 @@ export class FinalWhistleStack extends Stack {
     super(scope, id, props);
     // Created with its producer; consumers subscribe with their own rules (D6).
     const eventBus = new EventBus(this, 'EventBus', { eventBusName: EVENT_BUS_NAME });
-    new FetchResultsFunction(this, 'FetchResults', { eventBus });
+    const fetchResults = new FetchResultsFunction(this, 'FetchResults', { eventBus });
+    const matchChecks = new MatchCheckSchedules(this, 'MatchChecks', { checkFunction: fetchResults.lambdaFunction });
+    new PlannerFunction(this, 'Planner', { matchChecks });
     new NotifyFunction(this, 'Notify', { eventBus, sentMatches: createSentMatchesTable(this) });
   }
 }

infra/match-check-schedules.ts

@@ -0,0 +1,77 @@
+import { ArnFormat, Stack, Validations } from 'aws-cdk-lib';
+import { PolicyStatement, type Role } from 'aws-cdk-lib/aws-iam';
+import type { IFunction } from 'aws-cdk-lib/aws-lambda';
+import { ScheduleGroup } from 'aws-cdk-lib/aws-scheduler';
+import { Construct } from 'constructs';
+import {
+  MATCH_CHECK_FUNCTION_VARIABLE,
+  MATCH_CHECK_GROUP_VARIABLE,
+  MATCH_CHECK_ROLE_VARIABLE,
+} from '../src/scheduling/environment.js';
+import type { PipelineFunction } from './pipeline-function.js';
+import { createSchedulerInvokeRole, SCHEDULER_SERVICE } from './scheduler-invoke-role.js';
+
+const MATCH_CHECK_GROUP_NAME = 'final-whistle-match-checks';
+
+export interface MatchCheckSchedulesProps {
+  /** The fetch-results function each match check invokes. */
+  readonly checkFunction: IFunction;
+}
+
+/**
+ * The one-time match checks (D4): their schedule group, and the role Scheduler invokes fetch-results with.
+ * Lambdas allowed to schedule checks may only create schedules in this group, with this role.
+ */
+export class MatchCheckSchedules extends Construct {
+  /** Where a check goes, for the environment of a Lambda that schedules checks. */
+  readonly environment: Readonly<Record<string, string>>;
+  private readonly invokeRole: Role;
+
+  constructor(scope: Construct, id: string, props: MatchCheckSchedulesProps) {
+    super(scope, id);
+    new ScheduleGroup(this, 'Group', { scheduleGroupName: MATCH_CHECK_GROUP_NAME });
+    this.invokeRole = createSchedulerInvokeRole(
+      this,
+      'InvokeRole',
+      'EventBridge Scheduler: invoke fetch-results for a match check',
+    );
+    this.invokeRole.addToPolicy(
+      new PolicyStatement({ actions: ['lambda:InvokeFunction'], resources: [props.checkFunction.functionArn] }),
+    );
+    this.environment = {
+      [MATCH_CHECK_GROUP_VARIABLE]: MATCH_CHECK_GROUP_NAME,
+      [MATCH_CHECK_FUNCTION_VARIABLE]: props.checkFunction.functionArn,
+      [MATCH_CHECK_ROLE_VARIABLE]: this.invokeRole.roleArn,
+    };
+  }
+
+  /** Allows creating schedules in the group only, and passing the invoke role to Scheduler only. */
+  grantScheduleChecks(pipelineFunction: PipelineFunction): void {
+    const { lambdaFunction } = pipelineFunction;
+    lambdaFunction.addToRolePolicy(
+      new PolicyStatement({ actions: ['scheduler:CreateSchedule'], resources: [this.getSchedulesInGroupArn()] }),
+    );
+    lambdaFunction.addToRolePolicy(
+      new PolicyStatement({
+        actions: ['iam:PassRole'],
+        resources: [this.invokeRole.roleArn],
+        conditions: { StringEquals: { 'iam:PassedToService': SCHEDULER_SERVICE } },
+      }),
+    );
+    Validations.of(pipelineFunction).acknowledge({
+      id: `AwsSolutions::AwsSolutions-IAM5[Resource::arn:<AWS::Partition>:scheduler:<AWS::Region>:<AWS::AccountId>:schedule/${MATCH_CHECK_GROUP_NAME}/*]`,
+      reason:
+        'Each match check gets its own schedule, named at run time from the match id, so the names cannot be ' +
+        'listed in advance; the wildcard is limited to the match check group and to CreateSchedule.',
+    });
+  }
+
+  private getSchedulesInGroupArn(): string {
+    return Stack.of(this).formatArn({
+      service: 'scheduler',
+      resource: 'schedule',
+      resourceName: `${MATCH_CHECK_GROUP_NAME}/*`,
+      arnFormat: ArnFormat.SLASH_RESOURCE_NAME,
+    });
+  }
+}

infra/planner-function.ts

@@ -0,0 +1,53 @@
+import { fileURLToPath } from 'node:url';
+import { Stack, TimeZone, Validations } from 'aws-cdk-lib';
+import type { CfnFunction } from 'aws-cdk-lib/aws-lambda';
+import { Schedule, ScheduleExpression } from 'aws-cdk-lib/aws-scheduler';
+import { LambdaInvoke } from 'aws-cdk-lib/aws-scheduler-targets';
+import type { Construct } from 'constructs';
+import type { MatchCheckSchedules } from './match-check-schedules.js';
+import { PipelineFunction } from './pipeline-function.js';
+import { createSchedulerInvokeRole } from './scheduler-invoke-role.js';
+
+const HANDLER_ENTRY = fileURLToPath(new URL('../src/planner/handler.ts', import.meta.url));
+// One page at 10 s, plus one CreateSchedule call per match (about fifteen on a busy day).
+const TIMEOUT_SECONDS = 30;
+// Same base as fetch-results (Node runtime, AWS SDK client, one page parsed).
+const MEMORY_SIZE_MB = 256;
+/** 10:00 in Paris: the page of the day has had time to fill in, and quiet hours are over (D4). */
+const PLANNER_HOUR = '10';
+
+export interface PlannerFunctionProps {
+  readonly matchChecks: MatchCheckSchedules;
+}
+
+/** The planner Lambda, run every morning: it may only read the page and schedule match checks. */
+export class PlannerFunction extends PipelineFunction {
+  constructor(scope: Construct, id: string, props: PlannerFunctionProps) {
+    super(scope, id, {
+      entry: HANDLER_ENTRY,
+      roleDescription: 'planner Lambda: write its logs and schedule match checks',
+      timeoutSeconds: TIMEOUT_SECONDS,
+      memorySizeMb: MEMORY_SIZE_MB,
+      environment: props.matchChecks.environment,
+    });
+    props.matchChecks.grantScheduleChecks(this);
+    this.addMorningSchedule();
+  }
+
+  /** In the default group, so the planner's rights on the match check group never reach its own trigger. */
+  private addMorningSchedule(): void {
+    const role = createSchedulerInvokeRole(this, 'MorningScheduleRole', 'EventBridge Scheduler: run the planner');
+    new Schedule(this, 'MorningSchedule', {
+      description: 'Run the planner every morning (Paris time)',
+      schedule: ScheduleExpression.cron({ minute: '0', hour: PLANNER_HOUR, timeZone: TimeZone.EUROPE_PARIS }),
+      target: new LambdaInvoke(this.lambdaFunction, { role }),
+    });
+    const functionLogicalId: unknown = Stack.of(this).resolve((this.lambdaFunction.node.defaultChild as CfnFunction).logicalId);
+    Validations.of(role).acknowledge({
+      id: `AwsSolutions::AwsSolutions-IAM5[Resource::<${String(functionLogicalId)}.Arn>:*]`,
+      reason:
+        'The CDK Lambda target grants invoke on the planner and its versions and aliases (`:*`): ' +
+        'the wildcard never reaches another function.',
+    });
+  }
+}

infra/scheduler-invoke-role.ts

@@ -0,0 +1,15 @@
+import { Stack } from 'aws-cdk-lib';
+import { Role, ServicePrincipal } from 'aws-cdk-lib/aws-iam';
+import type { Construct } from 'constructs';
+
+export const SCHEDULER_SERVICE = 'scheduler.amazonaws.com';
+
+/** A role for EventBridge Scheduler that only schedules of this account may use (confused deputy). */
+export function createSchedulerInvokeRole(scope: Construct, id: string, description: string): Role {
+  return new Role(scope, id, {
+    assumedBy: new ServicePrincipal(SCHEDULER_SERVICE, {
+      conditions: { StringEquals: { 'aws:SourceAccount': Stack.of(scope).account } },
+    }),
+    description,
+  });
+}

package.json

@@ -37,6 +37,7 @@
   "dependencies": {
     "@aws-sdk/client-dynamodb": "3.1156.0",
     "@aws-sdk/client-eventbridge": "3.1156.0",
+    "@aws-sdk/client-scheduler": "3.1156.0",
     "@aws-sdk/client-ssm": "3.1156.0",
     "node-html-parser": "9.0.5"
   }

src/adapters/treize-mondial/parse-cells.ts

@@ -32,6 +32,11 @@ export function parsePageDay(text: string): PageDay | null {
   return { day: Number(groups.day), month: Number(groups.month) };
 }
 
+/** `{ day: 31, month: 10 }` → `31/10`, as in error messages. */
+export function formatPageDay(day: PageDay): string {
+  return `${String(day.day)}/${String(day.month)}`;
+}
+
 /** `5H40` → `05:40`; anything else (postponed, not set yet) → null. */
 export function parseKickOff(text: string): string | null {
   const groups = KICK_OFF_PATTERN.exec(text)?.groups;

src/adapters/treize-mondial/parse-matches-page.ts

@@ -1,6 +1,7 @@
 import { parse, type HTMLElement } from 'node-html-parser';
 import type { FinalScore } from '../../results/match-result.js';
 import {
+  formatPageDay,
   getCellText,
   isFinishedMark,
   parseKickOff,
@@ -55,10 +56,6 @@ function getRowPageDay(cells: Row): PageDay | null {
   return cells.map((cell) => parsePageDay(getCellText(cell))).find((day) => day !== null) ?? null;
 }
 
-function formatPageDay(day: PageDay): string {
-  return `${String(day.day)}/${String(day.month)}`;
-}
-
 function getRowCompetition(cells: Row): string | null {
   const [cell] = cells;
   return cells.length === COMPETITION_ROW_CELLS && cell !== undefined ? getCellText(cell) : null;

src/adapters/treize-mondial/treize-mondial-adapter.ts

@@ -1,6 +1,7 @@
-import type { MatchKey, MatchLookup } from '../../results/match-result.js';
+import type { DayMatch, MatchKey, MatchLookup } from '../../results/match-result.js';
 import { fetchMatchesPage } from './fetch-matches-page.js';
 import { findMatchResult, isPageForDate } from './find-match-result.js';
+import { formatPageDay } from './parse-cells.js';
 import { parseMatchesPage, type MatchesPage } from './parse-matches-page.js';
 
 export const TODAY_PAGE_URL = 'https://treizemondial.fr/matchs-du-jour-rugby-a-xiii/';
@@ -18,6 +19,18 @@ export async function getMatchResult(key: MatchKey): Promise<MatchLookup> {
   return page === null ? { status: 'not-found' } : findMatchResult(page, key);
 }
 
+/** Every match of the main page, which must be the page for `isoDate` (YYYY-MM-DD). */
+export async function listDayMatches(isoDate: string): Promise<DayMatch[]> {
+  const page = parsePage(TODAY_PAGE_URL, await fetchMatchesPage(TODAY_PAGE_URL));
+  if (!isPageForDate(page.day, isoDate)) {
+    throw new Error(`Treize Mondial page ${TODAY_PAGE_URL} shows ${formatPageDay(page.day)}, not ${isoDate}`);
+  }
+  return page.matches.map(({ homeTeam, awayTeam, kickOff }) => ({
+    key: { date: isoDate, homeTeam, awayTeam },
+    kickOff,
+  }));
+}
+
 /** Reads the pages in order and stops at the first one for the date; most calls need one request. */
 async function readPageForDate(isoDate: string): Promise<MatchesPage | null> {
   for (const url of PAGE_URLS) {

src/planner/handler.ts

@@ -0,0 +1,34 @@
+import { SchedulerClient } from '@aws-sdk/client-scheduler';
+import { listDayMatches } from '../adapters/treize-mondial/treize-mondial-adapter.js';
+import type { MatchKey } from '../results/match-result.js';
+import {
+  MATCH_CHECK_FUNCTION_VARIABLE,
+  MATCH_CHECK_GROUP_VARIABLE,
+  MATCH_CHECK_ROLE_VARIABLE,
+} from '../scheduling/environment.js';
+import { scheduleMatchCheck, type MatchCheckTarget } from '../scheduling/match-schedule.js';
+import type { ParisWallClock } from '../scheduling/paris-time.js';
+import { getRequiredEnv } from '../shared/get-required-env.js';
+import { createLogger, type LogFields } from '../shared/logger.js';
+import { planDay, type DayPlan } from './plan-day.js';
+
+const SERVICE_NAME = 'planner';
+
+/** Read once, at cold start. */
+const matchCheckTarget: MatchCheckTarget = {
+  scheduleGroupName: getRequiredEnv(MATCH_CHECK_GROUP_VARIABLE),
+  functionArn: getRequiredEnv(MATCH_CHECK_FUNCTION_VARIABLE),
+  roleArn: getRequiredEnv(MATCH_CHECK_ROLE_VARIABLE),
+};
+const scheduler = new SchedulerClient({});
+
+/** planner Lambda: only wires the adapter, Scheduler, the clock and the logs to `planDay`. */
+export async function handler(): Promise<DayPlan> {
+  return planDay({
+    listDayMatches,
+    scheduleMatchCheck: (key: MatchKey, checkAt: ParisWallClock) =>
+      scheduleMatchCheck((command) => scheduler.send(command), matchCheckTarget, key, checkAt),
+    createLogger: (context: LogFields) => createLogger({ service: SERVICE_NAME, ...context }),
+    now: () => new Date(),
+  });
+}