Skip to content

[BRK] Count the factor pairs present in DevelopmentCorrelation - #1579

Open
scks1 wants to merge 1 commit into
casact:mainfrom
scks1:fix-devcorr-1577
Open

scks1 wants to merge 1 commit into
casact:mainfrom
scks1:fix-devcorr-1577

Conversation

@scks1

@scks1 scks1 commented Nov 10, 2026

Copy link
Copy Markdown
Contributor

Summary of Changes

DevelopmentCorrelation now computes each T_k from the adjacent-factor pairs present in its triangle, weights it by one less than its number of pairs (Mack's description of the weights I − k − 1) and takes Var(T) as one over the total weight, per triangle. Complete triangles are unchanged: all 2,336 in the bundled samples are bit-identical to main. A triangle without a column of two pairs gets NaN and no flag, as #1262 did for empty diagonals.

t_variance is now a DataFrame shaped like t_expectation, and confidence_interval a tuple of two of them. A scalar cannot hold one variance per triangle, hence [BRK] with no deprecation period.

New tests cover a complete triangle, an empty one, an object with one empty member (auto), the late-start raa case from #1577, and 2 × 2 to 4 × 4 triangles.

Related GitHub Issue(s)

Closes #1577.

AI/LLM Usage

This change was made with Árbol, an AI assistant running on Muse, working under my direction. It wrote the fix and the tests, ran the checks below and drafted this description.

Additional Context for Reviewers

  • Below four origins this returns NaN per triangle rather than raising the ValueError proposed in the issue, so one small member does not fail a whole object, as jbogaardt suggested in Independence tests for Chain Ladder  #76.
  • Factors removed by drop or drop_high now count as missing. On raa with the 1982 factor at age 12 dropped, T goes from 0.0696 (main ignores the drop) to 0.1974, flagged at the default p_critical. With drop_high=True it goes from -0.3004 to 0.1966.
  • In clrd, 1,733 of 4,650 triangles change T and 837 become NaN, 219 of them empty.
  • Triangles whose factors never change, such as earned premium, still get T = 1 and a flag. Tied ranks cause this, and I raised them separately in [BUG] DevelopmentCorrelation gives fully tied columns full weight, so triangles that never develop get T = 1 #1578.
  • test_correlation.py passes (29 tests) with the uv.lock versions and with numpy 2.5.4 and pandas 3.1.0, and ruff is clean.
  • This PR conflicts with [FEAT]deprecate date dims #1304, which renames a line this PR deletes, and with [FEAT] Add one-year CDR standard errors to MackChainladder #1558, which adds a fixture at the same place in conftest.py. Taking this side of the first and keeping both fixtures in the second resolves them, and test_correlation.py passes on both merged trees.

Submitter's Checklist

  • I have reviewed and am adhering to the standards outlined in the project Governing Doc.
  • The PR subject title summarizes the changes, with one proper prefix ([FIX], [FEAT], [DOCS], [TST], [CHORE], or [BRK]).
  • I am a human (not a bot), and this PR form is written by a human.

Reviewer's Checklist

  • The implementation addresses the associated issue(s).
  • The implementation is appropriate, maintainable, and follows ARCHITECTURE.md.
  • PR subject title has the proper prefix and the subject is appropriate.
  • Relevant issue(s) are linked.
  • AI/LLM usage is disclosed and appropriate.
  • Documentation and tests are appropriate.
  • CI tests passed, or any failures are acceptable.
  • Leave a comment with the final recommendation (e.g. approve as is, request a secondary review, or flag an area for more review).

DevelopmentCorrelation took Mack's I from the number of development
columns and turned a missing Spearman coefficient into T_k = 1. A
triangle without link ratios got T = 1 and was flagged, and a triangle
with fewer origins holding data than the grid has columns got the wrong
T and variance.

Each T_k now uses the pairs (F_ik, F_i,k-1) present in its triangle:
ranks among those pairs, weight n_k - 1, and Var(T) = 1 / sum(n_k - 1),
per triangle. On a complete triangle n_k = I - k, and the results are
unchanged. A triangle without a column of two pairs gets NaN for T and
its variance and is not flagged, so 2 x 2 and 3 x 3 triangles no longer
raise.

t_variance is now a DataFrame shaped like t_expectation, and
confidence_interval is a tuple of two such DataFrames.

Closes casact#1577.

Developed with an AI assistant (Árbol, running Muse).
@github-actions

Copy link
Copy Markdown

Pyright Type Completeness

View the full pyright --verifytypes output for this commit

Project (full chainladder package, at this PR's head): 16.6% of exported symbols fully typed (265 / 1592)

Known Ambiguous Unknown Total
Project (head) 265 102 1225 1592

Other symbols referenced but not exported by chainladder: 24

Known Ambiguous Unknown Total
Other (head) 11 1 12 24

Symbols without documentation:

  • Functions without docstring: 326
  • Functions without default param: 0
  • Classes without docstring: 10

Patch (exported symbols added or changed by this PR): 0.0% fully typed (0 / 7)

Known Ambiguous Unknown Total
Patch 0 1 6 7
Patch symbol details
Symbol Status Change
chainladder.core.correlation.DevelopmentCorrelation.t ⚠️ ambiguous changed (was ❌ unknown)
chainladder.core.tests.test_correlation.test_dev_corr_complete_triangle_counts ❌ unknown new
chainladder.core.tests.test_correlation.test_dev_corr_counts_origins_with_data ❌ unknown new
chainladder.core.tests.test_correlation.test_dev_corr_empty_member ❌ unknown new
chainladder.core.tests.test_correlation.test_dev_corr_no_data ❌ unknown new
chainladder.core.tests.test_correlation.test_dev_corr_smallest_triangle ❌ unknown new
chainladder.core.tests.test_correlation.test_dev_corr_too_small ❌ unknown new

@codecov

codecov Bot commented Nov 10, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 94.60%. Comparing base (8e2e53c) to head (4817a28).

Additional details and impacted files
@@           Coverage Diff           @@
##             main    #1579   +/-   ##
=======================================
  Coverage   94.59%   94.60%           
=======================================
  Files          98       98           
  Lines        6233     6241    +8     
  Branches      799      800    +1     
=======================================
+ Hits         5896     5904    +8     
  Misses        216      216           
  Partials      121      121           
Flag Coverage Δ
unittests 94.60% <100.00%> (+<0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG] DevelopmentCorrelation reports T = 1 with no data and takes I from the development columns

1 participant