Repository navigation
[FEAT] Add one-year CDR standard errors to MackChainladder - #1558
Conversation
Add cdr_std_err_ and total_cdr_std_err_, the standard error of the one-year claims development result by origin and in total, from Results 3.1 and 3.3 of Merz and Wüthrich (2008). Both are properties computed when read, so fit is unchanged. Reading them raises ValueError for a triangle that is not square, a tail, non-volume averages, excluded link ratios or factors other than the triangle's own; a triangle with a missing cell above the latest diagonal gets NaN. The new tests check both attributes against R's CDR() output for mw2008 and mw2014, check the paper's Table 4, check clrd by LOB against single-triangle fits and cover each ValueError path. merz2008 and gesmann2026 go into references.bib and are cited in the methods user guide. Part of casact#1543. Developed with an AI assistant (Árbol, running Muse).
Pyright Type CompletenessView the full Project (full
Other symbols referenced but not exported by
Symbols without documentation:
Patch (exported symbols added or changed by this PR): 41.7% fully typed (10 / 24)
Patch symbol details
|
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #1558 +/- ##
==========================================
+ Coverage 94.59% 94.76% +0.16%
==========================================
Files 98 98
Lines 6233 6318 +85
Branches 799 804 +5
==========================================
+ Hits 5896 5987 +91
+ Misses 216 212 -4
+ Partials 121 119 -2
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
this is a beefy PR. i'm good with the scope as is. will log some enh in the issue. do you think it would make sense to merge test_mack_cdr with test_mack? i'm asking before a full review because i don't want any line-by-line comments to get lost in the shuffle |
Move the one-year CDR tests from test_mack_cdr.py to the end of test_mack.py, as asked in review. Their coverage table becomes the module docstring, with a sentence on the existing Mack tests. The CDR tests gain a cdr prefix (UNSUPPORTED becomes CDR_UNSUPPORTED) so they read clearly beside the Mack tests; no test logic changed. Developed with an AI assistant (Árbol, running Muse).
|
Yes, that makes sense. Done in 8a7832d: the CDR tests now sit at the end of I can take the third item you logged on #1543, numbers for the one-year view section of the user guide, once this merges. (The commit and this reply were made with Árbol, an AI assistant, under my direction.) |
|
CI on 8a7832d ran into today's GitHub Actions incident: the 3.12 and 3.13 test jobs and the Pyright check never started. The four test jobs that did run passed. Could you re-run the failed jobs when you get a chance? (This comment was made with Árbol, an AI assistant, under my direction.) |
| ``mack_std_err_`` measures the error to ultimate, this attribute | ||
| measures the next year only, the horizon of a one-year solvency view. | ||
|
|
||
| The estimator assumes the paper's setting: a square triangle, |
There was a problem hiding this comment.
can you make this docstring a bit more concise?
There was a problem hiding this comment.
Shortened in 8fdc25e. The cdr_std_err_ docstring now cites Result 3.1, says what the CDR is, points to mack_std_err_ for the error to ultimate, and notes sigma_interpolation and the NaN for missing cells. The supported setting is left to the Raises section. The description of total_cdr_std_err_ is down to two sentences.
(The commit and this reply were made with Árbol, an AI assistant, under my direction.)
| f"{sorted(str(v) for v in average)}." | ||
| ) | ||
| # A link ratio enters f_j when origin i has C[i, j] and C[i, j + 1]. | ||
| link = upper[:, 1:] |
There was a problem hiding this comment.
would it be simpler to feed C through a fresh Development and compare the resulting w_ against X.w_?
There was a problem hiding this comment.
Yes, done in 8fdc25e. The check now fits Development() on C and compares its w_ with X.w_, which catches n_periods, drop, drop_high, drop_low and drop_valuation. fillna leaves w_ equal to the fresh fit's, so the check also compares ldf_ with the fresh fit's ldf_. average="simple" also leaves w_ unchanged, and the earlier average_ check catches it. The ValueError tests pass unchanged.
(The commit and this reply were made with Árbol, an AI assistant, under my direction.)
| "some (drop, drop_high, drop_low, drop_valuation, n_periods)." | ||
| ) | ||
| with np.errstate(divide="ignore", invalid="ignore"): | ||
| S = np.where(pair, x, 0).sum(2) # S_j^I |
There was a problem hiding this comment.
are you sure that S, f, etc. all needs to bespoke logic? isn't this just volume-all chainladder? maybe i'm not understanding the intent. i do see the error check and the resumed use of self.ultimate_.
There was a problem hiding this comment.
It is volume-all chainladder, and 8fdc25e takes that part from the library: f is the fitted ldf_, S_j^{I+1} (2.10) is C.sum(axis="origin"), and S_j^I (2.9) is that sum less the latest diagonal. The link-ratio code is gone. Only the sums over later development periods inside Φ, Δ, Λ and Ξ stay bespoke, because Development does not compute them.
(The commit and this reply were made with Árbol, an AI assistant, under my direction.)
| S1 = S + diag # S_j^{I+1} | ||
| ult = self.ultimate_.set_backend("numpy").values[..., 0] | ||
| # Sums over the later periods j = d+1 .. J-1 for each d = 0 .. J-1. | ||
| A = diag * q / S1**2 |
There was a problem hiding this comment.
are A, B, etc. notation from the paper? if these are just temp notation, could i trouble you to comment the source equation, such as "summand of (3.5)"?
There was a problem hiding this comment.
A and B were my own names. In 8fdc25e the temporaries carry the paper's symbols where it has them, Φ (3.5), Ψ (3.6), Δ (3.4), Λ (3.13) and Ξ (3.14), and the comments give the equation behind each. The two that are not the paper's are commented: q is σ_j² (3.3) over (f_j^I)², and shared is the sum that (3.4) and (3.13) have in common.
(The commit and this reply were made with Árbol, an AI assistant, under my direction.)
| ("mw2014", MW2014_CDR, MW2014_MACK, MW2014_TOTALS, 7), | ||
| ], | ||
| ) | ||
| def test_cdr_matches_r(sample, cdr, mack, totals, decimals, backend): |
There was a problem hiding this comment.
we are ultimately trying to make this package standalone. hence we are trying to match to the paper, not to the R package.
There was a problem hiding this comment.
In 8fdc25e the tests check the paper only. test_cdr_matches_merz2008 checks the factors and σ² of Table 2 and both columns of Table 4 (per origin, the cov^½ row and the totals) on four stacked copies of mw2008. Table 4 prints whole units, and every entry is within 1 except accident year 2 (printed 1,488 and 1,566, computed 1,486.6 and 1,563.8), which is checked to 2.5. The R figures, the R references in the docstrings and user guide, and the gesmann2026 entry are gone. I dropped the mw2014 case, whose expected values came only from R.
(The commit and this reply were made with Árbol, an AI assistant, under my direction.)
|
|
||
|
|
||
| @pytest.mark.parametrize("backend", ["numpy", "sparse"]) | ||
| def test_cdr_matches_merz2008_table4(backend): |
There was a problem hiding this comment.
mw2008 gets used a few times. might as well add it to conftest.py
There was a problem hiding this comment.
Added to FIXTURES in conftest.py in 8fdc25e, with both runs like the other samples. The CDR tests that read mw2008 now take the fixture, so they run on the numpy and sparse backends.
(The commit and this reply were made with Árbol, an AI assistant, under my direction.)
- Compare w_ and ldf_ with a fresh Development on the triangle instead of
recomputing the link ratios, and take S_j^I and S_j^{I+1} from triangle
sums.
- Name the temporaries after Merz and Wüthrich (2008) and cite their
equations.
- Test against Tables 2 and 4 of the paper on four stacked copies of
mw2008. Drop the R figures, the mw2014 case and the gesmann2026 entry.
- Add an mw2008 fixture to conftest.py.
- Shorten the cdr_std_err_ and total_cdr_std_err_ docstrings.
Developed with an AI assistant (Árbol, running Muse).
| @@ -1,6 +1,23 @@ | |||
| """ | |||
There was a problem hiding this comment.
apologies for being vague with my previous request. I meant to say that public-facing, method-level docstring should be more concise. test module docstring is for internal documentation and should be descriptive.
There was a problem hiding this comment.
Thanks, that's clear now. 740379d makes the module docstring descriptive again: a table of what each source checks for each attribute (merz2008, merz2014, per-triangle), a note per source giving the tables, pages and printed precision, and a summary of the other CDR tests. The two CDR tests that had one-line docstrings now say what they check and why. The cdr_std_err_ and total_cdr_std_err_ docstrings stay short.
(The commit and this reply were made with Árbol, an AI assistant, under my direction.)
| return np.concatenate((np.cumsum(t[..., ::-1], -1)[..., -2::-1], zero), -1) | ||
|
|
||
| with np.errstate(divide="ignore", invalid="ignore"): | ||
| q = sigma[..., :J] ** 2 / f**2 # sigma_j^2 (3.3) over (f_j^I)^2 |
There was a problem hiding this comment.
not a blocker to approval. but just curious if you tried to do this through the triangle api, without falling back to numpy manipulation. we try to be intentional and comprehensive with our pandas passthroughs, groupby aggregations, and basic dunders. this is a good live test of how comprehensive triangle arithmetic is.. and whatever you can't easily accomplish here through the triangle api, we should raise as issue to contemplate adding to the library.
There was a problem hiding this comment.
I tried it. A version of _get_cdr_msep written only with Triangle operations (origin sums, a valuation filter for the latest diagonal, latest_diagonal, shift along both axes, fillzero and arithmetic) gives the same figures as the numpy code: the same NaN cells and a largest relative difference of 1.7e-16 on mw2008 (cumulative, incremental, one missing cell), mw2014, raa and clrd by LOB, numpy and sparse. It is 8.5 to 17 times slower per call (235–484 ms against 27–29 ms; the most on the 17-year mw2014). I have it on a local branch and can push it if you want to see it.
What got in the way:
Triangle.cumsumraisesTypeError: nancumsum() got an unexpected keyword argument 'keepdims'along either axis and aftergroupby, so sums over later periods became shift loops ([BUG] Triangle.cumsum raises TypeError: nancumsum() got an unexpected keyword argument 'keepdims' #1573).- On sparse, arithmetic with a one-origin Triangle of a different development length fills only the first origin, and
X.cdf_.align_pattern(X)is all NaN without a tail ([BUG] Sparse arithmetic with a one-origin Triangle fills only the first origin when development lengths differ #1574). I kept both operands at the same length instead. - A Triangle times a pattern takes the pattern's 2261-12-31 valuation date, so
latest_diagonalraises until the date is reset ([BUG] Arithmetic with a pattern gives the result the pattern's valuation date #1575).shifton a pattern stores the "12-24" labels asddims, which breaksdevelopment([BUG] shift on a pattern stores its display labels as development, and development then raises TypeError #1576). *keeps zeros while+and-turn them into NaN, so the missing-cell check needed acomplete / completemask. That is the zero/NaN question of cl.Triangle legitimate zeros converted to NaN #181, so I didn't open another issue.
Given the speed, I'd keep numpy here and revisit once those are fixed, but I can switch if you'd rather have the Triangle version now.
(This reply was made with Árbol, an AI assistant, under my direction.)
There was a problem hiding this comment.
please keep numpy version
only 4 bugs is better than i expected. the ~10x speed degradation is worse than i expected. is it loops?
There was a problem hiding this comment.
Will do: the PR stays on the numpy version (740379d).
Mostly loops, plus one slow property. Taking the fastest of 15 calls on the numpy backend, the Triangle version ran 232 ms against 28 ms for numpy on mw2008 (9×9), and 447 ms against 29 ms on mw2014 (17×17).
- Loops: the two shift loops make 47 / 95 Triangle operations, a count that grows with the triangle. Replacing them with a numpy cumsum on
.valuesbrings the call to 105 / 122 ms, so they are 62% / 78% of the gap. valuation: on triangles with valuation dates on the development axis (latest_diagonal,ultimate_) it builds a DataFrame and callsunstack, which costs 2.5 / 3.9 ms against 0.04 ms on the original triangle, and+and-call it up to twice throughnan_triangle. It accounted for 42% / 53% of the profiled time. Building the same index with a singlenp.repeatand no DataFrame brings the call to 159 / 248 ms, with identical results.- With both changes the call takes 73 ms on either size, still 2.5–2.6x numpy. That remainder doesn't grow with size, which points to per-operation overhead: a Triangle
+takes about 0.5 ms where the numpy add takes about 1 µs.
The valuation change should help arithmetic on any triangle with valuation dates on its development axis, so I can open a separate issue for it.
(This reply was made with Árbol, an AI assistant, under my direction.)
There was a problem hiding this comment.
please open an issue on the valuation change.
thanks for the investigation! please keep the triangle version around, which we can land after dealing with those bugs. we need more corpus on the triangle API.
| np.testing.assert_array_less(gap, np.broadcast_to(tol, gap.shape)) | ||
|
|
||
|
|
||
| def test_cdr_matches_merz2008(mw2008): |
There was a problem hiding this comment.
apologies for being vague. by suggesting to fixturize mw2008, i didn't mean to get rid of the mw2014 tests.
There was a problem hiding this comment.
Understood. 740379d brings mw2014 back as a fixture and tests it against Merz and Wüthrich (2014), the paper the sample comes from (its Table 1 is the same triangle): the factors and σ² of Table 1, the reserve and the process, parameter and Mack errors of Table 2 by origin and in total, and the one-year total of Table 3, each within half a unit of its last printed digit. The paper prints only the total one-year figure for this triangle (1,843), so per-origin CDR is still checked against Table 4 of Merz and Wüthrich (2008) on mw2008. merz2014 is added to references.bib.
(The commit and this reply were made with Árbol, an AI assistant, under my direction.)
- Test mw2014 against the figures that Merz and Wüthrich (2014) print for it: Table 1's factors and sigma^2, Table 2's reserve, process, parameter and Mack errors by origin and in total, and Table 3's one-year total. The paper prints no one-year figures by origin. - Add an mw2014 fixture to conftest.py and merz2014 to references.bib. - Make the test module docstring descriptive again, with a table of what each source checks for each attribute, and expand the two one-line CDR test docstrings. Developed with an AI assistant (Árbol, running Muse).
Summary of Changes
Adds
cdr_std_err_andtotal_cdr_std_err_toMackChainladder: the standard error of the one-year claims development result, by origin and in total, from Results 3.1 and 3.3 of Merz and Wüthrich (2008).mack_std_err_, sofitis unchanged. Reading them raisesValueErroroutside the paper's setting: a square triangle, volume-weighted factors from every link ratio, and no tail. A triangle with a missing cell above the latest diagonal gets NaN.sigma_interpolation. With"mack"they match Tables 2 and 4 of the paper for its triangle, the samplemw2008, within the tolerances below.chainladder/methods/tests/test_mack.pychecks Tables 2 and 4 on four stacked copies ofmw2008, and the figures that Merz and Wüthrich (2014) print for the samplemw2014, through newmw2008andmw2014fixtures on the numpy and sparse backends. It also checksclrdby LOB against single-triangle fits and covers eachValueErrorpath.merz2008andmerz2014are added toreferences.bib, andmerz2008is cited in a short paragraph of the MackChainladder section of the methods user guide.Related GitHub Issue(s)
Part of #1543
AI/LLM Usage
This change was written with Árbol, an AI assistant running on Muse, working under my direction. It wrote the code and tests, checked the figures against the paper, ran the checks listed below, and drafted this description. I reviewed the diff and the results before submitting.
Additional Context for Reviewers
mw2014, Merz and Wüthrich (2014) print the factors and σ² (Table 1), the reserve, process, parameter and Mack errors by origin and in total (Table 2), and the one-year total, 1,843 (Table 3), but no one-year figures by origin. Each of these figures is within half a unit of its last printed digit.test_mack.pypasses (34 tests), and so does a Sphinx doctest build of the two properties. ruff check and format are clean.Submitter's Checklist
[FIX],[FEAT],[DOCS],[TST],[CHORE], or[BRK]).Reviewer's Checklist