[BUG] DevelopmentCorrelation gives fully tied columns full weight, so triangles that never develop get T = 1 #1578

Are you on the latest chainladder version?

• Yes, this bug occurs on the latest version (main at 8e2e53c, chainladder 0.10.1, with the same output under numpy 2.3.3 / pandas 2.3.3 from uv.lock and under numpy 2.5.4 / pandas 3.1.0).

Describe the bug in words

DevelopmentCorrelation ranks the factors in each column with rankdata, which gives equal factors their average rank. A column whose paired factors are all equal has no ranking, but its T_k still gets a value and full weight. On a complete triangle that T_k is at least 0.5 when one column of the pair is fully tied (exactly 0.5 if the other has no ties) and 1 when both are. A triangle whose factors never change therefore gets T = 1 and is flagged whatever its data. All 1,353 complete earned-premium triangles in clrd (EarnedPremDIR, EarnedPremCeded, EarnedPremNet) do.

Loss triangles have the same problem in the tail. A fully developed column holds the same factor in every row, and such columns pull T up. Of the 873 complete loss triangles in clrd (IncurLoss, CumPaidLoss, BulkLoss), 264 have a fully tied column pair, most often in T_8, T_7 and T_6 (260, 160 and 91 of them).

Mack's Appendix G ranks each column from 1 to I − k and does not discuss ties.

How can the bug be reproduced?

import numpy as np

import chainladder as cl

clrd = cl.load_sample("clrd")

# Earned premium does not develop, so every factor is 1 up to rounding.
prem = clrd.loc["Aegis Grp", "ppauto"]["EarnedPremDIR"]
print(np.nanmin(prem.link_ratio.values), np.nanmax(prem.link_ratio.values))
dc = cl.DevelopmentCorrelation(prem)
print("T_2..T_8:", dc.t.iloc[0].round(4).tolist())
print(dc.t_expectation.iloc[0, 0], dc.t_critical.iloc[0, 0])

# Paid losses that stop developing after 84 months.
loss = clrd.loc["Kentucky Farm Bureau Mut Ins Grp", "comauto"]["CumPaidLoss"]
print(loss.link_ratio.to_frame().iloc[:4, -4:].round(4).to_string())
dc = cl.DevelopmentCorrelation(loss)
print("T_2..T_8:", dc.t.iloc[0].round(4).tolist())
print(round(dc.t_expectation.iloc[0, 0], 4), dc.t_critical.iloc[0, 0])

0.9999999999999999 1.0
T_2..T_8: [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0]
1.0 True
       72-84  84-96  96-108  108-120
1988  1.0000    1.0     1.0      1.0
1989  1.0054    1.0     1.0      NaN
1990  1.0085    1.0     NaN      NaN
1991  1.0005    NaN     NaN      NaN
T_2..T_8: [-0.1667, 0.2143, -0.4286, 0.3, 0.8, 0.5, 1.0]
0.1277 True

The premium triangle's T_k are all 1. In the loss triangle the factors from 84 months on are all 1.0, so T_7 = 0.5 and T_8 = 1, and T = 0.1277 is flagged.

What is the expected behavior?

prem_dc = cl.DevelopmentCorrelation(prem)
loss_dc = cl.DevelopmentCorrelation(loss)
assert np.isnan(prem_dc.t_expectation.iloc[0, 0]) and not prem_dc.t_critical.iloc[0, 0]
assert loss_dc.t.iloc[0, -2:].isna().all()  # T_7 and T_8 left out

Our reading, since the paper is silent on ties: a column pair whose paired factors are all equal in either column gets weight 0, as Mack leaves out T_(I-1) because there is "only one rank and therefore no randomness". A triangle with no other usable column gets NaN and no flag. Equal would mean within a relative 1e-12, because link_ratio returns 0.9999999999999999 for equal amounts (the premium triangle above, and origin 2002 of xyz's Premium). With the rule the loss triangle gets T = 0.063 and no flag.

On complete triangles the #1577 fix (#1579) gives the same results as main, so these counts measure the rule alone. The 264 loss triangles with a tied pair all change T (242 down, 13 up, 9 to NaN), 44 lose their flag and 27 gain one. The other 609 are unchanged, and all 1,353 premium triangles become NaN without a flag. Across the bundled samples, 1,662 of the 2,336 complete triangles have a tied pair.

Would you like this rule? The alternatives I see are leaving ties as they are, with or without a warning when a column is fully tied. A branch with the rule and two tests is ready on top of #1579.

Would you be willing to contribute this ticket?

• Yes, absolutely!
• Yes, but I would like some help.
• No.

(This issue was written with Árbol, an AI assistant, under my direction.)