Skip to content

[FEAT] Opt-in check for duplicate keys in the Triangle constructor #1567

Open

Description

@scks1

Description

Triangle adds together all input rows that fall in the same index, origin and development cell (_aggregate_data in chainladder/core/base.py, the groupby(...).sum() at L259–268 on main). Transaction data needs this: the triangle tutorial builds one paid triangle from the 34,244 claim transactions in prism.csv by leaving out index. When the input should hold one row per cell, a repeated key is a data error, and the constructor sums it without a message. Nothing in the built triangle records how many rows went into a cell, so the diagnostics proposed in #1252 can't detect it afterwards.

The bundled CAS samples have a case of this. clrd and clrd2025 are indexed by GRNAME and LOB, and in both files four name-and-line pairs cover two GRCODEs each, so load_sample adds two companies into one triangle. I mentioned the clrd2025 pairs in #1256 (comment). In clrd2025, Farmers Mut Ins Co/othliab has paid losses of 3 (GRCODE 10323) and 4 (GRCODE 17124) for accident year 1998 at 12 months, and the sample shows 7.

from pathlib import Path

import pandas as pd

import chainladder as cl

rows = pd.DataFrame(
    {
        "origin": [2020, 2020, 2020, 2021],
        "valuation": [2020, 2020, 2021, 2021],
        "paid": [100, 50, 180, 90],
    }
)
print(cl.Triangle(rows, origin="origin", valuation="valuation", columns="paid", cumulative=True))

data = Path(cl.__file__).parent / "utils" / "data"
for name in ["clrd", "clrd2025"]:
    raw = pd.read_csv(data / f"{name}.csv")
    codes = raw.groupby(["GRNAME", "LOB"])["GRCODE"].nunique()
    lines = len(raw[["GRCODE", "LOB"]].drop_duplicates())
    print(f"\n{name}: {len(cl.load_sample(name).index)} triangles for {lines} company-lines")
    print(codes[codes > 1].to_string())
         12     24
2020  150.0  180.0
2021   90.0    NaN

clrd: 775 triangles for 779 company-lines
GRNAME                    LOB    
Farmers Mut Ins Co        othliab    2
Farmers Union Mut Ins Co  comauto    2
                          othliab    2
Madison Mut Ins Co        othliab    2

clrd2025: 768 triangles for 772 company-lines
GRNAME                    LOB    
Farmers Mut Ins Co        othliab    2
Farmers Union Mut Ins Co  comauto    2
                          othliab    2
Madison Mut Ins Co        othliab    2

In the first triangle, the 2020 cell at age 12 is 100 + 50.

Is your feature request aligned with the scope of the package?

  • Yes, absolutely!
  • No, but it's still worth discussing.
  • N/A (this request is not a codebase enhancement).

Describe the solution you'd like, or your current workaround.

A constructor keyword, for example duplicates="sum" (the default, today's behavior) or duplicates="raise", which raises a ValueError naming the first few repeated index, origin and development keys. The names are open, and "warn" could be a third option. _aggregate_data already groups by those keys, so the check could count rows per group before summing.

Today the check has to run on the DataFrame before the constructor, with dates converted to the triangle's grain by hand.

The sample index needs a smaller, separate fix. With GRCODE appended to the index, clrd gives 779 triangles, and .loc["Adriatic Ins Co", "othliab"] still returns one triangle. I can include that change here or open it as its own issue, whichever you prefer.

Do you have any additional supporting notes?

pandas makes the same split: DataFrame.pivot raises ValueError: Index contains duplicate entries, cannot reshape, while pivot_table aggregates.

Would you be willing to contribute this ticket?

  • Yes, absolutely!
  • Yes, but I would like some help.
  • No.

(This issue was written with Árbol, an AI assistant, under my direction.)

Activity

  1. henrydingliu commented on Nov 10, 2026

    @henrydingliu
    Member

    thanks for the thorough write-up. looks ready for implementation

  2. updated Scope Codebase, Effort Medium and Priority Low on Nov 10, 2026
  3. henrydingliu commented on Nov 10, 2026

    @henrydingliu
    Member

    regarding the clrd2025 duplicates. is editing GRNAME a reasonable fix? i see 5 alternative names here. the "not listed" one can just get "(defunct)" tag at the end of the current GRNAME.

  4. scks1 commented on Nov 10, 2026

    @scks1
    ContributorAuthor

    Yes, and it's a smaller change than indexing by GRCODE. clrd has the same three names and the same four merged name-and-line pairs, so I'd rename in both files. With the listing names, four codes change: 10323 to Eagle Mut, 32670 to Agraria Ins Co, 14443 to Madison Mut Ins Company, and 17124 to Farmers Mut Ins Co (defunct); 28436 and 30449 already match the listing. That leaves Madison Mut Ins Company and Madison Mut Ins Co, which differ only in the last word, so for that pair I'd use Madison Mut Ins Co (IL) and Madison Mut Ins Co (NY) instead.

    clrd goes from 775 to 779 triangles and clrd2025 from 768 to 772, so the counts in the Triangle.shape and ClarkLDF docstring examples and the tutorials change too. I'll put the rename in the same PR as the duplicates check.

    (This comment was made with Árbol, an AI assistant, under my direction.)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    Priority

    Low

    Scope

    Codebase

    Effort

    Medium

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions