Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
18 commits
Select commit Hold shift + click to select a range
4ad699c
refactor(tests): migrate test infrastructure from TUnit to MSTest v3
Nucs Apr 15, 2026
ac02033
refactor(tests): migrate test attributes from TUnit to MSTest v3
Nucs Apr 15, 2026
cdfd2ac
chore: add git-blame-ignore-revs for TUnit migration commit
Nucs Apr 15, 2026
b1d1d54
refactor(tests): convert TUnit assertions to AwesomeAssertions
Nucs Apr 15, 2026
2a19b6d
chore: add assertion migration commit to git-blame-ignore-revs
Nucs Apr 15, 2026
e0db3c3
fix(tests): add [TestClass] attribute required by MSTest
Nucs Apr 15, 2026
469b5c6
chore: add [TestClass] commit to git-blame-ignore-revs
Nucs Apr 15, 2026
5a4a4c0
fix(tests): fix assertion and parameter compatibility issues
Nucs Apr 15, 2026
4eea964
refactor(tests): convert async Task to void for sync test methods
Nucs Apr 15, 2026
7b98edd
chore: add async-to-void commit to git-blame-ignore-revs
Nucs Apr 15, 2026
1b704e3
fix(ci): use --project flag for dotnet test (.NET 10 SDK requirement)
Nucs Apr 15, 2026
cb37e52
fix(ci): use --report-trx for MSTest Testing Platform
Nucs Apr 15, 2026
4547cf9
fix(ci): disable MSTest Testing Platform for .NET 10 SDK compatibility
Nucs Apr 15, 2026
39e696c
fix(ci): use bash shell for build, restore --project flag for tests
Nucs Apr 15, 2026
2d1e364
fix(ci): pass Testing Platform disable flags to dotnet test too
Nucs Apr 15, 2026
aafbccc
fix(ci): remove global.json test runner constraint
Nucs Apr 15, 2026
2b3aabf
fix(ci): remove Testing Platform mode entirely for .NET 10 SDK compat…
Nucs Apr 15, 2026
299a4b4
fix(tests): mark Vector512 Round/Truncate tests as OpenBugs
Nucs Apr 15, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
55 changes: 34 additions & 21 deletions .claude/CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -345,8 +345,7 @@ dotnet build -v q --nologo "-clp:NoSummary;ErrorsOnly" -p:WarningLevel=0

### Running Tests

Tests use **TUnit** framework with source-generated test discovery.
- `dotnet_test_tunit --filter "..."`: MSTest-style filter for TUnit (Category=, Name~, ClassName~, FullyQualifiedName~)
Tests use **MSTest v3** framework with source-generated test discovery.

```bash
# Run from test directory
Expand All @@ -356,10 +355,19 @@ cd test/NumSharp.UnitTest
dotnet test --no-build

# Exclude OpenBugs (CI-style - only real failures)
dotnet test --no-build -- --treenode-filter "/*/*/*/*[Category!=OpenBugs]"
dotnet test --no-build --filter "TestCategory!=OpenBugs"

# Run ONLY OpenBugs tests
dotnet test --no-build -- --treenode-filter "/*/*/*/*[Category=OpenBugs]"
dotnet test --no-build --filter "TestCategory=OpenBugs"

# Exclude multiple categories
dotnet test --no-build --filter "TestCategory!=OpenBugs&TestCategory!=HighMemory"

# Run specific test class
dotnet test --no-build --filter "ClassName~BinaryOpTests"

# Run specific test method
dotnet test --no-build --filter "Name~Add_Int32_SameType"
```

### Output Formatting
Expand All @@ -369,10 +377,10 @@ dotnet test --no-build -- --treenode-filter "/*/*/*/*[Category=OpenBugs]"
dotnet test --no-build 2>&1 | grep -E "^(failed|skipped|Test run| total:| failed:| succeeded:| skipped:| duration:)"

# Results with messages (no stack traces)
dotnet test --no-build 2>&1 | grep -v "^ at " | grep -v "^ at " | grep -v "^ ---" | grep -v "^ from K:" | sed 's/TUnit.Engine.Exceptions.TestFailedException: //' | sed 's/AssertFailedException: //'
dotnet test --no-build 2>&1 | grep -v "^ at " | grep -v "^ at " | grep -v "^ ---" | grep -v "^ from K:" | sed 's/AssertFailedException: //'

# Detailed output (shows passed tests too)
dotnet test --no-build -- --output Detailed
# Verbose output (shows passed tests too)
dotnet test --no-build -v normal
```

## Test Categories
Expand All @@ -383,36 +391,38 @@ Tests use typed category attributes defined in `TestCategory.cs`. Adding new bug
|----------|-----------|---------|-------------|
| `OpenBugs` | `[OpenBugs]` | Known-failing bug reproductions. Remove when fixed. | **EXCLUDED** via filter |
| `Misaligned` | `[Misaligned]` | Documents NumSharp vs NumPy behavioral differences. | Runs (tests pass) |
| `WindowsOnly` | `[WindowsOnly]` | Requires GDI+/System.Drawing.Common | Runtime platform check |
| `WindowsOnly` | `[WindowsOnly]` | Requires GDI+/System.Drawing.Common | Excluded on non-Windows |
| `HighMemory` | `[HighMemory]` | Requires 8GB+ RAM | **EXCLUDED** via filter |

### How CI Excludes OpenBugs
### How CI Excludes Categories

The CI pipeline (`.github/workflows/build-and-release.yml`) uses TUnit's `--treenode-filter` to exclude `OpenBugs`:
The CI pipeline (`.github/workflows/build-and-release.yml`) uses MSTest's `--filter` to exclude categories:

```yaml
- name: Test (net10.0)
run: |
dotnet run --project test/NumSharp.UnitTest/NumSharp.UnitTest.csproj \
dotnet test test/NumSharp.UnitTest/NumSharp.UnitTest.csproj \
--configuration Release --no-build --framework net10.0 \
-- --treenode-filter '/*/*/*/*[Category!=OpenBugs]'
--filter "TestCategory!=OpenBugs&TestCategory!=HighMemory"
```

This filter excludes all tests with `[OpenBugs]` attribute from CI runs. Tests pass locally when the bug is fixed — then remove the `[OpenBugs]` attribute.
This filter excludes all tests with `[OpenBugs]` or `[HighMemory]` attributes from CI runs. Tests pass locally when the bug is fixed — then remove the `[OpenBugs]` attribute.

### Usage

```csharp
// Class-level (all tests in class)
[TestClass]
[OpenBugs]
public class BroadcastBugTests { ... }

// Method-level
[Test]
[TestMethod]
[OpenBugs]
public async Task BroadcastWriteCorruptsData() { ... }
public void BroadcastWriteCorruptsData() { ... }

// Documenting behavioral differences (NOT excluded from CI)
[Test]
[TestMethod]
[Misaligned]
public void BroadcastSlice_MaterializesInNumSharp() { ... }
```
Expand All @@ -421,13 +431,16 @@ public void BroadcastSlice_MaterializesInNumSharp() { ... }

```bash
# Exclude OpenBugs (same as CI)
dotnet test -- --treenode-filter "/*/*/*/*[Category!=OpenBugs]"
dotnet test --filter "TestCategory!=OpenBugs"

# Run ONLY OpenBugs tests (to verify fixes)
dotnet test -- --treenode-filter "/*/*/*/*[Category=OpenBugs]"
dotnet test --filter "TestCategory=OpenBugs"

# Run ONLY Misaligned tests
dotnet test -- --treenode-filter "/*/*/*/*[Category=Misaligned]"
dotnet test --filter "TestCategory=Misaligned"

# Combine multiple exclusions
dotnet test --filter "TestCategory!=OpenBugs&TestCategory!=HighMemory&TestCategory!=WindowsOnly"
```

**OpenBugs files**: `OpenBugs.cs` (general), `OpenBugs.Bitmap.cs` (bitmap), `OpenBugs.ApiAudit.cs` (API audit), `OpenBugs.ILKernelBattle.cs` (IL kernel).
Expand Down Expand Up @@ -599,10 +612,10 @@ A: Core ops (`dot`, `matmul`) in `LinearAlgebra/`. Advanced decompositions (`inv
## Q&A - Development

**Q: What's in the test suite?**
A: TUnit framework in `test/NumSharp.UnitTest/`. Many tests adapted from NumPy's own test suite. Decent coverage but gaps in edge cases. Uses source-generated test discovery (no special flags needed).
A: MSTest v3 framework in `test/NumSharp.UnitTest/`. Many tests adapted from NumPy's own test suite. Decent coverage but gaps in edge cases. Uses source-generated test discovery (no special flags needed).

**Q: What .NET version is targeted?**
A: Library multi-targets `net8.0` and `net10.0`. Tests require .NET 9+ runtime (TUnit requirement).
A: Library multi-targets `net8.0` and `net10.0`. Tests also multi-target both frameworks.

**Q: What are the main dependencies?**
A: No external runtime dependencies. `System.Memory` and `System.Runtime.CompilerServices.Unsafe` (previously NuGet packages) are built into the .NET 8+ runtime.
Expand Down
10 changes: 5 additions & 5 deletions .claude/skills/np-tests/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ To interact/develop/create tests for np.* functions, high-level development cycl
Definition of Done:
- At the end of this step you understand 100% what numpy tests: inputs, outputs, edge cases, error conditions, dtype behaviors.
- You have identified all test files and test methods related to your function.
2. Migrate numpy's tests to C# following TUnit framework patterns in test/NumSharp.UnitTest. Match numpy's test structure and assertions exactly.
2. Migrate numpy's tests to C# following MSTest v3 framework patterns in test/NumSharp.UnitTest. Match numpy's test structure and assertions exactly.
Definition of Done:
- Every numpy test case has a corresponding C# test.
- We cover all dtypes NumSharp supports (Boolean, Byte, Int16, UInt16, Int32, UInt32, Int64, UInt64, Char, Single, Double, Decimal).
Expand All @@ -32,8 +32,8 @@ Use battletesting to validate behavior matches numpy: 'dotnet run << 'EOF'' and

### Test Patterns
```csharp
[Test]
public async Task FunctionName_Scenario_Dtype()
[TestMethod]
public void FunctionName_Scenario_Dtype()
{
// Arrange
var input = np.array(new[] { 3, 1, 2 });
Expand All @@ -42,8 +42,8 @@ public async Task FunctionName_Scenario_Dtype()
var result = np.sort(input);

// Assert - values from running actual numpy
Assert.That(result.IsContiguous, Is.True);
Assert.That(result.GetAtIndex<int>(0), Is.EqualTo(1));
Assert.IsTrue(result.IsContiguous);
Assert.AreEqual(1, result.GetAtIndex<int>(0));
}
```

Expand Down
11 changes: 11 additions & 0 deletions .git-blame-ignore-revs
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
# TUnit to MSTest v3 attribute migration
ac020336

# TUnit assertions to AwesomeAssertions migration
b1d1d543

# Add [TestClass] attributes for MSTest discovery
e0db3c3e

# Convert async Task to void for sync test methods
4eea9644
33 changes: 23 additions & 10 deletions .github/workflows/build-and-release.yml
Original file line number Diff line number Diff line change
Expand Up @@ -43,26 +43,39 @@ jobs:
- name: Build
run: dotnet build test/NumSharp.UnitTest/NumSharp.UnitTest.csproj --configuration Release -p:NoWarn=${{ env.DOTNET_NOWARN }}

# NOTE: Test project currently only targets net10.0. See TODO in NumSharp.UnitTest.csproj
# to re-enable net8.0 when TUnit compatibility is resolved.
#
# Test filtering:
# - OpenBugs: excluded (known-failing bug reproductions)
# - HighMemory: excluded (requires 8GB+ RAM, too much for CI runners)
# - WindowsOnly: auto-skipped at runtime via [SkipOnNonWindows] attribute
# - WindowsOnly: excluded on non-Windows runners
- name: Test (net8.0)
shell: bash
timeout-minutes: 10
run: |
echo "Starting test run (net8.0)..."
echo "dotnet version: $(dotnet --version)"
echo "Available memory: $(free -h 2>/dev/null || echo 'N/A')"
FILTER="TestCategory!=OpenBugs&TestCategory!=HighMemory"
if [[ "$RUNNER_OS" != "Windows" ]]; then
FILTER="$FILTER&TestCategory!=WindowsOnly"
fi
dotnet test test/NumSharp.UnitTest/NumSharp.UnitTest.csproj \
--configuration Release --no-build --framework net8.0 \
--filter "$FILTER" --logger "trx"

- name: Test (net10.0)
shell: bash
timeout-minutes: 10
run: |
echo "Starting test run..."
echo "Starting test run (net10.0)..."
echo "dotnet version: $(dotnet --version)"
echo "Available memory: $(free -h 2>/dev/null || echo 'N/A')"
# Note: TUnit --treenode-filter doesn't seem to exclude class-level categories.
# The HighMemory filter was attempted but didn't reduce test count.
# Ubuntu failures are allowed via continue-on-error while investigating.
dotnet run --project test/NumSharp.UnitTest/NumSharp.UnitTest.csproj \
FILTER="TestCategory!=OpenBugs&TestCategory!=HighMemory"
if [[ "$RUNNER_OS" != "Windows" ]]; then
FILTER="$FILTER&TestCategory!=WindowsOnly"
fi
dotnet test test/NumSharp.UnitTest/NumSharp.UnitTest.csproj \
--configuration Release --no-build --framework net10.0 \
-- --treenode-filter '/*/*/*/*[Category!=OpenBugs]' --report-trx
--filter "$FILTER" --logger "trx"

- name: Upload Test Results
uses: actions/upload-artifact@v4
Expand Down
173 changes: 173 additions & 0 deletions docs/MSTEST_FILTER_GUIDE.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,173 @@
# MSTest `--filter` Guide

## Filter Syntax

MSTest uses a simple property-based filter syntax:

```
--filter "Property=Value"
--filter "Property!=Value"
--filter "Property~Value" # Contains
--filter "Property!~Value" # Does not contain
```

Combine with `&` (AND) or `|` (OR):
```
--filter "Property1=A&Property2=B" # AND
--filter "Property1=A|Property2=B" # OR
```

## Available Properties

| Property | Description | Example |
|----------|-------------|---------|
| `TestCategory` | Category attribute | `TestCategory=OpenBugs` |
| `ClassName` | Test class name | `ClassName~BinaryOpTests` |
| `Name` | Test method name | `Name~Add_Int32` |
| `FullyQualifiedName` | Full namespace.class.method | `FullyQualifiedName~Backends.Kernels` |

## 5 Concrete Examples

### 1. Exclude OpenBugs (CI-style run)

```bash
dotnet test --no-build --filter "TestCategory!=OpenBugs"
```

Runs all tests EXCEPT those marked `[OpenBugs]`. This is what CI uses.

### 2. Run ONLY OpenBugs (verify bug fixes)

```bash
dotnet test --no-build --filter "TestCategory=OpenBugs"
```

Runs only failing bug reproductions to check if your fix works.

### 3. Run single test class

```bash
dotnet test --no-build --filter "ClassName~CountNonzeroTests"
```

Runs all tests in classes containing `CountNonzeroTests`.

### 4. Run single test method

```bash
dotnet test --no-build --filter "Name=Add_TwoNumbers_ReturnsSum"
```

Runs only the test named exactly `Add_TwoNumbers_ReturnsSum`.

### 5. Run tests by namespace pattern

```bash
dotnet test --no-build --filter "FullyQualifiedName~Backends.Kernels"
```

Runs all tests in the `Backends.Kernels` namespace.

## Quick Reference

| Goal | Filter |
|------|--------|
| Exclude category | `TestCategory!=OpenBugs` |
| Include category only | `TestCategory=OpenBugs` |
| Single class (exact) | `ClassName=BinaryOpTests` |
| Class contains | `ClassName~BinaryOp` |
| Method contains | `Name~Add_` |
| Namespace contains | `FullyQualifiedName~Backends.Kernels` |
| Multiple categories (AND) | `TestCategory!=OpenBugs&TestCategory!=WindowsOnly` |
| Multiple categories (OR) | `TestCategory=OpenBugs\|TestCategory=Misaligned` |

## Operators

| Op | Meaning | Example |
|----|---------|---------|
| `=` | Equals | `TestCategory=Unit` |
| `!=` | Not equals | `TestCategory!=Slow` |
| `~` | Contains | `Name~Integration` |
| `!~` | Does not contain | `ClassName!~Legacy` |
| `&` | AND | `TestCategory!=A&TestCategory!=B` |
| `\|` | OR | `TestCategory=A\|TestCategory=B` |

**Important:**
- Use `\|` (escaped pipe) for OR in bash
- Parentheses are NOT needed for combining filters
- Filter values are case-sensitive

## NumSharp Categories

| Category | Purpose | CI Behavior |
|----------|---------|-------------|
| `OpenBugs` | Known-failing bug reproductions | **Excluded** |
| `HighMemory` | Requires 8GB+ RAM | **Excluded** |
| `Misaligned` | NumSharp vs NumPy differences (tests pass) | Runs |
| `WindowsOnly` | Requires GDI+/System.Drawing | Excluded on Linux/macOS |
| `LongIndexing` | Tests > int.MaxValue elements | Runs |

## Useful Commands

```bash
# Stop on first failure (MSTest v3)
dotnet test --no-build -- --fail-on-failure

# Verbose output (see passed tests too)
dotnet test --no-build -v normal

# List tests without running
dotnet test --no-build --list-tests

# CI-style: exclude OpenBugs and HighMemory
dotnet test --no-build --filter "TestCategory!=OpenBugs&TestCategory!=HighMemory"

# Windows CI: full exclusion list
dotnet test --no-build --filter "TestCategory!=OpenBugs&TestCategory!=HighMemory"

# Linux/macOS CI: also exclude WindowsOnly
dotnet test --no-build --filter "TestCategory!=OpenBugs&TestCategory!=HighMemory&TestCategory!=WindowsOnly"
```

## Advanced Filter Examples

### Example A: Specific Test Method

```bash
dotnet test --no-build --filter "FullyQualifiedName=NumSharp.UnitTest.Backends.Kernels.VarStdComprehensiveTests.Var_2D_Axis0"
```

**Result:** 1 test

### Example B: Pattern Matching Multiple Classes

```bash
dotnet test --no-build --filter "ClassName~Comprehensive&Name~_2D_"
```

**Result:** Tests in `*Comprehensive*` classes with `_2D_` in method name

### Example C: Namespace + Category Filter

```bash
dotnet test --no-build --filter "FullyQualifiedName~Backends.Kernels&TestCategory!=OpenBugs"
```

**Result:** All Kernels tests except OpenBugs

### Example D: Multiple Categories (OR)

```bash
dotnet test --no-build --filter "TestCategory=OpenBugs|TestCategory=Misaligned"
```

**Result:** Tests that have EITHER `[OpenBugs]` OR `[Misaligned]` attribute

## Migration from TUnit

| TUnit Filter | MSTest Filter |
|--------------|---------------|
| `--treenode-filter "/*/*/*/*[Category!=X]"` | `--filter "TestCategory!=X"` |
| `--treenode-filter "/*/*/ClassName/*"` | `--filter "ClassName~ClassName"` |
| `--treenode-filter "/*/*/*/MethodName"` | `--filter "Name=MethodName"` |
| `--treenode-filter "/*/Namespace/*/*"` | `--filter "FullyQualifiedName~Namespace"` |
Loading
Loading