riscv64: 4 of 4: RVV element-wise and reduction kernels - #2602
Conversation
|
@kali these PRs all use rafitied and stable ISA, no vendor specific extensions of any kind (it is quite fragmented with competing proposals and implementations for various extensions) so these PRs add Foundational Risc-V to extend the reach of TRACT to this platform and possibily invite other contributions from the community as the ISA standardisation evolves and solidifies. from a CI point of view QEMU is plenty to verify correctness, I have used both 10.X and 11.X). the fact it runs InceptionV3 executing end-to-end is quite remarkable. |
tract had no RISC-V backend, so rv64gc ran the generic Rust kernels for every matmul. Add an RVV 1.0 f32 mmm tier, detecting V from the AT_HWCAP bit that Linux never sets for the incompatible 0.7.1 draft and VLEN from the vlenb CSR. Because VLEN is a runtime property while MR must be a const generic, each kernel fixes (MR, NR, LMUL) and pins vl to MR, which is correct wherever VLMAX >= MR and short below it, so dispatch is gated on the hart's VLMAX reaching MR and the kernel re-checks the granted vl before running. An assembler probe keeps toolchains predating RVV 1.0 on the generic fallback.
The RVV kernels are gated on the hart's vector length, so a single emulated width would leave half the kernel set untested. Add riscv64gc to the qemu cross-test platforms twice, at VLEN 256 and 128, which select disjoint halves. -cpu max rather than a profile model because the generic rv64 model cannot run Debian's riscv64 glibc at all.
3bc0336 to
9634989
Compare
The riscv64 tier covered f32 only, so f16 matmul fell back to the generic kernels even on parts with native half-precision vectors, which includes the SpacemiT X60 in the K1. Add an f16 mmm tier from the same template at SEW=16, where VLMAX doubles and so does every tile height for a given LMUL and VLEN. Zvfh is read from the /proc/cpuinfo isa line, since RVA23 mandates only Zvfhmin and that cannot hold an f16 accumulator, and a second assembler probe keeps toolchains predating Zvfh on the f32 tier alone.
The riscv64 tier had no integer kernels, so quantised models fell back to the generic ones for every matmul. Add an i32 accumulator tier handling both packings the frame offers: i32 x i32 at e32, and i8 x i8 through a loop at e16 where vle8.v picks EEW=8 off the instruction and vwmacc.vx widens straight into the e32 accumulators, which keeps VLMAX identical between the two so one vl serves both. QScale, RoundingShiftRight and ShiftLeft follow the sign-magnitude reference in generic/rounding.rs, widening to e64 for the multiply; the negative-value mask taken before the magnitude overwrites it also serves the MinusInf and PlusInf nudges, which differ only where the result is zero anyway.
Every element-wise binary op and every f32 reduction still ran the generic kernels on riscv64. Add RVV versions of by-scalar and unicast mul, add, sub, subf, min and max, plus the max, min and sum reductions. These strip-mine on vsetvli rather than fixing a tile, so unlike the matmul kernels they are vector-length agnostic and need no VLEN predicate, and being Rust asm! blocks rather than .S files they need only rustc's own assembler, so they sit outside the tract_rvv cfg that records whether an external one could encode RVV.
9634989 to
64de462
Compare
|
Rebased onto current This was downstream of #2599, whose conflict was a duplicate of fb4bf2f (the hwbench fallback, since landed upstream). Dropping that one commit cleared this PR too — everything else cherry-picked onto current Re-verified on the new base under qemu-riscv64 11.0.3: the full stack passes 2709 tract-linalg tests at VLEN 128, 256 and 512, on stock |
Stacked on the matmul tiers. Adds fifteen kernels: by-scalar and unicast
mul,add,sub,subf,min,maxover f32, and themax,minandsumreductions — filling theregister_all_by_scalar/register_all_unicastregistries and the
mul_by_scalar_f32,max_f32,min_f32,sum_f32slots.Two ways in which these differ from the matmul kernels
No VLEN predicate. They strip-mine on
vsetvliinstead of pinningvltoa tile height, so
vlis whatever the hart grants and the tail falls out of theloop condition. That makes them vector-length agnostic outright — nothing to
gate beyond
has_rvv().nris only the frame's chunking granularity here, nota tile width.
No external assembler. They are Rust
asm!blocks, as the correspondingarm64 kernels are, so
.option arch, +vgoes through rustc's own LLVM and nobinutils version can refuse them. They therefore sit outside the
tract_rvvcfg, which records only whether an external assembler could handle the
.Smatmul kernels. A toolchain too old for those still gets these.
The reductions keep the running result in element 0 of
v1and feed it back asthe reduction's scalar operand each round, so strip-mining needs no separate
accumulator vector and no final horizontal step — reduction operands are LMUL=1
whatever vtype says.
vfredusumis the unordered sum, reassociating as thearm64 kernels also do.
Testing
The frames' own proptests, with
has_rvv()as the condition so they staymeaningful on a hart without V. 2709 tract-linalg tests and tract-core's 270
pass at VLEN 128, 256 and 512, on stock RVA23, and with V absent, under qemu
10.2.1 and 11.0.3.
No CI change needed. Same caveat as the earlier PRs: correctness under
emulation, no hardware numbers.
🍍