Cannot use Phi3-small with DirectML / ONNX

[Craig Loewen (craigloewen-msft)]

Describe the bug
I am trying to run the Phi-3 small model using ONNX and cannot do so.

To Reproduce
Steps to reproduce the behavior:

1. Build the ONNX model with this command: python3 builder.py -m "microsoft/Phi-3-small-8k-instruct" -o ./output -p "int4" -e dml -c ./cache_dir
2. Run the ONNX model in a Visual Studio project
3. Then you will see an error.

Expected behavior
The ONNX model loads as other ONNX models would.

Screenshots

Desktop (please complete the following information):

• OS: Windows 11 Version 10.0.22635.4010]

Additional context
Full error:

Microsoft.ML.OnnxRuntimeGenAI.OnnxRuntimeGenAIException
  HResult=0x80131500
  Message=This is an invalid model. Type Error: Type 'tensor(float)' of input parameter (InsertedPrecisionFreeCast_/model/layers.0/attn/q_rotary/RotaryEmbedding/output_0) of operator (SparseAttention) in node (/model/layers.0/attn/SparseAttention) is invalid.
  Source=Microsoft.ML.OnnxRuntimeGenAI
  StackTrace:
   at Microsoft.ML.OnnxRuntimeGenAI.Result.VerifySuccess(IntPtr nativeResult)
   at Microsoft.ML.OnnxRuntimeGenAI.Model..ctor(String modelPath)
   at AdvancedPaste.Helpers.SLMRunner.<>c__DisplayClass10_0.<InitializeAsync>b__0() in E:\eDev\PowerToys\src\modules\AdvancedPaste\AdvancedPaste\Helpers\SLMRunner.cs:line 120
   at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)

[Baiju Meswani (baijumeswani)]

DML currently does not support the Phi-3 small model.

As you have found, the SparseAttention operator is not implemented for DML as of now.