Tracking Issue: Pluggable Scalar Functions #7881

This is a tracking issue for adding pluggable scalar functions to Vortex. Right now, it is impossible for a third-party extension type to add an implementation for scalar function execution and also reordering/optimization.

Motivation

As of writing, the main use case for this is TurboQuant, which we have decided to make a separate logical extension type with a TQDecode and TQEncode scalar function implementation that converts to and from a Vector array. Right now, there is no way for TurboQuant to implement specialized execution for vector scalar functions such as CosineSimilarity, InnerProduct, and L2Norm.

It's also unclear what function signature of the binary operations are (we need CosineSimilarity to accept a Vector on at least 1 of the sides, so does that mean the return type is always Vector but sometimes one of the children can be TurboQuant??)

And it is not obvious if this is actually the way we want to implement this functionality in Vortex (for example, we could just have TQCosineSimilarity and TQInnerProduct, but that seems duplicative). But this feels like a reasonable path forward (and I think in general we want to be more extensible in general, not less).

We would ideally like this sooner than later so that we can get TurboQuant back to a fully functional state (i.e., fully working in the vortex-search-bench benchmarks with the same performance), and then we can delete the implementation in vortex-tensor.

Design

Not really sure what this should look like. However this ends up, it needs to support pluggable return dtype, execution, reordering/optimization, etc. I think this is similar to how Arrays must be pluggable, but it has to additionally support a new return dtype (since Array encodings keep the same dtype).

Steps

• Unclear

Unresolved questions

• How do we do this?

Implementation history

cc @gatesn @joseph-isaacs