Detect the stream filters due filter in/out option

[arturminchukov]

Is your feature request related to a problem?

When a user clicks Filter for value / Filter out value in the log details view (Explore / Logs panel), the datasource always adds a regular field filter:

field:="value"

Regular field filters scan data blocks, so on large volumes such queries can be slow — even when the clicked field is actually a stream field (e.g. app, namespace, instance).

Describe the solution you'd like

Detect whether the clicked field is a stream field and, if so, add it as a stream filter instead:

{field="value"}

Stream filters are resolved via the stream index and are the most efficient filter type in LogsQL, so this should noticeably reduce query execution time for filter in/out actions on stream fields.

Implementation notes

• Detection can be synchronous and free: toggleQueryFilter receives ToggleFilterAction with an optional frame: DataFrame, and the backend already attaches per-row stream label maps to every logs frame (meta.custom.streams, built from the _stream field). The clicked key is a stream field iff it is present in those maps. This is the same mechanism Loki uses to detect label types.
• The filter chip can be marked with a flag (similar to the existing editor-only fromLevelFilter flag), and the serialization branch added in getFilterInsertValue, which already special-cases _stream / _stream_id. Both delivery paths (root-query prefix and extra_filters) accept LogsQL filters, so the stream syntax works in both.
• Operators map 1:1 — {...} supports =, !=, =~, !~.
• Fallback: if no frame is available in the action, keep the current behavior (regular field filter).

Things to verify

• Filter-out semantics: {k!="v"} may also match streams without the label k (PromQL-like semantics). Compare with the current k:!="v" behavior and pick the equivalent form (possibly -{k="v"}).
• Stream filters passed via extra_filters are optimized by the index the same way as inline ones.

[github-vincent-miszczak]

Filtering for field:="value" and {field="value"} can give different results within a time range when for instance the stream field was added somewhere in the time range.
It's not correct to assume it was always there. It will often work, but not always.

The user that clicks filter in/out does not necessarily know when a field was promoted.

We had a similar user experience case recently, where the user thought his logs were missing because Claude crafted a query on a stream field we introduced along the way, so logs before the cutover were missing.

That's not necessarily a blocker but should at least be mentioned somewhere.

[arturminchukov]

Filtering for field:="value" and {field="value"} can give different results within a time range when for instance the stream field was added somewhere in the time range. It's not correct to assume it was always there. It will often work, but not always.

The user that clicks filter in/out does not necessarily know when a field was promoted.

We had a similar user experience case recently, where the user thought his logs were missing because Claude crafted a query on a stream field we introduced along the way, so logs before the cutover were missing.

That's not necessarily a blocker but should at least been mentioned somewhere.

@github-vincent-miszczak thanks for your comment! Yeah, you're right. So when the user clicks filter in/ filter out, we check if this field is in the _stream of this log. If it is, we assume that we can apply this field as a stream filter. So, in this case, we will show similar logs to users (with this stream field). If not, we will show all logs by field: value filter.

[github-actions]

🎉 The pull request #702 that addresses this issue has been merged!

PR Details:
- Title: 691 detect the stream filters due filter inout option
- Author: @arturminchukov
- Merged by: @arturminchukov
- Link: #702

This feature will be included in the next release.

[hagen1778]

Related vmui ticket VictoriaMetrics/VictoriaLogs#1607

[hagen1778]

@arturminchukov even with the check on the current time range, it still could happen that stream/field filtering will cause confusion. For example, user applied filter as stream-field on the range -24h, and then zoomed-out to -30d where field=>stream_field transition happened. In this case, the search query would effectively filter out all log entries that had non-stream-field on the time range.

However, despite the chance of this confusion I still think this change is beneficial in long term. Transition between field types happens rarely, while filtering in UI happens all the time. More efficient filtering is better user experience.
We should probably deal with the confusion by making UI more clear for users:

1. Add more visual emphasis on difference of stream and regular fields
2. Add possibility to swtich filter type between stream and regular fields with one click.

wdyt?

[arturminchukov]

1. Add more visual emphasis on difference of stream and regular fields
2. Add possibility to swtich filter type between stream and regular fields with one click.

@hagen1778

1. Right now, I added a stream badge to the stream filters. I think it shows the difference between the two filter types.

2. It's a good suggestion, I will add the possibility to switch between stream and regular field. It is easy to do in one way when you are switching from a stream filter to a regular one. So, in the opposite case, we could check whether this field is in the stream fields, and then show the switch button.