set sourcetype without setting payloadkey

[matthiasgleichauf]

Hi,

In this commit it was added that sourceType requires payloadkey to be set for Splunk Outputs.
Is there a reason for this specific requirement? Or could we allow to set the sourceType without specifying the payloadkey or defining a reserved keyword (e.g. .) to take the whole payload?

Background:
We have a sourcetype configured in Splunk that is basically _json but allows for much longer lines than the default 10k characters to accommodate the logs of some workload that sometimes generates longer messages.
We still want to include all metadata the ClusterLogForwarder adds (like kubernetes, openshift etc) so reducing the payload to .message is also not an option. Since we don't want to allow longer lines for everything we ended up on the custom sourcetype.

Internal guidelines to contribute to external projects are cumbersome, so starting this issue to get your opinion first.

[jcantrill]

This epic captures the work associated with this feature.

Is there a reason for this specific requirement? Or could we allow to set the sourceType without specifying the payloadkey or defining a reserved keyword (e.g. .) to take the whole payload?

I'm confident the repo documentation here or official documentation should answer your questions.
The crux of this design is the log collector is responsible for workloads from across the cluster which are capable of producing logs in various formats (i.e. log4j, klog, etc). The field is meant to optimize the receiver's ingestion in order to extract meaningful metadata; you are essentially telling it what the layout of the 'splunk message' (which they could actually infer on their side). Given ViaQ is an organic, Red Hat Logging construct, there is no sourceType; the closest correct one is _json.

We have a sourcetype configured in Splunk that is basically _json but allows for much longer lines than the default 10k characters to accommodate the logs of some workload that sometimes generates longer messages.

Our observations have been that audit logs, specifically Kubernetes and OpensShift, are the primary culprits that generate large payloads. These can be reduced by either configuring the cluster to make them less verbose or by configuring the kubeAPIAudit filter. Additionally, there is a prune filter to remove specific fields in a more targeted fashion. IMO, if container logs are of the size identified here, then you should work with the workload provider to reduce those logs; it is questionable the value provided by an individual log line of this size.

As an aside, we have been investigating how best we might offer a configuration option to limit the max size of a payload.

[matthiasgleichauf]

Thank you very much for your response, it's appreciated.

The official doc I read, I hadn't found OBSDA-1383 yet though. From that I gather that the described customer was already using payloadKey anyway, maybe that's why it was designed like this? Other than that I could not identify a reason for the dependency, just that it should be that way.

So to restate my question: Is there a specific reason why it should not be possible to define the sourcetype and keeping the whole payload?

I get that its kind of smelly to have such long messages, but I also remember the pain of fusing lines again if we were to split them up just to satisfy some arbitrary maximum length requirement while absolutely not reducing the load in total, since we still send the data, just with more http requests and their overhead. And the metadata surrounding the log message accumulates already ~2300 characters in our environment so... 10k is reached rather fast if the logs are somewhat structured.

[jcantrill]

So to restate my question: Is there a specific reason why it should not be possible to define the sourcetype and keeping the whole payload?

Yes, I believe I answered that here:

Given ViaQ is an organic, Red Hat Logging construct, there is no sourceType; the closest correct one is _json.

What happens when a user configures sourceType (i.e. access_combined) which clearly does not match the incoming log stream? This is an intentional defensive mechanism for support and engineering. I assume this design comes from pre-kubernetes environments where the collector could be configured with a known type for the source. Frankly, I don't understand why this logic can not be inferred upon ingestion.

I get that its kind of smelly to have such long messages, but I also remember the pain of fusing lines again if we were to split them up just to satisfy some arbitrary maximum length requirement while absolutely not reducing the load in total, since we still send the data, just with more http requests and their overhead.

Our experience indicates that while the reduction in requests and overhead is a benefit, many receivers have a hard cap on the size of the payload or log line they are willing to accept. This limit is generally not arbitrary but born out of the understanding of the CPU and memory utilized upon receipt. Many of those services have strict QoS guarantees that are difficult to meet when these limits are exceeded.

[matthiasgleichauf]

In this specific workload being routed through the ClusterLogForwarder, we'd like to allow for more than 10k characters because the logmessages are that large and contain that much useful information. We also need the metadata that is being added by the ClusterLogForwarder like namespace name, pod name, IPs etc.
At the moment we can't do that because sourcetype cannot be configured without also limiting us to the log message itself. Do you consider that an issue or should we look into transferring those log messages in another way?

[jcantrill]

At the moment we can't do that because sourcetype cannot be configured without also limiting us to the log message itself. Do you consider that an issue or should we look into transferring those log messages in another way?

Have you considered looking at alternate ways to configure the ClusterLogForwarder? This documents the usage of the splunk output type. One could utilize the indexedFields and source to receive the information you require.

[matthiasgleichauf]

Splunk closes the HTTP Connection when the ClusterLogForwarder sends more data than the max line length for the given sourcetype allows. This results in errors like this:
2026-07-28T07:06:43.207538Z WARN sink{component_kind="sink" component_id=output_core_splunk_receiver component_type=splunk_hec_logs}: vector::sinks::util::retries: Retrying after error. error=Failed to make HTTP(S) request: connection closed before message completed 2026-07-28T07:07:02.455277Z WARN sink{component_kind="sink" component_id=output_core_splunk_receiver component_type=splunk_hec_logs}: vector::internal_events::http_client: HTTP error. error=connection closed before message completed error_type="request_failed" stage="processing"
Hence working with anything but the sourcetype is not helpful.