Skip to content

NdjsonSerializer adds a leading comma to lines 2+ of each batch, so parsers drop them #4

Description

@Pablo1

Version:

  • Release v3.0.0 (also on master at afa9797)

OS and platform:

  • Ubuntu 26.04
  • Technitium 15.4

Configuration:

{
  "sinks": {
    "maxQueueSize": 50000,
    "enableEdnsLogging": true,
    "console": {
      "enabled": false
    },
    "http": {
      "enabled": true,
      "endpoint": "https://victorialogs.example.com:9429/insert/jsonline",
      "headers": {
        "VL-Time-Field": "timestamp",
        "VL-Msg-Field": "question.questionName",
        "VL-Extra-Fields": "app_name=TechnitiumDNSServer,host.name=host.example.com,severity_text=INFO",
        "VL-Stream-Fields": "app_name,host.name"
      },
      "ndjson": true
    }
  },
  "pipeline": {
    "normalize": {
      "enabled": true
    },
    "tagging": {
      "enabled": false,
      "tags": [
      ]
    }
  }
}  

Error Message:

2026-09-17T08:15:45.207Z warn VictoriaLogs/app/vlinsert/jsonline/jsonline.go:79 jsonline: cannot read line #1 in /jsonline request: cannot parse JSON: cannot parse number: unexpected char: ","; unparsed tail: ",{\"answers\": 

Expected Behavior:

  • One JSON object on each line, no separators

Actual Behavior:

  • Lines 2 and after in a batch start with ,
{"answers":[...],"clientIp":"..."}
,{"answers":[...],"clientIp":"..."}

Steps to Reproduce:

  • Install the app
  • Enable the http sink with the above config
  • Make multiple queries in rapid succession so that one batch has more than one record
  • Examine the warnings from the log ingestion system and compare the ingested record count with the Technitium query count

The file and console sinks use the same serializer, so they probably have the same problem (not tested).

Root Cause:

  • With SkipValidation = true, the reused Utf8JsonWriter writes a separator before each top-level value after the first

Proposed Fix:

  • Call Utf8JsonWriter.Reset() after each record is flushed
  • I have a one-line fix tested on a branch. Would a PR be acceptable?

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions