fix(whelk-core): strip invalid characters in JsonLD2RdfXml

[andersju]

Some records, e.g. https://libris.kb.se/7qj7x9xk10qm1nb, have values that include control characters.

This is "fine" for the JSON-LD representation, Jackson escapes them:

~ curl -s https://libris.kb.se/7qj7x9xk10qm1nb/data.jsonld|gron|grep mainTitle
json["@graph"][1].hasTitle[0].mainTitle = "L\u0005âmoureuse histoire dȧuguste Comte et de Clotilde de Vaux";

And they're stripped in the Turtle representation (JsonLdToTurtle's cleanValue() does v.replaceAll("\\p{Cntrl}", '')):

~ curl -s https://libris.kb.se/7qj7x9xk10qm1nb/data.ttl|grep mainTitle
      :mainTitle "Lâmoureuse histoire dȧuguste Comte et de Clotilde de Vaux" ;

But in JsonLD2RdfXml doesn't strip them and such characters are forbidden in XML, so we get an error:

~ curl -s https://libris.kb.se/7qj7x9xk10qm1nb/data.rdf
{"status_code":500,"error_id":"688bd2e6-966f-43fc-8994-f64d46738921","message":"Internal server error. Please report the error_id if the problem persists.","status":"Server Error"}

org.apache.jena.shared.CannotEncodeCharacterException: cannot encode (char)  in context XML
  at org.apache.jena.rdf.model.impl.Util.substituteEntitiesInElementContent(Util.java:184) ~[rest.jar:?]
  at org.apache.jena.rdfxml.xmloutput.impl.Unparser.wValueString(Unparser.java:562) ~[rest.jar:?]
  at org.apache.jena.rdfxml.xmloutput.impl.Unparser.wPropertyEltValueString(Unparser.java:547) ~[rest.jar:?]
  at org.apache.jena.rdfxml.xmloutput.impl.Unparser.wPropertyEltValue(Unparser.java:525) ~[rest.jar:?]
  at org.apache.jena.rdfxml.xmloutput.impl.Unparser.wPropertyElt(Unparser.java:391) ~[rest.jar:?]
  at org.apache.jena.rdfxml.xmloutput.impl.Unparser.wPropertyEltStar(Unparser.java:855) ~[rest.jar:?]
  at org.apache.jena.rdfxml.xmloutput.impl.Unparser.wTypedNodeOrDescriptionLong(Unparser.java:841) ~[rest.jar:?]
  at org.apache.jena.rdfxml.xmloutput.impl.Unparser.wTypedNodeOrDescription(Unparser.java:772) ~[rest.jar:?]
  at org.apache.jena.rdfxml.xmloutput.impl.Unparser.wTypedNode(Unparser.java:744) ~[rest.jar:?]
  at org.apache.jena.rdfxml.xmloutput.impl.Unparser.wObj(Unparser.java:684) ~[rest.jar:?]
  at org.apache.jena.rdfxml.xmloutput.impl.Unparser.wPropertyEltValueObj(Unparser.java:581) ~[rest.jar:?]
  at org.apache.jena.rdfxml.xmloutput.impl.Unparser.wPropertyEltValue(Unparser.java:526) ~[rest.jar:?]
  at org.apache.jena.rdfxml.xmloutput.impl.Unparser.wPropertyElt(Unparser.java:391) ~[rest.jar:?]
  at org.apache.jena.rdfxml.xmloutput.impl.Unparser.wPropertyEltStar(Unparser.java:855) ~[rest.jar:?]
  at org.apache.jena.rdfxml.xmloutput.impl.Unparser.wTypedNodeOrDescriptionLong(Unparser.java:841) ~[rest.jar:?]
  at org.apache.jena.rdfxml.xmloutput.impl.Unparser.wTypedNodeOrDescription(Unparser.java:772) ~[rest.jar:?]
  at org.apache.jena.rdfxml.xmloutput.impl.Unparser.wTypedNode(Unparser.java:744) ~[rest.jar:?]
  at org.apache.jena.rdfxml.xmloutput.impl.Unparser.wObj(Unparser.java:684) ~[rest.jar:?]
  at org.apache.jena.rdfxml.xmloutput.impl.Unparser.wPropertyEltValueObj(Unparser.java:581) ~[rest.jar:?]
  at org.apache.jena.rdfxml.xmloutput.impl.Unparser.wPropertyEltValue(Unparser.java:526) ~[rest.jar:?]
  at org.apache.jena.rdfxml.xmloutput.impl.Unparser.wPropertyElt(Unparser.java:391) ~[rest.jar:?]
  at org.apache.jena.rdfxml.xmloutput.impl.Unparser.wPropertyEltStar(Unparser.java:855) ~[rest.jar:?]
  at org.apache.jena.rdfxml.xmloutput.impl.Unparser.wTypedNodeOrDescriptionLong(Unparser.java:841) ~[rest.jar:?]
  at org.apache.jena.rdfxml.xmloutput.impl.Unparser.wTypedNodeOrDescription(Unparser.java:772) ~[rest.jar:?]
  at org.apache.jena.rdfxml.xmloutput.impl.Unparser.wDescription(Unparser.java:730) ~[rest.jar:?]
  at org.apache.jena.rdfxml.xmloutput.impl.Unparser.wObj(Unparser.java:686) ~[rest.jar:?]
  at org.apache.jena.rdfxml.xmloutput.impl.Unparser.wObjStar(Unparser.java:364) ~[rest.jar:?]
  at org.apache.jena.rdfxml.xmloutput.impl.Unparser.wRDF(Unparser.java:345) ~[rest.jar:?]
  at org.apache.jena.rdfxml.xmloutput.impl.Unparser.write(Unparser.java:247) ~[rest.jar:?]
  at org.apache.jena.rdfxml.xmloutput.impl.Abbreviated.writeBody(Abbreviated.java:140) ~[rest.jar:?]
  at org.apache.jena.rdfxml.xmloutput.impl.BaseXMLWriter.writeXMLBody(BaseXMLWriter.java:502) ~[rest.jar:?]
  at org.apache.jena.rdfxml.xmloutput.impl.BaseXMLWriter.write(BaseXMLWriter.java:473) ~[rest.jar:?]
  at org.apache.jena.rdfxml.xmloutput.impl.Abbreviated.write(Abbreviated.java:125) ~[rest.jar:?]
  at org.apache.jena.rdfxml.xmloutput.impl.BaseXMLWriter.write(BaseXMLWriter.java:460) ~[rest.jar:?]
  at org.apache.jena.riot.adapters.AdapterRDFWriter.write(AdapterRDFWriter.java:51) ~[rest.jar:?]
  at org.apache.jena.riot.adapters.RDFWriterRIOT.write(RDFWriterRIOT.java:94) ~[rest.jar:?]
  at whelk.converter.JsonLD2RdfXml.convert(JsonLD2RdfXml.groovy:40) ~[rest.jar:?]
  at whelk.rest.api.ConverterUtils.convert(ConverterUtils.java:41) ~[rest.jar:?]
  ...

This problem can also affect the OAI-PMH server if rdfxml is chosen.

So, let's strip XML-illegal stuff from the XML output. For the Turtle representation we strip all \p{Cntrl} but that'd also remove characters that are valid in XML, such as tabs and newlines) so we have to be narrower here.

(We should also have a whelktool script to clean up this kind of crap in our data (maybe we already do?) but that's a separate issue.)

[kaipoykio]

thumbs up!