[SPARK-58794][SQL] Empty2Null, text, and Hive prune fallback for CHAR/VARCHAR

[srielau]

What changes were proposed in this pull request?

Follow-up on first-class CHAR/VARCHAR (spark.sql.charVarchar.standardSemantics.enabled) so a few string-family call sites treat CharType / VarcharType like STRING.

• V1Writes applies Empty2Null to nullable CHAR/VARCHAR partition columns (dataType.isInstanceOf[StringType]). CHAR(n>0) still pads '' to spaces, so the empty CHAR case is CHAR(0).
• The V2 text data source accepts CHAR/VARCHAR as a string-family type.
• Hive metastore filter conversion still refuses CHAR/VARCHAR partition keys (varcharKeys in SupportedAttribute: Hive's trailing-blank comparison is not Spark's). Under standard semantics that would scan every partition, so prunePartitionsFastFallback now also runs when a partition schema contains CHAR/VARCHAR. Pruning is then client-side with Spark's own predicates (CHAR compared without PAD SPACE; the test literal is 'a ' for CHAR(5)).

This is independent of #58130.

Why are the changes needed?

With first-class types, CHAR/VARCHAR stay in the plan instead of being rewritten to annotated STRING. Equality against StringType then skips them:

• empty partition values are not converted to NULL, so they become a distinct partition directory instead of __HIVE_DEFAULT_PARTITION__
• USING text rejects a CHAR/VARCHAR schema
• Hive partition filters on CHAR keys fetch every partition

The annotation-skipping idea was tried and dropped: ApplyCharTypePadding uses __CHAR_VARCHAR_TYPE_STRING as an idempotence marker, so the annotation is load-bearing.

Does this PR introduce any user-facing change?

Yes, only when spark.sql.charVarchar.standardSemantics.enabled is true (still off by default).

• Empty VARCHAR / CHAR(0) partition values become NULL like STRING.
• spark.read.schema("value CHAR(n)").text(...) is accepted.
• Hive CHAR partition filters prune to the matching partitions (client-side), and CHAR(5) = 'a' does not match a stored 'a ' unless the literal carries the pad or an RTRIM collation is used.

How was this patch tested?

• sql/testOnly *CharVarcharTestSuite *V1WriteCommandSuite: 164 succeeded
• hive/testOnly *HiveCharVarcharTestSuite: 58 succeeded
• hive/testOnly *HivePartitionFilteringSuite*: 360 succeeded (Hive 2.3-4.1)

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Cursor Grok 4.6