[SPARK-58973][SS] Fix state store batch read failure when StreamingQueryManager is not initialized

[shrirangmhalgi]

What changes were proposed in this pull request?

Make StateStoreProvider.coordinatorRef gracefully return None when the StateStoreCoordinator RPC endpoint is not available, instead of propagating SparkException. This is done by wrapping the forExecutor call in a try-catch that catches SparkException (which wraps RpcEndpointNotFoundException from awaitResult), logs a warning, and returns None.

Why are the changes needed?

PR #50123 added reportSnapshotUploadToCoordinator() to HDFSBackedStateStoreProvider.loadMap(), creating an unconditional dependency on the StateStoreCoordinator from the state store read path. The coordinator endpoint is only registered when StreamingQueryManager is instantiated — which is intentionally a lazy val in SessionState (since SPARK-29423). In a fresh session that only does batch reads of state store data (e.g., spark.read.format("statestore").load(path)), the endpoint doesn't exist and the read fails with [CANNOT_LOAD_STATE_STORE.UNCATEGORIZED].

Does this PR introduce any user-facing change?

Yes. spark.read.format("statestore").load(checkpointDir) now works in sessions that have never started a streaming query, without requiring the workaround of accessing spark.streams first or setting spark.sql.streaming.stateStore.coordinatorReportSnapshotUploadLag=false.

How was this patch tested?

Added StateStoreCoordinatorBatchReadSuite with two tests:

1. Unit test verifying StateStoreProvider.coordinatorRef returns None after coordinator shutdown
2. Integration test verifying batch state store reads succeed without the coordinator

Was this patch authored or co-authored using generative AI tooling?

Yes. Co-Authored using Claude Opus 4.8

Closes #58211

[shrirangmhalgi]

@HeartSaVioR / @anishshri-db / @Kimahriman could you please review. This fixes the batch state store read failure reported in #58211.