[spark] Add paimon spark4.1 module

[kerwin-zk]

Purpose

Add paimon spark4.1 module

Tests

CI

[Copilot]

Pull request overview

Adds a new Spark 4.1 build target to the Paimon Spark integration, wiring it into the spark4 Maven profile and introducing a Spark 4.1.1 module with matching unit tests/resources.

Changes:

• Register paimon-spark/paimon-spark-4.1 under the root spark4 Maven profile.
• Add new paimon-spark-4.1 Maven module targeting Spark 4.1.1.
• Add Spark 4.1 test suite wrappers/resources (copied/aligned with existing Spark 4.0 tests) plus optimizer rule source.

Reviewed changes

Copilot reviewed 29 out of 30 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
pom.xml	Adds paimon-spark/paimon-spark-4.1 to the spark4 profile modules.
paimon-spark/paimon-spark-4.1/pom.xml	Defines the new Spark 4.1.1 module and dependencies/shading setup.
paimon-spark/paimon-spark-4.1/src/main/scala/org/apache/paimon/spark/catalyst/optimizer/MergePaimonScalarSubqueries.scala	Spark 4.1 module implementation of the scalar-subquery merge rule.
paimon-spark/paimon-spark-4.1/src/test/scala/org/apache/paimon/spark/sql/AnalyzeTableTest.scala	Spark 4.1 test wrapper extending shared base tests.
paimon-spark/paimon-spark-4.1/src/test/scala/org/apache/paimon/spark/sql/DataFrameWriteTest.scala	Spark 4.1 test wrapper extending shared base tests.
paimon-spark/paimon-spark-4.1/src/test/scala/org/apache/paimon/spark/sql/DDLTest.scala	Spark 4.1 test wrapper extending shared base tests.
paimon-spark/paimon-spark-4.1/src/test/scala/org/apache/paimon/spark/sql/DDLWithHiveCatalogTest.scala	Spark 4.1 Hive-catalog DDL test wrappers extending shared base tests.
paimon-spark/paimon-spark-4.1/src/test/scala/org/apache/paimon/spark/sql/DeleteFromTableTest.scala	Spark 4.1 delete tests toggling v2-write config.
paimon-spark/paimon-spark-4.1/src/test/scala/org/apache/paimon/spark/sql/DescribeTableTest.scala	Spark 4.1 test wrapper extending shared base tests.
paimon-spark/paimon-spark-4.1/src/test/scala/org/apache/paimon/spark/sql/FormatTableTest.scala	Spark 4.1 test wrapper extending shared base tests.
paimon-spark/paimon-spark-4.1/src/test/scala/org/apache/paimon/spark/sql/InsertOverwriteTableTest.scala	Spark 4.1 test wrapper extending shared base tests.
paimon-spark/paimon-spark-4.1/src/test/scala/org/apache/paimon/spark/sql/MergeIntoTableTest.scala	Spark 4.1 merge-into variants toggling bucket/non-bucket and v2-write config.
paimon-spark/paimon-spark-4.1/src/test/scala/org/apache/paimon/spark/sql/PaimonCompositePartitionKeyTest.scala	Spark 4.1 test wrapper extending shared base tests.
paimon-spark/paimon-spark-4.1/src/test/scala/org/apache/paimon/spark/sql/PaimonOptimizationTest.scala	Spark 4.1 optimization tests with Spark shim usage for CTE refs.
paimon-spark/paimon-spark-4.1/src/test/scala/org/apache/paimon/spark/sql/PaimonPushDownTest.scala	Spark 4.1 pushdown test wrapper extending shared base tests.
paimon-spark/paimon-spark-4.1/src/test/scala/org/apache/paimon/spark/sql/PaimonV1FunctionTest.scala	Spark 4.1 test wrapper extending shared base tests.
paimon-spark/paimon-spark-4.1/src/test/scala/org/apache/paimon/spark/sql/PaimonViewTest.scala	Spark 4.1 view test wrapper extending shared base tests.
paimon-spark/paimon-spark-4.1/src/test/scala/org/apache/paimon/spark/sql/RewriteUpsertTableTest.scala	Spark 4.1 test wrapper extending shared base tests.
paimon-spark/paimon-spark-4.1/src/test/scala/org/apache/paimon/spark/sql/RowIdPushDownTest.scala	Spark 4.1 test wrapper extending shared base tests.
paimon-spark/paimon-spark-4.1/src/test/scala/org/apache/paimon/spark/sql/RowTrackingTest.scala	Spark 4.1 test wrapper extending shared base tests.
paimon-spark/paimon-spark-4.1/src/test/scala/org/apache/paimon/spark/sql/ShowColumnsTest.scala	Spark 4.1 test wrapper extending shared base tests.
paimon-spark/paimon-spark-4.1/src/test/scala/org/apache/paimon/spark/sql/SparkV2FilterConverterTest.scala	Spark 4.1 test wrapper extending shared base tests.
paimon-spark/paimon-spark-4.1/src/test/scala/org/apache/paimon/spark/sql/TagDdlTest.scala	Spark 4.1 tag-DDL test wrapper extending shared base tests.
paimon-spark/paimon-spark-4.1/src/test/scala/org/apache/paimon/spark/sql/UpdateTableTest.scala	Spark 4.1 update tests toggling v2-write config.
paimon-spark/paimon-spark-4.1/src/test/scala/org/apache/paimon/spark/sql/VariantTest.scala	Spark 4.1 variant tests toggling shredding-schema inference config.
paimon-spark/paimon-spark-4.1/src/test/scala/org/apache/paimon/spark/procedure/CompactProcedureTest.scala	Spark 4.1 procedure test wrapper extending shared base tests.
paimon-spark/paimon-spark-4.1/src/test/scala/org/apache/paimon/spark/procedure/ProcedureTest.scala	Spark 4.1 procedure test wrapper extending shared base tests.
paimon-spark/paimon-spark-4.1/src/test/resources/hive-site.xml	Spark 4.1 module Hive metastore test configuration.
paimon-spark/paimon-spark-4.1/src/test/resources/log4j2-test.properties	Spark 4.1 module test logging configuration.
paimon-spark/paimon-spark-4.1/src/test/resources/function/hive-test-udfs.jar	Spark 4.1 module test UDF jar used by Hive-related tests.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

In the spark4 Maven profile, adding the paimon-spark-4.1 module means the reactor build now pulls Spark 4.1.1 (from paimon-spark-4.1/pom.xml) while the profile properties still pin paimon-spark-common.spark.version / test.spark.* to 4.0.x. This mixes Spark minor versions within the same profile and can make it unclear which Spark version the shared paimon-spark4-common and any Spark-based tests/e2e tooling are targeting. Consider either updating the spark4 profile properties to 4.1.x (if 4.1 is now the baseline) or introducing a separate spark41 profile so the common module and test properties stay aligned with the included Spark modules.

[YannByron]

The MergePaimonScalarSubqueries in paimon-spark-4.1 is identical to the one in paimon-spark-4.0. Would it be better to move these version-agnostic implementations into paimon-spark4-common instead of duplicating them in each version-specific module?

[YannByron]

Maybe hive-test-udfs.jar、hive-site.xml and log4j2-test.properties, consider these as well.

[kerwin-zk]

@YannByron log4j2-test.properties and function/hive-test-udfs.jar have been removed from paimon-spark-4.1 — paimon-spark-ut already ships identical copies on the test classpath. hive-site.xml is kept per-module on purpose. paimon-spark-3.2 through paimon-spark-4.0 each carry their own copy with a distinct hive.metastore.uris port (3.2→9087, 3.3→9088, 3.4→9089, 3.5→9090, 4.0→9090) so the ITCases can run in parallel without port collisions; paimon-spark-4.1 follows the same pattern with 9091.