[SPARK-58322][CORE] Verify executor app ID matches driver at config fetch

[wangyum]

What changes were proposed in this pull request?

This PR adds an executor-side app-ID verification that runs immediately after the executor fetches the driver's Spark config via RetrieveSparkAppConfig, before SparkEnv is created. The new CoarseGrainedExecutorBackend.verifyAppId compares the executor's --app-id launch argument (assigned by the cluster manager) against spark.app.id from the driver's SparkConf (set in SparkContext), and throws SparkException on mismatch.

The check is invoked in both code paths that retrieve the driver config:

• CoarseGrainedExecutorBackend.run() — covers Standalone and YARN (YARN delegates to the shared run() via YarnCoarseGrainedExecutorBackend).
• KubernetesExecutorBackend.run() — separate implementation, so the call is added there explicitly.

The verification is unconditional (always on). It only fires when spark.app.id is present in the driver properties and differs from the executor's app ID — a condition that never occurs in normal operation.

Why are the changes needed?

When a driver hits a fatal error (e.g., OOM) that kills the RPC dispatcher thread, SparkContext.stop() releases the driver's RPC port while the driver process (and in YARN cluster mode, the ApplicationMaster) can remain in a zombie state — still heartbeating to the resource manager but effectively dead. During this window, the freed port can be rebound by another driver on the same host, and newly launched executors can connect to the wrong driver and register to the wrong application, risking data corruption.

Does this PR introduce any user-facing change?

Yes. In the rare case of an app-ID mismatch, the executor will now exit immediately at startup with a SparkException instead of registering to the wrong driver. The error message is:

Executor app ID <executorAppId> does not match driver app ID <driverAppId>.
This likely means the executor connected to the wrong driver. Exiting.

No config or API changes. No impact on normal operation.

How was this patch tested?

Unit tests.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: GLM 5.2.

[uros-b]

Thank you @wangyum!