From 50b6717d3d5acb229552d93562822f565160d8b7 Mon Sep 17 00:00:00 2001 From: Nicholas Chammas Date: Sat, 25 Jul 2026 13:51:49 -0400 Subject: [PATCH 1/2] typos fixes and updates to reflect complete standalone scripts --- docs/running-on-yarn.md | 2 +- docs/spark-standalone.md | 15 ++++++++++----- 2 files changed, 11 insertions(+), 6 deletions(-) diff --git a/docs/running-on-yarn.md b/docs/running-on-yarn.md index f7f356822f63f..bd82a84453888 100644 --- a/docs/running-on-yarn.md +++ b/docs/running-on-yarn.md @@ -773,7 +773,7 @@ Stage level scheduling is supported on YARN: - When dynamic allocation is disabled: It allows users to specify different task resource requirements at the stage level and will use the same executors requested at startup. - When dynamic allocation is enabled: It allows users to specify task and executor resource requirements at the stage level and will request the extra executors. -One thing to note that is YARN specific is that each ResourceProfile requires a different container priority on YARN. The mapping is simply the ResourceProfile id becomes the priority, on YARN lower numbers are higher priority. This means that profiles created earlier will have a higher priority in YARN. Normally this won't matter as Spark finishes one stage before starting another one, the only case this might have an affect is in a job server type scenario, so its something to keep in mind. +One thing to note that is YARN specific is that each ResourceProfile requires a different container priority on YARN. The mapping is simply the ResourceProfile id becomes the priority, on YARN lower numbers are higher priority. This means that profiles created earlier will have a higher priority in YARN. Normally this won't matter as Spark finishes one stage before starting another one, the only case this might have an effect is in a job server type scenario, so it's something to keep in mind. Note there is a difference in the way custom resources are handled between the base default profile and custom ResourceProfiles. To allow for the user to request YARN containers with extra resources without Spark scheduling on them, the user can specify resources via the spark.yarn.executor.resource. config. Those configs are only used in the base default profile though and do not get propagated into any other custom ResourceProfiles. This is because there would be no way to remove them if you wanted a stage to not have them. This results in your default profile getting custom resources defined in spark.yarn.executor.resource. plus spark defined resources of GPU or FPGA. Spark converts GPU and FPGA resources into the YARN built in types yarn.io/gpu) and yarn.io/fpga, but does not know the mapping of any other resources. Any other Spark custom resources are not propagated to YARN for the default profile. So if you want Spark to schedule based off a custom resource and have it requested from YARN, you must specify it in both YARN (spark.yarn.{driver/executor}.resource.) and Spark (spark.{driver/executor}.resource.) configs. Leave the Spark config off if you only want YARN containers with the extra resources but Spark not to schedule using them. Now for custom ResourceProfiles, it doesn't currently have a way to only specify YARN resources without Spark scheduling off of them. This means for custom ResourceProfiles we propagate all the resources defined in the ResourceProfile to YARN. We still convert GPU and FPGA to the YARN build in types as well. This requires that the name of any custom resources you specify match what they are defined as in YARN. # Important notes diff --git a/docs/spark-standalone.md b/docs/spark-standalone.md index ec1656b0348c8..39abb14b0d4de 100644 --- a/docs/spark-standalone.md +++ b/docs/spark-standalone.md @@ -21,7 +21,7 @@ license: | * This will become a table of contents (this text will be scraped). {:toc} -In addition to running on the YARN cluster manager, Spark also provides a simple standalone deploy mode. You can launch a standalone cluster either manually, by starting a master and workers by hand, or use our provided [launch scripts](#cluster-launch-scripts). It is also possible to run these daemons on a single machine for testing. +In addition to running on the [YARN cluster manager](./running-on-yarn.md) and [Kubernetes](./running-on-kubernetes.md), Spark also provides a simple standalone deploy mode. You can launch a standalone cluster either manually, by starting a master and workers by hand, or use our provided [launch scripts](#cluster-launch-scripts). It is also possible to run these daemons on a single machine for testing. # Security @@ -100,13 +100,16 @@ Once you've set up this file, you can launch or stop your cluster with the follo - `sbin/start-master.sh` - Starts a master instance on the machine the script is executed on. - `sbin/start-workers.sh` - Starts a worker instance on each machine specified in the `conf/workers` file. - `sbin/start-worker.sh` - Starts a worker instance on the machine the script is executed on. -- `sbin/start-connect-server.sh` - Starts a Spark Connect server on the machine the script is executed on. +- `sbin/start-connect-server.sh` - Starts a [Spark Connect](spark-connect-overview.html) server on the machine the script is executed on. +- `sbin/start-history-server.sh` - Starts the History Server, which lets you view logs for completed applications. Requires event logging to be enabled (see [Monitoring and Instrumentation](./monitoring.md)). - `sbin/start-all.sh` - Starts both a master and a number of workers as described above. - `sbin/stop-master.sh` - Stops the master that was started via the `sbin/start-master.sh` script. - `sbin/stop-worker.sh` - Stops all worker instances on the machine the script is executed on. - `sbin/stop-workers.sh` - Stops all worker instances on the machines specified in the `conf/workers` file. - `sbin/stop-connect-server.sh` - Stops all Spark Connect server instances on the machine the script is executed on. +- `sbin/stop-history-server.sh` - Stops the History Server. - `sbin/stop-all.sh` - Stops both the master and the workers as described above. +- `sbin/decommission-worker.sh` - Gracefully decommissions a worker, allowing in-progress tasks to finish and shuffles to be migrated before the worker exits. Note that these scripts must be executed on the machine you want to run the Spark master on, not your local machine. @@ -687,7 +690,7 @@ configurations, curl CLI command can provide the required header li ```bash $ curl -XPOST http://IP:PORT/v1/submissions/create \ ---header "Authorization: Bearer USER-PROVIDED-WEB-TOEN-SIGNED-BY-THE-SAME-SHARED-KEY" +--header "Authorization: Bearer USER-PROVIDED-WEB-TOKEN-SIGNED-BY-THE-SAME-SHARED-KEY" ... ``` @@ -748,7 +751,7 @@ worker during one single schedule iteration. Stage level scheduling is supported on Standalone: - When dynamic allocation is disabled: It allows users to specify different task resource requirements at the stage level and will use the same executors requested at startup. -- When dynamic allocation is enabled: Currently, when the Master allocates executors for one application, it will schedule based on the order of the ResourceProfile ids for multiple ResourceProfiles. The ResourceProfile with smaller id will be scheduled firstly. Normally this won’t matter as Spark finishes one stage before starting another one, the only case this might have an affect is in a job server type scenario, so its something to keep in mind. For scheduling, we will only take executor memory and executor cores from built-in executor resources and all other custom resources from a ResourceProfile, other built-in executor resources such as offHeap and memoryOverhead won't take any effect. The base default profile will be created based on the spark configs when you submit an application. Executor memory and executor cores from the base default profile can be propagated to custom ResourceProfiles, but all other custom resources can not be propagated. +- When dynamic allocation is enabled: Currently, when the Master allocates executors for one application, it will schedule based on the order of the ResourceProfile ids for multiple ResourceProfiles. The ResourceProfile with smaller id will be scheduled first. Normally this won’t matter as Spark finishes one stage before starting another one; the only case this might have an effect is in a job server type scenario, so it’s something to keep in mind. For scheduling, we will only take executor memory and executor cores from built-in executor resources and all other custom resources from a ResourceProfile; other built-in executor resources such as `offHeap` and `memoryOverhead` won't take any effect. The base default profile will be created based on the Spark configs when you submit an application. Executor memory and executor cores from the base default profile can be propagated to custom ResourceProfiles, but all other custom resources can not be propagated. ## Caveats @@ -760,6 +763,8 @@ Spark's standalone mode offers a web-based user interface to monitor the cluster In addition, detailed log output for each job is also written to the work directory of each worker node (`SPARK_HOME/work` by default). You will see two files for each job, `stdout` and `stderr`, with all output it wrote to its console. +For a richer monitoring experience across completed applications, [enable event logging and start the History Server](./monitoring.md#viewing-after-the-fact). + # Running Alongside Hadoop @@ -835,7 +840,7 @@ In order to enable this recovery mode, you can set SPARK_DAEMON_JAVA_OPTS in spa spark.deploy.recoveryDirectory "" The directory in which Spark will store recovery state, accessible from the Master's perspective. - Note that the directory should be clearly manually if spark.deploy.recoveryMode + Note that the directory should be cleared manually if spark.deploy.recoveryMode or spark.deploy.recoveryCompressionCodec is changed. 0.8.1 From 1226c5ccc1cf87362136d4e10332e35e80640cd9 Mon Sep 17 00:00:00 2001 From: Nicholas Chammas Date: Sat, 25 Jul 2026 14:45:33 -0400 Subject: [PATCH 2/2] we don't use jekyll-relative-links --- docs/spark-standalone.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/spark-standalone.md b/docs/spark-standalone.md index 39abb14b0d4de..b35e74d246adf 100644 --- a/docs/spark-standalone.md +++ b/docs/spark-standalone.md @@ -21,7 +21,7 @@ license: | * This will become a table of contents (this text will be scraped). {:toc} -In addition to running on the [YARN cluster manager](./running-on-yarn.md) and [Kubernetes](./running-on-kubernetes.md), Spark also provides a simple standalone deploy mode. You can launch a standalone cluster either manually, by starting a master and workers by hand, or use our provided [launch scripts](#cluster-launch-scripts). It is also possible to run these daemons on a single machine for testing. +In addition to running on the [YARN cluster manager](running-on-yarn.html) and [Kubernetes](running-on-kubernetes.html), Spark also provides a simple standalone deploy mode. You can launch a standalone cluster either manually, by starting a master and workers by hand, or use our provided [launch scripts](#cluster-launch-scripts). It is also possible to run these daemons on a single machine for testing. # Security @@ -101,7 +101,7 @@ Once you've set up this file, you can launch or stop your cluster with the follo - `sbin/start-workers.sh` - Starts a worker instance on each machine specified in the `conf/workers` file. - `sbin/start-worker.sh` - Starts a worker instance on the machine the script is executed on. - `sbin/start-connect-server.sh` - Starts a [Spark Connect](spark-connect-overview.html) server on the machine the script is executed on. -- `sbin/start-history-server.sh` - Starts the History Server, which lets you view logs for completed applications. Requires event logging to be enabled (see [Monitoring and Instrumentation](./monitoring.md)). +- `sbin/start-history-server.sh` - Starts the History Server, which lets you view logs for completed applications. Requires event logging to be enabled (see [Monitoring and Instrumentation](monitoring.html)). - `sbin/start-all.sh` - Starts both a master and a number of workers as described above. - `sbin/stop-master.sh` - Stops the master that was started via the `sbin/start-master.sh` script. - `sbin/stop-worker.sh` - Stops all worker instances on the machine the script is executed on. @@ -763,7 +763,7 @@ Spark's standalone mode offers a web-based user interface to monitor the cluster In addition, detailed log output for each job is also written to the work directory of each worker node (`SPARK_HOME/work` by default). You will see two files for each job, `stdout` and `stderr`, with all output it wrote to its console. -For a richer monitoring experience across completed applications, [enable event logging and start the History Server](./monitoring.md#viewing-after-the-fact). +To track and review logs across completed applications, [enable event logging and start the History Server](monitoring.html#viewing-after-the-fact). # Running Alongside Hadoop