Skip to content

[MINOR][DOCS] Fix typos and add missing references to standalone and YARN deployment docs #57536

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
nchammas wants to merge 2 commits into
base: master
Choose a base branch
from
+11 −6
Open

[MINOR][DOCS] Fix typos and add missing references to standalone and YARN deployment docs #57536

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
50b6717
typos fixes and updates to reflect complete standalone scripts
nchammas Jul 25, 2026
1226c5c
we don't use jekyll-relative-links
nchammas Jul 25, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/running-on-yarn.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -773,7 +773,7 @@ Stage level scheduling is supported on YARN:
- When dynamic allocation is disabled: It allows users to specify different task resource requirements at the stage level and will use the same executors requested at startup.
- When dynamic allocation is enabled: It allows users to specify task and executor resource requirements at the stage level and will request the extra executors.

One thing to note that is YARN specific is that each ResourceProfile requires a different container priority on YARN. The mapping is simply the ResourceProfile id becomes the priority, on YARN lower numbers are higher priority. This means that profiles created earlier will have a higher priority in YARN. Normally this won't matter as Spark finishes one stage before starting another one, the only case this might have an affect is in a job server type scenario, so its something to keep in mind.
One thing to note that is YARN specific is that each ResourceProfile requires a different container priority on YARN. The mapping is simply the ResourceProfile id becomes the priority, on YARN lower numbers are higher priority. This means that profiles created earlier will have a higher priority in YARN. Normally this won't matter as Spark finishes one stage before starting another one, the only case this might have an effect is in a job server type scenario, so it's something to keep in mind.
Note there is a difference in the way custom resources are handled between the base default profile and custom ResourceProfiles. To allow for the user to request YARN containers with extra resources without Spark scheduling on them, the user can specify resources via the <code>spark.yarn.executor.resource.</code> config. Those configs are only used in the base default profile though and do not get propagated into any other custom ResourceProfiles. This is because there would be no way to remove them if you wanted a stage to not have them. This results in your default profile getting custom resources defined in <code>spark.yarn.executor.resource.</code> plus spark defined resources of GPU or FPGA. Spark converts GPU and FPGA resources into the YARN built in types <code>yarn.io/gpu</code>) and <code>yarn.io/fpga</code>, but does not know the mapping of any other resources. Any other Spark custom resources are not propagated to YARN for the default profile. So if you want Spark to schedule based off a custom resource and have it requested from YARN, you must specify it in both YARN (<code>spark.yarn.{driver/executor}.resource.</code>) and Spark (<code>spark.{driver/executor}.resource.</code>) configs. Leave the Spark config off if you only want YARN containers with the extra resources but Spark not to schedule using them. Now for custom ResourceProfiles, it doesn't currently have a way to only specify YARN resources without Spark scheduling off of them. This means for custom ResourceProfiles we propagate all the resources defined in the ResourceProfile to YARN. We still convert GPU and FPGA to the YARN build in types as well. This requires that the name of any custom resources you specify match what they are defined as in YARN.

# Important notes
Expand Down
15 changes: 10 additions & 5 deletions docs/spark-standalone.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ license: |
* This will become a table of contents (this text will be scraped).
{:toc}

In addition to running on the YARN cluster manager, Spark also provides a simple standalone deploy mode. You can launch a standalone cluster either manually, by starting a master and workers by hand, or use our provided [launch scripts](#cluster-launch-scripts). It is also possible to run these daemons on a single machine for testing.
In addition to running on the [YARN cluster manager](running-on-yarn.html) and [Kubernetes](running-on-kubernetes.html), Spark also provides a simple standalone deploy mode. You can launch a standalone cluster either manually, by starting a master and workers by hand, or use our provided [launch scripts](#cluster-launch-scripts). It is also possible to run these daemons on a single machine for testing.

# Security

Expand Down Expand Up @@ -100,13 +100,16 @@ Once you've set up this file, you can launch or stop your cluster with the follo
- `sbin/start-master.sh` - Starts a master instance on the machine the script is executed on.
- `sbin/start-workers.sh` - Starts a worker instance on each machine specified in the `conf/workers` file.
- `sbin/start-worker.sh` - Starts a worker instance on the machine the script is executed on.
- `sbin/start-connect-server.sh` - Starts a Spark Connect server on the machine the script is executed on.
- `sbin/start-connect-server.sh` - Starts a [Spark Connect](spark-connect-overview.html) server on the machine the script is executed on.
- `sbin/start-history-server.sh` - Starts the History Server, which lets you view logs for completed applications. Requires event logging to be enabled (see [Monitoring and Instrumentation](monitoring.html)).

@nchammas nchammas Jul 26, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The section this is in is called "Cluster Launch Scripts", so I don't know if we want to leave mention of the History server out of here since it gets a dedicated mention in the monitoring page. The Thrift server also gets its own call out in the "Distributed SQL Engine" page (which is currently not in this list).

I like the idea of having a complete list of the sbin scripts users may be interested in, even if some of them get additional explanation elsewhere. In that case, I'd also add the Thrift server here.

What do you think, @uros-b?

- `sbin/start-all.sh` - Starts both a master and a number of workers as described above.
- `sbin/stop-master.sh` - Stops the master that was started via the `sbin/start-master.sh` script.
- `sbin/stop-worker.sh` - Stops all worker instances on the machine the script is executed on.
- `sbin/stop-workers.sh` - Stops all worker instances on the machines specified in the `conf/workers` file.
- `sbin/stop-connect-server.sh` - Stops all Spark Connect server instances on the machine the script is executed on.
- `sbin/stop-history-server.sh` - Stops the History Server.
- `sbin/stop-all.sh` - Stops both the master and the workers as described above.
- `sbin/decommission-worker.sh` - Gracefully decommissions a worker, allowing in-progress tasks to finish and shuffles to be migrated before the worker exits.

Note that these scripts must be executed on the machine you want to run the Spark master on, not your local machine.

Expand Down Expand Up @@ -687,7 +690,7 @@ configurations, <code>curl</code> CLI command can provide the required header li

```bash
$ curl -XPOST http://IP:PORT/v1/submissions/create \
--header "Authorization: Bearer USER-PROVIDED-WEB-TOEN-SIGNED-BY-THE-SAME-SHARED-KEY"
--header "Authorization: Bearer USER-PROVIDED-WEB-TOKEN-SIGNED-BY-THE-SAME-SHARED-KEY"
...
```

Expand Down Expand Up @@ -748,7 +751,7 @@ worker during one single schedule iteration.

Stage level scheduling is supported on Standalone:
- When dynamic allocation is disabled: It allows users to specify different task resource requirements at the stage level and will use the same executors requested at startup.
- When dynamic allocation is enabled: Currently, when the Master allocates executors for one application, it will schedule based on the order of the ResourceProfile ids for multiple ResourceProfiles. The ResourceProfile with smaller id will be scheduled firstly. Normally this won’t matter as Spark finishes one stage before starting another one, the only case this might have an affect is in a job server type scenario, so its something to keep in mind. For scheduling, we will only take executor memory and executor cores from built-in executor resources and all other custom resources from a ResourceProfile, other built-in executor resources such as offHeap and memoryOverhead won't take any effect. The base default profile will be created based on the spark configs when you submit an application. Executor memory and executor cores from the base default profile can be propagated to custom ResourceProfiles, but all other custom resources can not be propagated.
- When dynamic allocation is enabled: Currently, when the Master allocates executors for one application, it will schedule based on the order of the ResourceProfile ids for multiple ResourceProfiles. The ResourceProfile with smaller id will be scheduled first. Normally this won’t matter as Spark finishes one stage before starting another one; the only case this might have an effect is in a job server type scenario, so it’s something to keep in mind. For scheduling, we will only take executor memory and executor cores from built-in executor resources and all other custom resources from a ResourceProfile; other built-in executor resources such as `offHeap` and `memoryOverhead` won't take any effect. The base default profile will be created based on the Spark configs when you submit an application. Executor memory and executor cores from the base default profile can be propagated to custom ResourceProfiles, but all other custom resources can not be propagated.

## Caveats

Expand All @@ -760,6 +763,8 @@ Spark's standalone mode offers a web-based user interface to monitor the cluster

In addition, detailed log output for each job is also written to the work directory of each worker node (`SPARK_HOME/work` by default). You will see two files for each job, `stdout` and `stderr`, with all output it wrote to its console.

To track and review logs across completed applications, [enable event logging and start the History Server](monitoring.html#viewing-after-the-fact).


# Running Alongside Hadoop

Expand Down Expand Up @@ -835,7 +840,7 @@ In order to enable this recovery mode, you can set SPARK_DAEMON_JAVA_OPTS in spa
<td><code>spark.deploy.recoveryDirectory</code></td>
<td>""</td>
<td>The directory in which Spark will store recovery state, accessible from the Master's perspective.
Note that the directory should be clearly manually if <code>spark.deploy.recoveryMode</code>
Note that the directory should be cleared manually if <code>spark.deploy.recoveryMode</code>
or <code>spark.deploy.recoveryCompressionCodec</code> is changed.
</td>
<td>0.8.1</td>
Expand Down