ZOOKEEPER-5070: Support Single EKU certificates by PDavid · Pull Request #2429 · apache/zookeeper

PDavid · 2026-07-30T09:25:18Z

Goal

Support certificates with a single Extended Key Usage (EKU) - one cert with serverAuth for incoming connections, another with clientAuth for outgoing connections

Approach

Use separate SSLContext instances for client and server role instead of one. Each is initialized with standard PKIX managers from its own keystore / truststore.

If these new properties are not set, ZooKeeper falls back to the existing keyStore.* and trustStore.* configs (shared for both roles), so the change is fully backward-compatible.

Configuration changes

Introduces the following new config properties:

Client keystore

Presented when initiating connections (client role). Allows using a certificate with only clientAuth EKU.

zookeeper.ssl.client.keyStore.location=/etc/zookeeper/certs/client-keystore.jks
zookeeper.ssl.client.keyStore.password=clientKeystorePass
zookeeper.ssl.client.keyStore.type=JKS

Server truststore

Used when accepting connections (server role). Validates the connecting client's certificate during mTLS.

zookeeper.ssl.server.trustStore.location=/etc/zookeeper/certs/server-truststore.jks
zookeeper.ssl.server.trustStore.password=serverTruststorePass
zookeeper.ssl.server.trustStore.type=JKS

Quorum client keystore

zookeeper.ssl.quorum.client.keyStore.location=/etc/zookeeper/certs/quorum-client-keystore.jks
zookeeper.ssl.quorum.client.keyStore.password=quorumClientKeystorePass

Quorum server truststore

Validates clients when accepting connections

zookeeper.ssl.quorum.server.trustStore.location=/etc/zookeeper/certs/quorum-server-truststore.jks
zookeeper.ssl.quorum.server.trustStore.password=quorumServerTruststorePass

TODO

Add documentation

anmolnar

Please add documentation to zookeeper-website/app/pages/_docs/docs/_mdx/admin-ops/administrators-guide/communication-using-the-netty-framework.mdx

anmolnar

Please help me to understand the concept better.

You introduced only the following new properties:

ssl.(quorum).client.keyStore
ssl.(quorum).server.trustStore

Shouldn't we also have:

ssl.(quorum).client.trustStore
ssl.(quorum).server.keyStore

In my understand and EKU-supported zoo.cfg should look like this:

secureClientPort=2182
clientCnxnSocket=org.apache.zookeeper.ClientCnxnSocketNetty
ssl.server.keyStore.location=/Users/andor/work/ssl/server-keystore.jks
ssl.server.keyStore.password=password
ssl.server.trustStore.location=/Users/andor/work/ssl/server-truststore.jks
ssl.server.trustStore.password=password
ssl.client.keyStore.location=/Users/andor/work/ssl/client-keystore.jks
ssl.client.keyStore.password=password
ssl.client.trustStore.location=/Users/andor/work/ssl/client-truststore.jks
ssl.client.trustStore.password=password
...
sslQuorum=true
serverCnxnFactory=org.apache.zookeeper.server.NettyServerCnxnFactory
ssl.quorum.server.keyStore.location=/Users/andor/work/ssl/server-keystore.jks
ssl.quorum.server.keyStore.password=password
ssl.quorum.server.trustStore.location=/Users/andor/work/ssl/server-truststore.jks
ssl.quorum.server.trustStore.password=password
ssl.quorum.client.keyStore.location=/Users/andor/work/ssl/client-keystore.jks
ssl.quorum.client.keyStore.password=password
ssl.quorum.client.trustStore.location=/Users/andor/work/ssl/client-truststore.jks
ssl.quorum.client.trustStore.password=password

The original ssl.(quorum).keystore/truststore settings should be blank in the EKU case and vica versa.

PDavid · 2026-09-01T13:04:57Z

Please add documentation to zookeeper-website/app/pages/_docs/docs/_mdx/admin-ops/administrators-guide/communication-using-the-netty-framework.mdx

Thanks, this is a very good point, I'll add it. 👍

PDavid · 2026-09-01T13:22:23Z

Please help me to understand the concept better.

You introduced only the following new properties:
ssl.(quorum).client.keyStore
ssl.(quorum).server.trustStore
Shouldn't we also have:
ssl.(quorum).client.trustStore
ssl.(quorum).server.keyStore

Sorry, I did not mentioned this in the PR but my idea was to only introduce new TLS properties which were not yet present and use the existing TLS properties with the existing names.

Here is an example of the relevant TLS properties from zoo.cfg:

...

# === Existing TLS properties (unchanged) ===

# Server keystore — presented when accepting connections (server role)
zookeeper.ssl.keyStore.location=/etc/zookeeper/certs/server-keystore.jks
zookeeper.ssl.keyStore.password=serverKeystorePass
zookeeper.ssl.keyStore.type=JKS

# Truststore — used when connecting to other nodes (client role)
# Validates the remote server's certificate
zookeeper.ssl.trustStore.location=/etc/zookeeper/certs/client-truststore.jks
zookeeper.ssl.trustStore.password=clientTruststorePass
zookeeper.ssl.trustStore.type=JKS

# === New properties ===

# Client keystore — presented when initiating connections (client role)
# Allows using a certificate with only clientAuth EKU
zookeeper.ssl.client.keyStore.location=/etc/zookeeper/certs/client-keystore.jks
zookeeper.ssl.client.keyStore.password=clientKeystorePass
zookeeper.ssl.client.keyStore.type=JKS

# Server truststore — used when accepting connections (server role)
# Validates the connecting client's certificate during mTLS
zookeeper.ssl.server.trustStore.location=/etc/zookeeper/certs/server-truststore.jks
zookeeper.ssl.server.trustStore.password=serverTruststorePass
zookeeper.ssl.server.trustStore.type=JKS

And the quorum equivalents for inter-node communication:

# Quorum server keystore
zookeeper.ssl.quorum.keyStore.location=/etc/zookeeper/certs/quorum-server-keystore.jks
zookeeper.ssl.quorum.keyStore.password=quorumServerKeystorePass

# Quorum client keystore (New)
zookeeper.ssl.quorum.client.keyStore.location=/etc/zookeeper/certs/quorum-client-keystore.jks
zookeeper.ssl.quorum.client.keyStore.password=quorumClientKeystorePass

# Quorum truststore (validates servers when connecting as client)
zookeeper.ssl.quorum.trustStore.location=/etc/zookeeper/certs/quorum-client-truststore.jks
zookeeper.ssl.quorum.trustStore.password=quorumClientTruststorePass

# Quorum server truststore (New)
# validates clients when accepting connections
zookeeper.ssl.quorum.server.trustStore.location=/etc/zookeeper/certs/quorum-server-truststore.jks
zookeeper.ssl.quorum.server.trustStore.password=quorumServerTruststorePass

...

This approach makes less changes but might be a bit confusing as we don't have some client / server counterpart properties.

What do you think? Is this a good approach or should we introduce those additional new properties?

anmolnar · 2026-09-02T19:25:06Z

I've found (thanks to Codex/ChatGPT) the bug with client mTLS authentication. You missed to modify X509AuthenticationProvider to adopt the new configuration:

tm = x509Util.buildTrustManager(config);

should be

tm = x509Util.buildServerTrustManager(config);

Otherwise it builds the trust relation chain with the wrong truststore. You might also want to add a unit test to X509AuthTest to cover this scenario.

p.s. I'm still not sure about the config parameters. Your approach makes sense to me too.

anmolnar · 2026-09-02T19:38:39Z

I created a pull request with the code that ChatGPT generated:
PDavid#2

anmolnar

lgtm.

PDavid · 2026-09-10T14:03:59Z

Hi @anmolnar,

I added documentation about single EKU support and new config parameters to app/pages/_docs/docs/_mdx/admin-ops/administrators-guide/configuration-parameters.mdx.

EDIT:
Do you think it is OK like this? Or should we have a step-by-step how-to guide for single EKU like we have on the Netty page?

Added client keystore, server truststore config

…pport Replace custom ClientServerX509KeyManager and ClientServerX509TrustManager wrappers with two separate SSLContext instances — one for client role (outgoing connections) and one for server role (incoming connections). The custom wrappers were passed to SSLContext.init(), which is rejected by JVM-level FIPS providers that only accept their own validated manager implementations. The new approach initializes each context with standard PKIX managers from their respective keystores, making single-EKU certificate support fully FIPS-compatible. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

…d unit test (#2)

…client/server stores In the default deployment where ssl.client.keyStore.location and ssl.server.trustStore.location are not set, createSSLContextAndOptionsFromConfig loaded key/truststores twice

…ake cleanup Problem: finally block did not clear getSslKeystoreLocationProperty, leaving a deleted temp file path in system properties if serverSocket.close() throws. Fix: - Moved serverSocket.close() to the end of finally block - property cleanup now happens first, so an exception from close() can't skip it. - Added cleanup for the three shared keystore properties that were set in the test but never cleared.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

PDavid · 2026-09-11T07:37:58Z

Testing

Tested the changes with single EKU certificates locally.

With the help of Claude Code created a shell script to generate single EKU certs, keystores, truststores and config for a local quorum testing: Test ZooKeeper local quorum with single EKU certificates

Executed the script, then started quorum, performed the tests manually:

$ ls -la dev/single-eku-test/
total 80
drwxrwx--- 6 user user 4096 Sep 11 09:09 .
drwxrwx--- 4 user user 4096 Sep 11 09:09 ..
-rw-rw---- 1 user user 2576 Sep 11 09:09 ca.jks
-rw-rw---- 1 user user 1073 Sep 11 09:09 ca.pem
-rw-rw---- 1 user user  977 Sep 11 09:09 client.csr
-rw-rw---- 1 user user 4312 Sep 11 09:09 client-keystore.jks
-rw-rw---- 1 user user 4312 Sep 11 09:09 client.p12
-rw-rw---- 1 user user 1171 Sep 11 09:09 client.pem
drwxrwx--- 2 user user 4096 Sep 11 09:09 data
drwxrwx--- 3 user user 4096 Sep 11 09:10 node1
drwxrwx--- 3 user user 4096 Sep 11 09:10 node2
drwxrwx--- 3 user user 4096 Sep 11 09:10 node3
-rw-rw---- 1 user user  977 Sep 11 09:09 server.csr
-rw-rw---- 1 user user 4312 Sep 11 09:09 server-keystore.jks
-rw-rw---- 1 user user 1171 Sep 11 09:09 server.pem
-rw-rw---- 1 user user 1110 Sep 11 09:09 server-truststore.jks
-rw-rw---- 1 user user 1110 Sep 11 09:09 truststore.jks
$ dev/setup-single-eku-test.sh 
=== 1. Generate CA ===
Generating 2,048 bit RSA key pair and self-signed certificate (SHA256withRSA) with a validity of 365 days
	for: CN=TestCA
Certificate stored in file <zookeeper/dev/single-eku-test/ca.pem>
Certificate was added to keystore
Certificate was added to keystore
=== 2. Generate server cert (serverAuth EKU only) ===
Generating 2,048 bit RSA key pair and self-signed certificate (SHA256withRSA) with a validity of 365 days
	for: CN=localhost
Certificate was added to keystore
Certificate reply was installed in keystore
=== 3. Generate client cert (clientAuth EKU only) ===
Generating 2,048 bit RSA key pair and self-signed certificate (SHA256withRSA) with a validity of 365 days
	for: CN=zk-client
Certificate was added to keystore
Certificate reply was installed in keystore
=== 3b. Export client keystore to PKCS12 ===
Importing keystore zookeeper/dev/single-eku-test/client-keystore.jks to zookeeper/dev/single-eku-test/client.p12...
Entry for alias client successfully imported.
Entry for alias ca successfully imported.
Import command completed:  2 entries successfully imported, 0 entries failed or cancelled
=== 4. Verify EKUs ===
--- Server cert EKU (expect serverAuth only):
ExtendedKeyUsages [
  serverAuth
]
--- Client cert EKU (expect clientAuth only):
ExtendedKeyUsages [
  clientAuth
]
=== 5. Generate 3-node quorum configs ===
  wrote conf/zoo-quorum-1.cfg  (clientPort=2181, secureClientPort=2281, adminPort=8081, metricsPort=7071)
  wrote conf/zoo-quorum-2.cfg  (clientPort=2182, secureClientPort=2282, adminPort=8082, metricsPort=7072)
  wrote conf/zoo-quorum-3.cfg  (clientPort=2183, secureClientPort=2283, adminPort=8083, metricsPort=7073)
=== 6. Generate standalone config ===
  wrote conf/zoo-single-eku.cfg  (standalone, clientPort=2181, secureClientPort=2182)

============================================
  Setup complete!  Files in: zookeeper/dev/single-eku-test
============================================

Stores created:
zookeeper/dev/single-eku-test/ca.jks
zookeeper/dev/single-eku-test/client-keystore.jks
zookeeper/dev/single-eku-test/server-keystore.jks
zookeeper/dev/single-eku-test/server-truststore.jks
zookeeper/dev/single-eku-test/truststore.jks

# Start Quorum (3-node with TLS on client + quorum ports): -> Successful - OK

$ bin/zkServer.sh start conf/zoo-quorum-1.cfg
ZooKeeper JMX enabled by default
Using config: conf/zoo-quorum-1.cfg
Starting zookeeper ... STARTED
$ bin/zkServer.sh start conf/zoo-quorum-2.cfg
ZooKeeper JMX enabled by default
Using config: conf/zoo-quorum-2.cfg
Starting zookeeper ... STARTED
$ bin/zkServer.sh start conf/zoo-quorum-3.cfg
ZooKeeper JMX enabled by default
Using config: conf/zoo-quorum-3.cfg
Starting zookeeper ... STARTED
$ for i in 1 2 3; do bin/zkServer.sh status conf/zoo-quorum-$i.cfg; done
ZooKeeper JMX enabled by default
Using config: conf/zoo-quorum-1.cfg
Client port found: 2181. Client address: localhost. Client SSL: false.
Mode: follower
ZooKeeper JMX enabled by default
Using config: conf/zoo-quorum-2.cfg
Client port found: 2182. Client address: localhost. Client SSL: false.
Mode: leader
ZooKeeper JMX enabled by default
Using config: conf/zoo-quorum-3.cfg
Client port found: 2183. Client address: localhost. Client SSL: false.
Mode: follower
$

# Checked ZK log - no Exception - OK

# Connect (plaintext): -> Successful - OK

$ bin/zkCli.sh -server localhost:2181
Connecting to localhost:2181
2026-09-11 09:16:07,772 [myid:] - INFO  [main:o.a.z.Environment@98] - Client environment:zookeeper.version=3.10.0-SNAPSHOT-ea14b32a48dce4fe9a0af16b4ec7aca179a5a5a4-dirty, built on 2026-09-09 08:52 UTC
2026-09-11 09:16:07,773 [myid:] - INFO  [main:o.a.z.Environment@98] - Client environment:host.name=host
2026-09-11 09:16:07,774 [myid:] - INFO  [main:o.a.z.Environment@98] - Client environment:java.version=17.0.19
2026-09-11 09:16:07,774 [myid:] - INFO  [main:o.a.z.Environment@98] - Client environment:java.vendor=Eclipse Adoptium
2026-09-11 09:16:07,774 [myid:] - INFO  [main:o.a.z.Environment@98] - Client environment:java.home=/home/user/.sdkman/candidates/java/17.0.19-tem
...

# Connect (TLS / mTLS): -> Successful - OK

$ CLIENT_JVMFLAGS="\
      -Dzookeeper.ssl.keyStore.location=zookeeper/dev/single-eku-test/client-keystore.jks \
      -Dzookeeper.ssl.keyStore.password=changeit \
      -Dzookeeper.ssl.trustStore.location=zookeeper/dev/single-eku-test/truststore.jks \
      -Dzookeeper.ssl.trustStore.password=changeit \
      -Dzookeeper.client.secure=true\
      -Dzookeeper.ssl.client.enable=true \
      -Dzookeeper.clientCnxnSocket=org.apache.zookeeper.ClientCnxnSocketNetty \
    " bin/zkCli.sh -server localhost:2281

Connecting to localhost:2281
2026-09-11 09:18:25,043 [myid:] - INFO  [main:o.a.z.Environment@98] - Client environment:zookeeper.version=3.10.0-SNAPSHOT-ea14b32a48dce4fe9a0af16b4ec7aca179a5a5a4-dirty, built on 2026-09-09 08:52 UTC
2026-09-11 09:18:25,045 [myid:] - INFO  [main:o.a.z.Environment@98] - Client environment:host.name=host
2026-09-11 09:18:25,045 [myid:] - INFO  [main:o.a.z.Environment@98] - Client environment:java.version=17.0.19
2026-09-11 09:18:25,045 [myid:] - INFO  [main:o.a.z.Environment@98] - Client environment:java.vendor=Eclipse Adoptium
2026-09-11 09:18:25,045 [myid:] - INFO  [main:o.a.z.Environment@98] - Client environment:java.home=/home/user/.sdkman/candidates/java/17.0.19-tem

...

# Negative test (serverAuth cert as client — should FAIL): FAILS - as expected - OK

$ CLIENT_JVMFLAGS="\
      -Dzookeeper.ssl.keyStore.location=zookeeper/dev/single-eku-test/server-keystore.jks \
      -Dzookeeper.ssl.keyStore.password=changeit \
      -Dzookeeper.ssl.trustStore.location=zookeeper/dev/single-eku-test/truststore.jks \
      -Dzookeeper.ssl.trustStore.password=changeit \
      -Dzookeeper.client.secure=true\
      -Dzookeeper.ssl.client.enable=true \
      -Dzookeeper.clientCnxnSocket=org.apache.zookeeper.ClientCnxnSocketNetty \
    " bin/zkCli.sh -server localhost:2281
Connecting to localhost:2281
2026-09-11 09:19:23,057 [myid:] - INFO  [main:o.a.z.Environment@98] - Client environment:zookeeper.version=3.10.0-SNAPSHOT-ea14b32a48dce4fe9a0af16b4ec7aca179a5a5a4-dirty, built on 2026-09-09 08:52 UTC
2026-09-11 09:19:23,060 [myid:] - INFO  [main:o.a.z.Environment@98] - Client environment:host.name=host

....

2026-09-11 09:19:23,133 [myid:localhost:2281] - INFO  [main-SendThread(localhost:2281):o.a.z.ClientCnxn$SendThread@1154] - Opening socket connection to server localhost/127.0.0.1:2281.
2026-09-11 09:19:23,133 [myid:localhost:2281] - INFO  [main-SendThread(localhost:2281):o.a.z.ClientCnxn$SendThread@1156] - SASL config status: Will not attempt to authenticate using SASL (unknown error)
[zk: localhost:2281(CONNECTING) 0] 2026-09-11 09:19:23,317 [myid:] - INFO  [zkNetty-EpollEventLoopGroup-1-1:o.a.z.ClientCnxnSocketNetty$ZKClientPipelineFactory@454] - SSL handler added for channel: [id: 0xafba523a]
2026-09-11 09:19:23,325 [myid:] - INFO  [zkNetty-EpollEventLoopGroup-1-1:o.a.z.ClientCnxn$SendThread@1007] - Socket connection established, initiating session, client: /127.0.0.1:48914, server: localhost/127.0.0.1:2281
2026-09-11 09:19:23,326 [myid:] - INFO  [zkNetty-EpollEventLoopGroup-1-1:o.a.z.ClientCnxnSocketNetty$1@182] - channel is connected: [id: 0xafba523a, L:/127.0.0.1:48914 - R:localhost/127.0.0.1:2281]
2026-09-11 09:19:23,417 [myid:] - ERROR [zkNetty-EpollEventLoopGroup-1-1:o.a.z.ClientCnxnSocketNetty$ZKClientHandler@522] - Unexpected throwable
io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: (certificate_unknown) Received fatal alert: certificate_unknown
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:515)
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:296)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1357)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:868)
	at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:805)
	at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:501)
	at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:399)
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:1007)
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.base/java.lang.Thread.run(Thread.java:840)
Caused by: javax.net.ssl.SSLHandshakeException: (certificate_unknown) Received fatal alert: certificate_unknown
	at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:131)
	at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:117)
	at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:370)
	at java.base/sun.security.ssl.Alert$AlertConsumer.consume(Alert.java:293)
	at java.base/sun.security.ssl.TransportContext.dispatch(TransportContext.java:209)
	at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:172)
	at java.base/sun.security.ssl.SSLEngineImpl.decode(SSLEngineImpl.java:737)
	at java.base/sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:692)
	at java.base/sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:507)
	at java.base/sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:483)
	at java.base/javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:679)
	at io.netty.handler.ssl.SslHandler$SslEngineType$3.unwrap(SslHandler.java:308)
	at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1486)
	at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1377)
	at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1428)
	at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:545)
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:484)
	... 15 common frames omitted
2026-09-11 09:19:23,418 [myid:] - INFO  [zkNetty-EpollEventLoopGroup-1-1:o.a.z.ClientCnxnSocketNetty@249] - channel is told closing
2026-09-11 09:19:23,418 [myid:] - INFO  [zkNetty-EpollEventLoopGroup-1-1:o.a.z.ClientCnxnSocketNetty$ZKClientHandler@469] - channel is disconnected: [id: 0xafba523a, L:/127.0.0.1:48914 ! R:localhost/127.0.0.1:2281]
2026-09-11 09:19:23,418 [myid:localhost:2281] - WARN  [main-SendThread(localhost:2281):o.a.z.ClientCnxn$SendThread@1290] - Session 0x0 for server localhost/127.0.0.1:2281, Closing socket connection. Attempting reconnect except it is a SessionExpiredException or SessionTimeoutException.
EndOfStreamException: channel for sessionid 0x0 is lost
	at org.apache.zookeeper.ClientCnxnSocketNetty.doTransport(ClientCnxnSocketNetty.java:287)
	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1279)

# Negative test (no client cert — should FAIL): FAILS - as expected - OK


$ CLIENT_JVMFLAGS="\
      -Dzookeeper.ssl.trustStore.location=zookeeper/dev/single-eku-test/truststore.jks \
      -Dzookeeper.ssl.trustStore.password=changeit \
      -Dzookeeper.client.secure=true\
      -Dzookeeper.client.secure=true \
      -Dzookeeper.ssl.client.enable=true \
      -Dzookeeper.clientCnxnSocket=org.apache.zookeeper.ClientCnxnSocketNetty \
    " bin/zkCli.sh -server localhost:2281

Connecting to localhost:2281
2026-09-11 09:20:59,931 [myid:] - INFO  [main:o.a.z.Environment@98] - Client environment:zookeeper.version=3.10.0-SNAPSHOT-ea14b32a48dce4fe9a0af16b4ec7aca179a5a5a4-dirty, built on 2026-09-09 08:52 UTC
2026-09-11 09:20:59,933 [myid:] - INFO  [main:o.a.z.Environment@98] - Client environment:host.name=host

...

2026-09-11 09:20:59,997 [myid:localhost:2281] - INFO  [main-SendThread(localhost:2281):o.a.z.ClientCnxn$SendThread@1154] - Opening socket connection to server localhost/127.0.0.1:2281.
2026-09-11 09:20:59,997 [myid:localhost:2281] - INFO  [main-SendThread(localhost:2281):o.a.z.ClientCnxn$SendThread@1156] - SASL config status: Will not attempt to authenticate using SASL (unknown error)
2026-09-11 09:21:00,029 [myid:] - WARN  [zkNetty-EpollEventLoopGroup-1-1:o.a.z.c.X509Util@587] - zookeeper.ssl.keyStore.location not specified for X509KeyManager
[zk: localhost:2281(CONNECTING) 0] 2026-09-11 09:21:00,157 [myid:] - INFO  [zkNetty-EpollEventLoopGroup-1-1:o.a.z.ClientCnxnSocketNetty$ZKClientPipelineFactory@454] - SSL handler added for channel: [id: 0xee02e253]
2026-09-11 09:21:00,170 [myid:] - INFO  [zkNetty-EpollEventLoopGroup-1-1:o.a.z.ClientCnxn$SendThread@1007] - Socket connection established, initiating session, client: /127.0.0.1:34514, server: localhost/127.0.0.1:2281
2026-09-11 09:21:00,171 [myid:] - INFO  [zkNetty-EpollEventLoopGroup-1-1:o.a.z.ClientCnxnSocketNetty$1@182] - channel is connected: [id: 0xee02e253, L:/127.0.0.1:34514 - R:localhost/127.0.0.1:2281]
2026-09-11 09:21:00,256 [myid:] - ERROR [zkNetty-EpollEventLoopGroup-1-1:o.a.z.ClientCnxnSocketNetty$ZKClientHandler@522] - Unexpected throwable
io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: (certificate_required) Received fatal alert: certificate_required
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:515)
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:296)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1357)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:868)
	at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:805)
	at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:501)
	at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:399)
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:1007)
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.base/java.lang.Thread.run(Thread.java:840)
Caused by: javax.net.ssl.SSLHandshakeException: (certificate_required) Received fatal alert: certificate_required
	at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:131)
	at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:117)
	at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:370)
	at java.base/sun.security.ssl.Alert$AlertConsumer.consume(Alert.java:293)
	at java.base/sun.security.ssl.TransportContext.dispatch(TransportContext.java:209)
	at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:172)
	at java.base/sun.security.ssl.SSLEngineImpl.decode(SSLEngineImpl.java:737)
	at java.base/sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:692)
	at java.base/sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:507)
	at java.base/sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:483)
	at java.base/javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:679)
	at io.netty.handler.ssl.SslHandler$SslEngineType$3.unwrap(SslHandler.java:308)
	at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1486)
	at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1377)
	at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1428)
	at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:545)
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:484)
	... 15 common frames omitted
2026-09-11 09:21:00,259 [myid:] - INFO  [zkNetty-EpollEventLoopGroup-1-1:o.a.z.ClientCnxnSocketNetty@249] - channel is told closing
2026-09-11 09:21:00,259 [myid:localhost:2281] - WARN  [main-SendThread(localhost:2281):o.a.z.ClientCnxn$SendThread@1290] - Session 0x0 for server localhost/127.0.0.1:2281, Closing socket connection. Attempting reconnect except it is a SessionExpiredException or SessionTimeoutException.
EndOfStreamException: channel for sessionid 0x0 is lost
	at org.apache.zookeeper.ClientCnxnSocketNetty.doTransport(ClientCnxnSocketNetty.java:287)
	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1279)
2026-09-11 09:21:00,260 [myid:] - INFO  [zkNetty-EpollEventLoopGroup-1-1:o.a.z.ClientCnxnSocketNetty$ZKClientHandler@469] - channel is disconnected: [id: 0xee02e253, L:/127.0.0.1:34514 ! R:localhost/127.0.0.1:2281]

# AdminServer (HTTPS): -> Successful - OK

$ curl -k https://localhost:8081/commands/stat
{
  "version" : "3.10.0-SNAPSHOT-ea14b32a48dce4fe9a0af16b4ec7aca179a5a5a4-dirty, built on 2026-09-09 08:52 UTC",
  "read_only" : false,
  "server_stats" : {
    "packets_sent" : 17,
    "packets_received" : 17,
    "fsync_threshold_exceed_count" : 0,
    "client_response_stats" : {
      "last_buffer_size" : 16,
      "min_buffer_size" : 16,
      "max_buffer_size" : 112
    },
    "provider_null" : false,
    "uptime" : 907333,
    "server_state" : "follower",
    "data_dir_size" : 67109492,
    "log_dir_size" : 67109492,
    "last_processed_zxid" : 4294967301,
    "outstanding_requests" : 0,
    "avg_latency" : 5.625,
    "max_latency" : 46,
    "min_latency" : 0,
    "num_alive_client_connections" : 0,
    "auth_failed_count" : 5,
    "non_mtlsremote_conn_count" : 0,
    "non_mtlslocal_conn_count" : 2
  },
  "client_response" : {
    "last_buffer_size" : 16,
    "min_buffer_size" : 16,
    "max_buffer_size" : 112
  },
  "node_count" : 6,
  "connections" : [ ],
  "secure_connections" : [ ],
  "command" : "stats",
  "error" : null
}%   

# AdminServer with mTLS client cert: -> Successful - OK

$ curl --cert zookeeper/dev/single-eku-test/client.p12:changeit --cert-type P12 \
      --cacert zookeeper/dev/single-eku-test/ca.pem https://localhost:8081/commands/stat
{
  "version" : "3.10.0-SNAPSHOT-ea14b32a48dce4fe9a0af16b4ec7aca179a5a5a4-dirty, built on 2026-09-09 08:52 UTC",
  "read_only" : false,
  "server_stats" : {
    "packets_sent" : 17,
    "packets_received" : 17,
    "fsync_threshold_exceed_count" : 0,
    "client_response_stats" : {
      "last_buffer_size" : 16,
      "min_buffer_size" : 16,
      "max_buffer_size" : 112
    },
    "provider_null" : false,
    "uptime" : 952587,
    "server_state" : "follower",
    "data_dir_size" : 67109492,
    "log_dir_size" : 67109492,
    "last_processed_zxid" : 4294967301,
    "outstanding_requests" : 0,
    "avg_latency" : 5.625,
    "max_latency" : 46,
    "min_latency" : 0,
    "num_alive_client_connections" : 0,
    "auth_failed_count" : 5,
    "non_mtlsremote_conn_count" : 0,
    "non_mtlslocal_conn_count" : 2
  },
  "client_response" : {
    "last_buffer_size" : 16,
    "min_buffer_size" : 16,
    "max_buffer_size" : 112
  },
  "node_count" : 6,
  "connections" : [ ],
  "secure_connections" : [ ],
  "command" : "stats",
  "error" : null
}%                      

# Prometheus metrics (HTTPS + mTLS client cert): -> Successful - OK

$ curl --cert zookeeper/dev/single-eku-test/client.p12:changeit --cert-type P12 \
      --cacert zookeeper/dev/single-eku-test/ca.pem https://localhost:7071/metrics

# HELP add_dead_watcher_stall_time_total add_dead_watcher_stall_time counter
# TYPE add_dead_watcher_stall_time_total counter
add_dead_watcher_stall_time_total 0.0
# HELP approximate_data_size approximate_data_size
# TYPE approximate_data_size gauge
approximate_data_size 190.0
# HELP auth_failed_count auth_failed_count
# TYPE auth_failed_count gauge
auth_failed_count 5.0
# HELP avg_latency avg_latency
# TYPE avg_latency gauge
avg_latency 5.625

...

Test summary:

Start Quorum (3-node with TLS on client + quorum ports): -> Successful - OK
Checked ZK log - no Exception - OK
Connect with zkCli (plaintext): -> Successful - OK
Connect (TLS / mTLS): -> Successful - OK
Negative test (serverAuth cert as client — should FAIL): FAILS - as expected - OK
Negative test (no client cert — should FAIL): FAILS - as expected - OK
AdminServer (HTTPS): -> Successful - OK
AdminServer with mTLS client cert: -> Successful - OK
Prometheus metrics (HTTPS + mTLS client cert): -> Successful - OK

→ OK

anmolnar · 2026-09-11T16:12:29Z

Merged to master branch. Thanks for the contribution @PDavid !
Please create separate pull request for branch-3.9 if you'd like to backport it.

PDavid force-pushed the ZOOKEEPER-5070-single-eku branch from bad8dd7 to d056eeb Compare August 18, 2026 13:54

PDavid marked this pull request as ready for review August 19, 2026 08:28

anmolnar requested changes Aug 31, 2026

View reviewed changes

Comment thread zookeeper-server/src/main/java/org/apache/zookeeper/common/X509Util.java

anmolnar requested changes Aug 31, 2026

View reviewed changes

PDavid force-pushed the ZOOKEEPER-5070-single-eku branch from 008b056 to ea14b32 Compare September 3, 2026 18:46

anmolnar approved these changes Sep 3, 2026

View reviewed changes

PDavid and others added 7 commits September 10, 2026 16:07

ZOOKEEPER-5070: Support Single EKU certificates

b964592

Added client keystore, server truststore config

ZOOKEEPER-5070. Fix X509AuthenticationProvider for client mTLS and ad…

5227850

…d unit test (#2)

ZOOKEEPER-5070: Fix typo in log message in buildServerTrustManager

8f1f771

ZOOKEEPER-5070: Fixed loading key/truststores twice when no separate …

a8a7a4c

…client/server stores In the default deployment where ssl.client.keyStore.location and ssl.server.trustStore.location are not set, createSSLContextAndOptionsFromConfig loaded key/truststores twice

ZOOKEEPER-5070: Documented single EKU support and new config properties

8d39444

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

PDavid force-pushed the ZOOKEEPER-5070-single-eku branch from f096b2c to 8d39444 Compare September 10, 2026 14:07

anmolnar merged commit ebe5fd7 into apache:master Sep 11, 2026
21 of 22 checks passed

nikolettotvos62-web mentioned this pull request Sep 11, 2026

ZOOKEEPER-4871: ZooKeeper python module (zkpython) is incompatible with Python 3.12 #2199

Merged

Navigation Menu

Uh oh!

ZOOKEEPER-5070: Support Single EKU certificates - #2429

ZOOKEEPER-5070: Support Single EKU certificates#2429
anmolnar merged 7 commits into
apache:masterfrom
PDavid:ZOOKEEPER-5070-single-eku

PDavid commented Jul 30, 2026 •

edited

Loading

Uh oh!

anmolnar left a comment

Uh oh!

Uh oh!

anmolnar left a comment •

edited

Loading

Uh oh!

PDavid commented Sep 1, 2026

Uh oh!

PDavid commented Sep 1, 2026

Uh oh!

anmolnar commented Sep 2, 2026 •

edited

Loading

Uh oh!

anmolnar commented Sep 2, 2026

Uh oh!

anmolnar left a comment

Uh oh!

PDavid commented Sep 10, 2026 •

edited

Loading

Uh oh!

PDavid commented Sep 11, 2026

Uh oh!

Uh oh!

anmolnar commented Sep 11, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

PDavid commented Jul 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Goal

Approach

Configuration changes

Client keystore

Server truststore

Quorum client keystore

Quorum server truststore

TODO

Uh oh!

anmolnar left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

anmolnar left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

PDavid commented Sep 1, 2026

Uh oh!

PDavid commented Sep 1, 2026

Uh oh!

anmolnar commented Sep 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

anmolnar commented Sep 2, 2026

Uh oh!

anmolnar left a comment

Choose a reason for hiding this comment

Uh oh!

PDavid commented Sep 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

PDavid commented Sep 11, 2026

Testing

Uh oh!

Uh oh!

anmolnar commented Sep 11, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

PDavid commented Jul 30, 2026 •

edited

Loading

anmolnar left a comment •

edited

Loading

anmolnar commented Sep 2, 2026 •

edited

Loading

PDavid commented Sep 10, 2026 •

edited

Loading