check RSA key size for PSS

[kojo1]

Description

Don't select RSA-PSS in MatchSigAlgo when the RSA key is too small for the signature.
When a TLS 1.2 server holds a small RSA certificate (e.g. 512bit) and the ClientHello's signature_algorithms extension offers rsa_pss_rsae_sha256 (0x0804), the handshake is aborted even though the client also offers rsa_pkcs1_sha256 (0x0401), which the key can use.

Fixes zd#22091

Testing

Local test with 512bit cert and run time SetMinRsaKey_Sz.

Checklist

• added tests
• updated/added doxygen
• updated appropriate READMEs
• Updated manual and documentation

[github-actions]

MemBrowse Memory Report

gcc-arm-cortex-m0plus

• FLASH: .text +28 B (+0.0%, 64,183 B / 262,144 B, total: 24% used)

gcc-arm-cortex-m3

• FLASH: .text +24 B (+0.0%, 122,505 B / 262,144 B, total: 47% used)

gcc-arm-cortex-m4

• FLASH: .text +64 B (+0.0%, 200,384 B / 262,144 B, total: 76% used)

gcc-arm-cortex-m4-dtls13

• FLASH: .text +64 B (+0.0%, 180,952 B / 1,048,576 B, total: 17% used)

gcc-arm-cortex-m4-openssl-compat

• FLASH: .text +64 B (+0.0%, 771,212 B / 1,048,576 B, total: 74% used)

gcc-arm-cortex-m4-pkcs7

• FLASH: .text +192 B (+0.1%, 212,780 B / 262,144 B, total: 81% used)

gcc-arm-cortex-m4-pq

• FLASH: .text +64 B (+0.0%, 279,460 B / 1,048,576 B, total: 27% used)

gcc-arm-cortex-m4-rsa-only

• FLASH: .text +64 B (+0.0%, 325,120 B / 1,048,576 B, total: 31% used)

gcc-arm-cortex-m4-tls12

• FLASH: .text +64 B (+0.1%, 123,277 B / 262,144 B, total: 47% used)

gcc-arm-cortex-m4-tls13

• FLASH: .text +64 B (+0.0%, 236,146 B / 262,144 B, total: 90% used)

gcc-arm-cortex-m7-pq

• FLASH: .text +64 B (+0.0%, 280,036 B / 1,048,576 B, total: 27% used)

gcc-arm-cortex-m7-tls13

• FLASH: .text +64 B (+0.0%, 236,210 B / 262,144 B, total: 90% used)

linuxkm-pie

• Data: __patchable_function_entries +8 B (+0.0%, 25,560 B)

linuxkm-standard

• Data: __patchable_function_entries +8 B (+0.0%, 48,248 B)
  No memory changes detected for:
• gcc-arm-cortex-m4-baremetal
• gcc-arm-cortex-m4-crypto-only
• gcc-arm-cortex-m4-min-ecc
• gcc-arm-cortex-m4-sp-math
• gcc-arm-cortex-m7
• stm32-sim-stm32h753

[Frauschi]

🐺 Skoll Code Review

Overall recommendation: APPROVE
Findings: 4 total — 2 posted, 2 skipped

Posted findings

• [Medium] Key-size check only covers RSA-PSS/SHA-256, not SHA-384/SHA-512 — src/internal.c:31129-31136
• [Low] Magic number 66 hardcoded instead of derived from WC_SHA256_DIGEST_SIZE — src/internal.c:31065-31066

Skipped findings

• [Medium] No test added for the small-RSA-key PSS rejection path
• [Low] keySz is used as a proxy for emLen (loose by up to 1 byte)

Review generated by Skoll via Claude/Codex

[Frauschi]

Jenkins retest this please.

[Frauschi]

LGTM