Feature: Pull RSA large E support from zcrypto/zgrab2

[hdm]

Feature: Pull RSA large E support from zcrypto/zgrab2

Closes the long-standing gap where excrypto rejected RSA public exponents that
exceed 1 << 31 - 1 and therefore could not parse or inspect a meaningful
slice of pathological-but-real keys observed in the wild by zmap/zgrab2 and
runZero. Mirrors the approach pioneered in
zcrypto and the SSH-specific fixes from
zgrab2#714, then extends them.

Summary of changes

1. rsa.PublicKey.E is now *big.Int

The hard 31-bit ceiling on E is gone. crypto/rsa.PublicKey.E is a
*big.Int, and every internal site that previously typed it as int has been
updated:

• crypto/internal/fips140/rsa (NewPrivateKey, NewPrivateKeyWithPrecomputation, NewPrivateKeyWithoutCRT, Export, checkPublicKey, checkPrivateKey, encrypt, decrypt, cast.go, keygen.go)
• crypto/internal/fips140test/acvp_test.go
• crypto/rsa (rsa.go, fips.go, boring_test.go, pss_test.go, rsa_test.go)
• crypto/json (rsa.go, rsa_test.go)
• crypto/x509 (pkcs1.go, parser.go, x509_test.go)
• crypto/ssl3/tls (key_agreement.go, handshake_server_test.go)
• x/crypto/openpgp/packet (public_key.go, public_key_v3.go, encrypted_key_test.go)
• x/crypto/ssh (keys.go)
• x/crypto/ssh/agent (server.go, client.go)

ASN.1 PKCS#1 structs (pkcs1PrivateKey, pkcs1PublicKey) also use
*big.Int so the wire encoding round-trips arbitrarily large exponents.
crypto/json switches the JSON exponent representation to json.Number to
encode/decode values that don't fit in a Go int.

2. Modular exponentiation no longer truncates E

crypto/internal/fips140/rsa.encrypt/decrypt previously called
bigmod.Nat.ExpShortVarTime(x, uint(e), m), which silently caps e at the
size of a uint. They now use bigmod.Nat.Exp(x, e.Bytes(), m), which accepts
arbitrarily large exponents.

A tiny-key fix in checkPrivateKey: when e > p-1 (which happens for
tiny test keys, never for real keys), e is pre-reduced via
new(big.Int).Mod(e, p-1) before being loaded into a bigmod.Nat sized for
pMinus1. The same is done for qMinus1. This fixes TestTinyKeyGeneration,
TestEverything/32, and TestEverything/33.

3. Removed all residual size filters on E

Audited every parse path. Only correctness checks remain — no size ceilings:

• crypto/x509.ParsePKCS1PublicKey: only rejects pub.E.Sign() <= 0. The old
  pub.E > 1<<31-1 rejection is gone. TestMarshalRSAPublicKey case zcrypto merge #5
  updated to expect success.
• crypto/x509/parser.go (parsePublicKey): only rejects Sign() <= 0.
• x/crypto/ssh.parseRSA: dropped BitLen() > 24. Now rejects only
  E < 3 || E.Bit(0) == 0 (required by RFC 4253 for m^e mod n to be
  invertible — E must be an odd integer ≥ 3).
• x/crypto/ssh/agent.parseRSAKey, parseRSACert: dropped BitLen() > 30;
  aligned with parseRSA's odd-E ≥ 3 requirement.
• crypto/internal/fips140/rsa.checkPublicKey: a large E only flips
  fipsApproved = false. It no longer rejects the key.

4. Defensive copies of E at every trust boundary

A reviewer-noted safety concern: storing the caller's *big.Int E directly
inside long-lived state means the caller can mutate it later and corrupt
internal invariants. Every constructor that consumes a caller-supplied E
now defensively copies it via new(big.Int).Set(e):

• crypto/internal/fips140/rsa.NewPrivateKey, NewPrivateKeyWithPrecomputation, NewPrivateKeyWithoutCRT
• crypto/internal/fips140/rsa.PrivateKey.Export (returns a copy so the caller
  cannot mutate stored state through the returned value)
• crypto/rsa.fipsPublicKey
• x/crypto/ssh.parseRSA (copies both E and N from the wire-parsed struct)
• x/crypto/ssh/agent.parseRSAKey

A new test TestPublicKeyDefensiveCopy (crypto/rsa/rsa_test.go) verifies
that mutating priv.PublicKey.E after an operation does not break subsequent
operations on the same key.

5. Configurable DoS bound: rsa.MaxPublicExponentBitLen

Accepting unboundedly large E from untrusted sources is a DoS vector:
modular exponentiation is O(bitlen(E) · bitlen(N)²). Indicative cost on
Apple M3 Max with a 2048-bit modulus:

bitlen(E)	per-op
17 (e=65537, the common case)	~56 µs
256 (FIPS 186-5 ceiling)	~340 µs
1024 (this PR's default)	~1.25 ms
2048 (E as wide as N)	~2.5 ms

A new package variable, rsa.MaxPublicExponentBitLen (default 1024),
gates the public-key operations (EncryptPKCS1v15, EncryptOAEP,
VerifyPKCS1v15, VerifyPSS, etc.) in the single choke point
checkPublicKeySize. Parsing is never gated, so any key — no matter how
pathological — can still be inspected. Users can tune the bound:

rsa.MaxPublicExponentBitLen = 0    // accept any E
rsa.MaxPublicExponentBitLen = 256  // FIPS 186-5 ceiling
rsa.MaxPublicExponentBitLen = 31   // stdlib's historical 32-bit limit
rsa.MaxPublicExponentBitLen = 17   // effectively restrict to e ≤ 65537

Tests: TestMaxPublicExponentBitLen and BenchmarkVerifyByExponentBitLen.

6. SSH parity with zgrab2#714

Beyond the parseRSA BitLen > 24 removal already noted, the SSH agent
side mirrors the PR:

• x/crypto/ssh/agent/server.go: parseRSAKey and parseRSACert no longer
  cap E at 30 bits.
• x/crypto/ssh/agent/client.go: passes k.E directly (now *big.Int)
  instead of big.NewInt(int64(k.E)).
• x/crypto/ssh/keys.go: Marshal, parseOpenSSHPrivateKey, and
  MarshalPrivateKey thread E through as *big.Int.

Differences from upstream zcrypto / zgrab2

This branch is a strict superset of the upstream patches plus several
additions that surfaced from PR review:

• DoS protection. zcrypto switches E to *big.Int and otherwise leaves
  callers to defend themselves. This PR adds a configurable, documented
  MaxPublicExponentBitLen gate with a permissive default (1024) and concrete
  per-op cost numbers in the doc comment so callers can tune sensibly.
• Defensive copies. zcrypto installs the caller's *big.Int directly
  into internal state. This PR copies at every boundary, and also copies on
  the way out via Export. New TestPublicKeyDefensiveCopy covers this.
• bigmod.Nat.Exp instead of ExpShortVarTime. Required to actually
  evaluate operations with bitlen(E) > 64; without this change, large E
  parses but silently misbehaves on encrypt/verify.
• Tiny-key support. The checkPrivateKey path now reduces E modulo
  p-1 and q-1 before loading into bigmod.Nat, so test keys with
  bitlen(N) < bitlen(E) validate correctly.
• crypto/json uses json.Number for the exponent so values that exceed
  2^53 survive JSON round-trips intact.
• x/crypto/ssh/agent parity. zgrab2#714 only touches x/crypto/ssh;
  this PR extends the same treatment to x/crypto/ssh/agent's server and
  client, with the same odd, E ≥ 3 validation.

Validation

• go build ./... — clean
• go vet ./... — clean
• go test ./... — all packages pass except a pre-existing macOS
  ssh-agent Unix-socket-path-length environment failure (verified to
  reproduce on main independently of this branch).

See also: zgrab2#714 (the SSH
piece) and zcrypto's crypto/rsa.PublicKey.E *big.Int design.

[Copilot]

Pull request overview

This PR changes RSA public exponents from fixed-width int values to *big.Int across RSA, X.509, SSH, OpenPGP, TLS, JSON, and FIPS-related code paths to support unusually large RSA exponents.

Changes:

• Converts RSA exponent parsing, marshaling, comparisons, and tests to use *big.Int.
• Relaxes several previous exponent-size rejection paths.
• Updates FIPS/internal RSA plumbing and ACVP test adapters for big integer exponents.

Reviewed changes

Copilot reviewed 22 out of 22 changed files in this pull request and generated 13 comments.

Show a summary per file

File	Description
crypto/rsa/rsa.go	Changes rsa.PublicKey.E to *big.Int and updates key generation/equality/precompute paths.
crypto/rsa/fips.go	Updates FIPS-only exponent checks for *big.Int.
crypto/rsa/rsa_test.go	Updates RSA tests for pointer exponent values.
crypto/rsa/pss_test.go	Updates PSS vector exponent parsing.
crypto/rsa/boring_test.go	Updates BoringCrypto RSA test keys.
crypto/internal/fips140/rsa/rsa.go	Converts internal FIPS RSA exponent handling to *big.Int.
crypto/internal/fips140/rsa/keygen.go	Uses big.NewInt for generated FIPS RSA exponent.
crypto/internal/fips140/rsa/cast.go	Updates test FIPS RSA key exponent initialization.
crypto/internal/fips140test/acvp_test.go	Adapts ACVP RSA exponent inputs/outputs.
crypto/x509/pkcs1.go	Updates PKCS#1 RSA public/private exponent ASN.1 structures and validation.
crypto/x509/parser.go	Parses X.509 RSA public exponents into big.Int.
crypto/x509/x509_test.go	Updates X.509 RSA tests and large-exponent expectations.
crypto/ssl3/tls/key_agreement.go	Serializes/parses SSL3/TLS export RSA exponents as big.Int.
crypto/ssl3/tls/handshake_server_test.go	Updates TLS test RSA key exponent.
crypto/json/rsa.go	Encodes/decodes RSA JSON exponent as json.Number/big.Int.
crypto/json/rsa_test.go	Updates JSON RSA test key exponent.
x/crypto/ssh/keys.go	Updates SSH RSA public/private key parsing and marshaling.
x/crypto/ssh/agent/server.go	Updates SSH agent RSA key/cert parsing.
x/crypto/ssh/agent/client.go	Sends RSA agent exponent as big.Int.
x/crypto/openpgp/packet/public_key.go	Updates OpenPGP RSA exponent construction/parsing.
x/crypto/openpgp/packet/public_key_v3.go	Updates V3 OpenPGP RSA exponent construction/parsing.
x/crypto/openpgp/packet/encrypted_key_test.go	Updates OpenPGP test RSA key exponent.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.