FFC-DH Peer Validation Uses Attacker-Supplied q

Issue summary: When EVP_PKEY_derive_set_peer() is called with a DHX (X9.42)

peer key, the peer key is not properly checked for the subgroup membership.

Impact summary: A malicious peer which presents an X9.42 key carrying the

victim's p and g parameters, a forged q = r (a small prime factor of the

cofactor (p−1)/q_local), and a public value Y of order r can recover the

victim's private key after a small number of key exchange attempts.

When EVP_PKEY_derive_set_peer() is called with a DHX (X9.42) peer key, the

subgroup membership check Y^q ≡ 1 (mod p) is performed using the peer's

own q parameter, not the local key's q. The peer's domain parameters are

then matched against the domain parameters of the private key, but the value

of q is not compared.

A malicious peer who presents an X9.42 key carrying the victim's p, g,

a forged q = r (a small prime factor of the cofactor), and a public

value Y of order r passes all checks. The shared secret then takes only

r distinct values, leaking priv mod r. Repeating for each small-prime

factor of the cofactor and combining via CRT recovers the full private

key (Lim–Lee / small-subgroup-confinement attack).

The realistic attack surface is narrow: principally CMP deployments with

long-lived RA/CA DHX keys and bespoke enterprise or government applications

using X9.42 DHX static keys with interactive protocols and therefore this

issue was assigned Low severity.

The FIPS modules in 4.0, 3.6, 3.5, 3.4, 3.1.2 and 3.0 are affected by this

issue.