DTLS listener: bound the per-connection receive queue

The DTLS listener queues datagrams from a known peer into a per-connection
list (DTLS_RX::urxe_pending) with no limit on count or bytes. A reachable
peer that completes the ordinary cookie exchange can grow one parked
connection's queue without bound as long as the application keeps
the shared demux pumped, moving datagrams into userspace past the bound
the kernel socket buffer would otherwise provide.

Cap the queue at OSSL_DTLS_RX_DEFAULT_MAX_QUEUED (128) datagrams
per connection - a datagram count, not a byte cap. Every queued URXE
holds a buffer of at least the listener's maximum datagram size whatever
the datagram's length, so a full queue costs about 295 KB per connection
at the default 2000-byte size, roughly one default kernel socket receive
buffer; across a full 256-connection listener budget that is about 75 MB.
The queue is consumed FIFO, so on overflow the oldest datagram is evicted
rather than the newest: a DTLS flight can span several datagrams,
and under overload keeping the newest biases the queue toward the flight
the peer just sent instead of stale fragments of an older one.
The evicted URXE is unhooked under rx->mutex and released to the demux
after dropping it, so demux->mutex is never taken while rx->mutex
is held.

References: https://github.com/openssl/openssl/issues/33173
Assisted-by: Claude Code:moonshotai/kimi-k3
Reviewed-by: Tomas Mraz <[email protected]>
Reviewed-by: Ryan Hooper <[email protected]>
Reviewed-by: Eugene Syromiatnikov <[email protected]>
Merge-date: Wed Oct 14 10:54:54 2026
Merged-from: https://github.com/openssl/openssl/pull/33197

1 files changed, 11 insertions, 0 deletions

diff --git a/include/internal/dtls_record_rx.h b/include/internal/dtls_record_rx.h
index 44099b2477f..2be3b11a637 100644
--- a/include/internal/dtls_record_rx.h
+++ b/include/internal/dtls_record_rx.h
@@ -16,6 +16,17 @@
 
 #ifndef OPENSSL_NO_DTLS
 
+/*
+ * Maximum datagrams queued per connection in DTLS_RX::urxe_pending: a
+ * count, not a byte cap. On overflow the oldest datagram is evicted
+ * (DTLS permits datagram loss). Each queued URXE holds a buffer of the
+ * listener's maximum datagram size, so 128 entries cost about 295 KB per
+ * connection at the default 2000-byte size - roughly one default kernel
+ * socket receive buffer, the bound that queuing into userspace would
+ * otherwise bypass.
+ */
+#define OSSL_DTLS_RX_DEFAULT_MAX_QUEUED 128
+
 typedef struct dtls_rx_st {
     DGRAM_DEMUX *demux;
     DGRAM_URXE_LIST urxe_pending;