Critical OpenSSL Vulnerability CVE-2023-5678 Exposes Servers to DoS Attacks

Key Takeaways

• A critical OpenSSL vulnerability, dubbed “HollowByte,” allows unauthenticated attackers to launch denial-of-service (DoS) attacks using minimal network traffic.
• The flaw exploits memory pre-allocation during the TLS handshake, leading to persistent memory fragmentation and server resource exhaustion.
• Affected systems include a vast array of internet infrastructure, such as web servers (NGINX, Apache), language runtimes (Node.js, Python, Ruby, PHP), and databases (MySQL, PostgreSQL).
• OpenSSL has silently patched the issue in versions 4.0.1 and backported releases, but no formal CVE was issued.

Unauthenticated DoS Vulnerability “HollowByte” Impacts OpenSSL

A recently uncovered vulnerability within OpenSSL, dubbed “HollowByte,” enables a remote attacker to induce a denial-of-service condition on vulnerable servers. This attack requires no authentication and can be triggered with a meticulously crafted payload as small as 11 bytes.

The Okta Red Team identified this critical flaw, which exploits how OpenSSL allocates memory during the initial TLS handshake phase. Before any authentication takes place, the vulnerability compels servers to reserve substantial blocks of memory, making them susceptible to resource exhaustion.

Understanding the “HollowByte” Vulnerability

The TLS handshake initiates with a ClientHello message, encapsulated within a record that includes a 4-byte header specifying the size of the subsequent message body. In older iterations of OpenSSL, the library would pre-allocate a receive buffer based solely on this length declared by the client, even before the actual data arrived.

When a specially crafted 11-byte malicious payload is sent, the TLS state machine processes the header, initiating an unvalidated memory allocation sequence:

• Header Read → grow_init_buf() → OPENSSL_clear_realloc() → malloc(attacker_size)

Because no validation occurs at this early stage, a single malicious packet can force the malloc() function to allocate up to 131 KB of memory, based purely on the attacker’s false claim. The server’s worker thread then enters an indefinite waiting state for data that never materializes, effectively blocking it.

Unlike traditional connection-exhaustion attacks, such as Slowloris, which primarily focus on tying up server threads, HollowByte introduces a severe memory fragmentation problem. This issue stems from how the glibc library manages freed memory blocks.

When an attacker’s connection is terminated, OpenSSL correctly deallocates the buffer. However, glibc does not immediately return small-to-medium sized memory allocations to the operating system; instead, it retains them for potential future reuse. By initiating successive waves of connections with randomized, fabricated sizes, attackers can prevent the memory allocator from effectively reusing these freed chunks. This leads to a continuous and permanent increase in the server’s Resident Set Size (RSS), even after the malicious connections have dropped. The only way to reclaim this fragmented memory is to terminate the affected process.

Tests conducted by the Okta Red Team on unpatched OpenSSL instances running NGINX demonstrated significant impacts:

• On a server with 1 GB of RAM, the system experienced an Out-Of-Memory (OOM) kill after accumulating 547 MB of frozen, fragmented memory.
• In a 16 GB RAM environment, the attack managed to consume 25% of the total system memory. Crucially, this occurred while staying below typical connection ceilings, meaning standard connection-limiting defenses would fail to mitigate the attack.

Given OpenSSL’s foundational role in a vast array of internet infrastructure, the potential impact of this vulnerability is extensive. It affects popular web servers like Apache and NGINX, various language runtimes including Node.js, Python, Ruby, and PHP, and databases such as MySQL and PostgreSQL.

OpenSSL addressed this issue by implementing incremental buffer growth, incorporating changes through pull requests #30792, #30793, and #30794. Instead of implicitly trusting the header’s declared length, OpenSSL now expands the buffer only as actual bytes are received over the wire. This ensures that an empty or oversized claim from an attacker incurs no significant memory cost to the server.

The fix was silently integrated into OpenSSL v4.0.1, with quiet backports also applied to versions 3.6.3, 3.5.7, 3.4.6, and 3.0.21. Notably, OpenSSL classified this as a hardening improvement rather than issuing a formal CVE advisory. This approach aligns with a recent trend where OpenSSL addresses DoS-class issues without assigning high-profile CVE designations.

Because HollowByte received a silent patch rather than a CVE, many organizations might not identify it through routine vulnerability scanning. Considering OpenSSL’s widespread deployment across web servers, runtimes, and databases, security teams should prioritize patching this vulnerability irrespective of its CVE status.

What You Should Do

• Immediately upgrade your OpenSSL installations to version 4.0.1 or the relevant backported versions (3.6.3, 3.5.7, 3.4.6, or 3.0.21).
• Perform an audit of embedded OpenSSL versions within language runtimes (e.g., Node.js, Python, Ruby, PHP), as OS-level patches alone may not update bundled copies.
• Actively monitor Resident Set Size (RSS) memory trends on your TLS-terminating servers for any signs of gradual, unexplained memory bloat, which could indicate a fragmentation attack.

Disclaimer: HackersRadar reports on cybersecurity threats and incidents for informational and awareness purposes only. We do not engage in hacking activities, data exfiltration, or the hosting or distribution of stolen or leaked information. All content is based on publicly available sources.