AI & ML

Microsoft’s Hyenas Uncovers Long-Standing Rust Memory Safety Flaws

Microsoft researchers found three Rust standard library flaws, including an 11-year-old signal-stack bug and a stable Rust use-after-free risk.

Nov 10, 2026 ● 3 min read
Sign in to save

Rust’s safety guarantees depend partly on code the compiler can’t verify. Microsoft researchers used an AI-assisted scanner to find several flaws in that code, including a signal-stack bug that had persisted for nearly 11 years and a separate stable Rust issue that could expose applications to freed memory.

The findings came from Microsoft’s FORGE Lab, which used Hyenas, its research vulnerability-scanning system, to examine unsafe code in Rust’s standard library. Hyenas is a research version of codename MDASH, Microsoft’s agentic vulnerability-discovery system. Researchers checked the affected code, reproduced the bugs, assessed their security impact, and reported them to Rust maintainers.

Three ways unsafe code can go wrong

The team grouped its findings into three patterns: code that assumes an operation won’t panic, code that relies on user-provided implementations behaving as expected, and code that makes incorrect assumptions about the operating system or C library beneath Rust.

The panic-safety flaw affected MapWindows::clone. The function allocated a buffer before fully initializing it, then called a user-provided clone operation to fill the buffer. If that operation panicked, Rust’s unwinding process could run the buffer’s destructor while some contents were still uninitialized, leading to undefined behavior. Microsoft said the reported fix prevents destruction until initialization is complete.

A second issue involved str::replace and a custom implementation of the nightly Pattern trait. The function assumed the implementation would provide an ASCII byte, but a custom implementation could return a non-ASCII value. That could produce invalid UTF-8, leaving later string operations vulnerable to undefined behavior.

These two patterns have a history in Rust’s standard library. Microsoft said researchers were already studying similar weaknesses with Rudra, a static analyzer, in 2021. The library had also addressed panic-safety problems as early as 2015, yet related implementation mistakes continue to surface.

Operating system interactions reach stable Rust

The most persistent bug involved process::exit. If a child thread called the function, the runtime could remove the main thread’s alternate signal stack while that thread was still running. A later stack overflow on the main thread could then prevent the operating system from delivering the signal correctly, causing a segmentation fault instead of Rust’s intended stack-overflow handling.

The bug had been in the standard library for nearly 11 years. Microsoft said it found no reliable way to exploit it under the conditions examined and considered practical exploitation unlikely.

Researchers also found a flaw in CommandExt::exec, which interacted with the process-wide environment variable array. The function could modify this shared data while holding a read lock. Concurrent activity could expose inconsistent state and, in some circumstances, leave an environment pointer referring to freed memory.

Both the process::exit and CommandExt::exec findings affect stable Rust. When Microsoft published its findings, a proposed fix for the environment-variable issue was under review. It would avoid changing the shared array by calling execve directly. Two issues reported by the team remained open at that time.

What the findings mean for Rust users

Rust’s ownership model and type system are designed to prevent common memory-safety errors, but the standard library uses unsafe code internally for operations the compiler can’t check. Types such as Vec and String expose that work through safe interfaces, which must remain sound no matter how safe application code uses them.

Microsoft cautioned that each flaw requires unusual conditions, and a standard library bug becomes a serious vulnerability only when a real program reaches the affected API in the necessary way. That limits the practical risk, but it also illustrates why trusted library code deserves ongoing scrutiny: safe interfaces can still depend on subtle assumptions about panics, custom implementations, and system behavior.

Microsoft said the research can inform Microsoft Perception, its multi-agent security system, and MDASH, which brings agentic vulnerability discovery into Microsoft Defender. One example is the Web App Pen Test Agent, which can use MDASH findings to prioritize tests, assess exploitability against live applications, and help guide remediation. For Rust developers, the immediate takeaway is to track the reported fixes and keep auditing the unsafe foundations behind safe APIs.

Source: Joseph Davis · www.csoonline.com

Comments

Sign in to join the discussion.