The Solana network encountered a significant outage on Feb. 6 at 10:22 am UTC, resulting in the suspension of block production for over five hours until validators were able to restart the network. This is not the first time Solana has experienced such issues, as there have been approximately six major outages and 15 partial or major outage days since January 2022.
These incidents varied in severity, ranging from partial to complete network outages, and were caused by factors such as software vulnerabilities, denial-of-service attacks, and resource exhaustion attacks.
The most recent outage was attributed to a bug that caused an infinite loop error, which led to validators getting stuck on a particular block. An infinite loop error occurs when a program becomes trapped in an endless cycle, preventing validators in the case of Solana from verifying transactions on the network as they were unable to process the same block.
In a postmortem report released on Feb. 9, Anza, a Solana-focused software development firm, detailed the cause of the recent outage and explained how validators were able to restart the network. The report revealed that the bug occurred in Solana’s Just-in-Time (JIT) compilation cache, which compiles all programs before executing a transaction.
Previously, the JIT cache utilized “ExecutorsCache,” but with the v1.16 release, it was replaced by a new implementation called “LoadedPrograms.” The new loader program used accounting information stored in a program’s on-chain account to determine its most recent deployment slot and calculate the effective slot height.
However, this loader program encountered a legacy loader issue, causing the JIT cache to enter a loop with older programs and continuously recompile them instead of efficiently accessing and executing their instructions. This glitch resulted in a network stall, as the system became trapped in an infinite loop and was unable to process transactions or move forward.
Austin Federa, head of strategy at the Solana Foundation, stated that there was a process to replace the old loader system with a new motor system, but it was scheduled to be disabled with an updated version.
Solana engineers promptly identified and resolved the problem, updated the system’s processes, and identified the older programs to prevent future recompilation cycles. This solution involved reducing the deployment of problematic legacy loaders to ensure accurate and efficient access to all program instructions without falling into the recompilation trap.
Some critics have attributed Solana’s frequent outages to the lack of client diversity, which creates potential central points of failure. However, Federa argues that Solana has a more accurate architecture compared to other networks and explains that only Ethereum and Solana have independent validator clients. Currently, Solana has one major validator client built on one functional codebase.
The Solana blockchain, which was launched in March 2020, has made significant progress in the decentralized finance (DeFi) ecosystem over the past four years. It ranks as the fifth-largest DeFi chain in terms of total value locked, with over $1.7 billion allocated to various on-chain protocols. The native Solana (SOL) token has a market valuation of approximately $49 billion, making it the fifth-largest cryptocurrency.
Despite its impressive track record, Solana has never officially exited its beta phase and is still referred to as being in mainnet beta. However, the blockchain is open to everyone, with numerous decentralized applications being utilized to launch various products. The ecosystem also boasts a vibrant nonfungible token (NFT) ecosystem.
Although the code is still evolving, Solana has surpassed what can be considered beta software. The misconception arises from confusion between pre-launch testing and post-launch software maturity, as well as misunderstandings surrounding Solana’s messaging.
The beta status has attracted criticism, especially in light of the network’s outages in recent years. However, Federa argues that most blockchain systems are still in beta, and only Bitcoin and Ethereum can be considered complete mainnets.
While Solana’s downtime has been seen as a hurdle to its goal of becoming an Ethereum competitor, developers have been working to address the bugs that have caused past outages to ensure they don’t affect the network in the future. Despite its troubled history, Solana’s speed and cost advantages continue to attract new projects to the platform.