Background
Incentivised testnets have been one of the significant innovations in the crypto space. Cosmos led the way with Game of Stakes, and since then, it’s become a core activity for bootstrapping new networks.
Incentivized testnets are powerful in many ways. They enable validators to build up the skills they need to deploy, upgrade, and maintain the new network. But more than that, testnets help the validators learn how to operate as a community, where they work together to solve problems and make the network more robust. Often the incentives include rewards for maintaining high uptime or for passing a security audit. They may even include activities besides running validators e.g., rewards for content production. Incentivized testnets also give token holders a chance to evaluate the skills of each validator, to help them decide with whom to stake once the mainnet is launched.
A key part of running testnets is to break things. The premise is that the more successful attacks there are on a testnet, the more prepared a network is for mainnet. The best sign that a network has been put through its paces is to find the Holy Grail of bugs: a priority one, severe bug that is a showstopper for the network launch.
This week Chorus One engineer, Reisen, did just that. He demonstrated a critical flaw with the Solana network that allowed him to steal 500M Sol tokens on the testnet. What follows is the story of how he did it.
A Step Back
To start the story, we must go back to when Reisen joined the Chorus One team in June of 2019. Over the summer, Chorus was contracted by Solana to build out StrongGate, a solution that enables high-availability validators on the Solana network. This involved Reisen getting deep into some of the core Solana code. The Solana codebase would challenge any developer, even for a very experienced Rust and functional programming expert like Reisen. That is because the Solana project has innovated on so many fronts in parallel. Solana is built around eight vital technological innovations in one project (as described in this blog post). All these innovations were delivered in Version 1 of the Solana codebase, so when Reisen dove into the codebase, there was still a lot of development activity going on. And none of the eight technologies are easy to grasp. Proof of History is a new model for distributed timekeeping, Tower BFT is a new consensus algorithm, Turbine is a block propagation protocol influenced by BitTorrent, Archivers are a decentralized file system, and that’s only half of them. Anyone of these on its own would challenge any developer. So it was no easy task to dive into the codebase.
When Solana’s incentivized testnet (Tour de SOL) came around, Reisen was determined to come up with the best exploit. It was always in the back of his mind as we looked through the codebase in late 2019. But it wasn’t until Tour de SOL kicked off in February that he focused his attention on the problem.
The First Attacks
Designing distributed computing protocols in Byzantine environments is especially hard. The typical attacks we see on these networks involve crafting malformed packets or launching denial of service attacks. So this is where Reisen started.
His first attack was a joint effort with Certus One. Reisen had identified an issue at network launch time. Each node announces the height of the last block they have seen. The node with the most recent block is then responsible for sharing the latest block(s) to bootstrap the nodes that are joining the network. Reisen realized that this could be used to sabotage the network. By advertising a very high block height, he could control the launch. But the question was how best to exploit this? This is where the amazing Certus One team come in. They built a superb mechanism to slow down the snapshot delivery so that nodes would be effectively stalled trying to get access to the latest chain data before the network could start. They also prepared a compression (or zip) bomb that we deployed but unfortunately was never activated. And they designed the pièce de résistance, a beautiful piece of ASCII art to add to the mischief:
The attack worked! We had successfully attacked the network. Independently, Certus also launched other denial of service attacks on the network. It was such a pleasure for us the partner with Certus on this, as they have long been the rock stars of testnet attacks.
The fun had begun!
The Search For The Big Attack
So far, so good. But Reisen wanted more. He sensed there was a more significant attack possible. And then, last week, he spotted an issue in the code. At this point, it was just a suspicion that an attack might be possible. But he wasn’t sure.
So over a couple of sleepless nights, he set about setting up a test environment to build and test the exploit. Over the weekend, the Chorus team got early indications that the attack was viable. But could Reisen make it happen in the Tour de SOL testnet? We had to be patient as Reisen waited for the right opportunity.
This was the big one. It was the most critical exploit you can imagine in a crypto network: stealing unlimited funds. Reisen had found a bug that allowed a block producer to inject a transaction that could steal funds from any account without knowing the private key. And the network was utterly oblivious to anything being wrong.
But could we launch such a severe attack on Tour De SOL? Reisen felt that we should run it by one of the Solana team first. So we showed the exploit to Michael Vines, Head of Engineering at Solana. And it didn’t work at first. Reisen thought he’d messed up. Maybe there was something wrong with his test environment. He couldn’t reproduce it with Michael. But then, after seven attempts — it worked! We had reproduced it on the Solana Soft Launch testnet.
Now the big question. Could we make it work on Tour De SOL? Again the answer at first was no. It failed. Then it failed again. But at least Michael had seen the exploit working, so we still marked it down as a big success.
But as you may have guessed by now, this wasn’t good enough for Reisen. He needed to see this through. So he left the exploit on a loop. And an hour after we demonstrated the issue to Michael… The transaction went through. WE DID IT! We stole 500 Million SOL tokens from the genesis account on the Tour de SOL testnet.
Deep Dive Into the Issue
Let’s start with some terminology. Solana has slots, periods where a particular validator adds transactions. It’s helpful to think of a slot as a block in a more traditional blockchain context. Leaders are elected for a slot. They get to decide which transactions to include in that slot, during which time they broadcast these transactions to the other validators. Validators run in one of two modes: Broadcast and Replay. The leader broadcasts, while the other validators are in replay mode.
As is typical in blockchains, each transaction contains a set of public keys for the relevant accounts involved in the transaction. In the case of a simple token transfer, this is the sender and receiver public keys. For the token transfer transaction to be valid, it must be signed by the sender’s private key.
Simplified Transaction Format: [Signature, Sender Key, Receiver Key, Payload].
Check: Does Signature belong to Sender Key.
If you sign a transaction with a key that isn’t the correct one, then validators will fail the signature verification step. But this is not what happens when the leader injects a transaction.
The leader broadcasts the transactions they have included in the slot. Each validator receives those transactions, in what’s called replay mode. When transactions are replayed to a validator from the leader, the signature of each transaction should be verified. Instead the validator does a “light” check on the fields of the transaction. It looks to confirm that the correct keys are in place. There is a field in the transaction to indicate if the signatures have been previously checked (presumably by the node that accepted the transaction). But — and this is the critical flaw — the receiving validator does not actually re-run this validation of the signature i.e. there is no explicit check by the validator to verify that the owner of the account from which funds are being withdrawn has signed the transaction. In effect, the receiving validator trusts the transaction coming from the leader. This was the critical issue Reisen identified.
The steps he took were as follows:
But how come we had trouble reproducing the attack on the two testnets? The issue was that sometimes the malicious transaction the leader submitted never made it into the chain. The Solana network is so fast that it’s hard for a leader to inject transactions fast enough. But by retrying in a loop, the transaction was finally accepted into the chain. Reisen had succeeded through sheer grit and determination and found a way to steal 500M SOL on the Tour De SOL testnet.
The Solana Team Response
The Solana team’s response has been great.
The Solana codebase is excellent. The Rust compiler ensures type safety, which rules out whole classes of bugs. And the code is written defensively so that all inputs are checked. We just happened to find the one case where the robust checks we see everywhere else in the code were missed.
And now the hard work for Solana starts. Of course, questions must be asked on how this issue was missed in the recent security audit. But Anatoly (the Solana CEO) has given clear instructions to the team to review all the code, especially the crypto signing pieces.
We think this is the shock that every network needs as it prepares for the mainnet launch. We do not doubt that the Solana team will rise to the challenge to ensure that an issue like this never occurs again.
But at Chorus One we’re thrilled with our work! Reisen got his attack. And he got the much-deserved kudos from Anatoly (which is always lovely!):
And he got a new nickname: Reisen Hood, Prince of Crypto Thieves.
It’s been an amazing week for us. By a strange coincidence, Monday also saw some other big news for Chorus One. We launched Anthem, our multi-network staking platform, which allows token holders to track and manage Proof of Stake portfolios and earnings. Users can create a personalized staking dashboard for any Cosmos address, with detailed data and charts to cover your ATOM staking portfolio. Support for other networks is underway and will be coming soon. So please check it out at https://anthem.chorus.one.
We’re excited to unveil Anthem: a web application to track and manage your portfolio of Proof-of-Stake assets. Anthem supports staking on Celo, Cosmos, Oasis, Kava, and Terra.
Go to https://anthem.chorus.one now, connect your Ledger or any address on the supported networks and see your personal staking dashboard! 📈
The core motivation to build Anthem arose from our own needs as a staking provider:
Currently, most accounting and tax regimes require you to recognize revenues at the time received. When staking on Cosmos, this means that revenues occur every single block, or around 400,000 times a month! So far, there has been no public tool allowing you to see your historical staking rewards. Anthem changes that.
Anthem provides stakers with a consistent, personalized dashboard that allows them to track, manage, and optimize how their staking assets are utilized across many different networks.
The rest of this post will walk you through the current functionality of Anthem:
When visiting Anthem for the first time, you will be presented with the landing page. Press get started or scroll to the login options. Here you have two options:
Sign in with Ledger: Use this option to automatically see information for your Ledger account. When connecting your Ledger device you can also transfer and stake assets, as well as (on Celo) participate in governance.
Sign in with Address: Use this option to track balance and staking data for addresses on any supported network.
This is the core piece of Anthem. Here you can see your portfolio development, past transactions, as well as your current balance.
Anthem visualizes historical data on your total portfolio including the ability to filter staked and available tokens, as well as the rewards that you accrued while staking. Anthem also allows you to export a CSV file that you can use for tax reporting or analytics.
There’s a variety of ways to customize settings in Anthem. You can choose different fiat currencies to use as a base currency, and you can change the language in which Anthem is displayed. We currently support Chinese, English, German, Korean, and Spanish. There’s even an evil dark mode! 🌙
We’re always looking for feedback, so please don’t hesitate to contact us through our website or hit us up on Telegram.
We are excited to announce that Chorus One is joining Chainlink as a node operator. Starting today, our Chainlink node will assist in providing highly available and reliable data feeds to the crypto ecosystem.
The Chainlink team is providing a core piece of infrastructure for blockchain applications by allowing them to handle off-chain data. On their own, blockchains are unable to verify “real world” data that is created outside their native network. This creates a conundrum, as many blockchain use cases need to stay informed about external events to trigger the execution of smart contracts. For example, financial contracts need market data, insurance contracts need IoT data, and gaming applications need provable randomness.
Not only do smart contracts need off-chain connection, but they need that connection to be as secure as the underlying blockchain. Hence, there needs to be a way for external data to be fed into blockchains in a secure and reliable manner. This is widely referred to as the oracle problem, and Chainlink is leading the effort in making integrating off-chain data, such as price feeds, simple, yet secure for decentralized applications.
Chainlink relies on node operators providing streams of data to the network from external sources. These Chainlink-powered decentralized oracle networks are live on the Ethereum mainnet with additional plans to become blockchain agnostic over time. At the moment node operators are vetted by the Chainlink team to maintain Sybil resistance in the early stages of the network. A staking incentive model is in the works that will allow LINK holders to stake their tokens with trustworthy node operators. This will help create a permissionless, decentralized network focused on reliably providing accurate data to be consumed by blockchain applications.
We have valuable experience in working with networks that rely on price oracles. The Terra stablecoin protocol allows anyone to exchange their stablecoins with the network’s native staking token Luna through an automated market maker. The Terra community has implemented an incentivized, stake-based oracle protocol in which validators provide price data on-chain in frequent time intervals. While this approach may be suitable for some large applications and networks; it appears that a general, customizable solution that works across blockchains and applications will be more scalable, reliable, and ultimately trustworthy, if designed with the proper incentives.
Following Chainlink’s progress over the last couple of months, it is our belief that the protocol has a good chance of becoming the standard for providing off-chain data to blockchain applications. Our thesis is further supported by the transition of several major decentralized finance projects on Ethereum to the Chainlink network instead of running their own oracle implementations. Examples include Aave and Synthetix, which made this decision after their early price reference feed was exploited by a trader.
We are excited to join the Chainlink community to help make blockchain adoption for real-world applications a reality. We will be supporting the network with our infrastructure and knowledge of staking systems and look forward to publishing further content around Chainlink and the planned staking implementation in the future.
Chorus One is building validation and staking infrastructure for Proof-of-Stake networks.
We will offer staking for Chainlink when it goes live. You will be able to support our work and earn staking rewards by delegating LINK to our node.
Website: https://chorus.one
Twitter: https://twitter.com/chorusone
Telegram: https://t.me/chorusone
Chainlink is a decentralized oracle network that enables smart contracts to securely access off-chain data feeds, web APIs, and traditional bank payments. Chainlink is consistently selected as one of the top blockchain technologies by leading independent research firms such as Gartner. It is well known for providing highly secure and reliable oracles to great companies like Google, Oracle, SWIFT, and many other large enterprises, as well as many of the world’s best smart contract projects/teams such as Web3/Polkadot, Synthetix, Loopring, Kaleido, OpenLaw, Reserve, and many more.
Learn more by visiting the Chainlink website, Twitter or Reddit. If you’re a developer, visit the developer documentation or join the technical discussion on Discord.
We’re in the midst of a cambrian explosion of decentralized protocols. From peer-to-peer payments to non-state forms of money to decentralized organizational structures, blockchains are redefining human collaboration in the 21st century. But those new networks can’t rely on the enforcement of rules in the old, slow, fuzzy and nation-bound court system. Instead, they are embracing economic incentives and particularly the usage of cryptoassets as collateral.
At the forefront of this is Proof-of-Stake, which replaces the expensive mining process by using cryptoassets as collateral for enforcing consensus rules. In the last year, networks like Cosmos and Tezos have pushed Proof-of-Stake into the crypto mainstream. Today, we have ahead of us a wave of blockchain networks all launching based on the same design insight. To operate this infrastructure, a new industry of node operators has emerged. We went from theoretical protocols and academic papers to live networks with billions of dollars at stake. And even large exchanges like Coinbase and Binance are entering the fray and have begun offering staking to their customers.
But while it is clear that the Proof-of-Stake paradigm is upon us, it’s still very unclear how it will play out. When Satoshi created Bitcoin, he didn’t seem to have anticipated ASICs nor mining pools. Similarly, in the infinitely richer design space of Proof-of-Stake, we don’t know what the end state will be and what preconditions will lead to resilient, decentralized networks capable of sustaining censorship-resistance at scale.
One clear demand that is emerging is the ability to re-use and transfer staked collateral. If a user can borrow a stablecoin like Maker’s DAI against their interest-earning staked ATOMs, clearly those gain in value and utility. Similarly, the ability to transfer and sell tokens without going through an unbonding period has substantial value.
The centralized exchanges that have entered staking will easily be able to provide that and much more in their walled gardens. But in Proof-of-Stake, control over private keys means control over the consensus process. So can these networks sustain healthy communities and robust decentralization if token holders are driven to surrender custody to unlock additional capabilities and returns?
In the past six months, many approaches have been explored to unlock innovation around staked assets directly on-chain. These efforts can be subsumed under the term “ liquid staking”. One example of these efforts was built by Chorus One together with Sikka, when we implemented Delegation Vouchers as part of the Cosmos-SDK hackathon in Berlin. Since then, various other teams have risen to the challenge and came up with their own designs. Some examples include Everett, Stafi Protocol, Stake Capital DAO, and Staker DAO.
Today, we are proud to announce that we received support from the Interchain Foundation to form a working group and to produce a research report focused on approaches to liquid staking. The goal is to take a holistic look at the design space, evaluate the different approaches with respect to clear criteria and understand their impact on network security and sustainability. We will focus on aspects such as:
We will map out different paths for the future of Proof-of-Stake and hope to create a shared vision for how to ensure open innovation, sustainability and decentralization for Proof-of-Stake networks.
We’re inviting protocol designers, validators, token holders, and anyone else interested in working on this topic to join the working group and contribute to this crucial research and discussion. The publication of the final report is planned to happen by the end of April.
A core feature of the working group will be our bi-weekly community calls. Our first community call, which will serve as an introduction to the project, will take place January 15th at 5pm CET. During this call, you will be able to find out more about our plans and learn how to contribute. Sign up here to for the calendar invite to this first working group call.
In the meantime, you can find the Liquid Staking Working Group on Telegram and Github ( https://github.com/ChorusOne/liquid-staking), where we already began to gather relevant material. We want to thank the Interchain Foundation for realizing the importance of this topic and hope that our research will help inform stakeholders and improve Proof-of-Stake protocol design.
Your Chorus One Team
About Chorus One
Chorus One is operating validation infrastructure and staking services for Proof-of-Stake networks.
Website: https://chorus.one
Twitter: https://twitter.com/chorusone
Telegram: https://t.me/chorusone
About Interchain Foundation
The ICF is a Swiss foundation funding and pushing the frontiers of blockchain-related infrastructure.
Website: https://interchain.io
Twitter: https://twitter.com/interchain_io
Originally published at https://blog.chorus.one on January 9, 2020.
