El Salvador’s Chivo Wallet: a slapstick saga of software disaster

Salvadoran President Nayib Bukele wants to keep all the details of the introduction of bitcoin secret from the people of El Salvador. Journalists are told that basic government operations are commercially confidential. Critics are forced out of the country.

But Bukele keeps outsourcing his bad ideas to US companies — who file stuff with the SEC, or get taken to court, and then it all comes out.

Shaun Overton is the owner of ROI Developers, a.k.a. Accruvia, who were contracted to fix issues with Chivo, the official bitcoin wallet for El Salvador. Overton and his team worked on the Chivo system from September to November 2021:

We began working on the project with the government of El Salvador because software was provided to the government and they experienced a number of incredibly difficult circumstances where the program was functionally unusable. And so we were brought in to right the ship and fix all the problems.

Overton says that Athena failed to pay him $83,698.91 for his work, and he’s suing Athena. I mentioned this case in February 2022.

Athena filed a countersuit against Overton claiming shoddy work, which was dismissed in August 2022 for being in the wrong venue. [CaseText]

The transcript of Overton’s August 2022 deposition in the case was just written up by Guatemalan news site No-Ficción. They put a lot of effort into tracking down the details of what happened here, and this article wouldn’t exist without theirs. [No-Ficción, in Spanish]

I have a copy of the Overton transcript, and of the deposition with Eric Gravengaard, CEO of the Athena entities, the next day. The transcripts are messy scans, but they contain their fair share of comedy gold, which I live to share with you, courtesy a valued helper who has PACER access. [Document, PDF, in English; PACER case docket, paywalled] Update: docket now on Court Listener, and slowly getting filled out. [case docket]

The Gravengaard deposition — pages 58 to 95 of the document — is about venue issues and who did what, when, for what company, as Athena tries to argue that Overton is suing the wrong corporate entity.

Most of the Overton deposition —  pages 19 to 57 of the document — is about these venue issues. But there’s also an inside view of a couple of major disasters afflicting the Chivo wallet.

(I signed up to PACER myself trying to buy this document. [Thanks to my patrons for expenses!] You can’t even do a search on PACER until they send you a search access code … via physical US mail … within seven to ten days. Yes, all of this is on computers. At least I’ll have access in a week or two.)

The KYC that never was

When Bitcoin and the Chivo wallet were launched in El Salvador, everyone who created a Chivo wallet got a $30 signup bonus — which would be like getting $300 in the US.

The signup process required Know Your Customer (KYC) information: your DUI (national identity card) number, your date of birth, a photo of your DUI and a photo of yourself.

But the system didn’t do anything with the photos — it would accept any input. So you could sign up and claim your $30 with any DUI and birth date that matched — and many DUIs were public. Thousands of fraudulent signups followed. Many people tried signing up only to find that their DUI had already been used by someone else.

Overton tells how the KYC system crashed at 3am on launch day, after the first 150 users. But the authorities couldn’t let the system stay broken — so they let it come back up with no KYC system, so they could hit their target of 50,000 signups:

So when the program launched on September 7th, there was a vendor selected that I don’t recall the name of. And I know that QA Consultants, another vendor, had warned the government that this software — that this vendor was extremely likely to fail under load.

And within the first 150 sign-ups on the platform, the, the vendor for KYC compliance crashed. And government of El Salvador had, especially President Bukele, had staked pretty much his entire career on the successful rollout.

So the government of El Salvador made to eliminate the KYC, the KYC vendor from the process so that the Chivo Wallet registrations could continue. They wanted to hit their 50,000-user initial launch point. But because they turned off KYC, there was literally no supervision whatsoever. Anybody on the platform could sign up and get $30.

So we experienced — I know that Miguel Sabal, who was the direct advisor to President Bukele and who I worked with nearly every day, was coming in my office telling me about users, when they would take the selfie for their personal identification, he was saying people were taking pictures of the wall, of potted plants.

(The KYC vendor who Overton couldn’t remember was VU Security. [home page])

Overton’s orders came from two members of Bukele’s Venezuelan “shadow cabinet” — Sara Hanna and Miguel Sabal.

A few days after the crash, on 9 September, a contact of Overton’s from QA Consultants asked him to come to El Salvador urgently to work on “the deal of a lifetime.” Overton signed the contract on 10 September, and got into El Salvador for “the most stressful project of my life” — fixing the undocumented KYC system code:

And the very first day was dedicated to establishing our work environment. The project was very difficult to spin up because there was no documentation … Literally zero documentation.

… Lorenzo Rey, who was functionally the government’s CTO, informed me that Athena Bitcoin, Inc. had developed the software and sold it — licensed a copy to the government of El Salvador.

… So the government hired QA Consultants with the intent to use them for load testing. However, because of stalling from Athena, they were never actually able to run those load tests.

Sabal asked Overton to fix the KYC system and reconnect it to the Chivo system. Overton started by writing unit tests for the system, because there weren’t any of those either.

(For the curious: the KYC system was apparently written in Django, a Python-based web development framework. Yes, you can write a back-end system in Django.)

Overton’s team did eventually fix the KYC system. But Overton estimates that 10-20% of Chivo signups were fraudulent — though he doesn’t state how he got these percentages in this deposition — losing $12–$24 million:

And so when I mentioned that 10 to 20 percent number, by the time I left the project we were at 4 million registered users. So you have at least 10 percent of 4 million. So 400,000 people that each got 30 bucks, you’re talking about $12 million.

And for context, like for the American government, that would be chump change. But the government of the El Salvador, at the time they were negotiating with the IMF for a bailout for a billion dollars. And so when you’re losing anywhere from 12 to 24 million dollars, just throwing it away on bogus registrations, they did not have that kind of money to just throw away to fraud.

So when they would come to us and beg us to run these scripts to mark these people as fraudulent as fast as possible, it’s because they were bleeding money that they didn’t have.

Overton estimates the Chivo userbase at 4 million — out of 6.5 million people of all ages in El Salvador. The observed real world usage of Chivo makes it implausible that that many actual humans signed up, but I can quite believe there were that many registrations.

Thousands of cases of identity fraud were filed with the Fiscalía (the Attorney General) between September and November 2021. So far these remain unanswered.

No-Ficción tried to contact Miguel Sabal for his side of the story. The President’s Press Secretary, Ernesto Sanabria, told No-Ficción that Sabal didn’t work there, and that he didn’t even know who Sabal was.

 

El Diario de Hoy front page, Saturday 6 November 2021

 

Athena tries to buy out ROI

Lorenzo Rey was extremely unhappy with Athena:

But then Lorenzo pulled me aside and offered me the opportunity to — initially he offered me the opportunity to remove Athena Bitcoin, Inc. from the project entirely.

… He was irate and — with the quality of the code that was provided. He said — I don’t remember the exact term that he used. But he basically felt like they were sold a bill of goods, that — they were just severely disappointed.

Athena was unhappy that Overton and ROI had been offered a direct deal. Eric Gravengaard of Athena offered to buy ROI and make Overton the CTO of Athena, and they signed a term sheet — pages 3 to 5 of the document. This is what led to the arguments over billing and which Athena entity was liable to pay Overton.

Athena then tried to poach Overton’s employees, offering a 25% pay rise, a $50,000 sign-on bonus and equity:

Because every single developer came to me and said: Hey, Athena’s trying to poach us.

Q. And what did they tell you?

A. They said they weren’t interested because the work environment was so toxic that they didn’t want to have anything to do with Athena.

 

 

Become a bitcoin billionaire from day-trading!

If bitcoins are money, then volatility is bad. So the Chivo wallet would let users freeze the price for bitcoin for one minute when making transactions.

With the price in Chivo frozen for sixty seconds while the outside price on exchanges still changed, a day trader could exploit this arbitrage and make a few dollars.

The government complained about this “scalping,” and even described it as “fraude” — but had to admit it wasn’t actually illegal. Price freezing was disabled on 18 October 2021. [Twitter, archive; Elsalvador.com, 2021, in Spanish]

Overton was brought in a second time to fix the problem. One trader he saw started with $2,000 and traded it up to $400,000.

Sabal told Overton and Athena Bitcoin that scalping had to be blocked that day. Overton worked on the price change code. Eric Gravengaard of Athena gave the job of fixing the freeze problem to a novice programmer that Gravengaard (not Overton) had just hired, who had just started that day — and who was fired three weeks later.

You’d normally test such a change before deploying it to production. Athena didn’t do that. So on the night of 18 October 2021, the novice coder accidentally set the exchange rate to 1 dollar = 1 bitcoin — when BTC was actually around $60,000.

Overton tells the story:

And Miguel told me, like, this needs to be fixed right now, tonight. And I advised him that that’s a horrible idea because we don’t have time for QA. And he said skip it. And then Eric felt that pressure and overruled me.

So instead of, instead of having —— work on unit tests where he couldn’t break anything, Eric directed him to modify the control that controlled how the Bitcoin/dollar exchange rate worked.

And long story short is he screwed it up; because the code was a giant mess, full of the copy-and-paste no unit tests. And so —— mistakenly allowed for the exchange of 1 dollar to 1 Bitcoin. And the impact of 1 dollar at the time was actually worth 60,000 Bitcoin.

(I’ve removed the guy’s name, because he doesn’t deserve to be made searchable for getting stuck on a project like this.)

Chivo wallets showed bitcoin balances in the millions or billions of dollars. Programmers at the time assumed it was an arithmetic overflow in the system, because bitcoin balances showed as negative. Salvadorans assumed it was just an amusing error, and shared screenshots around. It even made the papers. [Elsalvador.com, 2021, in Spanish; Elsalvador.com, 2021, in Spanish]

But they could have actually taken that money out. And some did:

And so we found — by the time that that mistake was caught — Miguel frantically texted Edward Iskra saying: What is going on? There’s way too many Bitcoin going through the system.

We found that line of code. It resulted in — we saw — by the time we released the code and we stopped it about an hour later, there were about 3600 transactions that we estimated were about $50 in value each. And so instead of trading $180,000, that ——’s code trade 180,000 Bitcoin with a market value in excess of $10 billion, which made the government insolvent immediately.

And we had to pull in the Accruvia team, Jeremy Brenner, Eric Gravengaard. We did an all-night marathon session writing code to unwind that whole event so that — and we had to take the server completely off-line — like we killed the entire system, just turned it off — didn’t get it back on until about 7:30 in the morning.

And that was ——’s very first day working on the project.

Q. And did that ultimately get resolved? You said an all-nighter. Did that resolve it?

A. It did. I think the government probably lost a quarter million dollars on it. The government had some safety mechanism that, through sheer luck, prevented the disaster. It didn’t allow Bitcoin withdrawals from 10:00 p.m. until around 6:00 or 7:00 in the morning.

And because we did nightly releases, the transaction — we did releases at 10:00 p.m. So when we caught it around 11:00, 11:30, nobody could actually take the on-chain Bitcoin out of the system; they could only transfer it within Chivo. The only money that was able to leak was using Bitcoin Lightning. And so I just ballparked it at around a quarter of a million dollars.

The GDP of the United States ($20 trillion) is approximately a thousand times El Salvador’s GDP ($26 billion). So this is as if the US government lost a quarter billion dollars to fraud due to a software error in an internationally publicised flagship payment project, by telling a junior developer to commit straight to production on his first day.

The disasters that killed a payment system

The deposition is Overton speaking under oath — but we should keep in mind that this is just one guy’s view of the situation.

However, anyone who’s worked in software development will understand the tale Overton tells. This is what consulting companies like ROI do for a living — they come in and clean up business software messes. We know what sort of mess he would have seen in there.

The disasters that Overton describes were huge news in El Salvador — especially the extensive KYC fraud. His answers match the known timings and explain the observed problems in ways that make sense — what happened and why it would have happened.

One unanswered question is how Athena got a contract they seem to have been utterly unable to deliver on. One hypothesis is that it was the Bitcoin Beach guys who got Athena in touch with the government, who were already interested in Bitcoin Beach. The Bitcoin Beach guys had brought the first Athena bitcoin ATMs into El Salvador.

Athena lost the Chivo contract in December 2021. They were replaced by AlphaPoint.

But it was too late for bitcoin in El Salvador — with these and other dysfunctionalities, the users’ trust had been shattered. Almost nobody wanted to risk using the Chivo system, even to transmit dollars. Bitcoin is still legal tender in El Salvador, but its use is negligible.

Everyone anticipated that Chivo would be used to steal and launder money — but I don’t think anyone expected that this fate would be averted by Chivo just not working.