{"id":24242,"date":"2022-11-30T23:50:16","date_gmt":"2022-11-30T23:50:16","guid":{"rendered":"https:\/\/davidgerard.co.uk\/blockchain\/?p=24242"},"modified":"2022-12-01T13:14:30","modified_gmt":"2022-12-01T13:14:30","slug":"el-salvadors-chivo-wallet-disasters-a-slapstick-saga-of-software-disaster","status":"publish","type":"post","link":"https:\/\/davidgerard.co.uk\/blockchain\/2022\/11\/30\/el-salvadors-chivo-wallet-disasters-a-slapstick-saga-of-software-disaster\/","title":{"rendered":"El Salvador\u2019s Chivo Wallet: a slapstick saga of software disaster"},"content":{"rendered":"

Salvadoran President Nayib Bukele wants to keep all the details of the introduction of bitcoin secret from the people of El Salvador. Journalists are told that basic government operations are commercially confidential. Critics are forced out of the country.<\/a><\/p>\n

But Bukele keeps outsourcing his bad ideas to US companies \u2014 who file stuff with the SEC, or get taken to court, and then it all comes out.<\/p>\n

Shaun Overton is the owner of ROI Developers, a.k.a.<\/i> Accruvia, who were contracted to fix issues with Chivo, the official bitcoin wallet for El Salvador. Overton and his team worked on the Chivo system from September to November 2021:<\/p>\n

We began working on the project with the government of El Salvador because software was provided to the government and they experienced a number of incredibly difficult circumstances where the program was functionally unusable. And so we were brought in to right the ship and fix all the problems.<\/p><\/blockquote>\n

Overton says that Athena failed to pay him $83,698.91 for his work, and he\u2019s suing Athena. I mentioned this case<\/a> in February 2022.<\/p>\n

Athena filed a countersuit against Overton claiming shoddy work, which was dismissed in August 2022 for being in the wrong venue. [CaseText<\/em><\/a>]<\/p>\n

The transcript of Overton\u2019s August 2022 deposition in the case was just written up by Guatemalan news site No-Ficci\u00f3n. They put a lot of effort into tracking down the details of what happened here, and this article wouldn’t exist without theirs. [No-Ficci\u00f3n<\/i><\/a>, in Spanish<\/i>]<\/p>\n

I have a copy of the Overton transcript, and of the deposition with Eric Gravengaard, CEO of the Athena entities, the next day. The transcripts are messy scans, but they contain their fair share of comedy gold, which I live to share with you, courtesy a valued helper who has PACER access. [Document<\/i><\/a>, PDF, in English; <\/i>PACER case docket<\/i><\/a>, paywalled<\/i>] Update:<\/strong> docket now on Court Listener, and slowly getting filled out. [case docket<\/em><\/a>]<\/p>\n

The Gravengaard deposition \u2014 pages 58 to 95 of the document \u2014 is about venue issues and who did what, when, for what company, as Athena tries to argue that Overton is suing the wrong corporate entity.<\/p>\n

Most of the Overton deposition \u2014\u00a0 pages 19 to 57 of the document \u2014 is about these venue issues. But there\u2019s also an inside view of a couple of major disasters afflicting the Chivo wallet.<\/p>\n

(I signed up to PACER myself trying to buy this document. [Thanks to my patrons<\/a> for expenses!] You can\u2019t even do a search<\/i> on PACER until they send you a search access code … via physical US mail … within seven to ten days. Yes, all of this is on computers. At least I\u2019ll have access in a week or two.)<\/p>\n

The KYC that never was<\/h3>\n

When Bitcoin and the Chivo wallet were launched in El Salvador, everyone who created a Chivo wallet got a $30 signup bonus \u2014 which would be like getting $300 in the US.<\/p>\n

The signup process required Know Your Customer (KYC) information: your DUI (national identity card) number, your date of birth, a photo of your DUI and a photo of yourself.<\/p>\n

But the system didn\u2019t do anything with the photos \u2014 it would accept any input.<\/a> So you could sign up and claim your $30 with any<\/i> DUI and birth date that matched \u2014 and many DUIs were public. Thousands of fraudulent signups followed. Many people tried signing up only to find that their DUI had already been used by someone else.<\/p>\n

Overton tells how the KYC system crashed at 3am on launch day,<\/a> after the first 150 users. But the authorities couldn\u2019t let the system stay broken \u2014 so they let it come back up with no KYC system,<\/i> so they could hit their target of 50,000 signups:<\/p>\n

So when the program launched on September 7th, there was a vendor selected that I don’t recall the name of. And I know that QA Consultants, another vendor, had warned the government that this software \u2014 that this vendor was extremely likely to fail under load.<\/p>\n

And within the first 150 sign-ups on the platform, the, the vendor for KYC compliance crashed. And government of El Salvador had, especially President Bukele, had staked pretty much his entire career on the successful rollout.<\/p>\n

So the government of El Salvador made to eliminate the KYC, the KYC vendor from the process so that the Chivo Wallet registrations could continue. They wanted to hit their 50,000-user initial launch point. But because they turned off KYC, there was literally no supervision whatsoever. Anybody on the platform could sign up and get $30.<\/p>\n

So we experienced \u2014 I know that Miguel Sabal, who was the direct advisor to President Bukele and who I worked with nearly every day, was coming in my office telling me about users, when they would take the selfie for their personal identification, he was saying people were taking pictures of the wall, of potted plants.<\/p><\/blockquote>\n

(The KYC vendor who Overton couldn\u2019t remember was VU Security. [home page<\/i><\/a>])<\/p>\n

Overton\u2019s orders came from two members of Bukele\u2019s Venezuelan \u201cshadow cabinet\u201d<\/a> \u2014 Sara Hanna and Miguel Sabal.<\/p>\n

A few days after the crash, on 9 September, a contact of Overton\u2019s from QA Consultants asked him to come to El Salvador urgently to work on \u201cthe deal of a lifetime.\u201d Overton signed the contract on 10 September, and got into El Salvador for \u201cthe most stressful project of my life\u201d \u2014 fixing the undocumented KYC system code:<\/p>\n

And the very first day was dedicated to establishing our work environment. The project was very difficult to spin up because there was no documentation … Literally zero documentation.<\/p>\n

… Lorenzo Rey, who was functionally the government’s CTO, informed me that Athena Bitcoin, Inc. had developed the software and sold it \u2014 licensed a copy to the government of El Salvador.<\/p>\n

… So the government hired QA Consultants with the intent to use them for load testing. However, because of stalling from Athena, they were never actually able to run those load tests.<\/p><\/blockquote>\n

Sabal asked Overton to fix the KYC system and reconnect it to the Chivo system. Overton started by writing unit tests<\/a> for the system, because there weren’t any of those either.<\/p>\n

(For the curious: the KYC system was apparently written in Django, a Python-based web development framework. Yes, you can write a back-end system in Django.)<\/p>\n

Overton\u2019s team did eventually fix the KYC system. But Overton estimates that 10-20% of Chivo signups were fraudulent \u2014 though he doesn\u2019t state how he got these percentages in this deposition \u2014 losing $12\u2013$24 million:<\/p>\n

And so when I mentioned that 10 to 20 percent number, by the time I left the project we were at 4 million registered users. So you have at least 10 percent of 4 million. So 400,000 people that each got 30 bucks, you’re talking about $12 million.<\/p>\n

And for context, like for the American government, that would be chump change. But the government of the El Salvador, at the time they were negotiating with the IMF for a bailout for a billion dollars. And so when you’re losing anywhere from 12 to 24 million dollars, just throwing it away on bogus registrations, they did not have that kind of money to just throw away to fraud.<\/p>\n

So when they would come to us and beg us to run these scripts to mark these people as fraudulent as fast as possible, it’s because they were bleeding money that they didn’t have.<\/p><\/blockquote>\n

Overton estimates the Chivo userbase at 4 million \u2014 out of 6.5 million people of all ages in El Salvador. The observed real world usage of Chivo makes it implausible that that many actual humans signed up,<\/a> but I can quite believe there were that many registrations.<\/p>\n

Thousands of cases of identity fraud<\/a> were filed with the Fiscal\u00eda (the Attorney General) between September and November 2021. So far these remain unanswered.<\/p>\n

No-Ficci\u00f3n tried to contact Miguel Sabal for his side of the story. The President’s Press Secretary, Ernesto Sanabria, told No-Ficci\u00f3n that Sabal didn\u2019t work there, and that he didn\u2019t even know who Sabal was.<\/p>\n

 <\/p>\n

\"\"<\/a>El Diario de Hoy front page, Saturday 6 November 2021<\/i><\/small><\/p>\n

 <\/p>\n

Athena tries to buy out ROI<\/h3>\n

Lorenzo Rey was extremely unhappy with Athena:<\/p>\n

But then Lorenzo pulled me aside and offered me the opportunity to \u2014 initially he offered me the opportunity to remove Athena Bitcoin, Inc. from the project entirely.<\/p>\n

… He was irate and \u2014 with the quality of the code that was provided. He said \u2014 I don’t remember the exact term that he used. But he basically felt like they were sold a bill of goods, that \u2014 they were just severely disappointed.<\/p><\/blockquote>\n

Athena was unhappy that Overton and ROI had been offered a direct deal. Eric Gravengaard of Athena offered to buy ROI and make Overton the CTO of Athena, and they signed a term sheet \u2014 pages 3 to 5 of the document. This is what led to the arguments over billing and which Athena entity was liable to pay Overton.<\/p>\n

Athena then tried to poach Overton\u2019s employees, offering a 25% pay rise, a $50,000 sign-on bonus and equity:<\/p>\n

Because every single developer came to me and said: Hey, Athena’s trying to poach us.<\/p>\n

Q. And what did they tell you?<\/p>\n

A. They said they weren’t interested because the work environment was so toxic that they didn’t want to have anything to do with Athena.<\/p><\/blockquote>\n

 <\/p>\n

\"\"<\/a><\/p>\n

 <\/p>\n

Become a bitcoin billionaire from day-trading!<\/h3>\n

If bitcoins are money, then volatility is bad. So the Chivo wallet would let users freeze the price for bitcoin for one minute when making transactions.<\/p>\n

With the price in Chivo frozen for sixty seconds while the outside price on exchanges still changed, a day trader could exploit this arbitrage and make a few dollars.<\/p>\n

The government complained about this \u201cscalping,\u201d and even described it as \u201cfraude\u201d \u2014 but had to admit it wasn\u2019t actually illegal. Price freezing was disabled on 18 October 2021. [Twitter<\/i><\/a>, archive<\/a>; <\/i>Elsalvador.com<\/i><\/a>, 2021, in Spanish<\/i>]<\/p>\n

Overton was brought in a second time to fix the problem. One trader he saw started with $2,000 and traded it up to $400,000.<\/p>\n

Sabal told Overton and Athena Bitcoin that scalping had to be blocked that day<\/i>. Overton worked on the price change code. Eric Gravengaard of Athena gave the job of fixing the freeze problem to a novice programmer that Gravengaard (not Overton) had just hired, who had just started that day \u2014 and who was fired three weeks later.<\/p>\n

You\u2019d normally test such a change before deploying it to production. Athena didn\u2019t do that. So on the night of 18 October 2021, the novice coder accidentally set the exchange rate to 1 dollar = 1 bitcoin \u2014 when BTC was actually around $60,000.<\/p>\n

Overton tells the story:<\/p>\n

And Miguel told me, like, this needs to be fixed right now, tonight. And I advised him that that’s a horrible idea because we don’t have time for QA. And he said skip it. And then Eric felt that pressure and overruled me.<\/p>\n

So instead of, instead of having \u2014\u2014 work on unit tests where he couldn’t break anything, Eric directed him to modify the control that controlled how the Bitcoin\/dollar exchange rate worked.<\/p>\n

And long story short is he screwed it up; because the code was a giant mess, full of the copy-and-paste no unit tests. And so \u2014\u2014 mistakenly allowed for the exchange of 1 dollar to 1 Bitcoin. And the impact of 1 dollar at the time was actually worth 60,000 Bitcoin.<\/p><\/blockquote>\n

(I\u2019ve removed the guy\u2019s name, because he doesn\u2019t deserve to be made searchable for getting stuck on a project like this.)<\/p>\n

Chivo wallets showed bitcoin balances in the millions or billions of dollars. Programmers at the time assumed it was an arithmetic overflow<\/a> in the system, because bitcoin balances showed as negative. Salvadorans assumed it was just an amusing error, and shared screenshots around. It even made the papers. [Elsalvador.com<\/i><\/a>, 2021, in Spanish;<\/i> Elsalvador.com<\/i><\/a>, 2021, in Spanish<\/i>]<\/p>\n

But they could have actually taken that money out. And some did:<\/p>\n

And so we found \u2014 by the time that that mistake was caught \u2014 Miguel frantically texted Edward Iskra saying: What is going on? There’s way too many Bitcoin going through the system.<\/p>\n

We found that line of code. It resulted in \u2014 we saw \u2014 by the time we released the code and we stopped it about an hour later, there were about 3600 transactions that we estimated were about $50 in value each. And so instead of trading $180,000, that \u2014\u2014’s code trade 180,000 Bitcoin with a market value in excess of $10 billion, which made the government insolvent immediately.<\/p>\n

And we had to pull in the Accruvia team, Jeremy Brenner, Eric Gravengaard. We did an all-night marathon session writing code to unwind that whole event so that \u2014 and we had to take the server completely off-line \u2014 like we killed the entire system, just turned it off \u2014 didn’t get it back on until about 7:30 in the morning.<\/p>\n

And that was \u2014\u2014’s very first day working on the project.<\/p>\n

Q. And did that ultimately get resolved? You said an all-nighter. Did that resolve it?<\/p>\n

A. It did. I think the government probably lost a quarter million dollars on it. The government had some safety mechanism that, through sheer luck, prevented the disaster. It didn’t allow Bitcoin withdrawals from 10:00 p.m. until around 6:00 or 7:00 in the morning.<\/p>\n

And because we did nightly releases, the transaction \u2014 we did releases at 10:00 p.m. So when we caught it around 11:00, 11:30, nobody could actually take the on-chain Bitcoin out of the system; they could only transfer it within Chivo. The only money that was able to leak was using Bitcoin Lightning. And so I just ballparked it at around a quarter of a million dollars.<\/p><\/blockquote>\n

The GDP of the United States ($20 trillion) is approximately a thousand times El Salvador\u2019s GDP ($26 billion). So this is as if the US government lost a quarter billion dollars to fraud due to a software error in an internationally publicised flagship payment project, by telling a junior developer to commit straight to production on his first day.<\/p>\n

The disasters that killed a payment system<\/h3>\n

The deposition is Overton speaking under oath \u2014 but we should keep in mind that this is just one guy\u2019s view of the situation.<\/p>\n

However, anyone who\u2019s worked in software development will understand the tale Overton tells. This is what consulting companies like ROI do for a living \u2014 they come in and clean up business software messes. We know<\/i> what sort of mess he would have seen in there.<\/p>\n

The disasters that Overton describes were huge news in El Salvador \u2014 especially the extensive KYC fraud.<\/a> His answers match the known timings and explain the observed problems in ways that make sense \u2014 what happened and why it would have happened.<\/p>\n

One unanswered question is how Athena got a contract they seem to have been utterly unable to deliver on. One hypothesis is that it was the Bitcoin Beach guys who got Athena in touch with the government, who were already interested in Bitcoin Beach. The Bitcoin Beach guys had brought the first Athena bitcoin ATMs into El Salvador.<\/p>\n

Athena lost the Chivo contract in December 2021. They were replaced by AlphaPoint.<\/p>\n

But it was too late for bitcoin in El Salvador \u2014 with these and other dysfunctionalities, the users\u2019 trust had been shattered. Almost nobody wanted to risk using the Chivo system, even to transmit dollars. Bitcoin is still legal tender in El Salvador, but its use is negligible.<\/a><\/p>\n

Everyone anticipated that Chivo would be used to steal and launder money \u2014 but I don\u2019t think anyone expected that this fate would be averted by Chivo just not working.<\/i><\/p>\n

 <\/p>\n

\"Become<\/a>

Your subscriptions keep this site going. Sign up today!<\/a><\/i><\/p><\/div>","protected":false},"excerpt":{"rendered":"

Anyone who\u2019s worked in software development will understand the tale Overton tells.<\/p>\n","protected":false},"author":1,"featured_media":24251,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"inline_featured_image":false,"jetpack_post_was_ever_published":false,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":""},"categories":[1],"tags":[3329,2419,2366,2305,3330,2378,3331,3328,2448,3327],"class_list":["post-24242","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-uncategorised","tag-accruvia","tag-athena","tag-chivo","tag-el-salvador","tag-eric-gravengaard","tag-lorenzo-rey","tag-miguel-sabal","tag-roi-developers","tag-sara-hanna","tag-shaun-overton"],"jetpack_featured_media_url":"https:\/\/davidgerard.co.uk\/blockchain\/wp-content\/uploads\/2022\/11\/chivo-balances.jpg","jetpack_sharing_enabled":true,"_links":{"self":[{"href":"https:\/\/davidgerard.co.uk\/blockchain\/wp-json\/wp\/v2\/posts\/24242","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/davidgerard.co.uk\/blockchain\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/davidgerard.co.uk\/blockchain\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/davidgerard.co.uk\/blockchain\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/davidgerard.co.uk\/blockchain\/wp-json\/wp\/v2\/comments?post=24242"}],"version-history":[{"count":28,"href":"https:\/\/davidgerard.co.uk\/blockchain\/wp-json\/wp\/v2\/posts\/24242\/revisions"}],"predecessor-version":[{"id":24271,"href":"https:\/\/davidgerard.co.uk\/blockchain\/wp-json\/wp\/v2\/posts\/24242\/revisions\/24271"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/davidgerard.co.uk\/blockchain\/wp-json\/wp\/v2\/media\/24251"}],"wp:attachment":[{"href":"https:\/\/davidgerard.co.uk\/blockchain\/wp-json\/wp\/v2\/media?parent=24242"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/davidgerard.co.uk\/blockchain\/wp-json\/wp\/v2\/categories?post=24242"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/davidgerard.co.uk\/blockchain\/wp-json\/wp\/v2\/tags?post=24242"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}