In under 24 hours, news of the Iowa Democratic Party’s failed attempts to utilize their new Mobile App has reached headlines nationwide. The app, built by Shadow, inc and funded by ACRONYM (sounds evil, right?), was intended to help speed up the processing of data and replace the traditional phone system for calling in election results. Alas, the mobile app was reported to have failed completely for a number of reasons.
Let’s provide a brief background of Shadow and recap what happened to provide some insight on what mistakes were made along the way, and what could be done in the future to avoid these mistakes. Let’s dive into what happened in Iowa…
Background – The History of ACRONYM (Shadow’s parent company)
Shadow, the company that built the Mobile app used by the Iowa Democratic Party, is actually owned and invested in by a company called “ACRONYM”. ACRONYM was founded in 2017 and quickly became a frequent receiver of the Democratic donations after Hilary Clinton’s campaign in 2016. According to its website, ACRONYM is a nonprofit organization committed to building power and lasting digital infrastructure for the progressive movement.
According to the New York Times, last year Acronym unveiled a plan to spend $75 million on digital advertising to counter President Trump’s advertising campaigns. As part of that $75,000,000, an undisclosed couple of million dollars were invested in a company called Groundbase, which later renamed itself to Shadow.
Groundbase (Shadow’s) Funding from ACRONYM
Groundbase (now “Shadow”) was founded by Gerard Niemira and Krista Davis to build a texting-based platform that was intended to target campaigns. After a number of years Groundbase’s main texting product failed to catch on with users, and the company was nearing bankruptcy. Seeing an opportunity, ACRONYM stepped in to finance Shadow with a plan to refocus the organization to develop mobile apps for campaigns. The funding allowed Shadow to maintain itself long enough for additional projects, such as Lightrail; a tool built to help centralize data in the democratic party – as well as for Nevada and Wisconsin State Democratic parties – and presidential campaigns such as that of Joe Biden.
Although Shadow was able to garner some projects, there were numerous reports that the texting program was particularly programmatic and sources alluded to security concerns. An anonymous aide to Mr. Biden said the campaign attempted to use Shadow to text voters in Philadelphia last year, but the technology did not pass cybersecurity checks.
Nonetheless, the Iowa Democratic party still chose to use Shadow’s services after receiving bids from multiple bidders and paid two installments of $63,183.00 (a total of $126,366) to Shadow, which was given only limited time to design, build, and deploy an app from scratch.
Why did Shadow’s Iowa Application Fail?
Shadow was given an impossible task: given an insufficient budget; an inexperienced team of engineers; and a rushed timeline, Shadow was tasked to design and develop a mobile application to be used by the Iowa Democratic Party for recording election results in a timely and succinct manner. Employees of ACRONYM who spoke on the condition of anonymity said the app was so rushed, that there was no time to have the application approved by the App Store.
This is an extremely important step, and the party should have postponed the launch to a different election. For any readers wondering why skipping the App store is a red flag, let me elaborate: The App Store Review Process is (often) a grueling process that can last days or weeks; in which Apple employees will manually check a mobile application for bugs, user experience issues, security issues, and to see if anything within the app is violating Apple policy or best practice. While it can often be annoying to business owners and product teams, it does provide an extra ‘sanity’ check that has often allowed users of Apple’s App Store to be able to easily and readily access and understand all apps launched on their platform.
Simply put, the application failed because there was not enough time or budget allocated to go through proper design, development, testing, and stress testing of the application. When the App Store rejected the application, Shadow and the democratic party should have simply fallen back to the traditional approach.
According to Shadow
“The goal of the app was to ensure accuracy in a complex reporting process,” Shadow tweeted. “We will apply the lessons learned in the future, and have already corrected the underlying technology issue. We take these issues very seriously, and are committed to improving and evolving to support the Democratic Party’s goal of modernizing its election processes. We sincerely regret the delay in the reporting of the results of last night’s Iowa caucuses and the uncertainty it has caused to the candidates, their campaigns, and Democratic caucus-goers.”— Shadow, Inc. (@ShadowIncHQ) February 4, 2020
What could the Party and Shadow have done better?
First off, there are generally two types of applications: Mission-critical, and non-critical applications.
A non-critical application is an app in which there are margins of error allowed that will not dramatically impact people, businesses, or organizations. In a non-critical application, an error may occur with no lives affected or organizational consequences besides a small margin of unhappy users. As an example, let’s say Tinder servers go down for 4 hours for maintenance. Yes, they may lose some business at that time – but will people die or will they shut down? No.
A mission-critical application; however, is another story altogether.
A mission-critical mobile application is any app that has real-world consequences that relate directly to the survival of a person, business, or organization. In this case, the United States voting system would most definitely be considered a mission-critical application. These applications have a load of requirements that need to be met from a planning and architecture perspective, such as:
- Security Audits
- Any application that has sensitive data must go through appropriate security audits to ensure that data can not be manipulated, leaked, or otherwise compromised. As an example: for HIPPA-compliant apps; often companies such as Hurricane Labs would be hired to perform security and API checks.
- Data Redundancy & Backups
- Organizations and tools such as AWS (Amazon Web Services), Google Firebase, and Microsoft Azure have a large number of services revolving around data redundancy and backups; however, these all need set up by professionals. The engineering team at Shadow may not have had the expertise to provide these services.
- Offline Mode
- If you have mission-critical data; however, and a customer loses internet connection – what happens? There are systems that can be put in place to compensate for poor signal and temporary lack of internet service; especially in rural areas.
- Stress Testing
- Testing with a couple of team members is extremely different than go-live with millions of users. Trust me on this, I’ve been through it and seen it happen. If your architecture or hosting solution is not set up properly, it can cause a cascading effect across your app and cause users to be unable to utilize the services. There are ways to compensate for scale, but it takes experience and planning to prepare.
- Compliance Testing
- What compliance requirements are there for an election? Are there rules or regulations? These need to be followed to avoid liability.
- Proper Device & OS Testing
- Between iOS and Android there are thousands of combinations of Devices (iPhone X, XS, 5, 6, 7, Samsung S10, Note, etc) and operating systems (OS 12, 13, etc). These would all need to be tested thoroughly for your market before releasing. At OpenForge, we have found that while you can get 90% of the way there on a simulator, it is absolutely critical to have the physical devices for your Quality Assurance team to test on.
These are all things Shadow company (and the parent company, ACRONYM) could have done to prevent such a disaster. On the bright side though, I’m sure they have learned from the situation and will budget accordingly next time around. At the end of the day, you get what you pay for and mission-critical applications should never be taken lightly. In our experience, mission-critical applications can cost anywhere from $150,000 to $1,000,000, depending on complexity; with simpler apps hovering around the $250,000 – $350,000 range.
Can a company hire a team and build an app for less? Absolutely; it will simply be a matter of risk tolerance and timeline. More expensive firms have experienced engineers who have lived through these types of experiences before and have learned from past mistakes so that the same mistakes are not made again.
If I had to make recommendations to Shadow company (or any other party out there looking to attempt a new technology on a large scale) I’d suggest 3 things. First, allocate a proper budget for the scale and timeline of what you’re trying to accomplish. Second, timing is everything. It doesn’t matter how much money you spend; if you’re only given a couple of weeks to design and build an application you can not expect it to be bug-free or reliable. Good things take time; and large-scale applications that are mission-critical need to go through proper processes and appropriate testing in order to be considered “Go-Live Ready”. Third and finally, listen to the engineers and know when to pull out. If the team creating the app is not confident and you have not followed the checks listed above at least 2 weeks before launch – do NOT try to launch. Software Development is analogous to Construction – what happens when you ask your plumber to rush and cut corners? You get soaked.
We wish Shadow the best; it’s tough being put in a no-win situation, but we also hope that everyone learns from this case study. The most important lesson is that mission-critical apps should never be rushed.