The stories you want. The opinions that matter.
No, thanks
Tag: iowa democratic party
Inside The Iowa Democratic Party’s ‘Boiler Room’ Meltdown

Inside The Iowa Democratic Party’s ‘Boiler Room’ Meltdown

This article was produced by Voting Booth, a project of the Independent Media Institute.

The app and software that failed to report and count the Iowa Democratic Party’s 2020 presidential caucus results were not the only miscalculation by the IDP and its vendor.

Its “boiler room” or secret operations center was unprepared to handle the chaos that ensued. Fewer than 100 computers and phones were set up for IDP volunteers to receive the results and complaints. IDP help desk materials anticipated problems with the app, including that some caucus chairs would end up calling in their results. The IDP’s instructions were complex, but they were also lacking some basics, according to experts in voting technology systems.

“In general, there is no documentation of what to do if various other problems arise,” said David Jefferson, a voting systems software expert who has studied and critiqued electronic election infrastructure since the 1990s. He is an executive board member of Verified Voting, a national advocacy group founded by computer scientists and academics.

“The universal answer seems to be, ‘If you have a problem, call in your results,’” he continued. “That might make sense if the app and the server were so robust that there were almost never any problems, and if the training of the caucus chairs was thorough so that they were all confident that they knew how to handle it. But apparently that was not the case.”

New revelations about the IDP “boiler room” offer important lessons. Nevada’s 2020 Democratic Party presidential caucus starts with early voting on Saturday, February 15. Its party officials have ditched their plans to use the same reporting app and system that failed in Iowa’s 2020 presidential caucus. While the Nevada Democratic Party has said that it might use paper-based voting in precinct caucuses, it has not released further specifics.

Meanwhile, DNC Chairman Tom Perez has called on Iowa’s state party to recount its results amid complaints from campaigns—a process that the IDP’s 2020 rules only anticipated if a campaign filed a formal request and paid for it upfront.

(On Thursday, Bloomberg.com reported that Donald Trump’s supporters apparently “flooded a hotline used by Iowa precinct chairs” after finding its number online. This was a development that nobody in state or national party circles overseeing the caucuses had voiced beforehand. Instead, they had focused on more complex cybersecurity threats.)

Inside IDP’s Boiler Room

The vote counting and help desk operations center was tucked into two conference rooms in the mezzanine above Veterans Memorial Hall at the Iowa Events Center—the opposite end of the same complex where a media filing center hosted TV networks whose analysts spent hours Monday night trashing the IDP for failing to release the caucus results. As the night unfolded, the party announced that irregularities in the counting, software glitches and jammed phones all collided and pushed the IDP to abandon using its results-reporting app and switch to compiling results from paper summary sheets from nearly 1,700 precincts.

I found the boiler room on Wednesday. While IDP officials were continuing to compile results in one room, this reporter saw the other room. It was empty but still set up as an operations center. It had computers, phones, routers; training materials on desks, help desk scripts and forms for taking caucus results and other complaints; an “app FAQ” sheet for caucus chairs who experienced problems and other instructions on wall posters.

The training materials were for a session that occurred on Sunday—one day before the 2020 caucus. An agenda said it included a “brief description of the caucus night set up,” “boiler room set up” work shifts, rules in the room, phone security and questions. Voting Booth took photos and shared them with Verified Voting’s Jefferson for context.

Jefferson concluded that the boiler room had less than half the staff that it needed at peak hours to field calls; that its FAQ and other help desk materials omitted key information—even as it anticipated that caucus chairs would encounter problems logging in and using passwords. He also noted what appeared to be additional missing elements that added to the congestion and reporting meltdown, such as the app not saying that results had been successfully received, which prompted caucus chairs to call in to confirm that they had been received—further clogging the phone lines.

The operations center was a windowless rectangular room filled with a half-dozen rows of long desks covered in black tablecloth. On each desk sat two or three laptop computers and a phone, all hardwired by Ethernet cables to a server in the room. The front desks had piles of a “2020 Precinct Caucus Guide” that summarized procedures, piles of forms to report results and other problems, and red and blue folders with these documents and others—such as app FAQs and phone scripts.

The front-facing wall had hand-lettered posters. One said, “NO photos can be taken in this room!!! NO social media usage allowed at all!!!!” There also were reminders for volunteers “to confirm [precinct] ID,” to tell caucus chairs where to find a “Precinct PIN” in their materials (to be able to log in using the app), and an email address where the chairs were to send photos of the “caucus math worksheets.” That last document had the key results from two rounds of caucus voting in each of Iowa’s 1,678 precincts. It was what the party scrambled to collect to verify and redo the statewide count, after finding tabulation errors from the app and data-entry mistakes by boiler room volunteers (as reported by the New York Times).

Together, if both conference rooms were similarly equipped (this reporter was escorted out when approaching the second room), that meant there were fewer than 100 computer stations ready to receive precinct results and handle complaints. All calls came in via the same telephone number. Once a caucus chair with results got through, IDP volunteers were told to fill out a form “completely… [and] confirm each number by reading it back to the precinct leader.” If the call was reporting something else, labeled “general public issues” or “temp chair issues,” volunteers took briefer notes and were told to “tell the caller [that] they will be called back ASAP.”

The party knew the app would be problematic. Their materials listed at least 10 different problems with the reporting app, including: “Can’t Sign In,” “Wrong Pin,” “Password,” “2 Factor Authentication,” “Error Message,” “Calculation Issues,” “Screen is Frozen,” “Can’t Take a Picture” and “Satellite Caucus Reporting [from outside the state].”

The “App FAQ” anticipated more specific problems, such as caucus chairs not finding their precinct IDs or PIN numbers to log in, and PIN numbers not working. The FAQ said there were two different PINs that the chairs had to use, one to identify the precinct and another to report the results. If they cannot get into the system’s backend with the PINs, the FAQ said, “They’ll need to close out of the app and restart the login process.”

It described additional steps “if they’re on a two-factor authentication screen.” If they could not log in, it said, “If for some reason they’re not receiving the code via text after a couple of tries, they should call in their results.”

The FAQ also said the caucus chairs could not go backward through the screens if any error was made. “Once they move on from a screen, they cannot go back to a previous screen and they cannot restart the caucus as they could in the test mode,” it said. “Every screen will ask them to confirm their numbers before moving on. If they made an error on any step, they should call in their results.” (Emphasis in original.)

A Complex Process Without Snafus

Verified Voting’s Jefferson called the app and these details “a human interface disaster.”

“One thing that sticks out like a sore thumb is the discussion under ‘Can I go back while reporting in the app?’ It says that no, you cannot go back to previous screens when using the app. It says, ‘If they’ve made an error on any step, they should call in their results,’” he said. “There is no recovery from error! No ‘undo.’ No ‘go back.’ They cannot even start over and ‘restart the caucus.’”

“This, of course, is a human interface disaster,” he said. “It is hard to fathom how anyone could design and field such a rigid, fragile app, [so] unforgiving, and expect hundreds of people who have never used the app before to be successful at it. This feature alone would likely cause most of the caucus chairs to call in their results instead of using the app, adding to frustration and phone-in volume. This alone should have caused the IDP to completely reject the app.”

Jefferson also said that the FAQ omitted basics, such as including different answers for people using Apple and Android operating systems. It did not mention how caucus chairs were to download the app, which would be different on Apple and Android phones.

But the larger point was the IDP was understaffed to handle the breakdown that ensued when it had to shift from an app-based reporting system that it knew was problematic.

Jefferson, doing rough calculations, said that at best, each call should have taken 5 minutes. Of course, once the apps started failing and the calls cascaded (apart from any interference intentionally clogging lines from Trump supporters), the volume went up quickly and a backlog developed that lasted as long as 90 minutes.

Under the best of circumstances, it would have taken five or six hours to efficiently handle the telephone reporting, he said, noting that the volume of information collected in 2020 was much more than in past years. The IDP was seeking to be more transparent. It was collecting and planned to announce four figures for every caucus: the participant numbers, first alignment results, realignment results and delegates awarded.

Jefferson suggested that the IDP had half as many help desk stations as it needed. But the IDP never expected that it would have to abandon its app and software backend. On the Friday before Monday’s caucus, IDP Executive Director Kevin Geiken was upbeat about the system. When pressed by this reporter about the “worst-case scenario,” he said that it would take one-to-two days to count the paper “caucus math worksheets” submitted in person or by emailed photos from across the state.

The IDP has not responded to my inquiries since Monday’s caucus.

Up Next: Nevada’s Caucuses

In response to the Iowa meltdown, Nevada State Democratic Party officials have said that they were abandoning the use of two different apps from the same vendor that failed in Iowa, and were looking at reverting to a paper-based voting and counting process. The NDP did not respond to Voting Booth’s inquiries about its likely shift to paper, which would begin with early voting that starts on February 15.

The party had planned to use 1,000 party-owned digital tablets for those participants to vote online, according to the Nevada Independent. It had also planned on using the same app as in Iowa to send its early voting results to its caucus chairs at 250 sites at the start of the regularly scheduled caucus process on February 22, and to transmit the results once the caucuses were finished.

The Nevada Democratic Party, like that of Iowa, intended to have every caucus participant fill out a paper presidential preference card, and for caucus chairs to compile results summary sheets that every candidate’s precinct captain would sign to affirm the results. But the system to file those results, and then compile and report statewide totals, is yet to be seen.

Steven Rosenfeld is the editor and chief correspondent of Voting Booth, a project of the Independent Media Institute. He has reported for National Public Radio, Marketplace, and Christian Science Monitor Radio, as well as a wide range of progressive publications including Salon, AlterNet, The American Prospect, and many others.

Political Parties Don’t Count Votes As Well As Government Officials

Political Parties Don’t Count Votes As Well As Government Officials

Reprinted with permission from ProPublica.

Here’s the takeaway from the Iowa fiasco: Beware of caucuses run by political parties. But don’t panic about the integrity of most primaries and the general election, which are run by state and county election administrators.

As Tuesday morning wore on without results from Iowa’s Democratic caucuses, the long-awaited first test of the strength of President Donald Trump’s would-be challengers, both public officials and enraged commentators stoked fears that Iowa was a harbinger of chaos for the rest of the 2020 campaign. Some said it raises alarms about the broader condition of election security and the reliability of computer systems that record, tally and publish the votes. Trump campaign manager Brad Parscale even suggested on Twitter Monday, without evidence, that the process was “rigged.”

But there’s a marked difference between the Iowa caucuses and the upcoming primaries in New Hampshire and South Carolina, as well as the 14 state primaries on Super Tuesday. The Iowa Democratic Party ran the caucuses, much as its counterparts in Nevada, Wyoming and several territories will do in the next few months. Party officials have less training and experience in administering the vote than do state and local election administrators who oversee most of the primaries.

Reflecting such concerns, the Democratic nominating process includes fewer caucuses this year than it did in 2016. The Democratic National Committee has called for using government-run primaries rather than party-run caucuses.

“Caucuses are run by rank amateurs. Even though we have concerns about the capacity of election officials, at least this is what they do a lot of,” said Charles Stewart, who runs MIT’s election data and science lab. “Even in the smallest of jurisdictions you run a lot of elections — you have contingency plans. The parties, bless their hearts, they don’t do this very much and that’s the bottom line.”

Connecticut Secretary of State Denise Merrill, whose office will oversee the state’s primary in April, said, “The Iowa caucus is an excellent reminder of why important elections should be run by trained, skilled and experienced state and local election administrators, not political parties.” Connecticut’s results undergo a post-election audit, and all votes there are on paper.

“Connecticut’s voters should be confident that they can trust the results of our elections,” she said.

In retrospect, Iowa’s Democratic Party made one mistake after another. It introduced a new app, widely reported to have been made by a company called Shadow Inc., without sufficient testing, training of precinct captains or transparency. At the same time, it made reporting requirements more complex, so that the 1,600 Democratic volunteers who manage individual precincts were required to provide three times as many data points as in past caucuses on a brand new app many had never been trained to use. (There were also many more candidates this year, further multiplying the amount of information to be reported.) Party officials didn’t hire enough people to take reports by phone in case the system failed. And they managed expectations poorly, assuring the public that results would be published faster than ever before.

“These are probably the most prepared we’ve ever been as a party for these caucuses,” Iowa Democratic Party Chair Troy Price told CBS on Monday morning, while shrugging off concerns about the possibility of technical problems. “We’re ready.”

This is not the first time that administrative problems have plagued the Iowa caucuses. In 2012, Mitt Romney was declared the winner of the Republican caucuses shortly after 1:30 a.m. by eight votes over Rick Santorum. Two weeks later, a recount showed Santorum had actually won. And in 2016, Hillary Clinton’s campaign declared victory after 2:30 a.m., even though official counting was not completed until that afternoon.

This year, the brand-new technology, lack of training and overconfidence by the state party amounted to a perfect storm. Government officials said they became aware of problems in the late afternoon, when precinct chairs began to report problems logging into the app. Many gave up on the app and began calling results in — as they’d done in past elections — but the reduced number of staff meant wait times so long that precinct leaders went home before they could report the results.

“Everyone was having the same problem,” said one Des Moines official who declined to be named. “Early on, it was obvious there were going to be problems.”

The receptionist at a WeWork office building in Washington, D.C., where Shadow Inc. listed its office in campaign finance filings, told a ProPublica reporter Tuesday that the company had recently moved out. Shadow CEO Gerard Niemira did not respond to a text message seeking comment Tuesday, and the voicemail box on his cellphone was full. An email to ACRONYM — an affiliated company — went unreturned.

One reason that caucus results are difficult to count is because they have multiple tallies. If a candidate doesn’t get 15% of the vote the first time, his or her supporters can switch to a rival. Delegates are apportioned by a mathematical formula. Now, the party is going through the painstaking process of verifying three datasets: the first expression of preference, the realignment and the overall delegate numbers. Verifying each number from each precinct takes several minutes, and the process must be repeated for more than 1,600 precincts. Because the Democratic Party did take the precaution of backing up counts on paper ballots, the final results should be verifiable. State party and federal officials have expressed confidence that the outcome will be accurate and trustworthy.

In a two-minute call just after 1 a.m. with the media, Price said the party was “validating every piece of data we have within that paper trail” and would “report results with full confidence.”

“We have said all along: We had backups in place for exactly this reason,” he said.

In a statement, Iowa Secretary of State Paul Pate, a Republican, said he was “glad to hear [the Iowa Democratic Party has ] a paper trail for their votes, just as we use paper ballots in all official elections in the state of Iowa.”

“I support IDP while they take their time and conduct checks and balances to ensure the accuracy and integrity of the votes,” he said.

Records show the state’s Democratic Party paid $60,000 to Shadow Inc. in two installments in November and December. The app was introduced with the intent of speeding up reporting. While local and national media began asking about the app weeks ago, the party was largely silent about its mechanics and said little about testing or training. Appearing on “Fox & Friends” Tuesday morning, Acting Department of Homeland Security Secretary Chad Wolf, said that the Iowa Democratic Party had declined to allow DHS to conduct vulnerability testing on the app, though he said DHS saw no signs of malicious activity. The party has put out a statement that it has confirmed there were no intrusions and that the problems were the result of a “coding issue in the reporting system.”

The confusion in Iowa does raise concerns about the rest of 2020’s caucuses, as well as states — such as Hawaii and Alaska — where parties run primaries. The Nevada State Democratic Party, which paid Shadow Inc. $58,000 in August for “technology services,” will hold its caucuses on Feb. 22. The state party did not return a call for comment about the payment or whether it is using Shadow to report returns.

Experts said that using little-tested apps can raise the risk of security breaches because hackers could take advantage of an app’s poor computer coding. Some criticized the secrecy that shrouded the app itself.

“For critical software, I always look for documented, third-party security validation and transparency into the testing process the vendor used,” said Chris Wysopal, the chief technology officer at security firm Veracode and a prominent computer security expert. “It is a big, red flag if there is secrecy about the development process used to create and test an app.”

Election observers said one lesson of Iowa is that accuracy and clarity should be valued over speed. “It’s not about election integrity — the results will be verified with paper — it’s about satisfying our need to know immediately who won,” said David Becker, the executive director of the Center for Election Innovation & Research. “When we balance out what’s more important, speed or accuracy, it’s not even a close call. We should be expecting accuracy and adjusting our expectations in regards to speed.”

Photo Credit: Phil Roeder

Behind The Iowa Caucus Crash, A Simple Technology Glitch

Behind The Iowa Caucus Crash, A Simple Technology Glitch

This article was produced by Voting Booth, a project of the Independent Media Institute.

The electronic system used by the Iowa Democratic Party on Monday night to compile its 2020 presidential caucus results was only counting “partial data,” Iowa Democratic Party chairman Troy Price said in a statement Tuesday morning, giving the most specific clue about what went wrong.

That partial data—called a “coding error” in the most recent national press reports—was most likely tied to three different sets of figures that the IDP planned to release for the first time after the caucuses ended, but withheld due to what the IDP called unspecified “inconsistencies.” 

The state party announced at midday that the “majority” of results will be released at 4 PM Central Standard Time.

Those three sets of “inconsistent” figures — details of which the Iowa party has never released in previous cycles — could only refer to steps in the process where the number of participants and the votes cast in two consecutive rounds of caucus voting did not all match.

Iowa Democratic officials haven’t said more yet about what went wrong with the tabulating system software, which was never tested before on the scale used Monday. evening But it is possible to identify one discrepancy in the numbers that would have been reported via the IDP’s app to its software and system and could have caused the “partial” data and “inconsistent” analyses. 

A likely cause of “partial data” flaw may have been the process itself, according to my own eye-witness observations and assessment (based on undertaking Iowa’s caucus chair training and numerous interviews with top party officials, including a demo of the app last week).  

The “partial data” or data mismatch may have little to do with the app used by caucus chairs to report the winners in two consecutive rounds of voting and the resulting delegate allotments (although caucus chairs and campaign precinct captains had problems with getting online and logging into systems.) A more likely problem was simply that not everybody attending a caucus voted in the second round, if their top presidential choice was disqualified in the first round. 

That pattern of drop-off voting could have produced the “inconsistencies” that were seen by the IDP boiler room. In the demo by Iowa Democratic Party executive director Kevin Geiken, the app showed when too many participants went into its calculator (called an “overcount” on the app), but it didn’t report undercounting or intentional drop offs in the second round.

This very scenario was seen in Polk County’s 57th precinct in Des Moines, when 11 voters of the 385 attendees did not vote for another candidate in the realignment round—after their first presidential choice was eliminated. 

Those voters mostly came to vote for Joe Biden and were overheard saying that they could not vote for anyone else, especially after the race’s other centrist, Amy Klobuchar, also had been eliminated in the qualifying first round. They didn’t want to cast a vote for Bernie Sanders or Elizabeth Warren in the realignment.

In other words, they came and they voted. Their first choice lost, or at least wasn’t “viable” in that room. They didn’t pick another candidate. Thus, there were gaps between the number of participants in the two rounds of voting. That could account for tabulation software in the IDP’s electronic backend seeking balanced totals — via an app on each of 1,678 chairs’ smartphones — and reporting figures that didn’t match or balance out. 

Or these same voter fall-off numbers also could have appeared in called-in results that the state party was receiving, if the caucus chairs could not connect with their app to the party’s backend. That also happened in Polk-57, where Caucus Chair John McCormally could not get his app to log in before and during the event, despite trying several times.

So McCormally ran the caucus using pens, math, and paper, and then called in results. In his case, McCormally reported the results quickly (possibly because his wife was a volunteer there). But other chairs across the state encountered waits of 90 minutes before talking to the party

This quagmire deepened inside Democratic Party headquarters when it had to assess the growing problem. They had to quickly find a solution or use a backup plan, which top party officials had bullishly predicted only a day earlier that they would not need. The backup entailed gathering from every caucus chair the paper summary sheets that list the voting round totals and delegates won.

The summary sheets, signed by precinct captains from all the campaigns, were to be turned in to Democratic county chairs (along with presidential preference cards filled out by voters), according to the caucus training materials. The county chairs, in turn, were to turn in all their paper records to Iowa Democratic Party headquarters either in person or by mail. 

Before Caucus Night, Geiken was asked about worst-case scenarios in a demo of the caucus app that only two reporters attended—including this writer. It might take a day or two to physically collect and recount the full paper vote record, should the electronic system be jettisoned for whatever reason. He emphasized there would be a reliable and accurate count, but it might not be as fast as expected. 

The state party’s announcement early on Tuesday that the “majority” of votes would be released suggests that it made a dash to collect and count as many ballot summary sheets as possible. 

When I covered this ‘what comes next’ scenario before Iowa’s caucuses, few state and national party officials imagined that the reporting system would melt down. They expressed great confidence in the party’s voting system and its private contractors. (Caucuses are not run directly by government election officials but use rented voting systems.) 

Even hours before the caucuses began, these officials downplayed reports that some precinct chairs were having trouble signing onto the caucus app, as well as the possible consequences.

But voting technology experts predicted these problems. Reliability issues are to be expected when a new system debuts, especially one that has not been tested at scale and when its users encounter access issues (insufficient bandwidth and unable to log in) atop software glitches. That is why government election officials like to debut new voting systems in low-profile races.

Iowans and everyone else will get to see accurate results eventually. That is a silver lining.  The IDP will be using a paper trail and that paper trail will be more detailed than in any past caucus.

The only other silver lining might be the realization by national media outlets that it may not be possible to report fast and accurate results on election nights. Many other states will introduce new voting and reporting systems in caucuses and primaries this year. But Nevada, at least, has abandoned a system similar to the one that crashed in Iowa.

“Nevada Dems can confidently say that what happened in the Iowa caucus last night will not happen in Nevada on February 22nd,” said Nevada Democratic Party Chair William McCurdy II on Tuesday. “We will not be employing the same app or vendor used in the Iowa caucus. We had already developed a series of backups and redundant reporting systems, and are currently evaluating the best path forward.”

Steven Rosenfeld is the editor and chief correspondent of Voting Booth, a project of the Independent Media Institute. He has reported for National Public Radio, Marketplace, and Christian Science Monitor Radio, as well as a wide range of progressive publications including Salon, AlterNet, The American Prospect, and many others.