David Helkowski set out to be a whistle-blower; he now faces the feds and unemployment.
by Sean Gallagher
 |
| Aurich Lawson / Thinkstock |
David Helkowski stood waiting outside a restaurant in Towson, Maryland, fresh from a visit to the unemployment office. Recently let go from his computer consulting job after engaging in some “freelance hacking” of a client’s network, Helkowski was still insistent on one point: his hack, designed to draw attention to security flaws, had been a noble act.
The FBI had a slightly different take on what happened, raiding Helkowski’s home and seizing his gear. Helkowski described the event on reddit in a thread he titled, “IamA Hacker who was Raided by the FBI and Secret Service AMAA!” Recently Ars sat down with him, hoping to get a better understanding of how this whitehat entered a world of gray. Helkowski was willing to tell practically everything—even in the middle of an ongoing investigation.
Until recently, Helkowski worked for The Canton Group, a Baltimore-based computer consulting firm serving, among other clients, the University of Maryland. Helkowski’s job title at The Canton Group was “team lead of open source solutions,” but he began to shift his concerns toward security after identifying problems on a University of Maryland server.
That transformation from developer to hacker came to a head when Helkowski decided that the vulnerabilities had gone unfixed for too long. He set out to prove a point about computer security both to the University of Maryland and to his employers. In early March 2014, working from a computer in his Parkville, Maryland home, Helkowski said that he exploited a misconfigured Web server and some poor database security in order to duplicate the results of a recent data breach that exposed the Social Security numbers and personal information for more than 300,000 current and former University of Maryland students and staff.
On March 14, Helkowski made his point rather dramatically by posting the university president’s Social Security number and phone number to reddit. He then sent an anonymous e-mail to the members of the university’s newly formed security task force, telling them in no uncertain terms just how horrible their security was.
Though he claims the message was not meant to sound threatening, it included lines like, “Out of politeness I’ll give you a chance to respond directly about this to me, and I’ll consider pulling it off the public Internet…Your internal IDs are listed below to get your attention.” If the security task force wouldn’t work with him, Helkowski told them to “consider this your fair warning and last contact from me.”
Despite his use of proxy and VPN services, the FBI began asking questions around Helkowski’s workplace the very next day. On the afternoon of March 16, agents investigating the case obtained a warrant. They kicked in the door of Helkowski’s home at 7pm that night. He was not arrested during the search, but his electronics were seized, his dog was (temporarily) lost, and federal charges may well be forthcoming.
The picture Helkowski painted of the circumstances leading up to his brush with law enforcement—and the still-present possibility that he will face criminal and civil charges—bears a strong resemblance to stories of information systems projects gone wrong at other universities and institutions. It has particular resonance for the public sector, where contractors can sometimes find it easier and safer to turn a blind eye to security problems rather than disclose them.
An unlikely hacker
Helkowski looks a bit older than his 32 years, with a slight build and a few gray hairs among the wiry black ones pulled back into his ponytail. He’s clearly confident in his abilities and confident that what he’s done is morally, if not legally, right. Helkowski told Ars he would act in exactly the same way if he had to do it all again.
An avid Steam gamer and anime fan, Helkowski is also a self-described computer keyboard aficionado. At one point, he had a collection of more than 35 keyboards, starting with a Northgate OmniKey that he picked up at a yard sale for $10. When the FBI raided his house, the agents had difficulty dealing with his current preferred keyboard—a Japanese keyboard with an English alphabet that he bought in the Akihabara district of Tokyo. The characters for each key are printed on the front of the key instead of its top, and its symbols don’t properly map to US standards.
“I figured out how to make it work with Windows by going in and making Windows Registry changes,” he told me. “But now I’ve just memorized which key maps to which symbol.”
Helkowski’s travels to Japan weren’t just driven by a desire for a clicky keyboard. Helkowski’s wife, a pianist and music teacher, is from Japan. He met her while she was in the US studying at Berklee College of Music. They dated but once. A few years later, after another relationship ended, he started communicating with her over the Internet again, finding she moved back to Japan to get a degree in music therapy. He traveled to see her, and they married six years ago.
“She teaches piano to kids 3 to 12 years old, and we have two grand pianos in our living room,” he said. Helkowski has been a developer for a number of Maryland tech companies, doing everything from Web scripting to hardcore C development. Before joining The Canton Group last July, he worked as a contractor for a year at T. Rowe Price. But it was his experience in dealing with ColdFusion and doing data conversion work that greased the chain of events leading to the FBI.
(In what follows, all technical details were provided by Helkowski; neither the university nor The Canton Group agreed to speak with Ars about his account.)
In November of 2013, Helkowski said, he was asked by a co-worker on The Canton Group’s team responsible for work with the Drupal Web content management system to help migrate data from a legacy ColdFusion site belonging to the University of Maryland’s School of Public Health. Using the team’s access to the server, Helkowski said he downloaded the contents of the site’s directory from the server to his work computer.
That download set off his malware scanner. Helkowski started to investigate why, and he found that one of the files on the server was a PHP script file. The code in the file had been compressed and obfuscated to hide its purpose, so Helkowski began decompressing and analyzing the code to find out just what it did.
The script turned out to be a piece of Web server malware known as C99Shell—a backdoor script that allows a remote user to execute arbitrary commands on the server, search through the file system, and upload files among other things. Because of the configuration of the server, the remote user was able to execute commands with the permissions of the Web server (httpd) user account. That included access to other University of Maryland websites, which resided in other directories on the same file system. It also came with the ability to change file permissions. “It was pretty close to root access because the user was so widely capable,” Helkowski said.
Based on its creation date in the file system, Helkowski said, the backdoor script had been on the server since 2011—meaning that the server was breached at least once over the last two years. He found another similar script not detected by the malware scanner. It appeared that the scripts were both uploaded to the site through a Web interface that allowed site users to post images to the website. The directory the upload page put files in was configured to allow PHP scripts within it to be executed by the Web server. Not good.
Passwords
As he was looking at the ColdFusion site itself to figure out how to migrate the university data to Drupal, Helkowski found that ColdFusion was configured with passwords for a host of databases within the University’s network that weren’t directly related to the site itself.
“I made a dump of all the passwords to all the databases in November just to cover my ass,” Helkowski told me. “I didn’t do anything with them, but I wondered why ColdFusion had access to 80 plus databases—I assumed it was secure access and not sensitive stuff. But anything I came across, I made a copy of it and put it in an encrypted container. Just in case something bad happens, I have evidence of what happened.”
At that point, Helkowski said, he took what he learned to a co-worker and then to a Canton Group senior executive.
“I’ve been attempting up until recently to defend The Canton Group,” Helkowski said. “I think what happened was that when we communicated with the [executive], he assumed we communicated to the university, and we assumed that he had, so no one did. I’ve actually said that to other people, that I don’t want to throw them under the bus.”
But Helkowski said that, in February 2014, he learned the backdoors were in fact still on the university server—and they became part of a much bigger problem.
Once again into the breach
This was not a great winter for data security at universities. In March, Indiana University revealed that it accidentally stored the personal data of 146,000 recent and current students on a publicly accessible website for 11 months, and the data was crawled by search engines. The North Dakota University System had a data breach, including SSNs, that began in October of 2013 and continued until early February. And Johns Hopkins University faced a smaller breach, exposing the names and contact information of 1,300 biomedical engineering students. A message from someone claiming to be from Anonymous took credit for the JHU breach:
But the biggest breach of them all would be at the University of Maryland.
On February 18, Brian Voss, the university’s vice president of information technology, informed university President Wallace Loh that someone broke into one of those databases Helkowski noticed. The hacker obtained records for 309,079 current and former students, faculty members, and employees. The data included Social Security numbers, dates of birth, and university ID numbers.
In an open letter published the next day, Loh said that “the University of Maryland was the victim of a sophisticated computer security attack that exposed records containing personal information.” Loh apologized and promised immediate action. In a follow-up message, he announced the formation of a computer security task force which would, among other things:
- Conduct a scan of every database to find out where sensitive personal information might be located. Then the task force members will recommend the appropriate course of action: purge the data, or ensure its protection;
- Perform penetration tests of the security defenses of our central and local information systems to identify and seal any possible technological gaps through which cyber criminals could gain access. Further, recommend policies and procedures to establish these probes on an ongoing basis.As news of the breach spread, Helkowski said, he went back to check if the malicious PHP script files were still on the university’s server. They were still there, untouched. Helkowski then decided to check if the attacker could have used the information in the ColdFusion site to gain access to personal data.
“I was like, ‘I wonder if this has access to the Social Security numbers that everyone was complaining about,” he said. “And I said, if they were anywhere, I would bet they were in this database named LDAP.” (LDAP refers to the “Lightweight Directory Access Protocol,” which allows for single login accounts across networked computers and can also provide directory lookups on other users of the network.) As it turned out, the LDAP contained a field labeled “SSN.”“So I then said, let’s query it,” Helkowski said. “And [people on my team] said, ‘No, don’t dump that, you shouldn’t do that.’” But Helkowski knew someone who was a current student at the university, and he obtained the student’s authorization to check the database for his information.“So we did a live query [of the database] to pull up his information,” Helkowski said. “And he was able to confirm his Social Security number, date of birth, GPA, major, his student ID number as well as the barcode on the back of the student ID which would actually allow someone to forge student identification cards. That was all in the LDAP database.”After seeing this, one of Helkowski’s co-workers then “went directly to the CEO [of The Canton Group, Ethan Kazi] and told him,” Helkowski said. He then wrote up “a full report of how bad I believed that access to be,” including recommendations to correct them such as changing all of the database passwords that were hard-coded into the ColdFusion site.But according to Helkowski, he was soon told that his report wouldn’t actually be given to the university. This infuriated him, and he felt like the issue was being swept under the rug.“In my personal opinion, if we had been involved in [reporting the breach back in] November 2013 and they had listened to my advice and done the things I suggested, they would have patched the holes that the hacker would have used to get in [in February 2014],” he said. “It’s not the same hole I believe, but they would have set things in motion.”
Going rogue
Helkowski began drifting away from being a mere employee, becoming—in his own eyes—a righteous force for good. Determined to get his report into the hands of someone at the university that could act on the problem, Helkowski gave a copy to a student in University of Maryland’s cybersecurity program. He hoped it would be passed on to the University’s security team.“It was a Wednesday when we discovered [the vulnerability], and it was a Friday when we had a teleconference meeting with UMD, which they did not involve me with at first,” Helkowski told us. “But halfway through, UMD requested me to join to ask me questions and to demonstrate the vulnerability. During the part where they asked me to demonstrate the vulnerability, I said, ‘I can’t actively show you a dump of the SSNs because I don’t have authority to do so.’ I asked, ‘Do you guys give me authorization to reveal this information, because otherwise I shouldn’t be doing this,’ and multiple people on the other end of the line said, ‘Yes, we give you authorization.’”Helkowski then told them about the database named LDAP.“They said, ‘We don’t know what this is, we don’t know where it came from.’ Apparently it’s some sort of mirror that someone created for some other application—it wasn’t the live directory,” he said. “But I was able to subsequently prove that there was another table that was inside that LDAP database, that was all the user names and passwords that had permission to query that database. That table included clear text passwords for those people as well. And the HTTP user had the ability to see that. In other words, you could write a ColdFusion script, not do any hacking, just hit that database and dump out the main active directory credentials. You could hit that server from anywhere and download that entire directory structure, all 300k of those numbers.”After finally getting his message across to the university, Helkowski said he waited about two weeks more and checked to see if anyone addressed the problems. The backdoor scripts were archived in place so that they could no longer be executed, and a configuration file (an .htaccess file) had been added to the image upload directory to prevent new scripts loaded there from running. Still, problems remained.“It was still open access to the database,” Helkowski said. “They did not change any of the passwords. So that’s sort of what triggered me to look further into how bad their security was. Because I was like—if they’re not changing that, they aren’t changing anything.”Helkowski decided to take his work home with him and see exactly how far he could go in penetrating the University of Maryland’s network. It turned out that breaking into the network from outside was as easy as performing a Google search.Extra credit
Helkowski knew that the backdoor script he reported to the University was closed, but he suspected there might be more exploits in place on other websites within the UMD system that could grant him the same access to university databases.“I searched for other PHP scripts with the word ‘upload’ in their name by Googling it,” he said. “And then I went and looked at those files, and looked at the code, and they were vulnerable as well. And that’s when I found another shell script. So not only have I found another place that was vulnerable, but it’s already been attacked.”Using one of these backdoors, Helkowski didn’t have to do any real hacking to gain access to nearly everything on the server. With the door wide open for him, Helkowski started to see what he could do within the system.The answer turned out to be just about anything. He managed, using the access made available to him through the Web server’s privileges, to install and compile the Gnu C Compiler (gcc) on the server. He installed the Nmap security scanner on the server. Helkowski even established a remote Xwindow session to the server from his home computer and was able to run Oracle command-line utilities against databases using passwords he was able to read in the script files—including that database called “LDAP.”“I had access for quite a while. I could have escalated my access,” Helkowski said. “I did not do that but was considering that.”But Helkowski decided that he reached “the turning point—where I would go from whitehat to black hat,” he said. “I decided I needed to tell them immediately, and I gave them a full list of what I did in detail. That way the university would have full evidence of what I did and be able to distinguish me from a malicious person. I didn’t want to interfere with the investigation.”The way Helkowski chose to communicate that information, however, did not exactly cast him in the best light. He devised a plan that he thought would help both get the university’s attention about the severity of their security problems while keeping him out of hot water. As it turned out, he succeeded on the first count but failed miserably on the second.Using the LDAP database, Helkowski obtained the e-mail addresses of all of the members of President Loh’s newly formed security task force. He also obtained President Loh’s Social Security number and phone number. On the night of March 14, to demonstrate the seriousness of the situation, he posted the information to Pastebin and linked it on reddit. He then sent a lengthy e-mail to all of the members of the security task force from an anonymous e-mail service, claiming to be a hacker who obtained a copy of the security report that Helkowski leaked to the University.His e-mail, which was included with the eventual FBI search warrant affidavit, read in full:Helkowski didn’t think that this came off as threatening. “Based on their response, it seemed they wanted to cooperate, though they may have thought I was malicious,” he said. “I said, I need assurances you will not press charges against me, and they said absolutely.”In the meantime, Helkowski shared what he was doing with a number of his co-workers, and some of those chats took place through the computer gaming service Steam. On March 15, the FBI was already questioning employees of The Canton Group, including one co-worker who provided a log of his Steam chat conversation with Helkowski.By the next afternoon, the FBI had a warrant. At 7pm on March 16, the FBI and Secret Service literally knocked Helkowski’s door in.“Is there a party at my house?”
“So what they did was they opened my gate to my backyard,” Helkowski said, describing the raid. “They went to my side door and bashed the door in.”Frightened by the raid, Helkowski’s dog ducked out a doggy door into the backyard and escaped through the still-open gate. “And they were basically, ‘Oh well, the dog ran off,’” he said.Helkowski and his wife weren’t at home when the raid started. They were at dinner with acquaintances following a musical performance at a nearby university. As the couple arrived at their Parkville home around 8pm, they noticed that things were not as expected.“I saw the lights of my house all lit up,” Helkowski said. “I saw people walking around the front yard, cars and SUVs in front of my house [and] in my driveway, and I was like, ‘What, is there a party in my house?’ So I slowed down and pulled over a little bit—and they immediately yelled, ‘Pull the car over, stop the car, put the window down, and put both your hands out the window.’ And I saw they had a gun on me.”Helkowski told the agents he would cooperate, and he asked if someone would call his father to help retrieve his dog. An agent did call his father, posing as one of Helkowski’s friend, saying that he saw the dog loose. When Helkowski’s father arrived and opened his car door, the dog jumped in—and agents then brought his father inside while the search continued.Helkowski admitted to the agents that he dumped information from the University of Maryland servers to his computers. And there was no shortage of computer gear for the agents to search. “In my house, the main server I was using was a dual Opteron system with 64 gigabytes of memory,” Helkowski recounted. “They took that and all the USB drives and SD chips on that desk and anywhere near it—which was like 15. They took my PSP.”They also took the hard drive out of his wife’s computer—a system he built from a former server—and her USB drive. He later called the FBI and told them that there was nothing on his wife’s USB drive that had anything to do with him, and “they’ve since returned it to me,” he said.FBI agents skipped over other items, such as an old server that sat partially disassembled. “I had taken the memory out just because I was going to do something else with it,” Helkowski said. Agents also passed on collecting “about 400 LTO-3 backup tapes, about 15 terabytes’ worth.”After the FBI left, Helkowski realized that he still had a copy of some of the dumped data. “I relayed that information to the FBI because I’m cooperating with them,” he said. “They said, ‘We need to take that too.’ And I said, ‘Well, I have that on an encrypted drive which I’m not going to give you the password to because I have other stuff on it.’ So they came to the house and watched while I used a secure delete utility and they verified it was gone.”The next day, Helkowski went into work and told his employer about the raid. He was let go shortly afterward. Helkowski said he was laid off without a reason.We contacted The Canton Group and the University of Maryland to get their side of the story, but neither organization would discuss the case. Jason DeLoach, an attorney for The Canton Group, told us only that “The Canton Group continues to cooperate with law enforcement to aid in their investigation of David Helkowski. David is no longer an employee of The Canton Group. We are fully cooperating with The University of Maryland and conducting our own internal investigation into this matter. We will have no further comment due to the ongoing investigations.”End of line
Though he has not been charged with a crime, Helkowski has since contacted the office of the Federal Public Defender in Baltimore for representation. Even if no charges materialize, Helkowski knows that he may now be unemployable in his chosen profession.“I suspect that when potential employers learn of this, I may not be able to get a job,” he said. “I will probably remain unemployed, which will probably drive me into bankruptcy.”But that doesn’t mean Helkowski has any particular regrets about what he did. He feels that he would have been laid off anyway—even if he hadn’t escalated his security concerns to the level that involved federal law enforcement.“I definitely did not mean to cause any damage to the University or to The Canton Group,” he said, reflecting on the experience. “I really hate The Canton Group—I think they’re a bunch of idiots.”If the vulnerabilities Helkowski says he found were the ones used by attackers to gain access to personal information of students and staff, it wouldn’t be unusual. It’s not the first time a publicly facing website—with relatively low value to attackers on its own—was used as the gateway for a much more damaging intrusion. As for The Canton Group, its obligations regarding the overall security of the site are unclear, given that the firm was hired to migrate the University off the very site that was vulnerable.IT contractors face this sort of dilemma all the time. Pros are often faced with pre-existing systems that are poorly configured and potentially riddled with vulnerabilities, even as they are asked to integrate work with other systems over which they have little control.In the weeks since Helkowski originally posted what he did to reddit, we’ve spoken with a number of people who’ve worked in IT on both sides of the contractor/client relationship, including people who have worked at other universities. What Helkowski described resonated with all of them: contractors who complained that clients didn’t care about reported vulnerabilities and internal IT people who bemoaned legacy systems bolted together on the cheap with no one left to maintain them.But none of these people took matters into their own hands the way Helkowski did. Regardless of whether it was well-intentioned or otherwise, Helkowski’s attempt to create a teachable moment in the middle of an investigation—one involving the University of Maryland’s own IT department, federal officials, security experts from MITRE, and other outside organizations—ended up creating more confusion and cost. It’s a move that even Helkowski admits may keep him from ever holding a job in IT again.
