There are times when working in cyber-security feels like any other job. You show up every morning, have a cup of coffee and scan your emails. You review code and run tests, you write reports, you attend meetings, and then you go home. But other times, you stumble across something that scares the crap out of you. With a sinking feeling in the pit of your stomach, you pick up the phone and you call the FBI and the Secret Service. One of our cyber-security experts, Dana Taylor, had a morning like that, not very long ago. She discovered a vulnerability in versions of Oracle software being used by hundreds of high-profile organizations, both in the United States and around the world.

The vulnerability left critical data out in the open for anyone to read – simple as that. Reams of confidential information was exposed, unencrypted and unprotected, for anyone on the internet to view.  Within minutes, Dana realized the scale of what she’d found and called in the rest of the team. Even a brief review of vulnerability and the affected organizations demonstrated a cyber-security vulnerability of staggering proportions. The next few days were hectic, as Rhino Security Labs worked around the clock with the FBI – identifying affected organizations and closing the holes, one by one.

The list of affected organizations and the variety exposed information is chilling. One major city exposed arrest and prosecution information on hundreds of citizens. Another instance revealed records of young students at a large school that also included detailed bus route information, photographs of the students, their medical information, and even the social insurance numbers and marital status of the parents. There were full student records from more than one of the Big Ten universities, ranging from social security numbers and grade reports to full transcripts, and the salary information of hundreds of thousands of students and faculty – even the numbers of the dorm rooms where the students lived. A secretary of state’s office exposed business licenses, LLC formation documents, EIN and tax identification numbers. At a state election’s bureau, campaign finance records and donation records with donor names and amounts were left visible for anyone to see. In Texas, the Department of Family and Protective Services exposed an unknown number of files, and including details on the children’s living situations, written accounts of their traumatic experiences, case notes and court files, medical records, and personal data like social security numbers, full names, addresses, and dates of birth. Sadly, these examples are represent a small sampling of the total list of affected organizations. There were many others of similar kind, school districts and state-level departments and the like from all over the United States, each with their own litanies of exposed information.

The nature of the data exposed at many of the affected organizations make the cyber-security implications far more serious than mere numbers would suggest. Large-scale breaches like those at Target or Home Depot gather a lot of attention, with victim counts in the tens millions, but the information being stolen is relatively manageable. After all, credit card numbers are easily changed and banks have in place massive and sophisticated systems for detecting and stopping fraudulent activity. At worst, a victim of such breaches suffers a hit on their credit rating and may need to dispute charges with their bank. On the other hand, the information exposed by the vulnerability, and much more dangerous in the wrong hands. Social security numbers, dates of birth, salaries, personal histories – everything a criminal needs to steal another person’s identity and perpetrate crimes in their name. With this kind of information, a criminal can start businesses, take out loans and mortgages, file fraudulent tax returns, and worse. Identity theft can be deeply traumatic, financially ruinous, and can take years to recover from. When you look at it like that, suddenly a victim count in the “low millions” becomes far more terrifying.

The root cause of the vulnerability was almost as troubling as the potential consequences. As it turned out, this particular vulnerability had been found and fixed by Oracle developers two years ago – in 2012. A patch was issued, an advisory was posted on the Oracle blog, and the matter was considered closed. Only, the matter wasn’t closed. Hundreds of major organizations, whether through ignorance or carelessness, failed to see the warning or implement the fix.

It’s at times like these that we realize that cyber-security isn’t always like other jobs. We deal with very serious issues. Sometimes hundreds of thousands of people are at risk, their money and reputations and identities are on the line. Lives can be ruined. How well we do our job is what makes the difference. It’s impossible to say whether we got to this vulnerability before anyone else. If we did, then a very large number of well-known organizations and their millions of customers can breathe a collective sigh of relief. But regardless of the outcome in this one specific case, there’s a sobering moral to this story: this vulnerability sat undetected for years in a product developed by one of the top software companies in the world. Who knows how many other critical vulnerabilities lie dormant, or worse in active use by cybercrime, amongst the millions of unsecured servers?

We’ve said it before, but it’s worth saying again: any organization that handles or stores sensitive information absolutely must take steps to ensure its safety. They have a responsibility to the people whose information you have been entrusted with – to protect their information, and to protect them.