Defcon: The security penetration testing quagmire

LAS VEGAS -- The relationship between CISOs and security penetration testers is anything but clear-cut and raises ethical issues for both parties, a Defcon crowd heard from a former CISO.

Whether penetration testers should come in looking for the place where they can spectacularly break into the network or instead assess it clinically and point out potential vulnerabilities is the big decision CISOs have to make, says a CISO-turned penetration tester identified only as Shrdlu.
SELF TESTING: Metasploit 4.0 sets the stage for mass penetration testing
And the choice is the CISO's, she says, because the CISO is paying the bills. "It's not about your satisfaction," she told a crowd that included many penetration testers.
She says that often penetration tests are mandated by regulations, and the network must pass in order to comply. In that case, she prefers a light touch by the tester, telling her informally about technical security shortcomings but not including them in the formal report that goes to the compliance auditor. "Tell me verbally what's wrong and don't write it down," she says.
For example, if the help desk prompts users that they can't login because they've gotten their username wrong, that's a violation. But, she says, doing so saves a lot of help desk and employee time and is a good risk-business tradeoff. She doesn't consider the practice a major breach of good practice.
"There are things I do on purpose and are not high-impact," she says.
That drew protests from audience members, one of whom said it was unethical not to include security problems he finds and is possibly illegal because it is essentially lying to compliance auditors. "It sounds like avoiding regulatory scrutiny," he said.
"That's very fair," Shrdlu responded. But she says most compliance regulations are vague enough that reports can be vague as well, indicating an unspecified problem without detailing it. She says penetration testers can prepare two reports, one for her use and a second for the auditor.
She says these dual reports are useful for public organizations where the reports may become public record. The vague one that doesn't detail specific problems can be the public version and the detailed one can be called a working document and so avoid public scrutiny.
Another audience member said her approach could cause problems for penetration testers if a problem found but not mentioned is exploited. The tester would have no documentation that he'd done his job properly. Again, she fell back on the dual report, where the vague reference to the problem would provide cover for the penetration tester.
She says she's found frustration with penetration testers who haven't worked in corporate security and had to shore up problems testers have found. Often the problems present less of a risk to the organization than the time it would take to fix them is worth, she says. "I'm impatient with penetration testers that have never been on the fixing side," she says. They need to be more aware of the big impact and the business impact of remedies. "There are things that just plain aren't going to be fixed."
She says she doesn't want a broad penetration test that lists all the ills her network harbors. Rather, she wants to hear about the significant ones that she doesn't know about yet so she can create an internal list of what she wants to fix. "Don't tell me what I already know," Shrdlu says.