The Security Skeptic

More Thoughts on Damage Mitigation as the New Defense

Earlier this month, Kelly Jackson-Higgins interviewed me for a Dark Reading article, Damage Mitigation as the New Defense.  Kelly also sought perspectives from Richard Bejtlich (Mandiant), Neal Creighton (CounterTack), Bruce Schneier (BT Counterpane), George Kurtz (CrowdStrike), and Tim Rains (Microsoft Trustworthy Computing). In the article, Kelly explores “security’s new reality”, which put simply is a growing fatalistic acceptance that trying to prevent attacks by well funded and determined attackers is futile and that security professionals should shift focus to early detection, rapid mitigation, and damage containment.

The interviewed parties frame out the problem space and identify security technologies that are likely to play prominent roles in the solution space, but all acknowledge that the solution space is, at the moment, incomplete and that its efficacy is unproven. The resulting column gives a good insight into how the security industry may change course or re-invent itself in “era” of the Advanced Persistent Threat.

Editors often collect far more during interviews than they use or quote. With Kelly’s permission, I’m sharing a transcript of our interview as a complement to her column:

Jackson Higgins: “First, I’d love to get your thoughts on the premise that there’s now a philosophy of accepting that ‘you’ve already been breached, so focus on minimizing the damage.’”

Piscitello: “Organizations that are only now coming to the realization that their network perimeters have been compromised are late to the game. Malware ceased being obvious and destructive years ago. The criminal application of collected/exfiltrated data is now such an enormous problem that it's impossible to avoid. The notion that our only recourse is to focus on minimizing the damage, however, troubles me. It's a concession of defeat. I think this is wrong thinking. Would we respond to oil spills by only focusing on minimizing the damage? I'd rather have us adopt a more aggressive strategy where we actively seek out, identify (and where we discover) contain threats, identify root causes, and take measures to eliminate or mitigate these.”

Jackson Higgins: “Where did the security industry/vendors go wrong that led us to this phase? Did we fail as an industry? Why or why not?”

Piscitello: “Given the evidence, it's hard to argue we haven't failed. Plenty of blame and directions to point fingers and most of that is of little benefit.  Organizations have relied too long on traditional security architectures. Unified threat management, policy enforcement points, next generation firewalls are all incarnations of securing perimeters. They are relevant, necessary, but not sufficient primarily because they are mainly static defenses and most (especially the consumer, small business) largely rely on a knowledge based on what we've seen before. We need to be more proactive in looking for anomalous behavior.”

Jackson Higgins: “Where did organizations go wrong in how they deal with security?

Piscitello: “I think organizations have been too passive, placing too much trust in the security we've implemented to notify us when something's wrong with our network. We don't do enough active monitoring and analysis. Most organizations have little insight into what traffic is flowing across their network and through their firewalls. Most organizations don't really understand what the expected traffic mix looks like for their networks: they have no baseline to gauge "healthy and secure" We need this baseline to help us identify what is anomalous/suspicious and then we need to pursue these leads.

Jackson Higgins: “What is the next phase?

Piscitello: “Saying ‘the bad guy's in’ is marketing hype, not security practice. The solution involves more than re-packaging SIEM along with other or new monitoring, analysis and automated intervention tools. These play perhaps a bigger role in the future and it’s a good thing that we’re heightening awareness of how much they are needed. But we should also admit that we want too desperately to rely on automation. Adjustments to the human side – modifying user behavior, reducing administrative error, expanding the staff that defend networks and digital assets as well as their skill sets – are as desperately needed as new tech."

Dilbert on Firewalls: A teaching moment

In the May 24, 2011 comic, Dilbert discusses a firewall failure with boss man:

A teaching moment

Firewalls have two possible failure modes: closed, where no traffic passes through the firewall until the system is restored to a normal operating state, and open, where traffic does pass through while the system is in an error state.

Apparently, Dilbert's is a "fail open" firewall. The consequence of failing open is quite obvious: lots of stuff that would otherwise be blocked enters the network. This is the same effect as a breach of an outer wall of a fortress, (Historically, the assault would be led by a Forlorn Hope rather than mole people from another dimension but the mole people image probably imprints better in the 21st century. Well, technically, what Dilbert calls mole people would properly be called sappers, and they'd have been the cause of the breach. )

Fail open is bad. Of course, fail closed has the same effect as if you lose your connection to the Internet or are the victim of a denial of service attack so it's bad, too.

Another teaching moment

When you rely on one firewall to protect your network, you have a single point of failure in your security perimeter and you face the dilemma of having to choose between fail open or fail closed. If neither of these is a satisfactory solution for your organization (especially true if web presence is a business imperative), consider a high availability firewall deployment. Run two firewalls in tandem. The firewalls must be able to remain in constant touch with each other with a "heartbeat" protocol. If one firewall fails, the other firewall detects this and begins to process traffic according to your policy little or no interruption.

A number of commercial firewalls support high availability as well as redundant Internet conections. Search and ye shall find.

UTM vs NGFW vs WAF in IRC: WTH OMG LMAO

Captured from an IRC channel following RSAC

rS@DuD3: lol my BFFL haz nu ngfw stopz l33t haxxors c0ld grtnrG@l: sw33t rS@DuD3: next gen grtnrG@l: wut dat rS@DuD3: stops web hax grtnrG@l: dats WAF rS@DuD3: no ngfw haz m0r grtnrG@l: like wut rS@DuD3: app policy dude checks SS#s CC#s breaks haxxor crypt0 blocks malware spam u no it rawks grtnrG@l: that's UTM rS@DuD3: u r oot UTM is so l@st year grtnrG@l: itz dat kewl rS@DuD3: srsly grtnrG@l: Ill haz 2 make quadrant 4 it rS@DuD3: ^.^ srs $ 4 u lol grtnrG@l: roflmao always

Parsing hacker chat is almost as difficult as parsing market hype or analyst-speak. The takeaway from the conversation between grtnrg@l and rSaDuD3 is as timeless as the Shakesperean phrase "What's in a name?". The lesser quoted of Juliet's speech in that scene is even more apropos:

"Tis but thy name that is my enemy..."

Don't get distracted by analysts who make or break markets through exhortation or extortion, or  vendors who think it's easier to conjure up  a new market than compete in an existing one. It's entirely irrelevant whether your vendor is seeling you a unified threat management system, next generation firewall, or web application firewall. What really matters is whether the functionality the system offers meets your requirements for securing your application mix.

Whether you are shopping for groceries for a low cholestrol diet or a security features for application protection, you will always be best served by making informed choices. Unfortunately, the Internet can be the best or worst source for conducting research on any product because the "noise to signal" ratio (in this case, the amount of biased comparisons, uncorroborated testing and unqualified sources constitutes noise) is very high.

The best sources for informed choices are 3rd party, independent and trustworthy. As you research appication protection hardware or software, consider the source as well as the signal. What kind of track record does the analyst or consultant have? What kind of testing have they conducted (and are you satisfied that it's adequately scientific or empirical)? Who paid for the testing? Do the testing criteria appear unbiased? It often doesn't take a whole lot of work to weed out the charlatans and the effort can prove the difference when you commit to deployment.

Choke and screen firewall: the modern day concentric castle

Security experts often use terms of warfare to describe Internet security. Some of the terms come from the castle building era. Castle design evolved over time to incorporate multiple lines of defenses to defeat attacks and resist sieges. These principles are included in Internet security under the rubric

Defenses in Depth

Castles protected people, their property and possessions against attacks from rival lords and invading armies. Initially, castle fortifications consisted of a single line of defense. An earthwork called a motte protected a keep. The motte surrounded the area to be defended and was frequently located on a hilltop.	Photo by jayneandd
Durham County Council	The Motte The motte is similar to the field fortifications a Roman legion built around each encampment during a campaign. Legionnaires would dig a ditch  around their camp and pile the dirt behind the trench. This effectively doubled the height of the wall. Mottes were in many cases much larger and wider. The ditch was also often dug deeper and wider than a Romain camp entrenchment.
Castle designers were engineers. Like modern engineers, they learned from failures as often as successes. As siege tactics improved and the motte became too regularly breached, engineers determined that multiple lines of defense would offer more protection than any single wall, no matter how formidable that wall might be. They added a wall within the outer wall.	HistoryOnTheNet.com
Wikipedia	Concentric Castles Concentric castles relied on the outer wall to blunt the bulk of the attack. The inner wall was used to bolster the defenses of the outer wall and to defend against any attackers who managed to penetrate the outer wall. Checkpoints at gateways were used at both walls to permit access to residents and visitors and deny access to intruders.

Firewall Evolution

The early evolution of firewalls tracks closely with castle defenses. Initially, a single perimeter defense, the Internet firewall, protected an enterprise’s clients and servers against network attacks. This arrangement proved effective while organizations hosted applications primarily or exclusively for users behind the Internet firewall.

As Internet presence became a business imperative, organizations had to find ways to separate applications intended for internal users from external visitors or customers. Look carefully at concentric wall design and you can see that firewalls could be similarly deployed. An outer, Internet firewall would blunt or screen attack traffic and serve as a checkpoint to allow traffic to public facing applications intended for external visitors. These applications would be situated behind the Internet firewall. An inner firewall would serve as a choke point to ensure that no traffic intended for public facing applications gained access to the organization’s internal (private) applications and hosts.

Screen and Choke Firewall

The screen and choke firewall arrangement, like its concentric castle predecessor, proved effective. Evolving business needs (intranets, Wikis, secure application access), technology (WLANs) and attack targets (client as well as web applications and services) have forced us to think beyond the original deployment, the choke and screen firewall arrangement is remains popular. Today, web application firewalls (WAPs), firewall load balancers (FWLBs), and interdepartmental firewalls serve as the screening firewall in many screen and choke deployments.

5 Best Firewall Practices and How to Implement Them

Linda Musthaler's Network World article identifies a Top 5 best practices for firewall administrators. From the article, these are:
 

1. Document all firewall rule changes
2. Install all access rules with minimal access rights
3. Verify every firewall change against compliance policies and change requests
4. Remove unused rules from the firewall rule bases when services are decommissioned
5. Perform a complete firewall rule review at least twice per year

These are generally good advice.
 
Linda’s article tells you what practices to implement, but not how. I’ve seen “how” discussed many times on firewall mailing lists, and I’ve described how in earlier articles, but repetition is the key to learning, so let's consider how to implement these five practices again.
 
How do I document firewall rule changes? The answer is not quite as simple as recording rule changes in written or electronic forms. Firewall rule changes, a.k.a. Firewall policy life cycle management, should match security policy changes. Documenting security policy and firewall rule change are both elements of a change control process or workflow. As I mention in earlier articles (Blog 511, Blog 461), every firewall rule change must be proposed and approved as a security policy change. The change must then be implemented (entered into the firewall configuration), tested (confirmed by verifying that the traffic is allowed or blocked as intended by the policy). Log firewall events (blocked and allowed traffic) regularly. Periodically audit your firewall log files to assert that the rules you assert in your security policy are being enforced.
 
Install all access rules with minimal access rights. I’ve covered the outbound traffic side of this critical aspect of firewall rule composition in my article, Firewall Best Practices – Egress Traffic Filtering.  Configuring firewalls with minimal inbound access controls follows essentially the same logic as outbound: decide upon a security policy regarding the services you will allow, then implement firewall rule changes to allow traffic only to those services that you intend to make accessible. Where appropriate or practical, apply granular constraints as well (i.e, the addresses or users who are permitted access and the directions of traffic flow between security policy boundaries such as guest, DMZ, internal, and public).
 
Verify every firewall change against compliance policies and change requests. Matters of compliance must be reflected in your security policy. Your change control process must include checks that ensure you do not alter a firewall rule that will cause your security enforcement to fall out of compliance. With these measures in place, your firewall life cycle management should continue to serve your organization well.

Remove unused rules from the firewall rule bases when services are decommissioned. Firewall policy life cycle management covers adds, drops, and changes. Decommissioning a service can be a policy action. A change in your regulatory obligations might require that you migrate from one service to another, or your assessment team may conclude that continuing to support a certain service exposes you to unacceptable risk. Decommissioning a service may be an end of life action. Managing unused rules is essentially no different from managing new rules or revising existing rules. The basic mechanisms are covered in Skeptic Blog 511 and Blog 461. 

Perform a complete firewall rule review at least twice per year. Firewall rule review (auditing) ought to be done more regularly than twice per year. Firewall rule review is an ongoing activity. Confirm that the firewall configuration, deployment, and operation enforce the intended security policy and cannot be circumvented. This often involves more than firewall testing and may include modem “war-dialing”; managing WLAN frequency, range and security configuration; and testing to confirm your virtual and physical LAN configurations separate internal, DMZ, and guest subnets according to the policy you intend to enforce. This is a non-trivial activity. Consider whether your review is best accomplished by means of a bi-annual audit or by combination of auditing and real time monitoring, and whether you may be better served by outsourcing this activity to a trusted provider.

Thanks to Linda for identifying a Top 5 best firewall practices to implement and providing me an opportunity to write again on a favorite subject.