Shields Down Security Testing

Should security testing (vulnerability scanning, web app pentesting, etc.) be performed through a full protection stack (firewall, IPS, WAF, email filter, etc.) or not?

Some may answer with an initial gut reaction of, “Of course. It should be through whatever protection solutions are typically in place so it will most accurately reflect what an attacker will face”. Although that does make some logical sense, that viewpoint misses the purpose and value of security testing.

If you share this viewpoint, you are not alone. More times that not, my clients (initially) have that same view. I hope this article not only shows you how this viewpoint is wrong, but also shows that it is actually in total inverse to all things reasonable.

Purpose

When a consultant is evaluating your infrastructure or application during a security test, it’s not some sort of game to see if your defenses are better than their hacker skills. That’s called a Capture-The-Flag (CTF) training exercise. Why would you pay to train the consultant? That makes no sense. The consultant’s job is to find as many weaknesses in your systems as possible, and then help you secure them. The more you join forces, the better the outcome.

Value

When a client hires PEN Consultants to perform security testing, it’s rarely with an unlimited budget or time constraint. Okay, actually, it is never with an unlimited budget or unlimited time. The client ALWAYS wants the best possible return on investment (ROI). The faster I can identify the vulnerabilities, the cheaper it is for the client. This gets more pronounced when you hire someone to perform a vulnerability scan, but do not grant an exception in your defenses for the scanner to work effectively and efficiently. Automated vulnerability scans are already one of the lowest value security tests performed. Limiting its effectiveness nearly wipes out all value.

Who’s your enemy?

Who are you trying to defend against? The script kiddie that fires up Metasploit? If that is all you’re concerned with, leave your defenses up and just do a basic vulnerability scan to see what exposures you have…if any. If you are concerned with organized crime actors (nearly unlimited budget), hacktivist (lots of time to spend), insider threat (tons of access), nation states (all of the above), etc., then security testing will likely not give you the ROI you are looking for if you don’t make some concessions with testing.

Real Example

Here’s a concrete example. SQL Injection (SQLi) through a Web Application Firewall (WAF). Is it possible for an unauthenticated tester to find a somewhat obscure SQLi vulnerability in your web app with a WAF blocking and rate limiting nearly every request? Sure. But, at what cost? In other words, how many billable hours will it take to find that vulnerability compared to performing the same testing without WAF interference? I’ll take it a step further…how about compared to the tester being given testing credentials, full access to source code, backend command line access, etc.? The difference can be days or weeks versus seconds or minutes. Which one are you willing to pay for?

How sure are you an attacker will not find that same vulnerability given enough time, resources, or maybe just luck? What happens when your WAF fails? Fails to protect against attack X or maybe just croaks and fails open? At the end of the day, there IS a vulnerability in your web app. Why would you not want to find it as quickly, and cheaply, as possible? Clearly the more access one is given, the faster vulnerabilities will be discovered, thus reducing man-hours, and in turn reducing cost, which increases ROI.

What is being tested?

It’s worth asking, what are you having tested? If you are having your web app tested (for example), the WAF is not part of your web app. If it is, you have a lot bigger problems that need to be addressed. In fact, in most cases, the WAF is 3rd-party. Maybe even cloud hosted. Unless the tester has authorization from the WAF vendor, they could actually get into some legal trouble by attempting to circumvent it’s controls. I’ve tested WAFs before – it is a very different test than a web app pentest. If you want both tested, the testing should be split up.

Applicable Everywhere

This concept is not limited to web app testing through a WAF. This same concept applies to network vulnerability scanning and penetration testing (through firewalls, IDS/IPSs, WAFs, email filters, etc.), red teaming engagements (custom detections), wireless assessments (restricting physical access), social engineering engagements (names and contact info), etc. As a general rule, the closer one gets to full unmitigated access, the better the ROI.

What are others saying?

Is this just the opinion of a few people in the industry? Competitor example: https://contextualsecurity.com/2020/10/why-are-you-asking-to-be-whitelisted/.

Maybe we’re just lazy and don’t want to work as hard? A growing number of methodologies and standards require, or highly encourage, protection bypasses for scanning and testing.

The PCI DSS standard, for example, states, “In order to ensure that reliable scans can be conducted, the scan solution must be allowed to perform scanning without interference from active protection systems”. If it is not resolved, the vendor providing the testing must report it as a “failed scan”. Source: https://www.pcisecuritystandards.org/documents/ASV_Program_Guide_v3.0.pdf

I don’t want to bash the PCI DSS standard (it has a lot of strengths), but let’s be real. Even this standard is seen by most as bare minimum or basic. Imagine how much more important this topic becomes if you’re wanting advanced security and maturity.

Shields up?

Is it ever advisable to perform testing with your “shields up”? I say, yes!. But, it should always be in addition to full white-box, non-interference testing (as close as possible). Here’s what I feel is the most valuable approach. Grant the tester full, unmitigated access to all documentation, systems, accesses, protection bypasses, etc. Allow them to perform their testing and produce a list of findings. Now, put your “shields up” (and/or they come at you through a “normal” path) and let them see what is still exploitable. In some cases, new vulnerabilities may even show up! You now have your first cut at ranking the vulnerabilities in terms of risk and likelihood of attack, while at the same time, drastically reducing the hours/ costs of the testing. It’s a win-win for you!

Conclusion

Bottom line, the security testing consultant should be seen as your ally, not an adversary. How will you treat yours?

Shields Up == Adversary
Shields Down == Ally