Red Teaming vs Penetration Testing vs Vulnerability Scanning vs Vulnerability Assessments

What is the difference between “Vulnerability Scanning”, “Vulnerability Assessments”, “Penetration Testing”, and “Red Teaming”? What do they mean? Which one do you need performed on your systems?

The industry is very opinionated on the definitions of these four terms. This article will share our view of the similarities and differences between vulnerability scanning (vuln scan), vulnerability assessments (vuln assmnt), penetration testing (pentest), and red teaming. This article is not meant to “argue” with others’ opinions, but to ensure our clients and readers know what we mean when we use one of these terms in our marketing material, website, phone calls, etc.

BLUF (Bottom Line Up Front)

The most concise way one can compare/contrast these four terms is to first group vuln scans, vuln assmnts, and pentests into a category that we will call “technical focused”:

• Vuln scan – This is nearly 100% automated testing. Setup is somewhat manual, and, in some cases, some initial manual browsing is needed to “prime” the scanner(s).
• Vuln assmnt – This is a vuln scan, but it also has manual verification (i.e. exploitation) of each verified vulnerability. It could be said that the tester is “penetrating” the system. Starting to see the confusion?
• Pentest – This often includes all of the above, but it also includes semi-automated and manual testing to find vulnerabilities and attack vectors not otherwise detectable by automated tools.

The difference between technical focused tests and red teaming:

• Technical-focused testing (vuln scans, vuln assmnts, pentests)
  • Primarily tests (usually exclusively) for technical control gaps and misconfigurations
  • Attack prevention – firewalls, AV, WAFs, patched and securely configured systems, strong web app security, etc.
• Red Teaming
  • Can include the technical part (usually focused, not broad), but also tests for gaps in detection, response, people, and process, etc.
  • Includes alerting processes (i.e commercial and custom detections), human gullibility (i.e. social engineering), human response (i.e. DFIR), etc.

Different Methodologies

These four terms are different. With this one statement we have, believe it or not, alienated some readers who refer to some of these scans and tests synonymously. Furthermore, we call these “security testing methodologies”. They are ways to look for vulnerabilities, gaps, weaknesses, and attack vectors that could be used to compromise your systems, users, and data.

The first three methodologies (vuln scans, vuln assmnts, pentests) build on one another, whereas Red Teaming is a whole different methodology, as shown above. With this overview in mind, let’s drill into the details of each.

Vulnerability Scanning

Vulnerability Scanning, or vuln scanning for short, could potentially be leveraged by any of the other methodologies. Vuln scanning is nothing more than running an automated vulnerability scanner(s) against a system, network, application, source code, web service, etc. There is often some manual work to set things up, verify things are working correctly, or in some cases, prime the scanner with initial data (ex. manually browsing a website through a proxy), but outside of that, it is automated.

Time: At least a 21-day lead time, 45 days if you want to avoid a “rushed” fee being placed on testing. At a minimum, this is going to take at least 2-3 weeks for even a small scope. There are certain tests that cannot be sped up without causing account locks, overwhelming your resources, producing false negatives (i.e. missing findings), etc. Additionally, we often cannot start the day the client is ready for testing. Our testers are actively working other engagements, so it may take a few weeks to get the testing scheduled and in their queue once we have all of the deliverables from the client.

Deliverable: A raw vuln scan report of potential vulnerabilities which includes a brief description of the vulnerability, how an attacker may leverage it, and general guidance on how to mitigate said vulnerability. This raw report may be generated for network vuln scans from tools such as Nessus, OpenVAS, etc., or for a web app vuln scan from tools such as Burp, ZAP, etc. There is no manual analysis of scanner findings or anything beyond that. Although there may be risk severities assigned to each finding, that rating does not take into consideration the client environment and other potential mitigating controls. Unless the scanner report includes it, there are no recommendations given for fixing the problem. The recommendations certainly would not be tailored to the client’s environment.

Cost: Because this is almost fully automated, this testing methodology will typically only cost a few hundred dollars.

Benefits: Price. The price is the main benefit. It costs almost nothing (relativity speaking) to have this performed. Any true positives that come out of this would likely be considered an imminent risk, so you would want to know about it. Typically, these scans will help find the common and easy-to-discover technical gaps to mitigate, which will protect you against the “drive-by” attacks. Automated scanning will do a pretty decent job at finding the same vulnerabilities automated attack tools find.

Challenges: Depending on the scanner(s) used, there can be a high number of false positives. Vuln scanners are typically limited to finding the “low hanging fruit” – things that are trivial to find by a tester or attacker. They are not as likely to find things that require a manual analysis…even just a few minutes of manual analysis. Also, technical skills (such as a developer, sysAdmin, netAdmin, etc.) will be needed in order to understand the report and remediate the vulnerabilities.

Bottom line: If you are on a very tight budget and can only protect yourself against the “drive-bys,” consider this testing.

More Information: For a breakdown of sample pricing and example reports, see ”basic” services under “vulnerability scanning” at https://penconsultants.com/services.

Vulnerability Assessments

We define this as a vuln scan with manual verification and unique risk assessment. With this type of testing, one would perform the vulnerability scan(s), then manually verify (i.e exploit) the vulnerability to confirm it is a true positive. In some cases, you may not be able to verify – for example, a DoS (denial of service) vulnerability in a production environment. In those instances, one may create a test/cloned environment or make an educated guess as to the validity of the findings.

Time: At least a 30-day lead time, 60 days if you want to avoid a “rushed” fee being placed on testing. In addition to the time requirements mentioned for vulnerability scans (see above), this testing requires manual verification/exploitation, as well as the creation of a customized report.

Deliverable: The client will receive a custom Findings and Recommendations Report in which there is great certainty that every listed vulnerability exists and can be leveraged by an attacker. There is a manual risk calculation based on the specifics of the finding and the client environment. The mitigation recommendations are tailored to each client’s unique environment, and they are meaningfully prioritized and immediately actionable.

Cost: Because of the manual work involved in verifying each potential vulnerability and writing custom recommendations for you, this is more pricey than vuln scanning. Instead of a few hundred dollars, as with the vuln scan, this will be at least a few thousand dollars.

Benefits: “99%” true positive findings will ensure your organization is not having to chase down findings that end up being false positive. The tester is performing a portion of the labor for you and assisting you by giving tailored remediation recommendations based on their experience.

Challenges: Since vuln scanning is the sole foundation of this assessment, you are still only protecting yourself against the “drive-bys”. Although there is analysis based on scanner results, there is rarely any additional manual analysis that goes into this testing.

Bottom line: If you do not have the time, resources, or expertise to “verify” scanner findings, you will want at least this level of testing. If you lack the budget for a full pentest (described next), this is a good middle ground between vuln scanning and pentesting. In addition to receiving the boilerplate recommendations from the vendor/industry standard, the tester gives specific guidance to help streamline the remediation process.

More Information: For a breakdown of sample pricing and example reports, see “standard” services under “vulnerability assessments” at https://penconsultants.com/services.

Penetration Testing

Pentesting almost always involves the same vulnerability scanning and assessment as mentioned above. It not only exploits the vulnerability, but it also leverages the access obtained to further pivot deeper into the environment in order to discover additional vulnerabilities and determine the exposure that would occur. However, that is only a portion of this testing, and represents just a fraction of the time spent by the tester. The bulk of the tester’s time is spent on a great deal of manual analysis in which no scanner can perform. This testing will also follow an industry-standard testing methodology such as the OWASP Testing Guide (for web app testing), the Penetration Testing Execution Standard (for network testing), SANSs|DISA’s wireless audit (for wireless testing), etc. There may still be tools and scripts used for many of the tasks, but they require enough manual setup/ operation/ interpretation that prevents full automation. Pentesting may or may not include the following, or may only include a minimal/best effort: wireless assessments (typically a separate service), social engineering attacks (more red teaming), web app testing (typically a separate service), etc.

Time: At least a 45-day lead time, 60-75 days (min) if you want to avoid a “rushed” fee being placed on testing. In addition to the time requirements mentioned for vulnerability scans and assessments (see above), this testing requires extensive manual testing. Anyone that says they can do a pentest in <30 days is not conducting testing as thorough as we are, and are likely just performing an automated vulnerability scan. For example, a password spray takes ~30 days to run safely and cannot be sped up (would lock out accounts). That doesn’t include the discovery, prep, and setup time to start it, or the post-exploitation time after it is complete. We could give other examples, but that is usually the easiest one to understand.

Deliverable: The client will receive a custom Findings and Recommendations Report in which there is great certainty that every listed vulnerability and attack vector exists and can be leveraged by an attacker. There is a manual risk calculation based on the specifics of the finding and the client environment. The mitigation recommendation(s) are tailored to each client’s unique environment, and the recommendations are meaningfully prioritized and immediately actionable.

Cost: Pentesting is the most costly of the three because of the large number of hours needed to fully perform. Even for smaller organizations, this will start at around $10,000 and quickly escalate based on scope. On that note, if anyone ever offers you a “penetration test” for less than this, it is not a pentest…guaranteed.

Benefits: Like the other two services mentioned thus far, you will be aware of, and be able to, mitigate against “drive-by” attacks. In addition, you will also have a measure of visibility and opportunity to protect against more targeted and advanced attacks aimed at your organization.

Challenges: A pentest’s quality depends largely on who is testing (i.e. their skill set), what methodology they follow (if any at all), and the cooperation of the client (i.e. white box testing vs black box testing).

Bottom line: If you care about validating the technical security stance of your organization against more advanced attackers, pentesting is hands-down better than vuln scanning/assessments. Because of the above-mentioned challenges, we recommend hiring multiple pentesters (if you can afford it), and/or swapping out those that are not producing the quantity and quality of findings as another testing company. We also recommend treating your tester like part of your team (access and information) and not an attacker (blind testing).

More Information: For a breakdown of sample pricing and example reports, see “standard” services under “penetration testing” at https://penconsultants.com/services.

Red Teaming

The term Red Teaming is probably the most industry-debated term in this article. Red Teaming is entirely different in terms of scope, goals, and methodology as compared to the other three already discussed. The first three testing methodologies mentioned are all TECHNICAL evaluations. They are looking for technical ways to breach your systems and gain access to your data. Red Teaming, however, operates much more like a true attacker by leveraging any and all means possible (within safety and legal limits, of course) to achieve the objective.

In addition to testing technical controls, Red Teaming also tests an organization’s detection, response, people, and process. In other words, you discover which attacks succeeded (based on technical control failures) AND which ones were not detected. Depending on the engagement and scope, Red Teaming often also tests human gullibility (i.e. social engineering), human response (i.e. DFIR), physical access controls, etc.

As mentioned above, Red Teaming operates much more like a true attacker, simulating attackers and/or testing attacker techniques, tactics, and procedures (TTP). Red Teaming is divided into two separate categories: adversary simulation (objective-driven, stealthy, adversarial simulation) and technique simulation (testing individual attacker techniques in partnership with your blue team…aka “purple teaming”).

Time: There is no typical or average timeline for this. It could be as short as a few hours if we’re just jumping on a screen share with you and helping validate a detection (i.e. technique simulation), or could last an entire year (advanced adversary simulation).

Deliverable: For adversary simulation, you will receive a Findings and Recommendations Report detailing avenues in which the tester was able to breach your systems/data, along with recommendations tailored to your organization. There is also opportunity for your response team to exercise the incident response process in reaction to our attack(s). For technique simulation, much of the “findings and mitigations” take place real-time as the tester is working directly and constantly with your blue team. After the engagement, you will receive a report detailing all gaps discovered, those that have already been mitigated / verified, and those that are outstanding.

Cost: Red Teaming has the most varied cost of any service. It could run a few hundred dollars per month, to 10’s of thousands of dollars. It not only depends on the type of engagement (adversary simulation and technique simulation), but also on the scope of the engagement. Adversary simulation is usually best to run under a set-fee contracted price, whereas technique simulation is typically best billed against a retainer.

Benefits: Red Teaming, in general, gives the best ROI. Of the two types of Red Teaming – adversary simulation and technique simulation – technique simulation typically has the best ROI. Even if your organization has poor technical controls and has not invested money into blue team defenses (detections, people, processes, etc.), this testing may help highlight the gaps and the need to address them.

Challenges: The challenge with Red Teaming, and all testing to a certain degree, is that you must be willing to truly partner with the tester. The most effective and valuable red team knows everything your blue team knows – every detection, mitigation, etc. For example, performing technique simulation requires 100% transparency to be effective. There is no such thing as “black box red teaming,” as there is no “team” with “black box”. It is important to note that Red Teaming does not typically give you the breadth of coverage that a pentest will give you. You often will need both services performed.

Bottom line: Red Teaming tests the organization’s ability to prevent, detect and respond to non-technical and technical controls and processes. We would recommend a full scope pentest at least annually, followed by quarterly Red Teaming engagements that focus on your areas of greatest risk.

More Information: For more information about Red Teaming and an example report, see “Red Teaming” at https://penconsultants.com/services.

Conclusion

Security testing is a vital part of any security program. Selecting the best service(s) to meet your unique needs can be challenging sometimes. Our hope is this article has helped you to determine which testing service best meets your needs. Feel free to reach out to PEN Consultants for a free, no-obligation consultation to discuss your unique needs and how we may be able to help – https://penconsultants.com/contact.