Gray Box vs. Black Box vs. White Box Testing

The balance of white/black box security testing is a decision you will ultimately make based on your budget, risk concerns, internal policies, and possibly compliance requirements. This article is intended to help a client think through the benefits of white box testing and the downsides to full black box testing, as well as provide several real-world examples to demonstrate the points.

Note: The primary purpose of this article is to inform our clients of the differences between the three types of testing, but hopefully it will provide value to others as well.

Bottom line: White box testing is always going to give you the best ROI. A security tester’s objective is to help you find your weaknesses and address them. Additionally, you may be required to do so anyway, depending on various compliance standards you may fall under.

Gray Box Testing

• somewhere in the middle of black box and white box testing
• all penetration testing we perform would technically be considered gray box
• Note: We require, at minimum, IP addresses and domain names that are in scope for legal purposes. Red team engagements can sometimes delay that knowledge until later into testing, but, at some point, it has to be given/verified prior to any “hacking” activity.

Black Box Testing

• testing performed “just like the attacker”, meaning there is (typically) no access or assistance given or privileged information that is known ahead of time prior to testing
• pure black box testing either takes more time (and therefore money), or thoroughness has to be reduced, or both, in many cases; as a general rule, black box testing adds 50% to testing costs with one-tenth the thoroughness
• we would discourage you from wasting your money on pure black box testing

White Box Testing

• the tester has been granted full access to all systems/areas being tested and “knows what you know”
• testing is more thorough and the mitigation recommendations we give are more specific to your environment
• campus access badge, computer account (user level and admin level) and access to every detail of your environment (ex. technical documents, blueprints, etc.).

Bottom Line

White box testing is always going to give you the best ROI. A security tester’s objective is to help you find your weaknesses and address them. Being granted access to all portions of your environment and information greatly reduces the time spent on the recon and enumeration phases. Furthermore, every stage of testing benefits, as testing outcomes are much easier to determine and the mitigation recommendations we give are more specific to your environment.

A growing number of methodologies and standards require, or highly encourage, a gray or white box approach to testing.

• The PCI DSS standard, for example, states, “[White box] assessments yield more accurate results and provide a more comprehensive test of the security posture of the environment than a pure black-box assessment. Performing black-box assessment, when the entity provides no details of the target systems prior to the start of the test, may require more time, money, and resources for the deliverables to meet the requirements of PCI DSS”. Source: https://listings.pcisecuritystandards.org/documents/Penetration-Testing-Guidance-v1_1.pdf
• NIST SP 800-115 states: White box is “more efficient and cost-effective for finding security defects in custom applications than black box techniques”. Source: https://nvlpubs.nist.gov/nistpubs/legacy/sp/nistspecialpublication800-115.pdf

Common Oppositions to White Box Testing

• By granting the tester the access/information, they bypass the recon and enumeration phases. PEN Consultants does not bypass any phase or test when granted a higher level of access. 100% of the same testing occurs. Verification effort is what is greatly reduced.
• White box testing is testing insider threat as opposed to outsider threat. The two are actually independent of one another. An outsider’s perspective engagement will primarily focus on what an external threat is capable of, regardless of the level of inside knowledge we have ahead of time.
• An outside attacker will not have this access and information. How is this a fair assessment? This is where the bulk of the ROI is realized. An advanced threat will spend months (sometimes years) performing recon and testing various things in and against your environment. We want to ensure you do not have to pay for months/years of testing when days/weeks of testing can perfectly simulate what could have been done with an extended period of time and budget. The risk levels presented to you in the Findings and Recommendations Report for each finding take into account the difficulty for an attacker to come across a certain piece of information…even if we were given it directly. How sure are you an attacker will not find the information given enough time, resources, or maybe just luck?
• We want to know about vulnerabilities that are exploitable by an external unauthenticated attacker (or with a low-privilege level), not one with a privileged level of access. Mature organizations will want to be concerned with both. Without authenticated & privileged access, only one role can be tested, and only as thoroughly as time allows. Additionally, you have to consider real attackers often target privileged users (ex. phishing) and gain this level of access prior to pivoting/fully breaching your network. Having authenticated & privileged access during testing allows us to test for weaknesses that can be exploited by privileged, unprivileged, AND unauthenticated users. Another way to think of it is to imagine asking a mechanic to thoroughly inspect a car you are about to purchase, but only giving them access to look from outside/under the car, and not the ability to open the hood, look inside the cabin, provide them the keys, etc. If you want to know about all of the potential risks, you have to give all of the access.
• I don’t trust someone outside our organization with that level of information and access. Whether you grant access/information ahead of time or not, tester will gain access to much, if not all, of it anyway before the testing is over. When dealing with security testing vendors, you should thoroughly vet any organization that will be given (or gain) access to your privileged information. Read more about that here: https://penconsultants.com/compare

Real-world Examples:

• Post-authentication testing. Without credentials to your web application, VPN or remote access service, email portal, etc. we can typically only test the login page, but nothing beyond. There are a number of scenarios and tests that should be performed against any functionality post-login. An attacker will likely have this level of access (ex. compromised credentials, insider threat, etc.). Unless you provide credentials, those scenarios and tests cannot be performed, thus drastically reducing overall testing thoroughness and leaving you highly suspectable to the associated risks.
• Port scans and service detections are simplified when the testing includes access to netstat and tasklist/ps on the back-end.
• Lockout policy. We can often pick an account and count the number of times it takes for the account to become locked. Then, we would monitor how long it takes for the account to unlock. Repeating this process with timeouts between each attempt, we can determine the window of time in which a certain number of failed logouts must occur to lock the account. When you provide this policy, it saves hours of time (and cost).
• The ability to take a full inventory of the environment (ex. process, service, and file permissions) can only be achieved with privileged credentials. Otherwise, only a subset can be achieved.
• Old, stale DNS entries could allow an attacker to take over control of one of your subdomains. The black box method to find these involve dictionary lists of millions of common words and days/weeks to enumerate. With white box, however, you simply provide us an export from your DNS server/services so we can review them.
• Injection type attacks (ex. SQLi, XXE, etc.), are almost always blind to a certain degree. Depending on the degree of blindness, the white box approach can greatly speed up verification of suspected injection vulnerabilities.
• Sensitive content in web servers (ex. backup files) can be found by brute force means over the course of days/weeks and can expose things like credentials that could give access to the web server. This days/weeks time frame can be cut down to seconds if we have back-end access and can search the file system directly. The severity and impact to your web app are precisely the same, but the black box testing method is exponentially more expensive.
• Having access to and the ability to scan source code of an application is far superior than fuzzing the running application. The former is a scientific and repeatable process; the latter is bordering on luck to find vulnerabilities. Note: Both are complimentary, so both are recommended.
• For phishing/social engineering engagements, the collection and verification of the target email address, phone numbers, titles, etc. can be time-consuming, and therefore, costly. A client should ask themselves what their primary concern is – attacker generating a list of valid contacts vs. their users’ ability to distinguish between legit/evil messages – and provide the piece that is not of concern. In other words, why pay for the tester to generate and verify a list of contacts if your reaction to it will be, “So what, that information is mostly public anyway.”
• Another piece of a phishing/social engineering engagement that can contribute significantly to the costs is determining what your workstation load(s) looks like. The client providing a breakdown of client workstations (Win, Mac, Linux) and testing payloads for us prior to a phishing campaign saves us from having to carry out one (or more) recon campaigns to acquire the information on our own. As with an attacker, we certainly can go through all of the steps, but it really doesn’t provide you any value…and can double, or even triple, the cost for that service.
• Having a client-provided floor plan prior to a physical social engineering engagement saves us the time (and you the costs) of acquiring one through black box means (ex. surveillance ahead of time).

A real-world testing scenario

One of PEN Consultants’ developed pretext for a physical social engineering engagement is as follows:

1. Disable the air conditioner serving the IT closet the night before engagement
2. Wait for the phone call to the HVAC service company and intercept it
3. Impersonate a HVAC technician to gain physical access
4. Tap into the corporate network and/or other actions

Although this seems like a straight forward assessment, and is certainly real-world, there are a number of factors that can greatly influence the pricing – a few thousand dollars, to many thousands of dollars. Here are a few of the areas in which a client’s provided information/help can influence the pricing:

• Floor Plan
  • White box (no additional costs): Client provides the floor plan showing the location of the IT closet and schedules a site visit prior to testing (ex. the evening before).
  • Black box (extra costs, no ROI): Find the floor plan online, acquire through the building contractor, recon and covert site visits in the days/weeks before, etc.
• Name of the HVAC service company
  • White box (no additional costs): Client provides it
  • Black box (extra costs, no ROI): Weeks before, we disable an outdoor condenser and surveil the area until the HVAC service company arrives
• Location of the condenser (outside) that serves the IT closet
  • White box (no additional costs): Client provides it
  • Black box (extra costs, little/no ROI): Weeks before, we surveil all condensers and determine which serves the IT closet – the IT closet is generally the largest heat source in the building, and therefore, that unit will run most frequently
• Means to intercept the communication channel between the target and HVAC company
  • White box (no additional costs): Client’s maintenance employee (assuming there is one) can intercept and forward the call, or client could convince the HVAC company’s dispatcher to intercept and forward the call
  • Black box (extra costs, little/no ROI): Ex. We tap the Client’s phone line at the demarc, tap the HVAC company’s phone line at their demarc (real-world, but no legal way to perform this), etc.

Featured image is a derivative work from the following images: OpenClipart-Vectors @ https://pixabay.com/vectors/cardboard-box-cardboard-box-moving-147605/