Building a Security Testing Business

I am often asked, “How did you get started with your security testing business?” “What are some lessons learned?” “What are your current challenges?” I have been asked enough times that I decided to post my thoughts in blog format.

How did you decide it was the right time to take the step from W-2 employment to self-employment?

It was God…plain as day. Here’s the long version…

In 2009/2010, while working at NSA, I started volunteering my time to provide IT support to a few local churches and private schools. As that side hobby/community service grew, we came up with the name “PEN Consultants” in 2014, and have been doing business under that name ever since. In addition to serving the community with my spare time, I was very much involved in “research,” AKA hacking. Looking back, there are plenty of activities I would do differently now. More on that progression here: https://penconsultants.com/personalEthics

When I knew it was time to leave NSA (that’s a story for another time), we weren’t quite sure if it was good timing to pursue PEN Consultants as a fulltime job, or if it would ever be a good time. I was hesitant, so I applied to work for USAA in 2014 until we knew the direction we should go and the right timing. After receiving a CJO from USAA, and during the background check process, I disclosed the fact that I had this side company and was heavily involved in research activities on personal time. My soon-to-be-supervisor described that background check process as one of the lengthiest he could remember, due to the ethics portion of the background check. The decision finally came back, I had to agree to the following terms:

• Could not perform side work on company time
• Could not use company resources when performing the side work
• Could not utilize any confidential, proprietary, or otherwise protected, information obtained exclusively from USAA outside of USAA
• Could not “recruit” other USAA employees for the work
• Could not perform work for other FIs (financial institutions)

Other than a small disagreement with that last stipulation, those were all no-brainers and things I would have avoided anyway. It was easy to agree to those terms, so I soon found myself working for the best FI in the industry, USAA! As time went on, USAA became less friendly to employees performing research and consulting type activities on personal time. Examples: USAA demanded patents for things done on personal time, raised strong objection to research performed against a vendor product they use (Note: USAA is large…they use all the well-known vendors), required prepub review for blogs written on personal time based on personal research, wanted to limit the hours of work outside of USAA as to not impact on-the-clock performance, etc.

To be fair to them, part of these changes were likely due to an ignorance on their part (initially agreeing to things they had no prior working knowledge of), increased regulation, being such a large company with a ton of processes and expectations, and them not knowing how to get comfortable with the security research space. I won’t speculate other reasons that may have had a factor, but I have heard a few. In 2018, I spearheaded an effort to find a reasonable middle ground between HR/legal/ethics and we, the researchers. I called the effort “Cybersecurity Moonlighting,” received feedback from all/most of the parties involved (many dozens), and grouped those into “must haves”, “like to haves”, and “things for consideration”. At the same time, my wife and I were praying – a lot – about our PEN Consultants side business, whether the timing was right to do it full-time. Because I’m hard of hearing – physically and spiritually – I asked God for a clear, unmistakable sign.

Just before my “Cybersecurity Moonlighting” effort got sent to legal/HR/ethics/etc. for final review so it could make its way into policy, USAA formally reprimanded me for work I was doing on the side. Work I had permission to do! Additionally, they set a policy that all outside activity had to be approved through legal…on an individual basis – every blog post, every code release, every voluntary consulting engagement, each bug bounty find. Everything! The policy also stated that no outside research would be authorized against any vendor USAA uses, which essentially meant nothing against any well-known vendor would be approved.

Given that I do not prefer to be reprimanded for things I have authorization to do, and that security research is not only core to my being, but also core to my ability to remain highly skilled in this field, I submitted my resignation within hours of that decision. I agreed to stay on not only for the customary two weeks, but for as long as they needed me. The only caveat I gave them for me staying on longer was that I could not maintain compliance with the new policy during that time. Interestingly enough, USAA went the other way with it – I was immediately terminated when I submitted my resignation (accounts terminated within minutes). Side note: No infoSec employee before me had been denied a 2-week period to say goodbye and close things down gracefully.  When submitting this article for prepub to USAA, I was told only one other employee has been terminated immediately after notice of resignation, and it is certainly not the standard.

I have heard it said my immediate termination was likely due to the fact that I, being the red team lead, probably knew about more vulnerabilities in USAA systems than just about anyone else and was seen as a risk if I were kept on for an additional 2 weeks. In my case, I would never consider any sort of retaliation, but given the feelings I had at that time, I would totally understand others not maintaining certain ethical (and legal) standards.

The point of this long answer is that I asked God for a very clear and direct sign, and He gave it to me. It literally could not have been a clearer answer to my “Should I go out on my own?” prayer.

Note:

• I still have a high regard for USAA and even mentor/coach people on how to get on there.
• The majority of the people involved had nothing against me personally, or researchers in general. For the most part, everyone was simply trying to “do the right thing”.  In fact, most everyone on the infoSec side of things (all the way up to, and including, the SVP), went to bat for me on multiple occasions, provided me with top cover for some prior events that scared legal/ethics, and engaged the corporate level policy makers on more times than I’m even aware.  I will forever be grateful to those who supported, encouraged, and mentored me while I was there..  I still have ties to people at many levels of USAA, and even eat lunch with one of the execs on a quarterly basis.
• It’s simply a matter of USAA not being a compatible employer for those wishing to perform certain types of research activities on personal time. If you want to perform all of your infoSec/cyberSec work on-the-clock and care little about doing it on personal time, I would highly encourage you to apply for work there (https://www.usaajobs.com/). Feel free to reach out and I’ll even offer advice on how to get in and who to contact.

How did you form the legal entity you’re working under?

There are two primary places to go to for this: a lawyer or an online legal service (ex. LegalZoom, Rocket Lawyer). Due to costs, we went with the latter.

As far as going LLC vs Corporation vs some other legal entity, we went the LLC route for now since we are a small startup. The overhead and costs of going the corp route just didn’t make sense at this time, but may in the future. You certainly need to obtain legal advice on that if you are unsure.

What type of insurance coverage did you start out with?

It all depends on your risk profile which should be discussed with an insurance professional. I recommend checking here: https://www.daveramsey.com/elp/insurance

At minimum, you are going to want General Liability and Professional Liability (AKA errors and omissions). Although most minimum state requirements, if any, are likely lower, 1 million dollars per incident with 2 million aggregate for each policy seems to be typical for small firms (0-5 employees). This will run you around $2,000-3,000 per year. If you get up to 10-15 employees, you likely want to increase the aggregate as a function of the number of employees and amount of business you are performing.

You will also want to look into Cyber Liability if you plan to store any amount of client data. This will easily add another $2,000 to your policy costs (costs are increasing rapidly every year). Note: we avoid storing much more than screenshots of client data, which themselves are almost immediately redacted. I would certainly advise against exfilling client data to the greatest degree possible. During pentests and red team engagements, we even try to avoid staging client data on another internal host. If we come across credit card data, we will substitute fake CC data in before proving exfil (for example). In some cases, the data will fall under a regulation (ex. PCI, HIPAA, etc.) that specifically states data cannot be placed on an unapproved system. Assuming your attacker systems are not PCI/HIPAA/etc. certified, you would likely be in violation if you even attempted it.

There are other coverages, such as general umbrella coverages, which may be beneficial as well. So far, it hasn’t made sense for us to go with these as our risks simply are not there yet.

With all that said, one interesting thing you will occasionally come across is a client requesting more coverage than what your current risk profile requires. If it means adding another $1,000 of insurance to get a $15,000 job, that may make sense. When your client is asking for coverage that costs more than the invoiced amount for the job, that probably doesn’t make sense. What that percentage is for you may look different than ours. For us, we are willing to add additional insurance so as long as it does not cost more than 10% of the testing fee, or 20% if it’s something that another client has asked about before. Note: less than 10% of our clients even ask for proof of coverage at all, much less ask for more coverage. We are willing to lose clients in order to keep our costs low.

What is the hardest part about having your own company?

The bumps in the road are much more obvious. Working for someone else, especially a larger organization, you are nearly oblivious to those bumps as someone else is taking care of them.

On that note, you end up having to become a jack-of-all-trades, and not just a technical one. For someone with a technical and business background, this may not be a struggle. For me, a full blown introverted techy with zero business experience, the business aspects are literally some of the hardest things I’ve done in my life. What should be simple ends up becoming very complex and uncertain because everyone has their own way of doing something based on the business philosophy they were taught. I have found that trying to determine what decision is best for you is hard.

How do you find clients?

I’ll break that down into five areas:

Referrals

The best clients come by referrals, no question about it. It’s simple, clean, cost effective, efficient, a better experience (for both us and the client), etc. Before we have even heard of the client, someone is telling them about our services, how good a job we’ve done for them or someone they know, our credibility, etc. By the time the client makes it to us, they are already 99% convinced they should go with us. This is so valuable to us we give $250 cash, or $1,000 testing credit, for a referral: https://penconsultants.com/referral.

Another interesting way we have acquired clients is through other pentest vendors. It sounds odd, but what usually happens is a smaller client (ex. $100 million credit union or small startup software development company) engages a medium/large sized pentest firm for needed testing. Those firms are unable to perform even the smallest jobs for less than $10,000 to $15,000 due to their overhead and hourly rates. In many cases, that is the client’s entire IT security budget for the year. Since the large testing firms know they can never meet that need, they send them to smaller firms, like us.

Partnerships

The next best set of clients come from partnerships we have with other IT service providers – virtual CISOs, managed IT/security firms, vendors, etc.

Our preference, and the majority of our relationships, are just basic mutually beneficial referral relationships. Not only does this enable us to refer our clients to a trusted partner, but it also gives our partners a trusted security firm to refer their clients who are in need of quality security testing.

We occasionally enter into a subcontract service agreement with a prime and perform work for the client through the prime’s MSA. Our relationship with the client can be nearly identical to a direct relationship, other than the paperwork/payment is being handled through an intermediary (i.e. the prime). Subcontracted relationships can sometimes be fully subcontracted in that the client sees us as an employee of the prime and never sees “PEN Consultants” on emails, documents, etc. (i.e. white labeling). In the first 6 months, we worked under any/all types of subcontracting relationships because we needed to eat! Although we still enter into subcontract relationships on occasion, we no longer perform services under a white label relationship.

White labeling drawbacks to consider:

• Overhead is too great (refactoring forms and reports into prime’s templates)
• Communication is more complicated and error-prone (relaying everything through prime)
• Our brand is never seen (a key factor in running a business is growing your brand)
• Price is not optimal for the client (we charge for the extra overhead and the prime markups up even more)
• Makes future work and continued retainer type work less likely (you are contractually forbidden from going direct)
• etc.

Outsourced Sales

Another way to acquire clients is to outsource your sales efforts. In practice, this has proven difficult for the following reasons.

Cold calling: One type of outsourced sales is paying a sales firm for X hours per month (even up to full time) for an agent to call, email, connect with, etc. potential clients and try to convince them they need your services. Although we looked into this, we never went with one of these providers because everyone hates cold calls and that would just start things out on the wrong foot.

Commissioned sales: We have experimented with commissioned sales in a few different ways over the last year. The concept is simple – promise to pay someone X% for each client they get us connected with who signs a contract. The problem is, when that percentage is 10% (for example), it is impossible to find an experienced sales agent who is able to bring in a warm lead (warm lead = person has expressed interest, but no contract yet). The experienced sales agents work for 40-60% commission, just for a warm lead! I wish I was exaggerating, but I am not. We would have to double our pricing to make that work, which runs counter to our vision to remain the most affordable.

Direct Contact

Direct contact is essentially the same thing an outsourced sales team would do – a lot of cold calling and pulling on loosely connected threads to make connections with potential clients. The difference being we are doing it directly – no commission payouts, no intermediary, no sales pitches, etc.

Even though these are highly targeted, individualized, connection attempts utilizing a common bond (ex. previous employer, shared acquaintance, hobby, etc.), they have proven to be mostly ineffective at landing a client. We have made great efforts via email, phone, and LinkedIn. We are in the process of attempting mailers and in-person visits in our local region. TBH, we don’t expect that to be much more successful.

Bottom line: we are not sales and marketing people, so there’s close to zero chance of this working for us.

Marketing

Our marketing efforts are close enough to zero to say we don’t do marketing. As a techy working under management that frequently sent a “Hey, check out what I came across” message, I hated marketing, and that hate has transferred to our business. All evidence I’ve seen seems to indicate marketing is minimally effective compared to referrals and partnerships that can vouch for you.

Has the Coronavirus pandemic affected your business?

It has, for sure. It’s hard to say how much since, as of today, we are still in the middle of it. Our current estimate is about 20% client loss and 50% client postponements. We are only doing about 20-30% of the work we were before the government shut down private businesses. We are hopeful the majority of that will return.

The loss of life and people’s jobs is tragic. We can argue about what could have been avoided and who is responsible, but the bottom line is, it happened, it sucks, and people’s lives have been turned upside down. We are very fortunate to be debt free – both as a business and personally. That will enable us to ride out the storm a little better than some of the other firms.

How do you deal with an angry client?

Thankfully, we have not had an angry client as of yet. There have been a handful of occasions over the years where we negatively impacted a client’s network and it led to dropped VoIP based calls, network congestion, etc. But, because we provide real-time communication (https://penconsultants.com/informed), those types of issues are almost always identified and resolved within minutes. On one occasion, our testing knocked out the ability for the admins to gain remote VPN access until the appliance was bounced that evening. At most, a few clients may have become a little annoyed, but were understanding and forgiving.

In addition to our guarantee of high quality, standards-based testing at a fair price, we offer a satisfaction guarantee. If a client is ever unsatisfied, we commit to doing everything possible to resolve the issue.

The other part of it is we try to avoid demanding clients as demanding clients can often become angry clients. It’s pretty obvious a client will be demanding when they contact you and want a report on their desk by next week for a testing engagement that doesn’t even have a signed contract yet, much less started. Another warning sign we lookout for is a client that treats you like the proverbial red-headed stepchild from the get-go.

If we sense a bad match, we use a few undisclosed techniques to help the client realize we are not a good fit. We pour our heart and soul into what we do, and have no desire to work with someone who cannot appreciate that.

Do you have employees? How do you know when to hire?

We are a small startup family business. There are currently three people who work for PEN Consultants. Since those three people are under the same family unit from an IRS perspective, we are essentially a one-person firm.

With that said, just before the Coronavirus pandemic hit, we were already in pre-interview talks with a handful of candidates and planning to hire eminently. The future is uncertain now. It could be that we remain a small family business forever. Or, we may scale up to be a large firm. We are taking that one day at a time and seeing where it may lead.

Our metric was two full quarters of 60-80 hour per week workload before hiring. The closing of our country happened two weeks before pulling that trigger.

How hard was it to transition from red teaming back to pentesting?

This was one of the hardest things to cope with. But first, definitions: https://penconsultants.com/testingDiff

I greatly miss my five years of building and leading USAA’s red team and my nation state level exploitation at NSA. We infrequently get to perform either now due to the size and maturity of most of our clients. Social engineering (physical and remote) is about the extent of it so far, but the hope is to secure larger clients who will be in need of the level of red teaming I provided while at USAA, or the large focused research/dev effort comparable to my time at NSA.

Regardless, my experience has certainly given me a somewhat unique perspective on pentest findings. Many times a client provides previous pentest reports when we start the engagement – in some cases compliance requires it. More times than not, it is obvious the “pentester” simply ran a few tools and spit out a report without any consideration of actual likelihood of attack or considering anything outside of what the tool found.

Having been an attacker for so long, and playing the attacker at various levels, I am able to bring a much different perspective than your typical compliance-focused pentest. I feel, and we receive confirmation from clients, that we generally deliver a better prioritized list of findings. Examples: https://penconsultants.com/report

For our more mature clients, one service I like to push is our Technique Simulation service: https://penconsultants.com/redteaming. Even if the client is not ready for our Adversary Simulation service, nearly any client with even a basic detection capability can benefit from testing common attacker techniques to ensure their COTS solution is setup/configured optimally.

What percentage of testing vs consulting do you perform?

Our services are certainly weighted heavily towards testing and a detailed report. The recommendations portion of the report would obviously be a form of consulting. Approximately one-third of the hours spent on an engagement is, what I call, post-core testing. The most obvious task during that phase is writing the report, but also includes some additional adhoc testing (i.e. “oh, what if I did X”). The reason the report takes so much time is because the recommendations are custom tailored to the client. Example: depending on the level of access we have, we literally pull things like group policy and show the current config and what the config should be to prevent the attack demonstrated.

Additionally, there are often post-testing conference calls, debriefs, emails, etc. that drill into other angles of a finding and what the best route is for the client. We also provide a retainer service to our clients, branded as Cybersecurity Unlimited, which allows for those random adhoc consultations throughout the year. Details: https://penconsultants.com/cybersecurityUnlimited. Of the dozens of clients in a given year, less than a handful need/require that level of consulting. Given that I’m not a sales and marketing guy, I probably am just doing a poor job in offering that to clients. This type of service is one of the most beneficial for both us and the client – the client has direct and immediate access to advice and small testing requests, and we get a more steady, albeit smaller, source of income.

Venture Capital money or loan to get started?

We are debt free – personally and as a company – and do not enter into debt for any reason. We live by the Dave Ramsey principles which are based on scripture. Since we have strong faith, we live according to the Bible’s teachings on debt – that the borrower is a slave to the lender (Proverbs 22:7).

Although it is easier, and may be perceived as more reputable to clients, to borrow millions of dollars and hit-the-ground-running is an enormous amount of risk compared to building the business slowly (debt free). Scripture aside, since a business is more likely to fail than succeed, it seems foolish to gamble on that risk.

Another benefit of growing slowly is the ability to develop your culture in a more truthful manner. With venture capital-backed companies, for example, greed will inevitably drive certain decisions. We do not want to be like that. Our stated purpose is to help our clients and community…not to get rich. If we eventually make a lot of money, that might be fun. But, for now, we are just concerned with making enough to keep our family fed.

Are your rates too low?

In one word, “yes”, our prices are currently too low. They are so low, many mid (and certainly large) sized organizations question the quality of our work. Even though we post sample reports, offer client testimonials, post more details about our testing methodology than the firms they are likely using, etc., the fact that our prices, as of 2020, are still one-third to half of the comparable service quality elsewhere, understandably causes concern.

Although we have been increasing our prices by 10% per quarter, we will likely never charge as much as the industry average does for comparable work. Our stated vision has been, and continues to be, “to be the most highly skilled, ethical, effective, and biblically-centered security testing company in the industry, while remaining the most affordable”. We uphold this with our guaranteed fair price on superior quality, standards-based testing.

Given our hourly rate (which is half of the industry average), our vast experience (more experience reduces testing time), and low costs overhead (no marketing, office, etc.), we should be able to remain the most affordable on the market forever.

Any pricing models that seem to have worked out well? Anything to avoid?

Our pricing model is something that sets us apart from most testing firms. For one, we post our pricing for typical small/mid-sized engagements, which is nearly unheard of in this industry. In fact, it is rare to find a firm that is even willing to post its hourly rate. More about our pricing model, and transparency in general, can be found here: https://penconsultants.com/compare

At this time, most of our contracts are on a per-engagement basis. Some firms will request a 3-engagement/3-yr contract from the get-go, or even a 3-yr with quarterly testing. Although we do offer substantial discounts for multi-year contracts, we do not feel that is fair to a client to be forced into it from the get-go. With that said, I certainly see the business benefits of doing so.

Do you talk about your faith with clients?

We certainly do not hide the fact that our business is run by Bible-believing Christians. With that said, we have been the ones to bring up our faith exactly zero times in a client conversation. If a client asks a religious-related question, we do not hesitate to answer it. On a handful of occasions, our client has wanted to venture off into a 30/45 minute conversation on faith, after the testing-related conversations are over, and we are always happy to discuss it.

How do clients react to your faith and beliefs?

We are not aware of anyone that has been offended by our beliefs or anyone that has avoided our services due to our faith. As mentioned previously, it is not something we bring up unless a client does. If just the mere fact that we are Christian offends someone to the point they cannot do business with us, then they likely lack the tolerance of someone we would want to do business with anyway.

At the end of the day, our services are making people more secure, regardless of their religious beliefs or lack thereof. We are not a church; we are a security testing company.

Changes

• 15 May 2020: Initial draft, sent relevant section to USAA for prepub review.
• 16 May: A few additions based on USAA feedback – 1 (not 0) other infoSec employees have been immediately terminated after giving a 2-weeks notice, USAA’s processes and expectations were not as agile as one may hope for due to company size, and the fact that the disconnect really was not with the security team/leadership but more the corporate level policies and guidelines.
• 16 May: Sent new version back to USAA for review.