Crowdsourced vs. Traditional Pen Testing

A side-by-side comparison of key test features and when best to apply them based on the constraints within your budget and environment.

Crowdsourced security has recently moved into the mainstream, displacing traditional penetration-testing companies from what once was a lucrative niche space. While several companies have pioneered their own programs (Google, Yahoo, Mozilla, and Facebook), Bugcrowd and HackerOne now carve up the lion’s share of what is a fast-growing market.

How does crowdsourced pen testing compare with traditional pen testing, and how does it differ in methodology? Does this disruptive approach actually make things better? Read on for a side-by-side comparison.

Time-Limited vs. Open-Ended Engagements
One of the major downsides of pen testing today is that it doesn’t match the development speed of modern applications. Most companies pen test annually, but in today’s environments, applications are updated frequently, sometimes once a day, and sometimes even more than that. This results in your pen test being merely a snapshot of your security posture at a particular point in time. That’s it. Once you’ve updated your website or application, those findings are out of date, which means potential new vulnerabilities.

Crowdsourced pen tests are typically open ended, which maps better to how applications are built today and, most importantly, how attackers behave. An attacker can spend three to four months examining one of your assets if he pleases; a traditional pen tester doesn’t have that luxury. On the other hand, crowdsourced pen testers do, and it shows as they dig up highly critical bugs from live sites they have been pen testing for years.

I once found a vulnerability that took me over 50 hours to find (way longer than a pen test), and the vulnerability gave me access to the internal company network as well as all its data. This company used to run pen tests, but what surprised me most was that its crowdsourced program had been open for a year without anyone finding this particular bug, which proves another point: The more eyeballs you throw at something, the more things you’ll discover.

Proof-of-Concept vs. Theoretical Vulnerabilities
I’ve read dozens of pen-testing reports over the years filled with “junk” risk, where a vulnerability is listed as “high” just because a system is not on the latest patch but without showing how. When asked for a proof of concept on how this is exploitable, the report’s authors usually remove that from the report. This is now referred to as pen-tester syndrome — making things appear worse than they actually are. Garbage such as missing HTTP headers with absolutely no context as to how or why they are exploitable also falls into this category. In a crowdsourced pen test, you will only get exploitable vulnerabilities with actionable proof of concept. This does wonders for preventing companies from chasing phantom risk and focusing their remediation where it matters. Crowdsourced security really shines in this respect.

Pay per Pen Test vs. Pay per Vulnerability
Pen testing, for now, has held its ground against crowdsourced security because it’s cheap. Since you pay per day and a typical website will take you between four to five days, you know exactly how much you will pay up front, regardless of how many vulnerabilities are found. Crowdsourced pen tests, on the other hand, can vary, and because you have to pay both a platform fee and, on top of that, pay per vulnerability found, it can get expensive. While different providers now vary their models (some will charge just a platform fee so you don’t pay per vulnerability), it can be difficult to budget for the pay-per-vulnerability approach.

Testing Different Types of Assets
If you want someone testing from “inside” your network in a traditional pen-testing environment, a pen tester physically turns up at your office and just plugs in his or her laptop. In a crowdsourced scenario, it can get messy. Some of the engagements I’ve participated in require VPN or proxy setups, and you’re usually in a test environment, not a live environment with real users. This increases the cost for companies, not to mention doing this not just for one but dozens of testers. Other assets such as embedded and Internet of Things devices require a physical asset in hand, and while I have seen a few crowdsourced programs send out devices to testers in the mail, it’s more convenient and cheaper just to hand a single device over to a pen tester. For now, if you want to test anything inside your network or an IoT device, pen testing is just more convenient and cheaper!

Salaried Employee vs. Disposable Resource
While rarely considered, there is a glaring difference between both crowdsourced and traditional pen tests: how people are rewarded. In a traditional pen test, you know that work is carried out by a salaried employee who is remunerated correctly and paid regardless of whether he or she finds vulnerabilities or not. It’s likely this person has other “soft” benefits such as a pension plan and pen-testing tools paid for by the company, and probably gets regular training and sick pay.

Crowdsourced pen testers do not have any of that because they are paid per vulnerability. Referring back to my previous example of spending over 50 hours on a vulnerability, if I had turned up empty handed, I would have been rewarded nothing at all. Crowdsourced pen testers also have to fund their own training and their own tools. Want to test an iOS app? Better have your own test device set up. You’re sick? Too bad. Pension plan? What’s that? The crowdsourced industry is acutely aware of this criticism and has started offering standard flat fees for certain tests and certain researchers, so that if you don’t find any vulnerabilities, you still get paid.

Related Content:

Alex Haynes is a former pentester with a background in offensive security and is credited for discovering vulnerabilities in products by Microsoft, Adobe, Pinterest, Amazon Web Services and IBM. He is a former top 10 ranked researcher on Bugcrowd and a member of the Synack … View Full Bio

More Insights

Read More HERE