Bad bots now make up 20 percent of web traffic

April 17, 2019 TH Author

Bots can be valuable tools for webmasters seeking additional visibility into their domains but malicious variants are a constant headache for online services.

So-called “bad bots” can be tasked with performing denial-of-service (DoS) attacks, they can scrape and steal data, they may be used to automatically publish fake content or reviews, and also skew advertising and visitor metrics.

Bots, in general, are estimated to make up roughly 37.9 percent of all Internet traffic. In 2018, one in five website requests — 20.4 percent — of traffic was generated by bad bots alone.

According to Distil Networks’ latest bot report, “Bad Bot Report 2019: The Bot Arms Race Continues,” the financial sector is the main target for such activity, followed by ticketing, the education sector, government websites, and gambling.

Based on the analysis of hundreds of billions of bad bot requests over 2018, simple bots, which are easy to detect and defend against, accounted for 26.4 percent of bad bot traffic. Meanwhile, 52.5 percent came from those considered to be “moderately” sophisticated, equipped with the capability to use headless browser software as well as JavaScript to conduct illicit activities.

A total of 73.6 percent of bad bots are classified as Advanced Persistent Bots (APBs), which are able to cycle through random IP addresses, switch their digital identities, and mimic human behavior.

An example of this is mouse mimicry, in which the bot is able to simulate mouse events a genuine visitor may perform on a website domain. These tactics are used to try and appear as a legitimate user for the purposes of ad fraud, as well as brute-force attacks against online accounts, competitive data mining, transaction fraud, spam, and phishing campaigns.

Amazon is the leading ISP for bad bot traffic origins. In total, 18 percent of bad bot traffic came from the firm’s services, a jump from 10.62 percent in 2017.

Almost 50 percent of bad bots use Google Chrome as their user agent and 73.6 percent of bad bot traffic was recorded as originating from data centers, down from 82.7 percent in 2017.

CNET: Feel safer with these smart home security gadgets

The United States outstrips all other countries as a generator of bad bots. In total, 53.4 percent of bad bot traffic came from the US, followed by the Netherlands and China. The most blocked country by IP is Russia, together with Ukraine and India.

“Bot operators and bot defenders are playing an incessant game of cat and mouse, and techniques used today, such as mimicking mouse movements, are more human-like than ever before,” said Tiffany Olson Kleemann, CEO of Distil Networks. “As sophistication strengthens, so too does the breadth of industries impacted by bad bots. When critical online activity, like voter registration, can be compromised as a result of bad bot activity, it no longer becomes a challenge to tackle tomorrow. Now is the time to understand what bots are capable of and now is the time to act.”

Previous and related coverage