How a major electronics retailer ended the bot menace on its portal

It's May 2020, and a large electronic goods retailer realises its bot problem is going from bad to worse.

In a single week, its online store is hit with eight million bot visits to systematically scrape pricing and product information without authorisation, not to mention 53,000 customer account takeover attempts, 136,000 denial of inventory attacks, and 234,000 attempts to conduct affiliate link fraud.

Even without these malicious activities, bot traffic is tying up valuable resources in ways guaranteed to hurt the bottom line.

For a firm operating 300 retail stores that attract ten million shoppers a year to its website, this level of bot traffic is unsustainable. It's risking not only lost sales and rising costs but also damage to a brand image built with huge effort over many years.

Bad bots affect retailers in a range of sinister ways:

• Account takeover (ATO) fraud – Criminals breaking into a customer's account to carry out a range of frauds, including identity theft, stealing loyalty points, or making fraudulent transactions.

This often happens because of credential stuffing, which exploits the fact that many customers reuse the same username and password across multiple accounts. When those reused credentials are leaked or breached and then sold on the dark web, users' accounts are vulnerable to ATO.

• Denial of inventory – Filling baskets with products without paying for them. This ties up inventory, artificially reducing product availability while damaging the retailer's sales.
• Affiliate fraud – Earning rewards from a site by generating large amounts of junk traffic. Retailers end up paying for nothing.
• Wasting resources – Dealing with bot traffic makes it difficult for marketing teams to get accurate KPI data to plan for growth.
• Carding attacks – Using retail websites to 'test' stolen credit and debit card data. Retailers are often left with the cost of reimbursement ­— and a poor merchant reputation.
• Scalping – Using automated programs to grab desirable inventory before real customers can. The goods are then resold at inflated prices on the secondary market.
• Web scraping – Stealing pricing and other proprietary data for rivals or other malicious purposes. A variation on this is scraper bots that steal other website content, including reviews and product descriptions.

Oddly, the last two bot activities are not illegal in many countries. However, this doesn't mean that the retailer should tolerate them. For example, scalping can hurt the reputation of a retailer with genuine customers.

Bots can be hard to see until trouble strikes. Bad bots plague today's retailers all year round, but the extent of the problem often becomes even more magnified at peak times, such as holiday seasons when traffic naturally spikes.

For example, during Thanksgiving 2021, a group of six prominent e-commerce websites protected by my company's bot manager were flooded with bot traffic. Traffic volume ranged from more than four million bots per day to well over nine million bots per day during the week prior to the holiday.

To a retailer without bot protection for its website or apps, these numbers represent potential attacks that can wreak havoc on the user experience. Some retailers may experience website slowdowns that frustrate shoppers. Others may see a drain on the inventory of highly sought-after products that are snatched up by scalpers for resale rather than loyal customers.

Still others will field complaints on their support line about cashed-out gift cards and loyalty points. Not only does the customer experience suffer, but ultimately brands are damaged, and revenue is lost.

So, how did the bot battle end for the large electronics retailer mentioned earlier?

The retailer knew something was wrong, but what? How big was this problem? In search of a solution, the company made the decision to trial a leading web application firewall (WAF) and application protection solution.

The security vendor's analysts discovered that more than 50% of all visitors to the retail site were, in fact, bots. This was an unsustainable situation that, if left unchecked, could lead to the adverse consequences mentioned above and invite even more damaging bot attacks.

To further evaluate the situation, the retailer initiated a proof of concept (POC) trial with my company's bot manager, using our NGINX connector to integrate with its website. After analysing visitor traffic for a week, the bot manager went into 'active mode' and began to block over two million bad bots every day thereafter.

Suspected bots were shown a CAPTCHA to solve to enter the website. Only 0.25% of these challenges were solved, which meant that almost all bots were blocked, and genuine visitors were not shown a CAPTCHA while visiting the website.

CAPTCHA challenges, of course, are only an initial step in an overall bot detection process that is powered by a patented intent-based deep behaviour analysis technology. This technology offers unmatched accuracy in detecting sophisticated bots that emulate human behaviour as they traverse a website or application.

Soon after our successful engagement with this retailer, my company learned that earlier, they had also conducted a POC with a global provider of CDN and bot mitigation services, which fell short in two major areas when compared to our results:

• The competing solution detected approximately 20% fewer bad bots than we did. Considering we detected and blocked two million bots daily, this would theoretically mean 400,000 bad bots carried out attacks every day.
• The competing CDN-bundled bot mitigation solution required all website traffic to be rerouted through its servers for bot detection, which was an unacceptable proposition for this retailer (and any organisation serious about data privacy and protection).

In the end, the result was clear — a leading bot protection solution had proved itself more effective under real-world conditions.