I have noticed lately that many eCommerce guides are focused on the same tips:
While these are all great tips, I’m here to give you just one scraping hack that saved my business from shutting down. (If you’re not using web scraping for your online business already, check out this blog).
source: https://upload.wikimedia.org/wikipedia/commons/c/ca/Scraping_Lesson.jpg
How it all started
My background is in marketing, I’m an experienced SEO, PPC and affiliate marketer, so naturally, I understand a thing or two about driving traffic and turning it into sales.
It felt right to open an eCommerce shop to generate another source of income while keeping my day job as VP of marketing.
I had a programmer friend who agreed to code a scraper for me with a pricing optimization logic. The logic can be simplified into two basic rules. The first rule is that whenever a competitor is found lowering one of their prices, I get a notification so I can decide whether I’m going to lower my price or sit idle. The second rule is whenever a competitor runs out of stock, I get notified so I can decide whether to increase my prices or not.
After a few months of hard marketing work, I saw an increase in traffic to my store, which later turned into an increase in conversions. Life was good, money was coming in and my only tasks were to execute remarketing campaigns and keep my SEO neat.
However, the party didn’t last long, weeks passed and much to my surprise, I saw my conversion rates drop, and eventually, my expenses exceeded my earnings.
I tried to look for the source of the problem. I had not changed my store page nor my marketing campaigns before the decrease started, and my traffic’s volume remained stable.
Finding the cause
Ruling out the store and the marketing campaigns, I then checked my pricing and my scraper.
I compared the dates where my conversion rate dropped to a cached version of my competitor’s site and I found that my competitors had significantly reduced the prices of the products I was selling. For some reason, I had stopped receiving the notifications a few days before the drop in conversion rates.
No wonder I was losing sales – My store’s target audience is within the least loyal and most price-sensitive age group: 21-29. According to research by MarketTrack , 85% of them will switch stores and 86% will change brands if they find better prices elsewhere.
What Went Wrong with my scraping?
Recommended for You
Webcast, March 29th: How To Close Enterprise Deals: The Sales Hacker Story
Was I scraping too fast? Too aggressive? I used common practices to ensure I was not recognized by my competitor as a bot, and it worked fine in the first months of my operation.
When I researched the subject, I found out that target companies do attempt to protect themselves from being scraped. For example, a company called Shiledsquare helps eCommerce businesses by identifying scrapers due to unusual activity.
As businesses usually scrape other businesses through paid data centers with a specific, publicly listed range of IPs, the company can easily identify and block suspicious IPs owned by commercial data centers. This means that I was probably being blocked because someone else had scraped the same site poorly using the same IP range I was using …….and yet, I paid the price.
Finding a solution
At this point, I realized I needed a solution that fits my existing scraping infrastructure, one that was reliable and that would never get blocked.
Luckily I discovered residential IPs. Residential IPs are IPs that were given to a private user by an internet service provider company. The difference between residential IPs and data-center IPs are their chances of being identified and as a result, be blocked.
Apparently, the new generation of business proxy networks consists of P2P consumers. These consumers agree to allow the routing of some requests through their IP while their device is idle, connected to wifi and charging. In return for allowing these requests, they receive benefits in the form of free use of applications and ad-free applications to name a few.
Apparently, the new generation of business proxy networks consists of P2P consumers. These consumers agree to allow the routing of some commercial requests through their IP while their device is idle, connected to wifi and charging. In return for allowing these requests, they receive benefits in the form of free use of applications and ad-free applications to name a few.
App developers would usually want to use this kind of barter as growing and monetize their app with ads or paid subscription usually decreases the engagement.
This makes blocking scrapers almost impossible since the target websites have no interest in blocking regular consumers from accessing their site.
I found a company called Luminati who has built a huge residential IP pool, and I quickly switched my scraper’s infrastructure to continue scraping using their residential IPs.
The price alerts came back immediately, and together with the right price adjustments, my conversion rate jumped back up and I quickly became profitable again.
Conclusion
Scraping has two sides to it. On the one hand, it’s fully automated and once it’s working you can go into auto pilot mode. On the other hand, the sites you are scraping will try to block you and you must avoid this in order to ensure you are always getting reliable data, and that’s exactly what residential IPs allowed me to do.
