Bigbasket Data Scraping using Python

Big Basket is the biggest online Indian grocery retailer. It has become a very popular website in Indian metro cities and is slowly growing its base in tier 2 and tier 3 cities. It is like an online supermarket where you can find all the products for your household needs such as grocery, cleaning, personal care etc. One can grab all the products by automated way using Bigbasket scraper for product monitoring.

Just like any other ecommerce website there are a lot of useful data on big basket. But it is a java script heavy dynamic website which makes it difficult to scrape. When you open this website through your browser there goes an AJAX request on the backend which displays the data on website. So if you send a get request directly to the link that you used in browser you will not get any data. We can verify this by seeing the source code. Generally if we press ctrl+U on any website it shows us the source code of that particular page which will have all the HTML content. But for a java script heavy website like Big basket pressing ctrl+U will not give us the HTML source code.

So for this kind of website we need to find the backend AJAX request URL and send a get request to that to scrape data.

Normal website ctrl+U gives us HTML source code:

But not on a java script heavy dynamic website like big basket:

Now how to find this AJAX request? To do so click on any of the product and copy the product id from the URL as highlighted below:

Now go to inspect Network and pres ctrl+F to open search box and paste the product id, press enter, this will show you all the requests in which this product id was there:

Now hover your mouse on all these request and look for the one which has the word slug in it. That’s the backend URL we need for sending a get request:

https://www.bigbasket.com/custompage/sysgenpd/?type=pc&slug=potato-onion-tomato

Now that we have got the url let’s get to the code to built Bigbasket scraper and scrape some data. As the website is JavaScript heavy we will have to use json here. To know it in detail watch the video.

Below is the complete code:

import requests
from bs4 import BeautifulSoup as soup
header = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.61 Safari/537.36'}
r = requests.get('https://www.bigbasket.com/custompage/sysgenpd/?type=pc&slug=potato-onion-tomato',headers=header)
bsobj = soup(r.content)
import json
comp = json.loads(r.text)
name = []
mrp = []
sp = []
for j in a:
  name.append(j['p_desc'])
  mrp.append(j['mrp'])
  sp.append(j['sp'])

print(name)
print(mrp)
print(sp)

Output:

import pandas as pd
bigb = {'Product_name':name,'MRP':mrp,'Special_price':sp}
df = pd.DataFrame.from_dict(bigb)

df

Output:

However, you can use our services to satisfy your accurate and bulk products data need without involving in coding. Download sample data of Bigbasket data scraping for better understanding about data.