Product Data Extraction

A product page is written for shoppers, not for databases. The name and description sit at the top, specifications further down, sizes and colors in dropdown menus, and the price, stock message and photos around them. To use any of that in a catalog, spreadsheet or database, it has to be pulled together into one structured record per product.

That is what product data extraction does. Worth Web Scraping extracts product information from online stores, marketplaces, manufacturer websites and catalogs and organizes it into consistent records, with the same fields for every product.

Projects are built around the product fields, source websites, categories, variants and output structure your work calls for.

A spreadsheet with a product name and a price is a list. A product record goes further: it holds what is needed to identify the product, describe it, tell it apart from similar ones and present it, from brand and category to specifications, options, availability and images.

The aim is a catalog you can work with: one record per product, the same fields throughout, and the information from each page in its own column instead of buried in page text.

Which fields go into a record depends on the source and the project. Most product records draw on these groups:

• Identity – product name, product URL, product ID, SKU where displayed, brand and product type
• Classification – category, breadcrumb or category path, and collection name
• Description – the product description and listed features
• Specifications and attributes – dimensions, weight, capacity, materials, technical characteristics, compatible products and other details the page lists
• Variants – options such as size, color, capacity and pack size, each with its own SKU, price and availability where the page shows them
• Availability and price – stock status or availability message, current and original price, discount and currency
• Images and media – primary and gallery image URLs, plus video links where a page has them
• Ratings – average rating and number of reviews, where displayed

Price is one field among many in a record here. Tracking how prices change over time is a different kind of project.

One product page is easy to read. A whole catalog is harder, because it is rarely uniform. Products in different categories carry different attributes: clothing has sizes and materials, appliances have capacities and technical characteristics, and a given field may turn up in a different spot on each page template.

The extraction is set up so that records from every category end up with the same field names in the same file. Where a field does not apply to a product, the dataset can retain the same field structure so records remain consistent.

Projects usually start from one of these:

• A list of category or product-listing pages
• A list of individual product URLs
• A section of a website’s catalog, defined by its categories

Where the same fields are wanted from more than one source, such as a manufacturer website and a retailer, they are collected in the same layout so the results can be compared.

Variants are where product data gets complicated. A single product page can offer several sizes, colors, capacities, pack sizes, models or styles, and each combination may have its own SKU, price and stock status.

Take an insulated bottle sold in two capacities and two colors. It is one product page, but four variants:

• 500 ml, black
• 500 ml, blue
• 750 ml, black
• 750 ml, blue

Each of the four can carry its own SKU, price and availability, while the product name, brand, description and specifications are shared.

There are two common layouts. In one, each variant gets a row of its own, with the shared product details repeated on every row. In the other, there is one row per product and the variants sit in separate columns. The first suits catalogs and databases that store variants individually; the second suits a simple product list. The layout is agreed at the start of the project.

Sites also differ in how they present options. Some show every option on one product page, while others give each option its own URL. The output can follow one consistent structure whichever way a source does it.

Request a Data Collection Project

Product data is rarely uniform. One source may give weight in grams and another in kilograms, or write a size as “M” on one site and “Medium” on another. Where needed, duplicate records, inconsistent formats and missing values are reviewed before delivery.

The finished catalog has one column per field, whether that is a product name, a specification or a variant SKU. CSV and Excel files suit spreadsheets and catalog tools, while JSON and database-ready data suit applications and systems, with API or data feed access where applicable to the project. The layout follows what was agreed at the start.

The same product records can serve several purposes:

• Building or updating a catalog – structured product data created from stores, marketplaces, catalogs or manufacturer websites
• Product research and comparison – the same product fields collected from several sources so products can be compared side by side
• Completing existing records – selected fields such as descriptions, specifications, variants or images added to product information you already hold
• Feeding your own systems – structured product data loaded into databases, applications or analytics

A project can run once or repeat on a schedule defined for the project, depending on how often the source information changes.

Product data projects vary a great deal: one manufacturer’s product range, or several categories across a number of stores. Each is defined around your sources, fields and output structure rather than a preset package.

It helps to know:

• Which websites or catalogs the products are on, and which categories or product URLs to include
• The fields you need in each record
• How variants should appear: one row per variant, or one row per product
• The output format and layout you prefer
• Roughly how many products are involved, and whether this is a one-time job or a repeating one

If some of this is undecided, send what you know and we can go through it together and suggest how the product data could be structured.

Request a Data Collection Project