Web Data Extraction

Not every data project needs a whole website. Often the requirement is much narrower: the price and availability from a list of product pages, the address and category from a set of company profiles, the headline and publication date from a group of articles.

That is targeted web data extraction. You point to the pages, name the fields you want, and the information is pulled from those pages into structured records. Whatever else the pages display is left behind.

Worth Web Scraping builds extraction projects around the pages, fields, volume and output format you define. A project can be a one-time extraction, or it can be repeated on a schedule defined for the project.

An extraction project begins with the fields, not the website. Once you know what each record should contain, it is easier to see which pages hold the information and how the results should be laid out.

Fields tend to fall into a few kinds:

Names and identifiers. Product names, company names, listing titles and headlines, plus product IDs or SKUs where a page displays them.

Descriptions and attributes. Descriptions, categories, tags, specifications, colors, sizes and other details listed on the page.

Prices and numbers. Prices, original prices, discounts, currency, ratings and review counts.

Locations and business details. Addresses, cities and regions, business hours and services displayed on the page.

Dates and status. Publication dates, listing dates, availability and listing status, where a page shows them.

Text and reviews. Article summaries, review titles and review text, with authors and publishers where displayed.

Links and media. Page URLs, source URLs and image URLs.

These are examples, not a fixed menu. Other publicly displayed information can be added as a field where your project calls for it.

A web page usually carries far more than a project needs: navigation menus, banners, related items, footers. Extraction picks out the fields you have named and leaves the rest.

Take a single product page. The record for that page might hold only:

• Product name
• Page URL
• Price
• Currency
• Availability

Point the same extraction at a category page that lists many products, and each product on it becomes its own record with the same fields. Laid out this way, the data can be opened in a spreadsheet or loaded into a database.

Different kinds of pages give up their information in different ways.

One item per page: a product page, a company profile, an article or a single listing. Every field comes from the same page, so the work is mostly a matter of finding each field in the same place across many pages built from one template.

Many items sit on one page. Each item becomes its own record, and results that continue over several pages are worked through one page at a time.

A list page links to a separate page for each item. Fields from the list, such as a title and price, are combined with fields from each item’s own page, such as the full description or specifications, into a single record.

Some pages present values in tables or repeated blocks, such as specification tables. The rows and columns on the page are mapped to the fields you have defined.

Before anything is extracted, five things need to be settled. Writing them down first keeps the results close to what you expected.

1. The pages. The URLs, page types, categories or searches the data should come from.
2. The fields. What each record should contain, and the column name you would like each field to have.
3. The scope. How many pages are involved, whether one website or several, whether the extraction is one-time or repeated, and whether historical information is needed where the requested data is accessible.
4. The rules. How to handle values that are missing on some pages, and any formatting you expect for values such as prices, currencies and dates.
5. The output. The file format and layout the data should arrive in.

Request a Data Collection Project

Pages on the same website are not always identical. A field can sit in a different place on one template than on another, a value can be missing on some pages, and formats can change from one item to the next. The extraction has to be built around how the pages you specify are actually put together.

Situations to plan for include:

• The same field appearing in different places on different page templates
• Fields that are present on some pages and absent on others
• Product options such as sizes and colors, each with its own value
• Results that continue over several pages
• Content that only appears after the page has finished loading
• Values shown in mixed formats, such as prices with different currency symbols

Websites also change over time, so a repeated extraction may need to be looked at again when a source page is redesigned.

Extracted values often need a little preparation before they are useful. The information is organized into a consistent dataset, with each field in its own column and the same field names on every record. Where needed, duplicate records, inconsistent formats and missing values are reviewed before delivery.

The finished data can be delivered as CSV or Excel files, as JSON, as database-ready data, or through API or data feed access where that suits the project. The structure follows the fields and layout agreed at the start.

It is usually a good fit when:

• You know which pages, or which types of pages, hold the information
• You can list the fields you need from each page
• You need only part of what those pages display
• The same fields are needed from a defined set of pages or websites

It is a poorer fit when:

• You do not yet know which pages hold the data and need to find them first
• You want a complete copy of a website rather than selected fields
• The requirement is open-ended, such as “everything available on this topic”

In those cases a project usually starts with a broader collection step, and targeted extraction can be defined once the pages and fields are clear.

Extraction requirements are often specific. A field may need a particular name, a value may need to appear in a certain format, or a set of pages may not follow one pattern. The project is defined around those details.

The easiest brief is a handful of example URLs and a short list of the fields you want from each. If you have an example of the output you would like, even a few rows filled in by hand, send that as well. Add how often you need the extraction, roughly how many pages are involved and the format you prefer.

If some of this is still undecided, tell us what you have. We can review your requirements and recommend a suitable approach to the extraction.

Request a Data Collection Project