Using webcrawler techniques for improved market surveillance – new possibilities for compliance and energy policy

Panel: 7. Appliances, products, lighting and ICT

This is a peer-reviewed paper.

Authors:
Peter Bennich, Swedish Energy Agency, Sweden
Linn Stengård, The Swedish Energy Agency, Sweden
Signe Friis Christensen, Danish Energy Agency, Denmark
Teemu Hartikainen, Finnish Safety and Chemicals Agency
, Finland
Troels
Fjordbak Larsen, Big2Great ApS
, Denmark
Kasper Schäfer Mogensen, Big2Great ApS
, Denmark

Abstract

In recent years, web crawler techniques have been developed at a rapid pace. With intelligent software, it is possible to scrape large volumes of data from publicly available data sources at the Internet, basically in real time. In the realm of product policies, such as ecodesign and energy labelling, this offers alternative means to track products available at the market place, when compared to more traditional data collections offered by companies like GfK. Thus, typical information on products and model names, claimed performance on functionality and energy use, and finally purchase price, is thus cheap and relatively easy to collect in real time. With advanced modelling based on ranking (popularity indexes at sites like e.g. Pricerunner) it is even possible to estimate the sales volumes.

The new possibilities thus offered are currently being explored for policy evaluation and improved policy design, as well as for developing new and improved consumer tools when choosing and buying energy using products.

This paper focus on another use, namely on how webcrawler techniques can increase the effectiveness of market surveillance. In the project “Nordcrawler”, run under the Nordic Council of Ministers, market surveillance authorities (MSA:s) from the Nordic countries work together to develop a web-based application which each MSA can use to monitor their respective national market.

The advantages include the following aspects:

• Reduced cost for data collection

• Higher coverage and a better representativeness of the market

• Data collection in real time

• Improved sampling strategies when choosing products to check, be it random or targeted sampling

• Faster response times for targeting non-compliant products

• Overall improved effectiveness and reduced cost for market surveillance

This is a new way of working for MSA:s and the paper discusses lessons learnt. Apart from the advantages above, it also includes issues like retailer acceptance and legal aspects.