OpenHazWaste: Open EU Hazardous Waste Identification Benchmark

Proposal, version 0.1 · 20 October 2026

Several providers talk about good recognition rates for chemical products and hazardous waste. A percentage that does not say what was measured, on which images, and what happens when the system answers "don't know" cannot be compared with anything. This is an open proposal for a shared measurement that companies, municipalities, recycling sites and researchers can take part in on the same terms. Kemipilot wrote the draft and will be measured by the same rules as everyone else.

What is measured

A recognition rate can mean five different things. The proposal keeps them apart:

1. Identity: was the right product found?
2. Hazard: is the hazard right, that is the UN number, the ADR class and the hazard statements?
3. Fraction: was the right EU waste code chosen, and was the waste correctly judged hazardous or not hazardous?
4. Guidance: is the instruction for a specific site correct and safe?
5. Safety: how often is the answer dangerously wrong?

A claim about a recognition rate must say which of the five levels it refers to, and on which images it was measured.

Tasks

The benchmark has five tracks, reported separately:

• Track A, chemicals: consumer and professional products, from cleaning products and paint to glue and laboratory chemicals.
• Track B, batteries: chemistry, format and condition, including damaged lithium cells.
• Track C, gas cylinders, cartridges and aerosols: gas type, container type and borderline cases such as spray cans with a broken nozzle.
• Track D, guidance: is the specific instruction for a named site correct once the product has been identified correctly?
• Track E, field conditions: worn labels, decanted and unlabelled containers, foreign languages and a moving conveyor belt.

A shared language: the EU List of Waste

The truth and the answers are expressed as waste codes from the EU List of Waste (Commission Decision 2000/532/EC), as a judgement of hazardous or not hazardous, and as UN and ADR information. That way no provider's own fractions are favoured. The actual containers at a site differ from site to site and are therefore measured in their own track. The truth comes from the product's own safety data sheet (sections 13 and 14) or from an expert judgement, never from the system being measured.

How results are reported

• Every number has a 95% confidence interval, because a result on a couple of hundred images otherwise says less than it appears to.
• Both coverage and precision are reported, so a system that answers "don't know" to everything does not look better than one that answers.
• Every wrong answer gets a severity class from "correct" to "wrong guidance that can create danger on site". The requirement for the sealed test set is zero dangerous errors, not an average.
• Two systems are compared on the same images with a paired test, not by putting two percentages side by side.
• Results with and without lookup in the provider's own product catalogue are shown separately, because that difference is what says something about the recognition itself.
• Results are shown per track and never as a single overall number.

Data and privacy

The images show products and packaging and nothing else:

• Images with faces, hands, other people or readable personal information are left out.
• All metadata is removed, including GPS, time and device.
• Images from customers' staff are used only in the sealed test set, which is held by a neutral party and is not published.
• Retailer and producer images are not published, because they are protected by copyright.
• The public fields contain no names of customers, sites or employees.

How to take part

1. Read the protocol and run the scorer on the development set.
2. Submit your answers on the sealed set to the neutral evaluator.
3. The result is published per track with the system's version and date.
4. Everyone follows the same rules, Kemipilot included.

Write to kontakt@kemipilot.com to take part or to comment on the proposal.

Status

This is a proposal. No results have been published for any system yet, our own included.

• A pilot set is being built. Tracks B and C are thinner than track A, and we photograph what is missing ourselves.
• The severity classes must be reviewed by chemists before they are final.
• The neutral evaluator has not been agreed yet.
• The licence is proposed but not decided: open code (Apache-2.0) and an open protocol and taxonomy (CC BY 4.0).