Skip to content

Leakage-Free Splits and Consistent CVSS 4.0 Scoring for the VLAI Severity Classifiers

Read the technical report as a PDF.

We are publishing a new version of CIRCL/vulnerability-scores, the dataset behind the VLAI vulnerability severity classifiers used in Vulnerability-Lookup. The new version gives the models a stricter, more reliable evaluation and more consistent severity labels. A technical report describes the changes and the measurements behind them.

What is new

  • Leakage-free splits. A CVE record and its GitHub advisory, or two documents with the same text, now always land in the same split. No vulnerability group spans train, validation and test, and an automatic check verifies this after every rebuild.
  • Consistent CVSS 4.0 labels. Every CVSS 4.0 score is now read as CVSS-B (base metrics only), whatever the source, so the same vulnerability always gets the same severity label.
  • A validation split, so that the test split is used only once, for the final score.
  • Cleaner, reproducible data: repeated documents and rejected CVEs are removed, and a vulnerability keeps its split from one rebuild to the next.

What we learned

In the previous version of the dataset, 58% of the test rows had a training row with the same description, mostly a CVE record and the GitHub advisory for the same CVE. On the test rows it had never seen in any form, our RoBERTa model classifies 73% of the vulnerabilities correctly across four severity classes. That is the figure that matters for new vulnerabilities, and the one the retrained models will be measured against.

The new RoBERTa severity classifier, trained on the new dataset, scores 74.2% on its leakage-free test split, in line with that estimate. From now on, the accuracy reported for our severity classifiers comes from this stricter evaluation.

Thanks

These improvements come from a careful external review by Christian Bucher, who measured both problems and contributed the fixes and the leakage check. Thank you!

The changes are implemented in VulnTrain. Feedback is welcome at info@circl.lu or on GitHub.

Funding

EU Funding

AIPITCH aims to create advanced artificial intelligence-based tools supporting key operational services in cyber defense. These include technologies for early threat detection, automatic malware classification, and improvement of analytical processes through the integration of Large Language Models (LLM). The project has the potential to set new standards in the cybersecurity industry.

The project leader is NASK National Research Institute. The international consortium includes:

Funded by the European Union. Views and opinions expressed are however those of the author(s) only and do not necessarily reflect those of the European Union or the European Cybersecurity Competence Centre. Neither the European Union nor the European Cybersecurity Competence Centre can be held responsible for them.