New publication on Artificial Intelligence and Scientific Understanding

The paper ‘Towards a Benchmark for Scientific Understanding in Humans and Machines’ has been published in Minds and Machines. The article was written by Kristian González Barman, Sascha Caron, Tom Claassen and Henk de Regt and results from the project “Machine learning in science: bridging the gap between data and understanding”, funded by a voucher from the Faculty of Science’s Interdisciplinary Research Platform.

The abstract of the paper is as follows: Scientific understanding is a fundamental goal of science. However, there is currently no good way to measure the scientific understanding of agents, whether these be humans or Artificial Intelligence systems. Without a clear benchmark, it is challenging to evaluate and compare different levels of scientific understanding. In this paper, we propose a framework to create a benchmark for scientific understanding, utilizing tools from philosophy of science. We adopt a behavioral conception of understanding, according to which genuine understanding should be recognized as an ability to perform certain tasks. We extend this notion of scientific understanding by considering a set of questions that gauge different levels of scientific understanding, covering information retrieval, the capability to arrange information to produce an explanation, and the ability to infer how things would be different under different circumstances. We suggest building a Scientific Understanding Benchmark (SUB), formed by a set of these tests, allowing for the evaluation and comparison of scientific understanding. Benchmarking plays a crucial role in establishing trust, ensuring quality control, and providing a basis for performance evaluation. By aligning machine and human scientific understanding we can improve their utility, ultimately advancing scientific understanding and helping to discover new insights within machines.

Scale of properly answered questions, from no understanding to new scientific understanding (discovery)

Contact information

Organizational unit
Institute for Science in Society