This year, pay attention to all the qualities of scientists

Always welcome in January: hope for the future. I get that hope this year from a policy letter from the Advisory Council for Science, Technology and Innovation (AWTI), published just before Christmas, on the evaluation of science and scientists.

The House of Representatives had requested the letter because of concerns about “recognition and appreciation,” the new way universities try to evaluate their employees. Some scientists criticize this: they think it is too vague, too subjective and too different from what the rest of the world is doing, and therefore bad for the international position of Dutch science. This criticism reached the Chamber, hence the question to the AWTI: how can the quality of science be objectively determined?

No, the advisory board answered, and by the way, that’s the wrong question. At least that is my interpretation, the Council of course did not say that. They write political documents there, not chronicles.

Quality depends on what you consider important

But in the letter, they specify that the scientific quality does not exist. What constitutes scientific quality always depends on what you consider important. Should science solve social problems? Produce fundamental knowledge? Train students? make money? This makes quite a difference to what constitutes a meaningful measure of scientific quality – a successful start-up is a good outcome for a scientist who needs to contribute to Dutch earning power, but not necessarily for a scientist who wants to generate new knowledge. AWTI therefore prefers to speak of qualities, in the plural, which also requires several measures.

These measures can be both quantitative and qualitative. Quantitative measures have been particularly appreciated in recent decades: the number of publications, the number of citations, the amount of the grant received. In the new recognition and appreciation, universities also want to add more qualitative criteria, such as: “I write columns loyalty and in doing so, I contribute to the social debate about how scientists should be evaluated.

Some scientists worry about these qualitative criteria because they are “stories” and they are subjective. It’s true, writes the AWTI, but so are the numbers. Those who only want numbers, for example, put scientists who conduct research to help policy makers at a disadvantage: we don’t have a proper quantitative measure to contribute to policy. And the bare numbers hide the underlying story. Suppose Scientist A publishes three papers per year and Scientist B publishes only one, but Scientist B spends a lot of time reviewing other people’s papers and advising his colleagues, while Scientist A refuses every request, is scientist A really a better researcher?

No measure of quality is completely objective. Too bad, but such is life. Nevertheless, we can try to be as objective as possible and do as much justice as possible to all the different roles and circumstances that scientists have. The new recognition and assessment attempts to do this by combining quantitative and qualitative criteria.

A new way of evaluating is inevitable

The AWTI sees no reason to assume that this new evaluation method threatens the international position of Dutch science. On the contrary: it is precisely international developments that make a new assessment of scientists inevitable. The fear that the Netherlands is isolated internationally is, as the Council has shown, unjustified. However, the Netherlands, along with the UK, Finland and Norway, are leading the way, providing a good opportunity to determine the precise course of the new methods.

The positions, tool boxes, forms and protocols with this course are ready. What matters now is whether individual scientists will follow this path. Almost all scientists are also scientific evaluators: as a member of application or promotion committees, as an evaluator of grant applications, as head of department, as an approver of a dissertation.

My hope for 2023 is that scientists, whenever they assess their colleagues, have an eye for all the qualities that scientists may have, and then dare to estimate those qualities based on both words and on numbers.