New project to map the impact of artificial intelligence on science

It has only taken a few years for generative AI models like ChatGPT to establish themselves as important tools for researchers, academics and professionals creating text, illustrations, and scientific code. Even scientific papers that are co-authored with an AI are now being published.

But how are scientists in fact using these new AI tools also known as large language models? And how will the access to constantly improving artificial intelligence change the ways scientific knowledge is produced and regarded?

Such questions are the focus of a new project, ‘Quantifying the Prevalence and Diffusion of Generative AI in Science’, based at the Copenhagen Center for Social Data Science (SODAS), University of Copenhagen. Thanks to a grant of DKK 2,993,270 from the VILLUM FONDEN it aims at filling the present knowledge gap by combining expertise in computer science and the sociology of science.

“Generative AI has already entered our lives as scientists, academics, and teachers. AI co-authors publications, writes codes, and generates graduate level essays. Yet many uses of AI go undisclosed, leaving us in the dark about who's leveraging it, for what reasons, and to what extent. Our goal is to unveil these aspects, ensuring the scientific community is prepared for the broader implications of AI's pervasive integration,” says Professor Roberta Sinatra who is also affiliated with IT University of Copenhagen.

She is leading the project together with Associate Professor Mathias Wullum Nielsen from the Department of Sociology, University of Copenhagen. They operate with three key objectives:

• First, the project will provide evidence on overt, but also covert usage of large language models in science.
• Second, it will analyse trends in the usage and prevalence of the models across scientific disciplines.
• Finally, it will leverage this comprehensive understanding to predict the diffusion and adoption of generative AI in scientific networks.

AI can change the relationship between science and society

The two-year-long project will include a questionnaire targeting 200,000 publication-active scientists to investigate the use of large language models. The survey will cover both overt and covert, potentially problematic use of AI.

Moreover, the researchers will examine large collections of scientific texts (using so-called text mining) to quantify trends in both overt and covert use of AI-generated content. This will involve techniques that can identify text covertly written by AI models such as ChatGPT.

“We'll employ a blend of surveys, data analysis, and network modeling to estimate the undisclosed adoption of AI. This approach will let us determine whether AI's involvement is associated with more impact and consequently advances the progress of science,” Roberta Sinatra explains.

Large language models will likely increase the pace of scientific discoveries in many fields, but we need to ensure that scientists use these models in trustworthy and transparent ways.

Ultimately, the project hopes to map how the new technology is spreading across scientific communities and analyse its concrete impact, both now and in the future.

For Mathias Wullum Nielsen, this is important. Even more powerful AI models not only have the potential to drastically change the way scientists work and create new knowledge. It can also potentially change the whole relationship between science and the surrounding society.

“Large language models will likely increase the pace of scientific discoveries in many fields, but we need to ensure that scientists use these models in trustworthy and transparent ways. If they don’t, scientific knowledge may lose its legitimacy in broader society in the long run,” he says.