Using Large Language Model Applications to Determine Citizen Science Data Compatibility.
Summary
Metadata is essential for the findability and comparison of citizen science projects. Tagging datasets with scientific disciplines or keywords allows filtering and searching on citizen science platforms. However, certain citizen science platforms lack tagging features. Tag annotation can be a time-consuming process and is prone to mistakes. This research aimed to determine whether large language models can aid citizen science platforms with tag annotation, with a specific focus on geodata.
First, ontological frameworks consisting of tags were created by large language model applications. These were generated with different rules and hierarchical structures to determine a proper format for a tag framework. Next, the tags were assigned to citizen science projects based on their descriptions. This was done in six different ways to determine which assignment methods performed better. Finally, the validity of the tags was tested by determining the thematic overlap between projects from various domains. The tested projects were chosen for the use cases of personas.
The assigned tags proved unhelpful for three of four personas use cases. No particular combination of assignment method and ontological framework outperformed other combinations. Difficulties were experienced with generated ontological frameworks, as they all contained tags that were neither applicable to citizen science nor geodata. Assigning tags with large language model applications worked better with simple methods. Determining thematic compatibility with the tags was only helpful for the persona who wished to remove the least relevant projects from their collection. Finding the best matches, identifying a core discipline, or matching two compatible projects was unsuccessful with the assigned tags. This study shows that tagging with LLMAs is possible since one use case benefited from the tags assigned by LLMs. In addition, better results are expected when the ontological frameworks are replaced by smaller lists of tags focused on a narrower scope of topics.
The combination of methods used in this research did not produce quality tags that a citizen science platform would use. The instructions for large language model applications were often too complex, generating unhelpful tag sets. However, not all results were meaningless, and improving the methods used could make large language models a helpful tool capable of generating meaningful tag sets.
Other recommended methodological improvements, such as explainable AI and soft prompting, could enhance LLMA performance in annotating citizen science spatial datasets by refining prompts and mitigating low training data limitations. Additionally, exploring different LLM-based solutions, including fine-tuned or more recent models with updated training data, may yield better tagging outcomes.