Automating Research Synthesis with Domain-Specific Large Language Model Fine-Tuning

The process of conducting an SLR (Figure 1) begins with the clear articulation of the review’s purpose and intended outcomes. This foundational step ensures that the review’s scope and objectives are explicit to its audience [29]. Following this, the development of a detailed protocol and the training of the review team are essential to guarantee methodological rigor and consistency in the review execution. The screening process, both for inclusion and quality appraisal, demands transparency in the criteria used for study selection and exclusion, ensuring that only the most relevant and high-quality studies are considered for synthesis. The comprehensive search for literature, coupled with systematic data extraction from the included studies, lays the groundwork for the subsequent synthesis phase. This phase is where the insights gleaned from individual studies are integrated to form a coherent narrative or to distill new theoretical understandings.

Recent progress in AI, Natural Language Processing (NLP), and machine learning has significantly propelled the development of automation tools for supporting literature reviews. Techniques such as text mining and machine learning algorithms have demonstrated potential in automating various stages of the review process, from literature retrieval to data extraction [44]. These tools range from comprehensive research management platforms like Covidence²²2http://www.covidence.org, ASReview³³3https://www.asreview.nl, Mendeley⁴⁴4https://www.mendeley.com, and Zotero⁵⁵5https://www.zotero.org, to specialized tools aimed at specific literature review stages, such as ’connected papers’⁶⁶6https://www.connectedpapers.com/ for literature search and Elicit⁷⁷7https://elicit.com/ for individual paper analysis. Despite the growing availability of these tools, their adoption remains limited, with researchers citing challenges such as steep learning curves and inadequate support [45]. More advanced AI tools provide citation support like Scite ⁸⁸8https://scite.ai/, as well as ScholarAI ⁹⁹9https://scholarai.io/ integration tools with GPT models with numerous features for literature search support.

Meanwhile the body of research dedicated to automating SLR tasks is expanding, with several review papers categorizing these efforts [46, 47, 48, 8, 49, 50, 51, 52, 53, 54, 28]. Nonetheless, the focus on automating the knowledge synthesis phase of SLRs remains minimal [28] which is a reflection of the task’s inherent complexity. Most research in SLR automation employs NLP and Information Retrieval techniques, with tasks such as data extraction and monitoring often framed as text summarization challenges [14, 55].

Retrieval-Augmented Generation (RAG) [68] is a framework that combines the capabilities of large language models (LLMs) with external knowledge sources through a retrieval mechanism. Unlike traditional language models that generate text based solely on the text’s internal representation, RAG models retrieve relevant information from a knowledge base (like a database or the internet) and integrate this into the generation process. The core components of RAG include the retrieval mechanism, which fetches relevant documents or data, and the generative model, which synthesizes the retrieved information into coherent and contextually relevant responses. The theoretical principles underpinning RAG stem from the need to enhance language models with the ability to access and utilize external, structured knowledge [69]. This is in response to the limitations of traditional LLMs that rely solely on their pre-trained parameters for knowledge, which can be outdated or incomplete. The typical architecture of RAG systems involves a retriever that fetches relevant information from a database and a generator that incorporates this information into the final output, thus, the integration of these components allows the model to produce contextually enriched and factually accurate text [68]. The incorporation of RAG into the workflow of LLMs, therefore, presents a sophisticated approach to augmenting the model’s knowledge base beyond its pretraining, specifically tailored to the demands of the evolving nature of studies in SLRs. By allowing the model to access an external corpus of domain-specific literature the context within which the LLM operates, it attains the ability to be enriched while a critical countermeasure to the model’s propensity for generating plausible yet factually incorrect information - hallucination - is mitigated. In the realm of SLRs, where the precision of synthesized knowledge is paramount, RAG’s ability to draws upon relevant information from a targeted corpus helps ensure that the generative outputs of LLMs are anchored in verifiable data.

In training LLMs, they initially undergo pretraining on extensive, diverse datasets, acquiring a foundational grasp of both language and knowledge. This stage equips LLMs with a generalist understanding, however, the leap from generic text generation to specialized domain proficiency requires additional training in the form of fine-tuning which is a process where LLMs are further trained on relevant datasets to align their outputs with new requirements [70].

The literature review underscores the potential and gaps in the intersection of SLRs and LLMs. While advancements in text mining and machine learning herald efficiency in literature retrieval and screening, the synthesis phase remains a challenge that has not been sufficiently addressed, and made difficult by the phenomenon of LLM hallucinations and the inability to audit the source information from the LLMs’ responses. The review reveals the potential of fine-tuning LLMs with SLR-specific datasets to tailor these models for more accurate knowledge synthesis. However, work is required to ensure that both fidelity of synthesized knowledge and auditing of LLM responses can be achieved. Against this backdrop, this paper proposes an SLR-automation Framework to bridge this gap by leveraging domain-specific fine-tuning of LLMs via the most recent advances in PEFT, on selected SLR papers and targetting the knowledge synthesis stage of SLRs, together with novelties for embedding fidelity and mitigating hallucination in the LLM responses. The viability of the proposed framework is explored and illustrated using a case study that will seek to replicate a PRISMA-conforming SLR as a ground truth against which the proposed framework can be evaluated.

Based on the research gaps in the literature, this study attempts to answer the following research questions:

• •

  (RQ1) Can we leverage LLMs in combination with finetuning and/or RAG in order to facilitate the synthesis phase of an SLR process?

• •

  (RQ2) How can the extraction of finetuning datasets from an SLR corpus be effectively automated?

• •

  (RQ3) Can finetuning of LLMs be effectively conducted on relatively small datasets representing a very narrow domain?

• •

  (RQ4) Can we achieve high fidelity of LLM responses and ensure that we can audit and verify the source information of LLM responses, and what evaluation metrics are suitable?

• •

  (RQ5) Are the proposed SLR-automation approaches viable for replicating a previously published SLR study?

The framework initiates with a dual focus on SLR paper selection and research question articulation which is covered by the planning, selection and extraction stages of the SLR process (Figure 1 Steps 1 to 3). Once the target papers for an SLR are identified and the overarching and supporting research questions for the study have been defined, they both then serve as inputs to the next Step 2, and direct the automation process of data extraction and processing.

In this step, we developed a program ¹⁰¹⁰10The tool is publicly available here: https://github.com/peterjhwang/slr-helper that leverages the downstream summarization as well as the question and answering capabilities of an advanced LLM to extract a dataset from all the papers. This program systematically processes each selected paper, using the chosen LLM’s summarization capabilities to extract the core content of primary paper sections and subsections into concise summaries alongside the papers’ meta-data. From these distilled summaries, the program then uses an LLM to automatically generate a series of domain-relevant questions that are common denominators and can be answered by all studies. This set of questions is then used to interrogate each paper and extract answers via a chosen LLM, and the combination of the two forms question and answer (Q&A) pairs that represent our finetuning dataset. Simultaneously, we enable the customization of domain-specific, researcher-defined questions informed and motivated by the objectives and aims of an SLR being undertaken. These questions are tailored to the unique contours of the study’s domain and are formulated to draw out information that is particularly relevant for the SLR’s purposes. This dual strategy ensures that while our approach remains systematically consistent across the majority of academic disciplines, it is also flexible enough to accommodate the specialized requirements of the academic field under investigation.

With both sets of questions, we are then able to automatically generate paper-level Q&A pairs that encapsulate the insights of individual studies at different levels of detail and granularity. To achieve a synthesis that spans the corpus of all studies under investigation, we further collate key summaries from across all papers and from these, generate additional SLR-level Q&A pairs. These pairs are designed to provide responses that are relevant across the entire body of selected works, effectively offering a synthesized narrative that captures the collective information of the SLR papers. Through this process, we ensure that the resulting Q&A dataset is not only rich in detail and precision at the level of individual papers but also possesses the breadth and depth required to reflect the evidence synthesis of a systematic literature review.

The incorporation of distinct knowledge markers, referred to as unique tokens, into the fine-tuning process of LLMs is a theoretical construct designed to create explicit linkages between the model’s augmented neural network parameters and the new knowledge encapsulated from the SLR papers. The motivation behind this approach is to achieve a higher degree of fidelity and precision in the model’s output during inference. When an LLM is fine-tuned with additional data, it assimilates this new information by adjusting a small percentage of its existing parameters while adding a larger set of new parameters. Without a mechanism to distinguish between pre-existing knowledge acquired from pretraining and newly acquired information, the model risks providing responses based on its pretraining rather than the domain-specific insights gained from the subsequent fine-tuning step. Since it is imperative that during inference time, the model does not access facts from its knowledge base developed during pretraining, and instead, only responds with facts acquired during finetuning, an explicit mechanism is needed to ensure this occurs. Without such mechanisms, the results of the SLR risk being contaminated with knowledge that is outside of the scope of the selected papers, thus, invalidating the research. In essence, the aim of the proposed approach is to insulate the LLMs responses from its pretrained knowledge but to nonetheless leverage its natural language capabilities that it gained during pretraining.

Therefore, these tokens or markers serve as signposts within the neural network, directing the model to access and utilize the parameters associated with the SLR data and facts when generating responses. It also enables the model to explicitly state the source information for its responses so that the provenance of information can be audited. This approach provides a method for researchers to verify that the output is indeed based on the recent fine-tuning, enabling a level of interpretability and trust in the model’s responses that is critical for academic rigor and reliability and represents a strategic advancement in using LLMs for SLR.

As part of Step 2, to ensure that the LLMs can handle a variety of linguistic expressions during inference time, we introduce permutations in the Q&A pairs. For each original question, we create several semantically equivalent variations. Since accurate recall of facts is the primary objective of the system rather than style, linguistic quality, and variability of the text, we retain the same answer for each permutation of the corresponding questions. This step is critical for training the LLMs to recognize and correctly respond to different phrasings of the same question which different users are likely to generate.

In the third step of our SLR-automation framework, we propose specific finetuning methods for LLMs, complemented by the integration of RAG processes. To enhance the finetuning efficiency, we propose employing PEFT techniques [71], which modify only a subset of the model’s parameters. This is crucial, as fine-tuning all parameters of large LLMs is computationally prohibitive and may not yield proportionate improvements in performance. PEFT techniques offer a balance, concentrating the model’s learning capacity on those parameters that most significantly influence the domain-specific tasks, thereby optimizing both resource expenditure and model proficiency. RAG, on the other hand, serves as an augmentative mechanism during inference. It dynamically retrieves and incorporates information from the SLR corpus into the LLM’s generative process, potentially providing an additional layer of contextual support. This optional step may ensure that the model’s output is not solely reliant on internalized patterns but is also substantiated by direct data references, thereby potentially enriching the depth and factual accuracy of the synthesized content and responses.

The final phase of the proposed framework tests the framework to ensure that it has the ability to accurately generate responses with respect to the selected SLR papers. The test dataset is extracted from the main Q&A finetuning dataset and withheld from the finetuning process. An automated process is subsequently initiated after finetuning is completed which tests the LLMs ability to answer the selected questions correctly using metrics that are specifically designed for factual validation.

We selected a peer-reviewed and published PRISMA-SLR within the domain of learning analytics as the Gold Standard, with the goal of investigating whether the proposed SLR-automation framework has the capability to generate responses from which a manually conducted SLR can be replicated. The target SLR paper is titled “Learning Analytics Dashboard: A Tool for Providing Actionable Insights to Learners” [76]. This SLR is contemporary, being aligned with the recent advancements in learning analytics dashboards, and was chosen for its adherence to the PRISMA guidelines as well as due to authors’ familiarity with the work. The SLR itself focused on aspects of advancements in educational technology, delving into the impacts of learning analytics dashboards (LADs) on university student outcomes, scrutinizing the technological underpinnings, the analytics methodologies most commonly employed, and the demographics of the study cohorts used in trials. Furthermore, it examined the existing gaps and posited future trajectories for research in this field. This SLR encompassed 17 papers. Its selection was strategic not only because it aligned with the expertise of this study’s authors which assisted in guaranteeing the veracity of LLM responses, but also offered an insight into the viability and effectiveness of finetuning LLMs on relatively small datasets.

Step 2 of the proposed framework in Figure 2 was fully automated in order to generate a finetuning dataset. A software tool was built using Python to systematically parse the 17 PDF files representing the SLR papers and generate Q&A pairs for the finetuning dataset. To begin with, this tool extracted the metadata from each paper via OpenAI’s GPT-4 API calls and before data generation commenced, it created a high-level summary of each paper. The summaries of each paper were then collated, and GPT-4 was tasked with generating a set of questions that were relevant across all the papers. This produced a vehicle for automatically extracting domain-specific information that was common across all the studies. These questions were then augmented with a further set of researcher-defined questions that reflected the narrower aims of the SLR study and were closely aligned to the objectives and overarching questions of the SLR being conducted. Once this broad set of questions was collected, GPT-4 was used via API calls to generate answers to each one from every study. These initial Q&A pairs we refer to as paper-summary-level data points. These data points captured high-level paper summaries that were relevant to both the research domain and the aims of the SLR study.

The next phase involved the generation of another set of Q&A data points based on data from individual studies at different levels of granularity. In this phase, our processing divided each paper into chunks¹¹¹¹11The chunks usually comprised key sections of each paper., and asked GPT-4 to directly generate Q&A pairs based on the content of each chunk. This method yielded another dimension of information from each paper that reflected some more distinct aspects of each study alongside, like their expanded and more detailed contributions, idiosyncratic methodological details, key findings, and future research directions. This process was then repeated with even smaller chunks of text which roughly represented individual paper paragraphs. This more detailed approach allowed for the extraction of even more granular information from each study in the form of Q&A pairs, revealing yet finer details of the methodologies, analyses, and discussions presented in the studies. By dissecting the papers at a paragraph level, this method augmented the depth of the overall finetuning dataset with a more detailed set of Q&A pairs. This layer, we collectively refer to as paper-level Q&A pairs that offer a more granular insight into each study compared to paper-summary-level data points.

While the above strategy generated data points that would enable researchers to “talk to” the content of each paper, the subsequent strategy aimed to allow researchers to “talk across” all the studies simultaneously. These Q&A pairs we refer to as SLR-level data points which served the with the aim of facilitating the answering of questions that relate to the synthesis of knowledge across all the studies under investigation. From the implementation perspective, this second layer of data extraction involved collating answers from each paper that were generated from questions used for creating paper-summary-level Q&A pairs. Then, Q&A pairs were generated via GPT-4 based on all the collated answers. By combining paper and SLR-level Q&A pairs, our finetuning dataset not only facilitated equipping LLMs with in-depth facts from singular studies, but also attempted to make possible LLM prompts that support queries requiring a comprehensive synthesis of knowledge and evidence across the entire SLR corpus, achieving both the breadth and depth needed to support a systematic study of a corpus of literature and to answer specific research questions.

The initial training dataset following the data extraction and Q&A pairs generation comprised 1503 samples and of these, approximately 28% were paper-summary-level, 49% were paper-level extracted from larger chunks, 31% were paper-level extracted from paragraphs, and 2% from the SLR-level. To augment the robustness of the finetuned LLMs during inference, we implemented a novel refinement step: for each original question, we generated ten alternative formulations. These variants were designed to be semantically diverse yet structurally aligned, ensuring that despite the variability in phrasing, the essence of the questions remained intact, eliciting the same precise response which resulted in producing a final training dataset of 16423 samples. This strategic augmentation served a dual purpose. Firstly, it aimed to significantly enhance the model’s ability to comprehend and respond accurately to a broader spectrum of question formulations from different users, mirroring the diverse linguistic styles encountered in real-world academic discourse. Secondly, it aimed at increasing the size of the dataset. For the test dataset, we generated three additional permutations for each unique question to arrive at a final dataset of 4962 data points, where the distribution of paper-level and SLR-level questions was the same as that in the training dataset. Table 1 demonstrates an example of a text summary (answer), for which 11 different but semantically equivalent questions were generated using GPT-4 for the training dataset and an additional three for the test dataset.

In the experimentation of the proposed framework, the Mistral-7B base model [77] was selected as the foundation for fine-tuning due to its efficiency and high benchmark performance, particularly in English language tasks. This 7.3 billion parameter model leverages advanced techniques such as Grouped-query attention and Sliding Window Attention, enabling it to outperform larger models across a variety of benchmarks, including commonsense reasoning, making it an ideal candidate for fine-tuning to suit our specific SLR automation needs. Conversely, the Mistral-7B-Instruct variant, already fine-tuned for instructional tasks, was employed in conjunction with the RAG approach to leverage its capabilities in handling Q&A instructional content. The Instruct model’s demonstrable adaptability and performance on tasks like MT-Bench, coupled with its ability to generalize across tasks, provided an appropriate candidate for generating accurate and contextually relevant responses in our SLR framework experimentation.

In our study, we implemented two RAG systems, leveraging the Weaviate vector database (version 1.23.9) and the OpenAI Embedding model ‘text-embedding-3-large‘¹²¹²12This embedding model was notable for its efficiency and ranked 7th on the HuggingFace leaderboard at the time of writing this paper. for pre-retrieval processes. For the post-retrieval stage, the initial model used was OpenAI’s ‘gpt-3.5-turbo‘. The pre-retrieval phase involved optimizing the indexing process by chunking data using LLMs, focusing on contextually relevant segments. The vector database was structured with the attributes ‘text‘ and ‘source‘, where ‘text‘ was searchable and ‘source‘ was filterable. During the retrieval phase, we utilized Langchain’s self-querying retriever for its simplicity¹³¹³13Future implementation intentions are to upgrade to a hybrid search model for enhanced accuracy and more refined filtering mechanisms.. In the post-retrieval process, we employed an aggregation prompt template designed to integrate the retrieved context into a coherent and concise answer, limited to three sentences for any given question.

We experimented with five different combinations of finetuning and RAG in order to determine what the most effective approaches could be for the SLR-automation process. Our experimental setup investigated the following methodologies:

1. 1.

   Baseline: Evaluate the Mistral-7B-Instruct on its knowledge and ability to answer SLR-related test dataset questions.

2. 2.

   Fine-tuning LLMs using LoRA: Leveraging Low-Rank Adaptation for fast and efficient parameter adjustment.

3. 3.

   Fine-tuning LLMs using NEFTune: Introducing noise into embedding vectors to investigate effects on generalization improvements.

4. 4.

   Instruct LLM + RAG with Raw Articles: Combining LLMs with Retrieval-Augmented Generation, using unprocessed article text as the retrieval corpus.

5. 5.

   Instruct LLM + RAG with Auto-Extracted Data: Employing RAG with a knowledge base of automatically extracted data comprising the finetuning dataset for focused information retrieval.

6. 6.

   Best Finetuned LLMs + Best RAG Solution: Integrating the top-performing fine-tuning and RAG methods to optimize SLR automation.

Each method also summarised in Table 3, was evaluated for factually correct answers with respect to the SLR dataset.

Fine-tuning and RAG execution were performed on a Linux system powered by an AMD EPYC 7713 64-bit 8-Core Processor with 259GB of RAM. For the training, we utilized an NVIDIA A40 GPU, which is built on an 8 nm process and equipped with 48 GB of GDDR6 memory. This GPU features 10,752 shading units, 336 tensor cores, and 84 ray tracing cores, supporting a base clock speed of 1305 MHz and a boost clock up to 1740 MHz. The total power consumption of the GPU is 300 W. The training processes for both NEFTune and LoRA models took approximately 70 hours to complete 150 epochs each.

Our evaluation necessarily centered around assessing the factual accuracy of responses generated by the various methodologies applied to the SLR dataset. This assessment was conducted through a combination of quantitative and qualitative analyses, utilizing test sets.

We begin first by establishing the validity of the devised evaluation metrics. Tables 5 and 5 show the correlation matrices analyzing the inter-rater reliability among the two independent human evaluators’ (H1 and H2) scores as well as that of GPT-4, showing the utility and limitations of both CGS and FEVER within the context of factuality assessment.

The CGS metric displays moderate to strong correlations, particularly with the H2 rater exhibiting a robust 0.74, suggesting a substantial concurrence with the GPT-4 evaluations. H1 demonstrated a moderate agreement with a correlation of 0.65 with GPT-4. These figures are noteworthy, given the subjective nature inherent in qualitative evaluations. The inter-rater reliability, represented by a correlation of 0.67, although not indicating perfect agreement, nevertheless reveals a meaningful consensus between the human raters that overall affirms the CGS as a reliable tool within its application scope.

Contrastingly, the FEVER metric was characterized by lower correlation values and indicated a more modest agreement between evaluators – which is a reflection of the challenges faced in calibrating subjective analytical tools in general. With H1 at 0.49 and H2 at 0.6 correlation with GPT-4, there was discernible inter-evaluator consonance, albeit less pronounced than that within the CGS metric. The inter-rater correlation of 0.56, while significant, underscored the need for further evaluator calibration to mitigate interpretive disparities. Upon a deeper investigation into the divergences, the results showed that there were significant differences in the manner in which the human raters evaluated the ’NOT ENOUGH INFO’ category with respect to each other and GPT-4, with both human raters underrating this category and preferring either the SUPPORTED or REFUTED categories instead. Thus, while the FEVER metric offers meaningful insights into the veracity of claims for the analysis in this study, its application necessitates a more critical interpretation. For the purposes of this study, CGS can be relied upon to provide a broader spectrum of consistency in evaluation, and FEVER serving as a barometer for broader factual accuracy.

The quantitative outcomes, encapsulated in Table 6, show the performance across REFUTED, NOT ENOUGH INFO, and SUPPORTED categories, thereby offering a granular perspective on the factual alignment of each method. All results are rank-ordered with respect to the SUPPORTED category. We first draw attention to the results of the Baseline method which represents responses from the unmodified Mistral-Instruct LLM and indicates the potential existence of any SLR-relevant prior knowledge arising from its pretraining and instruct-tuning processes, while also serving as a benchmark to determining the degree to which proposed approaches have succeeded in achieving an improvement. The baseline indicates the ability to correctly respond to 14.5% of the test data, with the majority of the responses being categorised as NOT ENOUGH INFO. We see next that the NEFTune method emerged as the most effective, with 89.2% of its responses being SUPPORTED, indicative of its strong factual accuracy and alignment with the actual SLR corpus. LoRA, with an 87.7% ‘SUPPORTED‘ rate, also demonstrated significant reliability, albeit marginally lower than NEFTune. In contrast, the integration of RAG with a fine-tuned dataset (RAG + FD) presents a discernible shift in distribution across all three categories, with a notable increase in ‘NOT ENOUGH INFO‘ outcomes and a decrease in ‘REFUTED‘ responses compared to NEFTune and LoRA. The combination of NEFTune and RAG + FD, however, exhibited a clear reduction in factual accuracy, marked by a SUPPORTED rate of 63.2%. The sharply elevated NOT ENOUGH INFO and REFUTED rates may reflect the inability of the finetuned LLM to integrate additional text from the RAG process. The low RAG (raw) outcomes underscored the challenges inherent in leveraging unmodified RAG approaches and basic methodologies in achieving accurate retrieval for the purpose of synthesis of SLR-specific information that was generated during the data extraction process.

The CGS results, detailed in Table 7, offer a more fine-grained perspective on the consistency of the responses. This scale ranges from -2 (Contradictory) to 2 (Fully consistent), providing a more detailed view of the alignment between the generated responses and the SLR corpus. Similar to the FEVER results, in the CGS analysis NEFTune again emerged as the leader with the highest percentage of ’Fully consistent’ responses and an overall mean score of 1.75 (87.6%), indicating a predominance in the factual fidelity of the responses. LoRA closely followed with an overall mean score of 1.72 (86.0%). The method combining RAG with a fine-tuned dataset (RAG + FD) achieved a moderately lower mean score of 1.67 (83.7%), reflecting a competent performance but with room for possible improvement via the adjustable RAG parameters, particularly in enhancing the proportion of ‘Fully consistent‘ responses. NEFTune combined with RAG + FD exhibited a notable drop in performance, as evidenced by a mean score of 1.14 (57.2%), which suggests that the integration of RAG inputs with the finetuned models was likely introducing confusion in response generation. This significant decrease highlights the challenges in combining these two otherwise effective methodologies without compromising factual integrity. RAG (raw) and the baseline methods manifested the lowest consistency levels in line with previous results, with mean scores of 0.46 (23.2%) and 0.164 (8.2%) respectively.

The correlation analysis between the responses of various LLM methodologies is presented next in order to explore similarities and divergences in the LLM responses that may offer insights complementary nature of each where there is evidence of them capturing different aspects of the data, leading to the potential of combining them to achieve better accuracies. The correlation matrix represented in Table 8 offers a refined perspective on the ordinal associations between different LLM FEVER responses.

A positive k value between NEFTune and LoRA indicates a moderately similar ranking order in their response evaluations. Conversely, a near-zero k value between NEFTune and the baseline method suggests an independent performance trend, underscoring NEFTune’s distinct methodological improvements over the benchmark. The correlations between responses of all other methods were not significant.

Similar to the previous table, the CGS correlation Table 9 again showed a moderately positive correlation between NEFTune and LoRA (0.342), indicating that these methods share exhibited overlapping learning patterns on the underlying finetuning dataset. The near-zero correlation between NEFTune and the baseline (-0.004) reinforces the distinct advancement NEFTune contributes. This independence from the baseline method and other approaches highlights NEFTune’s effectiveness in effectively encoding the finetuning dataset SLR.

We next examine and compare responses of the top three best-performing approaches on two sample test questions from FEVER, showcasing an example where each method’s response was classified as NOT ENOUGH INFO and REFUTED. With respect to the NOT ENOUGH INFO responses in Table LABEL:tab:FEVERNoTEnough, we find variances in detail and emphasis. In response to the question asking what manner does the dashboard contribute to the advancement of virtual learning, the reference text highlights the dashboard’s role in managing lab activity schedules and resource usage on a shared AWS account, focusing on logistical aspects. In contrast, LoRA and NEFTune responses pivot towards the dashboard’s educational support, emphasizing feedback, self-regulation, and progress monitoring, with NEFTune slightly touching on resource management but not to the extent mentioned in the reference. RAG + FD, while partially aligning with the reference by discussing resource usage management, also diverged by emphasizing feedback and activity completion, omitting the specific detail about scheduling. Thus, none of the responses fully encapsulated the reference’s emphasis on time slot management and resource usage, illustrating a shift from logistical to educational functionalities in their interpretations. In these instances the classification of NOT ENOUGH INFO for all responses was correct.

Meanwhile, with respect to the examples of REFUTED responses for the same question which requested details of a specific study’s insights on how the learning analytics dashboard (LAD) drove the motivation among students in Table LABEL:tab:FEVERRefuted, we see that LoRA and NEFTune produced identical responses and closely match the reference text. However, while they correctly reported most details, they omitted the detail that intrinsic motivation actually decreased in both groups, as noted in the reference text. RAG + FD introduced a claim not present in the reference text, suggesting that LADs lead to a higher academic performance evidenced by higher final grades. Similar to LoRA and NEFTune, the response mentioned the increased extrinsic motivation in the treatment group and the lack of significant difference in changes in intrinsic motivation between the groups. However, it also omitted that intrinsic motivation decreased in both groups over time. Again, in these instances the FEVER classification of REFUTED derived by GPT-4 was correct for all responses.

For the best-performing model from the above results, Tables 12 and 13 offer a vantage point to assess NEFTune model’s proficiency across various levels of analysis. The NEFTune model’s capability is particularly observable at the paper-level Q&As, with 83.0% of responses achieving ’Fully Consistent’ ratings (CGS=2) as shown in Table 12. Likewise, at the paper-summary-level, NEFTune ability to respond with high fidelity is further confirmed by a ’Fully Consistent’ score in 95.0% of cases. When combined with ’Partially consistent’ results, NEFTune’s scores rise to 93% and 98% accuracy for paper-level and paper-summary-level respectively. However, for SLR-level questions, NEFTune’s ’Fully Consistent’ does significantly reduce to a score of 52%, but again, in combination with ’Partially consistent’ results, this rises to 84%, still indicating a strong level of dependability.

In the FEVER metric results in Table 13, the results do harmonize with the CGS findings, demonstrating the model’s robust accuracy with a majority of ’Supported’ classifications at both the paper level (87.7%) and paper-summary level (95.5%). Again, a divergent scenario unfolds at the SLR-level, where a considerable 42.7% of responses are categorized as ’Refuted’.

The analyses of CGS and FEVER metrics, taken conjointly, paint a comprehensive overview of NEFTune’s strengths. While the model’s performance is robust in factual replication at paper-level Q&As, the performance on SLR-level data points suggests an area for improvement in either the calibration of the FEVER metric, or in optimizing the learning for these data points by increasing the size of the finetuning dataset for these samples. However, it should be noted that despite the lower accuracies and a reduced level of confidence in the model’s outputs on the SLR-level questions, the SLR-level questions can nonetheless be verified and accurately reconstructed from more accurate paper-summary-level responses.

The qualitative evaluations conducted manually across the eleven NEFTune LLM responses to questions about key SLR findings in the target paper representing the ground truth, reveal a high level of fidelity and alignment between the reference and the LLM’s responses. An example of a typical response can be seen in Table 14. The analysis showed that in eight out of eleven cases, the key findings from the SLR ground truth could be faithfully extracted and reconstructed from the responses provided by the finetuned LLM’s responses. However, in two instances, a deeper understanding of certain domain-specific concepts such as prescriptive analytics and model interpretability within the context of learning analytics was crucial to be able to arrive at ground truth conclusions from the LLM responses. These instances highlighted a divergence in the conceptualization of certain technical terms between the LLM and the SLR study, necessitating domain expertise to align the LLM’s broader interpretations with the SLR’s findings. Nonetheless, it can be concluded that in 90% of the cases, the ground truth findings from the SLR study are supported by the LLM’s responses. However, there was one notable LLM response that introduced some inaccuracies in its response concerning the number of participants across all the studies in 17 papers. It was only in his response that a deviation was observed and in this instance, it could be confirmed that the LLM response did not support the ground truth findings in the SLR study. Example responses from the qualitative analysis can be seen in Appendix A.

The analyses consistently demonstrate that the finetuned LLM’s responses tend to provide a broader and more detailed perspective than ground truth which is often more concise and sometimes more definitive in its assertions. Below is a synthesis of the overarching patterns observed:

1. 1.

   Expansion and Detailing: The LLM frequently expands upon the information provided in ground truth introducing additional context, examples, and details that were not explicitly mentioned in the ground truth. This pattern is evident in analyses where the LLM’s responses offer a deeper dive into specific studies, technologies, and outcomes associated with LADs, enhancing the understanding of the subject matter beyond the baseline established by ground truth. However, the additional information and details are correct and do not indicate hallucination.

2. 2.

   Alignment with Core Assertions: Despite the additional details, the LLM’s responses generally align with the core assertions or findings of ground truth. This alignment indicates that the finetuned LLM successfully captures the essence of the ground truth findings, while adding value through elaboration and exemplification.

3. 3.

   Discrepancies in Emphasis and Scope: Some discrepancies arise primarily from differences in emphasis and scope. For instance, while the ground truth response might highlight a lack or absence of certain features or trends within the LADs research, the LLM occasionally points out exceptions or minor trends that counteract these broad strokes. These discrepancies do not necessarily contradict ground truth but rather suggest a more variegated picture, which can give the researcher more scope for expansion in an academic paper.

4. 4.

   Reconstruction and Completeness: The degree to which the ground truth can be reconstructed from the LLM’s responses is generally very high, reflecting the LLM’s ability to both mirror and in many responses, extend the ground truth in the target paper. In cases where the LLM’s detailed accounts align closely with ground truth, the reconstruction ability is high, indicating a strong corroboration of ground truth by the LLM.

This study’s exploration of LLMs in SLRs was constrained by model selection, focusing on Mistral without assessing larger models over 7 billion parameters that might offer improved outcomes. Future research should broaden its scope to include a variety of open-source LLMs, evaluating their efficacy in SLR contexts. Dataset access presents a significant limitation, as copyright restrictions by publishing houses impede the public sharing of even summarised content, challenging the reproducibility and validation of our findings. Addressing this issue requires navigating copyright laws to facilitate wider academic verification and exploration. The use of GPT-4 for evaluation reflects the current state of technology, yet the rapid advancement in LLMs suggests that future iterations will necessitate updated prompts and methodologies to maintain relevance and accuracy in assessments. A limitation of using proprietary models like GPT-4 is also cost, which can and should be bypassed in the future using locally running LLMs for data extraction and evaluation. The study’s focus on singular-topic questions limits the depth of evaluation regarding the models’ ability to handle complex queries involving multiple concepts. Enhancing the CGS and FEVER metrics calibration through detailed training could yield more reliable evaluations, a necessity for future studies to consider. Methodologically, the study leaned towards qualitative meta-synthesis, leaving a gap in quantitative meta-analysis exploration. Addressing this gap, alongside refining hyperparameters and expanding computational resources to better support extensive data processing, forms a critical pathway for subsequent research. Performing hyperparameter tuning was particularly challenging given the long training runtimes for model finetuning, and it is highly likely that suboptimal hyperparameters were used in this research as a result. Future work should encompass large-scale replication studies to confirm and extend these initial findings, incorporating both qualitative and quantitative SLR meta-analyses. This includes investigating LLMs’ multimodal capabilities in extracting information from diverse formats such as tables and figures.