Enhanced Short Text Modeling: Leveraging Large Language Models for Topic Refinement

Abstract.

Crafting effective topic models for brief texts, like tweets and news headlines, is essential for capturing the swift shifts in social dynamics. Traditional topic models, however, often fall short in accurately representing the semantic intricacies of short texts due to their brevity and lack of contextual data. In our study, we harness the advanced capabilities of Large Language Models (LLMs) to introduce a novel approach termed ”Topic Refinement”. This approach does not directly involve itself in the initial modeling of topics but focuses on improving topics after they have been mined. By employing prompt engineering, we direct LLMs to eliminate off-topic words within a given topic, ensuring that only contextually relevant words are preserved or substituted with ones that fit better semantically. This method emulates human-like scrutiny and improvement of topics, thereby elevating the semantic quality of the topics generated by various models. Our comprehensive evaluation across three unique datasets has shown that our topic refinement approach significantly enhances the semantic coherence of topics.

1. Introduction

Figure 1. Overview of topic refinement ($N=10$): ①Prompt Construction; ②LLM Request and Response; ③Iterative Refinement.

Short texts, such as online comments and news headlines, are influential in shaping public opinion and mirroring social trends (Pang et al., 2021; Zhang et al., 2022, 2023). Analyzing these texts can uncover underlying patterns and topics, providing insights into social discourse. Topic modeling emerges as a valuable tool to extract topics from vast data (Lin et al., 2020; Wang et al., 2023). Nevertheless, the sparsity of context in short texts poses substantial challenges for conventional topic models (Blei et al., 2003; Yang et al., 2021). This brevity leads to low information density (Qiang et al., 2022), making it difficult to accurately capture and represent the semantics of discussed topics.

Some advancements in existing methods have been made to alleviate data sparsity. Early studies (Cheng et al., 2014; Li et al., 2016) address this challenge by enriching word co-occurrence within the corpus, thus discovering broader linguistic patterns. Later, researchers begin to incorporate external knowledge sources (Grootendorst, 2022; Meng et al., 2022; Zhang and Lauw, 2022; Churchill et al., 2022), like word embeddings (Wang et al., 2020; Wu et al., 2023; Zuo et al., 2023), into topic modeling to sufficiently capture semantic relationships. Another line of work focuses on generating quantified topic distributions tailored for short texts (Wu et al., 2020, 2022). These methods effectively aggregate short texts with similar topics into the same topic distribution. Recently, contrastive learning has been explored to further mitigate the effects of data sparsity (Nguyen and Luu, 2021; Han et al., 2023; Zhou et al., 2023). These models rely on data augmentation to learn from both similar and dissimilar text pairs, thereby enhancing the learning signals from short texts. Despite these advancements, existing methods frequently yield suboptimal semantic coherence of topics and impair their interpretability in practical applications.

To narrow these gaps, we turn to the booming and widely concerned Large Language Models (LLMs) such as PaLM (Chowdhery et al., 2023), GPT (Brown et al., 2020; Ouyang et al., 2022), and others. With their unprecedented semantic comprehension and generating capabilities (van Dijk et al., 2023; Tan et al., 2023; Ye et al., 2023), trained across extensive datasets, LLMs are expected to further improve the quality of topic modeling for short texts. Capitalizing on this potential, we introduce a novel model-agnostic mechanism: Topic Refinement. More concretely, we focus not on involving in topic mining but rather leverage LLMs via prompt engineering to refine topics initially mined by base topic models. This mechanism systematically iterates over each topic and constructs prompts to request LLMs. It sequentially assumes a word as the potential intruder word in a topic while the remaining words represent that topic. Prompts with LLMs are then employed to identify the topic and assess if the word aligns with the semantic expression of the other words. If alignment is confirmed, the word is retained; otherwise, coherent words are generated as candidates to replace the intruder word. Essentially, topic refinement mimics the human process of identifying, evaluating, and refining the extracted topics to make them more coherent and easier for users to discern.

Overall, the main contributions of our work are as follows:

• We introduce Topic Refinement, a model-agnostic mechanism for short text topic modeling, pioneering a new pathway to improve topic quality after initial mining.

• Topic refinement innovatively integrates LLM-based prompt engineering with diverse topic models to precisely refine and optimize the representative words for each topic.

• Extensive experiments are conducted on three short text datasets to evaluate the effectiveness of our mechanism, particularly in enhancing the semantic coherence of topics.

2. Proposed Method

2.1. Problem Definition

Topic refinement aims to enhance the semantic coherence and clarity of topics generated from base topic models for short texts. Formally, given a set of topics $T=\left\{t_{1},t_{2},\cdots,t_{K}\right\}$ discovered by a base topic model, each topic consists of $N$ representative words, denoted by $t_{i}=\left\{w_{i1},w_{i2},\cdots w_{iN}\right\}$, where $1\leq i\leq K$. Within this context, an intruder word is defined as a word within a topic that is not semantically coherent with the other words, thus disrupting the coherence. The topic refinement mechanism seeks to transform each topic $t_{i}$ into a refined topic $t^{\prime}_{i}$ by correcting such intruder words $w_{ij}$ in its word set $t_{i}$. This transformation is guided by a refinement function $\mathcal{R}$, which determines whether to retain or replace the potentially intruder word $w_{ij}$ in $t_{i}$ as follows:

(1) $$\mathcal{R}:\left(t_{i}\setminus\{w_{ij}\},w_{ij}\right)\to w^{\prime}_{ij},$$

where $w^{\prime}_{ij}$ is the alternative word for topic $t_{i}$. We propose to model the refinement function via LLMs, leveraging their advanced capabilities in understanding and generating.

2.2. Technical Details of Topic Refinement

Table 1. Main results on three datasets. TR refers to the topic refinement mechanism.

Base Models	AGNews					TagMyNews					YahooAnswer
	$C_{A}$	$C_{P}$	$C_{V}$	$UCI$	$NPMI$	$C_{A}$	$C_{P}$	$C_{V}$	$UCI$	$NPMI$	$C_{A}$	$C_{P}$	$C_{V}$	$UCI$	$NPMI$
LDA	0.174	0.220	0.373	0.062	0.036	0.189	0.237	0.391	0.021	0.038	0.190	0.278	0.378	0.510	0.063
+ TR	0.200$\uparrow$	0.330$\uparrow$	0.394$\uparrow$	0.472$\uparrow$	0.064$\uparrow$	0.216$\uparrow$	0.384$\uparrow$	0.409$\uparrow$	0.615$\uparrow$	0.077$\uparrow$	0.220$\uparrow$	0.364$\uparrow$	0.396$\uparrow$	0.776$\uparrow$	0.086$\uparrow$
BTM	0.166	0.153	0.367	-0.034	0.023	0.177	0.142	0.389	-0.594	0.005	0.172	0.185	0.365	0.219	0.036
+ TR	0.205$\uparrow$	0.349$\uparrow$	0.395$\uparrow$	0.702$\uparrow$	0.074$\uparrow$	0.216$\uparrow$	0.357$\uparrow$	0.409$\uparrow$	0.445$\uparrow$	0.066$\uparrow$	0.204$\uparrow$	0.308$\uparrow$	0.389$\uparrow$	0.527$\uparrow$	0.065$\uparrow$
G-BAT	0.181	0.192	0.398	-0.622	0.012	0.216	0.332	0.407	0.302	0.061	0.213	0.265	0.399	0.304	0.062
+ TR	0.204$\uparrow$	0.337$\uparrow$	0.403$\uparrow$	0.215$\uparrow$	0.056$\uparrow$	0.237$\uparrow$	0.414$\uparrow$	0.424$\uparrow$	0.719$\uparrow$	0.087$\uparrow$	0.235$\uparrow$	0.386$\uparrow$	0.415$\uparrow$	0.730$\uparrow$	0.090$\uparrow$
CNTM	0.266	0.469	0.448	0.744	0.101	0.293	0.515	0.475	0.608	0.103	0.294	0.534	0.455	1.311	0.141
+ TR	0.279$\uparrow$	0.528$\uparrow$	0.458$\uparrow$	1.019$\uparrow$	0.118$\uparrow$	0.303$\uparrow$	0.575$\uparrow$	0.484$\uparrow$	1.029$\uparrow$	0.124$\uparrow$	0.301$\uparrow$	0.554$\uparrow$	0.461$\uparrow$	1.420$\uparrow$	0.147$\uparrow$
TSCTM	0.205	0.155	0.437	-1.705	-0.022	0.209	0.153	0.465	-1.911	-0.027	0.215	0.234	0.437	-0.919	0.016
+ TR	0.238$\uparrow$	0.379$\uparrow$	0.442$\uparrow$	-0.001$\uparrow$	0.060$\uparrow$	0.242$\uparrow$	0.332$\uparrow$	0.468$\uparrow$	-0.568$\uparrow$	0.039$\uparrow$	0.239$\uparrow$	0.394$\uparrow$	0.442$\uparrow$	0.173$\uparrow$	0.072$\uparrow$
BERTopic	0.228	0.295	0.442	-0.458	0.038	0.217	0.287	0.459	-1.090	0.012	0.252	0.421	0.448	0.533	0.094
+ TR	0.257$\uparrow$	0.433$\uparrow$	0.451$\uparrow$	0.401$\uparrow$	0.083$\uparrow$	0.235$\uparrow$	0.416$\uparrow$	0.469$\uparrow$	-0.257$\uparrow$	0.054$\uparrow$	0.258$\uparrow$	0.457$\uparrow$	0.451$\uparrow$	0.730$\uparrow$	0.105$\uparrow$
ECRTM	0.203	0.125	0.443	-2.387	-0.053	0.190	0.086	0.451	-2.623	-0.064	0.247	0.315	0.454	-0.829	0.027
+ TR	0.254$\uparrow$	0.464$\uparrow$	0.447$\uparrow$	0.243$\uparrow$	0.072$\uparrow$	0.235$\uparrow$	0.347$\uparrow$	0.443	-0.509$\uparrow$	0.036$\uparrow$	0.292$\uparrow$	0.496$\uparrow$	0.470$\uparrow$	0.590$\uparrow$	0.099$\uparrow$

Figure 1 depicts the overall architecture of our topic refinement mechanism. Prior to this mechanism, we extract a set of topics $T=\left\{t_{1},t_{2},\cdots,t_{K}\right\}$ and the vocabulary $V$ from short texts through a base topic model. This base model can utilize existing methods in the field, such as probabilistic topic models or neural topic models. Consequently, our mechanism unfolds in three phases: prompt construction, LLM request and response, and iterative refinement.

Figure 2. LLM prompt template for topic refinement.

Prompt Construction. Prompt engineering serves as the direct way to guide and control the output generated by LLMs. To model the refinement function $\mathcal{R}$, our prompt template is designed to guide LLMs to find intruder words in each topic and generate alternative words related to this topic. Given the topic $t_{i}$ and the words $\left\{w_{i1},w_{i2},\cdots w_{iN}\right\}$ in it, we structure the prompt template into two tasks. Firstly, we employ LLM $\mathcal{M}$ to identify the common topic shared by the ¡TOPIC_WORDS¿, which is represented by words $t_{i}\setminus\{w_{ij}\}$. Later, LLM $\mathcal{M}$ assesses whether the word $w_{ij}$, denoted as ¡WORD¿, is semantically consistent with the previously determined common topic. In cases of inconsistency, LLM $\mathcal{M}$ is also tasked with suggesting ten commonly used words closely related to that topic as candidate alternatives. For the clarity and diversity of topics, these candidate words should be easily recognizable and distinctly different from those in the provided list. Moreover, to facilitate subsequent processing of generated results, the prompt template also enforces the format of generated texts to be in JSON. The prompt template is defined as $p=\mathrm{Prompt}$(¡TOPIC_WORDS¿, ¡WORD¿), which is used to construct each iteration of prompts $p$ for LLM $\mathcal{M}$. The details of the prompt template are presented in Figure 2.

LLM Request and Response. After constructing the prompt $p=\mathrm{Prompt}(t_{i}\setminus\{w_{ij}\},w_{ij})$, we initiate requests to LLM $\mathcal{M}$ for its response. Essentially, the LLM models the topic refinement function by estimating the conditional probability of the generated text: $\mathrm{P_{LLM}}(s_{1},s_{2},\cdots,s_{n}\mid p)$, where $(s_{1},s_{2},\cdots,s_{n})$ is the generated word sequence with variable lengths. The generated sequence, formatted in JSON, is then converted into our desired results by a simple mapping function $(r_{1},r_{2},r_{3})=f(s_{1},s_{2},\cdots,s_{n})$. $r_{1}$ represents the topic identified by LLM $\mathcal{M}$, typically composed of one word or phrase summarizing this topic. $r_{2}$ is the judgment of whether the ¡WORD¿ in the prompt $p$ is semantically coherent with the identified topic, with the response being either ”Yes” or ”No”. $r_{3}$ is a list of ten candidate words for replacement, generated only if $r_{2}$ is ”No”; otherwise, it remains empty. The process above is formalized as follows:

(2) $$(r_{1},r_{2},r_{3})=\mathcal{M}(p)=\mathcal{M}(\mathrm{Prompt}(t_{i}\setminus\{w_{ij}\},w_{ij})).$$

Iterative Refinement. This phase, informed by the results from LLM $\mathcal{M}$, selects a more coherent alternative to replace the intruder word when necessary. If $r_{2}$ indicates that a word $w_{ij}$ lies outside the semantic boundary of the topic $t_{i}$, a list of ten alternative words is generated in $r_{3}$. From this list, the alternative word $w^{\prime}_{ij}$ is selected based on its inclusion in the vocabulary $V$. If none of the alternative words are found in the vocabulary, the mechanism opts for the most semantically similar word from $V$, which is not already included in the topic $t_{i}$. This ensures that the alternative word $w^{\prime}_{ij}$ is not only semantically coherent but also retains the lexical integrity of the dataset. The process iterates over each word in a topic in reverse order, for words are generally arranged according to topic relevance. By the end of this iteration, we arrive at a refined set of topics $t^{\prime}_{i}$. The specific procedure of the topic refinement mechanism is demonstrated in Algorithm 1.

Algorithm 1 Topic Refinement Mechanism

0:  a set of topics $T$, vocabulary $V$, large language model $\mathcal{M}$

0:  refined topics $T^{\prime}=\left\{t^{\prime}_{1},t^{\prime}_{2},\cdots,t^{\prime}_{K}\right\}$

1:  Initialize the set of refined topics $T^{\prime}\leftarrow\varnothing$;

2:  for each topic $t_{i}$ in $T$ do

3:     Initialize the refined topic $t^{\prime}_{i}\leftarrow t_{i}$;

4:     for each word $(w_{iN},w_{i(N-1)},\cdots,w_{i1})$ in $t^{\prime}_{i}$ do

5:        Construct the prompt $p\leftarrow\mathrm{Prompt}(t_{i}\setminus\{w_{ij}\},w_{ij})$;

6:        Generate the answer $(r_{1},r_{2},r_{3})$ by Eq. 2;

7:        if the answer $r_{2}$ is ”NO” then

8:           Select the alternative word $w^{\prime}_{ij}$ from $r_{3}$ and $V$;

9:           Replace $w_{ij}$ in topic $t^{\prime}_{i}$ with $w^{\prime}_{ij}$;

10:        else

11:           Retain $w_{ij}$ in topic $t^{\prime}_{i}$;

12:        end if

13:     end for

14:  end for

3. Experiments

Figure 3. Ablation study results with various LLMs. The error bars denote a 95% confidence interval for the statistical variability of results across seven base models.

3.1. Experimental Setup

Datasets. We employ the following short text datasets for evaluation: (1) AGNews (Zhang et al., 2015): more than one million news articles across four categories gathered by ComeToMyHead. We randomly sample 5,000 article titles from each category for experiments. (2) TagMyNews (Vitale et al., 2012): news titles from various news sources, annotated with seven labels. (3) YahooAnswer (Zhang et al., 2015): ten categories of questions and their corresponding best answers from the Yahoo Answers website. Similar to AGNews, we sample 2,500 question titles from each category. Table 2 shows the statistics of these datasets after preprocessing with reference to Wu et al. (2022).

Table 2. Statistics of datasets after preprocessing.

Datasets	#Docs	#Labels	#Vocab.	Avg. Length
AGNews	14,845	4	3,290	4.268
TagMyNews	27,369	7	4,325	4.483
YahooAnswer	12,258	10	3,423	4.151

Base Models. We utilize the topic refinement with the following base topic models: (1) LDA (Blei et al., 2003) that uses Gibbs sampling for parameter estimation and inference, with GibbsLDA++ implementation in our experiment. (2) BTM (Cheng et al., 2014) that is designed for short texts to enrich word co-occurrence within them. (3) Gaussian-BAT (Wang et al., 2020) that incorporates word relatedness information by multivariate Gaussian modeling of topics. (4) CNTM (Nguyen and Luu, 2021) that leverages contrastive learning and word embedding to capture semantic patterns of documents. (5) TSCTM (Wu et al., 2022) that is based on contrastive learning with quantified sampling strategies to improve topic quality in short texts. (6) BERTopic (Grootendorst, 2022) that utilizes pre-trained transformer-based language models to extract topics. (7) ECRTM (Wu et al., 2023) that forces each topic embedding to be the center of a word embedding cluster in the semantic space via optimal transport. It is noteworthy that we consider both probabilistic topic models, such as LDA and BTM, and advanced variants of neural topic models that integrate external knowledge. All base models are executed with their recommended hyperparameters.

Evaluation Metrics. Following previous mainstream work, we evaluate the topic quality by topic coherence metrics, measuring the coherence between the $N$ words of discovered topics. We adopt the widely-used Palmetto to calculate these coherence metrics $C_{A}$, $C_{P}$, $C_{V}$, $UCI$, and $NPMI$ (Röder et al., 2015), with higher values indicating better topic quality. This library employs Wikipedia articles as a reference corpus to estimate the co-occurrence probabilities of words.

Implementation details. For our topic refinement mechanism, we use GPT-3.5-turbo as the LLM $\mathcal{M}$ and set its temperature as 0 to lower the completion randomness. Keeping with prior work, each topic $t_{i}$ mined by base topic models is represented by $N$ words, where $N$ is typically set to 10.

3.2. Main Results

We utilize base topic models with $K=20$ and $50$ for experiments in this part. The average results, as depicted in Table 1, reveal the effectiveness of the topic refinement mechanism across various datasets and base models. Notably, incorporating topic refinement consistently enhances the topic coherence of all base models, with increases in nearly every evaluated metric. This improvement is especially pronounced in AGNews and TagMyNews datasets, where base models initially showed lower performance metrics. In effect, topic refinement indirectly mitigates the challenges of data sparsity in short texts by refining topics. Even for base models that initially exhibited high coherence, like CNTM and BERTopic, topic refinement also boosts their performance. Collectively, the uniform advancements prove the model-agnostic nature of our mechanism.

Moreover, Table 3 presents the average token costs for each topic across various datasets, showcasing the efficiency of our mechanism. The input token costs are relatively consistent across datasets due to the similar prompts. Interestingly, the output token costs suggest differences in the complexity of topic modeling for each dataset. For instance, the YahooAnswer dataset, which already exhibited relatively higher-quality topics in the base models, requires fewer tokens for refinement. The implication is that datasets with lower modeling complexity require less intervention from our mechanism.

Table 3. Average token costs for each topic.

Datasets	Input tokens	Output tokens
AGNews	1589.89	264.21
TagMyNews	1595.61	274.46
YahooAnswer	1579.12	226.27

3.3. Ablation Study and Visualization

We additionally carry out ablation studies to evaluate the impact of different LLMs, such as PaLM2, GPT-3.5, Gemini Pro, and GPT-4, on the TagMyNews dataset with $K=20$. Figure 3 provides a detailed comparison of coherence metrics and the number of altered words within the topics after topic refinement. The implementation of all tested LLMs clearly improves topic coherence, underscoring the efficacy and robustness of our mechanism. It is noteworthy that LLMs with more recent and superior capabilities exhibit a better performance for refinement, with GPT-4 emerging as the most effective among them. Although Gemini Pro and GPT-4 yield comparable results in coherence metrics, GPT-4 shows fewer alterations in the topics, indicating its precision in refining topics.

(a) Comparison of $NPMI$ for word $w_{ij}$ and $w^{\prime}_{ij}$.

(b) Comparison of average $NPMI$ for topic $t_{i}$ and $t^{\prime}_{i}$.

Figure 4. Coherence visualization.

Figure 4 displays the coherence visualization of ECRTM’s (Wu et al., 2023) topics on the AGNews dataset with $K=20$. In the visualization, we compute the metric $NPMI$ for each word in a topic against the other $N-1$ words to assess word coherence. Figure 4(a) compares the $NPMI$ for individual words within the topics before and after refinement, indicated by $w_{ij}$ and $w^{\prime}_{ij}$, respectively. The heatmaps show a visible shift towards higher values after refinement to evidence the enhancement in word-topic relatedness. Figure 4(b) further contrasts the average $NPMI$ from base topics to refined topics, while the refined topics generally reveal more positive $NPMI$ values. This improvement is proved by the error bars with 95% confidence interval, which are notably tighter for the refined topics.

4. Conclusion

This paper explores the potential of using LLMs to enhance the topic modeling quality in short texts. Our innovative mechanism, Topic Refinement, utilizes prompt engineering with LLMs to identify and correct semantically intruder words within extracted topics. Extensive experiments across the three datasets and seven base topic models conclusively demonstrate the effectiveness of this mechanism. Future research could involve the dynamic adjustment of the topic modeling process based on LLM feedback, directly tackling the challenge of data sparsity in short texts.