Human-Centric Topic Modeling with Goal-Prompted Contrastive Learning and Optimal Transport Rui Wang† , Yi Zheng† , Dongxin Wang† , Haiping Huang†∗ , Yuanzhi Yao‡ , Yuxiang Zhou♦ , Jialin Yu♥ , Philip Torr♥ † School of Computer Science, Nanjing University of Posts and Telecommunications ‡ School of Computer Science and Information Engineering, Hefei University of Technology
arXiv:2604.12663v1 [cs.AI] 14 Apr 2026
♦ School of Electronic Engineering and Computer Science, Queen Mary University of London
♥ Department of Engineering Science, University of Oxford email:{rui_wang, 1223045708, 1224045815, hhp}@njupt.edu.cn, [email protected], [email protected], {jialin.yu, philip.torr}@eng.ox.ac.uk ❌Goal-biased Topics
Abstract Existing topic modeling methods, from LDA to recent neural and LLM-based approaches, which focus mainly on statistical coherence, often produce redundant or off-target topics that miss the user’s underlying intent. We introduce “Human-centric Topic Modeling” (Human-TM), a novel task formulation that integrates a humanprovided goal directly into the topic modeling process to produce interpretable, diverse and goal-oriented topics. To tackle this challenge, we propose the Goal-prompted Contrastive Topic Model with Optimal Transport (GCTM-OT), which first uses LLM-based prompting to extract goal candidates from documents, then incorporates these into semantic-aware contrastive learning via optimal transport for topic discovery. Experimental results on three public subreddit datasets show that GCTM-OT outperforms state-of-theart baselines in topic coherence and diversity while significantly improving alignment with human-provided goals, paving the way for more human-centric topic discovery systems.
CCS Concepts • Information systems → Document topic models.
Keywords Human-Centric Topic Modeling, Opinion Mining, Contrastive Learning, Large Language Models ACM Reference Format: Rui Wang† , Yi Zheng† , Dongxin Wang† , Haiping Huang†∗ , Yuanzhi Yao‡ , Yuxiang Zhou♦ , Jialin Yu♥ , Philip Torr♥ . 2026. Human-Centric Topic Modeling with Goal-Prompted Contrastive Learning and Optimal Transport. In . ACM, New York, NY, USA, 11 pages. https://doi.org/10.1145/nnnnnnn. nnnnnnn * Corresponding author.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]. Conference’17, Washington, DC, USA © 2026 Copyright held by the owner/author(s). Publication rights licensed to ACM. ACM ISBN 978-x-xxxx-xxxx-x/YYYY/MM https://doi.org/10.1145/nnnnnnn.nnnnnnn
Corpus
LDA[2003], CAST[2025],etc. Modeling without Goals
whatsbotheringyou What's the thing that's bothering you ? Our GCTM-OT
Human
Human-Centric Topic Modeling
Limitations
Topic 1: house, sister, father, brother... Topic 2: wife, father, marry, house... Topic 3: happy, enjoy, hope, change... Topic 4: college, class, study, learn... ...
👍Goal-oriented Topics
Topic Redundancy Not Related to Goal Low Interpretability
Summary Explanation
[Health Issues] Topic 1: cancer, surgery, medical, injury... Topic 2: cheat, breakup, girlfriend, betray... [Breakup struggles] Topic 3: passion, motivation, purpose, success... [Life Direction]
Topic 4: exam, academic, grade, assignment... [Academic Stress] ...
Figure 1: Key differences between Traditional Topic Modeling and Human-Centric Topic Modeling (with human’s goal).
1
Introduction
Topic models [1, 46], such as the Latent Dirichlet Allocation (LDA)[3] and advanced variants including the Corpus-Aware Self-similarity enhanced Topic Model (CAST) [21], have been widely adopted for open information extraction [40, 41] and knowledge discovery [36, 49]. These approaches uncover coherent semantic patterns from large text corpora without supervision. However, when the analytical objective is guided by a specific human goal, these methods often fall short because they are not designed to adapt topic discovery to user-defined intentions without extensive manual intervention. Consider the example in Figure 1, where the human goal, expressed in question "What’s the thing that’s bothering you?", is to identify the main issues of concern within a collection of text responses. Traditional topic models, which are optimized solely for statistical coherence, often produce topics that, although internally consistent, are redundant, less interpretable, or unrelated to the goal. For example, topics containing terms such as ‘house’, ‘sister’, ‘father’ or ‘happy’, ‘enjoy’, ‘hope’ are coherent in a statistical sense but do not address the intended analytical objective. As a result, important patterns such as ‘breakup struggles’, ‘health issue’ or ‘academic stress’ may be overlooked or obscured, requiring substantial manual filtering and interpretation. To address these limitations, we propose a new task formulation called Human-centric Topic Modeling (Human-TM). As illustrated in Figure 1, given a text corpus (e.g. Subreddit corpus ‘Whatsbotheringyou’) 1 and a human-provided free-text goal (e.g. “What’s the thing that’s bothering you?”), Human-TM aims to automatically generate a set of goal-oriented topics. These topics should satisfy the 1 https://www.reddit.com/r/whatsbotheringyou/
Conference’17, July 2017, Washington, DC, USA
following three requirements: 1). Topics are semantically relevant to the human-provided goal. 2). Topics are semantically coherent and easily interpretable. 3). Topics are diverse (non-redundant) and collectively provide broad coverage of the corpus. To tackle the Human-TM task, we propose the Goal-prompted Contrastive Topic Model with Optimal Transport (GCTM-OT), which formulates topic modeling as a representation learning procedure based on advanced prompt engineering [10] and optimal transport theory [25]. The core idea is to incorporate goal information, identified through LLM-based prompting from texts, into the modeling process for goal-oriented topic extraction. Specifically, GCTM-OT first generates phrase-style goal candidates for each document using a goal-oriented summarization prompt and aggregates them into a corpus-level goal set. To improve topic interpretability, it integrates contextualized word representations from a transformer [9] to build contextualized document representations and models topics with topic representations to infer documenttopic distributions. Moreover, to embed goal information and encourage topic diversity, it forms goal representations by clustering goal candidates in the corpus-level goal set and employs optimal transport [25] to guide topic representation training. After training with the supervision signals provided by contrastive learning and optimal transport, GCTM-OT is capable of identifying interpretable and diverse goal-oriented topics. Additionally, it could generate a phrase-style topic summary for each topic, offering document-level topic interpretation. The main contributions of this work are summarized as: • We introduce Human-TM, a novel topic modeling formulation that takes human goals into account and generates interpretable, diverse and goal-oriented topics. • We propose the novel Goal-prompted Contrastive Topic Model with Optimal Transport (GCTM-OT), which leverages LLM-based prompting to identify goal candidates within texts and incorporates them into the modeling process via contrastive learning and optimal transport, enabling goaloriented topic extraction. • We evaluate GCTM-OT on three Subreddit datasets. Experimental results show that GCTM-OT outperforms state-ofthe-art baselines in topic coherence and diversity. In addition, it generates topics that closely align with human goals, as measured by Goal Similarity (𝐺𝑆), Goal-relevant Topic Rate (𝐺𝑇 𝑅) and Goal Coverage Rate (𝐺𝐶𝑅) metrics.
2
Trovato et al.
network to capture semantic patterns. Inspired by NVDM, Xu et al. proposed the VONT [47] by employing a mixture of von-Mises Fisher distributions as topic prior. Ma et al. utilized the word embeddings to filter out irrelevant words and proposed the Corpus-Aware Self-similarity Topic (CAST) [21] model. Fang et al. incorporated the dynamic word representations into the modeling process and proposed the Contextualized Word Topic Model (CWTM) [11]. To mine the public’s opinion towards ChatGPT, Wang et al. employed a topic disentangle mechanism and proposed the Disentangled Contextualized Topic Model (DisCTM) [38]. However, all of these approaches are not able to incorporate the human’s goal into the modeling process, resulting in generic topics that do not match specific human needs.
2.2
Advances in Large Language Models (LLMs) have made them a thriving research topic and have been explored for their robust capabilities in NLP tasks [32, 44, 50]. Meanwhile, Prompt Engineering [30] has emerged as an indispensable approach to extend the capability of LLMs. Firstly, Radford et al. proposed the Zero-shot prompting [27] to offer a paradigm shift in leveraging LLMs. By providing several input-output examples to induce the target of the given task, Brown et al. proposed the Few-shot Prompting [4]. Along this line, Wei et al. introduced the Chain-of-Thought (CoT) [45] prompting to guide LLMs and form a step-by-step reasoning scheme. Recently, scholars have also explored using prompting for topic extraction. Pham et al. employed the LLM-prompting and proposed the TopicGPT [26] for topic phrase generation. Mu et al. proposed the Large Language Model Topic Extraction (LLM-TE) [23], which is an automatic topic generation framework based on LLM. Chang et al. proposed to improve topic quality by designing a topic refinement mechanism [6] with LLM-based prompting.
3
(1) Goal-Oriented: Each topic 𝑡𝑘 (𝑘 ∈ {1, 2, ..., 𝐾 }) must be semanticlly related to human’s goal. For example, if the goal H𝑔 =“What’s the thing that’s bothering you?”, extracted topics should be “Breakup Struggles”, “Acdamic Stress” and etc. (2) Interpretable: Each topic 𝑡𝑘 (𝑘 ∈ {1, 2, ..., 𝐾 }) must be represented by a list of semantically coherent words 𝑡𝑘𝑤 and a human-readable summary explanation 𝑡𝑘𝑠 , such as [‘cheat’, ‘breakup’, ‘girlfriend’, ‘betray’, ‘ring’, ‘break’, ‘divorce’, ‘relationship’, ‘partner’, ‘devastate’] and “breakup struggles”. (3) Diverse and Maximum Coverage: The topics in 𝑇 should be semantically diverse, and their union should cover the majority of goal candidates retained in the corpus.
Related Work
Neural Topic Modeling
Modeling topics using generative neural networks, such as Variational Autoencoders (VAE) [17] and Generative Adversarial Networks (GAN) [14], has emerged as an active area of research. Miao et al. proposed the Neural Variational Document Model (NVDM) [22], which is the pioneering work in this field, based on the VAE. Likewise, Wang et al. proposed the Adversarial-neural Topic Model (ATM) [39] and the Bidirectional Adversarial Topic (BAT) [37] model in an adversarial manner and utilized a generator
Human-TM Task Formulation
Given a collection of 𝑁 documents 𝐷 ={𝑥 1, 𝑥 2, ..., 𝑥 𝑁 } and a humanprovided goal description H𝑔 , human-TM aims to mine a set of 𝐾 topics 𝑇 ={𝑡 1, 𝑡 2, ..., 𝑡𝐾 } that satisfy the following requirements:
Our work is relevant to neural topic modeling and Large Language Model based prompting.
2.1
Large Language Model based Prompting
4
Methodology
As illustrated in Figure 2 (a), our proposed Goal-prompted Contrastive Topic Model with Optimal Transport (GCTM-OT) comprises five components: 1). Goal-oriented Summary Generation (topleft): Extracts human-goal–related phrases from each document
Human-Centric Topic Modeling with Goal-Prompted Contrastive Learning and Optimal Transport workflow of summary knowledge workflow of OT Alignment
Conference’17, July 2017, Washington, DC, USA
Detailed Design of Text Representation Procedure
Goal-oriented Summary Generation
Goal-oriented Optimal Transport Alignment
Document
Document-goal Distributions
Goal Clusters
Goal-candidates Set
GoalSummarization Prompt
Goal Representation Matrix
OT Alignment
.
Clustering
{
Summary
}:
"I'm trapped by student debt, supporting four post-dad's illness on minimum wage with crushing workload leaving no social life.
:
{Student debt limiting independence; Supporting family on minimum wage; No social life due to workload}
Pretrained Transformer
.
OT Alignment
...
...
Maximize Similarity
.
Student debt ...
0.1
0.7
0.1
0.2
0.1
0.3
Supporting family...
0.1
0.3
0.1
0.1
0.2
0.5
No social life...
0.1
0.3
0.1
0.3
0.3
0.7
Similarity Matrix
Topic Representation Matrix
...
Maximum and Normalize
Weight sum
...
Document-topic Distributions
Augmented Documents
n so o ci al lif e
.
...
Weight Sharing
I'm
...
Text Augmentation Prompt
tra pp e ... d
OT Cost Matrix
0.1
0.3
0.1
0.1
0.1
0.3
Attention Weight
Document Representations
Text Augmentation and Representation
Topic Inference and Prior Matching
Contrastive Learning
(a) Overall framework of Goal-prompted Contrastive Topic Model with Optimal Transport (GCTM-OT).
(b) Goal-oriented Text Representation Mechanism.
Figure 2: The framework of GCTM-OT (a) and details of Goal-oriented Text Representation Mechanism (b). so that later topic discovery stays focused on the target intent. It uses the goal summarization prompt P𝑠 to generate 𝐺𝑥 for each document 𝑥, which are then aggregated into the corpus-level goal set 𝐺. 2). Text Augmentation and Representation (bottom-left): Produces diverse yet semantically consistent views of each document so that contrastive learning is effective. Using the text augmentation prompt P𝑎 , it generates pair (𝑥 𝑎 , 𝑥 𝑏 ), then integrates goal information from 𝐺𝑥 into a transformer T to produce goal-oriented representations (ℎ®𝑎𝑥 , ℎ®𝑏𝑥 ). 3). Topic Inference and Prior Matching (bottommiddle): Maps goal-aware document representations to topic distributions while maintaining interpretability. It models topics using the trainable topic representation matrix 𝐸𝑡 and infers the document-topic distributions 𝜃®𝑎 and 𝜃®𝑏 for 𝑥 𝑎 and 𝑥 𝑏 , respectively. These distributions are then matched to a Dirichlet prior, leveraging its multiple-peak property to improve topic interpretability. 4). Semantic-aware Contrastive Learning (bottom-right): Encourages topic representations to capture distinctive semantic boundaries between documents. It optimizes 𝐸𝑡 through contrastive learning, bringing semantically similar document pairs closer and separating unrelated pairs in the learned topic space. 5). Goal-oriented Optimal Transport Alignment (top-right): Aligns the learned topics with human goals to ensure relevance and diversity of topics. It first clusters the corpus-level goal set 𝐺 to form a goal representation matrix 𝐸𝑔 , infers document–goal distributions (𝜆®𝑎 , 𝜆®𝑏 ), and then applies optimal transport to align the document–topic distributions (𝜃®𝑎 , 𝜃®𝑏 ) with the corresponding document–goal distributions (𝜆®𝑎 , 𝜆®𝑏 ), directly injecting goal structure into topic learning. Meanwhile, Figure 2 (b) illustrates the details of the goal-oriented text representation mechanism. The functionalities of these components are described in detail below.
4.1
Goal-oriented Summary Generation
To ensure semantic alignment between the mined topics and the human-provided goal H𝑔 , we first identify and extract potential goal candidates from the corpus. For each document 𝑥 in the corpus 𝐷, we use an LLM [12] with a goal-summarization prompt P𝑠 to 𝑁 generate document-specific goal candidates 𝐺𝑥 = {𝑔𝑥1 , 𝑔𝑥2 , ..., 𝑔𝑥 𝑔𝑥 }.
Detailed Prompt for Goal-oriented Summarization <s>[INST] messages: [ {role: system, content: Act as a subreddit content framework designer. Your task is to identify documents (posts) in "r/{subreddit_name}" that align with the goal: "{goal}".}, {role: user, content: Process this document in two steps: Step 1:Relevance Check -If the text is **NOT relevant** to the goal, output exactly: `[irrelevant]` -If relevant, proceed to Step 2. Step 2 - Core Extraction -Extract 3-5 Core Elements from this text related to the goal({goal}): Format: {{elements1|elements2|...}}.each elements follows "<Entity Action/Status>" structure, without explanations information.}, {role: assistant, content: Sure, please provide the document.}, {role: user, content: {Text}} ] Output: [irrelevant] / {element1|element2|element3} [/INST]
Figure 3: Details of Goal Summarization Prompt P𝑠 . Detailed Prompt for Text Augmentation <s>[INST] messages: [ {role: system, content: You are a helpful assistant that extract critical information, rephrase text and make sentence smooth.}, {role: user, content: I will give you a document . Please rephrase it as little as possible, ensure the output starts with the prefix 'REPHRASED:' and is free of any additional labels, headers, or explanatory text.}, {role: assistant, content: Sure, please provide the sentence you would like me to rephrase.}, {role: user, content: {Text}} ] [/INST] Output: <Augmented Text>
Figure 4: Details of Text Augmentation Prompt P𝑎 .
Here, 𝑁𝑔𝑥 denotes the number of goals in document 𝑥, ranging from 3 to 5 in our experiments. If a document is deemed irrelevant to the goal H𝑔 , it is annotated as ‘irrelevant’ and excluded from the corpus. The details of the prompt P𝑠 are shown in Figure 3, which follows the OpenAI-released API guidelines2 . We then aggregate document-specific goal sets 𝐺𝑥𝑛 (𝑛 ∈ {1,2,. . . ,𝑁 }) to construct the corpus-level goal set 𝐺 = 𝐺𝑥1 ∪ 𝐺𝑥2 ∪ · · · ∪ 𝐺𝑥𝑁 , which is later used to produce the goal representation matrix 𝐸𝑔 (Section 4.5). 2 https://platform.openai.com/docs/guides/prompt-engineering
Conference’17, July 2017, Washington, DC, USA
Trovato et al.
where 𝑒®𝑛′ (𝑛 ∈ {1, 2, ..., 𝑁𝑥′ }) denotes the contextualized word repre-
representation matrix 𝐸𝑡 ∈ R𝐾 ×𝐻 , where 𝐻 denotes the dimensionality of the transformer representation. Specifically, for the augmented document 𝑥 𝑎 , GCTM-OT infers its document-topic distribution 𝜃®𝑎 with: 𝜃®𝑎 = softmax([cos(ℎ®𝑎 , 𝑒®𝑡1 ), cos(ℎ®𝑎 , 𝑒®𝑡2 ), ..., cos(ℎ®𝑎 , 𝑒®𝑡𝐾 )]) (6) where 𝑒®𝑡𝑘 (𝑘 ∈ {1, 2, ..., 𝐾 }) represents the topic representation of the 𝑘-th topic. Likewise, the document-topic distribution of 𝑥 𝑏 could be obtained by feeding it with ℎ®𝑏 . Moreover, to ensure topic interpretability, we match the inferred document-topic distributions to the Dirichlet prior via Maximum Mean Discrepancy (MMD) [15]. Specifically, given two batches of inferred document-topic distributions Θ ={𝜃®1, 𝜃®2, ..., 𝜃®2𝑀 } =Θ𝑎 ∪ Θ𝑏 = 𝑎 , 𝜃®𝑏 , 𝜃®𝑏 , ..., 𝜃®𝑏 } and two batches of random samples {𝜃®1𝑎 , 𝜃®2𝑎 , ..., 𝜃®𝑀 1 2 𝑀 ′ ′ ′ ′ } drawn from the Dirichlet prior Dir(𝜃®′ |𝛼), Θ = {𝜃®1, 𝜃®2, ..., 𝜃®2𝑀 ® we match the inferred document-topic distributions Θ to the Θ′ with the formula: 1 ∑︁ ® ® 1 ∑︁ ® ®′ 𝑀𝑀𝐷 (Θ, Θ′ ) = [k(𝜃 𝑖 , 𝜃 𝑗 ) +k(𝜃®𝑖′, 𝜃®′𝑗 )] − k(𝜃 𝑖 , 𝜃 𝑗 ) (7) 𝐴 𝑖≠𝑗 2𝑀 2 𝑖,𝑗
sentation of the 𝑛-th word in the document. Next, we encode the document-specific goal candidates with:
where 𝐴 =2𝑀 (2𝑀 − 1), 𝑀 denotes the batch size, k(·, ·) is the kernel function [33], and 𝛼® is the hyperparameter of the Dirichlet prior.
4.2
Text Augmentation and Representation
To ensure that the augmented documents preserve the semantic meaning of the original documents, which is critical for effective contrastive representation learning [42], we propose using a text augmentation prompt P𝑎 to generate the augmented documents. Specifically, for each document 𝑥 in the batch 𝑋 , we employ a text augmentation prompt P𝑎 to generate semantically consistent documents 𝑥 𝑎 and 𝑥 𝑏 by rephrasing 𝑥 using a large language model. The details of the augmentation prompt P𝑎 are illustrated in Figure 4, which also follows the OpenAI released API guideline 2 . Then, for each augmented document 𝑥 ′ ∈ {𝑥 𝑎 , 𝑥 𝑏 }, we generate a goal-oriented contextualized representation ℎ®𝑥′ using a transformer [9] T (Figure 2b). This model integrates the contextual information from the document with its corresponding goal can𝑁 didates 𝐺𝑥 = {𝑔𝑥1 , 𝑔𝑥2 , . . . , 𝑔𝑥 𝑔 }. Specifically, we first represent the document 𝑥 ′ = [𝑤 1′ , 𝑤 2′ , . . . , 𝑤 𝑁′ ′ ] using transformer T : 𝑥
[® 𝑒 1′ , 𝑒®2′ , ..., 𝑒®𝑁′ ′ ] = T ([𝑤 1′ , 𝑤 2′ , ..., 𝑤 𝑁′ ′ ]) 𝑥
′
′
(1)
𝑥
′𝑁
[® 𝑒𝑔1 ; 𝑒®𝑔2 ; ...; 𝑒®𝑔 𝑔𝑥 ] = [ T̄ (𝑔𝑥1 ); T̄ (𝑔𝑥2 ); ...; T̄ (𝑔𝑥 𝑔𝑥 )] 𝑁
(2)
′
where 𝑒®𝑔𝑖 (𝑖 ∈ {1, 2, . . . , 𝑁𝑔𝑥 }) is the contextualized embedding of the 𝑖-th goal 𝑔𝑥𝑖 , and T̄ (·) denotes the mean of token embeddings produced by the model T for that goal phrase. Then, for the 𝑛th (𝑛 ∈ {1, 2, ..., 𝑁𝑥′ }) word 𝑤𝑛′ , we could calculate its semantic correlations with each goal candidate and assign its attention weight 𝑎𝑛 as the maximum cosine similarity: ′
′
′𝑁
𝑎𝑛 = max( [cos(® 𝑒𝑛′ , 𝑒®𝑔1 ), cos(® 𝑒𝑛′ , 𝑒®𝑔2 ), ..., cos(® 𝑒𝑛′ , 𝑒®𝑔 𝑔 )]) (3) where cos(·, ·) means the cosine similarity. The use of the maximum operation ensures that each word is associated with its most relevant goal, preserving the strongest semantic link without dilution from less-related goals. This highlights words with at least one strong goal alignment while ignoring weaker associations. Finally, we obtain the contextual representation ℎ®𝑥′ of document 𝑥 ′ with: [𝑎ˆ1, 𝑎ˆ2, ..., 𝑎ˆ𝑁𝑥′ ] =softmax([𝑎 1, 𝑎 2, ..., 𝑎 𝑁𝑥′ ]) (4) ∑︁𝑁𝑥′ ℎ®𝑥′ = 𝑎ˆ𝑛 · 𝑒®𝑛′ (5) 𝑛=1
where softmax(·) means the softmax normalization. Thus, we could generate contextualized document representations ℎ®𝑎 and ℎ®𝑏 for the LLM-augmented documents 𝑥 𝑎 and 𝑥 𝑏 . By incorporating contextual and goal candidate information in this way, these representations emphasize goal-relevant semantics and facilitate the extraction of interpretable and goal-oriented topics.
4.4
Semantic-aware Contrastive Learning
To enable the topic representation matrix 𝐸𝑡 to capture semantic patterns and group semantically similar words into clusters, we train GCTM-OT using semantic-aware contrastive learning [7]. Concretely, given two batches of inferred document-topic dis𝑎 , 𝜃®𝑏 , 𝜃®𝑏 , ..., 𝜃®𝑏 } from augtributions Θ = Θ𝑎 ∪ Θ𝑏 = {𝜃®1𝑎 , 𝜃®2𝑎 , ..., 𝜃®𝑀 1 2 𝑀 𝑎 𝑏 𝑎 𝑎 , ..., 𝑥 𝑏 , 𝑥 𝑏 , ..., 𝑥 𝑏 }, mented documents 𝑋 = 𝑋 ∪ 𝑋 = {𝑥 1 , 𝑥 2𝑎 , ..., 𝑥 𝑀 1 2 𝑀 for each document-topic distribution 𝜃®𝑖𝑎 , we first select the corresponding 𝜃®𝑖𝑏 to form the positive pair (𝜃®𝑖𝑎 , 𝜃®𝑖𝑏 ). Then, we select 𝜃®¬ ∈ Θ from a semantically irrelevant 𝑥 ¬ ∈ 𝑋 to form negative pairs (𝜃®𝑖𝑎 , 𝜃®¬ ). Here, 𝜃®¬ should satisfy the following requirements: 1). 𝜃®¬ ∉ {𝜃®𝑖𝑎 , 𝜃®𝑖𝑏 }. 2). cos( T̂ (𝑥𝑖𝑎 ), T̂ (𝑥 ¬ )) < 𝛿, where T̂ (·) denotes the document representation obtained by averaging the contextualized word representations produced by the transformer T , 𝛿 is the similarity threshold hyperparameter, set to 0.6 in the experiments. Thus, the contrastive objective of the augmented document 𝑥𝑖𝑎 could be computed with: ®𝑎 ®𝑏
𝑙𝑖𝑎 =−log
e [s(𝜃𝑖 ,𝜃𝑖 )/𝜏 ] 𝑀 Í [s(𝜃®𝑎,𝜃®𝑎 )/𝜏] [s(𝜃®𝑎,𝜃®𝑏 )/𝜏] Δ+ [ind(𝜃®𝑖𝑎,𝜃®𝑎𝑗 )e 𝑖 𝑗 +ind(𝜃®𝑖𝑎,𝜃®𝑏𝑗 )e 𝑖 𝑗 ] 𝑗=1 ®𝑎 ®𝑏
4.3
Topic Inference and Prior Matching
Given the contextualized document representations (ℎ®𝑎 and ℎ®𝑏 ) from the previous stage, this module aims to infer document-topic distributions (𝜃®𝑎 and 𝜃®𝑏 ) for the augmented documents (𝑥 𝑎 and 𝑥 𝑏 ). Additionally, it matches the inferred document-topic distributions to the Dirichlet prior [19], leveraging the multiple peaks property of its density to improve topic interpretability [35]. To capture semantic patterns in the contextualized document representations, GCTM-OT models topics using a trainable topic
(8)
where s(·, ·) is the cosine similarity, Δ = e [s(𝜃𝑖 ,𝜃𝑖 )/𝜏 ] , and 𝜏 represents the temperature parameter (set to 0.05), the indicator function ind(𝜃®𝑖𝑢 , 𝜃®𝑣𝑗 ) is defined as: ( 1, 𝑖 ≠ 𝑗 𝑎𝑛𝑑 cos( T̂ (𝑥𝑖𝑢 ), T̂ (𝑥 𝑣𝑗 )) < 𝛿 𝑢 ®𝑣 ® ind(𝜃 𝑖 , 𝜃 𝑗 ) = (9) 0, 𝑜𝑡ℎ𝑒𝑟𝑤𝑖𝑠𝑒 where 𝑢, 𝑣 ∈ {𝑎, 𝑏}. Overall, the contrastive objective of two batches of augmented documents 𝑋 = 𝑋 𝑎 ∪ 𝑋 𝑏 could be computed with: 𝑀 1 ∑︁ 𝑎 𝑏 𝐿𝑋 = (𝑙 + 𝑙𝑖 ) (10) 2𝑀 𝑖=1 𝑖 where 𝑀 denotes the batch size.
Human-Centric Topic Modeling with Goal-Prompted Contrastive Learning and Optimal Transport
4.5
Goal-oriented Optimal Transport Alignment
To inject goal information into the learned topics more directly and ensure topic diversity, we identify key goal candidates from the corpus-level goal set 𝐺 via clustering and use them to guide model training through goal-oriented optimal transport alignment. Specifically, given the constructed goal candidates set 𝐺, we first identify a set of 𝐾 key goals by grouping them into clusters with: [𝐶 1, 𝐶 2, ..., 𝐶𝐾 ] = KMeans( T̄ (𝐺)) (11) where T̄ (·) denotes the mean of token embeddings produced by T for goal candidates in 𝐺. 𝐶𝑘 (𝑘 ∈ {1, 2, . . . , 𝐾 }) represents the 𝑘-th cluster of goal candidates. For each cluster 𝐶𝑘 , we use its centroid to form the 𝑘-th goal representation 𝑒®𝑔𝑘 ∈ 𝐸𝑔 . Here, 𝐸𝑔 ∈ R𝐾 ×𝐻 denotes the goal representation matrix, 𝐻 is the dimensionality of the transformer representations. Then, for each augmented document 𝑥 𝑎 , we infer the corresponding document-goal distribution 𝜆®𝑎 with: 𝜆®𝑎 = softmax( [cos(ℎ®𝑎 , 𝑒®1 ), cos(ℎ®𝑎 , 𝑒®2 ), ..., cos(ℎ®𝑎 , 𝑒®𝐾 )]) (12) 𝑔
𝑔
𝑔
where ℎ®𝑎 denotes the contextualized representation of the augmented document 𝑥 𝑎 , softmax(·) and cos(·, ·) denote the softmax normalization and cosine similarity. Thus, to inject the goal information into the modeling process and influence the learning of the topic representation matrix 𝐸𝑡 , we align the document-topic distribution 𝜃®𝑎 with the documentgoal distribution 𝜆®𝑎 using optimal transport (OT) [25], where the distance 𝑑 M̂ (𝜃®𝑎 , 𝜆®𝑎 ) is formulated as: 𝑑 (𝜃®𝑎 , 𝜆®𝑎 ) = min ⟨P, M̂⟩ (13) M̂
P∈𝑈 (𝜃®𝑎 ,𝜆®𝑎 )
where ⟨·, ·⟩ means the Frobenius dot-product, P ∈ R𝐾>0×𝐾 is the transport matrix, 𝑈 (𝜃®𝑎 , 𝜆®𝑎 ) is the transport polytope of 𝜃®𝑎 and 𝜆®𝑎 , which is the polyhedral set of 𝐾 × 𝐾 matrix: 𝑈 (𝜃®𝑎 , 𝜆®𝑎 ) :={𝑃 ∈ R𝐾>0×𝐾 |𝑃 1®𝐾 = 𝜃®𝑎 , 𝑃 𝑇 1®𝐾 = 𝜆®𝑎 }, and 1®𝐾 is the 𝐾-dimensional vector of ones. M̂ ∈ R𝐾>0×𝐾 is the cost matrix of the transport, where the element 𝑚𝑖,𝑗 in the 𝑖-th row and 𝑗-th column is formed with: 𝑚𝑖,𝑗 = 1 − cos(® 𝑒𝑡𝑖 , 𝑒®𝑔𝑗 ) (14) where cos(·, ·) denotes cosine similarity, and 𝑚𝑖,𝑗 ∈ [0, 2], 𝑒®𝑡𝑖 and 𝑒®𝑔𝑗 denote the 𝑖-th topic representation and the 𝑗-th goal representation, respectively. Similarly, we could calculate the OT distance for the augmented document 𝑥 𝑏 using Eq. 13 by feeding in 𝜃®𝑏 and 𝜆®𝑏 . As direct computation via Eq. 13 is time-consuming for large-scale problems, we follow an entropy-regularized estimator [8] and utilize the Sinkhorn iteration [24] for the OT distance estimation. Overall, given two batches of document-topic distributions Θ = Θ𝑎 ∪ Θ𝑏 and the corresponding document-goal distributions Λ = Λ𝑎 ∪ Λ𝑏 , generated from the augmented documents 𝑋 𝑎 ∪ 𝑋 𝑏 , their optimal transport objective could be formulated as: 𝑀 1 ∑︁ 𝑂𝑇 (Θ, Λ) = [𝑑 (𝜃®𝑎 , 𝜆®𝑎 ) + 𝑑 M̂ (𝜃®𝑖𝑏 , 𝜆®𝑖𝑏 )] (15) 2𝑀 𝑖=1 M̂ 𝑖 𝑖 where 𝑀 denotes the batch size, 𝜃®𝑖𝑎 and 𝜃®𝑖𝑏 are the 𝑖-th documenttopic distributions in Θ𝑎 and Θ𝑏 , 𝜆®𝑖𝑎 and 𝜆®𝑖𝑏 denote the 𝑖-th documentgoal distributions in Λ𝑎 and Λ𝑏 .
4.6
Conference’17, July 2017, Washington, DC, USA
Training Objective and Learning Procedure
To extract goal-relevant, interpretable and diverse topics, our proposed GCTM-OT should take the following factors into account: • Maximizing the similarities between positive pairs 3 and minimizing similarities between negative pairs 4 using semanticaware contrastive learning, enabling topic representation matrix 𝐸𝑡 to capture semantic patterns. • Matching the inferred document-topic distributions Θ = Θ𝑎 ∪ Θ𝑏 to the Dirichlet prior Dir(𝜃®′ |𝛼), ® ensuring topic interpretability. • Aligning the inferred document-topic distributions 𝜃®𝑎 and 𝜃®𝑏 with the document-goal distributions 𝜆®𝑎 and 𝜆®𝑏 using optimal transport, ensuring both semantic relevance to the goals and topic diversity. Thus, we formulate the training objective of GCTM-OT as: L = L𝑐 + 𝜂L𝑃𝑀 + 𝜁 L𝑂𝑇 (16) where L𝑐 denotes the contrastive learning objective, computed using Eq. 10. L𝑃𝑀 and L𝑂𝑇 represent the prior matching and optimal transport objectives, calculated using Eq. 7 and Eq. 15. 𝜂 and 𝜁 are coefficient hyperparameters, we set them to 1.0 in our experiments. The detailed training procedure of GCTM-OT is shown in Algorithm 1. In our experiment, batch size 𝑀 is set to 32, learning rate 𝛼 1 is set to 2e-3, the optimal transport objective L𝑂𝑇 is computed using the GeomLoss 5 library with default configuration. GCTM-OT is optimized by Adam [16] optimizer. Algorithm 1 The training procedure of GCTM-OT. Input: Corpus 𝐷 , human’s goal H𝑔 , topic number 𝐾 , batch size 𝑀 , learning rate 𝛼 1 . 1: for each document 𝑥 in 𝐷 do 2: Generate the document-specific goal candidates set 𝐺𝑥 = P𝑠 (𝑥 ) . 3: Add goal candidates in 𝐺𝑥 to the corpus-level goal set 𝐺 . 4: end for 5: Randomly initialize the topic representation matrix 𝐸𝑡 . 6: Obtain goal representation matrix 𝐸𝑔 via Eq.11. 7: Calculate the cost matrix M̂ of optimal transport using Eq.14. 8: for each batch of documents 𝑋 in 𝐷 do 9: for each document 𝑥 in 𝑋 do 10: Generate the augmented document pair via 𝑥 𝑎 = P𝑎 (𝑥 ), 𝑥 𝑏 = P𝑎 (𝑥 ) . 11: Construct document representations ℎ®𝑎 and ℎ®𝑏 for 𝑥 𝑎 and 𝑥 𝑏 via Eq. 5. 12: Infer document-topic distributions 𝜃®𝑎 and 𝜃®𝑏 for 𝑥 𝑎 and 𝑥 𝑏 via Eq.6. 13: Infer document-goal distributions 𝜆®𝑎 and 𝜆®𝑏 for 𝑥 𝑎 and 𝑥 𝑏 via Eq.12. 14: end for 15: Calculate contrastive objective L𝑐 via Eq. 10. 16: Draw a set of 2𝑀 random samples Θ′ from 𝐷𝑖𝑟 (𝜃®′ | 𝛼® ) . 17: Calculate the prior matching objective L𝑃 𝑀 via Eq. 7. 18: Calculate the OT objective L𝑂𝑇 in Eq. 13 through Sinkhorn Iterations [25]. 19: Calculate the overall training objective L via Eq. 16. 20: Update topic representation matrix 𝐸𝑡 with gradient descent. 21: end for Output: Learned topic representation matrix 𝐸𝑡 .
4.7
Topic Extraction
Leveraging the learned topic representation matrix 𝐸𝑡 and the constructed goal representation matrix 𝐸𝑔 , we could extract goalrelevant topics represented by semantically coherent keywords and interpretable topic summaries. 3 Both (𝜃®𝑎 , 𝜃®𝑏 ) and (𝜃®𝑏 , 𝜃®𝑎 ). 4 Both (𝜃®𝑎 , 𝜃® ) and (𝜃®𝑏 , 𝜃® ). ¬ ¬ 5 https://www.kernel-operations.io/geomloss/
Conference’17, July 2017, Washington, DC, USA
Trovato et al.
Specifically, for the 𝑣-th (𝑣 ∈ {1, 2, ..., 𝑉 }) word 𝑤 𝑣 in the vocabulary, we first use the transformer T to generate the contextualized ′ word representations 𝑒®𝑛𝑣 (𝑛 ′ ∈ {1, 2, ..., 𝑁 𝑣′ }) of all its appearances ′ and collect them into the word representation matrix 𝐸 𝑣 ∈ R𝐻 ×𝑁 𝑣 . ′ Here, 𝑁 𝑣 denotes the number of times 𝑤 𝑣 appears in the corpus, 𝐻 is the dimensionality of word representations. We then compute the semantic correlations 𝑐®𝑣 between word 𝑤 𝑣 and topics using: ′ ′ ′ ′ 𝑐®𝑛𝑣 = softmax( [cos(® 𝑒 𝑛𝑣 , 𝑒®𝑡1 ), cos(® 𝑒 𝑛𝑣 , 𝑒®𝑡2 ), ..., cos(® 𝑒 𝑛𝑣 , 𝑒®𝑡𝐾 )]) (17) 1 ∑︁𝑁 𝑣′ 𝑛′ 𝑐®𝑣 = ′ 𝑐® (18) 𝑛 ′ =1 𝑣 𝑁𝑣 ′ 𝑛 where 𝑐®𝑣 denotes the local semantic correlations between the word 𝑤 𝑣 and topics in its 𝑛 ′ -th appearance, and 𝑒®𝑡𝑘 denotes the topic representation of the 𝑘-th topic. Similarly, we construct a correlation matrix Ĉ ∈ R𝐾 ×𝑉 , which stores the semantic correlations between topics and words in the vocabulary. The topic-word distribution 𝜙®𝑘 of the 𝑘-th topic (𝑘 ∈ {1, 2, . . . , 𝐾 }) is then computed with: 𝜙®𝑘 = norm(® 𝑐𝑘,· ) (19) where 𝑐®𝑘,· denotes the 𝑘-th row of Ĉ, and norm(·) denotes the normalization function. Moreover, to enhance topic interpretability, GCTM-OT generates a human-readable summary 𝑡𝑘𝑠 for the 𝑘-th topic using: ˆ 𝑒®𝑔𝑘 )) 𝑡𝑘𝑠 = arg max (cos( T̄ (𝑔), ˆ 𝑘 𝑔∈𝐶
(20)
ˆ is the where 𝑔ˆ denotes a goal candidate in the 𝑘-th cluster 𝐶𝑘 , T̄ (𝑔) ˆ and mean of token embeddings produced by transformer T for 𝑔, 𝑒®𝑔𝑘 is the 𝑘-th goal representation.
5
Experiments
We first introduce the experimental setup, including descriptions of the datasets, baselines, metrics and implementation details. We then present the topic evaluation results along with the corresponding analysis, followed by hyperparameter analysis and ablation studies.
5.1
Experimental Setup
5.1.1 Datasets. We evaluate the performance of GCTM-OT for Human-TM task on three subreddit datasets: ‘Bothering’ 6 , ‘TeslaModel3’ 7 and ‘AskAcademia’ 8 . Specifically, the ‘Whatsbotheringyou’ dataset is a collection of posts about personal concerns, such as ‘breakup struggle’ and ‘health issue’, and is abbreviated as ‘Bothering’ in our experiments. ‘TeslaModel3’ dataset contains user-generated posts covering various aspects of this electric vehicle, such as ‘charging speed’ and ‘autopilot’. ‘AskAcademia’ dataset contains posts about academic life, including topics like ‘work–life balance’ and ‘research challenges’. All datasets are sourced from Pushshift’s Reddit data dumps9 . For each dataset, to ensure data quality, we remove duplicate posts with the same post_id, lowengagement posts with scores below 10, and irrelevant posts tagged by moderators. We then perform a series of preprocessing steps on
each selected post, including spell checking, stemming and stopword removal, using the SpaCy10 library. The statistics of the processed datasets are presented in Table 1. Datasets
#Doc
#Vocab
Avg_Len
Bothering TeslaModel3 AskAcademia
1,483 1,991 3,348
7,700 5,634 8,122
56.48 32.82 36.16
Table 1: The statistics of processed datasets. 5.1.2 Baselines. We choose the following approaches as baselines: • LDA [3] is a probabilistic topic model, which models documents with a mixture of topics. We use the Mallet 11 implementation in the experiments. • BAT [37] is a neural topic modeling approach based on bidirectional adversarial training. We implement the approach following the default parameter configurations. • CTMNeg [2] is a neural topic model based on VAE and negative sampling, we use the official implementation 12 . • vONT [47] is a neural topic model, which models topics with a mixture of von-Mises Fisher distributions. We use the official implementation 13 . • CWTM [11] is a topic model based on the contextualized representations, we use the official implementation 14 . • DisCTM[38] is a contextualized neural topic model for short text modeling. We implement the approach following the default parameter configurations. • HiCOT[34] is a neural topic model based on optimal transport, we use the official implementation 15 . • CAST[21] is a topic model that leverages word embeddings to remove functional words for topic extraction16 . • LLM-TE [23] is a topic mining approach based on prompt engineering, we use the official implementation 17 . • LLM-ITL [48] is a neural topic model that incorporates an LLM-based topic refinement mechanism 18 . 5.1.3 Evaluation Metrics. Human-TM’s performance is evaluated using topic coherence and diversity metrics. In addition, we propose three novel metrics to assess the relevance between humanprovided goals and the extracted topics. To measure topic interpretability, we follow [29] and employ four widely used topic coherence metrics: 𝐶𝑃 , 𝐶𝐴 , 𝑁 𝑃𝑀𝐼 and 𝑈 𝐶𝐼 . All coherence values are computed using the Palmetto 19 library, with higher values indicating greater interpretability. For topic diversity evaluation, we follow [41] and use the Unique Term (𝑈𝑇 ) metric, defined as: 𝑈𝑇 = #𝑁𝑢 /(10 × 𝐾) (21) where #𝑁𝑢 denotes the number of unique words among the topic words, and 𝐾 is the number of topics. 10 https://spacy.io/ 11 https://github.com/mimno/Mallet 12 https://github.com/adhyasuman/ctmneg 13 https://github.com/xuweijieshuai/Neural-Topic-Modeling-vmf 14 DBLP:conf/coling/00110P24
6 https://www.reddit.com/r/whatsbotheringyou/
15 https://github.com/HoangTran223/HiCOT
7 https://www.reddit.com/r/TeslaModel3/
16 https://github.com/yananma1029/CAST
8 https://www.reddit.com/r/AskAcademia/
17 https://github.com/GateNLP/LLMs-for-Topic-Modeling
9 https://www.reddit.com/r/pushshift/comments/1akrhg3/separate_dump_files_for_
18 https://github.com/Xiaohao-Yang/LLM-ITL
the_top_40k_subreddits/
19 https://github.com/dice-group/Palmetto