Topic Modeling as Multi-Objective Contrastive Optimization
CoRR(2024)
Abstract
Recent representation learning approaches enhance neural topic models by
optimizing the weighted linear combination of the evidence lower bound (ELBO)
of the log-likelihood and the contrastive learning objective that contrasts
pairs of input documents. However, document-level contrastive learning might
capture low-level mutual information, such as word ratio, which disturbs topic
modeling. Moreover, there is a potential conflict between the ELBO loss that
memorizes input details for better reconstruction quality, and the contrastive
loss which attempts to learn topic representations that generalize among input
documents. To address these issues, we first introduce a novel contrastive
learning method oriented towards sets of topic vectors to capture useful
semantics that are shared among a set of input documents. Secondly, we
explicitly cast contrastive topic modeling as a gradient-based multi-objective
optimization problem, with the goal of achieving a Pareto stationary solution
that balances the trade-off between the ELBO and the contrastive objective.
Extensive experiments demonstrate that our framework consistently produces
higher-performing neural topic models in terms of topic coherence, topic
diversity, and downstream performance.
MoreTranslated text
Key words
neural topic model,contrastive learning
AI Read Science
Must-Reading Tree
Example
Generate MRT to find the research sequence of this paper
Chat Paper
Summary is being generated by the instructions you defined