Breaking the Length Barrier: LLM-Enhanced CTR Prediction in Long Textual User Behaviors
CoRR(2024)
Abstract
With the rise of large language models (LLMs), recent works have leveraged
LLMs to improve the performance of click-through rate (CTR) prediction.
However, we argue that a critical obstacle remains in deploying LLMs for
practical use: the efficiency of LLMs when processing long textual user
behaviors. As user sequences grow longer, the current efficiency of LLMs is
inadequate for training on billions of users and items. To break through the
efficiency barrier of LLMs, we propose Behavior Aggregated Hierarchical
Encoding (BAHE) to enhance the efficiency of LLM-based CTR modeling.
Specifically, BAHE proposes a novel hierarchical architecture that decouples
the encoding of user behaviors from inter-behavior interactions. Firstly, to
prevent computational redundancy from repeated encoding of identical user
behaviors, BAHE employs the LLM's pre-trained shallow layers to extract
embeddings of the most granular, atomic user behaviors from extensive user
sequences and stores them in the offline database. Subsequently, the deeper,
trainable layers of the LLM facilitate intricate inter-behavior interactions,
thereby generating comprehensive user embeddings. This separation allows the
learning of high-level user representations to be independent of low-level
behavior encoding, significantly reducing computational complexity. Finally,
these refined user embeddings, in conjunction with correspondingly processed
item embeddings, are incorporated into the CTR model to compute the CTR scores.
Extensive experimental results show that BAHE reduces training time and memory
by five times for CTR models using LLMs, especially with longer user sequences.
BAHE has been deployed in a real-world system, allowing for daily updates of 50
million CTR data on 8 A100 GPUs, making LLMs practical for industrial CTR
prediction.
MoreTranslated text
AI Read Science
Must-Reading Tree
Example
Generate MRT to find the research sequence of this paper
Chat Paper
Summary is being generated by the instructions you defined