RB-Modulation: Training-Free Personalization of Diffusion Models using Stochastic Optimal Control
CoRR(2024)
Abstract
We propose Reference-Based Modulation (RB-Modulation), a new plug-and-play
solution for training-free personalization of diffusion models. Existing
training-free approaches exhibit difficulties in (a) style extraction from
reference images in the absence of additional style or content text
descriptions, (b) unwanted content leakage from reference style images, and (c)
effective composition of style and content. RB-Modulation is built on a novel
stochastic optimal controller where a style descriptor encodes the desired
attributes through a terminal cost. The resulting drift not only overcomes the
difficulties above, but also ensures high fidelity to the reference style and
adheres to the given text prompt. We also introduce a cross-attention-based
feature aggregation scheme that allows RB-Modulation to decouple content and
style from the reference image. With theoretical justification and empirical
evidence, our framework demonstrates precise extraction and control of content
and style in a training-free manner. Further, our method allows a seamless
composition of content and style, which marks a departure from the dependency
on external adapters or ControlNets.
MoreTranslated text
AI Read Science
Must-Reading Tree
Example
![](https://originalfileserver.aminer.cn/sys/aminer/pubs/mrt_preview.jpeg)
Generate MRT to find the research sequence of this paper
Chat Paper
Summary is being generated by the instructions you defined