diffvg+CLIP: Generating Painting Trajectories from Text

Given a sentence, the goal of our project is to find out the optimized trajectories for a painting robot to paint an image that illustrates the context of the sentence. We investigate two solution approaches. In the first approach, we consider two stages: in the first stage a set of candidate images are generated in a classification task to match the given sentence, and in the second stage a candidate raster image is simplified into a robot-paint-image along with a vector trajectory with a color labels. For the first stage, we implemented CLIP (Contrastive Language-Image Pre-training), a computer vision system released by OpenAI to generate candidate images. In the second stage, we proposed two different approaches: (i) using an off-the-shelf differentiable vector graphics rasterizing library named diffvg and (ii) applying a simple image segmentation task SLIC and backpropagating through a probability-based network to achieve a set of probability measure, which can be represented as the trajectory. In the second approach, we pair diffvg with CLIP and back-propogate through the entire chain to produce vector graphics which are described by a sentence. Additional details are provided in Section 2.