Duplication Scheduling with Bottom-Up Top-Down Recursive Neural Network.

Scientific workflows can be represented as directed acyclic graphs (DAGs) with nodes corresponding to individual tasks and directed edges between the nodes signifying the order of task execution. The nodes contain informative attributes related to task-specific data transfer/storage and scheduling length. Given an available amount of (cloud) computational resources, the overall workflow scheduling length can sometimes be reduced by making certain "critical tasks" run in parallel (task duplication) on multiple resources. In this way, a carefully designed spread of computational effort across the resources can result in a more efficient computational structure and hence a shorter scheduling length. However, task duplication algorithms deciding which tasks to duplicate in a given workflow can themselves be computationally expensive. Here we propose a novel Bottom-Up Top-Down Recursive Neural Network (BUTD RecNN) model that is able to learn from historical duplication decisions on workflows (represented as DAGs) to efficiently produce duplication recommendations for new unseen workflows. The approach is tested on collections of Montage workflows.