Federated selective aggregation for on-device knowledge amalgamation

In the current work, we explored a new knowledge amalgamation problem, termed Federated Selective Aggregation for on-device knowledge amalgamation (FedSA). FedSA aims to train an on-device student model for a new task with the help of several decentralized teachers whose pre-training tasks and data are different and agnostic. The motivation to investigate such a problem setup stems from a recent dilemma of model sharing. Due to privacy, security or intellectual property issues, the pre-trained models are, however, not able to be shared, and the resources of devices are usually limited. The proposed FedSA offers a solution to this dilemma and makes it one step further, again, the method can be employed on low-power and resource-limited devices. To this end, a dedicated strategy was proposed to handle the knowledge amalgamation. Specifically, the student-training process in the current work was driven by a novel saliency-based approach which adaptively selects teachers as the participants and integrated their representative capabilities into the student. To evaluate the effectiveness of FedSA, experiments on both single-task and multi-task settings were conducted. The experimental results demonstrate that FedSA could effectively amalgamate knowledge from decentralized models and achieve competitive performance to centralized baselines.