Feature refinement: An expression-specific feature learning and fusion method for micro-expression recognition

•A new three-layer feature refinement framework is proposed to recognize micro-expression.•Salient and discriminative features with attention and proposal loss are learnt.•The first work which evaluated on two public protocols of microexpression recognition.•Feature refinement framework achieves considerable and competitive results comparing to the state-of-the-art algorithms.