Hierarchical Attention Network for Action Segmentation

•End-to-end Fine-grained action segmentation model.•Attention based hierarchical network modelling for capturing relationships at frame-level and video segment-level.•Efficient temporal modelling that capture long-term dependencies within the input video.•Significant performance improvement compared to the state-of-the-arts in multiple challenging public benchmark datasets, including MERL Shopping, 50 salads, and Georgia Tech Egocentric datasets.•Ablation experiments to elaborate the importance of segment-level and frame-level embeddings.