A comprehensive mutation study in wide deep-rooted R1b Serbian pedigree: mutation rates and male relative differentiation capacity of 36 Y-STR markers

•120 male contemporaries (6349 pairs of relatives) related by 1–20 meiosis (overall 485) were genotyped using 36 Y-STRs.•An average mutation rate was 10.06 × 10−3 (95% CI: 8.65 × 10−3 - 11.61 × 10−3).•RM13, Yfiler + and PPY23 differentiated 95.9%, 65.5% and 57.4% pairs of relatives.•Extra-pair paternity rate was 11.9 × 10−3 (95% CI: 4.4 × 10−3 - 25.8 × 10−3).•A positive correlation between the fathers’ ages and mutability rates was observed.