Modelling the Dynamics in a Predator–Prey System with Allee Effects and Anti-predator Behavior

This paper studies a predator–prey model with strong and weak Allee effects and anti-predator behavior using a dynamical system approach. We perform a detailed bifurcation analysis including saddle-node bifurcation, Hopf bifurcation of codimension 3, cusp of codimension 3, cusp type Bogdanov–Takens bifurcation of codimension 3, and codimension-2 cusp of the limit cycle. The involvement of strong and weak Allee effects provides a new regime shift mechanism, which indicates the transition from a homoclinic cycle to a new heteroclinic cycle connecting two boundary equilibria induced by the Allee effect and the carrying capacity. The role of anti-predator behavior is fully uncovered by studying the interaction with the Allee effect. It is the first time that we find a codimension-2 cusp of infinitesimal limit cycle in the predator–prey system, which indicates the existence of a coexistence region of three limit cycles due to the weak Allee effect. Different from the scenario in the reference (Aguirre et al. in SIAM J Appl Math 69(5):1244–1262, 2009), it is a new generating mechanism of limit cycle bifurcating from one Hopf bifurcation point with two saddle-node bifurcation points on the limit cycle branch, and the double limit cycle curve originates from a codimension-2 degenerate Hopf bifurcation point and disappears at another one. The dynamics of the model with the Allee effect and anti-predator behavior are shown to be more complicated than those for other predator–prey systems. The biological interpretations of the bifurcation diagram and phase portrait are also provided.