Radiative transfer distortions of Lyman α emitters: a new Fingers-of-God damping in the clustering in redshift space

Complex radiative transfer (RT) of the Lyman alpha photons poses a theoretical challenge to galaxy surveys that infer the large-scale structure with Lyman alpha emitters (LAEs). Guided by RT simulations, prior studies investigated the impact of RT on the large-scale LAE clustering, and claimed that RT induces a selection effect which results in an anisotropic distortion even in real space but in an otherwise negligible effect in redshift space. However, our previous study, which relies on a full RT code run on the Illustris simulations, shows that the anisotropic selection effect was drastically reduced with higher spatial resolution. Adopting the same simulation framework, we further study the impact of RT on the LAE clustering in redshift space. Since we measure LAE's radial position through a spectral peak of Lyman alpha emission, the frequency shift due to RT contaminates the redshift measurement and hence the inferred radial position in redshift space. We demonstrate that this additional RT offset suppresses the LAE clustering along the line of sight, which can be interpreted as a novel Fingers-of-God (FoG) effect. To assess the FoG effect, we develop a theoretical framework modelling the impact of the RT similar to that of the small-scale peculiar velocity which is commonly studied in the context of the redshift space distortion (RSD). Although our findings strongly encourage a more careful RSD modelling in LAE surveys, we also seek a method to mitigate the additional FoG effect due to RT by making use of other information in a Lyman alpha spectrum.