Collaborative development of the Lidar Processing Pipeline (LPP) for retrievals of atmospheric aerosols and clouds

Atmospheric lidars can simultaneously measure clouds and aerosols with high temporal and spatial resolution and hence help understand cloud-aerosol interactions, which are the source of major uncertainties in future climate projections. However, atmospheric lidars are typically custom-built, with significant differences between them. In this sense, lidar networks play a crucial role as they coordinate the efforts of different groups, provide guidelines for quality-assured routine measurements and opportunities for side-by-side instrument comparisons, and enforce algorithm validation, all aiming to homogenize the physical retrievals from heterogeneous instruments in a network. Here we provide a high-level overview of the Lidar Processing Pipeline (LPP), an ongoing, collaborative, and open-source coordinated effort in Latin America. The LPP is a collection of tools with the ultimate goal of handling all the steps of a typical analysis of lidar measurements. The modular and configurable framework is generic enough to be applicable to any lidar instrument. The first publicly released version of the LPP produces data files at levels 0 (raw and metadata), 1 (averaging and layer mask), and 2 (aerosol optical properties). We assess the performance of the LPP through quantitative and qualitative analyses of simulated and measured elastic lidar signals. For noiseless synthetic 532 nm elastic signals with a constant lidar ratio (LR), the root mean square error (RMSE) in aerosol extinction within the boundary layer is about 0.1 %. In contrast, retrievals of aerosol backscatter from noisy elastic signals with a variable LR have an RMSE of 11 %, mostly due to assuming a constant LR in the inversion. The application of the LPP for measurements in Sao Paulo, further constrained by co-located AERONET data, retrieved a lidar ratio of 69.9 +/- 5.2 sr at 532 nm, in agreement with reported values for urban aerosols. Over the Amazon, analysis of a 6 km thick multi-layer cirrus found a cloud optical depth of about 0.46, also in agreement with previous studies. From this exercise, we identify the need for new features and discuss a roadmap to guide future development, accommodating the needs of our community.