Using Probe-Based Speed Data and Interactive Maps for Long-Term and COVID-Era Congestion Monitoring in San Francisco

Probe data that provide roadway speeds and travel times are increasingly being used for a variety of purposes in the transportation domain. A key use of these datasets has been roadway performance monitoring by state and local transportation agencies that are mandated to measure and report performance of their transportation networks. The San Francisco County Transportation Authority (SFCTA) monitors roadway performance as a part of the biennial Congestion Management Program (CMP) and primarily uses probe-based speed data for that purpose. Despite considerable savings in time and effort for data collection, integrating and processing the probe data still required a significant amount of manual work. This study highlights these challenges and proposes a data processing pipeline which includes an automated network conflation process, an efficient large data processing framework, and an interactive web-based visualization. In addition, all the scripts and code developed were made open source and are readily accessible from a public repository on GitHub. The value of the pipeline is demonstrated through the development of web-based interactive maps to monitor both long-term and short-term congestion in San Francisco. The short-term congestion monitoring application is timely given the spread of the COVID-19 pandemic and the region's rapidly changing traffic conditions. Several valuable lessons learned from use of probe data for roadway performance monitoring are shared. Developing tools to ensure consistency of the data product and to reduce reliance on any one data vendor is of key importance.