Cost-Effective Scraping and Processing of Real-time Traffic Data for Route Planning

The emergence of e-commerce has increased the demand for fast parcel delivery. In order to service their customers, a logistics company will normally set up a number of outlets in different areas of a city so that the senders can submit their parcels to the nearest outlets. Upon receiving the parcels, logistics company will then sort and send these parcels to the outlets that are close to the recipients. The collection/delivery of parcels between outlets needs a fleet of vehicles, and this problem then can be formulated as a vehicle routing problem. However, most of the existing algorithms proposed to solve vehicle routing problem assume that the travelling time between places are static. This may not be realistic because the traffic in the real world is rather dynamic which causes the travelling time from one place to another varies over time. This affects the accuracy of time-cost estimation for the logistics company during their parcel delivery process. However, the acquisition of accurate time-cost estimation is normally very expensive, and it might not be affordable to logistics company. Thus, this paper will mainly focus on a low-cost solution to effectively scrap, preprocess, and analyze the real traffic data in order to provide route planning algorithms with a set of highly accurate time-cost inputs to improve the accuracy of time-cost estimation for the logistic company. The importance of effective and efficient scraping is also stated as the path provided by real-time traffic map’s website is not optimal when traffic condition is heavier.