Methodology - Data Processing

[MonicaBrandeis]

We would like to update the data processing page with the data slicing process as well as the new input enhancement.

• Two slices from OSM full history
• Fully capturing the city boundaries
• MQM processing by city boundaries

[MonicaBrandeis]

@Bentleysb can you give me the section here, I can add to #68

[Bentleysb]

@MonicaBrandeis, here is a first draft. Feel free to make modifications or request changes as necessary.

OSM PBF Extraction:
The OSM data used was extracted from a full OSM history PBF file (source: https://planet.openstreetmap.org/). The goal of the initial phase of data processing was to have two extracts for each city, one containing 2018 data and the other 2019 data. This was achieved using a tool called Osmium (https://osmcode.org/osmium-tool/).
The first step was to cut the full planet OSM history PBF into a history file for each city using Osmium's 'extract' tool. The tool used geojson boundaries of each city to perform a soft clip for each, generating the new city history PBFs.
Next, each of the city history files were time sliced. Using Osmium's 'time-filter' tool each history file was filtered so that the data appeared the same as it would have at a specific point in time in OSM. For each city this was done twice, creating a PBF for April 7th at 10am UTC for both 2018 and 2019. At this point, the new time slice PBFs were in standard OSM PBF format, as opposed to OSM history files.
Finally, extraneous Nodes were cleaned from the time slice PBFs using a simple python script. The Nodes in question did not have an spatial information, as they were remnants of features that had been deleted at that point in time.

MQM Extent Enhancement:
An addition to MQM was incorporated into our methodology this year, to better compare differences over time. The additional feature allows the extent of MQM results to be locked to a specified bounding box, as opposed to being dynamically set by the extent of the input data. This allowed the results of two MQM runs to share the same extent despite small variances in the spatial bound of the input data.

[MonicaBrandeis]

Updated in #68