Sidewalk Data Quality

[aseemdeodhar]

The sidewalks width field is a text field (!) and has to be cleaned up. The vast majority of rows are clean, with only 20 out of ~24000 rows having undecipherable text values. Some entries with widths in the 90s (widths are in feet) seem to be erroneous entries where a decimal point is missing.

[aseemdeodhar]

Did manual cleaning of widths data.

Example errors:

• letter O instead of numeral 0
• > sign instead of numeral 7
• Manual edits required for checking if decimal point was missing or not. eg 30ft sidewalks exist, but some 3ft sidewalks are entered as 30 with decimal point missing
• ',' as decimal separator instead of '.'

Used the sf package in R to convert SWK_WIDTH column from text to numeric:

library(sf)
library(dplyr)

sidewalk_shp <- read_sf("sidewalks_clean/Sidewalk_Inventory.shp")
sidewalk_shp <- sidewalk_shp %>% mutate(SWK_WIDTH = as.numeric(SWK_WIDTH))
st_write(sidewalk_shp, "sidewalks_clean/Sidewalk_Inventory.shp", delete_layer = TRUE)

[patwater]

O instead of 0 is a major facepalm! Great stuff

[vr00n]

@aseemdeodhar can you install and run https://github.com/pandas-profiling/pandas-profiling on the sidewalk datafile and upload the resulting html here? Thanks!