Urbanixm Content Classification uses machine-learning models to identify relevant web pages and annotate passages about urban policy and interventions. It automates parts of Urbanixm’s content-curation pipeline so the collection can be processed at scale.
Context
Urbanixm harvests web documents and extracts information about urban policy and interventions. The pipeline needs to distinguish useful sources from irrelevant pages, then identify relevant topics and passages within the sources.
My role and contribution
Machine learning engineer. I developed the models and surrounding pipeline end-to-end.
How it works
The pipeline extracts relevant text passages from web pages and annotates them with metadata, including urbanism topics and whether a passage describes a policy or infrastructure implementation or its impact. Classifiers are applied to both whole documents and passages. Examples of classification tasks include:
- Estimating whether a web page is relevant to urbanism.
- Estimating whether a page is quotable—that is, whether it contains informative material rather than being an archive or search-results page.
- Identifying urbanism topics, such as segregated cycle lanes or environmental gentrification.
- Identifying passages that describe an implementation or its impact.
Document classifiers were implemented as multiple binary classifiers using scikit-learn. Passage classifiers were implemented as multi-label classifiers using a Hugging Face pipeline and a RoBERTa-base model.
Outcome
The classifiers were developed to automate content curation. The code is available on GitHub; the data used to build the production model is proprietary. The classifiers are evaluated using rigorous methodology and the ones with high enough accuracy are deployed to automatically classify content for the Urbanixm Knowledge Discovery Engine.
Technology
Python, PyTorch, scikit-learn, Hugging Face, machine learning