Urban Complaint Pattern Mining on NYC 311 Data (21M+ Records)

Project Overview

Urban Complaint Pattern Mining is a full-cycle data mining project on the NYC 311 Service Requests dataset (21.3 million records). Built for a Data Science course, it combines FP-Growth association rule mining and K-means clustering to answer two open research questions: do complaint types co-occur on individual buildings beyond chance, and do neighborhoods form stable complaint typologies? Every discovered pattern was validated against an independent source — 11 million official housing inspection violations — through a dual-window design built to separate real signal from administrative echo.

The Problem: Data That Records Reports, Not Reality

311 data has a fundamental epistemic limitation: it records what residents report, not what actually exists. Reporting behavior varies systematically across neighborhoods and socio-economic lines. Worse, in New York, housing complaints are forwarded to the HPD agency for inspection — so the violation records that would naturally serve as ground truth are themselves partially caused by the complaints. Naive validation would be circular. The entire project was designed around this problem.

Technical Architecture

A two-phase pipeline separating exploration from confirmation:

• Phase 1 — Pure discovery (311 only) — Polars pipeline cleans 21.3M records to 18.85M, aggregates them into 764,114 building-level "baskets" (one basket = unique complaint types per building), then mines frequent itemsets with FP-Growth.
• Phase 2 — External validation — Discovered patterns are enriched with PLUTO property attributes (size, ownership), validated against HPD violations through two time windows, and contextualized with ACS census data.
• Compute — Python 3.12.8, Polars 1.41.2 as the main engine (pandas only for mlxtend interop), mlxtend 0.24.0 FP-Growth, scikit-learn 1.9.0, fixed seed 42 for all stochastic steps.

Data Quality Archaeology

Three silent data bugs were caught by inspecting raw values before type conversion — each would have corrupted results:

• PLUTO building areas used periods as thousands separators, so "1.683" (1,683 sq ft) parsed as roughly a thousand times too small.
• 15 complaint-type pairs differed only in letter casing, creating fake co-occurrences.
• Affordable-housing owner names contained stray whitespace, breaking ownership matching.

Missing data was handled by mechanism, not reflex: the building-ID field (BBL, 11.66% missing) is MNAR/MAR — street complaints structurally lack a building — and since it identifies the analysis unit, listwise deletion was correct where imputation would be meaningless.

Finding 1: The Interior Maintenance Syndrome

Six complaint types — Heat/Hot Water, Unsanitary Condition, Plumbing, Paint/Plaster, Water Leak, Door/Window — consistently dominate the most frequent combinations. The distribution has a striking tail: more buildings have all six complaints (19,624) than have four (11,267) or five (10,639). Neglect arrives as a package, not one problem at a time.

The fifteen pairs among these six items average a lift of 8.48 (bootstrap 95% CI [7.44, 9.55], 10,000 resamples), with the strongest pair (Paint/Plaster - Door/Window) at 12.40.

An honesty note that survived into the final report: the highest-lift rules in the entire dataset are not the syndrome — they're a commercial food-service cluster (Food Establishment / Food Poisoning, lift 22-33). The syndrome is distinguished by frequency and verified validation, not by maximal lift. Reporting this distinction accurately mattered more than a cleaner story.

Finding 2: Validation That Survives Circularity

The core validation uses a dose-response design across two time windows:

Violation rates per housing unit rise monotonically with syndrome completeness within every size band — so building size can't explain it — and the pattern holds in the prior window, where violations predate the complaints and cannot be their administrative echo. The syndrome marks real, chronic neglect.

The Finding I Killed

In the concurrent window, hazardous violation classes appeared to rise dramatically (over 200x) with syndrome completeness — a headline-worthy "severity specificity" claim. But in the independent prior window, all severity classes rose equally (~14-17x). The specificity existed only where circularity could manufacture it, so the claim was rejected and documented as a rejected finding. This deliberate kill is the part of the project I'm proudest of.

Finding 3: Who Owns Neglect

Ownership concentration curves for syndrome buildings nearly overlap the general ownership baseline (top-5% gap: +0.85pp) — neglect is structurally dispersed, not the work of a few slumlords, though 85 owners neglect their entire portfolios. However, the burden pools in affordable housing stock (HDFC): 16-27 percentage points more syndrome prevalence than private ownership at equal building sizes, surviving size control and the prior window. Direction remains honestly unresolved — underfunding vs. distressed buildings being converted to HDFC — so the claim stays at "burden locus," not causation.

Finding 4: Three Sticky Neighborhood Typologies

K-means on deviation-space ZIP compositions yields three gradated types: mixed-commercial (Manhattan), interior-residential (Bronx) — the spatial face of the syndrome, with 20.3% median syndrome concentration vs 1.5-6.7% elsewhere — and vehicle/outdoor (Queens & Staten Island). Character is remarkably sticky: 84.2% of ZIPs never change type across six years. Complaint-composition shifts show no significant correlation with socio-economic change (all Spearman |ρ| < 0.18 across 12 tests) — a defensible null result that closes off the easy "gentrification signal" interpretation.

Model selection was honest: k=2 actually maximizes silhouette (0.2661 vs 0.2167), but merges domain-distinct types; k=3 was chosen for interpretability and documented as such, with stability verified over 20 seeds (silhouette std 0.0001).

Temporal Anatomy

Complaint timestamps reveal two accumulation modes: 43% of syndrome buildings report 2+ complaint types on the same day (silent decay, then a trigger releases everything), while others accumulate over years (20.8% reach four items within a month; 47.6% take over a year; median 332 days). Water-related complaints tend to be reported first; Paint/Plaster is a lagging indicator (first-report ratio 0.33-0.44) — reporting order, not physical causation.

Key Takeaways

• Dual-window validation is a general pattern for any dataset where the outcome variable is administratively downstream of the input signal.
• Stratification + per-unit pooled rates can control confounders credibly when regression is out of scope.
• Rejecting your best-looking finding when it fails independent testing is worth more than the finding itself — it makes every surviving claim credible.
• Polars over pandas made 21M-row iteration interactive instead of painful; the whole pipeline runs on a laptop.
• Null results are results: the ACS zero-correlation finding is one of the report's most defensible claims.

Course: Data Science, Hasanuddin University, 2026 · Report: 47-page academic report (FTUH 2023 format) with pre-registered success criteria, confounder audit, and rejected-experiments log