A Scalable Cloud-Native Lakehouse Architecture for Epidemiological Forecasting and Constrained Resource Allocation

Abstract

Disease surveillance systems in resource-constrained public health settings face two distinct challenges: detecting outbreak anomalies in real time, and translating those detections into operationally defensible resource-allocation decisions. Existing approaches typically address one challenge in isolation. This paper presents a cloud-native, end-to-end healthcare analytics platform that bridges both. The system ingests 190,845 rows of national-scale epidemiological data — 49,068 COVID-19 records and 141,777 US infectious disease records — through a distributed PySpark ETL pipeline, stores processed output in an Azure Data Lake Storage Gen2 lakehouse queried via Azure Synapse Analytics serverless SQL, and trains two independent model families for binary outbreak classification: Random Forest (ROC-AUC: 0.967, Recall: 0.793) and XGBoost (ROC-AUC: 0.980, Precision: 0.904), evaluated via 5-fold stratified cross-validation on a severely imbalanced target (4.97% outbreak-positive). The system additionally incorporates ARIMA time-series forecasting with stationarity testing, a pre-trained TrOCR handwriting recognition module for multi-modal clinical document fusion (50% exact-match accuracy, with documented root-cause analysis), and a prescriptive decision layer using constrained linear programming (PuLP) to translate predictions into operational budget allocations. A key engineering contribution is the mid-build pivot from a Kimball star schema to a serverless lakehouse pattern after discovering that Azure Synapse serverless pools do not support DDL operations. Results are delivered via a Power BI executive dashboard and a live public Streamlit application at https://healthcare-outbreak-dw.streamlit.app.