"Analyzing the Impact of Concept Drift on Twitter Sentiment Classification using the Full Sentiment140 Dataset (1.6M Tweets)"

Abstract

This study examines concept drift — used here, as defined precisely in Section 1.6, in the broad applied-ML sense of a train-to-test performance gap over time, which may reflect covariate shift, label shift, or concept shift in the stricter technical sense — in sentiment classification on the complete Sentiment140 dataset of 1,600,000 tweets collected between April 6 and June 25, 2009: a fourfold increase in coverage over our earlier 400k-sample study. All findings are scoped strictly to this three-month window of 2009 Twitter data and should not be read as claims about modern social media or other domains; Section 1.5 states this boundary explicitly. All tweets are processed in strict chronological order with no sampling, enabling genuine calendar-month train/test splits. A Logistic Regression classifier with TF-IDF features (max_features=30,000; bigrams; min_df=5) is evaluated across four temporal splits. Test accuracy ranges from 0.7966 to 0.8069 (Wilson 95% CI half-widths ≤0.001 in every split, given sample sizes of 576k–1.28M). The most temporally clean experiment — training on April, testing on May (10-day gap, 100k train / 576k test) — shows a 4.75 percentage-point accuracy drop relative to training accuracy, with ROC-AUC = 0.8729. Training on April+May and testing on June shows a 2.40 pp drop with ROC-AUC = 0.8881. All degradation results are statistically significant by a chi-squared test of independence (p < 0.001) with a moderate-to-large effect size (Cramér's V = 0.43–0.54 across the four splits, Cohen's 1988 convention), indicating the train/test gap is not merely an artifact of large sample size. Vocabulary-level analysis shows OOV rates of 62–81% across splits and Jaccard similarities of 0.17–0.23. An OOV ablation — masking out-of-vocabulary tokens at inference time — produces negligible performance change (≤0.07 pp); because our TF-IDF features already exclude rare tokens via a minimum document-frequency threshold, this null result should be read narrowly, as evidence that OOV masking adds little on top of existing feature selection, rather than as a general claim that vocabulary novelty cannot matter for sentiment drift. We consider class distribution shift (the positive-sentiment ratio falls from ≈60% in April–May to 42.3% in June, and to 0% in the final June week) the most plausible single contributing mechanism among those tested, but emphasize this is not a demonstrated causal mechanism; the rebalancing experiment that would test it directly was not run and is identified as the highest-priority follow-up. DDM identified 7 alarms across June's weekly stream; ADWIN identified none. Both detectors observed only 500 sequential samples per week (0.22% of June's 923,608 tweets) with state reset weekly, a verified design limitation that plausibly contributes to ADWIN's null result but was not isolated from ADWIN's delta parameter as an independent explanation; we report the discrepancy and its most likely contributing factor rather than asserting a single resolved cause. Periodic and ADWIN-triggered retraining produce average accuracy and F1 statistically indistinguishable from a static model within this three-month window; one weekly batch contains no positive-labeled tweets at all, which depresses the F1 average for every strategy alike and is itself direct evidence of within-month class shift. Relative to the prior 400k sample, the full 1.6M corpus yields higher test accuracy and smaller accuracy drops on the two analogously defined (though not content-matched) splits common to both studies, consistent with — though not, from two splits alone, proof of — broader training coverage reducing drift sensitivity.