ReviewAid: AI-Driven Full-Text Screening and Data Extraction for Systematic Reviews and Evidence Synthesis

Abstract

[Detailed Docs for v3.0.0 Release] Overview This update introduces a fundamental architectural shift from probabilistic AI self-assessment to deterministic verification based on the source text. It also includes several changes to the JSON parsing pipeline to prevent infinite loops and silent crashes. These changes improve the reliability of the confidence scoring system, reduce false positives (hallucinations), and improve stability during large batch processing. 1. Code Changes: The New Tier 1 Deterministic Verification Files Affected: confidence.py, extractor.py, screener.py What was added: Exact String Matching (Check A): For structured/numeric data (DOIs, Sample Sizes, P-values, Years). Semantic Token Overlap (Check B): For paraphrased semantic data (Conclusions, Interventions). The system extracts unique tokens from the AI's output and calculates the percentage of overlap with the source text. Negation Detection (Check E): A context-window check that looks 20 characters before and after a matched token for words like "not", "no", "failed", or "unable". Override Logic: A conditional statement in both screener.py and extractor.py that compares the AI's self-reported confidence against the new Tier 1 deterministic score. Why it was added: Previously, the Extractor module relied entirely on the LLM to grade its own homework (Tier 2 LLM Self-Assessment). LLMs can sometimes give high confidence scores even when the extracted information is incorrect, leading to a "Medium Anomaly" where the AI was confidently wrong (hallucinating) but still reported a high confidence score (0.6-0.8). How it increases accuracy: Prevents False Positives: If the AI hallucinates a conclusion that isn't in the source text, the Token Overlap will be near 0%. Catches Meaning Reversals: Negation detection checks whether the source text says "Natalizumab is not safe" while the AI extracts "Natalizumab is safe". In this case, the system detects the nearby negation word and drops the confidence score. Mathematical Grounding: The Override Logic ensures that if the AI claims >0.5 confidence but the Tier 1 math finds <0.5 overlap, the system overrides the AI's score downward and flags the paper for mandatory human review. This helps prevent the "confidently wrong" anomaly. 2. Code Changes: Parsing Pipeline Fixes File Affected: parser.py What was changed/removed: Removed json5.loads() entirely from the parsing pipeline. Removed Regex-Based String Processing: Deleted cleaned = re.sub(r'"(?:\\.|[^"\\])*"', replace_newlines_in_strings, cleaned). Added strict=False: Changed json.loads(cleaned_json) to json.loads(cleaned_json, strict=False). Simplified clean_json_response: Replaced Regex-based markdown removal with simple string replacements (raw_str.replace("```json", "").replace("```", "")). Why it was changed: The previous parser was vulnerable to Catastrophic Backtracking. When the AI returned a slightly malformed JSON response (e.g., missing a single closing quotation mark), the old Regex pattern and the json5 library could get stuck calculating millions of combinations. This caused the Streamlit application to hang indefinitely without throwing an error, resulting in "0 papers processed" and silent crashes. How it increases accuracy: Fewer Silent Failures: By using Python's native strict=False parameter, the parser can safely handle unescaped newlines inside JSON string values without using Regex. Faster Execution: The simplified cleaning pipeline runs in milliseconds. If standard parsing fails, the system moves to the "AI Repair" or "Regex Fallback" steps instead of getting stuck in the previous parsing loop. 3. Code Changes: Memory Management & Execution Flow Files Affected: extractor.py, screener.py What was changed: Moved the del full_text_backup and gc.collect() memory cleanup commands to after the estimate_confidence() function call. Why it was changed: To prevent RAM overload during batch processing of 20+ PDFs, the code deletes the full_text_backup string from memory. However, it was being deleted before the new Tier 1 deterministic check could use it. This caused an UnboundLocalError: cannot access local variable 'full_text_backup', which crashed the processing loop. How it increases accuracy: Ensures the Tier 1 verification function always has the source text available to compare against the AI's output, allowing the mathematical checks to run successfully before memory is cleared. 4. Documentation & UI Updates Files Affected: screener.py, extractor.py, README.md, index.html What was changed: Updated the Confidence Score Interpretation Table in both the UI and documentation to reflect the new mathematical framework. Updated the descriptions of Tier 1 and Tier 2 in the README and HTML files to accurately describe the new Deterministic Verification and Override Logic. Why it was changed: To ensure transparency. If a reviewer from Research Synthesis Methods looks at the tool, they need to know that a 0.9 score no longer means "the AI feels 90% confident" - it explicitly means "the extracted data was verified using exact string matching or high token overlap with the source text." A 0.3 score explicitly means "AI score overridden by Tier 1 math due to likely hallucination." Summary of Impact for Research Methodology We move from an LLM Self-Assessment paradigm to a Deterministic Verification paradigm. The AI performs the initial extraction, while deterministic checks compare the extracted information against the source text. This provides an additional safety check for high-confidence extractions and helps identify low-confidence or potentially unsupported results for human review. Full Changelog: https://github.com/aurumz-rgb/ReviewAid/compare/v2.3.1...v3.0.0