Self-Healing Workflow Orchestration Framework Using System-Aware Adaptive Recovery for Fault-Tolerant Distributed Systems

Abstract

This research presents the design, implementation, and experimental evaluation of a modular self-healing workflow orchestration framework for fault-tolerant distributed systems. Developed using Go, Redis, and PostgreSQL, the framework integrates asynchronous task queuing, persistent workflow state management, concurrent worker execution, controlled fault injection, and automated recovery mechanisms. Four recovery strategies are implemented and evaluated within a common execution environment: No Recovery, Fixed Retry, Exponential Backoff, and System-Aware Adaptive Recovery. The adaptive strategy uses deterministic, runtime-aware decision rules based on cumulative workflow failure rates and Redis queue depth to select an appropriate retry policy. The experimental evaluation comprises 20 benchmark runs and 2,000 workflow executions, with five runs of 100 workflows per recovery strategy. Results show that automated recovery substantially improves workflow reliability compared with the No Recovery baseline. Fixed Retry achieved the highest observed mean success rate of 84.4%, compared with 82.6% for Adaptive Recovery, 82.4% for Exponential Backoff, and 44.8% without recovery. Statistical analysis found no significant differences in success rates among the three recovery-enabled strategies. Under the independently injected failure model, retry timing does not influence the probability of successful execution, making recovery time and throughput the primary distinguishing factors. Fixed Retry exhibited the lowest average execution duration and highest throughput among the recovery-enabled strategies. The principal contribution of this work is a modular experimental framework for systematically comparing recovery policies under consistent workloads and execution conditions. The findings establish a foundation for future investigations into temporally correlated failures, predictive recovery models, reinforcement learning, and resilient cloud-native workflow orchestration.