PyActiveStorage: Energy-Efficient Workflows for Exascale Data

Abstract

As climate modeling enters the exascale era, data volumes are outstripping the capacity of traditional networks. The conventional approach of transferring petabytes from remote storage to local compute is increasingly non-viable due to high latency and the unsustainable energy costs of massive data movement. Data-proximate computing, or Active Storage, offers a vital alternative by performing critical analysis tasks at the storage layer. Even when true active storage is not possible, many of the advantages of active storage can be achieved with relatively little effort by the storage system providers.PyActiveStorage provides a near-production implementation of this technology, specifically engineered for high-performance research environments. The architecture utilises a modular provider pattern, allowing a unified API to interact with diverse backends. Currently deployed on the JASMIN high-performance object store and optimised for NetCDF4 and HDF5 datasets, the library pushes compute tasks directly to the storage system or an adjacent node. Using an MPI-style reduction interface, PyActiveStorage executes complex operations in situ. This prescribed implementation allows storage providers to manage security and load balancing while returning significantly smaller data outputs to the user. By bypassing traditional I/O bottlenecks, PyActiveStorage simplifies the transition to near-data processing. We demonstrate how this architecture ensures that as datasets reach the exascale, analysis capabilities and environmental sustainability keep pace.