ABSTRACT
This chapter discusses the classes of techniques aimed at addressing the challenges discussed earlier using some of the biggest simulations performed to date as case studies. Driven by the continued increase in available computing resources, scientists are simulating ever more complex phenomena, at higher spatial and temporal resolutions. The resulting massive datasets are increasingly difficult to handle, and consequently, advanced data management, analysis, and visualization techniques are becoming mandatory to gain new scientific insights. The solution to the out-of-core processing problem is typically divided into two parts: algorithm analysis, to understand data access patterns and, when possible, redesign to maximize data locality; storage of data in secondary memory using a layout consistent with the access patterns of the algorithm, amortizing the cost of individual I/O operations over several memory access operations. A series of basic tests was performed to verify the performance of the approach using a general purpose paging system.
