1、Kathryn Mohror,The Scalable Checkpoint/Restart Library (SCR): Overview and Future Directions,Increased component count in supercomputers means increased failure rate,Todays supercomputers experience failures on the order of hours Future systems are predicted to have failures on the order of minutes
2、Checkpointing: periodically flush application state to a file Parallel file system (PFS) Bandwidth from cluster to PFS at LLNL: 10s GB/s 100s TB to 1-2 PB of storage Checkpoint data size varies 100s GB to TB,Writing checkpoints to the parallel file system is very expensive,Parallel File System,Hera,
3、Atlas,Zeus,Gateway Nodes,Compute Nodes,Network Contention,Contention for Shared File System Resources,Contention from Other Clusters for File System,Failures cause loss of valuable compute time,BG/L at LLNL 192K cores Checkpoint every 7.5 hours Achieved 4 days of computation in 6.5 days Atlas at LLN
4、L 4096 cores Checkpoint every 2 hours 20 - 40 minutes MTBF 4 hours Juno at LLNL 256 cores Average 20 min checkpoints 25% time spent in checkpointing,Node-local storage can be utilized to reduce checkpointing costs,Observations: Only need the most recent checkpoint data. Typically just a single node
5、failed at a time. Idea: Store checkpoint data redundantly on compute cluster; only write a few checkpoints to parallel file system. Node-local storage is a performance opportunity AND challenge+ Scales with rest of system- Fails and degrades over time- Physically distributed- Limited resource,SCR wo
6、rks for codes that do globally-coordinated application-level checkpointing,int main(int argc, char* argv) MPI_Init(argc, argv);for(int t = 0; t TIMESTEPS; t+)/* . Do work . */checkpoint();MPI_Finalize();return 0; ,void checkpoint() int rank;MPI_Comm_rank(MPI_COMM_WORLD, ,SCR works for codes that do
7、globally-coordinated application-level checkpointing,int main(int argc, char* argv) MPI_Init(argc, argv);SCR_Init();for(int t = 0; t TIMESTEPS; t+)/* . Do work . */int flag;SCR_Need_checkpoint( ,void checkpoint() SCR_Start_checkpoint();int rank;MPI_Comm_rank(MPI_COMM_WORLD, ,SCR utilizes node-local
8、storage and the parallel file system,SCR_Start_checkpt(); SCR_Route_file(fn,fn2); fwrite(data,); SCR_Complete_checkpt();,SCR_Start_checkpt(); SCR_Route_file(fn,fn2); fwrite(data,); SCR_Complete_checkpt();,SCR_Start_checkpt(); SCR_Route_file(fn,fn2); fwrite(data,); SCR_Complete_checkpt();,SCR_Start_c
9、heckpt(); SCR_Route_file(fn,fn2); fwrite(data,); SCR_Complete_checkpt();,SCR_Start_checkpt(); SCR_Route_file(fn,fn2); fwrite(data,); SCR_Complete_checkpt();,X,SCR_Start_checkpt(); SCR_Route_file(fn,fn2); fwrite(data,); SCR_Complete_checkpt();,SCR uses multiple checkpoint levels for performance and r
10、esiliency,Aggregate checkpoint bandwidth to node-local storage scales linearly on Coastal,Parallel file system built for 10GB/s,Speedups achieved using SCR with PF3d,SCR can recover from 85% of failures using checkpoints that are 100-1000x faster than PFS,Level 1: Local checkpoint sufficient,42 Temp
11、orary parallel file system write failure (subsequent job in same allocation succeeded),10 Job hang,7 Transient processor failure(floating-point exception or segmentation fault),Level 2: Partner / XOR checkpoint sufficient,104 Node failure(bad power supply, failed network card, or unexplained reboot)
12、,Level 3: PFS checkpoint sufficient,23 Permanent parallel file system write failure (no job in same allocation succeeded),3 Permanent hardware failure (bad CPU or memory DIMM),2 Power breaker shut off,Observed 191 failures spanning 5.6 million node hours from 871 runs of PF3d on 3 different clusters
13、 (Coastal, Hera, and Atlas).,31%,54%,15%,Create a model to estimate the best parameters for SCR and predict its performance on future machines,Several parameters determine SCRs performance: Checkpoint interval Checkpoint types and frequency, e.g. how many local checkpoints between each XOR checkpoin
14、t Checkpoint costs Failure rates Developed a probabilistic Markov model Metrics Efficiency: How much time is spent actually progressing the simulation Accounts for time spent checkpointing, recovering, and recomputing Parallel file system load: Expected frequency of checkpoints to the parallel file
15、system,How does checkpointing interval affect efficiency?,C: Checkpoint Cost F: Failure Rate 1x: Todays Values,Maximum efficiency depends on checkpoint cost and failure rates,How does multi-level checkpointing compare to single-level checkpointing to the PFS?,Todays Cost,PFS Checkpoint Cost, Levels,
16、Multi-level checkpointing requires less writes to the PFS,Todays Cost,More expensive checkpoints are rarer,Higher failure rates require more frequent checkpoints,Multi-level checkpointing requires fewer writes to parallel file system,Todays Failure Rate,Expected Time Between Checkpointing to PFS (se
17、conds),PFS Checkpoint Cost, Levels,Summary,Multi-level checkpointing library, SCR Low-cost checkpointing schemes up to 1000x faster than PFS Failure analysis of several HPC systems 85% of failures can be recovered from low-cost checkpoints Hierarchical Markov Model that shows benefits of multi-level
18、 checkpointing: Increased machine efficiency Reduced load on the parallel file system Advantages are expected to increase on future systems. Can still achieve 85% efficiency on 50x less reliable systems,Current and future directions - Theres still more work to do!,Parallel File System,Contention,Use
19、 an overlay network (MRNet) to write checkpoints to the PFS in a controlled way,Parallel File System,“Forest” of writers,Average Total I/O Time per checkpoint with and without SCR/MRNet,Single writer Every checkpoint to the parallel file system,SCR/MRNet Integration,Still work to do for performance
20、Current asynchronous drain uses a single writer Forest Although I/O time is greatly improved, theres a scalability problem in SCR_Complete_checkpoint Current implementation uses a single writer and takes too long to drain the checkpoints at larger scales,Compress checkpoints to reduce checkpointing
21、overheads,Parallel File System,A0=,A1=,A2=,A3=,Partition array A,Interleave array A,Compress array A,70% reduction in checkpoint file size!,Comparison of N-N and N-M Checkpointing,Summary of Compression Effectiveness,Comp Factor = (uncompressed compressed) / compressed * 100,The MRNet nodes add extr
22、a levels of resiliency,Parallel File System,Geographically disperse nodes in an XOR set for increased resiliency,X,Thanks!,Adam Moody, Greg Bronevetsky, Bronis de Supinski (LLNL) Tanzima Islam, Saurabh Bagchi, Rudolf Eigenmann (Purdue) For more information kathrynllnl.gov Open source, BSD license: http:/ Adam Moody, Greg Bronevetsky, Kathryn Mohror, Bronis R. de Supinski, “Design, Modeling, and Evaluation of a Scalable Multi-level Checkpointing System,“ LLNL-CONF-427742, SC10.,
