Connecting the impact of silent data corruption with different training characteristics: An empirical study

Despite the nonnegligible occurrence of silent data corruption (SDC) during largescale training of large language models (LLMs), SDC impact on training lacks systematic understanding. This article empirically analyzes the connections between different training characteristics and the impact of SDC on LLM training. Using deterministic training workloads on real-world SDC-affected hardware, we quantify SDC impact by measuring the difference from the baselines on healthy hardware and provide insights into training robustness against SDC by systematically controlled experiments. We find that SDC impact correlates strongly with training stability and loss landscape regions, with not-a-number (NaN) occurring during training larger models. We further study if setting elementwise gradient bounds can mitigate SDC impact considering that SDC can change gradients by large magnitudes. Our results show that a proper bound can reduce the relative performance gap caused by SDC but cannot avoid SDC-induced NaN and may compromise training performance on healthy hardware.