This paper investigates the thermal characteristics of typical Micro Electro-Mechanical System (MEMS) Inertial Measurement Units (IMUs) with a reliable thermal test procedure. Test results show that MEMS sensor errors, not only biases, but also scale factors and non-orthogonalities, may vary significantly with temperature. Also, MEMS sensor errors can have significant inconsistent curves under different temperature changing profiles. The existence of such inconsistencies posed a challenge to the following assumption of thermal calibration: the thermal drift of a sensor error is only related to the temperature of the sensor core. A robust way to mitigate this issue is given by using the sensor data during both heat-and-stay and cool-and-stay processes to establish the final thermal models. The performance of both IMUs and inertial navigation systems improved significantly after compensation with the established thermal models. Additionally, the variation of the IMU thermal parameters with time was observed, which suggests that periodical thermal calibration is necessary for MEMS IMUs.