In the integrated navigation system using extended Kalman filter (EKF), the state error conventionally uses linear approximation to tackle the commonly nonlinear problem. However, this error definition can diverge the filter in some adverse situations due to significant distortion of the linear approximation. By contrast, the nonlinear state error defined in the Lie group satisfies the autonomous equation, which thus has distinctively better convergence property. This work proposes a novel strapdown inertial navigation system (SINS) nonlinear state error defined in the Lie group and derives the SINS equations of the Lie group EKF (LG-EKF) for the MIMU/GNSS/magnetometer integrated navigation system. The corresponding measurement equations are also derived. A land vehicle field test has been conducted to evaluate the performance of EKF, ST-EKF (state transformation extended Kalman filter) and LG-EKF, which verifies LG-EKF's superior estimation accuracy of the heading angle as well as the other two horizontal angles (pitch and roll). The LG-EKF proposed in this paper is unlimited in the choice of sensors, which means it can be applied with both high-end and low-end inertial sensors.