Smectite illitization is an important diagenetic phenomenon of mudstones, but only rarely has the influence of organic matter (OM) on this process been examined. In the present study, hydrothermal experiments were conducted with smectite (M1, total organic carbon (TOC) <0.3%) and a smectite and N,N-dimethylhexadecylamine (16DMA) complex (M2, TOC >1%). X-ray diffraction (XRD), infrared, X-ray fluorescence (XRF), and organic carbon analyses were employed to characterize the mineralogy and OM of the samples and the effect of OM on smectite illitization. The XRD patterns showed changes in clay mineral parameters with increased temperature. These changes varied in both M1 and M2 and indicated a difference in the degree of smectite illitization. Moreover, the OM in M2 was mainly adsorbed in smectite interlayers, the OM was largely desorbed/decomposed at temperatures above 350°C, and the OM was the main reason for differences in the degree of smectite illitization between M1 and M2. Bulk mineral composition, elemental content, and infrared absorption band intensities were changed with increased temperature (especially above 350°C). This indicated the formation of new minerals (e.g., ankerite). Overall, OM entered the interlayer space of smectite in M2 and delayed the exchange of K+ by interlayer cations, and thus, suppressed the transformation of smectite to illite and resulted in differences in smectite illitization of M1 and M2. In particular, the formation of CO2 after the decomposition of OM at temperatures above 300°C led to the formation of ankerite in M2. This demonstrated the effect of organic-inorganic interactions on smectite illitization and mineral formation. The disparities in smectite illitization between M1 andM2, therefore, were linked to differences in the mineral formation mechanisms of a water-rock system (M1) and a water-rock-OM system (M2) in natural environments. The insights obtained in the present study should be of high importance in understanding organic-mineral interactions, hydrocarbon generation, and the carbon cycle.