Atomic force microscopy (AFM) can determine mechanical properties, associated with surface topography and structure, of a material at the nanoscale. Force–indentation curves that depict the deformation of a target specimen as a function of an applied force are widely used to determine the elastic modulus of a material based on a contact model. However, a hysteresis may arise due to friction between the AFM tip and a specimen. Consequently, the normal force detected using a photodetector during extension and retraction could be underestimated and overestimated, respectively, and the extension/retraction data could result in a significant difference in the elastic modulus measurement result. In this study, elastic modulus and friction coefficient values were determined based on an in situ theoretical model that compensated for the effect of friction on force–indentation data. It validated the proposed model using three different polymer specimens and colloidal-tipped probes for the force–indentation curve and friction loop measurements. This research could contribute to the accurate measurement of mechanical properties using AFM by enhancing the interpretation of force–indentation curves with friction-induced hysteresis. Furthermore, the proposed approach may be useful for analyzing in situ relationships between mechanical and frictional properties from a fundamental tribological perspective.