In this paper, a new algorithm is developed based on the homogenization method integrating with the newly developed Hybrid Treffe FEM (HT-FEM) and Hybrid Fundamental Solution based FEM (HFS-FEM). The algorithm can be used to evaluate effective elastic properties of heterogeneous composites. The representative volume element (RVE) of fiber reinforced composites with periodic boundary conditions is introduced and used in our numerical analysis. The proposed algorithm is assessed through two numerical examples with different mesh density and element geometry and used to investigate the effect of fiber volume fraction, fiber shape and configuration on the effective properties of composites. It is found that the proposed algorithm is insensitive to element geometry and mesh density compared with the traditional FEM (e.g. ABAQUS). The numerical results indicate that the HT-FEM and HFS-FEM are promising in micromechanical modeling of heterogeneous materials containing inclusions of various shapes and distributions. They are potential to be used for future application in multiscale simulation.