This paper presents a new order selection technique of matrix memory polynomial technique that models the nonlinearities of single-branch and multi-branch transmitters. The new criteria take into account the complexity of the model in addition to its mean-square error in the selection criteria. The quasi-convexity of the proposed criteria was proven in this work. By using this proposed Akaike information criterion (AIC) and Bayesian information criterion (BIC) criteria, the model order selection was cast as a cost minimization problem. To minimize the criteria, modified gradient descent and simulated annealing algorithms were utilized which resulted in a considerable reduction in the number of search iterations. The performances of the criteria were shown by comparing the normalized mean square error (NMSE) of a higher-order model and the optimum model. It has been shown that the NMSE difference is <0.5 dB, but the complexity is much smaller.
