This paper provides an assessment of current rotor design comparison practices. First, the employed CFD method is validated for a number of rotor designs and is shown to achieve accurate performance predictions in hover and high-speed forward flight. Based on CFD results, a detailed investigation is performed in terms of comparing different rotor designs. The CFD analysis highlighted the need of high fidelity methods due to the subtle aerodynamics involved in advanced planforms. Nevertheless, the paper suggests that the correct basis for comparison in terms of performance metrics must be used to inform decisions about the suitability of the rotor blades designs for specific applications. In particular, when comparing blades of advanced planforms, direct torque and thrust comparisons are better than the commonly used lift to drag ratio and figure of merit.