Increasing barley production in the face of declining soil fertility on smallholder farms is a critical challenge in Ethiopia. The objectives of this study were to (1) determine the optimum rates of nitrogen (N), phosphorus (P), potassium (K), and sulfur (S) fertilizers under a balanced fertilization of other macro- and micronutrients (zinc and boron) and (2) quantify the agronomic efficiency of N and P on Cambisols and Vertisols. The trials were conducted for three consecutive years (2014–2016) on farmers’ fields on 83 sites under rain-fed conditions. On each farm, six rates of N, P, K, and S with a balanced application of the other nutrients were laid out separately in a randomized block design with three replicates. Balanced fertilization resulted in yield increments of 26–59% on Cambisols and 18–74% on Vertisols, as compared with the control (0 N). Application of P increased grain yield from 11% to 30% and from 7% to 14% due to the interaction of P by soil type as compared with the control. The application of K and S, however, had no significant effect on barley yields on either soil type. The highest agronomic efficiency of N (AEN) was obtained from 46 kg N ha−1, with AEN increasing by 117% on Cambisols and 33% on Vertisols, as compared with the highest N rate of 230 kg N ha−1. The maximum agronomic efficiency of P (AEP) was recorded with 10 kg P ha−1 on Vertisols, with AEP increasing by 73% on Cambisols and 452% on Vertisols, as compared with the application of 50 kg P ha−1. With balanced application of other nutrients, use of 230 kg N and 20 kg P ha−1 was recommended for tepid sub-moist mid-highlands and Vertisols. For tepid sub-humid mid-highlands, application of 46 kg N and 20 kg P ha−1 with balanced application of other nutrients was recommended. For Cambisols, application of 230 kg N and 40 kg P ha−1 was recommended with balanced application of the other nutrients. Similarly, application of 230 kg N, 40 kg P, and 10 kg S ha−1 was tentatively recommended for cool sub-most mid highlands. However, response to K was inconsistent and therefore a firm recommendation could not be proposed for the study sites. Further research needs to be conducted on more soil types and locations.