There is some evidence that perinatal factors, specifically birth weight (BW), may be related to the onset of prostate cancer (PRCA). This case–control study, nested within the Malmö Diet and Cancer Cohort Study, used archived birth record data from 308 incident PRCA cases diagnosed between 1991 and 2005, and 637 age-matched controls among 4781 men born (1923–1945) in Malmö and Lund, Sweden. We applied conditional logistic regression to examine the birth size–PRCA association, including tumour subtypes, adjusting for perinatal and adult factors. Compared with controls, cases had a non-significantly higher mean BW and were more likely to have high (>4000 g) BW (21% v. 18%), but did not differ in other birth size measures, nor in mean adult body mass index . We observed a non-linear association between BW and PRCA risk. Compared with BWs between 3000 and 3500 g (reference), the fully adjusted odds ratios (OR, 95% CI) were 0.55 (0.33–0.91) for <3000 g, 0.86 (0.61–1.22) for 3500–4000 g and 0.98 (0.64–1.50) for >4000 g. Among men with aggressive tumours, the reduction in risk for those with BWs <3000 g (OR 0.26, 95% CI 0.09–0.72) was stronger than the rate of risk for PRCA overall. Crude risk estimates were minimally attenuated when adjusted for gestational age, maternal age, birth order and adult factors. Birth length, head circumference and placental weight were not associated with prostate cancer. Our results indicate a protective effect of lower BW on risk of total and aggressive prostate cancer, rather than any direct effect of larger birth size.