Prenatal adversity has been linked to later psychopathology. Yet, research on cumulative prenatal adversity, as well as its interaction with offspring genotype, on brain and behavioral development is scarce. With this study, we aimed to address this gap. In Finnish mother–infant dyads, we investigated the association of a cumulative prenatal adversity sum score (PRE-AS) with (a) child emotional and behavioral problems assessed with the Strengths and Difficulties Questionnaire at 4 and 5 years (N = 1568, 45.3% female), (b) infant amygdalar and hippocampal volumes (subsample N = 122), and (c) its moderation by a hippocampal-specific coexpression polygenic risk score based on the serotonin transporter (SLC6A4) gene. We found that higher PRE-AS was linked to greater child emotional and behavioral problems at both time points, with partly stronger associations in boys than in girls. Higher PRE-AS was associated with larger bilateral infant amygdalar volumes in girls compared to boys, while no associations were found for hippocampal volumes. Further, hyperactivity/inattention in 4-year-old girls was related to both genotype and PRE-AS, the latter partially mediated by right amygdalar volumes as preliminary evidence suggests. Our study is the first to demonstrate a dose-dependent sexually dimorphic relationship between cumulative prenatal adversity and infant amygdalar volumes.