This study investigated the effect of phonological neighborhood density (PND) on the lexical encoding of perceptually confusable segmental contrasts and the extent to which the precision of encoding is modulated by phonetic categorization and vocabulary size. Korean learners of English and native speakers of American English completed an auditory lexical decision task that contained words and nonwords with /ɛ/, /æ/, /f/, and /p/ (/æ/ and /f/ do not exist in Korean), two phonetic categorization tasks (/ɛ/−/æ/ and /f/−/p/), and a vocabulary test. For the Korean group, participants’ categorization of /f/−/p/ was the only significant predictor of /f/−/p/ nonword rejection. For /ɛ/−/æ/, nonword versions of high PND words were rejected more accurately than low PND. Additionally, vocabulary size and phonetic categorization significantly interacted so that as perception abilities improve, the benefits that come from having a large vocabulary grow as well.