This study investigates second language (L2) phonetic categorization and phonological encoding of L2 words (hereafter, phonolexical encoding1) with phonemic and allophonic cross-linguistic mismatches. We focus on the acquisition of Spanish /ɾ/-/l/ and /ɾ/-/t/ contrasts among Spanish learners with American English (AE) and Mandarin Chinese (hereafter, Chinese) as first languages (L1s). [ɾ] and [t] are positional allophones in AE but separate phonemes in Spanish. The phoneme /ɾ/ is absent in Chinese. AE learners showed nativelike phonetic categorization and little between-contrast difference in phonolexical encoding, suggesting that L1 positional allophony does not necessarily impede L2 contrast acquisition. Chinese learners showed persistent perceptual difficulties with both contrasts due to perceptual similarity. Phonetic categorization significantly predicted phonolexical encoding for /ɾ/-/t/ contrasts for Chinese learners bidirectionally, while AE learners showed this relationship only when /t/ was incorrectly replaced by /ɾ/ in Spanish words. This asymmetry can be driven by the fact that [t] is the dominant allophone of /t/ in AE, while [ɾ] is a positional allophone. It suggests L1 allophonic knowledge heightens perceptual monitoring when evaluating substitutions that conflict with L1 phonological expectations. This study calls for more nuanced treatment of L1 influence in L2 phonological acquisition models, especially at the allophonic level.