Many experiments with infants suggest that they possess quantitative abilities, and many experimentalists believe that these abilities set the stage for later mathematics: natural numbers and arithmetic. However, the connection between these early and later skills is far from obvious. We evaluate two possible routes to mathematics and argue that neither is sufficient: (1) We first sketch what we think is the most likely model for infant abilities in this domain, and we examine proposals for extrapolating the natural number concept from these beginnings. Proposals for arriving at natural number by (empirical) induction presuppose the mathematical concepts they seek to explain. Moreover, standard experimental tests for children's understanding of number terms do not necessarily tap these concepts. (2) True concepts of number do appear, however, when children are able to understand generalizations over all numbers; for example, the principle of additive commutativity (a+b=b+a). Theories of how children learn such principles usually rely on a process of mapping from physical object groupings. But both experimental results and theoretical considerations imply that direct mapping is insufficient for acquiring these principles. We suggest instead that children may arrive at natural numbers and arithmetic in a more top-down way, by constructing mathematical schemas.