Heterogeneous asphalt mixture is treated as a two-phase composite consisting of asphalt mastic, namely a mixture of asphalt and fine aggregates, and coarse aggregates in this paper. A novel three-dimensional (3D) random modeling frame for asphalt mixture is developed, and as its applications, some numerical samples involving various polyhedric coarse aggregates with given gradations are generated. Viscoelastic asphalt mastic is simply characterized by the generalized Maxwell model whose parameters are determined by fitting the uniaxial compressive creep experimental data of asphalt mastic. After validation, the 3D random numerical samples are used to perform a series of numerical experiments in order to evaluate the influences of coarse aggregate distribution, content, average size, size deviation, shape and loading rate on the mechanical behaviors of asphalt mixture quantitatively. Finally, some important conclusions are given.