In the investigation of MgO–Y2O3–B2O3 system, six three-phase regions, five binary compounds, and one ternary compound MgYB5O10 were confirmed in the subsolidus phase relations. Single-phase powder sample of MgYB5O10 was successfully prepared through solution synthesis method. By using the Rietveld method from the step-scanning X-ray powder diffraction data, the crystal structure of MgYB5O10 was determined. It crystallizes in the monoclinic system with the space group P121/c1 and lattice parameters a = 8.5113(2) Å, b = 7.5892(2) Å, c = 12.2460(3) Å, β = 130.200(1)°, and Z = 4. The infrared spectrum of MgYB5O10 at room temperature demonstrates the existence of BO3 and BO4 groups. The UV–visible spectrum shows a wide absorption band within the range of 190–400 nm, while the absorption in the visible region is negligible. According to the electronic structure derived by first-principles calculations, MgYB5O10 is an insulator with a wide indirect energy band gap of about 5.95 eV. Layered structural characteristics, existence of one-dimensional YnO8n+2 chains, and the large band gap should be the immanent reason why MgYB5O10-based materials have exhibited outstanding performances in the luminescence field.