Fluorescent quantum dots (QDs) modified with polyethylene glycol (PEG) and albumin bovine serum (BSA) have profound application in the detection and treatment of hepatocellular carcinoma (HCC) cells. In the present study, the effects and mechanism of PEG and BSA modification on the cytotoxicity of QDs have been explored. It was found that the diameter of the as-prepared QDs, PEG@QDs, BSA@QDs is 3–5 nm, 4–5 nm, and 4–6 nm, respectively. With increase of the treatment time from 0 to 24 h, the HCC cell viability treated with QDs, PEG@QDs, and BSA@QDs obviously decreases, showing a certain time-dependent manner. When the concentration of several nanomaterials is increased from 10 to 90 nM, the cell viability decreases accordingly, exhibiting a certain concentration-dependent manner. Under the same concentration change conditions, the reactive oxygen species contents of cells treated by QDs, PEG@QDs, and BSA@QDs also rise from 7.9 × 103, 6.7 × 103, and 4.7 × 103 to 13.2 × 103, 14.3 × 103, and 12.3 × 103, respectively. In these processes, superoxide dismutase does not play a major role. This study provides strong foundation and useful guidance for QD applications in the diagnosis and treatment of HCC.