Pancreatic cancer is currently one of the most lethal tumors because of delayed diagnosis and treatment. Aminopeptidase N (CD13/APN), expressed in pancreatic cancer cells, is closely related to the malignant biological behavior, for instance, angiogenesis formation, tumor proliferation, and metastasis. In this study, asparagine–glycine–arginine (Asn–Gly–Arg, NGR), selectively binding to CD13 receptor, was modified to construct a novel contrast agent of QDs@Gd3+-NGR for targeted diagnosis and treatment of pancreatic cancer. It consists of QDs-unit for fluorescence imaging, Gd3+-unit for magnetic resonance imaging (MRI), and NGR for binding to CD13 receptor. PANC-1 cells labeled by QDs@Gd3+-NGR showed significant red fluorescence and high intensity on fluorescence and MR imaging, respectively. Besides, it was confirmed that QDs@Gd3+-NGR could inhibit theproliferation, metastasis, and invasion of PANC-1 cells, and increase reactive oxygen species production and death rate in vitro. Reasonably, we believe the targeted contrast agent of QDs@Gd3+-NGR can sensitively detect pancreatic cancer via MR-fluorescence dual-modality imaging, and plays an active role in inhibition of tumor progression. The promising results in this study provide integration of diagnostic and therapeutic strategy for the management of pancreatic cancer in future.