Ship anomaly detection is a vital aspect for monitoring navigational safety in specific water areas. Considering the effect of water channel boundaries, we propose the detection of an abnormal ship trajectory based on the complex polygon (DATCP) method to detect ship anomalies in this study. With the automatic identification systems (AIS) data from the Yangtze River estuary, a case study is created to verify the effectiveness of the proposed DATCP method. The case study results reveal that the proposed DATCP method can provide higher detection accuracy than the conventional A* algorithm. The feature analysis results indicate that ship anomalies are significantly influenced by ship type, time period, weather conditions and ship traffic characteristics.