The metallic return lines of ±500 kV Zhangbei VSC-HVDC grid have no voltage during normal situation, and there are two kinds of line forms, including single-loop line and double-loop line. At the connection point of single-loop line and double-loop lines, the wave impedance is discontinuous, thus traditional methods are not suitable for the fault monitoring of the metallic return lines. The wave impedance and traveling wave characteristics are analyzed, and a fault monitoring system and waveform comparison analysis method are presented, which are based on differential mode Time Domain Reflection (TDR) method. The system applies differential mode pulse to monitor the fault, and uses waveform comparison method to eliminate the wave reflection caused by inherent impedance discontinuity of the line and highlight the fault characteristics. The simulation and field tests show that the system and method can adapt to the operating condition and wave impedance discontinuity of the metallic return lines of Zhangbei VSC-HVDC grid. The method has high reliability and sensitivity for fault monitoring, and the location accuracy is better than 300 m.