The hosts, who beat Championship side Reading in the third round, were just two minutes away from becoming the first sixth-tier club to eliminate a Premier League team in the competition's 151-year history.
However, a 91st-minute equaliser from substitute Declan Rice cancelled out Alex Penny's 19th-minute opener and forced the tie into extra time, before Jarrod Bowen's 121st-minute winner broke Kidderminster's hearts and prevented penalties.
Kidderminster, the lowest ranked team left in the FA Cup, held their own against a West Ham side filled with international stars, and they deservedly took the lead when right-back Penny pounced on a loose ball inside the penalty area and tapped in from close range.
After an underwhelming first 45 minutes, Moyes was forced to call upon Rice and Craig Dawson at half time before Aaron Cresswell, Tomas Soucek and Pablo Fornals all followed after the hour mark to provide a much-needed spark.
West Ham continued to struggle in their effort to breach Kidderminster's backline, with goalkeeper Luke Simpson making a number of important saves to keep the tie alive, but it was Rice who eventually handed Moyes' men a lifeline with a rifled strike into the roof of the net from eight yards out just seconds before full time.
Penalties were then looming, but Russ Penn's spirited side were dealt another devastating blow when Bowen tapped into an empty net with virtually the last kick of the game, sending the Hammers into the fifth round. body check tags ::