The Clarets' skipper popped up at the far post to smash the ball past Lloyd Irish to send the Melbourne Community Stadium into vibrant celebrations.
City manager Rod Stringer made six changes to the side that cruised past Barking 7-1 mid-week in the BBC Essex Senior Cup: Matt Young, Adam Bailey-Dennis, Luke Daley, Max Cornhill, Billy Bricknell and Frankie Merrifield were replaced by Ashley Miller, Chris Bush, Robbie Willmott, Anthony Church, Shamir Mullings and Josh Rees.
Taunton Town were looking to reach the First Round of the FA Trophy for the first time since 1982, when they were comfortably beaten by Hendon.
The visitors had the majority of the opening chances, but the Clarets threatened when Max Porter's tame effort was deflected out to Josh Rees whose effort was turned wide for a City corner. Shamir Mullings' towering jump met Willmott's cross but the ball was deflected out for another corner which was eventually cleared.
Manager Rod Stringer was forced into making a substitution on 11 minutes as an earlier collision between Mark Haines and a Taunton midfielder left the defender impaired and needing to be replaced by Phil Walsh.
The visitors threatened midway through the first half as Ross Staley's in-swinging corner flew across the face-of-goal, taking a deflection as it went, before going out.
The hosts felt hard-done-by as Willmott's driving run on the right was cut short by a poor Taunton tackle, however, the Clarets quickly started a counter-attack but Whistler Ian Rathbone called City back, ending their advantage.
The first half ended incredibly flat, with it appearing that either side were unaware that they were playing a Cup fixture. The Clarets opened the second half brightly but were still unable to break the deadlock.
Full-back Daniel Blanchett expertly turned to beat two Taunton defenders on 47 minutes, before pulling the ball back to Anthony Church, but the Clarets' skipper effort flew wide of the goal.
The second half imitated the first when, as time ticked on, the game became incredibly flat and even. However the hosts remained the dominant side and proved their quality with a swift counter-attacking move. Shaun Jeffers connected with Rees' expertly weighted through-ball before being mauled down by Owen Irish and – much to the disgust of the City supporters – the visiting full-back remained unscathed on 71 minutes.
Irish eventually found his name written into the book three minutes later. Jeffers used his pace to dart between the Taunton defenders before being brought down by Irish on the left wing.
The Clarets pushed their way forward in an attempt to get a last-ditch winner but Willmott's darting run was cut short by Matt Villis on the right-wing, seeing the Taunton defender earn himself a place in the referee's book.
City eventually found the net with one minute left as Bush's long throw fell to Church who instinctively sent a thunderous effort past Irish.
Taunton pressed late on for an equaliser but Rod Stringer's team held on to consolidate their place in the next round of the FA Trophy at the expense of Taunton Town.
1. Ross Fitzsimons
2. Ashley Miller
3. Daniel Blanchett
4. Max Porter (60' Jeffers)
5. Chris Bush
6. Mark Haines (11' Walsh)
7. Robbie Willmott
8. Anthony Church ©
9. Shamir Mullings
10. Josh Rees
11. Elliott Buchanan
12. Shaun Jeffers
14. Phillip Walsh
15. Luke Daley
16. Max Cornhill
17. Frankie Merrifield
1. Lloyd Irish
2. Shane White
3. Ross Staley
4. Owen Irish
5. Ed Palmer
6. Matt Villis
7. Ollie Chamberlain
8. Brett Trowbridge
9. Craig Duff (54' Veal)
10. Jordan Rogers
11. Matt Buse
12. Craig Veal
14. Josh Searle
15. Alex Hoyle
16. Michael Meaker.
Chelmsford: Shamir Mullings (43'), Ashley Miller (86'), Shaun Jeffers (89').
Taunton: Jordan Rogers (45'), Ed Palmer (70'), Owen Irish (74'), Matt Villis (88').
Chelmsford: Anthony Church (89')
Man of the Match: Anthony Church
Referee: Ian Rathbone, assisted by Wayne Chalmers and Shaun Barry.