Oldham dominated the first half but a Lee Hughes penalty miss and more wasted chances paved the way for Rovers to extend their fine recent away form.
Duffy's eighth goal of the season sparked a massive change in the balance of play before defender Anthony celebrated his recall to the starting 11 by finishing off the stuttering Latics.
It was only Oldham's second home defeat this term but manager John Sheridan slammed their second-half performance and branded the goals conceded "diabolical".
Rovers had the first opening when Jeff Hughes blazed off target but visiting keeper Steve Phillips was almost left red faced after trying to shield the ball out of play and nearly presenting Deane Smalley with a free shot.
Phillips was guilty of another mistake on 29 minutes when he rushed off his line and pulled down Hughes, who seemed to be going nowhere in pursuit of a long ball from left-back Scott Golbourne.
Referee Colin Webster pointed to the spot and Hughes, Oldham's 15-goal top scorer, took responsibility.
But Phillips made amends in style as he dived to his left and beat out a poor spot-kick.
Rovers had another let-off eight minutes later when Oldham's Dean Windass seized on a loose ball and crashed his volley against the underside of the bar from ten yards.
Hughes was a split second from converting the rebound, only for Anthony's timely challenge to shut him out.
Phillips then saved from Smalley before the second half opened with a golden chance for the mid-table visitors.
Chris Lines dragged it wide with only keeper Jan Budtz to beat, yet Rovers quickly capitalised on Oldham's poor defending as Duffy latched onto a pass from Ricky Lambert and hit a 12-yard snapshot which took a deflection and beat Budtz.
The visitors suddenly hit their stride and should have gone 2-0 up through Lambert before Oldham were caught cold again.
The Latics' massed ranks were unable to cope with a right-wing free-kick from Stuart Campbell and Anthony ended the goalmouth scramble by prodding into the net from close range.