A penalty in the first period of extra time put Oldham into the third round of the Worthington Cup and sent Derby crashing out of the competition.
The First Division club took a 17th minute lead but Oldham hit back two minutes later and were the better side for much of the 90 minutes.
Derby were sunk when Clyde Wijnhard scored from the spot in the 97th minute and Gregory's ended a miserable night with ten men when goalkeeper Mart Poom was sent off for a second yellow card.
Oldham came into the tie on a run of six straight wins but Derby drew first blood. Clint Hill brought down Izale McLeod and Danny Higginbotham scored from the spot.
David Eyres, hit back with a shot from the edge of the area that wrong-footed Poom and Oldham had more chances as they played the better football.
Eyres had a deflected free-kick saved by Poom and Fitz Hall fired over from close range before Derby finally got their act together.
The introduction of Lee Morris created string of chances, but in the 97th minute Horacio Carbonari fouled Hill and Wijnhard beat Poom with a coolly taken penalty kick.
Poom was booked for protesting and in the 105th minute he was sent off when he clearly punched the ball outside his area.
Derby pressed hard in the second period of extra time but could not break through as Oldham hung on for a deserved victory