Kris Commons gave Celtic the lead away from home in the fourth minute on Saturday with a powerful low drive from outside the penalty area after being teed up by Gary Hooper.
But St Johnstone struck back in the 18th minute when Gregory Tade found room in the box to swivel and shoot.
Celtic were furious just eight minutes later when St Johnstone defender Steven Anderson clattered into Commons inside the penalty area but the referee refused to award a spot kick to the visitors.
Both sides had good chances to score again with Celtic particularly wasteful before Vine scored the winning goal in the 80th minute.
Substitute Nigel Hasselbaink drove through Celtic's defence before laying the ball off to Vine and the 29-year-old striker cut onto his right foot and curled his shot inside the far post.
Motherwell remain top of the standings after Michael Higdon scored twice to hand them a come-from-behind victory away to Dundee.
Colin Nish headed in the opening goal in the 20th minute to give the home side the lead but Motherwell managed to keep their unbeaten start to the season going with two goals in the last 16 minutes.
Higdon's first goal came after a cross from Henrik Ojamaa, which he bundled into the net, while his second was a powerful header from a corner with five minutes to play.
Hibernian remain just a point behind Motherwell in the standings after Leigh Griffiths scored twice in the Edinburgh club's 2-1 victory against Kilmarnock.
In other results, St Mirren knocked off Hearts 2-0 and Inverness Caledonian Thistle held on for a 1-1 draw with Aberdeen.