Both sides had chances in an open and entertaining first half, with Mark McGeown on top form to deny on-loan Gretna front man Dene Shields and Myles Hogarth twice saving efforts from Ryan Caddis.
The first goal was always going to be crucial and Ayr got it seven minutes after half-time in controversial fashion. Chris Robertson's header from a David Dunn corner was flicked on by Vareille and was ruled to have crossed the line, although keeper Hogarth was adamant he got there in time.
Thereafte, Ayr never looked in too much danger and Vareille pounced again in 65 minutes when he seized on a defensive error to fire home.
Stirling gave themselves a lifeline with eleven minutes left when Colin Cramb scored on his debut with a soaring header from Slovakian midfielder Marek Tomana's cross.
Just two minutes later, Robertson bustled through the middle of the home defence and picked his spot in the corner of the net. Albion looked as if they badly missed the suspended duo Stewart Devine and Ross Forsyth.