Rory broke his leg playing for Scotland against France in 2012 and has not been able to regain fitness.
He re-joined Glasgow Warriors in December 2011, after spending three years at the club between 2004 and 2007.
During his first spell with the club he scored 19 tries in 55 games, before moving to Sale Sharks and then Toulon.
Since retuning to the Warriors he has made six appearances, scoring one try.
He made his Scotland debut against Wales in 2005 and has 29 caps for his country.
Rory Lamont said: “I’m sad to be retiring from professional rugby but I’ve been struggling with injury for some time now and it’s clear I won’t be able to get back to the level I was playing at before.
"Despite a tough time personally it’s been a privilege to see the boys do so well and make such good progress this season.
“It’s fantastic to have been at the club during this exciting time and I wish them all the very best for the remaining games this season.”
Glasgow Warriors head coach, Gregor Townsend said: “Rory has had a frustrating time with injury and everyone at the club feels for him not being able to get back to fitness. We were all hoping he'd make a full recovery and show what he could do in a Warriors jersey.
“I’d like to thank him for his contribution to the club over his two spells here and he goes with our very best wishes for the future.”