Fagerson was one of Scotland's standout performers as he went toe-to-toe with former World Player of the Year Pieter-Steph du Toit and admitted it was one of the most physical matches he has experienced.
"The collisions were massive," Fagerson told Scotland Rugby News. "It felt in the first 10-15 minutes, we probably had to wear a few of them, especially as a forward pack. They were coming off the line, we were maybe playing a bit too deep and we were getting caught a little bit.
"In that middle 50 or 60 we started playing on top of them and a lot of gaps to open up.
"We knew that if we stuck to our guns we could get a result I think that's probably the disappointing thing, that we probably let it slip a little bit and let it get out of our grasp."
Despite pushing the world champions all the way, Fagerson was left frustrated Scotland could not convert their chances into a statement victory.
"I think part of the frustration is that we played bloody well and it was probably just a few soft penalties here and there and a few decisions that probably didn't go our way.
"We had opportunities and they were on their line. I would think that if you asked any of the Springboks they'd say it was a bloody tough game.
"But they're a quality side, they're a two-time [back-to-back] World Cup winners. It was a rotated side but they don"t hit any less hard."
The back-row has made 27 appearances for club and country over the past year, but with Fiji visiting Murrayfield this weekend, Fagerson has no thoughts of an early break.
"The schedule is a bit draining on the body and on the mind, and obviously spending time away from the family, but we're a very tight group and we've had a great three weeks together.
"We hope we can push on and get a win next week as well."
Fagerson believes Scotland's response to adversity against both Argentina and South Africa highlights the progress made since last autumn's collapse against Los Pumas and the Six Nations defeat to Italy.
"Last week Argentina scored early then we came back and we put them away comfortably although we leaked a couple at the end.
"Even on Saturday we were down 14-0 away from home against the Springboks and we didn't cave in like we saw England do it last week.
"We spent a lot of time as a group together and the hurt we had from Argentina in the autumn, we felt we'd righted those wrongs in the Six Nations and we'd become a group that could recognise momentum swings and we were trying to play on that in the middle third as well.
"If we'd gone over the line on a couple of those chances it might be a different scoreline.
"I think the lessons learned from Argentina I feel like we have moved on as a group. We're building something pretty special here.
"Over the last five or six years we've locked in what our identity is and who we are. I think everybody's on the same page as in that."
Saturday's match may officially be a Fiji home fixture, but Fagerson is looking forward to ending a long campaign in front of a packed Murrayfield crowd.
"The way the schedule has worked out we've got the best part of the draw that we've been away for two or three weeks and then we get to finish up at home and play in front of our friends and family."