The imposing defender progressed through the Gloucester City youth academy before departing for Witney & District League Two side Chipping Norton FC during the 2005-2006 season, in search of senior football.
He then signed for Southern League Division One Midlands team Bishops Cleeve in the summer of 2006, and went on to make 95 appearances as well as scoring two goals for them.
He left for Witney & District Premier club Carterton Town midway through the 2008-2009 season.
However, Avery was soon back with Southern League South & West side Bishops Cleeve, rejoining them for the start of the 2009-2010 campaign, playing 93 times and scoring one goal over the next two seasons.
In the summer of 2011, Avery had a trial with National League side Forest Green Rovers, however was unsuccessful and eventually returned to Bishops Cleeve that same summer.
He continued to perform at a consistent rate for the Southern League side over the next two seasons, and was snapped up by Southern League South & West team Shortwood United in the summer of 2013, going on to appear 51 times and score two goals for them with eight assists as well as being an unused substitute once.
In May 2014, Avery returned full circle and went back to his first club, National League North side Gloucester, and became a regular at the heart of their defence as well as wearing the armband for the last 18 months with the Tigers. He made 240 appearances for them and was an unused substitute 14 times for the Tigers. Avery scored 12 goals as well as assisted six goals for his hometown club, before being released in January 2020.
Less than a week after his Tigers departure, Avery linked up with the Seagulls and has gone on to become a regular in the heart of defence as he heads into his fourth full season with the club. Sam was part of the Seagulls side that saw the club win the Southern League Premier South title in 2023 and return to the National League South after four years away.