Kempster began his football career with the North Somerset Regional Talent Centre from the age of 11 through to the age of 16, with his final year including scoring two goals in five games for Clevedon under-18s in the Western Counties Floodlight League Premier Division.
He joined Weston in the summer of 2019 as part of the partnership with Weston College, and he went on to score four goals in 16 games for the under-19s during 2019-2020. His form for the youth team saw him rewarded with a first-team debut in the Southern League as a late substitute in the 1-0 win over Walton Casuals in February 2020.
Kempster was involved in the 2020 pre-season with the first-team squad, putting in some promising performances on the pitch which saw him promoted to the first-team.
He joined Southern League Division One South side Willand Rovers on loan for a month in September 2020, which was then extended in October 2020. He made nine appearances and was an unused substitute twice, scoring one goal and assisting one for the Devon side before the season was curtailed.
On 24th September 2021, Kempster returned to Southern League Division One South side Willand Rovers on loan until January. On 21st January 2022, Bailey extended his loan with Willand until the end of the season. Kempster made 22 appearances, scoring four goals and assisting one goal for his loan club across his loan spell.
Bailey was recalled early by Weston from his Willand loan on 1st February 2022 to feature in the 2-1 Somerset Premier Cup defeat to Frome Town, playing 72 minutes of the tie at left-back.
After the Frome game, Bailey returned to Willand on loan until the end of the 2021-2022 season where he made a further 11 appearances and was an unused substitute once, scoring one goal.
On 2nd September 2022, Kempster joined Southern League Division One South side Willand Rovers on loan for a fourth time, for the next month. The young winger made five appearances for Willand and assisted one goal.
He returned to the club and played the entire 90 minutes against Shepton Mallet in the Somerset Premier Cup.
On 7th October 2022, Kempster returned to Southern League Division One South side Willand Rovers on loan. Across the 2022-2023 season, he made 29 appearances for Willand and scored three goals.
On 23rd September 2023, Bailey joined Southern League Premier South side Tiverton Town on loan. As of 23rd November 2023, Bailey has made nine appearances for Tiverton.