I've always done 99% on the grinder, it's the way I was taught so I didn't know any better. It's always puzzled me why people spend so much time hand sanding. Anyway....be open to new tricks to ease your pain. Grind closer to finish. PBC compound is a great help, on carbon steels it allows me to heat treat with a 600 grit belt finish and be a step ahead after. Same with foil on stainless. Although I mostly do hollows, but a thin cork lined platen made a big difference at grits over 1000. You just need to really learn your machine and apply the right finesse where it's needed. Neccesity is the mother of invention.
Rudy