I have a similar problem on my home built grinder. I've found that minute adjustments of the tracking knob is multiplied at least 10x when using a small wheel. Belt speed is another consideration, as the smaller bearings aren't as robust as are found on the 12" wheel. A lite touch is required on my grinder for sure.
Are you saying the tracking adjustment doesn't move the belt on the small wheel. I'll have to look at mine but I am not having any issue with tracking. Hate to hear that. I agree with Axeman above, the adjustment for the smaller wheel is finicky.
Interesting. Try increasing belt tension and see what effect if any that has? You might also loosen the bolts on the holder and twist it left or right to compensate the tracking. When the belt centers to your liking, tighten the bolts.
A 2nd comment to agree with AR - just a very slight twist to the small wheel holder can really throw tracking off. Put tracking wheel in center, then twist the small wheel bracket a bit to center belt on small wheel and see if that doesn't fix the problem.
It has to be out of square or severely off centerline. Each of my KMG arm tools had to be set up for the centerline of my machine's wheels. If you have to adjust tracking a lot every time you change your attachments then I'd suggest re-looking at your centerline. The tip-off is that your tracking roller shouldn't have to be cockeyed when the belt is running centered.