The larger the diameter of the drive wheel, the more SFPN (surface feet per minute), in other words, the larger the diameter of the drive wheel, the faster the belt moves. The other side of that coin is.... as the diameter of the drive wheel increases, the faster the belt speed, but the lower the torque (in general).
If you're building a variable speed machine (with a VFD) the difference may not have a significant impact on the grinder's use, however, if you're building a single speed machine, I would recommend the 4" drive wheel. A slower belt speed might require a little more time at the grinder, but you won't "burn" things nearly as readily as you would with a faster belt speed....especially when using finer grit belts.