Back the piece, sharp bit and easy on the pressure when it is about to cut thru!!
Another thing I have done is clamp the scales on the handle and drill the hole with a small bit first. Then drill the scales on both sides with a larger size bit from the outside to inside. That way if you get chip out it will be on the inside!!
Once you have the larger hole drilled. Line the holes in the scales and the handle with a bit of that size on both ends and now drill with the final size you need, (whatever that may be)! This makes for perfect alignment for the pin! You don't want to get all glued up and the pin will not go thru!!
It sounds like a lot of hassle but when dealing with wood that easily chips out it is well worth the trouble!!