I would. In fact I would charge more for anything that took more time, attention to detail, skill, etc....
Not that you're trying to make so much per hour, but you do want to make something.... especially if you use more consumables to do it. I.E., sandpaper.
I think a there's a general rule of thumb for what to charge for a knife. Something like for what you sell one knife for, you should be able to make at least two more just like it? That includes material (steel, wood, micarta, pins, kydex, rivets, leather, or whatever...) as well as consumables (sandpaper, grinding wheels, electricity, gas, drill bits, epoxy, etc...) Time is rarely factored in, as it might take you 8 hours to make a type of knife, where another maker might be able to make one in 4.
That's not to say you should eliminate it from the equation, as some things do inherently take more time than others, but just make sure you're not losing money at the end of the day when all things are considered.
Another thing is that obviously the custom knife market will dictate prices to an extent based on the type of knife, fit and finish, materials, etc... but you should still be able to make money on it when all said and done.