Keep doing what you do. It is a good filter method which protects and helps you get rid of clients that want cheap labor or quick fixes.
I cannot find anything wrong with your method of estimating the amount of time you find necessary for getting the work done.
If this is what you need to deliver quality, then there is no reason to change. Not if your business generates profit.
Yes, there are faster workers. And, yes, there is cheaper labor force.
This being said, stay who you are and negotiate what you think is best for your business in order to maintain a high rate of client satisfaction (and implicitly generate word of mouth) by delivering quality and high standards.
The market has an invisible hand and it is capable of reaching an equilibrium without the need that you lower your prices or rush the work you do (just because one of ten clients says so). If your clients want someone else (faster, cheaper or just better) let them have this other person.
"Is it worth not doing the project?"
YES. If you lose money by lowering your standards or by making too many concessions then yes, it is worth not doing the project.
"How do you deal with these types?"
You negotiate what's best for you. If you feel you need a certain amount of time and if you aim for a certain amount of money, than defend your interests by being the better negotiator between you and your prospective client.