Probably, everyone who's in freelance/outsourcing business have had these problems. I'm not thinking my answer will contain an exhaustive list of your possible mistakes, but here are several ones that caught my eye when reading the question.  
Prepend each item with "possibly", "IMHO", "YMMV", etc.

* **Too high quality standards**. It well may be that you are requesting too high quality from the very beginning. For instance, if it's a mockup/prototype, your contractor may spend too much effort while developing it with full quality when it is not needed. Altogether it would lead towards losing time when working an actual product. I would expect that each subcontractor in a line may produce lower-quality result, and your role is to make it higher (finally, you are getting paid specifically for that).
* Don't be afraid to **delegate**. *"If you want to do it right, do it yourself"* seems to be a wrong approach. Some tasks, indeed, require maximal quality your business entity (yourself plus subcontractors) can reach, hence they require your best attention, but it does not mean they can't be delegated — at least, partially.
* **Poor communication**. *"leave contractors alone until they reach deadlines"* — I can't really think of a worse thing.  
**"Fire-and-forget" subcontractor is a treasure**, I have barely seen a dozen of such people working as subcontractors as they usually quickly rise towards opening their business which is larger than mine. Finally, *they* are in position to hire me, not vice versa.  
Usually, we are working with *average* people (not in a negative sense). People tend to make mistakes, and *continuous communication is the way to let average people to produce great results*.  
So, [Schedule Your Communication](http://freelancing.stackexchange.com/a/197/61). Don't set up yourself for a failure by [breaking deadlines, ever](https://workplace.stackexchange.com/a/5589/2145).

---
A bit of philosophy to answer a question raised in a comment. This is a pure opinion.

There's no ultimate answer on how to keep with own tasks.  
My (a bit idealistic) understanding is that *if I'm getting paid for a job, the job is mine*. I can outsource it partially, but I can't avoid personal involvement, quality check, etc.

Think of a large company (call them A) outsourcing at a subcontractor (B). B well may have different quality standards (not necessarily weaker than A's, but anyway). In order to place A's label on a product they **have** to be involved, e.g. run their own testing. This is why you may find cheaper "copies" of electronic equipment labeled B (or illegally labeled A). Chances are that you will find absolute copy, but out of a shipment of 100 items, say 5 would fail, while the A's quality allows maybe 1.