He paid you.. what's the problem??
Since he paid you, he clearly has full intent on following through. I also don't see 2 meetings falling through as "impossible to get ahold of". Don't forget you are in a service industry.
If your time zones are wildly different, that may be the issue. And, in fact you may want to give him a bit more leeway due to this. I realize you state you are "waking up early" but just schedule something so you don't have to. Find a common ground. You can't be more than 8-9 hours apart. So you should be able to schedule something at the start of your day and the end of his or vice versa.
Being annoyed that someone paid you and you've had simple meeting scheduling conflicts for a couple weeks is a bit over the top in my opinion. Good customer service would be of a mindset that he's paid, lets work with him, not get annoyed that a meeting is proving difficult to schedule.
It's one thing if he hasn't given you anything, but it's another matter entirely if he's proven intent by paying a deposit.
All that being posted... I too get annoyed and cancelled meetings at the last minute or no-shows. That's a different matter.
See here: What to do about negligent clients
-- but that does deal with a client prior to any commitment (deposit) so with a deposit I'd expand to allow at least 3 cancellations before I say anything.
Then I'd just express that my time is valuable and I have a fairly full schedule. I can't continually put things aside for meetings which are then cancelled at the last minute. If he's not prepared to move forward, I'll happily return the deposit.