Since there was an upfront payment, I'm assuming there was some sort of contract or understanding at the very least. So, follow the instructions in the contract.
If there was no contract or the current situation is not addressed by the contract, it becomes less obvious whether you should refund money.
Many factors need to be considered in addition to any potential legal issues. How much do you need the money? How was your relationship with the client and do you expect more work from them? Can and might they hurt you financially in some way?
Personally, I would never give money back until the matter has been settled, as you will never get that money back. Having received money gives you leverage, which should be used fairly to get what you at least deserve considering the time spent.