I only bill hourly, so my perspective is going to be a bit different. To start off, in your specific case, I would have probably done the same thing.
What's your reputation worth? You did spend some time, and there's no guarantee (that I'm aware of) on oDesk's fixed bid jobs - so you could have kept it. But what would that have cost you in reputation (not just with that client, but in general)? In your specific case, considering that oDesk's ratings are directly tied to the amount spent, it makes sense to refund (at least some) when the project failure falls on you (you increase the potential of positive feedback, while lessening the impact of any negative feedback on your rating).
What did the client get? In some cases, just knowing something isn't possible is a worthwhile outcome to the client. In this case, you seem to be saying that another freelancer could have completed it - it was just more than you were comfortable. So the only thing the client got was a better understanding of your abilities - and they shouldn't have to pay for that.* On the other hand, maybe I'm misunderstanding the situation, and the client asked for something that just may not be possible. If that's the case, and the client was aware of that possibility, there's less reason to refund - especially if they're not asking for it.
In cases like that, I generally let the client know I'm going to take a few hours to test their concept, and see if it's possible. That's certainly billable time, but the likelihood of failure is made clear.
However, with fixed big projects, you're selling the final product (and a product with parameters that someone else has mostly defined) - so it's harder to do that. Since I only sell my time, it's easier for clients to understand what they're paying for (but that's a topic of it's own).
[*This may sound harsh, I don't mean it to. You realized this at the right time, a much better time than when the project is supposed to be reaching completion.]