He says that he wants me and his design team to work together and our agreement was only for programming, not for design (he would provide
the flow and images).
This implies a change of project scope.
If he is talking about a single day, then I suggest you consider a separate invoice for your time - apply a discount since some of your time is paid under the existing contract. Present the customer with the quote, explain to him the additional costs help cover your travel time, and your hours of work which were previously flexible are less flexible. Perhaps not charge first time, but explain to him that any other onsite visits will be chargable.
Working with customers, like any relationship, can be tricky. You don't want them to waste your time, nor do you want to under sell yourself. Consider a rate for offsite work, and consider a rate for onsite work, and share that with the client. Explain that you have expenses to cover just like they do and the flexible working practices (like location/time) help you to offer a cost affordable service. When your working practice becomes less flexible, you have to adjust the cost accordingly.
If the customer balks and walks, examine your contract and find the exit clause. My contracts stipulate that my contract can be cancelled immediately if the project is cancelled. If your contract is the same, their walk should cost them some money (and cost you the chance of any further work from them for the next year).
Best thing though is to avoid rocking the boat - find compromise - especially if you think there is the possibility of additional work to follow.
When you talk with the customer - smile - be pleasant - your body language helps just as much as the words you use. Your customer did not wake up one morning deciding to make your day bad - their requirements changed so work with them to help them deliver the goals and you will profit along the way.
Lastly, for your next contract, include an hourly rate for onsite visits. Perhaps budget that 20% of your time will be onsite. If/when the customer asks you to be onsite, then go, but advise them that after x number of hours/days onsite, your time becomes chargable. I never say no to a customer - I just become more expensive, it gives them something to think about before they ask me.
Best of luck