for some reason the client prefers to pay an agency a lot more than go out and look for developers
The reason is very simple, and is the same reason because companies prefer head hunters instead of direct hiring. In the reality, reasons are more than one. First of all they save a lot of time. Testing several people to choose the right one is a big time wasting for a company, if someone can do this for you, you'll be glad to pay even a lot of money. Moreover you may not have enough expertise to evaluate candidates. Agencies have this expertise. If you hire directly and the employee decides to leave (may be even suddenly), you're left on your own devices, and need to repeat the hiring process from scratch. If that person was through an agency, it's and agency's problem to find another one for you, and normally they do very quickly. Summarizing, most companies prefer to buy a service, instead of searching a person, even if it costs more.
Apart all of this, I'm a freelance that work directly with end customers. Yes, it's possible, but it's a world of trust. It's very very hard (not to say impossible) to go to one company and say "Hey, I'm the best freelancer in town, please hire me". If you want to be hired, company must trust in you. How can you achieve this ? In my opinion, the best way is word of mouth. I began working through an agency, but I used to have a strong relationship with end customers. They were appreciating me and happened, 2 or 3 times, that some of them recommended me to friends owning other companies. Because these other companies were not customers of my agency, it was perfectly ethical to work directly with them (while it's not ethical, if not even illegal, to stole a customer to your agency). If you're enough professional, you may see that word of mouth may take you to several customers in few months.
Another way is social activity. I don't mean direct searching of customers through LinkedIn or similar. This is a very hard job, and, as you told, almost totally in the hands of agencies. I mean having your own blog, where you post, with some regularity, interesting facts about your jobs (but never post "I want to be hired", just post information that may help your typical customer). Then spread your posts through social (LinkedIn, Twitter, Facebook, etc.). If you can, you can even write a book and publish on Amazon. You'll never raise money from a book (if you're not Dan Browne), but you may get popularity and you'll be authorized to say "I even wrote a book on this matter". Normally this impresses a lot. Most entrepreneurs think that who write a book is a guru of the matter. If, some of these, read your book, they may try to contact you directly. So link, in some way, your book to your blog, and be sure to be contactable through it. Anyway, social activity, for me is just an idea I never leveraged, so I may not be the right person to give you suggestions....
One final word regarding the way of working directly. Normally, the bigger the company is, more likely it'll go through agencies. So, if you plan to go alone, be prepared to address smaller companies. Small companies, normally, don't hire developers as is. They need ways to solve problems, so you'll need to be first a consultant, then an analyst, and finally a developer. You may be forced to change your way of working. Don't expect a small company to give you a detailed analysis document to start from. You need to prepare the analysis document by yourself, and then discuss it whit company's boss, and adjust it to his wills. You may even discover (it was not my case, but maybe) that direct customer pay you less than agency. A big customer may pay 100 to agency, and you may get 70. A small customer may want to pay just 60 for the same job. It's not uncommon. Another caveat of working alone is that you need to manage yourself. Now is an agency job to decide when and where you've to work. If you're alone you've to plan this by yourself. You may have weeks when you turn your thumbs, and other when you don't know how to fulfill all the requests. Be prepared for this. Best way to fill empty hours is to develop for the future. You may do thinks that no one ordered you, but may become very useful in the future. it's a bet, sometimes you win, sometimes you waste time.
Hope you can find your way.
