Collecting payment via a credit card is convenient for you and your customers, but it definitely comes at a cost, so it's better to not think about 'cheap', think about 'added convenience factor' and charge accordingly. Plan on it costing you 3-4% of every transaction, then don't worry about the cost.
With that out of the way, there are lots of ways to accomplish the task. If your business qualifies, a traditional merchant account can offer both standard swipe services and access to features for online payments. Merchant service companies usually have deals with ISO's (independent service organizations) to push their services. An ISO makes a commission off your credit card volume. I've become an ISO myself when I did some work setting up a client's online store and I now collect a small monthly check from all their CC business.
For 'less-traditional' businesses, you may have a hard time finding a service company willing to set you up. In that case, look at services like Square or PayPal Here (there are others -- I think Intuit has one that integrates directly to Quickbooks). These services typically cost more per transaction, but sometimes lack the monthly fees and qualification issues that make it hard to obtain or justify a standard merchant account.
Know that some businesses are simply not desired by card service providers: online subscriptions, selling firearms, etc. Read the fine print on any service agreement before you start using it. If your business violates any of the articles, you may find that they can close your account, seize your funds, and even pull funds from your back account for past transactions.