While I wouldn't call it eclectic, my taste in music is somewhat off kilter. I'm probably one of the few people in the world who can listen to Aretha Franklin followed by Bolt Thrower followed by Sibelius followed by Thomas Dolby and not think anything of it.
While I can't play an instrument or sing to save my life, I do enjoy discovering new music. My two favourite places to do that are Bandcamp and Jamendo.
Bandcamp is site that allows independent artists to upload and sell their music. You can preview songs, and download whole albums as a set of MP3 files. You can buy those albums for as little as $9.
Jamendo operates on a different model. All music is available under a Creative Commons license. That means you can play, copy, and share the tracks as long as you credit the musician and don't sell them. Jamendo also lets you purchase royalty-free licenses for music.
You can spend hours, maybe longer, exploring both Bandcamp and Jamendo. Admittedly, not every artist and their music will appeal to you. But I'm sure you'll find something. I always do.
And if you'll permit me, I'd like to make two recommendations. At Jamendo, I stumbled across Italian jazz man Paolo Pavan. Over at Bandcamp, an artist I keep going back to is Icelandic cellist Hildur Guðnadóttir.