It can be done!
Jason Becker from Gabcast pointed me towards the Coral Content Distribution Network. If used properly, the Coral CDN can remove almost all (if not all) bandwidth considerations for every file on the Internet - podcasts and video casts included.
You'll need enough disk space to hold your own audio files, but bandwidth won't be an issue which would allow podcasters to start using less expensive podcast hosting solutions.
So how do you use it? Simple. When you upload an episode (or any file, really), modify your enclosure tag and add the .nyud.net:8080 to the end of the domain. As soon as the first person accesses your file via that URI, the Coral CDN will start to mirror it.
For example, let's say I have an audio file on my jonwatson.ca server named episode1.mp3. The normal link to that file would be http://www.jonwatson.ca/episode1.mp3 (link won't work). If I wanted the Coral CDN to mirror my show and provide the bandwidth, I would have to change all my links in my enclosure tags and anywhere else I might link to that file to http://www.jonwatson.ca.nyud.net:8080/episode1.mp3.
I wouldn't necessarily recommend using Coral CDN for this, but it makes for a nice example. You can simply append the .nyud.net:8080 to any host name for any file and Coral will start to mirror it for you.
Many possibilities, but be aware of privacy issues. As soon as you publish a link with the Coral CDN suffix on it, that file will be mirrored all over the world and any hope of privacy will be lost. Ensure you only use it for stuff that you want to be widely disseminated.