As suggested, there's
PlayOn, ($) and free scripts and plugins for it, as well as PLEX (free). There is also a free channel from PlayOn called "My Media" that lets you serve media from your computer. All require you to run server software on a computer on your network. There are also some apps for android and i-devices that let you control your Roku over your network.
But a direct hack to run a different OS, or a web browser, NO.
I use both
PlayOn (worth every penny for Hulu free and YouTube alone!) and Plex, as well as a number of "private" channels for my Roku.
Here's a blog post on the subject:
Cache Free TV: What Can You Do With a Roku?