I put Fedora 9 on mine, but it's far from an easy install. I believe F10 is easier. And getting wireless to work was a pain. Put it this way, it took me a DAY to install it!
You get full control over the PPU and 6 SPUs. There is a full SDK available.
RSX can be used with firmware <= 2.10. But if you want to do anything else online you'll be wanting to upgrade your firmware.
I haven't really had the time or inclination to code much for PS3. I wanted to do a distributed Mandelbrot renderer as an example of how to code the SPUs, but only got as far as prototyping it on Windows - even with a dual core I got about 2x speed up by multithreading over a single core.
Jim