It's a decision that I guess we'll all need to make, do you go with GFXLIB2, SDL, Tinyptc, Allegro, whatever.
Personally I've decided to go down the software render route for now but this is because most of the fun for me is making those boring little routines to draw polys, render gourad triangles, do texture maps etc.
Once I have them sorted I may well start to explore the other avenues.
For all the messing about involved in removing unwanted stuff from the GFXLIB2, you may just as well write your own stuff or use someone elses Clyde. I will be releasing my libs as I develop them if that's any help.
It's not as if the exe size is diminished all that much after it's been crunched anyway. If you're happy with GFXLIB2, use it

Lots of people do and there's nothing wrong with that stuff.