It's not just a waste of disk space. It can speed up your system.
I just read an article on the O'Reilly Network about the troubles with hardware configuration with Linux and how their caused. I always wondered why each and every kernel revision broke the drivers ...