Sometimes my students wondered why would i prefer to use Slackware and compile my own package from source if it's not provided by the upstream distribution maintainer instead of using binary packages which usually comes with a lot of precompiled binary like Fedora or Ubuntu.
Well, latest Distrowatch newsletter has finally gives a good explanation about the advantages (and of course disadvantages) of compiling source and using a binary package. I would recommend you to read it.
If you are looking for an optimized Linux distributions, i would suggest you to go for Gentoo, source based distribution which lets the user decide how to optimize their system in many ways. They don't have any binaries and all of their packages are compiled from source, thus it's way more optimized than any other Linux distributions.