Why you should care
Every visitor to your website is a potential customer. And every potential customer that has to wait for your website to load is a customer you might lose to one of your competitors and with it real revenue.
What you should do
Make sure your website performs excellent! Monitor your site's performance closely and fix problems immediately. Realizing that there is a problem with your website will save you real money.
Even if your website is just a minor marketing tool for your products and/or services, improving overall customer satisfaction will proof to be beneficial in the long run.