I agree, but with the caveat that a well designed PC/laptop site should automatically look good on a full size tablet. If it does not then you did not make it responsive enough for the various PC/laptop configurations your surfers may have.
But with 'phones and pocket tablets you need to be more creative. It is already good traffic, and with time will get better. Not designing for smaller devices is throwing traffic away.
However a lot of people use CSS, and that is so non standard between devices that it is near impossible to make a fully compatible CSS site for all 'phones/tablets (or all PC browsers come to that).
That said, sometimes it is impossible even with pure HTML. If you look at my most recent work:
http://women-in-bondage.info/cruel-bondage-movies/ you will see that the header is responsive enough for 'phones but the rest of the site is not. This is because I started intending for it to be 'phone ready, but soon realised that as I was using hosted videos I could not control my sponsors code so could not make the video pages responsive. And I cannot think of a quicker way to piss off surfers that to make the video menu responsive, then, when they try to watch a movie, for them to find the film does not work on their device.
Also, I am finding that many of my old sites require total rebuilding to make them 'phone ready, so I am leaving them as is and building new sites that are 'phone ready instead.
In short what I am saying is that I agree with you, but I do not 100% practise what I preach!