Ride less, cycle faster

Training books will all tell you that it is the rest days between cycling that are just important as the training rides themselves. This is because 'your body needs time to recover', they tell you, or 'your muscles develop when they are resting'.
It's one of those things that I have heard very often but always found it slightly hard to believe, unless your daily rides are really very intensive training rides. If, like me, your main priority is just to get out riding as often as possible, is that really making you slower?
When the weather is good I get out cycling most days, often just for an hour although sometimes for two or three hours. Occasionally, due to weather, health or holidays, I have periods of a couple of weeks where I do much less cycling. Like must people I suppose, except I am lucky in that I can cycle most days. I also have several routes that I ride regularly and know very well, and know my average speed for these routes very well.
I've been looking back at my cycling records for the last year or so and unfortunately one thing is glaringly obvious - my best (record) times are almost all immediately after a week or two where I have only cycled a couple of times rather than when I have been cycling every day.
So although I don't usually feel as if the previous day's ride has left me unable to give my best, in truth it seems to be the case. I am reasonably certain that the text books are correct and that cycling less often - perhaps three times a week instead of five or six - can leave you cycling faster than before.
Why do I say 'unfortunately'? Because the main reason I cycle is because I like cycling, not because I want to increase my speed, but it's nice to think I'm getting slightly better each time. It seems a shame that all that pleasure is slowing me down - and I'm a bit concerned how slowly I'll end up cycling if we have a long warm summer ahead of us and I'm out every day!
Note: another possible way to read these results is that going out every day is really a good thing for building overall strength and fitness, even if it does over-tire the legs and hold back peak performance, since you are building these qualities in readiness for your 'big days'. Then when you know you are having days where performance really counts (annual club competition ride etc) you are best to do no cycling at all in the two or three days before the event if you really want to be in the best condition possible.