Um. ... A lot of people will say that it's a very silly or shallow or superficial concern. After all, "fashion" is really an artificial construct - the important part is the function of the clothing, that it perform for protection, safety, and comfort.
However, I'm not among those people - in life in general, certain things are important to me, and one of them is how I dress, clean, stylish, color coordinated. I have almost 300 ties ... Probably almost that many shirts, and probably 50 suits. I like to make sure everything color coordinates when I'm going out of the house. The only time you will find me looking like a slob is when I'm doing serious heavy dirty work - fir example, just power washed and bleached decks, patios, siding and wore horrid old clothes because I get bleach spray drift.
I don't know about where you live, but here in the American Midwest, we have a great lack of sense of propriety in dress, grooming etc among the "general public" - go to any "big box" store such as a Walmart and it's often not attractive ... Especially the 300 lb women trying to dress in skimpy clothing like some young hottie.
Same sense of appearance is important to me on my bike. Luckily the bikes I ride most, hybrid and road, are both black with just a little red accent detail, so many/most color schemes work .., my winter bike is white with black accents ... Truly anything goes...
... Or can be made to work. Items like socks, jackets, gloves, and hats can be mixed and matched to make it work.
There is a lot of truth in "dress for success"
A lot of people say a lot of things. What is important to you is what matters first. Naysayers always exist to drag you down. Disregard them or they will drag you down.