I used to feel the same way about cycling clothes, but when I started
doing rides over ~40 miles, my butt got really, really sore. I finally
broke down and tried a pair of cheap cycling shorts (< $20). I felt
silly wearing them at first, but I was amazed at how much better I felt
during a long ride. If it weren't for that discovery, I don't think
I'd ever have ridden a century.
I usually wear a t-shirt for any rides less than ~90 minutes; but much
longer than that and I wear something that "wicks", i.e., a cycling
jersey (again, cheapo's). An added bonus is that I can carry snacks in
the back pockets.
I guess it depends on a person's body type/shape -- perhaps some people
can ride for 4-5 hours without any padding and still feel comfortable,
but I can't.
JR