There are a variety of stretches and strengthening exercises you can do. Some guys will tell you that stretching and core exercise is a waste of time, but they're generally the one's who are yet to experience low-back pain. First, how's your hamstring flexibility? Can you reach down and palm the ground with your knees locked, or at least touch your toes? If not, the typical racebike position might be too extreme for now. And for core strength, can you do leg raises easily without pain, and hold yourself in the "plank" position (elbows and toes on ground) for at least 30 seconds?
You can work on hamstring flexibility by sitting on the floor with legs straight and extending your arms/fingers to grab your toes. Do it gently, hold for 20-30 seconds, then release and relax. Repeat two more times. Next one I do is sitting on a bench, lower your upper body so that your shoulders are on or under your knees. As you improve, you'll be able to sit on the floor with straight legs and get your shoulders close to your knees. Again, 2 or 3 times, holding for 20-30 seconds each works well. Don't push it too hard, just to the point of really feeling the stretch. Final one I like is the "grapevine", where you sit on the floor, put one leg with bent knee over the other straight leg. Then, you twist your upper body to put your opposite arm over the bent knee. Once or twice for 20-30 sec, then repeat on other side.
For ab strength, I like leg raises and crunches, as well as heavier weights at the gym once a week. You can start with one-legged raises, keeping the other leg bent with foot flat on the floor, then after a few months move up to two legs at a time. On these as well as crunches, keep your back flat on the floor, pressing the small of your back down to start the move.
For lower back, I like old-fashioned "good mornings", just bending forward at the waist with hands folded in back of your neck. Planks are a good finishing move, since they work abs and lower back together. Just get down like you're doing a pushup, but rest your weight on your elbows and hold for 30-60 seconds.
OK, that's just what I do. A buddy here, former Cat I, uses an exercise ball as well as some side plank exercises and some different stretches. Important thing is to take your time and do this stuff daily. It only takes five minutes, and riding pain-free is worth the time and effort.