If I understand this correctly, these are used because many tubular rims are deep and a normal tire's valve is too short to extend past the rim height.
So if I use an extension to clear the rim height and inflate the tire. Can I remove the extension and then tape over the hole in the rim where the extension was? (To keep an aero profile)
Or do valve extensions stay affixed to the tire's valve/stem?
Yes that is what valve extenders are used for. With all of the valve extensions i have used u can take off the extender once the tyre has been pumped and tape over the hole. You shouldnt have a problem.