My go to answer is always.... YOU have to live with the horse so if you are not enjoying a horse with testicles, then geld. Testicles must be earned.
In Europe the stallions are shown, the mares have babies. Some ride the mares and show them but it's not as common as riding stallions. So having a bunch of stallions in a ring is not uncommon. But in the US the trend is Geld or buy mares.
Stallions are fantastic to ride. You don't have to breed a stallion for them to stay a stallion you can just ride them and enjoy the macho that comes with the extra testosterone. HOWEVER, you should have them properly housed and fenced so that no accidental pregnancies occur. Or accidental visitors that would get the stallion in trouble. At our farm our stallions have an extra rope or halter tied
Other considerations, not many boarding facilities allow stallions and some shows have age restrictions on who can handle stallions. I know in Australia there were even shows that didn't allow stallions and extra rules for stallions if they did. Like green tags on the bridles (made for ugly show photos, but warned the unaware mare owners you were there). Also Stallions had to be double tied, these were outside trailer tied shows so it was not set up for stabling a stallion like some US shows. I know some US shows have trailer tying, some have stalls in barns. Each show is different. One suggestion if you want to alert people to the presence of a stallion in this gelding/mare show world. I have gone out and bought normal saddle tress, or saw horses from the hardware store (home depot or lowes et all) painted them bright colors and wrote stallion across them. When I set them up on either side of the stallion or alley way it makes people look before just walking up on him with their in heat mare, or just flat out not paying attention. It's not that my stallion will do anything wrong, it's that it's rude to tease him if he's standing there getting tacked up or waiting on his class.
Now if you do decide the time has come where you want a gelding instead of a stallion then that is great. Don't feel bad, you are the one that has to live with them. If he's so absolutely genetically important for the breed to survive and excel then it might be an option to sell him or lease him out. Or even collect, freeze and then geld to preserve the genetics. BUT most of the time they are amazing to you, and your special friend. you can still preserve his DNA, but not every stallion needs offspring. There are plenty of horses in the slaughter sales that you don't need to be adding more. If you have a purpose and destination for the offspring great, but if that stallion is a terror to be around.... will the offspring also be the same way... so is it worth it.
Timing for gelding. There are vets that geld when the foal is born. I personally don't recommend doing it that early since there is so much growth associated with reproduction hormones. But if you can see both then they can be removed. I also don't like to put foals through such trauma, they already have such a hard time in the first 6 months of life. I would wait till after 6 months of age at min. because horses have immunity from their dam's colostrum, that has to last them till they are about 6 mo and can start forming their own immune system. If they use up that immunity fighting castration, then you'll likely have to do a plasma transfusion to get them through till they are 6 mo. Also I like my colts to have tetanus and some vaccinations before they go under the knife. 6 mo is usually when vaccinations are affective, and depending on the label on the vaccination might be the soonest the vaccine can be given.
You can wait till they are 2 or 3 to see what kind of man they turn into. I have stallions that don't even know they are studs, so they don't bother me to stay studs even if not breeding. I also have some that turn into studs on steroids and need their egos checked all the time. I don't mind it, but I know it would be a lot more than some could handle. If they get violent or untrustworthy, game over, jewels are gone.
Stallions that get into their teens and are not gelded will have more chrest development and larger testicles. They can be gelded, but it's at a risk of the chrest falling. But if a stallion comes to me that is not breeding quality, is not quiet to be around and is not safe then there is no reason to cry over a crest, bye by jewels.
Stallions that are in their 20's, now that is a riskier surgery because of their age, it can be done, but if they are safe and can be stabled properly and they don't have anything wrong with their testicles I would just leave them. But even a 20 yo jerk can have his jewels removed.
So long story short. What are you willing to live with, put up with, how do you want your time spent with your horse. Make sure your decision is for your sanity. You can always collect and freeze if DNA preservation is necessary (some don't freeze or don't have conceptions with frozen so have that tested prior to removal, but most have some ability to make a foal if needed). And if you choose to keep your stallion make sure that you only breed with a purpose and plan for that foal.