GSBR: Geometric Support Behavior Regularization for Online RL

Online RL needs guidance for efficient exploration without being overly constrained by priors. In robotics, we can leverage the increasingly capable pretrained policies as prior, but improving on them might require discarding poor action modes and exploring around promising ones. Existing approaches are too restrictive in one of two ways: behavior-cloning tie the actor to the base-policy distribution, including suboptimal modes; support constraints allow mode selection but exclude better actions outside the support. GSBR is a better behavior regularizer that preserves both axes of improvement. It estimates a support region from base-policy samples and applies no behavior penalty inside it. Outside the region, the penalty grows smoothly with distance, allowing improvement when the value gain justifies the cost.