Head the busybodies off by making up rumors abut yourself, and spreading them around. Make them outrageous and shocking, ten times worse than anything they could come up with. Nobody will believe the talk anyway, and you'll walk away with your reputation unscathed.