Rumors are swirling that Simon Cowell is second-guessing his decision to bring Britney Spears on as an X Factor judge. After Britney walked off the X Factor set several times since the show's second season has started filming, Simon Cowell has had it.
