Leaving Trump's side didn't make Elon Musk much more popular