Harry 'devastated' as he quits charity set up in Diana's honour