I cannot claim that this is more than speculation laced with some prejudice. I haven't been interviewed for a senior public appointment since the 1980s (when I was applying for professorships - the universities of Reading and Warwick and Leeds Polytechnic took a look at me). And I haven't interviewed others since the 1990s when I was on panels appointing university lecturers.
Anyway, here are my speculations, triggered by reading about the latest NHS scandal involving the Care Quality Commission:
1. People who apply for senior public appointments include a high proportion - let's say over half - who would not be fit for purpose if appointed; they would be no good at the job. But they want the money and the status or they just want to avoid being sacked from their current job. (Head-Hunting - Sponsored Mobility - is thus a good practice; it broadens the pool of candidates to include able people too modest to advance their own cause).
2. A high proportion of candidates - let's say over half - actually selected for interview would not be fit for purpose if appointed to the job. They would be no good at it, but they are good enough at filling up forms to get through to interview. They also have friends to write them nice references or enemies who want to get rid of them to do the same.
3. The interview process itself is not fit for purpose and is to a significant degree unable to identify and eliminate candidates who would turn out to be bad choices if appointed. This is in large measure due to the practice of what I will call Wooden Po-Faced Interviewing : for example, when each member of the interviewing panel is allowed One Question with no follow-up.
In addition, as part of something probably called an Equalities Policy, members of the panel will have been told that most of the interesting questions you could ask a candidate may not be asked.
The opportunity for an interview to reveal that someone is decisive, imaginative, firm, or inspiring are deliberately reduced to zero. Likewise, the opportunity for an interview to reveal that someone is weak, muddled, paranoid, or a complete shit.
4. The weakness of the interview process is probably compounded where there are pressures to avoid appointing yet another white male, since the field of choice is then tacitly reduced - white males may be called to interview (that's the Equalities Policy at work) but it's as far as they get (that's the Equalities Policy at work). In the worst cases, there is only one candidate in the room.
5. As a result of all this, at the end of the day maybe half of all senior public appointments are filled by people who just aren't up to the job.
Human Resources has a lot to answer for. Of course you must nurture staff, help them develop, acquire new skills, gain wider experience, prepare them. But Nice Guy stuff is only half the job. The other half is making sure that those who end up in senior public posts can do the job and want to do it, that they are willing to act decisively (and will be supported when they do), that they can think beyond their pension entitlement, and that if they actually have some spark of life about them - well, that that will be at least tolerated by the organisation.
Postscript 20 June:
I wrote and published the above before the names of those at the centre of the Care Quality Commission scandal were released. Now we have the names - Cynthia Bower, Jill Finney, Anna Jefferson [ I have added them to the labels for this Post] - and you can Google them. Start with Bower and you will very quickly discover that she already had a track record of incompetence before her appointment to lead the CQC and that this was a matter of published comment at the time, back in 2009 ... Next question, Who did the Interviews for this post and made the appointment decision?