Well, having the ability to fight doesn't necessarily make anyone worthy of leadership. Otherwise your average school bully would be an effective President, King or Prime Minister. 'tis nonsensical to think so: the sole virtue of direct strength is that it shows you might have the ability to protect those you are responsible for (i.e. the subjects of the Empire). Absent that, it's sole virtue is in getting rid of those idiots that think physical strength is all it takes to rule.