Prophet Muhammad (SAW) was prophet of Allah when only Mawla Ali (AS) believed in him, and he will be prophet of Allah if no one believes in him, or everyone believes in him. Prophet Muhammad (SAW) only became the leader of the Islamic estate when the people gave him that position.
In other words, political leadership cannot be possible without the consent of the people. During the era of Abu Bakr, Umer and Usman, they had the consent of the people, due to which Mawla Ali did not try to over throw them, neither support them. As the consent of the people was with the rulers of the time, it was not Mawla Ali's duty to lead the nation, when the nation did not want him to lead it.
When Mawla Ali became the caliph, he fought Mawiya. You must note that Mawiya had been ruler of Syria during the era of second caliph, Umer, but Mawla Ali (AS) didn’t fight Mawiya. As soon as Mawla Ali became the caliph, he called on Mawiya for accountability of Public treasury and thus the battles began. Because Mawla was given a political post by the people now it was now his duty to rectify things as the head of the state. If Mawla Ali was not given khilafat by the people, he would have never fought Mawiya. In other words, if Mawla Ali was not chosen by the people to lead them, he would have remained silent under Mawiya's rule as well, just like he remained silent against Mawiya for his seventeen year rule over Syria during the Caliphate of Umar and Usman. Silence is not loyalty or support all the time. It is a boycott as well.
The thing which u call loyalty, I call patience and dissociation. Mawla Ali (AS) went into political dormancy which is the reason he and Zulfiqar did not participate in any “jihad” under the first three caliphs.