1. Could Ned's evidence about the bastardy of Cersei's children have convinced a Great Council, such as gave Aegon V his crown? Is this evidence completely unequivocal in the context of this fantasy world (it wouldn't be in our own)?
Having just given a real life deposition in a court case in which I am a witness, I can tell you that no evidence is "completely unequivocal," either in Westeros or the USA. Lawyers will argue about anything.
And the Great Council is a very rare event. Westeros has no equivilent of a Parliament meeting at regular time. If Ned had been able to establish his authority as regent and summon a Great Council... well, the lords would have presumably heard the evidence and decided...
2. Was Littlefinger saying the truth, when he told Ned that Tyrells and Redwynes would rise against Stannis's succession?
No comment.
3. Do members of one Great House have a legal right to arrest and judge members of the other? I.e. was Catelyn's abduction of Tyrion, given all the incriminating evidence, legal?
It was a bit dicey. A lord administers justice in his own lands. Catelyn would have had a much stronger claim if she had taken Tyrion in the north. Even in our own world, there are always dangers in taking on the rich and powerful, regardless of the legality of your auction or how much evidence you have... and the high lords of Westeros are a deal more prickly about their honor.