The short answer to all your questions is that no one really knows what happened, if anything.
It's just not part of public knowledge and it's doubtful we will ever really know.
Morning Musume's situation reminds me of another notable case of a popular act getting the cold shoulder for years on end.
In 1978, Pink Lady at the height of their popularity tried to do their own new year's show, in competition with Kouhaku.
After that, they weren't invited back for the next 10 years.
Make no mistake, the people who run this show are far from impartial.
As you suggested, it could very well be that someone on the song selection committee is carrying such a grudge for UFP acts.
Alternatively, certain other popular idol groups could have threatened not to appear if our girls were invited.
I have no basis for this, but would anyone be surprised if this were the case?
Some acts appear every year for decades, despite not releasing anything new for years.
Meanwhile, Morning Musume gets #1s for 3 weeks out of the year and gets no recognition.
No one knows exactly how the song selection process goes, but it's clear to me that it isn't based on merit or sales figures.
For me, there's no such thing as "deserves to be invited" anymore--it's just lost all meaning.
Kouhaku hasn't given me a reason to watch for most of the last decade, and in fact has given me dozens of reasons NOT to watch.
At this point, my opinion of the show has gotten so low that I'm not even sure I'd be all that happy if Morning Musume was re-invited.
Honestly, I see Kouhaku as a corrupt circle-jerk that isn't worth getting worked up about.
Your questions are interesting ones for discussion, but I just don't care that much anymore.