When you say "all," your statement really isn't true, based on his question. If you switch Aldor/Scryer factions, yes, rep can decrease for the opposing faction. But in general, the statement above is correct. Some factions, such as the Bloodsail Buccaneers, will decrease as you kill members of their factions. This isn't a problem as there generally isn't a reward for getting friendly with them. Timbermaw Hold, though, is another example of a faction that can decrease on you when you might actually want something from that faction. If you kill Timbermaw furbolgs, your rep will decrease, so you'll want to get friendly with them. At minimum, this lets you pass the cave between Felwood and Winterspring without fear, but higher levels of faction give the ability to purchase items and trade skill recipes that will be useful to some.
At this time, the statement above is correct for The Consortium. Once you have Consortium rep, it does not go away.
This post is faded because it has a negative score.