Investment bankers that make a market in flow servicing rights increasingly are worried about the role Fannie Mae and Freddie Mac play in the space and are beginning to wonder if the two government-sponsored enterprises are creeping beyond their charters.