FHFA Director Mark Calabria has vowed not to monkey with the 30-year FRM or lower loan limits in high-cost states like California. Still, certain stakeholders worry he may try to adjust other areas of businesses.