Much of the decline came right at the top as Wells Fargo reported a drop of $19.85 billion in total mortgage servicing during the first quarter.