In a recent video answer, Google’s head of search spam Matt Cutts said that page speed is not any more important in terms of a ranking factor for mobile search than it is for desktop search.
Matt Cutts said:
It is not that in mobile, we apply that [page speed ranking factor] any more or less than desktop search.
But Matt Cutts did say that Google knows mobile users expect faster loading pages, and Google will continue to look into that. If it makes sense for them to tweak their algorithms just for mobile search, Google reserves the right to do so.
Here is the video:
I would not be surprised if Google does indeed make page speed a more important factor for mobile search in the future.