Does readability really improve if absolutely everything grows when the font size grows?

If the side margins grow, there will only be room for less text per line. Therefore, side margins/padding should be specified in pixels, and top/bottom margins/padding in rem. Read more here.

Another alternative is to change the media queries to use rem instead of px, so that you get mobile view on large screens if you have a large font size.