You can check how fast your site is by going and entering it here in at the PageSpeed Insights tool. On one of my wordpress blogs, a pretty standard template I was getting only 61/100 on the Desktop test.
1./ By simply installing and enabling the Async JS and CSS Plugin that went straight to 70/100.
2./ By installing the WP Super Cache plugin I went up to 72/100
3./ Compression was the last easy win on my list so I installed the GZip Ninja Speed Compression plugin and retested my site 86/100!!
I recommend that you test out your sites and do what you can to score as high as possible.
(it will take 2 or 3 weeks to see any difference in search results as Google takes it’s time to update the rankings)