Drain potatoes and return to pot to cool slightly. Roughly smash potatoes in pot with a masher or a large fork.
Heat a large nonstick skillet over high heat. Add the bacon and cook until bacon is crispy and has given up most of its fat. Lower the heat to medium-high, add the rosemary and season with salt and pepper. Make sure bacon and rosemary are evenly distributed in the pan then add the potatoes.
Pat down with a spatula so the potatoes are in an even layer. Season the top of the potatoes with salt and pepper.