Place the cubed potatoes in a pot with enough salted water to cover. Bring to a boil, and cook until tender, about 30 minutes; drain. Add butter to drained potatoes to melt; set aside.
While the potatoes are cooking, place bacon in a large, deep skillet. Cook over medium high heat until very crisp, about 5 minutes. Remove bacon to paper towels to drain; allow to cool and crumble. Leave 2 tablespoons bacon drippings in skillet and return it to the heat.
Cook the onion and garlic in the bacon drippings until tender, about 4 minutes. Stir in the rosemary and cook for 1 minute more.
Mash the potatoes with the melted butter. Mix in the crumbled bacon, the onion mixture, and milk; season with salt and pepper.