Heat oven to 375°F. Scrub potatoes; pierce with fork to allow steam to escape. Wrap garlic cloves in foil. Bake potatoes and garlic 1 hour or until potatoes are soft.
In 8-inch skillet, heat oil and rosemary over medium heat 2 to 3 minutes, stirring frequently, until rosemary is fragrant.
Cut potatoes in half; carefully spoon potatoes into large bowl. Save skins for another use or discard. Slip skins off cloves of garlic; discard skins. Add garlic, oil mixture and salt to potatoes. Mash until fluffy, adding milk until desired consistency.