Place potatoes in a large pot and cover with water by 1 inch. Bruise rosemary with the dull side of a knife blade; add to potatoes in the pot. Add salt, garlic, and bay leaves; bring to a boil. Reduce heat and simmer for 5 minutes. Drain and allow potatoes to air dry.

Preheat the oven to 375 degrees F (190 degrees C).

When potatoes are thoroughly dry, return them to the pot. Drizzle with olive oil and season with cayenne pepper and kosher salt; stir to coat well. Transfer potatoes to a rimmed sheet pan.

Bake in the preheated oven until potatoes are golden brown and crusty and flesh is creamy and soft, 35 to 45 minutes.