Preheat the oven to 400 degrees F (200 degrees C). Lightly grease a 9x13-inch baking dish.

Whisk together Alfredo sauce and milk in a medium bowl. Spread 1/4 cup sauce in the bottom of the prepared baking dish. Layer 1/3 of the potatoes over sauce in the dish. Sprinkle with 1 tablespoon Parmesan cheese; salt and pepper to taste.

Combine ham, broccoli, and 1 1/2 cups Swiss cheese in a separate medium bowl; mix well. Spread 1/3 of the ham mixture over potatoes in the baking dish. Top with another layer of potatoes, followed by ham mixture, finally topping all with remaining Swiss cheese and Parmesan cheese. Pour remaining Alfredo sauce on top.

Cover and bake in the preheated oven for 45 minutes.

Reduce oven temperature to 350 degrees F (175 degrees C). Uncover the dish and continue baking until potatoes are tender, about 25 more minutes. Let stand 10 to 15 minutes before serving.