Place bacon in a large, deep skillet, and cook over medium-high heat, turning occasionally, until evenly browned, about 10 minutes. Drain bacon slices on a paper towel-lined plate, crumble, and set aside.

Cook and stir onion and celery in the remaining bacon grease over medium heat until the onion is translucent and tender. Drain excess grease, then stir in potatoes. Add water to cover all but 1 inch of the potatoes. Bring to a boil over medium-high heat, then reduce to medium-low, and simmer until potatoes are tender, about 15 minutes, stirring often.

Stir in evaporated milk, and continue cooking until warmed through. Season with salt and pepper. Stir in bacon just before serving.