Heat a 3-quart saucepan over medium heat. Cook and stir bacon in the hot saucepan until crisp, about 5 minutes; remove to paper towels to drain.

Cook and stir celery and onion in drippings in the saucepan until softened, about 5 minutes. Stir in flour; cook 1 minute.

Add potatoes, clam juice, milk, and clams with their liquid. Bring to a boil, then reduce heat to low and simmer for 10 minutes to blend flavors. Season with salt and white pepper.

Ladle soup into bowls, garnish with bacon, and serve with oyster crackers.