Combine shrimp shells, onion, celery, celery leaves, lemon, garlic, peppercorns, thyme sprigs, and bay leaves in a saucepan. Pour in water and bring to a boil over medium-high heat. Reduce the heat to low and simmer until reduced by half, skimming off foam as necessary, 45 minutes to 1 hour.

Strain and discard solids. Use immediately, or let cool to room temperature and refrigerate or freeze for later use.