Combine broth, rice, celery, onions, water, and bouillon in a large pot over high heat; bring to a boil. Reduce heat to low, cover, and simmer until the rice has absorbed most of the liquid, about 30 minutes. Remove from heat and set aside.

Melt butter in a medium saucepan over medium heat; gradually add flour, stirring continuously to make a thick paste. Whisk in 4 cups of milk, 1/2 cup at a time, whisking constantly, until smooth.

Stir sauce and chopped chicken into rice mixture; add some or all of the remaining 2 cups of milk to reach desired consistency. Season with salt and pepper; simmer over low heat, stirring every 15 minutes, for 1 hour.