Cook bacon in a large, deep skillet over medium-high heat, turning occasionally, until evenly browned, about 10 minutes. Drain on a paper towel-lined plate. Crumble bacon; set aside.

Preheat the oven to 350 degrees F (175 degrees C). Coat a baking dish with cooking spray.

Heat olive oil in a skillet over medium heat. Add chopped mushroom stems, jalapeño, and garlic; cook and stir until mushrooms release moisture and soften, about 10 minutes. Transfer to a bowl; stir in cream cheese, Cheddar cheese, and bacon. Season with salt and black pepper. Spoon cheese mixture generously into mushroom caps; place in the prepared baking dish.

Bake in the preheated oven until cheese begins to brown, 15 to 20 minutes.