Combine mayonnaise, mustard, honey, horseradish, hot pepper sauce, and cayenne pepper in a bowl. Cover and refrigerate.

Mix ground turkey, grated onion, jalapeño, barbeque sauce, Worcestershire sauce, liquid smoke, steak seasoning, and mesquite seasoning in a large bowl. Form into 5 patties.

Heat olive oil in a skillet over medium heat. Stir in the sliced onion; cook and stir until onion has softened and turned translucent, about 5 minutes. Reduce heat to medium-low, and continue cooking and stirring until onion is very tender and dark brown, 15 to 20 minutes more.

Cook patties in a medium skillet over medium heat, turning once, to an internal temperature of 180 degrees F (85 degrees C), about 6 minutes per side. Serve on buns topped with prepared mayo and caramelized onions.