Whisk mayonnaise, lime zest, lime juice, rice vinegar, garlic, sweet chili sauce, and sugar in a large bowl, stirring to dissolve sugar. Mix cilantro and red onion into dressing. Stir cabbage into dressing mixture, about 1 cup at a time, until all cabbage is coated.