The colored traveling salesmen problem is a node routing problem with multiple salesmen, where the cities are divided into $m$ exclusive city sets and one shared city set. The objective is to minimize the total traveling distance of $m$ Hamiltonian circuits (routes) under the following constraints: each exclusive city is to be visited by the corresponding salesman, while each shared city can be visited by any salesman. In this work, we present the first grouping memetic algorithm for solving this challenging problem. The algorithm includes three main components: (i) a greedy randomized heuristic for population initialization; (ii) a dedicated local search procedure for local optima exploration; (iii) a backbone-based crossover operator for solution recombination. We show computational results on three sets of 65 popular benchmark instances to demonstrate the competitiveness of our algorithm. We especially report improved upper bounds for 38 instances (for more than 58% cases).