Colored was widely used as late as the 70's in the USA. So, that would be the 20th century.
But, OP, you would be right and saying most people, at least most American blacks, would find that very offensive.
For one, Jim Crow laws had Colored and White distinctions from drinking fountains to restrooms, from schools to neighborhoods. So, people that would find "colored" offensive would equate it to a time of second class citizenry.
Secondly, to ME, colored implies "other". That white is the norm and colored means outside the box of normal. One could also take it to mean that black, brown, yellow and red people have no identity. They're simply filled in using a crayon.
But I think the first reason for offense is usually why it's offensive. That's how the term African-American came about. White people decided what to call blacks since they were brought over here and African-American was the first time blacks, themselves, decided what they were called. The terms ******, nigra, negro, colored and black were all coined by the dominant culture, read enslavers. I assume terms like Afro-Cuban, etc. were coined by those in that population.