Wittgenstein said that language isn’t actually semantic in its natural state. Instead, language serves as a sort of game to accomplish social ends.

“I could care less” serves the social end of telling everyone you have no critical thinking skills and shouldn’t be taken seriously.