From my point of view, Categories, Ratings, and Colour ratings should be all handled the same way.
Currently, when selecting a Rating and using "Remove from category", the image becomes invisible.

One might think that the image has been removed from the rating, actually the image hasn't been removed from the rating because ratings are not handled as categories. One can see this when selecting a different rating and then the former rating.
The current behaviour is confusing. Either the image should be really removed from the rating or it should not dissappear when using "Remove from category" on a rating or colour label.