Additionally, if you deselect the parent it actually selects all the children, even the ones that weren't selected.
1) Expand a category, then select another one, then select (with ctrl) the expanded category
2) Deselect a couple of the children, as if you wanted most but not all of them, hence selecting the parent (to select them all) then deselecting the ones you don't want
3) Deselect the parent, which causes ALL of the children to be selected. Regardless of how you feel selection of children should behave, this is definitely not right behavior in any definition

4) Go back to step 1 and select the expanded category THEN select another one (with ctrl). Now the children aren't selected, so you get entirely different results despite doing essentially the same thing (just in a different order).
I do see the failure to deselect the children as a problem, especially since if the parent isn't expanded you wouldn't even know it and would likely assume they're deselected. But I also agree that deselecting them all automatically when deselecting the parent could be bad, because you could spend a while selecting which ones you want then lose it all by accidentally clicking on the parent. Then again, you would just as easily lose all that work and more by accidentally clicking anywhere without holding ctrl, so there has to be some level of "if you're not careful you're going to screw yourself." This is why checkboxes are useful. With a checkbox next to each item, you would just check item one, then check item two (the non-expanded parent), then uncheck the children you don't want (which would make the checkmark next to the parent something else to indicate not everything under it is checked, like a grayed-out check or something), and unchecking the parent would uncheck all children.
See attached image for example: