Move assertion where it actually belongs, since it does not apply to the buffer cleanup case (which semantically jumps over it with a 'goto').