I remember when I did a restore of my iPod without restoring from a backup the first few times I backed up they were like 6mb. Now they're 160 mb. I deleted some apps thinking that their data were taking up some of the space but the size went down only a few tenths of a mb. I deleted my Mail account thinking the email messages were taking up space but it didn't change the size at all
I read the title as "what determines the size of...". I was like, "oh, that seems like an... interesting discussion, yet I am curious", and then it was just this, and I was slightly disappointed.