This is for the gentlemen:
If you put your wallet in your back pocket and then sit on it, that side of your pelvis will be elevated, and your pelvis and spine then has to compensate for this. If you do this often enough and for long enough, your spine will eventually adopt this as the most efficient position, which then in turn makes your spine more vulnerable  to injury when you are not currently sitting on your wallet. I have been reliably informed by gentlemen that it is perfectly possible for a wallet to  travel in a front pocket.

This is for the ladies:
Carrying handbags is a similar story. If you carry a handbag on the shoulder, that shoulder hitches up and causes strain to the neck (where the hitching-up muscles are). If you put the strap over the head (eg. carrying the bag on your right hip and the strap goes around your neck on the left), this will put pressure on the spine at the base of the neck and put it out (personal experience!!).

Ideally, bags should be carried with the least amount of leverage on the body, ie. closest to the centre of gravity, which would meaning carrying your bag in front of your belly (with both hands). Failing that, go for a backpack type handbag, which has the advantage of leaving your hands free.
Also: Handbags end up full regardless of size! SO: Go for a little one!!! The less weight you lug around, the less strain (as little as 5 kg makes a difference to wear and tear in your joints).