To be fair Phil, those of us that lived in the suburbs needed to spend a good deal of time in, or travelling through, the city centre. That said, I don't recall there being particular 'trouble' in the city centre in my days (60s/70s) (the pub bombings apart). Sure it used to get a bit lively at night but that is town and city life for you.
I certainly didn't move away because of any dislike of Brum, but for career progression.