There's many buildings that have gone up & down in my lifetime (nearly 52), you mentioned the Post & Mail building, then there's the Central Library, many office blocks & tower blocks, and the previous incarnation of the Bull Ring was only a few years before I was born.
I can only speculate, but maybe the types of construction used haven't quite stood up to the test of time, that's certainly been the case with some concrete buildings.
I can well imagine that some offices built in the '60's & '70's don't meet with current expectations & standards - the one I work in now is certainly less dark & dingy than the one I worked in when I first started work.
Building standards surrounding insulation & energy efficiency have probably paid a part too. And some buildings were just an eyesore!