Not so fast mac45, when I said the Castle was in Leopold St I was correct, but I was mixing it up with the Crown. I should know better because I have done the same thing before.
Although the Castle was in Leopold St it was on the corner of Upper Highgate St, and if your Nan lived at number 5 Upper Highgate St near the other corner then in truth she did live across the road from the Castle.
Here's a photo looking from Moseley Rd to Upper Highgate St and you can just see the Castle on the corner.