The only remaining building of a once-bustling historic gold mining town has been damaged in a disappointing display of vandalism.