Prostate cancer blood tests can lead to men being over-tested, while those who need help may be missed, a large study has found.