I called HMPO last week about moving my current application for renewal to fast track and was told that I do not need to bring my passport with me. I would need to write a letter stating that I want to withdraw my current application and as my passport was already received by HMPO, they would be able to see it on the system and proceed with fast Track. I never went to fast track appointment but this might work for you.