We already have the refspecs parsed the git_remote, and loading them
lazily means we'll be parsing them again, and possibly parse a different
set from the ones we used to have loaded.
Make the loading eager, and pass the RemoteSafeHandle to more places, as
we need to use it in order to provide a single snapshot of the data.
