The idea of making a change and rebuilding is about proving that the build process does not require a network connection and that you have properly resolved the depencenies on that machine.
If you’ve built once using the standard go module functionality then you will be able to rebuild as long as you don’t pull in more dependencies. Naturally you can move the cache around.