If we do RUN go mod download then later when we try to build the image again, Image build will be faster if there are no changes to go.mod/sum, because docker will cache the result of RUN go mod download and use it next time.
Here you are building your go application, dependencies will be fetched every time you build the image, which will result in slower image build.