First of all, I’d like to thank everyone in this welcoming community.
I want to index nearly every package currently available and display them on a single search page(similar to what pkgsite does.).
In other words, I am making an alternative API search engine for Go called Goggle, so I need to index every package regularly.
a module mirror which implements the module proxy protocol. For users downloading large numbers of modules (e.g. for bulk static analysis), the mirror supports a separate endpoint
/cached-onlythat instructs the proxy to return only cached content.
I know I can download packages without cloning every git history via proxy.golang.org, however, I’m unsure how to obtain a full list of packages to index.
index.golang.org works like ‘timeline’, making it challenging to acquire only the latest versions. I have to search backward to do this, which is not straightforward due to the presence of only a ‘since’ parameter, rather than a ‘before’.
So, what approach did you use to quickly get a list of only the latest packages? Is there an automated tool or library that has already implemented this? Or are there other online proxies that allow retrieval of only the latest versions without having to crawl index.golang.org each time?
Any suggestions are welcome and any help would be greatly appreciated!