How to collect information in each package's source code directory when building a project

[Lslightly (Lqw)]

Purpose

I’m trying to modify the golang compiler(release 1.17.7) to collect information located in source code directory of packages which are imported by current project, and gather the information under the root directory(i.e. the main package directory) of the module.

What I do

What I do is that I add a flag -s to allow the go compiler(gc) to dump information in each package source directory. I achieve it.

And then I try to collect these information. In the runBuild function which is the entry of the gc dispatching compilation tasks, I try to add the following code to collect information dumped in each package’s source code directory to the main package’s directory after b.Do(ctx, a) Line448 and b.Do(ctx, a) Line459 which is the real function to do the compilation tasks. I think all imported packages will be compiled after b.Do(ctx, a) finished. But the collect failed.

		b.Do(ctx, a)
		var main_path string
		for _, p := range pkgs {
			if p.Name == "main" {
				main_path = p.Dir  // get the main package's path
			}
		}
        // collect information under each pkg.Dir and move files from pkg.Dir to 
		collect(pkgs, main_path)

What happened?

Take the following project as an example

pkg_export # the main package
|---main.go
|---mypkg # the mypkg package---imported by main.go
      |---utils.go

In my experiment, using command go build -a -gcflags=all=-s ., each package can properly dump information in each package’s source code directory, but the collect function can’t collect information from each package’s source directory to the main_path. In the example project, it appears like:

pkg_export # the main package
|---main.go
|---info1.txt
|---mypkg # the mypkg package---imported by main.go
      |---utils.go
      |---info2.txt

the info2.txt under mypkg directory is not merged into the info1.txt under pkg_export directory after collect(pkgs, main_path) function call.

What expected

the info2.txt under mypkg directory is not merged into the info1.txt under pkg_export directory after collect(pkgs, main_path) function call.

My questions

My questions are:

1. Is my code inserted in the right place? My purpose is to collect information when every dependency of the module is compiled and information is generated. Does all dependency compilation finish after b.Do(ctx, a) Line 547 and b.Do(ctx, a) Line 536?
2. Do I use the correct field p.Dir of the load.Package struct?
3. Is the compilation process parallel? If so, does the parallelism cause the failure of merging information?

Example source code

An example project is in Lslightly/pkg_export (github.com)

The rebuild of gc is to use ./clean.bash and then ./make.bash under src directory of golang’s source code and the new go compiler will be generated.

I really appreciate any helps and ideas. Thanks!

[skillian (Sean Killian)]

Hi, @Lslightly, is there a reason you’re modifying the compiler to do this instead of using a package like golang.org/x/tools/go/packages?

[Lslightly (Lqw)]

I’m not sure whether golang.org/x/tools/go/packages can get accurate import graph, so I don’t try it. Besides, In the Package struct in golang.org/x/tools/go/packages, there is no std related field, as I also want to filter out standard packages.

Anyway, thanks for your advice. I will try this package next time when I need the import information as this package is lightweight.

[Lslightly (Lqw)]

I have figured out why I didn’t merge info2.txt under mypkg directory into the info1.txt under pkg_export directory. The reason is that when compiling using go build -a -gcflags=all=-s . under pkg_export directory, the pkgs slice in the following code only contains one package(i.e. main package)

		b.Do(ctx, a)
		var main_path string
		for _, p := range pkgs {
			if p.Name == "main" {
				main_path = p.Dir  // get the main package's path
			}
		}
        // collect information under each pkg.Dir and move files from pkg.Dir to 
		collect(pkgs, main_path)

The Action a in b.Do(ctx, a) is a LinkAction generated by AutoAction due to main package. Action a contains a CompileAction and the CompileAction will append all
actions to compile the imported packages as its dependency. The code is at go/action.go#L459
as following.

for _, p1 := range p.Internal.Imports {
	a.Deps = append(a.Deps, b.CompileAction(depMode, depMode, p1))
}

So what I need to do is to use p.Internal.Imports to collect imported packages, use pkg.Dir to get source code directories and finally merge information under pkg.Dir to main_path, like the following code.

func collectImportPackages(a *Action, pkg_set map[*load.Package]bool) {
	for _, dep := range a.Deps {
		if pkg_set[dep.Package] || dep.Package.Standard {
			continue
		} else {
			pkg_set[dep.Package] = true
		}
		fmt.Println(dep.Package.Name)
		collectImportPackages(dep, pkg_set)
	}
}

all imported packages excluding standard packages will be collected as keys of the key_set map

[Mateo_Hudson]

Analyse the content of each source code file to extract the desired information. The exact information you want to collect will depend on your project’s requirements. Some common examples include function or method names, class definations, variable declaration, imports and comments.