How to generate a call graph for projects with multiple files

Getting Help
1

For example,

https://github.com/bluesea147/6.824/tree/master/src/mapreduce

One way to run it is to:

export GOPATH=/home/XXX/6.824
go get GitHub - pkg/profile: Simple profiling for Go
go test -run Sequential

I tried :
export GOPATH=/home/cpchung/learn/6.824
go get GitHub - pkg/profile: Simple profiling for Go
go test -run Sequential -cpuprofile=./c.pprof

Then how I can use the c.pprof? I cannot go build this project since there is no main() there.

I can profile one single file and generate the call graph.
But for projects like this, how can I generate the call graph or even profile the project?

I think generating a call graph is a good way to learn unfamiliar code base.

2

Two things,

  1. You don’t need go get github.com/pkg/profile, just leave that out.

  2. What happens when you run this command?

    go test -run Sequential -cpuprofile=./c.pprof

Can you please provide the complete output you see, and what you expected to see.

3

Hi Dave,

I expect to see a non-empty c.pprof so that I can generate the call graph using “web” in pprof, but now it is empty.

My goal is to generate the call graph even when the computation is not intensive, profiling is the next goal

cpchung:mapreduce$ ls
common.go mapreduce.go master.go test_test.go
c.pprof mapreduce.test run.sh worker.go
cpchung:mapreduce$ sh -x run.sh

  • export GOPATH=/home/cpchung/learn/6.824
  • go get github.com/pkg/profile
  • go test -run Sequential -cpuprofile=./c.pprof
    PASS
    ok _/home/cpchung/profileGo/6.824/src/mapreduce 0.006s
    cpchung:mapreduce$ more c.pprof
4

Try

 go test -run Sequential -bench=. -cpuprofile=./c.pprof
5

Oh, actually, the reason your profile is empty is your program only ran for 6 milliseconds. Your program need to run for longer to generate a profile.

6

cpchung:6.824$ tree
.
├── Makefile
├── README.md
└── src
├── main
│ ├── diff.out
│ ├── kjv12.txt
│ ├── lockc.go
│ ├── lockd.go
│ ├── mapreduce
│ │ ├── pkg
│ │ │ └── linux_amd64
│ │ │ └── github.com
│ │ │ └── pkg
│ │ │ └── profile.a
│ │ └── src
│ │ └── github.com
│ │ └── pkg
│ │ └── profile
│ │ ├── AUTHORS
│ │ ├── example_test.go
│ │ ├── LICENSE
│ │ ├── profile.go
│ │ ├── profile_test.go
│ │ ├── README.md
│ │ ├── trace16.go
│ │ ├── trace.go
│ │ └── trace_test.go
│ ├── mr-testout.txt
│ ├── pbc.go
│ ├── pbd.go
│ ├── pkg
│ │ └── linux_amd64
│ │ └── github.com
│ │ └── pkg
│ │ └── profile.a
│ ├── run.sh
│ ├── src
│ │ └── github.com
│ │ └── pkg
│ │ └── profile
│ │ ├── AUTHORS
│ │ ├── example_test.go
│ │ ├── LICENSE
│ │ ├── profile.go
│ │ ├── profile_test.go
│ │ ├── README.md
│ │ ├── trace16.go
│ │ ├── trace.go
│ │ └── trace_test.go
│ ├── test-wc.sh
│ ├── viewd.go
│ └── wc.go
└── mapreduce
├── common.go
├── c.pprof
├── mapreduce.go
├── mapreduce.test
├── master.go
├── run.sh
├── test_test.go
└── worker.go

20 directories, 41 files

cpchung:main$ time sh -x run.sh

  • export GOPATH=/home/cpchung/profileGo/6.824
  • go run wc.go master kjv12.txt sequential
    Merge: read RunSingle
    Split kjv12.txt
    DoMap: read split mrtmp.kjv12.txt-0 966967
    DoMap[Done]: read split mrtmp.kjv12.txt-0 966967
    DoMap: read split mrtmp.kjv12.txt-1 966941
    DoMap[Done]: read split mrtmp.kjv12.txt-1 966941
    DoMap: read split mrtmp.kjv12.txt-2 966974
    DoMap[Done]: read split mrtmp.kjv12.txt-2 966974
    DoMap: read split mrtmp.kjv12.txt-3 966970
    DoMap[Done]: read split mrtmp.kjv12.txt-3 966970
    DoMap: read split mrtmp.kjv12.txt-4 966905
    DoMap[Done]: read split mrtmp.kjv12.txt-4 966905
    DoReduce: read mrtmp.kjv12.txt-0-0
    DoReduce: read mrtmp.kjv12.txt-1-0
    DoReduce: read mrtmp.kjv12.txt-2-0
    DoReduce: read mrtmp.kjv12.txt-3-0
    DoReduce: read mrtmp.kjv12.txt-4-0
    DoReduce: read mrtmp.kjv12.txt-0-1
    DoReduce: read mrtmp.kjv12.txt-1-1
    DoReduce: read mrtmp.kjv12.txt-2-1
    DoReduce: read mrtmp.kjv12.txt-3-1
    DoReduce: read mrtmp.kjv12.txt-4-1
    DoReduce: read mrtmp.kjv12.txt-0-2
    DoReduce: read mrtmp.kjv12.txt-1-2
    DoReduce: read mrtmp.kjv12.txt-2-2
    DoReduce: read mrtmp.kjv12.txt-3-2
    DoReduce: read mrtmp.kjv12.txt-4-2
    Merge phaseMerge: read mrtmp.kjv12.txt-res-0
    Merge: read mrtmp.kjv12.txt-res-1
    Merge: read mrtmp.kjv12.txt-res-2
  • rm mrtmp.kjv12.txt mrtmp.kjv12.txt-0 mrtmp.kjv12.txt-0-0 mrtmp.kjv12.txt-0-1 mrtmp.kjv12.txt-0-2 mrtmp.kjv12.txt-1 mrtmp.kjv12.txt-1-0 mrtmp.kjv12.txt-1-1 mrtmp.kjv12.txt-1-2 mrtmp.kjv12.txt-2 mrtmp.kjv12.txt-2-0 mrtmp.kjv12.txt-2-1 mrtmp.kjv12.txt-2-2 mrtmp.kjv12.txt-3 mrtmp.kjv12.txt-3-0 mrtmp.kjv12.txt-3-1 mrtmp.kjv12.txt-3-2 mrtmp.kjv12.txt-4 mrtmp.kjv12.txt-4-0 mrtmp.kjv12.txt-4-1 mrtmp.kjv12.txt-4-2 mrtmp.kjv12.txt-res-0 mrtmp.kjv12.txt-res-1 mrtmp.kjv12.txt-res-2

real 0m9.417s
user 0m9.340s
sys 0m2.200s

This is the way to run it.

So how can I use go test without modifying the folder structure or anything else?

If I need to modify sth, how can I modify least amount of things?

7

  1. Edit wc.go and add this line

    defer profile.Start().Stop()

    At the top of your main() function.

  2. Use go build, not go run, so you get a binary

    go build wc.go

  3. run the program (might not be called wc, it’ll depend on the name of the directory you were in at the time)

  4. Now you have your program and your profile in /tmp/someprofilefile you can run a profile

    go tool pprof your_program /tmp/someprofilefile

  5. As a bonus, please take the time to read through https://golang.org/doc/code.html. I know you want to organise your code in a specific way, but that is not what the Go tool expects and it will cause you continual problems and you will not be able to ask for help from others because they will not be able to help you until you restructure your code. I’m sorry, this is the cost of using the go for things like profiling, you have to play by its rules.

1 Like
8

Thank you Dave!

I saw the built binary! It is very close now. But the program takes arguments to run. So how to pass the arguments into go test -run?

cpchung:main$ sh -x ./run.sh

  • export GOPATH=/home/cpchung/profileGo/6.824
  • go get github.com/pkg/profile
  • go build wc.go
  • go test -run ./wc master kjv12.txt sequential -cpuprofile=./c.pprof
    can’t load package: package master: cannot find package “master” in any of:
    /usr/lib/go-1.6/src/master (from $GOROOT)
    /home/cpchung/profileGo/6.824/src/master (from $GOPATH)
    can’t load package: package kjv12.txt: cannot find package “kjv12.txt” in any of:
    /usr/lib/go-1.6/src/kjv12.txt (from $GOROOT)
    /home/cpchung/profileGo/6.824/src/kjv12.txt (from $GOPATH)
    can’t load package: package sequential: cannot find package “sequential” in any of:
    /usr/lib/go-1.6/src/sequential (from $GOROOT)
    /home/cpchung/profileGo/6.824/src/sequential (from $GOPATH)
9

This program is not a test, you should not use go test. Use

 ./wc master kjv12.txt sequential -cpuprofile=./c.pprof
10

For reasons unknown,./wc master kjv12.txt sequential -cpuprofile=./c.pprof cant be used as -cpuprofile=./c.pprof is considered as the 4th argument to ./wc

So to summarize your effort, it works!

cpchung:main$ sh -x ./run.sh

  • rm -rf /tmp/profile403416606 /tmp/profile419178204
  • export GOPATH=/home/cpchung/profileGo/6.824
  • go get github.com/pkg/profile
  • go build wc.go
  • ./wc master kjv12.txt sequential
    2016/12/08 22:06:15 profile: cpu profiling enabled, /tmp/profile194612212/cpu.pprof
    Merge: read RunSingle
    Split kjv12.txt
    DoMap: read split mrtmp.kjv12.txt-0 966967
    DoMap[Done]: read split mrtmp.kjv12.txt-0 966967
    DoMap: read split mrtmp.kjv12.txt-1 966941
    DoMap[Done]: read split mrtmp.kjv12.txt-1 966941
    DoMap: read split mrtmp.kjv12.txt-2 966974
    DoMap[Done]: read split mrtmp.kjv12.txt-2 966974
    DoMap: read split mrtmp.kjv12.txt-3 966970
    DoMap[Done]: read split mrtmp.kjv12.txt-3 966970
    DoMap: read split mrtmp.kjv12.txt-4 966905
    DoMap[Done]: read split mrtmp.kjv12.txt-4 966905
    DoReduce: read mrtmp.kjv12.txt-0-0
    DoReduce: read mrtmp.kjv12.txt-1-0
    DoReduce: read mrtmp.kjv12.txt-2-0
    DoReduce: read mrtmp.kjv12.txt-3-0
    DoReduce: read mrtmp.kjv12.txt-4-0
    DoReduce: read mrtmp.kjv12.txt-0-1
    DoReduce: read mrtmp.kjv12.txt-1-1
    DoReduce: read mrtmp.kjv12.txt-2-1
    DoReduce: read mrtmp.kjv12.txt-3-1
    DoReduce: read mrtmp.kjv12.txt-4-1
    DoReduce: read mrtmp.kjv12.txt-0-2
    DoReduce: read mrtmp.kjv12.txt-1-2
    DoReduce: read mrtmp.kjv12.txt-2-2
    DoReduce: read mrtmp.kjv12.txt-3-2
    DoReduce: read mrtmp.kjv12.txt-4-2
    Merge phaseMerge: read mrtmp.kjv12.txt-res-0
    Merge: read mrtmp.kjv12.txt-res-1
    Merge: read mrtmp.kjv12.txt-res-2
    2016/12/08 22:06:22 profile: cpu profiling disabled, /tmp/profile194612212/cpu.pprof
  • go tool pprof wc /tmp/profile194612212/cpu.pprof
    Entering interactive mode (type “help” for commands)
    (pprof)

Thank you so much!!

11

-cpuprofile is only used when running go test, your program is not a test, it cannot be used as a test.

If you want to profile a program, you can use something like github.com/pkg/profile to generate the profile.

I recommend watching this presentation http://talks.godoc.org/github.com/davecheney/presentations/seven.slide

12

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.