Issue while creating a package inside /src, that needs to be accessed from main package

Manoj_Jayaram · June 6, 2017, 2:01pm

My directory structure is as below.

> ├── pkg
> └── src
 >     └── util

All my source code is present in /src folder, here

> main.go --(Invokes)->cmain.c---(invokes)-->main.go

My main.go file is shown below

> package main
> import (
>     /*
>        #include "cmain.h"
>     */
>     "C"
>     "os"
>     "util"`
>     "unsafe"
 > )

> func main() {
>     argc := C.int(len(os.Args))
>     argv := make([]*C.char, argc)
>     for i, arg := range os.Args {
>         argv[i] = C.CString(arg)
>     }
>     C.cmain(argc, (**C.char)(unsafe.Pointer(&argv[0])))
> }

> func sample_f(filename *C.char) int {
>     cs := C.GoString(filename)
>     util.Validation(cs)
>     util.XML_Read(cs)
>     return 0
> }

My cmain.c file is as shown below

> #include <stdio.h>
> #include "_cgo_export.h"
> void cmain(int argc, char** argv) {
   >   if( argc == 2 ){
>       sample_f(argv[1]);
>   } 
> }

NOTE: I have all the required .h for .c files
The sample_f() function is present in sample.go, for which i am trying to create a separate package called “util” in /src,
in sample.go (path shown below), i have included package util at the top of file.
├── cmain.c
├── cmain.h
├── main.go
├── makefile
└── util
└── sample.go

When ever i do a make i always get below error.

> env: tool: No such file or directory
> make: *** [_cgo_export.c] Error 127

My makefile is shown below,

> GOROOT=/usr/local/go
> GO=/usr/local/go/bin/go
> GOPATH=$(shell pwd)
> PKG_DIR=`pwd`/../pkg/linux_amd64
> GO_TOOL_FLAGS=-v -D ${PKG_DIR}
> CC=gcc


> CGO_SRCS= main.go util/sample.go

> GENFILES_GO=_cgo_gotypes.go \
> 	    $(CGO_SRCS:.go=.cgo1.go) 
> GENFILES_C=_cgo_export.c \
> 	   $(CGO_SRCS:.go=.cgo2.c) 
> EXPORT_H=_cgo_export.h
> GENFILES= $(GENFILES_GO) \
> 	  $(GENFILES_C) \
> 	  $(EXPORT_H) \
> 	  _cgo_main.c \
> 	  _cgo_flags

> all: _cgo_.o libSample.a SAMPLE

> .c.o:
> 	${CC} -c -o $@ -fPIC $<

> clean:
> 	rm -f $(GENFILES) *.o libSample.a ../bin/SAMPLE _cgo_import.go

> $(GENFILES): $(CGO_SRCS) cmain.h
> 	env CGO_LDFLAGS=${GO} tool cgo -objdir .  \
> 		-importpath $(CGO_SRCS)
> 	rm _cgo_.o

> cmain.o: cmain.c cmain.h

> _cgo_.o: $(GENFILES_C:.c=.o) $(EXPORT_H) _cgo_main.o cmain.o
> 	${CC} -o _cgo_.o $(GENFILES_C:.c=.o) _cgo_main.o cmain.o

> _all.o: $(GENFILES_C:.c=.o) $(EXPORT_H) cmain.o
> 	${CC} -fPIC -nostdlib -o _all.o -Wl,-r $(GENFILES_C:.c=.o) cmain.o

> _cgo_import.go: _cgo_.o
> 	${GO} tool cgo -dynimport _cgo_.o -dynout _cgo_import.go \
> 		-dynpackage main util

> libSample.a: $(GENFILES_GO) _cgo_import.go main.go _all.o
> 	${GO} tool compile ${GO_TOOL_FLAGS} -pack -o libSample.a $(GENFILES_GO) _cgo_import.go
> 	${GO} tool pack r libSample.a _all.o 

> SAMPLE: libSample.a
> 	/usr/local/go/pkg/tool/linux_amd64/link -L ${PKG_DIR} -buildmode exe -o ../bin/SAMPLE libSample.a

Can anyone give some pointers as to how to resolve this issue.

nathankerr · June 6, 2017, 2:26pm

Why aren’t you using go build or go install?

Manoj_Jayaram · June 6, 2017, 2:48pm

Since my requirement demands this and my flow is like this, main.go --(Invokes)->cmain.c—(invokes)–>main.go—(Invokes)–>sample.go , which is in util directory.

nathankerr · June 6, 2017, 3:01pm

That invocation pattern works with go build.

With the following files in the same directory (I happened to call mine from-main, run:

go build

This produces an executable from-main. Running

./from-main hello

produces:

main.go:27: hello

I think the only part of your situation that you have explained is that your util package was not used. I left the relevant lines in the code so you can easily try them out and see if you get a similar error.

Other than this invocation flow, what are your build requirements?

`main.go`

package main

/*
   #include "cmain.h"
*/
import "C"

import (
	"log"
	"os"
	"unsafe"
)

func main() {
	log.SetFlags(log.Lshortfile)

	argc := C.int(len(os.Args))
	argv := make([]*C.char, argc)
	for i, arg := range os.Args {
		argv[i] = C.CString(arg)
	}
	C.cmain(argc, (**C.char)(unsafe.Pointer(&argv[0])))
}

//export sample_f
func sample_f(filename *C.char) int {
	cs := C.GoString(filename)
	log.Println(cs)
	// util.Validation(cs)
	// util.XML_Read(cs)
	return 0
}

`cmain.h`

void cmain(int argc, char** argv);

`cmain.c`

#include <stdio.h>
#include "_cgo_export.h"

void cmain(int argc, char** argv) {
	if( argc == 2 ){
		sample_f(argv[1]);
	} 
}

nathankerr · June 6, 2017, 3:55pm

As for the error from the Makefile:

env: tool: No such file or directory
make: *** [_cgo_export.c] Error 127

This comes from this rule:

$(GENFILES): $(CGO_SRCS) cmain.h
	env CGO_LDFLAGS=${GO} tool cgo -objdir .  \ 
                    -importpath $(CGO_SRCS)
	rm _cgo_.o

env takes a list of NAME=VALUE pairs, then a command and its arguments. The command you have here is tool. env cannot find an executable in the PATH called tool; thus the error.

After “fixing” that error, the next error comes from not using go tool cgo properly.

Having to figure out how to use all the tools that underly go build is why I am pushing you to use go build if it is possible.

Manoj_Jayaram · June 7, 2017, 6:24am

Previously all my .go and .c files were residing in the same directory (/src) and it was building successfully, with the makefile. All of those .go files were included in the main package. The main.go file calls the cmain.c which in turn calls the sample_f() which is in the sample.go file. Now, as per my requirement, i need to move the sample.go file containing sample_f() which in turn calls Validation() and XML_Read functions, to util directory, forming util package. The issue that i am facing now is i am unable to link the directory structure with the main.go which is in src.
The previous structure was:
src
├── cmain.c
├── cmain.h
├── main.go
├── makefile
└── sample.go (XML_Validation and XML_Read)

Now my sample.go and main.go are in package main and this structure worked perfectly fine using the makefile.
But, the current requirements needs to create a package util and cmain.c will call the function in util package. So the directory structure will be:

src
├── cmain.c
├── cmain.h
├── main.go
├── makefile
└── util
│      └── sample.go 
├── github.com
       ├── lestrrat
              ├── go-libxml2

My overall Structure is as below

XML
├── src
       ├── cmain.c
       ├── cmain.h
├── main.go
├── pkg
        ├── linux_amd64
                ├── github.com
                ├── golang.org
                ├── gopkg.in
├── cmain.h
├── makefile

For this, I am unable to build. Please provide us some pointers to tackle this issue.

cmain.c

#include <stdio.h>
#include "_cgo_export.h"
void cmain(int argc, char** argv) {
  if( argc == 2 ){
    util.Sample_f(argv[1]);
  } 
  return;
}

cmain.h
void cmain(int argc, char** argv);

main.go

package main
import (
    /*
       #include "cmain.h"
    */
    "C"
    "os"
    "unsafe"
    "util"
)

func main() {
    argc := C.int(len(os.Args))
    argv := make([]*C.char, argc)
    for i, arg := range os.Args {
        argv[i] = C.CString(arg)
    }
    C.cmain(argc, (**C.char)(unsafe.Pointer(&argv[0])))
 }

makefile

GOROOT=/usr/local/go
GO=/usr/local/go/bin/go
GOPATH=$(shell pwd)
PKG_DIR=`pwd`/../pkg/linux_amd64
GO_TOOL_FLAGS=-v -D ${PKG_DIR}
GO_UTIL=$(GOPATH)/util
CC=gcc


CGO_SRCS= main.go util/sample.go

GENFILES_GO=_cgo_gotypes.go \
	    $(CGO_SRCS:.go=.cgo1.go) 
GENFILES_C=_cgo_export.c \
	   $(CGO_SRCS:.go=.cgo2.c) 
EXPORT_H=_cgo_export.h
GENFILES= $(GENFILES_GO) \
	  $(GENFILES_C) \
	  $(EXPORT_H) \
	  _cgo_main.c \
	  _cgo_flags

all: _cgo_.o libsample.a SAMPLE

.c.o:
	${CC} -c -o $@ -fPIC $<

clean:
	rm -f $(GENFILES) *.o libsample.a ../bin/SAMPLE _cgo_import.go

$(GENFILES): $(CGO_SRCS) cmain.h
	env CGO_LDFLAGS=${GO} tool cgo -objdir .  \
		-importpath $(CGO_SRCS)
	rm _cgo_.o

cmain.o: cmain.c cmain.h

_cgo_.o: $(GENFILES_C:.c=.o) $(EXPORT_H) _cgo_main.o cmain.o
	${CC} -o _cgo_.o $(GENFILES_C:.c=.o) _cgo_main.o cmain.o

_all.o: $(GENFILES_C:.c=.o) $(EXPORT_H) cmain.o
	${CC} -fPIC -nostdlib -o _all.o -Wl,-r $(GENFILES_C:.c=.o) cmain.o

_cgo_import.go: _cgo_.o
	${GO} tool cgo -dynimport _cgo_.o -dynout _cgo_import.go \
		-dynpackage main util

libsample.a: $(GENFILES_GO) _cgo_import.go main.go _all.o
	${GO} tool compile ${GO_TOOL_FLAGS} -pack -o libsample.a $(GENFILES_GO) _cgo_import.go structure.go
	${GO} tool pack r libsample.a _all.o 

SAMPLE: libsample.a
	/usr/local/go/pkg/tool/linux_amd64/link -L ${PKG_DIR} -buildmode exe -o ../bin/SAMPLE libsample.a

In /src/util/

sample.go

package util
import (
    "C"
    "encoding/xml"
    "fmt"
    "./github.com/golang.org/x/text/encoding/japanese"
    "./github.com/golang.org/x/text/transform"
    "github.com/lestrrat/go-libxml2"
    "github.com/lestrrat/go-libxml2/xsd"
    "io"
    "io/ioutil"
    "os"
)
func doSample(filename *C.char) int {

    // Read XML File. Note! This file is SJIS encoding.
    cs := C.GoString(filename)
    XML_Validation(cs)
    XML_Read(cs)
    return 0
}

structure.go, this files contains structure that is uses in XML_Validation() and XML_Read().

Our final goal is to be able to call the sample_f function from cmain.c. It is not mandatory to use makefile, use of go build or go install doing it is also fine.
Can you please help us in resolving this?

nathankerr · June 7, 2017, 7:05am

Is the top-level directory (XML) intended to be GOPATH?

Manoj_Jayaram · June 7, 2017, 7:10am

Yes , the top-level directory is GOPATH
have set GOPATH= /home/xxx/XML
In XML folder i have the following folder bin conf inc pkg src , inside src the structure is as shown in the above query.

nathankerr · June 7, 2017, 7:13am

Do you need to keep third-party code (e.g., github.com/lestrrat/go-libxml2) separate from your code? Or can it all be mixed in XML/src?

EDIT: I am working on the structure I would use.

Manoj_Jayaram · June 7, 2017, 7:24am

Yes, i would like to keep third-party code (github.com/lestrrat/go-libxml2) separate.

nathankerr · June 7, 2017, 8:17am

I think I covered everything. Tell me if I missed anything.

This sample demonstrates:

calling c from go
calling go from c
util package
third-party packages
compilation using the go toolchain, called from make (to make what is happening explicit)

To separate your code and third-party code, I used a multi-element GOPATH:

GOPATH=/home/xxx/XML/third-party:/home/xxx/XML

By placing the third-party entry first, go get will put the packages it gets into third-party instead of XML/src.

First, the structure as reported by tree:

XML
├── Makefile
├── src
│   ├── cmain
│   │   ├── cmain.c
│   │   ├── cmain.go
│   │   └── cmain.h
│   ├── sample
│   │   └── main.go
│   └── util
│       └── util.go
└── third-party
    └── src
        └── github.com
            ├── cheggaaa
            │   └── pb
            │       ├── LICENSE
            │       ├── README.md
            │       ├── example_copy_test.go
            │       ├── example_multiple_test.go
            │       ├── example_test.go
            │       ├── format.go
            │       ├── format_test.go
            │       ├── pb.go
            │       ├── pb_appengine.go
            │       ├── pb_nix.go
            │       ├── pb_solaris.go
            │       ├── pb_test.go
            │       ├── pb_win.go
            │       ├── pb_x.go
            │       ├── pool.go
            │       ├── pool_win.go
            │       ├── pool_x.go
            │       ├── reader.go
            │       ├── runecount.go
            │       ├── runecount_test.go
            │       ├── termios_bsd.go
            │       └── termios_nix.go
            └── mattn
                └── go-runewidth
                    ├── LICENSE
                    ├── README.mkd
                    ├── runewidth.go
                    ├── runewidth_js.go
                    ├── runewidth_posix.go
                    ├── runewidth_test.go
                    └── runewidth_windows.go

11 directories, 35 files

I used github.com/cheggaaa/pb to demonstrate third party support because I like it and it was easy to write a visible demonstration that it worked.

After setting your GOPATH as described, run:

go get github.com/cheggaaa/pb

This creates the files and directories under third-party.

I put all the cgo code in package cmain. I did this to improve compile times and error messages when working in the Go code.

Makefile

PWD=$(shell pwd)
GOPATH=$(PWD)/third-party:$(PWD)

all: bin/sample

run: bin/sample
	bin/sample this_is_a_filename

.PHONY: bin/sample # dependencies are handled by Go, not by make
bin/sample:
	go install -v sample

clean:
	rm -rf bin pkg third-party/bin third-party/pkg

src/cmain/cmain.c

#include <stdio.h>
#include "_cgo_export.h"

void cmain(int argc, char** argv) {
	if( argc == 2 ){
		int i;
		i = util_sample_f(argv[1]);
		printf("from c: %d\n", i);
	} 
	return;
}

src/cmain/cmain.go

// Package cmain interfaces between C and Go.
// The Go code in other packages should not need to import "C"
//
// write wrappers for every interaction between C and Go
package cmain

// #include "cmain.h"
import "C"
import (
	"unsafe"

	"util"
)

func Cmain(args []string) {
	argc := C.int(len(args))
	argv := make([]*C.char, argc)
	for i, arg := range args {
		argv[i] = C.CString(arg)
	}
	C.cmain(argc, (**C.char)(unsafe.Pointer(&argv[0])))
}

//export util_sample_f
func util_sample_f(filename *C.char) C.int {
	return C.int(util.DoSample(C.GoString(filename)))
}

src/cmain/cmain.h

void cmain(int argc, char** argv);

src/sample/main.go

// Package main will be installed as sample
// (because it is in a directory called sample)
//
// It is the entry point to the rest of the program
package main

import (
	"cmain"
	"fmt"
	"os"
)

func main() {
	fmt.Println("running with os.Args:", os.Args)
	cmain.Cmain(os.Args)
}

src/util/util.go

package util

import (
	"fmt"
	"time"

	"github.com/cheggaaa/pb"
)

func DoSample(filename string) int {
	fmt.Println("from go:", filename)

	n := 10

	bar := pb.StartNew(n)
	defer bar.Finish()

	for i := 0; i < n; i++ {
		time.Sleep(500 * time.Millisecond)
		bar.Increment()
	}

	return 1
}

Building and running with make run produces:

go install -v sample
github.com/mattn/go-runewidth
github.com/cheggaaa/pb
util
cmain
sample
bin/sample this_is_a_filename
running with os.Args: [bin/sample this_is_a_filename]
from go: this_is_a_filename
 10 / 10 [==============================================================] 100.00% 5s
from c: 1

The only thing tricky I did was the multi-value GOPATH. The building itself, while launched from make, is a straight-forward invocation of go install (the -v tells it to list the packages it installs, you will notice the packages listed correspond to those that have changed or depend on those that changed).

system · September 5, 2017, 8:17am

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.