setting CGO_ENABLED=1 will result in a binary compiled, but then fails due to libs missing when Lambda executes the binary
G
Another alternative would be to refactor your code to utilize a Go native Kafka package like kafka package - github.com/segmentio/kafka-go - Go Packages. Not necessarily convenient, but would avoid the CGO.
Hi there
I’ve tried, but ran into problems, the version of kafka that the native version supports <> the confluent Kafka / CCloud…
for me the problem is yes, confluent… but surely there is a working how to do this, that works, does not matter which external library is used, at the moment I’m getting allot of try this. but none of them end working.
not being difficult… just need to get one working, which I can then re-use.
G
would really appreciate it if someone that has experience with this would be willing to jump onto zoom or teams.
G
Just curious, have you checked the size of the executable when compiling with and without the link mode? The one compiled with linkmode should be significantly larger, if not there is another issue.
so with CGO=1 and =0 the binary is the same size.
below is the make file,
BINARY_NAME=main
export GOOS=linux
export GOARCH=amd64
export CGO_ENABLED=1
export AWS_REGION=af-south-1
.DEFAULT_GOAL := deploy
deploy:
go build -o ${BINARY_NAME} .
zip -r function.zip main
aws lambda update-function-code --function-name "S3JSONDecomposer-Golang" --zip-file fileb://function.zip --region=${AWS_REGION} | jq .
run:
go run ${BINARY_NAME}.go
re this… I tried… using the below confluent kafka producer configuration. if you can maybe try specifying the same params using straight kafka class, as you said, maybe that will work.
As much as I need the entire deploy into Lambda resolved, my priority is to get this working, so if we can get around it for now by not using confluent stack that i’m ok with that.
cm := kafka.ConfigMap{
"bootstrap.servers": os.Getenv("kafka_bootstrap_servers"),
"broker.version.fallback": "0.10.0.0",
"api.version.fallback.ms": 0,
"client.id": os.Getenv("George"),
"sasl.mechanisms": os.Getenv("kafka_sasl_mechanisms"), // PLAIN
"security.protocol": os.Getenv("kafka_security_protocol"), // SASL_SSL
"sasl.username": os.Getenv("kafka_sasl_username"),
"sasl.password": os.Getenv("kafka_sasl_password"),
}
// Create Kafka producer
producer, err := kafka.NewProducer(&cm)
if err != nil {
logger.Error("Failed to create Kafka producer:", err)
return err
}
So I spent some time building an environment to test your specific use case. Earlier tests on OSX created statically linked executables so the issue didn’t show.
I built an environment running Debian 11 with 2 nodes. The first running Kafka and the second to build the code. The test code I used is very rudimentary but accomplishes the goal. When dynamically linked the executable was 9677848 and when statically linked it was 10613232. As expected it was larger.
Both versions of the code ran with sucess. The statically linked code worked on a system without the clibs.
The source I used is:
package main
import (
“fmt”
“github.com/confluentinc/confluent-kafka-go/kafka”
)
func main() {
// producer config
config := &kafka.ConfigMap{
“bootstrap.servers”: “10.1.10.56:29092”, // kafka broker addr
}
topic := “coordinates” // target topic name
// Create Kafka producer
producer, err := kafka.NewProducer(config)
if err != nil {
panic(err)
}
// write 10 messages to the “coordinates” topic
for i := 0; i < 10; i++ {
value := fmt.Sprintf(“message-%d”, i)
err := producer.Produce(&kafka.Message{
TopicPartition: kafka.TopicPartition{Topic: &topic, Partition: kafka.PartitionAny},
Value: []byte(value),
}, nil)
if err != nil {
fmt.Printf(“Failed to produce message %d: %v\n”, i, err)
} else {
fmt.Printf(“Produced message %d: %s\n”, i, value)
}
}
// Close Kafka producer
producer.Flush(15 * 1000)
producer.Close()
}
Not sure if all this will help on your target system but it passes the basic tests of being statically linked and executable.
The following was used to build the code: go build -ldflags=“-extldflags=-static”
hi there
awesome, thanks for helping, mind sharing your build scripts.
G
The afore mentioned ‘go build -ldflags=“-extldflags=-static”’ was all that I used to build no additional scripting.
If you are truly trying to cross-compile and cross architecture at the same time there are a set of problem cross-compiling when C-libs are required to build for a target. To avoid this build on a like platform. Else look into xgo command - github.com/kevinho/xgo - Go Packages. This attempts to provide such functionality and the are various permutations of the project. It seems it might be easier just build on the target platform unless you plan on doing this often.
Regards
hi hi
What i’m doing re the build, i’m building the binary on a AWS Linux Instance, to try and remove “murphy” from the game.
will give this a try when i have a second, will feed back,
thanks.
G
ok… so did some testing…
the below is now 
export CGO_ENABLED=0
make deploy
**Makefile**
BINARY_NAME=main
export GOOS=linux
export GOARCH=amd64
export CGO_ENABLED=1
export AWS_REGION=af-south-1
.DEFAULT_GOAL := deploy
deploy:
go build -ldflags=“-extldflags=-static -o main .
# jsondecomposer
./main.go:57:17: undefined: kafka.Producer
./main.go:169:14: undefined: kafka.ConfigMap
./main.go:181:25: undefined: kafka.NewProducer
./main.go:271:23: undefined: kafka.Message
./main.go:272:28: undefined: kafka.TopicPartition
./main.go:272:86: undefined: kafka.PartitionAny
make: *** [deploy] Error 1
the below is now export CGO_ENABLED=1
make deploy
go build -ldflags=“-extldflags=-static -o main .
# runtime/cgo
linux_syscall.c:67:13: error: call to undeclared function 'setresgid'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
linux_syscall.c:67:13: note: did you mean 'setregid'?
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/unistd.h:593:6: note: 'setregid' declared here
linux_syscall.c:73:13: error: call to undeclared function 'setresuid'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
linux_syscall.c:73:13: note: did you mean 'setreuid'?
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/unistd.h:595:6: note: 'setreuid' declared here
make: *** [deploy] Error 1
This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.