How to compile Go apps with no source code and only .a files
Last weekend, a Gopher shared a problem with me regarding the compilation of Go programs. His request, which was not complicated at all, was that the API package provided by the partner company included only the golang archive (.a file built with go build -buildmode=archive), and no source code for the Go package. How can I link this .a to the final executable program built by the project?
For C, C++, Java programmers, only provide static link library or dynamic link library (including header files), jar file without providing the source code of the API is very common. But for Go, only provide Go package archive (.a) file, but not provide Go package source code is extremely uncommon. The reason for this is simply that go build or go run is not supported!
Is there really no way to build a Go application without source code, only based on .a files? Not exactly. There are indeed some hacky methods that can achieve this, and this article will discuss these methods from a technical perspective. However, it is not recommended to use them!
1. go build doesn’t support “no source, only .a”
Let’s first review how go build behaves with “no source, only .a”. To do this, we’ll set up a lab environment with the following directory and file layout
// api package with no external dependencies: foo
$tree goarchive-nodeps
goarchive-nodeps
├── Makefile
├── foo.a
├── foo.go
└── go.mod
$tree library
library
└── github.com
└── bigwhite
└── foo.a
// An app project that depends on the foo package
$tree app-link-foo
app-link-foo
├── Makefile
├── go.mod
└── main.go
Here we have built the app-link-foo dependency foo.a (via go build -buildmode=arhive) and placed it in the corresponding directory under library.
Note: The composition of
foo.acan be viewed by using thear -x foo.acommand.
Now we use go build to build the app-link-foo project.
$cd app-link-foo
$go build
main.go:6:2: no required module provides package github.com/bigwhite/foo; to add it:
go get github.com/bigwhite/foo
We see: go build analyzes the dependencies of app-link-foo and asks for the code of the foo packages on which it depends, but we can’t satisfy go build’s request!
Some people might say: go build supports passing parameters to the compiler and linker, so can we just tell the compiler and linker where foo.a is located and build it? Let’s try it:
$go build -x -v -gcflags '-I /Users/tonybai/Go/src/github.com/bigwhite/experiments/build-with-archive-only/library' -ldflags '-L /Users/tonybai/Go/src/github.com/bigwhite/experiments/build-with-archive-only/library' -o main main.go
main.go:6:2: no required module provides package github.com/bigwhite/foo; to add it:
go get github.com/bigwhite/foo
make: *** [build] Error 1
We see that even after passing gcflags and ldflags to go build and telling it the search path for foo.a, go build still throws an exception and still asks for the source code of the foo package! In other words, go build throws an exception before it even gets to the point of calling go tool compile and go tool link!
go build doesn’t support linking .a without source, so we have to bypass go build!
2. Bypassing go bulid
go build essentially invokes the go tool compile and go tool link commands to complete the build process of a go application. You can see the details of the go build build process by using go build -x -v.
Next, we’ll take on the role of “go build” and manually call go tool compile and go tool link to see if we can achieve the goal of building successfully without the source code of the dependencies.
Let’s take foo.a, a go archive with no external dependencies of its own, and build the app-link-foo project manually.
First, make sure that foo.a, built with -buildmode=archive, is placed correctly under library/github.com/bigwhite.
Next, let’s compile app-link-foo via go tool compile:
$cd app-link-foo
$go tool compile -I /Users/tonybai/Go/src/github.com/bigwhite/experiments/build-with-archive-only/library -o main.o main.go
We see that manually executing go tool compile compiles the object file correctly when passing in the .a file of the dependencies via -I. The go tool compile manual tells us that the -I option provides a directory for compile to search for package import paths.
$go tool compile -h
... ...
-I directory
add directory to import search path
... ...
Next, we use the go tool link to link main.o and foo.a together to form the executable binary file main.
$cd app-link-foo
$go tool link -L /Users/tonybai/Go/src/github.com/bigwhite/experiments/build-with-archive-only/library -o main main.o
Using go tool link and passing in the link path to foo.a with -L, we successfully linked main.o and foo.a together to form the final executable, main.
The -L option of go tool link provides the path for link to search for .a:
$go tool link -h
... ...
-L directory
add specified directory to library path
... ...
Execute the compiled and linked binary main and we will see the same output as expected.
$./main
invoke foo.Add
11
Some people may encounter a fmt.a or fmt.o not found error when running go tool compile! This is because Go 1.20 and later, Go installers will not install the standard library .a collection under \$GOROOT/pkg/\$GOOS_\$GOARCH by default, so go tool compile will not be able to find the fmt.a dependency of app-link-foo under this path.
➜ /Users/tonybai/.bin/go1.20/pkg git:(master) ✗ $ls
darwin_amd64/ include/ tool/
➜ /Users/tonybai/.bin/go1.20/pkg git:(master) ✗ $cd darwin_amd64
➜ /Users/tonybai/.bin/go1.20/pkg/darwin_amd64 git:(master) ✗ $ls
The solution is as simple as compiling and installing the standard library .a file manually with the following command.
$GODEBUG=installgoroot=all go install std
➜ /Users/tonybai/.bin/go1.20/pkg/darwin_amd64 git:(master) ✗ $ls
archive/ database/ fmt.a index/ mime/ plugin.a strconv.a time/
bufio.a debug/ go/ internal/ mime.a reflect/ strings.a time.a
bytes.a embed.a hash/ io/ net/ reflect.a sync/ unicode/
compress/ encoding/ hash.a io.a net.a regexp/ sync.a unicode.a
container/ encoding.a html/ log/ os/ regexp.a syscall.a vendor/
context.a errors.a html.a log.a os.a runtime/ testing/
crypto/ expvar.a image/ math/ path/ runtime.a testing.a
crypto.a flag.a image.a math.a path.a sort.a text/
This way both the go tool compile and the go tool link will find the corresponding standard library package!
In this example, foo.a depends only on the standard library, not on third-party libraries, which is relatively simple. Usually the packages in .a provided by the partner are dependent on third-party packages, so let’s see if the above method of compiling and linking will work if .a has third-party dependencies!
3. The .a file to be linked also depends on third-party packages.
The bar.a in the goarchive-with-deps directory is a go archive file that also depends on third-party packages. It depends on uber’s zap logging package and the zap package’s dependency chain, and the following is the contents of the go.mod file for bar:
// goarchive-with-deps/go.mod
module github.com/bigwhite/bar
go 1.20
require go.uber.org/zap v1.25.0
require go.uber.org/multierr v1.10.0
Let’s start by compiling and linking app-link-bar along the lines of app-link-foo.
$cd app-link-bar
$make
go tool compile -I /Users/tonybai/Go/src/github.com/bigwhite/experiments/build-with-archive-only/library -o main.o main.go
go tool link -L /Users/tonybai/Go/src/github.com/bigwhite/experiments/build-with-archive-only/library -o main main.o
/Users/tonybai/.bin/go1.20/pkg/tool/darwin_amd64/link: cannot open file /Users/tonybai/.bin/go1.20/pkg/darwin_amd64/go.uber.org/zap.o: open /Users/tonybai/.bin/go1.20/pkg/darwin_amd64/go.uber.org/zap.o: no such file or directory
make: *** [all] Error 1
The above error is expected because zap.a is not yet in build-with-archive-only/library. Next, we build a zap.a based on the uber zap source code and place it in the specified directory. The version of uber zap that bar.a depends on is v1.25.0, so we git clone uber zap, checkout v1.25.0 and build it.
$cd go/src/go.uber.org/zap
$go build -o zap.a -buildmode=archive .
$cp zap.a /Users/tonybai/Go/src/github.com/bigwhite/experiments/build-with-archive-only/library/go.uber.org/
Compile app-link-bar again.
$make
go tool compile -I /Users/tonybai/Go/src/github.com/bigwhite/experiments/build-with-archive-only/library -o main.o main.go
go tool link -L /Users/tonybai/Go/src/github.com/bigwhite/experiments/build-with-archive-only/library -o main main.o
/Users/tonybai/.bin/go1.20/pkg/tool/darwin_amd64/link: fingerprint mismatch: go.uber.org/zap has b259b1e07032c6d9, import from github.com/bigwhite/bar expecting 8118f660c835360a
make: *** [all] Error 1
We see a go tool link exception with the message “fingerprint mismatch”. This error means that the fingerprint of the zap package expected by bar.a does not match the fingerprint of the zap package we provided in the Library directory!
Let’s revisit the bar.a build with go build -v -x.
$go build -x -v -o bar.a -buildmode=archive
WORK=/var/folders/cz/sbj5kg2d3m3c6j650z0qfm800000gn/T/go-build3367014838
github.com/bigwhite/bar
mkdir -p $WORK/b001/
cat >/var/folders/cz/sbj5kg2d3m3c6j650z0qfm800000gn/T/go-build3367014838/b001/importcfg << 'EOF' # internal
# import config
packagefile fmt=/Users/tonybai/Library/Caches/go-build/d3/d307b52dabc7d78a8ff219fb472fbc0b0a600038f43cd4c737914f8ccbd2bd70-d
packagefile go.uber.org/zap=/Users/tonybai/Library/Caches/go-build/00/006d48e40c867a336b9ac622478c1e5bf914e6a5986f649a096ebede3d117bba-d
EOF
cd /Users/tonybai/Go/src/github.com/bigwhite/experiments/build-with-archive-only/goarchive-with-deps
/Users/tonybai/.bin/go1.20/pkg/tool/darwin_amd64/compile -o $WORK/b001/_pkg_.a -trimpath "$WORK/b001=>" -p github.com/bigwhite/bar -lang=go1.20 -complete -buildid mIMNOXMPJH00mEpw6WVc/mIMNOXMPJH00mEpw6WVc -goversion go1.20 -c=4 -nolocalimports -importcfg $WORK/b001/importcfg -pack ./bar.go
/Users/tonybai/.bin/go1.20/pkg/tool/darwin_amd64/buildid -w $WORK/b001/_pkg_.a # internal
cp $WORK/b001/_pkg_.a /Users/tonybai/Library/Caches/go-build/60/604b60360d384c49eb9c030a2726f02588f54375748ce1421e334bedfda2af47-d # internal
mv $WORK/b001/_pkg_.a bar.a
rm -r $WORK/b001/
We see that in compiling bar.a, the go tool compile uses -importcfg to get the location of go.uber.org/zap, and the printout shows that go.uber.org/zap points to a file in the go module cache: packagefile go.uber.org/zap=/Users/tonybai/Library/Caches/go-build/00/006d48e40c867a336b9ac622478c1e5bf914e6a5986f649a096ebede3d117bba-d.
So is it possible to use this same go.uber.org/zap in build app-link-bar and get through the go tool link process? Let’s try it.
$cd app-link-bar
$make build-with-importcfg
go tool compile -importcfg import.link -o main.o main.go
go tool link -importcfg import.link -o main main.o
$./main
invoke foo.Add
{"level":"info","ts":1693203940.0701509,"caller":"goarchive-with-deps/bar.go:14","msg":"invoke bar.Add\n"}
11
The compilation and linking of app-link-bar were indeed successful using -importcfg, and its execution results also met expectations! Note that we abandoned the previously used -I and -L options. Even if -I and -L are used together with -importcfg, go tool compile and link will prioritize the use of -importcfg!
Now there is another question in front of us, that is, what is the content of the file import.link in the above command line, and how is it generated? The import.link file is very large, with more than 500 lines, and its content is roughly as follows.
// app-link-bar/import.link
# import config
packagefile internal/goos=/Users/tonybai/Library/Caches/go-build/fa/facce9766a2b3c19364ee55c509863694b205190c504a3831cde7c208bb09f37-d
packagefile vendor/golang.org/x/crypto/chacha20=/Users/tonybai/Library/Caches/go-build/e0/e042b43b78d3596cc00e544a40a13e8cd6b566eb8f59c2d47aeb0bbcbd52aa56-d
... ...
packagefile github.com/bigwhite/bar=/Users/tonybai/Go/src/github.com/bigwhite/experiments/build-with-archive-only/library/github.com/bigwhite/bar.a
packagefile go.uber.org/zap=/Users/tonybai/Library/Caches/go-build/00/006d48e40c867a336b9ac622478c1e5bf914e6a5986f649a096ebede3d117bba-d
packagefile go.uber.org/zap/zapcore=/Users/tonybai/Library/Caches/go-build/e0/e0d81701b5d15628ce5bf174e5c1b7482c13ac3a3c868e9b054da8b1596eaace-d
packagefile go.uber.org/zap/internal/pool=/Users/tonybai/Library/Caches/go-build/bf/bfa96ebb89429b870e2c50c990c1945384e50d10ba354a3dab2b995a813c56a3-d
packagefile go.uber.org/zap/internal=/Users/tonybai/Library/Caches/go-build/33/33cb66c30939b8be915ddc1e237a04688f52c492d3ae58bfbc6196fff8b6b2b5-d
packagefile go.uber.org/zap/internal/bufferpool=/Users/tonybai/Library/Caches/go-build/68/68e58338a5acd96ee1733de78547720f26f4e13d8333defbc00099ac8560c8e8-d
packagefile go.uber.org/zap/buffer=/Users/tonybai/Library/Caches/go-build/7b/7bf00a1d4a69ddb1712366f45451890f3205b58ba49627ed4254acd9b0938ef8-d
packagefile go.uber.org/multierr=/Users/tonybai/Library/Caches/go-build/e7/e7cc278d56fc8262d9cf9de840a04aa675c75f8ac148e955c1ae9950c58c8034-d
packagefile go.uber.org/zap/internal/exit=/Users/tonybai/Library/Caches/go-build/18/187b2b490c810f37c3700132fba12b805e74bd3c59303972bcf74894a63de604-d
packagefile go.uber.org/zap/internal/color=/Users/tonybai/Library/Caches/go-build/e4/e419c93bea7ff2782b2047cf9e7ad37b07cf4a5a5b7f361bf968730e107a495b-d
Here, it includes the actual locations of the package .a files for the standard library packages that app-link-bar depends on, as well as bar.a and all the third-party packages that the bar package depends on. It is evident that most of these are package caches within the Go module cache.
So how do you get this import.link? Go has an importcfg.go file in the golang.org/x/tools package, and based on the Importcfg function in that file, you can get the package link information for all the packages related to the standard library. I’ve put the file under build-with-archive-only/importcfg, so you can use it.
The importcfg generated most of the package links, but there are still some missing links for third-party packages that bar.a depends on. When linking with go tool link, it throws an exception indicating the package import paths that cannot be found, such as go.uber.org/zap/internal/exit and go.uber.org/zap/internal/color. We can use the following go list command to find the link locations of these packages in the local Go module cache:
$go list -export -e -f "{{.ImportPath}} {{.Export}}" go.uber.org/zap/internal/exit go.uber.org/zap/internal/color
go.uber.org/zap/internal/exit /Users/tonybai/Library/Caches/go-build/18/187b2b490c810f37c3700132fba12b805e74bd3c59303972bcf74894a63de604-d
go.uber.org/zap/internal/color /Users/tonybai/Library/Caches/go-build/e4/e419c93bea7ff2782b2047cf9e7ad37b07cf4a5a5b7f361bf968730e107a495b-d
You can then manually copy this information into import.link. The import.link file is generated in this automated+manual process (of course you can write your own tool to get all the package link information that app-link-bar needs).
4. Summary
At this point, we have achieved the compilation of an executable program with no source code and only .a files.
However, this is a purely technical exploration, not a standard answer, nor an ideal one. The above explorations have reinforced my view that Don’t build go apps with just .a.
However, if you’re the provider of .a, and considering the fingerprint mismatch scenario, you’ll probably want to consider providing .a along with import.link, copies of all the go module cach e’s that you used to build .a, and the scripts to install those copies on the target host. This way your .a users can complete the linking process for .a files using the same dependency version.
The code tested in this article was compiled and linked under Go version 1.20. Will it fail if the version of Go used to compile .a is different from the version of Go used to compile the linked executable?
The code covered in this article can be downloaded from here.
Reference: https://tonybai.com/2023/08/30/how-to-build-with-only-archive-in-go/