One afternoon I was trying to build a C library for an embedded project when I started getting errors that looked like gibberish to me. Undefined references to __libc_start_main, undefined symbol warnings from dlopen, a couple of “multiple definition” errors for functions I only defined once. The compiler was confused, and so was I. By the time I’d worked through it, I’d learned more about ld than I wanted to.

The first kind of error:

/usr/bin/ld: /tmp/foo.o: in function `main':
foo.c:(.text+0x1e): undefined reference to `sqrt'

Classic. sqrt is in libm, which isn’t linked by default. gcc foo.c -lm fixes it. But the error gets subtler when transitive dependencies are involved. Suppose you have:

  • libfoo.a which calls sqrt
  • Your binary which calls foo_do_thing from libfoo

Historically you’d write gcc main.c -lfoo -lm and it would work. But with -Wl,--as-needed (which many distros enable by default now), the linker only keeps a library if a symbol from it is referenced BEFORE it’s encountered on the command line. So:

  • gcc main.c -lm -lfoo — -lm comes before any reference to sqrt, so the linker thinks it’s unneeded and drops it. Then libfoo references sqrt, which is now unresolved. Boom.
  • gcc main.c -lfoo -lm — libfoo is processed first, pulls in sqrt as an undefined symbol, then -lm resolves it. Works.

The order matters. With --as-needed, it matters even more. I use:

# Put the object files and your own libs first, then system libs
gcc -o app main.o mylib.a -lfoo -lm -lpthread

The rule: right-to-left ordering of dependency. If A depends on B, A must appear BEFORE B on the command line.

The second kind of error:

/usr/bin/ld: multiple definition of `global_config'; foo.o:(.bss+0x0): first defined here

Usually this means you declared a global variable in a header file without extern. In C:

// config.h - WRONG
int global_config = 42;

Every .c file that includes this header defines global_config, and the linker sees multiple definitions. The fix:

// config.h - RIGHT
extern int global_config;

// config.c
int global_config = 42;

Historically, GCC let you get away with the multiple-tentative-definition case (if the variable is declared but not initialized in multiple files, the linker picks one). This was called “common symbols” and it was default. GCC 10 changed the default to -fno-common, which now treats those as errors. Good change, surprised a lot of people including me.

The third kind, the weird one:

/usr/bin/ld: /tmp/foo.o: in function `bar':
foo.c:(.text+0x1a): undefined reference to `_ZN3std6thread7Builder5spawnIFvvEvE17hxxxE'

That mangled name is from a Rust library. Mixing languages through the C ABI is always more exciting than it should be. Rust’s mangling scheme changed at one point, and if you have two .rlib files built with different mangling schemes, they don’t link cleanly. The solution is usually “rebuild everything with the same toolchain.” Same story for mixing C++ versions — std::string has different ABIs with _GLIBCXX_USE_CXX11_ABI=0 vs 1, and mixing libraries causes undefined references.

A few tools I learned to use that afternoon:

nm prints symbols. nm -C libfoo.so lists all symbols in the library with C++ names demangled:

$ nm -C libmy.so | grep sqrt
                 U sqrt@@GLIBC_2.2.5

The U means “undefined” — this library references sqrt but doesn’t define it. That tells you which libraries need to be linked with it.

ldd shows dynamic dependencies of an executable or shared library:

$ ldd ./app
    linux-vdso.so.1 (0x00007ffd...)
    libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f...)
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f...)

If something says “not found,” that library isn’t in the loader’s search path at runtime.

objdump -d disassembles. Useful when you’re trying to verify that a specific function exists or wasn’t inlined away.

readelf -d shows ELF dynamic section entries, including DT_RPATH and DT_RUNPATH:

$ readelf -d ./app | grep PATH
  0x000000000000001d (RUNPATH)     Library runpath: [$ORIGIN/../lib]

If your binary has DT_RUNPATH set to $ORIGIN/../lib, it searches for shared libraries there (relative to the binary) before falling back to the system paths.

patchelf can modify ELF metadata post-hoc — adding rpaths, changing the interpreter, renaming dependencies. Helpful when the build system got something wrong:

patchelf --set-rpath '$ORIGIN/../lib' ./app

A specific pattern that bit me:

/usr/bin/ld: /tmp/foo.o: undefined reference to `dlopen'

dlopen is in libdl, on some glibc versions. On glibc 2.34+, it’s folded into libc, so -ldl is no longer needed and doesn’t hurt. On older systems you need -ldl. The difference caused CI failures when we upgraded our build base image.

Finally, the nuclear option for “I have no idea why the linker is doing this”:

gcc ... -Wl,--verbose

This makes ld print the linker script it’s using and how it’s resolving each symbol. It’s a wall of text, but when you’re stumped, grep it for your failing symbol and you can see exactly where it expected to find it.

Linking feels arcane because historically it was the last-added step in the compilation pipeline, and it accrued weird flags over decades. But once you’ve spent a day with it, it becomes boring. Boring is good.

For the macOS equivalent pain, see my post on Mach-O segments.