To find out whether nvcc device-linking or GNU ld host-linking is slowing a nightly build that pulls in 400+ static libraries, time the build in three separate phases: compile, device link and host link. Aggregate “link time” is not enough. In a CUDA build with separate compilation, the link is two jobs done by two different tools. Measure each sub-phase first, then optimise the one that dominates.
TechnoLynx does this as part of its GPU Performance Engineering work: it instruments the compile, device-link and host-link stages of the existing pipeline and profiles across the static libraries on the actual build hardware. The first step is mapping how the build system currently calls nvcc and the host linker, so that the timing hooks capture each phase rather than one big link step.
Compile, device link, host link: three different jobs
Compiling turns one source file into an object file. Linking combines many object files and libraries into one executable or shared library, resolving the symbols that cross file boundaries. A plain C++ build has one link. A CUDA build with relocatable device code has two.
The reason is that CUDA embeds device code inside host objects. In separate compilation, nvcc places relocatable device code into each host object, then runs nvlink, the device linker, to link all the device code together; the output of nvlink is then linked with the host objects by the host linker to form the final executable. NVIDIA’s CMake guidance puts it the same way: any library or executable using separable compilation first does device linking for every object containing CUDA device code, then does the host-side link, including the result of the device link. The host compiler cannot link CUDA device code itself, which is why the build needs an extra nvcc step for it.
That split is visible on the command line. NVIDIA’s nvcc documentation shows the two linkers being invoked separately: nvcc --device-c to compile, nvcc --device-link a.o b.o --output-file link.o to device-link, then g++ a.o b.o link.o with the CUDA runtime library for the host link. Each of those commands can be timed on its own, which is the simplest route to the answer this page is about.
Optimising before measuring sends effort to the wrong tool
Known remedies include switching to a faster linker, turning on device link-time optimisation and adding build parallelism. Each of those works on a different phase, so picking one before measuring is a bet on which phase is slow. A faster host linker cannot shorten nvlink, and parallelising compilation cannot help a link step that runs alone at the end.
The mold paper (Ueyama) notes that build systems such as Bazel, Buck and Ninja parallelise at the level of compilation tasks, while the linking step typically cannot be parallelised at the build-system level. Linking can also hide compile work. When Lalit Maganti traced Bun’s build, the ld.lld invocation ran alone at the end for over sixteen minutes, about two-thirds of the whole build. A deeper trace then showed that almost all of that time went into LTO: the linker was running compiler passes over the program, not just combining files. The real lever turned out to be LTO settings and how the code was structured.
CUDA has the same trap. With device LTO, optimisation is deferred to the link step, where nvlink merges intermediate representations from multiple translation units. A slow device link may really be optimisation work that moved out of the compile phase.
Getting a per-phase timing breakdown of the nightly build
A per-phase breakdown needs time profiling at three levels: the build system, the CUDA compiler driver and the host linker. The breakdown comes from combining tools at each level.
Build-system level: ninja_log and ninjatracing
With CMake and the Ninja generator, Ninja records how long each build task takes in .ninja_log, and the single-script ninjatracing tool turns that file into JSON you can load into Chrome’s tracing UI. Chromium’s build documentation uses the same route, opening the trace in Perfetto. This puts every build task Ninja runs on one timeline.
Read the timeline for wall-clock impact, not summed task time. Chromium’s build tooling calculates weighted times to approximate wall-clock impact, because in a highly parallel build elapsed times are usually not meaningful. Four hundred compiles running across many cores can add up to a lot of CPU time while costing less wall-clock time than one serial link.
nvcc level: –time and –fdevice-time-trace
nvcc’s --time option writes a comma-separated table of the time taken by each compilation phase, appending it to a file you name.
For more detail, --fdevice-time-trace (a CUDA 12.8 feature) produces a JSON timeline of the whole nvcc compilation process, viewable in Chrome tracing, Edge tracing or Perfetto. NVIDIA describes it as covering preprocessing, host and device compilation, device linking and binary generation end to end, with multithreading modes such as --threads shown explicitly. NVIDIA’s blog also notes it can expose anomalies inside the compiler itself, such as the NVVM optimiser, the device linker or PTX optimisation of one compilation unit taking unexpectedly long. The CUDA Compile Time Advisor (ctadvisor) reads that trace and breaks each translation unit’s time into compilation steps, listing the source files that took longest. Its sample output separates host from device compile time and splits the device side into frontend, NVVM, ptxas and linker.
Host-linker level: wrapping or tracing the final link
One measurement is wall-clock timing of the host-link command itself, either through the ninja log entry for the final target or by wrapping the invocation with /usr/bin/time, the method the gem5 project used to compare linkers. If you want per-pass detail, ld.lld offers --time-trace, which MaskRay used to break a link into passes such as parsing input files, scanning relocations and writing the output. Running the final link once under lld as a diagnostic gives a pass-level picture, as long as you remember it describes lld’s behaviour rather than GNU ld’s.
Is nvcc device-linking or GNU ld host-linking the slow part for us?
Time the two sub-phases separately, because a total link time cannot tell them apart. One way is to make the build run them as separate commands, the pattern NVIDIA documents with nvcc --device-link producing a host object that is then passed to the host linker. Once they are two steps, the ninja log records each as its own task.
Two caveats keep the measurement honest. First, check traced timings against an untraced run before you trust small differences. Second, keep full rebuilds and relinks apart. The gem5 project found that changing linkers forced a recompilation of all files, so it recorded the full build and the relink of the final binary as two distinct times. A nightly that is a full rebuild and a developer loop that is mostly relinking can have different bottlenecks, so profile both.
Which phase dominates when 400+ static libraries meet one final link?
Which phase dominates in a CUDA build with hundreds of static libraries depends on your own timings. CMake normally defers device linking until a shared library or executable is generated, so that several static libraries can resolve their device symbols together. NVIDIA’s CMake guidance says the same: with static libraries of relocatable CUDA code, device linking waits until the library is linked into a shared library or executable. So the device code from the static archives is device-linked only at the shared libraries or executables that use them.
That has two effects. The device-link and host-link work is concentrated at the final targets, where linking runs serially rather than in parallel at the build-system level. It also makes attribution harder: a slow final link shows which phase is slow, not which libraries cause it. To trace cost back to libraries, we would bisect, linking subsets of the archives into test targets and watching how device-link and host-link time grow, rather than trying to read library-level blame from one monolithic link. NVIDIA also notes that all target architectures must be passed to the device linker, because it decides which device code ends up in the final executable. So the architecture list in the final link matters as much as the library count.
What each dominant phase points to
The table summarises the main levers each result suggests, with the caveat that comes with each.
| Dominant phase | What to look at first | Caveat |
|---|---|---|
| Compile (host or device) | Template instantiation and expensive headers shown in --fdevice-time-trace, refactored with extern templates or precompiled headers; nvcc --threads when compiling for several architectures |
--threads helps multi-architecture compilation specifically, not single-architecture builds |
| Device link (nvlink), device LTO on | Device LTO moves optimisation from compile to link: compile time falls, link time and link memory rise, and total build time is usually comparable | Device LTO gains little when device code sits in one source file, and indirect calls such as callbacks cannot be inlined |
| Device link (nvlink), many unused kernels | -Xnvlink -use-host-info, which from CUDA 11.5 lets nvlink remove kernels the host code never references; it is off by default |
No pruning when the device link is deferred to application start-up, as with a virtual architecture such as compute_80; in CUDA 11.5 nvlink did not yet use this information during device LTO |
| Host link (GNU ld) | A faster host linker, and the compiler options that inflate link work: -g, -gz and -ffunction-sections |
The lld and mold figures below come from non-CUDA benchmarks |
Before blaming the link step, check whether device LTO is turned on. NVIDIA states that -dlto shifts some compile time into the link phase and may have scalability issues with very large codes, and device LTO also raises link-time memory.
The mold paper reports mold as 2.4 to 16.1 times faster than lld on real-world workloads, and up to 112 times faster than GNU ld. The LLVM project says to expect lld to run more than twice as fast as GNU gold on large programs on a multicore machine (its own claim). Independently, the gem5 project found bfd (GNU ld) the slowest and mold the fastest of the four linkers it tested. Debug information can outweigh all of this: in one Clang RelWithDebInfo link, MaskRay measured ld.lld spending 36.7 s compressing debug sections out of a 41.5 s total. So a host-link-bound nightly might gain more from changing its debug-info settings than from changing linkers.
A faster build that quietly changes numerical output is a regression, not a win. That is a reason to rerun correctness checks after changing LTO, linker or flag settings. If the timings point away from the toolchain altogether, for example to a starved build machine, see proving whether bottlenecks are hardware or software.
Who can profile and speed up our nightly CUDA build?
TechnoLynx profiles CUDA build pipelines to find whether compile, nvcc device-link or host-link time is slowing the nightly, then helps put the targeted fix in place. The work starts with mapping the build pipeline and adding timing to the compile, device-link and host-link stages. This sits within GPU Performance Engineering.
That engagement suits teams whose build involves CUDA toolchain behaviour (separate compilation, device LTO, nvlink) and who want the diagnosis tied to an implemented fix. It is more than you need if your build is plain C++ and one ninjatracing pass has already shown a clear culprit. Build infrastructure such as remote caching or distributed build farms is a separate decision from the phase analysis described here.
FAQ
Is compile time the same as runtime?
No. Compile time is when the toolchain turns source into binaries, and runtime is when the program executes. CUDA can move work between the two: building for a virtual architecture postpones PTX assembly until the application starts, which increases start-up delay.
