Reach for -O2, ship it, assume the compiler did its job. That habit works for a desktop demo and quietly fails the moment you cross-compile the native part of an agent for phone-class hardware. The flags you passed for the build host either won’t run on the target at all, or they run fine and silently eat the memory and latency headroom your on-device inference budget was counting on. Treat GCC flags as an explicit contract, not a magic incantation. A flag set is a set of promises about three things you can measure: how big the binary is and how fast it cold-starts, how the tail of your per-inference latency behaves, and whether the same source produces a usable native binary on a target that is not your laptop. Get those promises wrong and the native portion of your agent blows its footprint or its latency ceiling — and then someone spends a week rewriting code the compiler could have handled with the right invocation. What GCC flags actually do, in practice GCC flags are arguments passed to the compiler that change how it transforms your source into machine code — which optimisations it applies, which CPU it targets, and how it links the result. Most engineers know -O2 turns on optimisation and -g keeps debug symbols. Fewer treat the flag set as a deliberate lever over the outcome, and that is where edge builds go wrong. The confusion starts because -O2 is genuinely good at what it does on the machine that compiled it. The compiler tunes instruction selection, loop transformations, and inlining decisions for the host by default. On a desktop that host and the deployment target are the same chip, so the default is fine. At the edge they are almost never the same chip. Your CI runner is an x86-64 server; your target is an ARM Cortex core in a phone. The flag that produced a fast binary in CI can produce one the target cannot execute — or one it executes without ever using the SIMD units you assumed it had. So the useful mental model is not “optimisation level = speed”. It is: each flag moves one or more of four edge-aware axes — binary size, cold-start footprint, per-inference latency, and target portability — and moving one axis usually costs you another. That trade-off is the whole game. This is the same set of runtime constraints that governs model optimization for edge inference at the model level; the compiler flags are the corresponding lever at the build level. Which optimisation flag: -O2, -O3, or -Os? The optimisation level is the first decision, and the instinct to grab the highest number is wrong for edge. -O3 adds aggressive optimisations on top of -O2 — more inlining, more loop unrolling, more auto-vectorisation. On a fat desktop workload that sometimes helps. On an edge inference binary it frequently hurts: unrolling and inlining bloat the code, which enlarges the binary, worsens cold-start (more pages to fault in), and pressures the instruction cache — and instruction-cache pressure is one of the quieter causes of tail-latency regressions on small cores. In configurations we’ve tested, -O3 on a compute kernel that was already vectorised produced a larger binary with no measurable throughput gain (observed pattern across edge porting work, not a published benchmark). -Os optimises for size — it applies most of -O2’s optimisations but skips the ones that trade size for speed. For the native portion of an on-device agent, where binary size and cold-start footprint are hard constraints, -Os is often the right default, then you selectively re-enable speed optimisations on the hot inference path. Here is the decision surface, self-contained: Flag Optimises for Binary size Cold-start Hot-path latency Sensible edge use -O0 nothing (debug) large slow poor never ship it -O2 balanced speed moderate moderate good safe general default -O3 max speed larger worse sometimes better, often flat only on a profiled hot kernel -Os size smallest fastest good edge default for the whole binary -Oz aggressive size smallest fastest can regress when size is the hard wall The honest answer for most edge agents is -Os for the bulk of the binary, -O2 (rarely -O3) applied per-function on the measured hot path via a function attribute or a separate translation unit. Do not sprinkle -O3 across the whole build because it sounds faster. What -flto and dead-code elimination buy you Link-time optimisation is the single flag with the best size-to-effort ratio for an edge binary. -flto defers optimisation to link time so the compiler can see across translation-unit boundaries — inline across files, propagate constants, and, critically, prove that whole functions are unreachable and drop them. Pair it with section garbage collection. Compile with -ffunction-sections -fdata-sections so each function and data object lands in its own section, then link with -Wl,--gc-sections so the linker discards any section nothing references. On a binary that statically links a chunk of an inference runtime, this removes code paths you never call. The combination of -Os, -flto, and -Wl,--gc-sections is the standard recipe for cutting a native inference binary’s size and cold-start footprint measurably — that is axis 4 (binary size and cold-start) moving in the direction you want. The linker matters here too, not just the compiler. Which linker resolves and garbage-collects those sections affects both link time and the final layout; we cover that trade-off in the gold linker for edge agent binaries, which is the natural companion to the flags described here. One caveat worth stating plainly: -flto makes builds slower and can surface latent bugs that only appear once cross-TU inlining exposes them. It is worth it for a shipping edge binary; it is annoying during tight inner-loop development. Turn it on for release builds, not necessarily for every local iteration. How -march and -mtune change latency and portability This is the pair that most often produces a binary that “works in CI, crashes on device”, so it deserves care. -march=X tells GCC it may emit instructions specific to architecture X. -mtune=X tells GCC to schedule code favourably for X without emitting instructions the baseline lacks. The distinction is the whole portability story: -march changes what instructions exist in the binary. If you build with -march=native on an AVX-512 CI server and run on a target without those extensions, the binary hits an illegal instruction and dies. -march=native is the single most dangerous flag in a cross-compilation pipeline — it means “the host I happen to be on”, which is exactly the wrong reference. -mtune changes only scheduling. A -mtune mismatch costs you some latency; it does not crash. It is the safe knob when you are unsure. For cross-compilation you set -march to the target’s baseline, not the host’s. For a broad ARM phone target you might build for a conservative -march=armv8-a (a baseline nearly all 64-bit ARM devices support) and use -mtune for the specific core family you expect most, so devices with that core get well-scheduled code and older devices still run. Getting -march target-correct is what keeps per-inference latency inside the hundreds-of-milliseconds budget an on-device agent lives under — vectorised kernels only vectorise if the target’s SIMD extensions are actually enabled by the flag. The vectorisation flags ride on top of -march. Auto-vectorisation (on at -O2/-O3) can only emit NEON or SVE instructions if -march says the target has them. Enabling the wrong SIMD width is the difference between an inference kernel that hits its latency budget and one that runs scalar and misses it by a wide margin. This is closely related to the accelerator-selection question in CUDA applications for edge-bound agent inference and the cross-platform path in what OpenCL is for cross-platform edge inference — the CPU flags govern the fallback path those articles route around. Cross-compilation checklist Before you trust a cross-compiled edge binary, confirm: You are using a cross-toolchain (e.g. aarch64-linux-gnu-gcc), not the host gcc with a -march override — the wrong C library and headers cause subtle ABI breakage. -march names the target’s baseline, and -march=native appears nowhere in the release build. The float ABI matches the target (-mfloat-abi, -mfpu on 32-bit ARM) — a soft/hard-float mismatch produces a binary that links but computes garbage or won’t load. You ran the binary on the actual target class, not an emulator that lies about available instructions. Sanitisers and -flto are consistent between the objects and the link step. When is -ffast-math safe for an inference kernel? -ffast-math is the flag people either love or fear, usually without knowing why. It relaxes IEEE 754 floating-point guarantees so the compiler can reorder operations, assume no NaNs or infinities, and use faster reciprocal approximations. On a vectorisable inference kernel it can be a real latency win. The cost is that it changes your numerical results. Reassociation means (a + b) + c and a + (b + c) may now differ; the no-NaN assumption means a NaN that would have propagated as a signal now silently produces nonsense. For a lot of neural-network inference this is tolerable — the model already tolerates quantisation error far larger than the reordering error, so the output classification does not change. That is the case where -ffast-math is safe: post-quantisation inference where you have measured that the output is stable. It is not safe when your kernel relies on NaN/inf as sentinels, when you do numerically sensitive reductions (large softmax denominators, running statistics), or when you must reproduce a reference implementation bit-for-bit for validation. If you use it, scope it narrowly — apply it to the specific inference translation unit, not the whole binary, and diff the outputs against an IEEE-strict build on a representative input set before trusting it. Numerical-precision trade-offs like this are a first-class concern, not an afterthought; the same discipline of measuring whether a precision relaxation actually changed the answer applies here as it does in quantisation. How do I verify the flags did what I expected? Choosing flags is a hypothesis. Verifying is the job. Do not trust the flag names; measure the axes. Binary size and cold-start (axis 4): compare size output and on-device cold-start time between builds. Use bloaty to see what grew or shrank by section. If -Wl,--gc-sections isn’t actually dropping code, the map file (-Wl,-Map) tells you what is still referenced. Latency (axis 3): measure per-inference latency distribution on the target hardware, not the host — look at the tail (p95/p99), not the mean, because instruction-cache and scheduling effects show up in the tail first. This is an operational measurement on the deployed binary, and it is the only number that decides whether the flag set survived. Portability: run on the actual target class and confirm no illegal-instruction faults. Disassemble a hot function (objdump -d) and check the SIMD instructions you expected are actually there — if you see scalar code where you expected NEON, your -march is wrong. Numerical stability: diff -ffast-math outputs against an IEEE-strict build across representative inputs; confirm the downstream decision (the classification, the token, the detection) is unchanged. A useful worked example, with assumptions stated: suppose a native inference binary built at -O2 for the host, cross-compiled naively, measures 14 MB and cold-starts in a few hundred milliseconds, running scalar because -march never enabled NEON. Rebuild with the cross-toolchain, -Os -flto -ffunction-sections -fdata-sections -Wl,--gc-sections, -march=armv8-a, and -O2 restored on the profiled matmul kernel. The expected direction: binary shrinks (dead code dropped, size-optimised), cold-start improves (fewer pages), and the hot kernel now vectorises so per-inference latency drops. Every one of those is a number you confirm on the target — not a claim you take from the flag name. FAQ What should you know about gcc flags in practice? GCC flags are compiler arguments that control how source becomes machine code — which optimisations run, which CPU the code targets, and how it links. In practice, treat the flag set as an explicit contract over binary size, cold-start footprint, per-inference latency, and portability; each flag moves one of those axes, and moving one usually costs another. Which GCC optimisation flags (-O2, -O3, -Os, -flto) matter most when the goal is a small, fast on-device inference binary? -Os is usually the right default for the bulk of an edge binary because size and cold-start are hard constraints, with -O2 restored per-function on the measured hot path. -flto combined with -ffunction-sections -fdata-sections -Wl,--gc-sections gives the best size-to-effort ratio by dropping unreachable code. -O3 is rarely worth it — it tends to enlarge the binary and pressure the instruction cache without a matching throughput gain. How do -march and -mtune affect latency and portability when cross-compiling for phone-class edge hardware? -march changes which instructions the binary contains; -mtune only changes scheduling. Set -march to the target’s baseline (e.g. armv8-a), never -march=native, which pins to the build host and can produce an illegal-instruction crash on device. A -mtune mismatch only costs some latency; a wrong -march either crashes or leaves your kernels running scalar and missing the latency budget. Which flags reduce binary size and cold-start footprint without breaking the target ABI? -Os (or -Oz) plus -flto and section garbage collection (-ffunction-sections -fdata-sections -Wl,--gc-sections) shrink the binary and cold-start footprint. Protect the ABI by using a proper cross-toolchain rather than the host gcc, and by matching the target float ABI (-mfloat-abi, -mfpu on 32-bit ARM) — a float-ABI mismatch links but computes garbage. When is -ffast-math safe for an inference kernel, and when does it change numerical results enough to matter? It is safe on post-quantisation inference where you have measured that output decisions stay stable, because quantisation error already dwarfs the reordering error. It is not safe when the kernel relies on NaN/inf sentinels, does numerically sensitive reductions, or must match a reference bit-for-bit. Scope it to the specific translation unit and diff outputs against an IEEE-strict build before trusting it. How do I verify that my chosen GCC flags actually produced the footprint and latency gains I expected on the edge target? Measure the axes on the target, not the flag names. Compare size/bloaty output and on-device cold-start for footprint, per-inference tail latency (p95/p99) for latency, objdump -d to confirm the expected SIMD instructions are present, and an output diff against an IEEE-strict build to confirm -ffast-math didn’t change decisions. How do these compile-time flag choices feed back into the four edge-aware axes of agent framework selection in the parent hub? Correct flags directly move axis 4 (binary size and cold-start) via -Os/-flto/--gc-sections and axis 3 (bounded inference latency) via target-correct -march/-mtune and safe vectorisation. Those are the same metrics that decide whether a native agent component survives the edge boundary, which is why the compile-time contract is part of the framework decision rather than an implementation detail bolted on afterward. Where this leaves the build decision The compiler will do a great deal for you, but only inside the contract you hand it. On a desktop the default contract is fine because host and target are the same. At the edge boundary — where GPU acceleration is a fallback and the native path governs the GPU-side and CPU-side inference budget — the contract is yours to write, and the generative AI systems that run on-device depend on you writing it deliberately. So the sharp question is not “which optimisation level is fastest”. It is: for this target chip, this footprint ceiling, and this latency budget, which flags keep the native portion of the agent inside its box — and did you measure the four axes on the device to prove it, or did you trust the flag names and hope? The failure class is a build that passes CI and misses its footprint or latency budget on hardware; the compile-time contract is the artifact that prevents it.
