A GPU compute node that compiles OpenCL code but reports zero devices at runtime is not a broken node in any way your monitoring will tell you. It passes the network checks. It answers pings. The interconnect is wired, the topology is validated, and everything looks green — right up until a workload lands on it and finds nothing to run on. That node then sits in the cluster contributing zero GPU compute while every dashboard insists it’s healthy. This is the failure that sudo apt install optimism produces. The naive path is one command, a clean exit code, and the assumption that OpenCL now “works.” The problem is that a successful package install proves almost nothing about whether the runtime can actually see your hardware. On Ubuntu, OpenCL is not a single thing you install — it’s a loader plus at least one vendor runtime, and the two have to agree about which devices exist. Get the split wrong and you get a node that builds kernels flawlessly and executes them on nothing. What “opencl install ubuntu” actually installs When people search for how to install OpenCL on Ubuntu, they’re usually picturing one package. In practice there are two independent layers, and the reason installation fails is almost always that one of them is missing or mismatched. The first layer is the ICD loader — the Installable Client Driver loader, shipped on Ubuntu as ocl-icd-libopencl1 (the runtime) and ocl-icd-opencl-dev (the headers and the libOpenCL.so your code links against). The loader is vendor-neutral. It implements the OpenCL API surface your application calls, but it does no compute itself. Its entire job is to look up which real drivers are installed on the machine and dispatch calls to them. The second layer is the vendor runtime — the actual driver that talks to your GPU. This is where the compute happens, and it is specific to the silicon in the box: NVIDIA’s driver for NVIDIA cards, ROCm or the Mesa Rusticl/Clover path for AMD, and Intel’s Compute Runtime (NEO) for Intel GPUs. Each vendor runtime registers itself by dropping a small .icd text file into /etc/OpenCL/vendors/, containing the path to its implementation library. The loader discovers vendor runtimes by reading /etc/OpenCL/vendors/*.icd at runtime. This is the single most important mechanism to understand, because it explains the whole class of silent failures: if that directory is empty, the loader finds no runtimes, and clGetDeviceIDs returns zero devices — with no error from the loader itself. Your code compiled and linked against the loader; the loader loaded fine; there just wasn’t anything behind it. For the deeper mechanics of how the loader and runtimes interact once code is running, our walkthrough of how OpenCL works for ML inference on Linux covers the execution model in more detail. Why do you need both the ICD loader and a vendor runtime? Because they solve different problems, and OpenCL was designed around exactly this separation. The loader gives you one stable API and one library to link against, regardless of which GPU vendor’s hardware ends up in the node. The vendor runtime gives you the hardware-specific implementation. This is what lets a single OpenCL binary run on an NVIDIA node, an AMD node, and an Intel node without recompilation — the portability that motivates using OpenCL in the first place, as we discuss in what OpenCL is and why it matters for cross-vendor compute. The trade-off is that installing the loader and installing a runtime are two separate acts. Ubuntu’s ocl-icd-opencl-dev gives you a functioning API layer that reports zero devices, because it ships no vendor runtime and never could — the loader vendor can’t know what card you’ll install. Conversely, some vendor driver packages bundle their own loader, and having two loaders on PATH produces its own confusion. The clean model on a provisioned compute node: install the distribution’s ocl-icd loader once, then install exactly the vendor runtime that matches the installed GPU. Which packages install OpenCL on Ubuntu per vendor? The correct package set depends entirely on the silicon in the node. Installing an AMD runtime on an NVIDIA box gives you an ICD file pointing at a library that finds no matching device — the compile-but-zero-devices failure again. Match the runtime to the hardware. GPU vendor ICD loader (install once) Vendor runtime package(s) Registers ICD at NVIDIA ocl-icd-libopencl1, ocl-icd-opencl-dev NVIDIA driver (e.g. nvidia-driver-XXX) — bundles OpenCL /etc/OpenCL/vendors/nvidia.icd AMD (ROCm) ocl-icd-libopencl1, ocl-icd-opencl-dev rocm-opencl-runtime (via AMD’s ROCm repo) /etc/OpenCL/vendors/amdocl64.icd AMD (Mesa) ocl-icd-libopencl1, ocl-icd-opencl-dev mesa-opencl-icd (Clover / Rusticl) /etc/OpenCL/vendors/mesa.icd Intel GPU ocl-icd-libopencl1, ocl-icd-opencl-dev intel-opencl-icd (Compute Runtime / NEO) /etc/OpenCL/vendors/intel.icd A few notes that save provisioning hours. The NVIDIA OpenCL runtime rides along with the graphics/compute driver — you don’t install a separate OpenCL package, but you do need the driver actually loaded (nvidia-smi should enumerate the GPU before OpenCL will). For AMD, the ROCm runtime and the Mesa runtime are two different paths with different maturity for compute workloads; on a dedicated compute node the ROCm path from AMD’s own repository is usually what you want, and mixing both can leave conflicting ICD files. Intel’s intel-opencl-icd is packaged in recent Ubuntu releases but the version in the distribution repo often lags Intel’s own releases meaningfully — worth checking against Intel’s published Compute Runtime versions if your GPU is recent. If you’re standing up nodes that also need the SDK and headers for building, our note on getting the OpenCL runtime and SDK in place covers the development-side packages. How do you verify OpenCL sees your GPU after installation? This is the step the naive path skips, and it’s the entire point of doing this at provisioning time rather than under production load. Do not trust the install exit code. Enumerate devices explicitly and read what the runtime actually reports. The standard tool is clinfo (package clinfo). Run it and read three things: $ clinfo Number of platforms 1 Platform Name NVIDIA CUDA Number of devices 1 Device Name NVIDIA A100-SXM4-40GB Device Type GPU Max compute units 108 The diagnostic checklist for a node before it joins the cluster: Platform count > 0. If clinfo reports zero platforms, the loader found no ICD files. Check that /etc/OpenCL/vendors/ contains a .icd file and that the library path inside it exists. Device count > 0 on the right platform. A platform with zero devices means the runtime loaded but couldn’t talk to hardware — usually a driver-not-loaded or permissions problem, not an install problem. Device Type is GPU, not CPU. Some runtimes register a CPU fallback device. A node reporting only a CPU OpenCL device is compute-invisible for GPU purposes even though clinfo shows a device. Device Name matches the installed hardware. If the name is wrong or generic, you may have the wrong vendor runtime registered. The unprivileged service user sees the same devices. Run clinfo as the account that will actually run workloads, not just as root. Device node permissions (/dev/dri/*, /dev/nvidia*) frequently differ, and a device visible to root but not to the workload user is a classic silent-under-load failure. Only when all five hold should the node be marked ready. The measurable payoff is direct: the count of nodes with correct device enumeration before cluster join, versus provisioning-time failures caught against the far more expensive failures discovered under production load. In our experience standing up compute nodes, the debugging hours spent chasing “device not found” runtime errors collapse once this five-point check moves from post-incident to pre-join — an observed pattern across engagements, not a benchmarked figure. Why does OpenCL compile but report zero devices at runtime? Because compilation and device enumeration are governed by two different things, and only one of them was actually satisfied. Your build succeeded because it linked against the ICD loader’s libOpenCL.so and the OpenCL headers — both of which exist the moment you install ocl-icd-opencl-dev. Nothing in that build step consults a GPU or a vendor runtime. The compiler is happy with an API surface. Device enumeration happens at runtime, when clGetPlatformIDs and clGetDeviceIDs ask the loader for real hardware, and the loader walks /etc/OpenCL/vendors/. If no vendor runtime registered an ICD file there — or the file points at a missing library, or the driver isn’t loaded, or the workload user lacks device permissions — you get zero devices and no compile-time warning. The symptom looks like a code bug; the cause is a provisioning gap. The fix follows the diagnosis: Empty /etc/OpenCL/vendors/ → the vendor runtime was never installed. Install the package matching the GPU (see the table above). .icd file present but zero devices → check the driver is loaded (nvidia-smi, or dmesg for AMD/Intel) and that the library path inside the .icd file resolves. Devices visible as root but not to the service user → fix device-node group membership and permissions for the workload account. Only a CPU device appears → the GPU vendor runtime is missing or mismatched; a CPU-only ICD (like pocl-opencl-icd) has taken over. None of these are exotic. They’re the ordinary consequences of treating a two-layer system as a one-command install. How does per-node OpenCL setup fit into provisioning a GPU cluster? It’s the last-mile software check that sits after the fabric work and before the node is trusted. The interconnect design, the topology, the cabling — how MLOps architecture for GPU clusters frames the orchestration layer — all of that establishes that the node can communicate. None of it establishes that the node can compute. Those are independent properties, and the OpenCL device check is what closes the gap. The reason this matters at provisioning time specifically is that a compute-invisible node is a silent failure. It doesn’t fault. It doesn’t drop off the network. It contributes zero GPU work while presenting as healthy, and the cost of finding that out shifts by orders of magnitude depending on when you find it. Caught during provisioning, it’s a package fix on one node. Caught under production load, it’s a debugging session across a cluster that was supposed to be validated, chasing a symptom that points at code rather than at the node’s software stack. Folding OpenCL device verification into the node-readiness gate — the same gate that validates the GPU acceleration stack for the workload — is what keeps a correctly-wired cluster from quietly running short of the compute it was built to deliver. FAQ How should you think about installing OpenCL on Ubuntu in practice? Installing OpenCL on Ubuntu means installing two independent layers: the vendor-neutral ICD loader (ocl-icd-libopencl1 / ocl-icd-opencl-dev) that your code links against, and a vendor runtime that matches the GPU in the node. A successful apt install of the loader alone gives you a working API surface that reports zero devices, because the loader does no compute — it only dispatches to whatever vendor runtimes have registered themselves. What is the difference between the OpenCL ICD loader and a vendor runtime, and why do you need both? The ICD loader implements the OpenCL API and, at runtime, discovers installed vendor runtimes by reading /etc/OpenCL/vendors/*.icd. The vendor runtime is the hardware-specific driver that actually executes compute on the GPU. You need both because the loader gives you one stable, portable API to link against while the vendor runtime provides the implementation for the specific silicon — the split is what lets one OpenCL binary run across NVIDIA, AMD, and Intel hardware. Which packages do you install on Ubuntu for NVIDIA, AMD, or Intel GPUs to get OpenCL working? Install the ocl-icd loader once for any vendor, then add the matching runtime: the NVIDIA driver bundles OpenCL for NVIDIA cards, rocm-opencl-runtime or mesa-opencl-icd covers AMD, and intel-opencl-icd (the Compute Runtime / NEO) covers Intel GPUs. Match the runtime to the installed hardware — an AMD runtime on an NVIDIA node registers an ICD that finds no matching device. How do you verify that OpenCL sees your GPU devices after installation? Run clinfo and confirm five things: platform count above zero, device count above zero on the correct platform, Device Type reported as GPU rather than CPU, the Device Name matching the installed hardware, and the same devices visible to the unprivileged workload user — not just root. Only mark the node ready when all five hold; the exit code from the install command proves none of this. Why does OpenCL code compile but report zero devices at runtime, and how do you fix it? Compilation only needs the ICD loader’s library and headers, neither of which consults a GPU, so the build succeeds even with no vendor runtime present. Zero devices at runtime means the loader found no usable ICD in /etc/OpenCL/vendors/ — the runtime wasn’t installed, its library path is broken, the driver isn’t loaded, or the workload user lacks device permissions. Fix it by installing the correct vendor runtime, verifying the driver is loaded, and checking device-node permissions for the service account. How does per-node OpenCL setup fit into provisioning nodes for a GPU cluster? It is the last-mile software check that runs after the interconnect and topology are validated and before the node is trusted to join. Fabric work proves a node can communicate; the OpenCL device check proves it can compute — independent properties. Folding device verification into the node-readiness gate turns a compute-invisible node into a one-node package fix at provisioning time rather than a cluster-wide debugging session discovered under production load. A node that answers every network probe and runs no GPU work is the most expensive kind of healthy. The question worth carrying into your provisioning pipeline is not “did OpenCL install?” but “which devices does the workload user see before this node joins?” — and whether your readiness gate is actually asking it.
