Resolution: Smallest Measurable Difference in Sensor-Based Metrology

A measurement system with 1 µm depth resolution detects height differences of 1 µm and cannot distinguish height differences smaller than 1 µm. Resolution is a property of the complete measurement system — sensor, optics, signal chain, and processing algorithm — not of the detector element alone.

Resolution, accuracy, and precision address 3 distinct metrological concepts:

• Resolution defines the smallest detectable change in the measured quantity
• Accuracy defines the closeness of a measured value to the true value
• Precision defines the spread of repeated measurements under identical conditions

A laser profile sensor achieves 5 µm lateral resolution and simultaneously exhibits a systematic offset of 50 µm from the true profile position. The sensor resolves fine surface features while delivering inaccurate absolute position data. Resolution and accuracy are independent parameters; both require specification in a complete measurement system description.

Resolution governs the 4 primary measurement purposes relevant to this topical context: determining, comparing, monitoring, and verifying physical quantities. A sensor with insufficient resolution fails to determine feature dimensions within a required tolerance band, fails to compare surface profiles at sub-feature level, and fails to verify compliance with geometric dimensioning and tolerancing (GD&T) specifications that demand finer discrimination.

Resolution contributes directly to the measurement uncertainty budget. A measurement system with 10 µm resolution introduces a quantisation uncertainty contribution of ±5 µm — half the resolution step — into the overall uncertainty. The Guide to the Expression of Uncertainty in Measurement (GUM) classifies this as a Type B uncertainty component, evaluated by means other than statistical analysis of a measurement series.

The quantisation uncertainty \( u_q \) is calculated as:

\[ u_q = \frac{\text{resolution}}{2\sqrt{3}} \]

For a sensor with 10 µm resolution: \( u_q = \frac{10}{2 \times 1.732} \approx 2.89 \) µm (standard uncertainty, rectangular distribution assumed).

Lateral resolution is the smallest distance between 2 neighbouring surface features in the X-Y plane that a sensor separates into 2 distinguishable data points. Lateral resolution is a function of pixel pitch, lens magnification, and working distance.

A 3D laser profile sensor with 10 µm lateral resolution separates surface structures spaced 10 µm apart. At a larger working distance, the same sensor projects a larger laser spot, reducing lateral resolution to 25 µm or more. Lateral resolution is therefore a working-distance-dependent parameter, not a fixed sensor constant.

Lateral resolution directly limits the detection capability in 3 industrial inspection tasks: surface scratch detection, weld seam width measurement, and edge position determination. A scratch narrower than the lateral resolution of the inspection system produces no detectable signal change — the defect passes undetected regardless of scratch depth.

Depth resolution is the smallest height or distance difference along the Z-axis that a sensor distinguishes as 2 separate measurement values. Depth resolution is the dominant performance parameter in laser triangulation, structured-light measurement, and time-of-flight (ToF) sensing.

In laser triangulation, depth resolution depends on 3 system parameters: the triangulation angle, the detector pixel pitch, and the lens focal length. Sensors with a 30° triangulation angle achieve 4 times finer depth resolution than sensors with a 7° angle at equivalent pixel pitch. Increasing the triangulation angle improves depth resolution and simultaneously reduces the shadow-free measurement range on steep object slopes.

Depth resolution of industrial 3D laser profile sensors ranges from 0.1 µm in high-precision short-range configurations to 500 µm in long-range structural monitoring applications — covering 3 orders of magnitude.

Angular resolution is the smallest angular separation between 2 point targets that a scanning or array sensor resolves as distinct. Angular resolution applies primarily to LiDAR systems and rotating laser scanners. In a LiDAR system with 0.1° angular resolution, 2 objects separated by less than 0.1° at the operating range merge into a single detection point.

Temporal resolution is the minimum time interval between 2 consecutive measurement acquisitions at which a measurement system captures independent, non-blurred data points. Temporal resolution constrains the detection of dynamic events — vibrations, impact processes, fluid flow fronts, and thermal transients.

A profile sensor operating at 50 kHz profile rate achieves 20 µs temporal resolution, detecting surface events that occur within a 20 µs window.

The optical system determines lateral resolution through 3 parameters: pixel pitch of the detector, effective focal length of the imaging lens, and sensor format.

Pixel pitch is the centre-to-centre distance between adjacent detector pixels. A detector with 5 µm pixel pitch maps each pixel onto 5 µm × magnification in the object plane. A 1× magnification lens delivers 5 µm lateral resolution at the object; a 0.5× magnification lens delivers 10 µm lateral resolution from the same detector.

Lens quality limits resolution through diffraction and optical aberrations. The Rayleigh criterion defines the diffraction limit as:

\[ \delta_{\text{lat}} = \frac{0.61 \cdot \lambda}{\text{NA}} \]

A lens with \(\text{NA} = 0.1\) operating at \(\lambda = 660\) nm laser wavelength reaches a diffraction-limited lateral resolution of \(\delta_{\text{lat}} = \frac{0.61 \times 0.66}{0.1} \approx 4\) µm. Aberrations — chromatic aberration, field curvature, distortion — degrade achievable resolution below this physical limit in practice.

Sensor format determines the trade-off between field of view and lateral resolution at a fixed pixel pitch. A 5 Mpx sensor (2448 × 2048 pixels) at 5 µm pixel pitch covers a 12.2 mm × 10.2 mm field of view at 1× magnification with 5 µm pixel-level lateral resolution. Increasing the field of view by reducing magnification degrades lateral resolution proportionally.

The signal-to-noise ratio (SNR) of the detector output determines the effective resolution — the resolution achievable in practice rather than the theoretical optical limit. A sensor with 1 µm theoretical depth resolution and SNR of 10 achieves effective depth resolution of approximately 10 µm, because noise fluctuations mask sub-10 µm signal differences.

SNR depends on 4 factors: incident light intensity, detector quantum efficiency, read-out noise, and dark current. Cooling the detector reduces dark current and improves SNR in low-signal applications such as thermography. Signal processing steps — filtering, averaging, sub-pixel interpolation — improve effective resolution beyond the single-acquisition theoretical limit.

Resolution degrades with increasing measurement distance in all non-telecentric optical measurement systems. In a laser triangulation sensor, lateral resolution scales linearly with working distance: doubling the working distance doubles the projected pixel size and halves the lateral resolution.

Depth resolution in triangulation degrades with the square of the working distance, following the disparity gradient relationship:

\[ \Delta z = \frac{z^2 \cdot p}{b \cdot f} \]

where \( z \) is working distance, \( p \) is pixel pitch, \( b \) is baseline (distance between laser and detector), and \( f \) is focal length. A sensor achieving 5 µm depth resolution at 100 mm working distance achieves only 20 µm at 200 mm working distance.

The field of view (FOV) and resolution exhibit an inverse relationship at a fixed detector format. Selecting a larger FOV to cover a wider inspection area reduces lateral resolution for a given detector.

Surface optical properties affect the detectable signal and thereby the achievable effective resolution. 3 material-related factors reduce effective resolution: specular reflection, low reflectivity, and surface translucency.

• Specular reflection from polished metal surfaces or glass saturates detector pixels locally, creating blooming artefacts that broaden the apparent laser line profile and degrade depth resolution.
• Low reflectivity materials — blackened rubber, carbon fibre, dark anodised aluminium — return insufficient signal photons to achieve SNR above the resolution-limiting noise floor.
• Translucent materials — glass, thin polymers, silicone — transmit and scatter light inside the material, shifting the apparent surface position and creating a depth resolution artefact that appears as a broadened or bifurcated laser line profile.

Illumination wavelength, laser line width, and ambient light level each influence achievable resolution in optical measurement systems.

Laser line width directly maps to lateral resolution in laser triangulation: a 30 µm wide laser line cannot resolve features smaller than 30 µm regardless of detector pixel pitch. Laser line width is a function of laser beam quality factor M², focusing optics, and working distance.

Ambient light reduces SNR by introducing a DC offset into the detector signal, raising the noise floor and reducing effective depth resolution. Ambient light suppression techniques include: narrow-band optical filters, modulated illumination with synchronous detection, and high-speed shutters.

High Dynamic Range (HDR) imaging and Multipeak detection algorithms extend usable resolution to surfaces with mixed reflectivity within a single measurement scene.

A measurement system achieves fine resolution with poor accuracy: the sensor detects 1 µm height differences and simultaneously places each measurement 50 µm away from the true surface position. A system achieves high accuracy with coarse resolution: the sensor places absolute measurements within ±5 µm of true position and cannot detect height differences finer than 20 µm.

Repeatability and resolution differ in 1 fundamental respect: repeatability describes the statistical spread of repeated outputs for a fixed input; resolution describes the minimum input change that produces any output change. A sensor with 1 µm repeatability and 5 µm resolution reliably distinguishes 5 µm height changes in repeated measurements while all repeated measurements of a fixed surface scatter within a ±1 µm band.

Geometric Dimensioning and Tolerancing (GD&T) per ISO 1101 and ASME Y14.5 specifies the permissible variation of features of size, form, orientation, location, and runout. The measurement system resolution must be finer than the tolerance to be verified.

The 10:1 rule (gauge maker’s rule) in industrial metrology states: the expanded measurement uncertainty of the verification system must be no larger than 10 % of the tolerance band being verified.

A sensor with 50 µm depth resolution cannot reliably verify a 100 µm flatness tolerance, because the resolution alone consumes 50 % of the tolerance budget before any other uncertainty source contributes.

Resolution contributes a quantisation uncertainty component to the combined measurement uncertainty. The GUM framework classifies this as a Type B, rectangular distribution contribution:

\[ u_q = \frac{\text{resolution}}{2\sqrt{3}} \]

For a sensor with 10 µm resolution: \( u_q = \frac{10}{2 \times 1.732} \approx 2.89 \) µm standard uncertainty. The combined measurement uncertainty \( u_c \) aggregates this contribution with repeatability, thermal drift, calibration uncertainty, and other Type A and Type B components through root sum of squares (RSS) combination.

Laser profile sensors in industrial quality control achieve 3 performance classes, defined by working distance and corresponding resolution range:

A 3D laser profile sensor operating at 250 mm working distance with a 100 mm measurement range achieves typical depth resolution of 5–15 µm and lateral resolution of 40–80 µm. These values represent system resolution under controlled illumination and nominal surface reflectivity. Resolution degrades on specular or low-reflectivity surfaces.

Infrared cameras measure spatial resolution through 2 distinct parameters: the Instantaneous Field of View (IFOV) and the effective spatial resolution after detector noise filtering.

IFOV is the solid angle subtended by a single detector pixel. An IR camera with 640 × 512 pixel detector and 25 mm lens at 1 m working distance achieves an IFOV of approximately 0.9 mrad, corresponding to 0.9 mm ground sampling distance. IFOV is the theoretical spatial resolution of the IR camera.

NETD (Noise Equivalent Temperature Difference) is the thermal sensitivity — the smallest temperature difference that produces a detector output change exceeding the noise floor. NETD values of 20–50 mK are typical for uncooled microbolometer IR cameras in industrial thermography.

NETD and IFOV are independent specifications: a camera achieves fine IFOV and poor NETD in low-contrast thermal scenes. Both parameters require joint specification for a complete resolution characterisation of an IR measurement system.

Automated Optical Inspection (AOI) systems in electronics manufacturing specify lateral resolution requirements based on the smallest feature to be detected. 3 representative AOI resolution requirements illustrate the relationship:

• PCB solder joint inspection for 0.4 mm pitch QFN pads requires lateral resolution ≤ 30 µm to detect solder bridges
• Component presence verification for 0201 components (600 µm × 300 µm) requires lateral resolution ≤ 60 µm
• Fine-pitch BGA ball inspection for 0.3 mm pitch requires lateral resolution ≤ 15 µm

AOI system resolution is determined at system design stage by selecting the camera sensor format, pixel pitch, and lens magnification to deliver the required lateral resolution across the required inspection field of view.

DIN/EN ISO 10360-10 specifies the acceptance testing and reverification testing procedures for optical 3D measuring systems using area and line scanning sensors. The standard defines the Probing Form Error (PFE) and the Sphere Distance Error (SDE) as the primary performance parameters for optical 3D systems.

ISO 10360-10 does not directly specify “resolution” as a test parameter. Instead, it measures the repeatability of probing — which is functionally linked to effective depth resolution. A system with coarse depth resolution exhibits large PFE values because individual point measurements scatter across the resolution step interval.

The test procedure uses a calibrated reference sphere of known diameter. The PFE captures the form deviation of all probed points from the best-fit sphere surface, expressed as a range value in µm. A PFE of 5 µm indicates that the system’s combined resolution and repeatability places all measurement points within a ±2.5 µm band around the true sphere surface.

Industrial sensor datasheets specify resolution under 3 different labels that carry distinct metrological meanings:

A sensor specifying 0.5 µm pixel resolution and 3 µm single-point repeatability achieves an effective system resolution of approximately 3–5 µm — not 0.5 µm. Pixel resolution is a lower bound; system resolution is the performance-relevant parameter for application qualification.

Verification of resolution claims against standardised procedures requires measurement system analysis methods including Gage R&R studies. A Gage R&R study quantifies the contribution of the measurement system — equipment variation and appraiser variation — to the total observed process variation. Systems with a Gage R&R ratio above 30 % of the tolerance band fail measurement system capability requirements per AIAG MSA guidelines.