Linearity in Metrology: Evaluating Sensor Measurement Behavior Across the Measurement Range

The Best-Fit Line is calculated by the method of least squares: it minimizes the sum of squared residuals across all calibration data points. The End-to-End Line connects 2 fixed points — the zero-point output and the full-scale output — in a straight line. The Best-Fit method yields a smaller stated linearity error because the reference line is positioned to minimize maximum deviation. The End-to-End method typically produces a larger stated linearity error because the reference line is constrained to pass through 2 fixed endpoints.

Practical consequence for sensor datasheets: 2 sensors from different manufacturers can have identical physical linearity performance but different stated linearity errors if one manufacturer uses the Best-Fit method and the other uses the End-to-End method. Before comparing linearity specifications, engineers must identify which reference line method the manufacturer applies.

Full Scale Output (FSO) is the difference between the sensor output at the maximum measurement position and the output at the minimum measurement position. Linearity error expressed as % FSO is independent of the absolute measurement range — a sensor with 0.1 % FSO linearity error maintains the same relative performance whether its measurement range is 10 mm or 500 mm. Absolute linearity error in µm scales proportionally with the measurement range.

The relationship between relative and absolute linearity error is expressed as:

\[ E_{\text{abs}} \; [\mu m] = \frac{\text{Linearity} \; [\% \text{FSO}]}{100} \times \text{Measurement Range} \; [\mu m] \]

For a 3D triangulation sensor with a 200 mm measurement range and a specified linearity of \( 0.05\,\% \) FSO, the maximum absolute deviation from the ideal output line is 0.1 mm.

Linearity error in industrial sensors is determined by 4 primary factor categories: optical path characteristics, electronic signal processing, thermal behavior, and mechanical deformation.

Optical path non-linearity occurs in laser triangulation sensors when the relationship between the measured object position and the spot position on the detector is not perfectly linear — a result of lens distortion, detector geometry, and optical axis alignment. Electronic non-linearity originates in amplifier stages, analog-to-digital converter (ADC) transfer characteristics, and signal conditioning circuits. Thermal effects cause linearity drift when the sensor operates outside its calibrated temperature range — thermal expansion of optical components and temperature-dependent gain changes in detector circuits contribute to this drift. Mechanical deformation under mounting stress or vibration can introduce additional position-dependent output deviations that appear as linearity error.

Linearity error is reported in 3 standard formats in industrial sensor specifications: peak-to-peak deviation, maximum deviation, and RMS (Root Mean Square) deviation. Each format communicates a different aspect of the output curve quality.

The RMS value is always smaller than the peak-to-peak value for the same sensor. A sensor datasheet that reports linearity as RMS appears more favorable than one reporting the same sensor’s performance as peak-to-peak. Engineers must confirm the reporting format before comparing specifications across manufacturers.

The evaluation of sensor characteristic curves and linearity specifications references 3 primary standards in industrial metrology: IEC 60770-1 for transmitters used in process control, VDI/VDE 2634 Part 1 for optical 3D measuring systems, and ISO 12012 for laser trackers. These standards define terminology, test procedures, and reporting requirements — they do not prescribe which reference line method must be used. The automotive standard IATF 16949 requires measurement system analysis (MSA) documentation that includes linearity assessment for all measurement systems used in production.

3D laser triangulation sensors are verified for linearity using 3 types of calibration references: gauge blocks (step gauges) for discrete position calibration, optical flats for surface reference, and motorized precision stages with encoder feedback for continuous range scanning. Step gauges with nominal step heights traceable to national length standards provide the highest calibration accuracy. A typical linearity test covers a minimum of 10 equidistant calibration positions across the full measurement range.

The residual plot from a triangulation sensor linearity test visualizes the deviation of the sensor output from the reference line at each calibration position. A well-performing sensor produces a residual plot with no systematic pattern — deviations are distributed symmetrically around zero without a repeating shape. An S-curve or saddle pattern in the residual plot indicates a systematic optical non-linearity that requires factory correction or software-based curve correction.

Infrared cameras are verified for linearity across their temperature measurement range using blackbody radiation sources. A blackbody radiator is a calibration reference that emits thermal radiation at a precisely controlled and traceable temperature, providing a known radiance level at each calibration point. The linearity test for an infrared camera records the digital output value (detector counts or temperature output) at a minimum of 8 temperature setpoints distributed across the camera’s full measurement range.

Thermal sensors characteristically exhibit higher linearity error at the extreme ends of their temperature measurement range — near the lower and upper range limits — than in the mid-range region. This behavior results from the non-linear relationship between blackbody radiation and temperature described by Planck’s radiation law. The detector’s response function and the camera’s signal processing chain introduce additional non-linearity, particularly at high-radiance input levels near the upper measurement limit.

Linearity is one of the 5 MSA characteristics evaluated for a measurement system alongside bias, stability, repeatability, and reproducibility. Linearity assessment for infrared cameras in production environments follows the MSA Reference Manual (AIAG) procedure for linearity studies, which requires measurements at a minimum of 5 reference values and the calculation of linearity bias over the range.

Calibration as a corrective measure: If linearity verification identifies a systematic error pattern, factory calibration applies a correction polynomial or lookup table to the sensor output to reduce the linearity error to within the specified limit. Calibration intervals for 3D sensors and infrared cameras are typically 12 months in industrial environments, with interim verification checks after mechanical shocks, temperature excursions, or replacement of optical components.

3 industry standards define linearity requirements for measurement systems used in production: VDI/VDE 2634 Part 2 specifies acceptance and reverification tests for optical 3D measuring systems — linearity is evaluated as part of the probing error and length measurement error tests. ISO 10360-10 addresses coordinate measuring systems using laser trackers and triangulation sensors — it defines maximum permissible linearity errors as a function of the measurement volume. IATF 16949 (Automotive) requires that all measurement systems used in production and process control undergo MSA studies that include linearity evaluation — the acceptable linearity bias limit is typically defined as less than 5 % of the process tolerance.

In robot-guided measurement applications, linearity error has additional significance: the robot positions the sensor at varying distances from the measurement target depending on the workpiece geometry. If the sensor’s linearity error is not corrected or compensated, the robot path determines not only the measurement position but also the systematic measurement error. This coupling between robot positioning and sensor linearity error is a critical source of uncertainty in flexible automated measurement cells.