Spectral Total-Variation Local Scale Signatures for Image Manipulation and Fusion

Ester Hait and Guy Gilboa

We propose a unified framework for isolating, comparing and differentiating objects within an image. We rely on the recently proposed total-variation transform, yielding a continuous, multi-scale, fully edge-preserving, local descriptor, referred to as spectral total-variation local scale signatures. We show and analyze several useful merits of this framework. Signatures are sensitive to size, local contrast and composition of structures; are invariant to translation, rotation, flip and linear illumination changes; and texture signatures are robust to the underlying structures. We prove exact conditions in the 1D case. We propose several applications for this framework: saliency map extraction for fusion of thermal and optical images or for medical imaging, clustering of vein-like features and size-based image manipulation.

Concept and Algorithm:

Spectral TV Local Scale Signatures: a multi-scale spectral TV per-pixel description, sensitive to  size, local contrast and composition of structures, invariant to translation, rotation, flip and linear illumination change, with texture invariance to underlying structure. Pixels with common features (strawberry seeds) have similar signatures; different pixels can be differentiated by their distinct signatures.
Image manipulation and image fusion using spectral TV local scale signatures: algorithm flowchart.

Some experimental Results:

Salient structure extraction for a repetitive image.
Salient structure extraction and image manipulation for a repetitive image.
Image manipulation using size differentiation for repetitive images.
Thermal / RGB fusion.
Left: thermal / RGB fusion. Right: medical MRI-T2/T1 fusion.
Thermal / grayscale fusion.

Paper, Supplementary and Matlab Code:

Hait, Ester, and Guy Gilboa. “Spectral Total-Variation Local Scale Signatures for Image Manipulation and Fusion.” IEEE Transactions on Image Processing 28.2 (2019): 880 – 895.

Early access (peer-reviewed and accepted) version in IEEE Transactions on Image Processing:

Earlier version of the paper in HAL


Demo Matlab Code

Contact us at: