arXiv 2606.04493 · two-view correspondence learning

Smooth inliers,erratic outliers.

SFMambaNet: Frequency-Aware Mamba for Two-View Correspondence Learning

Correct matches between two photos shift together; wrong ones jump against their neighbours. SFMambaNet scans the matches cluster by cluster, where that difference becomes one of frequency, and filters its Mamba scans with learnable frequency-domain gates.

Zhihua Wang1,2,*, Yifeng Gao1, Yunhao Feng3, Rui Pan1, Zhouyuan Ma1, Baiheng Zhao1, Zixing Chen1, Yanping Li4, Yizhang Liu5, Xingjun Ma1,6,†

1Fudan University· 2University of Shanghai for Science and Technology· 3Ant Group· 4Shanghai Jiao Tong University· 5Fuzhou University· 6Shanghai Innovation Institute

*Work completed during an internship at Fudan University · †Corresponding author

Pruning, liveBuckingham Palace · YFCC100M · unseen scene
556inliers1,444outliers
Buckingham Palace, first view Buckingham Palace, second view
inlieroutlierPutative: all 2,000 nearest-neighbour SIFT matches · each method: the matches it keeps · lines as drawn in the paper
  • 73.83%mAP@5° on unseen YFCC100M scenes · SIFT, no RANSAC
  • +3.60points over MatchMamba, at fewer parameters and FLOPs
  • 2.01Mparameters · 5.01 GFLOPs · 4.80 ms per pair
  • 84%of inlier energy in the lowest tenth of frequencies, scanned by cluster (outliers: 15%)

01 · Insight

Frequency needs an order.

Correspondences form a set, and a set has no spectrum until it is read in some order. Read at random, where each match lands in the second image is noise for inliers and outliers alike. Read cluster by cluster, the inliers move in smooth, low-frequency steps while the outliers stay erratic.

a · Putative matchesNotre-Dame · 150 of 2,000 SIFT matches drawn
Notre-Dame de Paris, south facade, first view Notre-Dame de Paris, south facade, second view
Scan orderrandom order
First view in gray, keypoints coloured by scan position
firstcolour = position in the scanlast
Where each match landsu′ in the second view
● outliers — inliersscan position →
Spectrum of u′relative power · log–log
frequency, cycles per match →dotted: flat spectrum
9.9%of the inliers' energy sits in the lowest tenth of frequencies
10.0%of the outliers' energy sits there
10%is what a flat, noise-like spectrum puts there; a random order gives exactly that to both classes

Rebuilt from the paper's first figure with its own data: N = 2,000 SIFT matches between two CC0 photographs of Notre-Dame de Paris by Chabe01 (Wikimedia Commons), labelled with an estimated epipolar geometry (462 inliers, 1,538 outliers). The cluster order visits 32 k-means clusters of first-view keypoints one after another; spectra average 200 orders. Details in Appendix K.

02 · Method

Prune twice, verify once.

Two pruning blocks each keep the more reliable half of the matches. A verification block then fits the essential matrix with a weighted eight-point solver and re-checks all N matches, which recovers inliers dropped along the way. Frequency enters every pruning block twice: locally in LSGA, globally in SIGM.

Click a block · strands are matches: green inliers, red outliers

Inside each pruning block · local

Local Spectral-Geometric Feature Extractor

Embeds the N × 4 coordinates, links every match to its k nearest neighbours, and refines it with its neighbourhood through LSGA.

  • Up projection
  • ResNet ×4
  • KNN graph
  • ResNet ×4
  • LSGA ×1
Inside each pruning block · global

Spectral-Global Context Aggregator

Gathers global consensus along the cluster order with three SIGM blocks, predicts an inlier weight for every match, and keeps the top half.

  • ResNet ×4
  • SIGM ×3
  • ResNet ×4
  • weights Wk
Verification

Verification Block

Estimates Ê from the surviving matches, measures the epipolar distance of all N matches, and keeps every match below a threshold.

  • W2, C2
  • weighted 8-point
  • Ê
  • distance of all N
  • keep if below threshold
Local · LSGA

Local Spectral-Geometric Attention

Relative neighbour offsets ΔC are lifted into Fourier features with a fixed Gaussian projection B, so a small difference in motion becomes a large difference in the encoding. A local attention weighs neighbours by feature similarity and geometric consistency, and DiffPool groups the matches into clusters whose order Π defines the scan.

Efreq = [ sin(2πBΔC) ; cos(2πBΔC) ]  ·  Π = Order(DiffPool(FM))  ·  FG = Π−1(BiMamba(Π FM))

Global · SIGM

Spectral-Integrated Global Mamba

A forward Mamba scan runs along the cluster order. Its output is gated in the frequency domain, flipped, scanned backwards and gated again, so the second scan reads filtered features. The complex-valued gates learn amplitude and phase per channel and frequency, and leave Mamba's state transition untouched.

FSfwd = IFFT( FFT(Mamba(Π FGL)) ⊙ W1 )  →  Flip  →  Mamba  →  ⊙ W2  →  LN( · + · + · )

O(DN log N) per FFT O(DN) per scan 12 spectral gates · < 5% of the parameters 2.01M parameters · 5.01 GFLOPs

03 · Results

More accurate, and lighter.

Without RANSAC, SFMambaNet is the most accurate pruner on every split of relative pose estimation, with SIFT and with SuperPoint. It also gives the best precision and F-score in outlier removal and the best homography accuracy, with slightly fewer parameters and FLOPs than MatchMamba.

Relative pose estimationmAP@5° (%) · no RANSAC

Accuracy vs. cost

YFCC100M unknown · SIFT · no RANSAC · circle area ∝ parameters · baseline costs as reported for MatchMamba's comparison

Homography · HPatches
64%74%

Accuracy (%) at corner error below the threshold · trained on YFCC100M with SIFT · axis zoomed to the range shown

Outlier removal · SIFTprecision · recall · F-score (%) · brighter = higher within a column · outlined = best

04 · Analysis

Frequency, not capacity.

Controlled experiments ask why it works. Point-wise layers of exactly the gates' size lose most of the gain, a parameter-free low-pass filter keeps more of it, and the learned gates, left alone, become low-pass. Every number below is a five-seed mean.

Components

What each module adds

The baseline swaps LSGA and SIGM for identities. From known to unseen scenes the gain of LSGA shrinks while that of SIGM holds, so global frequency gating is what carries the model to new scenes.

On unseen YFCC100M (mAP@5°), LSGA adds 8.84 points and SIGM another 8.93.

Gate controls

Filtering beats capacity

Same network, four gates. Point-wise MLPs with exactly the gates' parameter count recover little; a fixed low-pass filter with no parameters does better; learning the response adds the rest.

Over identity gates on unseen YFCC100M: MLP +0.68 · low-pass +1.55 · learned +3.16

Encoding control

Sinusoids, not dimensions

Inside an unchanged attention, a linear lift to the same 128 dimensions gains almost nothing over raw coordinates, while Fourier features roughly triple the gain that raw coordinates bring over no explicit geometry.

Over no explicit geometry on unseen YFCC100M: raw +0.80 · linear lift +0.89 · Fourier +2.81

Inside the trained gate

The learned gate is low-pass

Nothing in training asks for a frequency response, yet the isolated first-stage gate keeps most of the low band and little of the high band. The network relies on that shape, not on energy.

Low-band energy keptinput → output of the gate58%
High-band energy keptconcentrated at outlier positions13%
AP lost · gate removedpoints, five seeds2.21
AP lost · high-freq gains restoredshape broken, rest kept1.84
AP lost · rescaled to equal energyenergy changed, shape kept0.06

Along the learned order, an outlier position carries 2.19× the high-band energy of an inlier position.

0.08pp
vs 0.62 with identity gates

Gating also tames the scan order. Mamba reads a sequence, but matches form a set. Shuffle all inputs twenty times and the pose accuracy of the gated model moves by only 0.08 points on YFCC100M and 0.06 on SUN3D, while the matches it keeps overlap by at least 98.5%. Along the cluster order, a shuffle only reorders matches inside a cluster, a high-frequency change that the gates remove.

a shuffle moves matches only inside their cluster

05 · Code

Run it yourself.

The repository holds the data-preparation, training and evaluation code. Dump YFCC100M or SUN3D correspondences with the OANet protocol, then evaluate a checkpoint with RANSAC turned off.

1 · environment
git clone https://github.com/Kirito14IT/SFMambaNet.git
cd SFMambaNet
conda create -n sfmambanet python=3.10 -y
conda activate sfmambanet
pip install torch==2.1.1 --index-url https://download.pytorch.org/whl/cu118
pip install opencv-contrib-python==4.7.0.72 mamba-ssm==1.2.0.post1
pip install h5py matplotlib numpy six tensorboard thop tqdm
2 · YFCC100M correspondences
# raw data from OANet's Google Drive (see the README):
# raw_data/yfcc100m and raw_data/pairs
mkdir -p data_dump
cd dump_match
python extract_feature.py --input_path ../raw_data/yfcc100m
python yfcc.py --raw_data_path ../raw_data/ --dump_dir ../data_dump/
cd ..
3 · evaluate without RANSAC
cd core
python main.py --run_mode=test \
  --data_te=../data_dump/yfcc-sift-2000-test.hdf5 \
  --model_path=/path/to/checkpoint_directory \
  --res_path=../outputs/yfcc_sift/test \
  --log_base=../outputs/yfcc_sift \
  --use_ransac=False --gpu_id=0

Availability. The public repository is a code snapshot: data preparation, training and evaluation are included, but the model is not yet fully synchronized with the revised manuscript, and no pretrained checkpoint is released. The numbers on this page are the ones reported in the paper.

Cite

BibTeX

wang2026sfmambanet
@misc{wang2026sfmambanet,
  title         = {Smooth Inliers, Erratic Outliers: Frequency-Aware Mamba for Two-View Correspondence Learning},
  author        = {Wang, Zhihua and Gao, Yifeng and Feng, Yunhao and Pan, Rui and Ma, Zhouyuan and
                   Zhao, Baiheng and Chen, Zixing and Li, Yanping and Liu, Yizhang and Ma, Xingjun},
  year          = {2026},
  eprint        = {2606.04493},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CV},
  url           = {https://arxiv.org/abs/2606.04493}
}

Frequency is a natural language
for agreement between matches.