When using the nanoVNA at frequencies above 300MHz it switches to harmonics mode
First attached picture shows the harmonics till 1600MHz of the SI5351 output when set at 100MHz.
The 9th harmonic is at 900MHz only 34dB below the fundamental at 100MHz.
To validate the performance of the SA612 I soldered two wires to output of the REF mixer and observed the output using a balanced USB audio input
Second picture shows the 8k audio FFT when the nanoVNA is set to 1MHz. About 100dB SNR at this small audio BW.
The nanoVNA uses less points (48) in its calculations and has therefore substantial less SNR
The third picture shows the output when set to 2GHz. The SNR is reduced to something between 20 and 30dB
This 80dB signal reduction is much larger as the reduction of power in the harmonics but can probably be attributed to the mixing loss in the SA612 as it it operated way out of spec (max 500MHz)
The fourth picture shows the roughly 50dB SNR you get when you operate the nanoVNA at 2GHz from a clean 5V supply. A very welcome improvement making S21 measurements possible.
A quick calculation based on the schematic suggests the REF mixer has the highest signal levels while the TRAN mixer is 10dB lower and REFL is 20dB lower
If I have enough courage I may change the attenuators for the REFL and TRAN mixer to get more balanced power levels.