Victor,
Let me preface my comments by saying that I have built several test jigs for measuring impedance using the S21 series method and tried several ways to compensate for errors. None have been very successful and I now just use the S11 reflection method which works up to several thousand ohms of complex impedance with a decent test jig.
I believe the NanoVNA-H4 is just not the right tool for the job for several reasons that I have mentioned before....
- the source and termination impedances are not 50 + j0 ohms and vary with frequency
- the current firmware and PC programs only use 6 term error correction not 12 term
Others have found similar issues using the S21 series method using the NanoVNA and even with higher performance HP VNA equipment. Here are some interesting links. The K6JCA experiments find differences similar to what you have measured.
In my experiments I found that parasitic capacitance had considerable affect on the S21 measurements and I tried several different test jigs to see what happened. The issue of where to establish the reference plane took some consideration. S21 Gain and S21 phase determine the complex impedance of the DUT and even with some type of source (Port1) and termination (Port2) correction the phase was often incorrect. Phase measurement was the problem and in my tests was due to an offset delay across the jig. Another issue I found was that for some reason high DUT impedance resulted in common mode current on the surface of the coax cables connecting the NanoVNA to the test jig. I tried snap-on ferrites and the results changed.
You mentioned that you could do a S11 reflection and S21 series measurement at the same time. You will get different results when the DUT is in series between Port 1 and Port2 compared to when it is across Port 1 alone. Best case is that it will be 50 ohms higher. TIn reality the complex impedance measured when in series is the DUT plus the input impedance of the cable attached to Port 2 (which will deviate from 50 ohms due to Port 2 impedance not being 50 ohms) and any parasitic inductance or capacitance between the DUT and the Port 2 cable.
I wish you good luck in your quest and hope you find a method that works for you and that you find a solution that eluded me.
Roger