Check that Instrument Mode is set to Reflection and RF I-V; Transmission mode should not be used with RF I-V mode.
Also check that calibration mode is set to Response Calibration.
These settings should be applied automatically when RF I-V mode is selected but they may have been inadvertently changed.
Also check that the two co-axial cables are not transposed; I once did this when I first used the fixture with my VNWA.
As I recall, the results were incorrect but, surprisingly, "in the ballpark", probably due to the awesome power of calibration.
There was a discussion of the RF I-V fixture some years ago on the old N2PK Yahoo group; Bill Carver was having a problem with his, I think.
To assist, I did some measurements of a 5k1 0805 resistor and obtained results as seen in the attachment; Paul opined that this was very good, the kind of result that might be expected.
Kerry VK2TIL.