Sherwood on ARRL Testing Methodology

[Bill W4ZV]

WA6HHQ:

>I discussed the issue of the missing IMDDR3 numbers (Third Order Dynamic
Range) with Michael Tracy at the ARRL a couple of weeks ago. I expressed
my strong concern that they had dropped these from the review data and
were only posting a footnote showing how to calculate them from the IMD
level and MDS. I pointed out that we, and many others, use the IMDDR3
numbers as the primary IMD Dynamic range comparison between rigs and
that the IP3 numbers were not as useful for receiver comparisons, since
they can artificially be inflated by turning on the attenuator, or
making a receiver have low sensitivity (deaf).

    I agree completely.  It was very surprising they omitted the IMD
numbers from the tables yet included them in the bar graphs at the
very front of the article.

>Also, the multi-level IP3 numbers were in direct response to requests to
the ARRL from several members of the advisory group that Rob mentions
below.

    I know W8JI who is on the group believes IP3 is simply
a subject of much confusion and abuse.  My guess is that
one of the European VHF guys in the group lobbied for this.  
But I agree it's mostly a meaningless theoretical number
better represented by measuring MDS and IMDDR3 separately.

>Michael Tracey and the ARRL test lab are above reproach in my opinion.

    I agree completely but I wish more of his objectivity were
shared by some of the folks writing the QST reviews.  I feel
some of Sherwood's criticisms are completely justified, but
this is of course not Michael's responsibility.  Rob also has
his own axe to grind at times so you have to take some
of what he says with a grain of salt.  I had an Orion for 4
years and *never* heard the AGC hang artifact even though I
spend 90% of my time on 160 meters where there are plenty
of BIG QRN crashes.  Rob can also go a bit overboard into
conspiracy theories about QST advertisers.  I don't buy that.

    It sounds like ARRL responded appropriately.  I also
found it humorous that W1ZR in the very same issue was
describing IMDDR3 and its importance yet it was omitted
in the FT-2000 tabes!  :-)  None of us is perfect.

    On the plus side, I commend ARRL for gradually
moving toward more realistic testing...such as posting
2 kHz spaced measurements in the QST articles rather
than simply burying the data in an obscure graph in
the Expanded Test Reports.  Same for TX signal purity
(keyclicks and phase noise).  They're getting better but
IMHO they need to tone down the fluffiness in the QST
articles.  My personal favorite is G3SJX in RSGB's Radcom.
It would almost be worth the cost of joining RSGB just to get
his reviews.

    BTW Sherwood measured the SDR-5000's IMDDR3 at
5 kHz as 96 dB.  It's going to be a real horse race to see if
the K3 may top that but if not it should be very close.

                                    73,  Bill  W4ZV
_______________________________________________
Elecraft mailing list
Post to: [hidden email]
You must be a subscriber to post to the list.
Subscriber Info (Addr. Change, sub, unsub etc.):
 http://mailman.qth.net/mailman/listinfo/elecraft   

Help: http://mailman.qth.net/subscribers.htm
Elecraft web page: http://www.elecraft.com

[Bill VanAlstyne W5WVO]

Bill Tippett wrote:

> (keyclicks and phase noise).  They're getting better but
> IMHO they need to tone down the fluffiness in the QST
> articles.

I agree. I find the articles themselves to be mostly non-critical rah-rah
stuff that the Marketing Department at the radio company could just as well
have written.

> BTW Sherwood measured the SDR-5000's IMDDR3 at
> 5 kHz as 96 dB.  It's going to be a real horse race to see if
> the K3 may top that but if not it should be very close.

I don't know what the test configuration of the SDR-5000 was -- I don't think
it uses any discrete hardware roofing filters, does it? -- but Elecraft is
already claiming that the IMDDR3 at 5 kHz spacing (400 Hz CW roofing filter)
will be greater than 100 dB. (See K3 Specs page) I don't believe Eric would
have let that spec be on a public web page unless he was pretty certain it
would still be true when the final numbers come out.  :-)

Bill / W5WVO


_______________________________________________
Elecraft mailing list
Post to: [hidden email]
You must be a subscriber to post to the list.
Subscriber Info (Addr. Change, sub, unsub etc.):
 http://mailman.qth.net/mailman/listinfo/elecraft   

Help: http://mailman.qth.net/subscribers.htm
Elecraft web page: http://www.elecraft.com

[Tom Hammond-2]

Bill:

I'm assuming you saw Rob Sherwood's recent posting to the FT-2000
reflector. It was recently re-printed to the elecraft reflector but
Bill Tippett, W4ZV.

Here's some comments on that article from Eric Swartz, WA6HHQ, of Elecraft.

73,

Tom
__________________________

Hi Bill,

I discussed the issue of the missing IMDDR3 numbers (Third Order
Dynamic Range) with Michael Tracy at the ARRL a couple of weeks ago.
I expressed my strong concern that they had dropped these from the
review data and were only posting a footnote showing how to calculate
them from the IMD level and MDS. I pointed out that we, and many
others, use the IMDDR3 numbers as the primary IMD Dynamic range
comparison between rigs and that the IP3 numbers were not as useful
for receiver comparisons, since they can artificially be inflated by
turning on the attenuator, or making a receiver have low sensitivity (deaf).

They agreed that they could add the IMDDR3 numbers back in explicitly
in the data table for future reviews. There is no sinister plot here.
:-) They absolutely did NOT remove them at the request of any
manufacturer. Quite the opposite. They were almost unintentionally
removed in the post lab test formatting at the layout level for the
review to save space as the review was rushed to print. After
discussing this with Michael and Joel Hallas, the reviews manager,
they said that they would add them back in for future reviews.

Also, the multi-level IP3 numbers were in direct response to requests
to the ARRL from several members of the advisory group that Rob
mentions below. The primary reason for doing so, as mentioned in the
sidebar of the FT-2000 review, was to show how some radios depart
from the ideal IP3 curve at different levels. Personally, as I
mentioned above, I feel that IP3, while useful for testing stand
alone amplifier stages, is not useful for receiver comparisons. (Rob
also alludes to this in his comments.) The same IMD level test is run
for IMDDR3 and IP3. The IP3 number is just calculated differently
from the exact same IMD data. The best comparison still is the full
IMD Dynamic range, IMDDR3, which can not be as easily manipulated by
artificially changing a receiver's sensitivity, as it can for IP3. I
can make the K2 have a huge IP3 by reducing its front end gain by
another 10 dB. Would anyone want that? No.

The better test is to just take the signal generator level necessary
to create an IMD product equal to the MDS, and compare that generator
level to the MDS for the receiver. The net difference is the IMDDR3.

Michael Tracey and the ARRL test lab are above reproach in my
opinion. They take great pains to try to make accurate and useful
measurements of each rig, and they are ethical to an extreme. Michael
spends an incredible amount of time and care making each set of test
measurements. They do not change any data for big advertisers and
they buy the rigs they test on the open market.  All of the recent
changes were made as a result of input from the test advisory group
that Rob is a member of. I'm sure there is still a lot of room for
discussion and improvement of the tests they run, but the ARRL does a
great service for the amateur community with their tests (as does
Rob). Their ONLY goal with these latest changes is to make the data
even more useful and repeatable between rigs.

73, Eric   WA6HHQ
-----


Bill Tippett wrote:

>         Rob Sherwood gave his permission to post this on the
>FT-2000 list.  Since that is public information I am posting
>it here also.  BTW I agree with his comments on the "new"
>methodology.
>
>                                 73,  Bill  W4ZV
>
>
>What has gone wrong with the ARRL's new Product Reviews in QST?
>
>For several years I participated as part of a group of hams who were
>trying to improve the testing of radios reviewed in QST.  Several
>reviews in the past had included questionable data on receiver
>performance.  A sincere effort was initiated to correct testing
>problems within the lab, and hopefully to also improve the "hands
>on" portion of the report.
>....
>
>What has changed in the ARRL reports?
>
>Some of the changes are of minor interest, like measuring the noise
>figure of an HF radio.  Noise figure is generally used by VHF and
>UHF enthusiasts, but adding these data points certain hurts
>nothing.  Is noise figure, or noise floor, or sensitivity a
>significant issue in today's receivers?  The 75A-4 has an excellent
>noise floor, as reported in the January 2006 QST Annual Vintage
>Issue.  Few of us have such a quiet location that atmospheric and
>galactic noise don't overshadow the noise floor of a modern receiver.
>
>What we did get was additional confusion in the ethereal world of
>third-order intercept (IP3), in place of real dynamic-range
>data.  The League used to measure it one way, then a second way, and
>now three ways.  Is this supposed to be helpful?  The old way
>(measured at the noise floor) was acceptable. The second way
>referenced an imprecise S5, now defined as -97 dBm, and a third new
>way at 0 dBm.  Zero dBm is really strong, something we don't likely
>ever see, unless we are working Field Day or Multi-Multi contests
>from near-by transmitters.
>(I am assuming we are not living in Europe with their 5 megawatt AM
>broadcast transmitters.)
>
>0 dBm is S9 + 73 dB, assuming any S meter reads that level
>accurately.  (The Flex 5000A would actually do that.)  On my IC-781,
>0 dBm reads S9 + 50 with 30 dB of internal attenuation, or something
>like S9 + 80 dB with the attenuators off, if the S meter  would read
>that high, which is does not.  What happens when you put two 0 dBm
>signals into an IC-781 at 20 kHz spacing?  The IMD reads S9 + 18 dB.
>At 2 kHz spacing the IMD reads S9 + 60 dB!  The 781 is not a radio
>with performance problems, so what do these new and improved
>measurements really mean?
>
>If you look at the FT-2000 chart for IP3 at 2 kHz with the preamp
>off, you see the IP3, measured at the noise floor, is -19 dBm.  This
>is not a good number, particularly since a Yaesu radio with "IPO"
>enabled (no preamp) is similar to most other radios with the 10 dB
>attenuator enabled.  Yet if you measure the FT-2000 at 0 dBm, the
>IP3 calculates out to +15 dBm, which sounds good.  This new
>information is meaningless at best, or misleading at its worst.  Why
>is the IP3 so high at 0 dBM?  Because the inter-modulation is so
>strong (S9 + 60 dB) the AGC has basically turned the gain of the radio off.
>
>Most operators will run an FT-2000 with preamp 1 enabled, since it
>gives a reasonable noise floor, sensitivity and AGC threshold. Yet
>no information is available with this typical setting for the newly
>touted IP3 reporting method, which at 2 kHz would be about -30 dBm
>for the League's sample.  (The FT-2000 I measured was considerably
>worse.) To get a meaningful dynamic-range number, the reader now has
>to subtract two numbers.  Why is this important data now missing, or
>at least obfuscated?  Could it be the big advertisers in QST didn't
>like seeing 2 kHz dynamic-range numbers that are typically around 70
>dB? Only the League could take a measured 2 kHz dynamic range of 69
>dB at 2 kHz and calculate it into a +15 dBm intercept at 0
>dBm.  Talk about smoke and mirrors!
>
>Th

_______________________________________________
Elecraft mailing list
Post to: [hidden email]
You must be a subscriber to post to the list.
Subscriber Info (Addr. Change, sub, unsub etc.):
http://mailman.qth.net/mailman/listinfo/elecraft

Help: http://mailman.qth.net/subscribers.htm
Elecraft web page: http://www.elecraft.com

_______________________________________________
Elecraft mailing list
Post to: [hidden email]
You must be a subscriber to post to the list.
Subscriber Info (Addr. Change, sub, unsub etc.):
 http://mailman.qth.net/mailman/listinfo/elecraft   

Help: http://mailman.qth.net/subscribers.htm
Elecraft web page: http://www.elecraft.com

[Goody K3NG]

Another thing to note is that ARRL buys all the gear they review, right
off the shelf just like any other ham would.  So no vendor, advertiser
or otherwise, has an opportunity to submit an individually tuned or
"souped up" unit to get a better review.  ARRL periodically auctions off
review units,  presumably at a loss.  I think this further increases the
validity of ARRL tests and just about dismisses any "sinister plot"
theories.

Tom Hammond wrote:
>
>
> Here's some comments on that article from Eric Swartz, WA6HHQ, of
> Elecraft.
>
Blog: http://thek3ngreport.blogspot.com/

_______________________________________________
Elecraft mailing list
Post to: [hidden email]
You must be a subscriber to post to the list.
Subscriber Info (Addr. Change, sub, unsub etc.):
 http://mailman.qth.net/mailman/listinfo/elecraft   

Help: http://mailman.qth.net/subscribers.htm
Elecraft web page: http://www.elecraft.com

[N2EY]

-----Original Message-----
From: Goody K3NG <[hidden email]>

>Another thing to note is that ARRL buys all the gear they review,
right off the shelf just like any other ham would.

And it's bought in such a way that the seller doesn't know it's going
to be an ARRL test unit.

--

As for "fluff" in ARRL Product Reviews, I attribute that to the wide
'dynamic range' of the readership's technical
knowledge.

On the one end are hams who can recite testing methodologies for all
sorts of performance metrics, including
pitfalls and claimed-vs-observed numbers. At the other end are hams who
don't understand why you'd want
passband tuning, narrow filters, or what the attenuator/preamp switch
does. And everything in between.

On top of that are complaints that QST is "too technical" and/or "aimed
only at the contest/DX/big station hams",
etc., etc.

So we get Product Reviews that are part lab test, part feelings, and
part an attempt to Elmer. All jammed into a
 limited space, and on a limited time, to be done by a limited staff.

The K2 got an expanded report that only appeared online. Went into more
detail than the mag report. K3 should
get the same.

One ham's fluff is another ham's main interest.

73 de Jim, N2EY
________________________________________________________________________
Email and AIM finally together. You've gotta check out free AOL Mail! -
http://mail.aol.com
_______________________________________________
Elecraft mailing list
Post to: [hidden email]
You must be a subscriber to post to the list.
Subscriber Info (Addr. Change, sub, unsub etc.):
 http://mailman.qth.net/mailman/listinfo/elecraft   

Help: http://mailman.qth.net/subscribers.htm
Elecraft web page: http://www.elecraft.com