Fw: K3 problem with logging program after download upgrade

[Ken Kopp]

Could this account for the "Intel Hex file ran out of bytes before Intel
End file record was read" -and- "Intel Hex Read error: Expected start
characher ":" not found" messages when attempting (unsuccessfully)
to load the new upgrade into my K3?

73! Ken Kopp - K0PP
[hidden email]



----- Original Message -----
From: Don Rasmussen
To: [hidden email]
Sent: Tuesday, 04 December, 2007 05:40
Subject: [Elecraft] K3 problem with logging program after download
upgrade


Hi, Buck.

The K3 Utility attempts to communicate with the MCU initially to
establish its baud rate.  This is done by sending a small command
at each of the supported speeds. <snip>

Soapbox ON.

IMO - This is a tragic WASTE of BITS.

And this happens every time there is an update?

Calculate the number of updates over the life of the
radio, now how big is the bit bucket - remembering
ASCII is 7 bits per character plus parity plus start
bits and stop bit.

Could be some major overflow problems when folks least
expect it...


_______________________________________________
Elecraft mailing list
Post to: [hidden email]
You must be a subscriber to post to the list.
Subscriber Info (Addr. Change, sub, unsub etc.):
 http://mailman.qth.net/mailman/listinfo/elecraft   

Help: http://mailman.qth.net/subscribers.htm
Elecraft web page: http://www.elecraft.com

[Dick Dievendorff]

No, that sounds like you got a bad file from the Elecraft file server.  I'd
recommend erasing the hmcu*.hex file and re-do the "Copy files from
elecraft".  If you open the hex file in Notepad, every line should start
with a colon.

73 de Dick, K6KR


-----Original Message-----
From: [hidden email]
[mailto:[hidden email]] On Behalf Of Ken Kopp
Sent: Monday, December 03, 2007 9:52 PM
To: [hidden email]
Subject: Fw: [Elecraft] K3 problem with logging program after download
upgrade

Could this account for the "Intel Hex file ran out of bytes before Intel
End file record was read" -and- "Intel Hex Read error: Expected start
characher ":" not found" messages when attempting (unsuccessfully)
to load the new upgrade into my K3?

73! Ken Kopp - K0PP
[hidden email]



----- Original Message -----
From: Don Rasmussen
To: [hidden email]
Sent: Tuesday, 04 December, 2007 05:40
Subject: [Elecraft] K3 problem with logging program after download
upgrade


Hi, Buck.

The K3 Utility attempts to communicate with the MCU initially to
establish its baud rate.  This is done by sending a small command
at each of the supported speeds. <snip>

Soapbox ON.

IMO - This is a tragic WASTE of BITS.

And this happens every time there is an update?

Calculate the number of updates over the life of the
radio, now how big is the bit bucket - remembering
ASCII is 7 bits per character plus parity plus start
bits and stop bit.

Could be some major overflow problems when folks least
expect it...


_______________________________________________
Elecraft mailing list
Post to: [hidden email]
You must be a subscriber to post to the list.
Subscriber Info (Addr. Change, sub, unsub etc.):
 http://mailman.qth.net/mailman/listinfo/elecraft   

Help: http://mailman.qth.net/subscribers.htm
Elecraft web page: http://www.elecraft.com

_______________________________________________
Elecraft mailing list
Post to: [hidden email]
You must be a subscriber to post to the list.
Subscriber Info (Addr. Change, sub, unsub etc.):
 http://mailman.qth.net/mailman/listinfo/elecraft   

Help: http://mailman.qth.net/subscribers.htm
Elecraft web page: http://www.elecraft.com

[Julian, G4ILO]

On Dec 4, 2007 6:31 AM, Dick Dievendorff <[hidden email]> wrote:
> No, that sounds like you got a bad file from the Elecraft file server.  I'd
> recommend erasing the hmcu*.hex file and re-do the "Copy files from
> elecraft".  If you open the hex file in Notepad, every line should start
> with a colon.
>

Surely the uploader should do some kind of integrity check, instead of
attempting to upload a duff file to the radio? It's not exactly rocket
science. There's a thing called MD5 that's very good for this...

--
Julian, G4ILO K2 s/n: 392  K3 s/n: ???
G4ILO's Shack: www.g4ilo.com
Zerobeat Ham Forums: www.zerobeat.net/smf
_______________________________________________
Elecraft mailing list
Post to: [hidden email]
You must be a subscriber to post to the list.
Subscriber Info (Addr. Change, sub, unsub etc.):
 http://mailman.qth.net/mailman/listinfo/elecraft   

Help: http://mailman.qth.net/subscribers.htm
Elecraft web page: http://www.elecraft.com

[Dan Barker]

md5sum anyone?

Dan / WG4S / K2 #2456

-----Original Message-----
From: [hidden email]
[mailto:[hidden email]] On Behalf Of Dick Dievendorff
Sent: Tuesday, December 04, 2007 1:31 AM
To: 'Ken Kopp'; [hidden email]
Subject: RE: [Elecraft] K3 problem with logging program after download
upgrade

No, that sounds like you got a bad file from the Elecraft file server.  I'd
recommend erasing the hmcu*.hex file and re-do the "Copy files from
elecraft".  If you open the hex file in Notepad, every line should start
with a colon.

73 de Dick, K6KR
<snip />

_______________________________________________
Elecraft mailing list
Post to: [hidden email]
You must be a subscriber to post to the list.
Subscriber Info (Addr. Change, sub, unsub etc.):
 http://mailman.qth.net/mailman/listinfo/elecraft   

Help: http://mailman.qth.net/subscribers.htm
Elecraft web page: http://www.elecraft.com

[Bob Cunnings NW8L]

Well,  From the error message I infer that the file is in Intel Hex
format. This format contains a built in integrity check, namely a
checksum for every line. So it seems the uploader did indeed perform
an integrity check - and detected a problem. Relax, these guys *do*
know what they are doing!

For years, at work, I've been using this format for deployment of
firmware images used to update products in the field, it's plenty
robust, nothing else is required, insofar as verifying the file read
in from disk.

Bob NW8L

On Dec 4, 2007 2:24 AM, Julian G4ILO <[hidden email]> wrote:
>
> Surely the uploader should do some kind of integrity check, instead of
> attempting to upload a duff file to the radio? It's not exactly rocket
> science. There's a thing called MD5 that's very good for this...
_______________________________________________
Elecraft mailing list
Post to: [hidden email]
You must be a subscriber to post to the list.
Subscriber Info (Addr. Change, sub, unsub etc.):
 http://mailman.qth.net/mailman/listinfo/elecraft   

Help: http://mailman.qth.net/subscribers.htm
Elecraft web page: http://www.elecraft.com

[Dick Dievendorff]

Thanks, Bob.  But I didn't detect the corrupt file problem until after the
K3 Utility had started to load the MCU.  I still think I might need to pass
the file twice, once to verify that it's complete and checksums correctly,
and then again to actually load the firmware into the radio.  

Dick, K6KR
 

-----Original Message-----
From: [hidden email]
[mailto:[hidden email]] On Behalf Of Bob Cunnings
Sent: Tuesday, December 04, 2007 7:26 AM
To: elecraft
Subject: Re: [Elecraft] K3 problem with logging program after download
upgrade

Well,  From the error message I infer that the file is in Intel Hex
format. This format contains a built in integrity check, namely a
checksum for every line. So it seems the uploader did indeed perform
an integrity check - and detected a problem. Relax, these guys *do*
know what they are doing!

For years, at work, I've been using this format for deployment of
firmware images used to update products in the field, it's plenty
robust, nothing else is required, insofar as verifying the file read
in from disk.

Bob NW8L

On Dec 4, 2007 2:24 AM, Julian G4ILO <[hidden email]> wrote:
>
> Surely the uploader should do some kind of integrity check, instead of
> attempting to upload a duff file to the radio? It's not exactly rocket
> science. There's a thing called MD5 that's very good for this...
_______________________________________________
Elecraft mailing list
Post to: [hidden email]
You must be a subscriber to post to the list.
Subscriber Info (Addr. Change, sub, unsub etc.):
 http://mailman.qth.net/mailman/listinfo/elecraft   

Help: http://mailman.qth.net/subscribers.htm
Elecraft web page: http://www.elecraft.com

_______________________________________________
Elecraft mailing list
Post to: [hidden email]
You must be a subscriber to post to the list.
Subscriber Info (Addr. Change, sub, unsub etc.):
 http://mailman.qth.net/mailman/listinfo/elecraft   

Help: http://mailman.qth.net/subscribers.htm
Elecraft web page: http://www.elecraft.com

[dave.wilburn]

On the note of bad files, are hash checks performed on the downloaded
files?
-  

David Wilburn
[hidden email]
K4DGW
K2 S/N 5982


On Mon, 2007-12-03 at 22:31 -0800, Dick Dievendorff wrote:

> No, that sounds like you got a bad file from the Elecraft file server.  I'd
> recommend erasing the hmcu*.hex file and re-do the "Copy files from
> elecraft".  If you open the hex file in Notepad, every line should start
> with a colon.
>
> 73 de Dick, K6KR
>
>
> -----Original Message-----
> From: [hidden email]
> [mailto:[hidden email]] On Behalf Of Ken Kopp
> Sent: Monday, December 03, 2007 9:52 PM
> To: [hidden email]
> Subject: Fw: [Elecraft] K3 problem with logging program after download
> upgrade
>
> Could this account for the "Intel Hex file ran out of bytes before Intel
> End file record was read" -and- "Intel Hex Read error: Expected start
> characher ":" not found" messages when attempting (unsuccessfully)
> to load the new upgrade into my K3?
>
> 73! Ken Kopp - K0PP
> [hidden email]
>
>
>
> ----- Original Message -----
> From: Don Rasmussen
> To: [hidden email]
> Sent: Tuesday, 04 December, 2007 05:40
> Subject: [Elecraft] K3 problem with logging program after download
> upgrade
>
>
> Hi, Buck.
>
> The K3 Utility attempts to communicate with the MCU initially to
> establish its baud rate.  This is done by sending a small command
> at each of the supported speeds. <snip>
>
> Soapbox ON.
>
> IMO - This is a tragic WASTE of BITS.
>
> And this happens every time there is an update?
>
> Calculate the number of updates over the life of the
> radio, now how big is the bit bucket - remembering
> ASCII is 7 bits per character plus parity plus start
> bits and stop bit.
>
> Could be some major overflow problems when folks least
> expect it...
>
>
> _______________________________________________
> Elecraft mailing list
> Post to: [hidden email]
> You must be a subscriber to post to the list.
> Subscriber Info (Addr. Change, sub, unsub etc.):
>  http://mailman.qth.net/mailman/listinfo/elecraft   
>
> Help: http://mailman.qth.net/subscribers.htm
> Elecraft web page: http://www.elecraft.com
>
> _______________________________________________
> Elecraft mailing list
> Post to: [hidden email]
> You must be a subscriber to post to the list.
> Subscriber Info (Addr. Change, sub, unsub etc.):
>  http://mailman.qth.net/mailman/listinfo/elecraft   
>
> Help: http://mailman.qth.net/subscribers.htm
> Elecraft web page: http://www.elecraft.com

_______________________________________________
Elecraft mailing list
Post to: [hidden email]
You must be a subscriber to post to the list.
Subscriber Info (Addr. Change, sub, unsub etc.):
 http://mailman.qth.net/mailman/listinfo/elecraft   

Help: http://mailman.qth.net/subscribers.htm
Elecraft web page: http://www.elecraft.com

[David Woolley (E.L)]

Julian G4ILO wrote:

> Surely the uploader should do some kind of integrity check, instead of
> attempting to upload a duff file to the radio? It's not exactly rocket
> science. There's a thing called MD5 that's very good for this...

MD5 is overkill.  It's intended to detect deliberate tampering, although
it's also useful as a fast way of comparing multiple files, as there is
a high level of confidence that files with the same MD5 digest have the
same content, given the presumption that they might.  It's origins are
in encryption, rather than in error detection.  CRC codes are faster to
compute and give good protection against random errors.

In general, all the high risk steps in getting the firmware from
Elecraft to the K3 are protected by error detection codes already, with
the exception of the final hop, over the serial link.  Typically the hex
file is going to be processed incrementally as it arrives over that
link, so most of the protection at that stage has to by short check
codes that can be applied frequently, not a long one for the whole file.

The other high risk area is probably the PC memory, if it doesn't have
parity, which most, for cost, and speed reasons, doesn't.  However, if
the PC memory unreliable, you cannot trust that error detection code is
being run correctly, or that errors won't be introduced after the check.
     Similarly for CPU faults, although these should be less likely for
a system running within specification.

Incidentally, there are known problems in MD5's tamper detection
credentials and best current practice, at least in the open literature,
is to use both MD5 and another cryptographic checksum algorithm called
SHA (secure hash algorithm).
>


--
David Woolley
Emails are not formal business letters, whatever businesses may want.
RFC1855 says there should be an address here, but, in a world of spam,
that is no longer good advice, as archive address hiding may not work.
_______________________________________________
Elecraft mailing list
Post to: [hidden email]
You must be a subscriber to post to the list.
Subscriber Info (Addr. Change, sub, unsub etc.):
 http://mailman.qth.net/mailman/listinfo/elecraft   

Help: http://mailman.qth.net/subscribers.htm
Elecraft web page: http://www.elecraft.com

[Dick Dievendorff]

Julian makes a good point, but we're not going to use MD5.

The intel hex format contains sufficient internal integrity checks that we
can very likely tell that it has been corrupted. It has a line-by-line star
character, a per-line checksum byte, and it has a definite end of file
record.  The sorts of corruption that we'll see in file transfer operations
will most likely be caught by the existing verification.

However the K3 Utility wasn't verifying the file until the MCU load had
started.  This is inconvenient, because the error recovery then consists of
erasing the corrupt file from your PC, re-doing the FTP, and trying it
again.  If the FTP operation is at some distance from the radio, it becomes
more inconvenient.

The current DSP files are in a binary format, and the load process depended
on a post-load internal verification, plus a command query to see that the
processor was healthy enough to respond to a few simple commands.

Based on K0PP's experience, the next revision of the K3 Utility (being
evaluated by Field Test now) will verify Intel Hex files right after FTP
copy, repeating the file copy if necessary. It will be verified again by a
pre-pass before beginning the MCU load.  

The loader contains changes that permit the DSPs to be loaded in either
binary or intel hex format.  Subsequent revisions of DSP firmware will be
released in intel hex format to permit this verification.

The DSP firmware files are larger (> 2 times larger), but the load time to
the K3 isn't increased dramatically.  It will be slightly slower.

Use of MD5 would have required us to maintain a side file of MD5 hashes or
create a new variant of the intel hex file format with an internal digest.
I suggested that keeping the side files in sync with the distribution of the
firmware would be a continuing distribution problem.  I did use an internal
digest technique (with a CRC-32) for the configuration file save format, but
that format was one of my creation.  intel hex files are processed by a fair
number of existing tools, not all of which are easily modified.
 

73 de Dick, K6KR


-----Original Message-----
From: [hidden email]
[mailto:[hidden email]] On Behalf Of David Woolley
Sent: Friday, December 07, 2007 12:40 AM
To: elecraft
Subject: Re: [Elecraft] K3 problem with logging program after download
upgrade

Julian G4ILO wrote:

> Surely the uploader should do some kind of integrity check, instead of
> attempting to upload a duff file to the radio? It's not exactly rocket
> science. There's a thing called MD5 that's very good for this...

MD5 is overkill.  It's intended to detect deliberate tampering, although
it's also useful as a fast way of comparing multiple files, as there is
a high level of confidence that files with the same MD5 digest have the
same content, given the presumption that they might.  It's origins are
in encryption, rather than in error detection.  CRC codes are faster to
compute and give good protection against random errors.

In general, all the high risk steps in getting the firmware from
Elecraft to the K3 are protected by error detection codes already, with
the exception of the final hop, over the serial link.  Typically the hex
file is going to be processed incrementally as it arrives over that
link, so most of the protection at that stage has to by short check
codes that can be applied frequently, not a long one for the whole file.

The other high risk area is probably the PC memory, if it doesn't have
parity, which most, for cost, and speed reasons, doesn't.  However, if
the PC memory unreliable, you cannot trust that error detection code is
being run correctly, or that errors won't be introduced after the check.
     Similarly for CPU faults, although these should be less likely for
a system running within specification.

Incidentally, there are known problems in MD5's tamper detection
credentials and best current practice, at least in the open literature,
is to use both MD5 and another cryptographic checksum algorithm called
SHA (secure hash algorithm).
>


--
David Woolley
Emails are not formal business letters, whatever businesses may want.
RFC1855 says there should be an address here, but, in a world of spam,
that is no longer good advice, as archive address hiding may not work.
_______________________________________________
Elecraft mailing list
Post to: [hidden email]
You must be a subscriber to post to the list.
Subscriber Info (Addr. Change, sub, unsub etc.):
 http://mailman.qth.net/mailman/listinfo/elecraft   

Help: http://mailman.qth.net/subscribers.htm
Elecraft web page: http://www.elecraft.com

_______________________________________________
Elecraft mailing list
Post to: [hidden email]
You must be a subscriber to post to the list.
Subscriber Info (Addr. Change, sub, unsub etc.):
 http://mailman.qth.net/mailman/listinfo/elecraft   

Help: http://mailman.qth.net/subscribers.htm
Elecraft web page: http://www.elecraft.com