|
I wanted to share a little information on how I'm going to go about restructuring the T41 code.?
?
The most difficult bugs to track down in the current code have been those caused by the radio being in an unknown state -- some line that controlled a switch on the RF board or a parameter in the DSP chain was inserted or omitted somewhere in the code. Because all variables are global, this could be literally anywhere in the nearly 30,000 lines of code. Finding the source of the bug without the ability to use a debugger is a challenge. It also possible to insert an entirely valid line of code that places the radio in an invalid or undesirable state -- for instance, imagine turning the CAL switch on when trying to transmit SSB.
?
I'm going to solve this problem by using state machines. In the new code structure, the state of the radio hardware will be controlled entirely by a radio mode state machine. This state machine is the only place in the code where the hardware state is changed. Using a state machine to control the hardware ensures that all hardware is always in a known configuration state. I'm working my way through the hardware boards, defining the valid configuration states for that hardware. I will then write functions that put the boards into each valid configuration state. The only way the rest of the code can affect the radio hardware is by calling one of those functions, and the only part of the code that will call those functions is the radio mode state machine. Here's an example of some of the states for the RF board (does not include the calibration states).
?
?
?
State machines can be written entirely in C code, but it¡¯s easier to understand how the state machine operates through a visual diagram. I've found an open-source solution, , that allows you to draw the state machines in a graphical environment (I'm using draw.io) and then automatically generate the C code that implements the state machine. Here's the state machine I've created for the radio hardware.
?
?
?
I'm working on a separate state machine to control the state of the graphical display.
?
Here's how I imagine them all working together:
?
?
The software runs in a loop as shown in the diagram above. It performs three major functions:
- Handle interrupt events: if an interrupt was registered by, for example, a button being pressed, then pass the appropriate event on to the state machines to change the hardware and UI states.
- Perform the appropriate signal processing, based on the current radio mode.
- Update the display, based on the current UI state.
?
Then go back to step 1 and repeat. This loop should take at most 10ms to execute in order to avoid buffer overflows in the IQ buffers.
?
I'm building this up from a blank canvas. I've spent most of my effort over the last few weeks learning how to use StateSmith and how to build an automated test environment using . I've now started to write code, starting with the signal processing, and writing unit tests for the code as I go. I've found the unit tests to be very helpful. If I decide to change a function's prototype or modify what the function does, I'll immediately know if that change breaks something somewhere else in the code. It should make the process of getting the code running on the Teensy much faster.
?
I'm going to clean up the code as I move it over, piece by piece, to this new structure -- remove dead code, apply consistent style formatting, minimize the use of global variables, and write tests for every function so we know what it's supposed to do and when it stops doing this.
?
?
?
?
?
|
Thanks Oliver. This is a huge effort! And, very exciting! dave,n3ds
toggle quoted message
Show quoted text
On Apr 24, 2025, at 8:52?PM, Oliver KI3P via groups.io <oliver@...> wrote:
? I wanted to share a little information on how I'm going to go about restructuring the T41 code.?
?
The most difficult bugs to track down in the current code have been those caused by the radio being in an unknown state -- some line that controlled a switch on the RF board or a parameter in the DSP chain was inserted or omitted somewhere in the code. Because all variables are global, this could be literally anywhere in the nearly 30,000 lines of code. Finding the source of the bug without the ability to use a debugger is a challenge. It also possible to insert an entirely valid line of code that places the radio in an invalid or undesirable state -- for instance, imagine turning the CAL switch on when trying to transmit SSB.
?
I'm going to solve this problem by using state machines. In the new code structure, the state of the radio hardware will be controlled entirely by a radio mode state machine. This state machine is the only place in the code where the hardware state is changed. Using a state machine to control the hardware ensures that all hardware is always in a known configuration state. I'm working my way through the hardware boards, defining the valid configuration states for that hardware. I will then write functions that put the boards into each valid configuration state. The only way the rest of the code can affect the radio hardware is by calling one of those functions, and the only part of the code that will call those functions is the radio mode state machine. Here's an example of some of the states for the RF board (does not include the calibration states).
?
?
?
State machines can be written entirely in C code, but it¡¯s easier to understand how the state machine operates through a visual diagram. I've found an open-source solution, , that allows you to draw the state machines in a graphical environment (I'm using draw.io) and then automatically generate the C code that implements the state machine. Here's the state machine I've created for the radio hardware.
?
?
?
I'm working on a separate state machine to control the state of the graphical display.
?
Here's how I imagine them all working together:
?
?
The software runs in a loop as shown in the diagram above. It performs three major functions:
- Handle interrupt events: if an interrupt was registered by, for example, a button being pressed, then pass the appropriate event on to the state machines to change the hardware and UI states.
- Perform the appropriate signal processing, based on the current radio mode.
- Update the display, based on the current UI state.
?
Then go back to step 1 and repeat. This loop should take at most 10ms to execute in order to avoid buffer overflows in the IQ buffers.
?
I'm building this up from a blank canvas. I've spent most of my effort over the last few weeks learning how to use StateSmith and how to build an automated test environment using . I've now started to write code, starting with the signal processing, and writing unit tests for the code as I go. I've found the unit tests to be very helpful. If I decide to change a function's prototype or modify what the function does, I'll immediately know if that change breaks something somewhere else in the code. It should make the process of getting the code running on the Teensy much faster.
?
I'm going to clean up the code as I move it over, piece by piece, to this new structure -- remove dead code, apply consistent style formatting, minimize the use of global variables, and write tests for every function so we know what it's supposed to do and when it stops doing this.
?
?
?
?
?
|
Hi Oliver-
?
One thing I recommend is writing a script to "de-globalize" the code.? I did this and removed hundreds of globals.? There were even globalized local variables!
What the script does is take a variable from SDT.h, and then does a search on all of the other files (I dont' think the .ino file was included).? Any variable which
shows up only once is a candidate for de-globalization.? The first step is to pull that extern variable out of SDT.h and make it a file scope variable in the file where it belongs.
Once that process is complete, then copy the file scope variables into a header file and make a class.? Not everything is a class yet, but the file scope variables are mostly
sorted to their home files.
?
The script will get a bunch of them.? But I didn't refine it enough to get all of them.? The rest have to be handled one at a time, manually.? It's tedious.
To get an idea of how much progress I have made, the number of lines in T41EEE.9/SDT.h = 1057.? SDTVer066-9/SDT.h = 2640.
There is still a huge amount of work to be done.? But a few variables will have to remain global.
?
The other thing I have done is to "modalize" the code.? I've used scoped enums to great effect.? This was a huge change in T41EEE.9.? I wasn't sure how it would work out,
but it is working extremely well and it's a clear way forward to more robust code.? There are still some things not completely modalized, but I think the next version will get it there.
A lot of things have been working in a state machine pattern for the last several versions.? It's less than perfect, but the modalization was a big step forward in improving that.
?
StateSmith looks pretty cool.? I bought a big artist's pad at Walmart and draw everything out with a pen.
?
--
73 Greg KF5N
|
Greg:
Curious about the line:
To get an idea of how much progress I have made, the number of lines in T41EEE.9/SDT.h = 1057.? SDTVer066-9/SDT.h = 2640.
What was the impact if you add up the class header file lines. How much net change was there?
Jack, W8TEE
On Thursday, April 24, 2025 at 10:10:13 PM EDT, Greg KF5N via groups.io <greg.electricity@...> wrote:
Hi Oliver-
?
One thing I recommend is writing a script to "de-globalize" the code.? I did this and removed hundreds of globals.? There were even globalized local variables!
What the script does is take a variable from SDT.h, and then does a search on all of the other files (I dont' think the .ino file was included).? Any variable which
shows up only once is a candidate for de-globalization.? The first step is to pull that extern variable out of SDT.h and make it a file scope variable in the file where it belongs.
Once that process is complete, then copy the file scope variables into a header file and make a class.? Not everything is a class yet, but the file scope variables are mostly
sorted to their home files.
?
The script will get a bunch of them.? But I didn't refine it enough to get all of them.? The rest have to be handled one at a time, manually.? It's tedious.
To get an idea of how much progress I have made, the number of lines in T41EEE.9/SDT.h = 1057.? SDTVer066-9/SDT.h = 2640.
There is still a huge amount of work to be done.? But a few variables will have to remain global.
?
The other thing I have done is to "modalize" the code.? I've used scoped enums to great effect.? This was a huge change in T41EEE.9.? I wasn't sure how it would work out,
but it is working extremely well and it's a clear way forward to more robust code.? There are still some things not completely modalized, but I think the next version will get it there.
A lot of things have been working in a state machine pattern for the last several versions.? It's less than perfect, but the modalization was a big step forward in improving that.
?
StateSmith looks pretty cool.? I bought a big artist's pad at Walmart and draw everything out with a pen.
?
--
73 Greg KF5N
-- Jack, W8TEE
|
Hi Jack-
?
It's a metric of the reduction of global variables.? Note that the .ino file was also greatly de-cluttered in concert with SDT.h.
There were also a bunch of variables which only existed in the SDT.h file.? I suspect they were remnants of earlier functions, tests, and experiments which were never cleaned up.
?
It was a large clean-up job, but the result was worth it.? The code is much easier to work with and debug.? Not really different in effect than that first step you Al took to break the
SDR Convolution code from a single to multiple files.
?
--
73 Greg KF5N
|
Oliver - Thanks for update.? Your diagrams are very helpful.
?
On Thu, Apr 24, 2025 at 07:10 PM, Greg KF5N wrote:
The rest have to be handled one at a time, manually.? It's tedious.
I took this approach.? Yes, it's tedious.? I've gone pretty far with my approach.? My SDT.h is only 209 lines.? This doesn't reflect a massive reduction in global variables but more of a redistribution into the modules primarily related to the variable.? Often these could be made local to specific functions or global to just that module.? When needed outside that module I use module specific header file with the required variable declared as external.? These header files also contain that module's function prototype declarations needed by other modules.? This basically doubles the number of files but makes it easier to find stuff.
|
Great work Terrance!? That is what I aspire to.? It's going to take a few more iterations to get T41EEE that good.
I wholeheartedly agree it makes it easier to find things specific to a particular task.? Debugging is much easier!
?
--
73 Greg KF5N
|
Mornin' Greg:
Yes, Al and I found a lot of "dead" variables when we started out. We could tell by their names that some of them were clearly used as temporary debug variables that just didn't get cleared out. We also found a bunch of "dead" functions that were never called anywhere. We tried to remove all of those, but I know some were not removed. At one time, I thought some of these might be resued later on, but ended up not being used. I just never when back to clean them out.
"De-globalizing" the code is a little harder when the variable is used in multiple unrelated functions. In those cases, the variable could be defined in the ino file, but passed to the other functions (as a pointer?) as needed. That does de-clutter things, but is a pain to do. However, if you can isolate the variable to only the file that uses it, the problem is more manageable. Example: If a variable is only used in the display.cpp source file, it can be localized to that file. It sounds like that's what you've done: create a header file for each "functional" cpp file. The reason I didn't do this is because, as you pointed out, it means each cpp file has its own header file, which has the effect of doubling the number of file which is its own form of clutter, albeit it is organized clutter.?
For such global variables that are only used in one cpp file, instead of placing them their own header file (e.g., display.h which ties to display.cpp), you could place those "local globals" in the cpp file, but use the static storage specifier for them. This keyword is more of an attention-gettter to the programmer than a difference in the variable's placement in memory. I say that because, if you define the variable in the display.cpp file outside of a function without the static specifier, it has global scope, but only for the file in which it is defined. If that variable is not called in any other cpp file, you won't get the "undefined" error message. The only thing the static keyword does is allow you to use that same variable name in multiple cpp files and still not generate a "multiply defined" error message. (I don't know why you'd want to do that, but you could!) The only reason I would add the static keyword in this case is to serve as a sentinel to tell other programmers that this is a "local global".
Jack, W8TEE
On Thursday, April 24, 2025 at 11:26:11 PM EDT, Greg KF5N via groups.io <greg.electricity@...> wrote:
Hi Jack-
?
It's a metric of the reduction of global variables.? Note that the .ino file was also greatly de-cluttered in concert with SDT.h.
There were also a bunch of variables which only existed in the SDT.h file.? I suspect they were remnants of earlier functions, tests, and experiments which were never cleaned up.
?
It was a large clean-up job, but the result was worth it.? The code is much easier to work with and debug.? Not really different in effect than that first step you Al took to break the
SDR Convolution code from a single to multiple files.
?
--
73 Greg KF5N
-- Jack, W8TEE
|
Yes, Jack, you've got it!? There is some nuance when someone uses the term "global", and what I was trying to flush out was the "super globals" versus "local file globals".? When I see "extern" that is a pretty good hint.
So what I've been doing, and I think Oliver is doing something similar, is isolating the "local file globals" first.? So that's what I do, is to put them in the existing .cpp file, in their home turf.
At some point it makes sense to re-factor into a class.? That is when a unique new header file is added.? The "local file globals" are scooped up and dropped into that new header file.
Externs are removed from the SDT.h file, and the same goes for function declarations which can be "delisted" from the SDT.h.
?
The one exceptional header file is the T41EEE's AudioSignal.h file.? This is a super-important and special header file which contains the Teensy Audio and Open Audio related code.? This header is included into the T41EEE.ino file.
?
--
73 Greg KF5N
|
Yep...we're on the same page.
Jack, W8TEE
On Friday, April 25, 2025 at 10:26:51 AM EDT, Greg KF5N via groups.io <greg.electricity@...> wrote:
Yes, Jack, you've got it!? There is some nuance when someone uses the term "global", and what I was trying to flush out was the "super globals" versus "local file globals".? When I see "extern" that is a pretty good hint.
So what I've been doing, and I think Oliver is doing something similar, is isolating the "local file globals" first.? So that's what I do, is to put them in the existing .cpp file, in their home turf.
At some point it makes sense to re-factor into a class.? That is when a unique new header file is added.? The "local file globals" are scooped up and dropped into that new header file.
Externs are removed from the SDT.h file, and the same goes for function declarations which can be "delisted" from the SDT.h.
?
The one exceptional header file is the T41EEE's AudioSignal.h file.? This is a super-important and special header file which contains the Teensy Audio and Open Audio related code.? This header is inserted into the T41EEE.ino file.
?
--
73 Greg KF5N
-- Jack, W8TEE
|
On Fri, Apr 25, 2025 at 07:26 AM, Greg KF5N wrote:
The one exceptional header file is the T41EEE's AudioSignal.h file.? This is a super-important and special header file which contains the Teensy Audio and Open Audio related code.? This header is included into the T41EEE.ino file.
There isn't anything particularly special about this code.? It can be placed in its own cpp file along with a header file for the global externs and function prototypes.? That's what I've done.
|
I'm using a different meaning of the word "special".? The code in the AudioSignal.h file is the software heart and lungs of the radio.
?
--
73 Greg KF5N
|
No question that configuring the audio state is an important function.? I've used this method before, usually for things I didn't want to convert into a library or when linker support was deficient or non-existent.
?
The code base has gotten so big that what you've done does make the code more accessible and readable and that is helpful.? But I don't see the need to treat this differently than other modules, other than it's more work to get all of the global variables handled correctly.? Your method has the advantage of not having to bother with that.? I have AudioConfig.h referenced in 8 modules.
|
This is fantastic Oliver! I am a retired EE. I have limited knowledge or experience with software and programming code, but I know all about state machines and I am wanting to learn C programming with the aid of T41-EP. I am particularly interested in the visual diagram because this will help tremendously when I come to design and implement my own functions and button mapping into the code.
Keep up the good work!
?
John, G0SDF
|
Hi John:
If you want some help learning C, take a look at Amazon B08GCVV6TG. Obviously, I'm biased, but read several of the reviews and see if you think it might help. I suggested this book over some others because if does talk about the Teensy, which is used in? the T41.
Jack, W8TEE
On Tuesday, April 29, 2025 at 04:59:16 AM EDT, John G0SDF via groups.io <jatkins295@...> wrote:
This is fantastic Oliver! I am a retired EE. I have limited knowledge or experience with software and programming code, but I know all about state machines and I am wanting to learn C programming with the aid of T41-EP. I am particularly interested in the visual diagram because this will help tremendously when I come to design and implement my own functions and button mapping into the code.
Keep up the good work!
?
John, G0SDF
-- Jack, W8TEE
|
John
?
Also Amazon 1484209419 is a very detailed beginner guide to C. I have found it to be excellent. Yet another great book by Jack.
?
Regards
Phil VK2KKZ
|
Thanks, Phil!
Jack, W8TEE
On Tuesday, April 29, 2025 at 02:32:35 PM EDT, Phil Taylor via groups.io <philbyt1@...> wrote:
[Edited Message Follows]
John
?
Also Amazon 1484209419 is a very detailed beginner guide to C. I have found it to be excellent. Yet another great book by Jack.
-- Jack, W8TEE
|
I must say as not yet retired EE, still have a few years before I earn that privilege.
?
I have dabbled in C several times over my career from university and onwards, back then the standard book was "The C Programming Language", by Brian Kernighan?and?Dennis Ritchie. Still a great reference book, but not really directly relevant to Arduino or Teensy. I think the best thing about programming now is the numerous libraries and functions available at your fingertips nowadays, that wasn't the case back when I was at university, the internet was still maturing - GitHub is a new millennium invention.
?
I've found it's the old "use it or lose it" paradigm, it's quite hard to maintain the C skills without using them regularly on top of all the other regularly used skills as an EE. I've also tried Python and JavaScript but lost most of that knowledge quickly. I hope to spend more time in the "developers world" myself, let's see how that goes.
?
All the best John on your journey.
?
Regards
Phil VK2KKZ
|
Phil & Jack. Thank you both for your book recommendations! I now have all 3 books, so no stopping me now!
?
Regards
John, G0SDF
|
Keep me posted on what you think about the books!
Jack, W8TEE
On Wednesday, April 30, 2025 at 03:36:09 AM EDT, John G0SDF via groups.io <jatkins295@...> wrote:
Phil & Jack. Thank you both for your book recommendations! I now have all 3 books, so no stopping me now!
?
Regards
John, G0SDF
-- Jack, W8TEE
|
jjpurdum