Questions & answers

Debugger or calibration tool?

Somewhere in between.

Most developers are familiar with the concept of a debugger, a tool that allows for setting breakpoints, stepping through code, and watching variable values. A calibration tool is a term far less popular than a debugger. It doesn't have the same purpose although some functionalities can overlap.

Calibration tools are primarily used in the aerospace and automotive industries. These software tools are designed to send configuration parameters to a device and monitor its behavior with the newly applied parameters. A common example would be the tuning of a PID controller, where the P, I, and D gains are adjustable values. Developers test the system response under certain gain values, observe how the variables change over time, and then adjust the gains accordingly. This can be seen as a high-level debugging process.

These adjustable parameters are often identified by unique IDs known to the calibration tool. Reading and writing operations are performed through a communication channel (such as a serial port, CAN bus, or socket) by sending read or write commands for the given parameter ID. This is similar to how Scrutiny handles its Runtime Published Values (RPVs).

But what if all variables could be tunable parameters? Instead of explicitly exposing them in the embedded firmware by assigning IDs, they could be automatically discovered using the debug symbols of a binary.
This is the approach taken by the aerospace and automotive industries using protocols such as XCP and the A2L file format. Such a tool can be invaluable for monitoring program execution and identifying defects, bringing it closer to a debugger.

Scrutiny is doing the same, using the debugging symbols to become a calibration tool on steroids. While it cannot be used to step through code or watch local variables, these limitations can be circumvented by using embedded graphs and temporarily making variables static during the debugging phase.

I only see static and global variables. What about locals?

Local variables on the stack are not accessible to Scrutiny. Make them static volatile, but avoid reentrance.

Because it works through instrumentation, the memory is accessed by Scrutiny only when the Scrutiny instrumentation code is executed. At that moment, the stack does not carry variables from other functions.

To view your local variables with Scrutiny, you can temporarily make them static and volatile. Making them static moves the variable outside of the stack, assigning them a fixed memory location that Scrutiny can inspect. Declaring the variable as volatile ensures that the compiler does not optimize the variable out.

Consider the code example below.


int my_func(int x) {
    int my_var = 0;
    if (x > 0){
        my_var = 1
    }
    my_var = do_something(my_var);
    return my_var;
}

First, it is impossible to know the value of my_var at any point in the code. Making the variable static volatile will let us see the value at the exit of the function only. We can look at intermediate values by inserting additional static variables


int my_func(int x) {
    static volatile int my_var;  // Do not initialize here
    static volatile int debug_var; 
    my_var = 0; // Initialization is moved in a different statement to keep the behavior of a local variable.
    if (x > 0){
        my_var = 1
    } 
    debug_var = my_var; // Catch the value of my_var for Scrutiny
    my_var = do_something(my_var);
    return my_var;
}

As you can see, we now have a hook that allows us to determine if my_var was 0 or 1 before calling do_something(). We can inspect my_var with Scrutiny to get the output of do_something(), and debug_var to get the value of my_var after the condition.

Another consideration is that the rate and order at which these two variables are read is not guaranteed. It largely depends on your communication link and the number of variables being watched simultaneously. To achieve a synchronized reading of the two variables, utilize the datalogging feature and create an embedded graph of them. This will allow you to see, for each execution loop, the values of my_var and debug_var.

Lastly, when making your variables static, be cautious to avoid reentrant functions; static variables will alter their behavior since each call will share the variable with the calling function. This is rarely an issue, unless you have a task that can interrupt another one and both are using the same piece of code.

I'm having an issue with Scrutiny, how can I pinpoint the problem?

Most features are managed by the Python server which can be launched using the Scrutiny CLI. The CLI accepts a --loglevel parameter that can make it much more verbose.

To launch a server with debug logging, use the following command:
scrutiny launch-server --config my_config.json --loglevel debug --logfile scrutiny.log

Note that the --loglevel option is applicable to all commands, including those used to run unit tests. Executing a failing unit test with debug logging is an effective method for identifying defects.

By default, error messages are suppressed during unit test execution to prevent confusion when running tests that are expected to return an error. To run a specific test with debug logging, use the following command:
python -m scrutiny runtest --module test.server.foo.bar.baz --loglevel debug

Notice how we did not use the scrutiny command to run the tests, but python -m scrutiny. This is because unit tests are not included in the release package when you install Scrutiny. Unit tests are only available in a development environment.

Why do you use a custom protocol instead of something like XCP?

Nobody would really benefit from that.

Supporting an existing protocol would primarily serve to make a device, previously compatible with a paid tool such as Vector Canape, compatible with the Scrutiny UI. When Scrutiny was developed, the goal was to introduce calibration tool/instrumentation debugging to sectors where it was not commonly used. Industries that require protocol compatibility are typically already deeply invested in paid tools, and they can continue to use these. Newcomers have the option to fully adopt Scrutiny or not.

Moreover, supporting compatibility with a standard is a time-consuming process, and development time is a scarce resource in a voluntary project. It would simply be an effort put at the wrong place.

Finally, having a tailored protocol, embedded and web API, is highly beneficial for development. All the commands are meaningful and perfectly suited to their tasks. Once the protocol is understood, development becomes much easier, faster, and cleaner.

Can I use Scrutiny for NVM configuration?

Yes, but you need to add a hook in your code

static volatile bool save_eeprom=false;
static volatile EEPROM_t eeprom_input;

if (save_eeprom){
    my_eeprom_driver.write_config(&eeprom_input, sizeof(eeprom_input));
    save_eeprom = false;
}

In the given example, the configuration code will not be executed under normal circumstances. But, The clients now has access to both eeprom_input and save_eeprom. These can be manually written or modified through the SDK.

To enhance usability, it's possible to create an alias to abstract this variable. This allows these items to appear at a user-friendly location, such as /EEPROM/Save & /EEPROM/Content. This feature is especially useful when the individual configuring the system is different from the one developing the software (for instance, the engineering team versus the production team). It also ensures that the EEPROM hooks consistently appear at the same location in the variable tree, irrespective of the software version, thereby preventing backward compatibility issues in the configuration scripts.

Note that the SDK also has the capability of writing memory chunk directly, which can be useful for automation.

Does scrutiny-embedded uses dynamic allocation?

No, static allocation only