Test Results & Traceability¶
Overview¶
This section demonstrates how OSQAr auto-imports test results into the documentation, establishing a complete compliance artifact chain from safety goals through implementation code to automated test reporting.
The traceability flow is:
ISO 26262 Safety Goal
↓
Safety Requirement (REQ_SAFETY_*)
↓
Functional Requirement (REQ_FUNC_*, ARCH_*)
↓
Implementation Code (src/* with requirement IDs)
↓
Unit Tests (tests/* with TEST_* IDs)
↓
JUnit XML Test Results
↓
Sphinx Auto-Import & HTML Report
↓
Compliance Artifact Package
Test Suite Execution¶
The test suite can be run locally to generate compliance artifacts:
# Pick one language example and run its end-to-end script.
# This generates JUnit XML + builds HTML docs with imported test results.
cd examples/<language>_hello_world
./build-and-test.sh
# The generated docs live under the example directory.
open _build/html/index.html
Test Configuration File¶
The test results are automatically imported via Sphinx configuration:
# conf.py configuration
extensions = [
'sphinx_needs', # Requirements traceability
'sphinxcontrib.test_reports', # Auto-import JUnit XML
'sphinxcontrib.plantuml', # Diagrams
]
# Point to the JUnit XML file
test_reports = ['test_results.xml']
This configuration tells Sphinx to parse test_results.xml and create a searchable, linked test report within the documentation.
Test Requirements Mapping¶
This section describes how each test requirement maps to implementation code and safety/functional requirements. All TEST_* needs are defined in the Verification & Test Plan document; this section provides the execution results and detailed traceability analysis.
Key Points:
TEST_CONVERSION_001 ((TEST) TEST_CONVERSION_001:... (TEST_CONVERSION_001)): Sensor Driver Tests
TEST_FILTER_001 ((TEST) TEST_FILTER_001: Noi... (TEST_FILTER_001)): Filter Noise Rejection
TEST_THRESHOLD_001 ((TEST) TEST_THRESHOLD_001: ... (TEST_THRESHOLD_001)): Threshold Detection
TEST_HYSTERESIS_001 ((TEST) TEST_HYSTERESIS_001:... (TEST_HYSTERESIS_001)): Hysteresis Deadband
TEST_END_TO_END_001 ((TEST) TEST_END_TO_END_001:... (TEST_END_TO_END_001)): End-to-End Latency
TEST_FAIL_SAFE_001 ((TEST) TEST_FAIL_SAFE_001: ... (TEST_FAIL_SAFE_001)): Fail-Safe on Persistent Errors
See Verification & Test Plan for detailed test case specifications and acceptance criteria.
Traceability Matrix¶
The following matrix demonstrates the complete traceability chain from requirements through code to tests. All IDs are clickable hyperlinks to requirement definitions:
Requirement ID |
Requirement Description |
Test Case(s) |
Code Implementation |
|---|---|---|---|
Prevent thermal damage to equipment |
(TEST) TEST_THRESHOLD_001: ... (TEST_THRESHOLD_001), (TEST) TEST_END_TO_END_001:... (TEST_END_TO_END_001) |
|
|
Detect overheat within 100ms |
|
||
Report safe state recovery reliably |
|
||
Convert 12-bit ADC to 0.1°C units |
|
||
Filter sensor noise (≥80% reduction) |
|
||
Detect 100°C threshold |
|
||
Apply 5°C hysteresis deadband |
|
||
Sensor driver component (100Hz sampling) |
5 class methods in |
||
Filter component (5-sample MA) |
3 class methods in |
||
State machine component (hysteresis) |
(TEST) TEST_THRESHOLD_001: ... (TEST_THRESHOLD_001), (TEST) TEST_HYSTERESIS_001:... (TEST_HYSTERESIS_001) |
4 class methods in |
|
Hysteresis state machine (100°C/95°C thresholds) |
|
||
Fail-safe error handling (10-error threshold) |
Error counter in |
Automated Test Reporting¶
The JUnit XML output from your test runner (pytest for the Python example, or a native runner for C/C++/Rust) is processed by Sphinx and rendered directly in this chapter.
Imported JUnit results¶
pytest¶
Tests: 13, Failures: 0, Errors: 0, Skips: 0
Time: 0.113
class |
name |
status |
reason |
|---|---|---|---|
examples.python_hello_world.tests.test_tsim.TestSensorDriver |
test_conversion_full_range |
passed |
|
examples.python_hello_world.tests.test_tsim.TestSensorDriver |
test_conversion_accuracy |
passed |
|
examples.python_hello_world.tests.test_tsim.TestTemperatureFilter |
test_filter_noise_rejection |
passed |
|
examples.python_hello_world.tests.test_tsim.TestTemperatureFilter |
test_filter_stabilization |
passed |
|
examples.python_hello_world.tests.test_tsim.TestStateMachine |
test_threshold_detection |
passed |
|
examples.python_hello_world.tests.test_tsim.TestStateMachine |
test_hysteresis_deadband |
passed |
|
examples.python_hello_world.tests.test_tsim.TestStateMachine |
test_state_output_bit |
passed |
|
examples.python_hello_world.tests.test_tsim.TestTSIMIntegration |
test_end_to_end_latency |
passed |
|
examples.python_hello_world.tests.test_tsim.TestTSIMIntegration |
test_detection_within_100ms |
passed |
|
examples.python_hello_world.tests.test_tsim.TestTSIMIntegration |
test_safe_state_recovery |
passed |
|
examples.python_hello_world.tests.test_tsim.TestTSIMIntegration |
test_error_recovery |
passed |
|
examples.python_hello_world.tests.test_tsim.TestTSIMIntegration |
test_fail_safe_on_persistent_errors |
passed |
|
examples.python_hello_world.tests.test_tsim.TestConfiguration |
test_hysteresis_constraint |
passed |
|
Notes:
The build workflow generates
test_results.xml.If tests are not executed, OSQAr generates a small placeholder file so docs builds stay robust.
Code Coverage¶
OSQAr supports embedding code coverage evidence alongside the JUnit test report.
If coverage tooling is available for the selected language example, the build workflow generates a
coverage_report.txtand embeds it here.If coverage tooling is not available, a placeholder report is generated so documentation builds remain robust.
Name Stmts Miss Cover Missing
-----------------------------------------------
src/__init__.py 2 0 100%
src/tsim.py 93 1 99% 326
-----------------------------------------------
TOTAL 95 1 99%
Complexity report¶
The build workflow generates a cyclomatic complexity report (complexity_report.txt) and embeds it here.
================================================
NLOC CCN token PARAM length location
------------------------------------------------
6 2 23 1 7 __post_init__@48-54@src/tsim.py
4 1 25 1 5 __init__@82-86@src/tsim.py
21 3 139 2 48 read_adc@88-135@src/tsim.py
2 1 12 1 3 __init__@150-152@src/tsim.py
8 3 68 2 26 update@154-179@src/tsim.py
4 2 40 2 5 __init__@195-199@src/tsim.py
12 5 110 2 33 evaluate@201-233@src/tsim.py
2 1 14 1 11 get_state_output@235-245@src/tsim.py
6 2 53 2 7 __init__@266-272@src/tsim.py
10 3 85 2 40 process_sample@274-313@src/tsim.py
2 1 16 1 12 get_safe_unsafe_output@315-326@src/tsim.py
13 2 65 1 23 test_conversion_full_range@39-61@tests/test_tsim.py
7 2 43 1 15 test_conversion_accuracy@63-77@tests/test_tsim.py
19 5 130 1 38 test_filter_noise_rejection@87-124@tests/test_tsim.py
6 3 39 1 13 test_filter_stabilization@126-138@tests/test_tsim.py
10 1 57 1 21 test_threshold_detection@148-168@tests/test_tsim.py
12 1 73 1 31 test_hysteresis_deadband@170-200@tests/test_tsim.py
5 1 34 1 14 test_state_output_bit@202-215@tests/test_tsim.py
8 2 71 1 27 test_end_to_end_latency@225-251@tests/test_tsim.py
15 4 96 1 26 test_detection_within_100ms@253-278@tests/test_tsim.py
10 3 63 1 21 test_safe_state_recovery@280-300@tests/test_tsim.py
7 2 56 1 16 test_error_recovery@302-317@tests/test_tsim.py
7 2 39 1 16 test_fail_safe_on_persistent_errors@319-334@tests/test_tsim.py
9 1 71 1 18 test_hysteresis_constraint@344-361@tests/test_tsim.py
4 file analyzed.
==============================================================
NLOC Avg.NLOC AvgCCN Avg.token function_cnt file
--------------------------------------------------------------
102 7.0 2.2 53.2 11 src/tsim.py
16 0.0 0.0 0.0 0 src/__init__.py
0 0.0 0.0 0.0 0 tests/__init__.py
145 9.8 2.2 64.4 13 tests/test_tsim.py
===============================================================================================================
No thresholds exceeded (cyclomatic_complexity > 10 or length > 1000 or nloc > 1000000 or parameter_count > 100)
==========================================================================================
Total nloc Avg.NLOC AvgCCN Avg.token Fun Cnt Warning cnt Fun Rt nloc Rt
------------------------------------------------------------------------------------------
263 8.5 2.2 59.2 24 0 0.00 0.00
Building Compliance Artifacts¶
The complete compliance artifact package is generated via:
# 1. Run tests, emit JUnit XML, build docs
cd examples/<language>_hello_world
./build-and-test.sh
# 2. Output contains:
# - Linked requirements and tests
# - Architecture diagrams with PlantUML
# - Test results integrated into HTML
# - Searchable traceability matrix
# - Compliance documentation suitable for assessment/audit
Compliance Artifact Checklist¶
Use this checklist to verify complete traceability:
[✓] Requirements documented with needs IDs (REQ_*, ARCH_*)
[✓] Architecture diagrams with PlantUML (SVG format)
[✓] Implementation code with requirement docstrings
[✓] Test suite with TEST_* IDs mapped to requirements
[✓] JUnit XML test results generated
[✓] Sphinx imports test results into documentation
[✓] HTML documentation includes traceability matrix
[✓] All requirements have ≥1 test case
[✓] All test cases linked to ≥1 requirement
[✓] No orphaned requirements (untested, unimplemented)
[✓] No orphaned tests (unlinked to requirements)
[✓] Build succeeds without errors
[✓] HTML documentation is searchable and indexed
Next Steps¶
Configure CI/CD: Add GitHub Actions to auto-run tests and rebuild documentation on commits
Add Domain Examples: Create medical_device, automotive, robotics subdirectories with domain-specific requirements
Extend Test Coverage: Add performance benchmarks, fault injection tests, environmental stress tests
Implement Requirements Gateway: Create automated checks that fail builds if requirements lack tests