Skip to content

Commit f1127cb

Browse files
macpijanmacpijan
committed
deploy: 08da96d 
0 parents  commit f1127cb

23 files changed

+3296
-0
lines changed

‎.nojekyll

Whitespace-only changes.

‎README.md

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,24 @@
1+
<!--
2+
SPDX-FileCopyrightText: 2024 3mdeb <[email protected]>
3+
4+
SPDX-License-Identifier: Apache-2.0
5+
-->
6+
7+
# OSFV documentation README
8+
9+
Introducing new platform to the Open Source Firmware Validation infrastructure
10+
requires acquiring crucial parameters of the Device Under Tests (DUT). To
11+
leverage this process, users can refer to the following diagrams and answer the
12+
accompanying questions.
13+
14+
## Power State requirements
15+
16+
![power-state](./img/power-state-v0.1.0.svg)
17+
18+
## Power Supply requirements
19+
20+
![power-supply](./img/power-supply-v0.1.0.svg)
21+
22+
## Serial requirements
23+
24+
![serial](./img/serial-v0.1.0.svg)

‎adding-and-naming-test-cases.md

Lines changed: 219 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,219 @@
1+
<!--
2+
SPDX-FileCopyrightText: 2025 3mdeb <[email protected]>
3+
4+
SPDX-License-Identifier: Apache-2.0
5+
-->
6+
7+
# Tests naming Convention
8+
9+
The tests in the Open Source Firmware Validation repository use the following
10+
naming convention:
11+
12+
`<suite_id><case_id>.<environment_id>`
13+
14+
where:
15+
- `suite_id` - It's typically 3 to 4 letters, uppercase, identifies the
16+
test suite. Should be related to what the test suite accomplishes.
17+
Examples:
18+
- `PSW001.001` - the `PSW` test suite tests the UEFI setup password functionality.
19+
Name is clearly derived from the word `Password`.
20+
- `CBO001.001` - the suite tests customizing the boot order. The name is an acronym
21+
of `Custom Boot Order`
22+
- `case_id` - It's a three digit number with leading zeros. Identifies test
23+
cases in a test suite. Typically test cases are numbered incrementally
24+
starting from `001`.
25+
Examples:
26+
+ `VBO001.001`
27+
+ `VBO002.001`
28+
+ `VBO003.001`
29+
+ `VBO004.001`
30+
- `environment_id` - A three digit number, unambiguously identifies the
31+
environment in which the test case is performed. The IDs of environments
32+
are separated into groups based on the leading digit:
33+
+ `1xx` - Firmware
34+
Example: `101` - EDK2 UEFI
35+
+ `2xx` - Linux
36+
Example: `201` - Ubuntu
37+
+ `3xx` - Windows
38+
Example: `301` - Windows 11
39+
40+
**❗Note:** test IDs must be never reused, even if the test no longer exists.
41+
This is required since old releases may still have results referencing it. All
42+
IDs, current and obsolete, are listed in `test_cases.json` file, which is
43+
described [below](#synchronization-with-database). This file must always be kept
44+
in sync with the code.
45+
46+
All the `environment_id`s of all the tested environments are defined as
47+
Robot Framework variables in the `os-config/environment-test-ids.py` file.
48+
49+
**❗Note:** in old test cases the `environment_id` segment was used more
50+
loosely. `environment_id`s with leading `0` don't use the convention above.
51+
52+
## Transitioning
53+
54+
When updating a test case ID to follow the convention,
55+
append the old ID in the keyword documentation after `Previous IDs:`
56+
to help with identifying the test cases during transition. Example (not from
57+
actual test case):
58+
59+
```robot
60+
DSP002.201 - External HDMI display in OS (Ubuntu)
61+
[Documentation] Check whether an external HDMI display is visible in
62+
... Linux OS. An external HDMI display must be provided in
63+
... the platform config.
64+
...
65+
... Previous IDs: ABC042.001 DSP002.001
66+
(...)
67+
```
68+
69+
If the ID is changed again for the same test, the newer `Previous ID` is added
70+
to the end of line, with single space as a separator (i.e. no comma or semicolon
71+
between them). This format is expected by a script described [later](#validate).
72+
In the example above, `ABC042.001` was changed to `DSP002.001`, which was later
73+
changed to `DSP002.201`.
74+
75+
# Synchronization with database
76+
77+
`test_cases.json` is the source of truth when it comes to test cases. It is the
78+
task of the developer that works on test cases to update this JSON in the same
79+
pull request as changes to the code.
80+
81+
Few examples of test cases in the mentioned JSON file:
82+
83+
```json
84+
[
85+
...
86+
{
87+
"doc": {
88+
"_id": "APU006.002",
89+
"name": "Enabling \"Enable PCIe power management features\" enables ASPM",
90+
"module": "Dasharo Compatibility"
91+
}
92+
},
93+
{
94+
"doc": {
95+
"_id": "AUD001.001",
96+
"name": "Audio subsystem detection (Ubuntu)",
97+
"module": "Dasharo Compatibility",
98+
"changed_to": "AUD001.201"
99+
}
100+
},
101+
...
102+
{
103+
"doc": {
104+
"_id": "AUD001.201",
105+
"name": "Audio subsystem detection (Ubuntu)",
106+
"module": "Dasharo Compatibility"
107+
}
108+
},
109+
...
110+
{
111+
"doc": {
112+
"_id": "ECR017.201",
113+
"name": "Keyboard (function key: flight mode) in OS (Ubuntu)",
114+
"module": "Dasharo Compatibility"
115+
}
116+
},
117+
...
118+
{
119+
"doc": {
120+
"_id": "WLE003.202",
121+
"name": "Bluetooth scanning (Fedora)",
122+
"module": "Dasharo Compatibility"
123+
}
124+
}
125+
]
126+
```
127+
128+
Important to note:
129+
130+
- This must be kept as a valid JSON file, so the last fields of an object and
131+
the last object of an array must not contain a trailing comma.
132+
- Double quotation marks in the test name must be escaped, but parentheses,
133+
colons and single quotation marks must not. Other fields aren't expected to
134+
have any of those characters.
135+
- Format must be preserved, `_id`, `name` and `module` fields are required. For
136+
ease of navigation, keep the as first fields of an object, in that order. New
137+
fields may be added in the future.
138+
- `module` must be one of `Dasharo Compatibility`, `Dasharo Performance`,
139+
`Dasharo Security` or `Dasharo Stability`, case sensitive - both words start
140+
with a capital letter.
141+
- `changed_to` is optional, if it is present, it indicates that the `_id` is
142+
obsolete and shouldn't be used in new releases. When it exists, it must contain
143+
another existing test case ID (which may also have a `changed_to` field,
144+
creating a chronological chain of ID modifications).
145+
- The file is ordered alphabetically by `_id`, keep it that way. This can be
146+
tested with the following command:
147+
148+
```shell
149+
diff -q <(jq 'sort_by(.doc._id)' test_cases.json) <(jq '.' test_cases.json)
150+
```
151+
152+
Before **creating** a new test case, check if the ID isn't and never was in use.
153+
If the ID isn't present in JSON, it may be used, but keep in mind that other
154+
developers may work on other PRs concurrently and may want to use the same ID.
155+
Since both the test development and JSON update is done in the same PR, a merge
156+
conflict clearly shows that a different ID has to be used.
157+
158+
Each time a test name, ID or module is **modified**, the change must be reflected
159+
in `test_cases.json` file. This includes previous IDs, which are saved in
160+
`changed_to` key. Note that there is only one ID in that key. Each ID change
161+
results in _creation_ of a new object in JSON file, and _modification_ of the
162+
existing one by pointing to the new ID.
163+
164+
Test IDs are **never removed**. This is required to keep references in the old
165+
releases valid. It also makes sure that the ID won't be reused.
166+
167+
## Scripts
168+
169+
### Validate
170+
171+
Two scripts are used for listing existing test cases, both active ones as well
172+
as those with deprecated IDs:
173+
174+
- `scripts/list-tests-from-robot.sh` - finds active (lines starting with
175+
something resembling an ID) and obsolete (`Previous IDs`, see
176+
[Transitioning](#transitioning)) IDs in files in `dasharo-*` directories
177+
(i.e. test cases in all Dasharo `*.robot` files), and prints a sorted list of
178+
all of them.
179+
- `scripts/list-tests-from-json.sh` - finds active (without `changed_to` field)
180+
and obsolete (with that field) IDs in `test_cases.json` file, and prints a
181+
list of all of them. The script doesn't sort the output, which indirectly
182+
checks that JSON file is properly sorted.
183+
184+
Output of both of those scripts consists of ID followed by either full test name
185+
(if the test is active) or `DEPRECATED` (if the test ID is obsolete). Comparison
186+
of the outputs can be used to check whether test cases in source files and their
187+
copy in `test_cases.json` are in sync. Possible use cases:
188+
189+
- CI that tests whether PR can be merged:
190+
191+
```shell
192+
diff -q <(./scripts/list-tests-from-robot.sh) \
193+
<(./scripts/list-tests-from-json.sh) || \
194+
(echo "Detected inconsistency between source files and test_cases.json" && false)
195+
```
196+
197+
- manual inspection of differences between the two:
198+
199+
```shell
200+
diff --side-by-side -W200 <(./scripts/list-tests-from-robot.sh) \
201+
<(./scripts/list-tests-from-json.sh) | less
202+
```
203+
204+
**❗Note:** neither of those scripts validates whether `Previous IDs`/`changed_to`
205+
is pointing from/to proper test case, nor that they match each other. The
206+
responsibility for making sure the mapping is valid is shared between the author
207+
and the reviewer.
208+
209+
### Synchronize
210+
211+
The synchronization is performed by `scripts/synchronize-db.py`. The script is
212+
to be started from top directory and doesn't take any parameters, but it reads
213+
database IP (and port) from environmental variable `DB_SERVER_IP`. It will
214+
interactively ask for user credentials. The user must have _Test case developer_
215+
permissions. Example usage, assuming database server running on localhost:
216+
217+
```shell
218+
DB_SERVER_IP=127.0.0.1:5984 ./scripts/synchronize-db.py
219+
```

‎adding-new-platforms.md

Lines changed: 123 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,123 @@
1+
<!--
2+
SPDX-FileCopyrightText: 2024 3mdeb <[email protected]>
3+
4+
SPDX-License-Identifier: Apache-2.0
5+
-->
6+
7+
# Adding new platforms
8+
9+
Depending on what type of platform you're adding, the instructions here will
10+
vary.
11+
12+
- If no similar board is yet supported, follow the steps in
13+
[Adding a brand new platform](#adding-a-brand-new-platform)
14+
- If the board is a variant of another, similar, already supported board, follow
15+
the steps in
16+
[Adding new variant of an existing platform](#adding-new-variant-of-an-existing-platform)
17+
18+
## Generating config variables
19+
20+
To simplify filling in variables in config, you can use
21+
`scripts/get-robot-variables.sh` on the target device. Remember to fill in the
22+
hardware slots (such as WiFi) with the peripherals to be used during tests. It
23+
is assumed that this script is executed on the Ubuntu OS. You can use these
24+
automatically generated variables as another input to the cases described below.
25+
26+
Also, if there already is a Dasharo FW binary for the platform, you
27+
might want to extract the `.config` from the binary and try to run it through
28+
our [config parser](config-parser.md) to get at least some flags automatically.
29+
30+
## Adding a brand new platform
31+
32+
- Create a new file for your mainboard in `platform-configs/`. For most
33+
platforms this file will be called `[platform-vendor]-[platform-model].robot`.
34+
- Copy the contents of `include/default.robot` to your platform config
35+
- Modify the file for your platform:
36+
+ Modify the settings appropriately for your mainboard
37+
+ Remove any unmodified lines - they will be sourced from `default.robot`
38+
+ Add the following at the top of your platform config - this will ensure
39+
defaults are used for unspecified options:
40+
41+
```robot
42+
*** Settings ***
43+
Resource default.robot
44+
```
45+
46+
- Add the platform configuration to `variables.robot:
47+
+ Create a new configuration of RTE, if you are using one, e.g.:
48+
49+
```robot
50+
&{RTE11}= ip=192.168.10.174
51+
... platform=apu4 platform_vendor=PC Engines
52+
```
53+
54+
+ Add the RTE to the list:
55+
56+
```robot
57+
@{RTE_LIST}= &{RTE05}
58+
... &{RTE06} &{RTE07} &{RTE08} &{RTE09} &{RTE10}
59+
... &{RTE11}
60+
```
61+
62+
+ Do the same for any modules installed in the platform
63+
+ Create a new CONFIG containing the RTE and modules used for testing, and
64+
append it to the list:
65+
66+
```robot
67+
@{CONFIG04}= &{RTE11} &{SSD06} &{CARD06} &{USB03}
68+
... &{MODULE06} &{ADAPTER01} &{MODULE10}
69+
70+
@{CONFIG_LIST}= @{CONFIG01} @{CONFIG02} @{CONFIG03} @{CONFIG04}
71+
```
72+
73+
+ Run a simple test to verify the config is working correctly - for example
74+
custom boot menu key:
75+
76+
```robot
77+
robot -v snipeit:no -L TRACE -v rte_ip:192.168.10.174 -v device_ip:0.0.0.0 -v config:pcengines-apu4 dasharo-compatibility/custom-boot-menu-key.robot
78+
```
79+
80+
If everything went right, the output may look something like this
81+
82+
```bash
83+
==============================================================================
84+
Custom-Boot-Menu-Key
85+
==============================================================================
86+
CBK001.001 Custom boot menu key :: Check whether the DUT is config... | PASS |
87+
------------------------------------------------------------------------------
88+
CBK002.001 Custom setup menu key :: Check whether the DUT is confi... | PASS |
89+
------------------------------------------------------------------------------
90+
Custom-Boot-Menu-Key | PASS |
91+
2 tests, 2 passed, 0 failed
92+
==============================================================================
93+
Output: /home/michal/Development/Dasharo/osfv/output.xml
94+
Log: /home/michal/Development/Dasharo/osfv/log.html
95+
Report: /home/michal/Development/Dasharo/osfv/report.html
96+
```
97+
98+
## Adding new variant of an existing platform
99+
100+
Some boards come in multiple variants, where the majority of properties and
101+
features can be shared. For these cases, we have shared "base" configs in
102+
`platform-configs/include/`. This way we don't need to copy-paste entire config
103+
files, making maintenance easier. In this example we'll be adding a new PC
104+
Engines apu variant, using an existing pcengines base config:
105+
106+
- Create a config file in `platform-configs` for your platform
107+
- Add the following to your platform config
108+
109+
```robot
110+
*** Settings ***
111+
Resource include/pcengines.robot
112+
```
113+
114+
- Add variant-specific settings for your platform - in this case, only the
115+
SMBIOS product name field:
116+
117+
```robot
118+
*** Variables ***
119+
${DMIDECODE_PRODUCT_NAME}= apu4
120+
```
121+
122+
- Proceed with adding the platform to `variables.robot` as per
123+
[Adding a brand new platform](#adding-a-brand-new-platform).

0 commit comments

Comments
 (0)