pkg/cover: source coverage reports are overly optimistic

[dvyukov]

I see that lots of kernel interfaces have suspiciously high coverage based on source coverage reports. Lots of them have 100% coverage while we usually don't cover all error paths.

Looking at a concrete function nl80211_set_mcast_rate.
In the current source coverage report, it has 100% coverage:

There are 10 instrumented lines in total, none of them are related to error paths.

The old coverage reports show more realistic picture:

This says the coverage is ~55% (5 out of 9 blocks).

We cannot properly merge and preserve coverage points that point to the same source line across different kernel builds. But we can make sense of them within a single build, and perhaps somehow merge better than we are now.

For example, consider:

• build 1 has 3 coverage points that belong to line 42, one is covered, 2 are not
• total coverage for build 1 is 33%
• build 2 has the same
• total coverage for build 2 is also 33%

When we currently merge we conclude that resulting coverage is 100%, while that can't possibly be true.
Real coverage is at most 66^%, but likely is 33% as well.

cc @tarasmadan

[dvyukov]

Example interface coverage info is in #5925, e.g. for these netlink commands:

NETLINK	NL80211_CMD_SET_BSS	func:nl80211_set_bss	loc:76	coverage:100	access:ns_admin	manual_desc:true	auto_desc:true	file:net/wireless/nl80211.c	subsystem:wireless
NETLINK	NL80211_CMD_SET_CHANNEL	func:nl80211_set_channel	loc:355	coverage:92	access:ns_admin	manual_desc:true	auto_desc:true	file:net/wireless/nl80211.c	subsystem:wireless
NETLINK	NL80211_CMD_SET_COALESCE	func:nl80211_set_coalesce	loc:163	coverage:10	access:ns_admin	manual_desc:true	auto_desc:true	file:net/wireless/nl80211.c	subsystem:wireless
NETLINK	NL80211_CMD_SET_CQM	func:nl80211_set_cqm	loc:224	coverage:76	access:ns_admin	manual_desc:true	auto_desc:true	file:net/wireless/nl80211.c	subsystem:wireless
NETLINK	NL80211_CMD_SET_FILS_AAD	func:nl80211_set_fils_aad	loc:31	coverage:37	access:ns_admin	manual_desc:false	auto_desc:true	file:net/wireless/nl80211.c	subsystem:wireless
NETLINK	NL80211_CMD_SET_HW_TIMESTAMP	func:nl80211_set_hw_timestamp	loc:33	coverage:25	access:ns_admin	manual_desc:false	auto_desc:true	file:net/wireless/nl80211.c	subsystem:wireless
NETLINK	NL80211_CMD_SET_INTERFACE	func:nl80211_set_interface	loc:1322	coverage:55	access:ns_admin	manual_desc:true	auto_desc:true	file:net/wireless/nl80211.c	subsystem:wireless
NETLINK	NL80211_CMD_SET_KEY	func:nl80211_set_key	loc:308	coverage:98	access:ns_admin	manual_desc:true	auto_desc:true	file:net/wireless/nl80211.c	subsystem:wireless
NETLINK	NL80211_CMD_SET_MAC_ACL	func:nl80211_set_mac_acl	loc:83	coverage:31	access:ns_admin	manual_desc:true	auto_desc:true	file:net/wireless/nl80211.c	subsystem:wireless
NETLINK	NL80211_CMD_SET_MCAST_RATE	func:nl80211_set_mcast_rate	loc:60	coverage:100	access:ns_admin	manual_desc:true	auto_desc:true	file:net/wireless/nl80211.c	subsystem:wireless
NETLINK	NL80211_CMD_SET_MESH_CONFIG	func:nl80211_update_mesh_config	loc:180	coverage:78	access:ns_admin	manual_desc:true	auto_desc:true	file:net/wireless/nl80211.c	subsystem:wireless
NETLINK	NL80211_CMD_SET_MPATH	func:nl80211_set_mpath	loc:30	coverage:70	access:ns_admin	manual_desc:true	auto_desc:true	file:net/wireless/nl80211.c	subsystem:wireless
NETLINK	NL80211_CMD_SET_MULTICAST_TO_UNICAST	func:nl80211_set_multicast_to_unicast	loc:28	coverage:91	access:ns_admin	manual_desc:true	auto_desc:true	file:net/wireless/nl80211.c	subsystem:wireless
NETLINK	NL80211_CMD_SET_NOACK_MAP	func:nl80211_set_noack_map	loc:21	coverage:100	access:ns_admin	manual_desc:true	auto_desc:true	file:net/wireless/nl80211.c	subsystem:wireless

[tarasmadan]

What we have requires name. It is not a source coverage and is not a line coverage.

It is a coverage of the lines where we have instrumentation points.
It will be higher than basic block coverage because 1 line can have 10 blocks and only 1 hit is needed to +1 this line.

We have the line:pos of the basic block. Thinking about the line with 3 blocks and hitcount 5, 0, 0...
The number we show may be something like "5/0/0" instead of current "5".

I can't easily show the end of the block. But it is probably another topic.
I don't see the end of the block in dwarf now. And I don't want to manually find the end of the instructions chain...

The game changer here will be the clang source code coverage with mc/dc like this.

[dvyukov]

The number we show may be something like "5/0/0" instead of current "5".
I can't easily show the end of the block. But it is probably another topic.

"5/0/0" would work for me, that would mean 33% coverage. I don't care about HTML representation at this point.