An "eBPF extension" is a Windows kernel driver or component that implements eBPF hooks or helper functions. The design of eBPF for Windows is such that an extension providing an implementation for hooks and helper functions can be developed and deployed without the need to modify either the eBPF execution context or the eBPF verifier.
The eBPF for Windows project uses the Network Module Registrar (NMR) feature in Windows to develop eBPF extensions that are decoupled from the core eBPF for Windows framework. NMR facilitates attaching software modules to each other. The Network Programming Interface (NPI) defines the contract between the modules. The eBPF extensions would implement the Provider Modules for two types of NPIs as discussed in detail below. Even though the NMR component and the relevant APIs have the word "Network" in them, these software modules are completely agnostic of Windows Networking stack and non-networking eBPF extensions can be developed using these.
The audience is assumed to be familiar with NMR Architecture and the various aspects of developing NMR modules as described in NMR documentation.
eBPF Extensions need to implement provider modules for two types of NPIs. They are the Program Information NPI provider and the Hook NPI provider. The following section explains when an extension must implement these providers.
The eBPF Program Information NPI contract is used to provide information about an eBPF program type. Program types are defined as the ABI contracts that eBPF programs are written to conform to. This information is consumed by the eBPF verifier to ensure that any eBPF programs of a given type is safe to load and execute. The ABI contract includes both a description of the "context" parameter passed to the eBPF program as well as the list of specific helper functions that are available to such eBPF programs.
eBPF extensions must implement a different Program Information NPI provider module for each program type for which it provides implementation of hooks or helper functions specific to that program type.
The Hook NPI contracts are used by extension to invoke eBPF programs when OS events occur. A given eBPF hook may have several attach types. The eBPF extension must register a separate Hook NPI provider module for each attach type it supports for an eBPF hook. Note that, there can be more than one attach types for a given program type. If an extension is adding a new attach type for an existing program type, then it only needs to implement the Hook NPI Provider.
The steps for authoring an eBPF extension are:
- Register the NPI provider.
- Author any program type specific Helper Functions.
- Invoke eBPF programs from hook(s).
- Register program and attach types.
The following sections describe these steps in detail.
When registering itself to the NMR, the Program Information NPI Provider should have the
NPI_REGISTRATION_INSTANCE
initialized as follows:
NpiId: This should be set toEBPF_PROGRAM_INFO_EXTENSION_IIDdefined inebpf_extension_uuids.h.ModuleId: This should be set to the eBPF program type GUID.NpiSpecificCharacteristics: Pointer to structure of typeebpf_program_data_t.
This is a mandatory header that is common to all data structures needed by eBPF extensions to register with the eBPF framework.
version: Version of the extension data structure.size: Size of the extension data structure, not including any padding.total_sizeTotal size of the extension data structure, including any padding.
When populating these data structures, the correct version, size and total_size fields must be set. The set of current version numbers and the
size for the various extension structures are listed in ebpf_windows.h. For example:
#define EBPF_PROGRAM_TYPE_DESCRIPTOR_CURRENT_VERSION 1
#define EBPF_PROGRAM_TYPE_DESCRIPTOR_CURRENT_VERSION_SIZE \
(EBPF_OFFSET_OF(ebpf_program_type_descriptor_t, is_privileged) + sizeof(char))
#define EBPF_PROGRAM_TYPE_DESCRIPTOR_CURRENT_VERSION_TOTAL_SIZE sizeof(ebpf_program_type_descriptor_t)When initializing the ebpf_extension_header_t struct, instead of using the individual values listed above, macros like below can also be used for convenience.
#define EBPF_PROGRAM_TYPE_DESCRIPTOR_HEADER \
{ \
EBPF_PROGRAM_TYPE_DESCRIPTOR_CURRENT_VERSION, EBPF_PROGRAM_TYPE_DESCRIPTOR_CURRENT_VERSION_SIZE, \
EBPF_PROGRAM_TYPE_DESCRIPTOR_CURRENT_VERSION_TOTAL_SIZE \
}NOTE: Extension developers must not set the
sizefield of these structures tosizeof()of the corresponding type. Instead, theCURRENT_VERSION_SIZEmacros defined inebpf_windows.hshould be used.
The various fields of this structure should be set as follows:
header: Version and size.program_info: Pointer toebpf_program_info_t.program_type_specific_helper_function_addresses: Pointer toebpf_helper_function_addresses_t. This structure provides the helper functions that are exclusive to this program type.global_helper_function_addresses: Pointer toebpf_helper_function_addresses_t. This structure provides helper functions that override the global helper functions provided by the eBPF runtime.context_create: Pointer toebpf_program_context_create_tfunction that creates a program type specific context structure from provided data and context buffers.context_destroy: Pointer toebpf_program_context_destroy_tfunction that destroys a program type specific context structure and populates the returned data and context buffers.required_irql: IRQL at which the eBPF program is invoked by bpf_prog_test_run_opts.capabilities: 32-bit integer describing the optional capabilities / features supported by the extension.- No capabilities are currently defined.
- The first bit is reserved and must be zero.
Extensions are required to add a context header at the start of each context passed to the eBPF program.
This is required for all extensions to support for runtime state needed by helpers.
To support this, the extension can use the macro EBPF_CONTEXT_HEADER to include
the context header at the start of the program context structure. The context pointer passed to the
eBPF program should point immediately after the context header.
Example
Below is an example of a sample extension including the eBPF context header at the start of the original context structure:
// Original sample extension program context that is passed to the eBPF program.
typedef struct _sample_program_context
{
uint8_t* data_start;
uint8_t* data_end;
uint32_t uint32_data;
uint16_t uint16_data;
} sample_program_context_t;
// Program context including the context header.
typedef struct _sample_program_context_header
{
EBPF_CONTEXT_HEADER;
sample_program_context_t context;
} sample_program_context_header_t;The extension passes a pointer to context inside sample_program_context_header_t and not a pointer to
sample_program_context_header_t when invoking the eBPF program. The header is not accessible
by the program.
The various fields of this structure should be set as follows:
header: Version and size.program_type_descriptor: Pointer toebpf_program_type_descriptor_t.count_of_helpers: The number of helper functions that are implemented by this extension for the given program type.helper_prototype: Pointer to array ofebpf_helper_function_prototype_t.
The various fields of this structure should be set as follows:
header: Version and size.name: Friendly name of the program type.context_descriptor: Pointer of typeebpf_context_descriptor_t.program_type: GUID for the program type. This should be the same as theNpiIdinNPI_REGISTRATION_INSTANCEas noted above.bpf_prog_type: Set to the equivalent bpf program type integer. If there is no equivalent bpf program type, either add a value to thebpf_prog_typeenum and assign it here or this field should be set to0 (BPF_PROG_TYPE_UNSPEC).is_privileged: Set toFALSE.
This structure (as the name signifies) provides a description of the context parameter that a hook passes when invoking an eBPF program. The various fields of this struct are as follows.
header: Version and size.size: Size of the context structure.data: Offset (in bytes) to the field in the context structure that is pointing to the beginning of context data.end: Offset (in bytes) to the field in the context structure that is pointing to the end of context data.meta: Offset (in bytes) to the field in the context structure that is pointing to the beginning of context metadata.
For example, for the XDP_TEST program types, the context data structure is as follows:
// XDP_TEST hook. We use "struct xdp_md" for cross-platform compatibility.
typedef struct xdp_md
{
void* data; ///< Pointer to start of packet data.
void* data_end; ///< Pointer to end of packet data.
uint64_t data_meta; ///< Packet metadata.
/* size: 12, cachelines: 1, members: 3 */
/* last cacheline: 12 bytes */
} xdp_md_t;The corresponding context descriptor looks like:
const ebpf_context_descriptor_t g_xdp_context_descriptor = {sizeof(xdp_md_t),
EBPF_OFFSET_OF(xdp_md_t, data),
EBPF_OFFSET_OF(xdp_md_t, data_end),
EBPF_OFFSET_OF(xdp_md_t, data_meta)};If any of the data or metadata pointer fields are not present on the context structure, the offset value is set to -1 in the context descriptor.
This structure is used to describe the prototypes of the various helper functions implemented by the extension.
typedef struct _ebpf_helper_function_prototype_flags
{
bool reallocate_packet : 1;
} ebpf_helper_function_prototype_flags_t;
typedef struct _ebpf_helper_function_prototype
{
ebpf_extension_header_t header;
uint32_t helper_id;
const char* name;
ebpf_return_type_t return_type;
ebpf_argument_type_t arguments[5];
ebpf_helper_function_prototype_flags_t flags;
} ebpf_helper_function_prototype_t;header: Version and size.helper_id: Integer signifying the helper function ID. (See section 2.6). Helper function IDs for different program types need not be unique.name: Helper function name.return_type: Set the appropriate value for theebpf_return_type_tenum that represents the return type of the helper function.arguments: Array of (at most) five helper function arguments of typeebpf_argument_type_t.flags: Bit field of flags.reallocate_packet: Flag indicating if this helper function performs packet reallocation.
implicit_context: Flag indicating the extension requires implicit context for this helper function.
Note about implicit_context:
With the implicit_context feature, an extension can choose to get the program context as the 6th argument to the
helper function. In case the helper function does not require all the original 5 arguments (program context being
the 6th argument), the helper function should declare dummy arguments as placeholders for the unused arguments.
Note that this new change does not require any change in the helper function prototype that is needed for the
program verification.
Example
Below is an example of a helper function only takes 1 argument arg as input, but to get also program context
as input, also declares the remaining 4 dummy arguments (dummy_param1 to dummy_param4).
static int64_t
sample_ebpf_extension_helper_implicit_2(
uint32_t arg,
uint64_t dummy_param1,
uint64_t dummy_param2,
uint64_t dummy_param3,
uint64_t dummy_param4,
_In_ const sample_program_context_t* context)This enum describes the various argument types that can be passed to an eBPF helper function. This is defined in the PREVAIL Verifier project.
typedef enum _ebpf_argument_type {
EBPF_ARGUMENT_TYPE_DONTCARE = 0,
EBPF_ARGUMENT_TYPE_ANYTHING,
EBPF_ARGUMENT_TYPE_CONST_SIZE,
EBPF_ARGUMENT_TYPE_CONST_SIZE_OR_ZERO,
EBPF_ARGUMENT_TYPE_PTR_TO_CTX,
EBPF_ARGUMENT_TYPE_PTR_TO_MAP,
EBPF_ARGUMENT_TYPE_PTR_TO_MAP_OF_PROGRAMS,
EBPF_ARGUMENT_TYPE_PTR_TO_MAP_KEY,
EBPF_ARGUMENT_TYPE_PTR_TO_MAP_VALUE,
EBPF_ARGUMENT_TYPE_PTR_TO_MEM,
EBPF_ARGUMENT_TYPE_PTR_TO_MEM_OR_NULL,
EBPF_ARGUMENT_TYPE_PTR_TO_UNINIT_MEM,
} ebpf_argument_type_t;This enum describes the various return types from an eBPF helper function. This is defined in the PREVAIL Verifier project.
typedef enum _ebpf_return_type {
EBPF_RETURN_TYPE_INTEGER = 0,
EBPF_RETURN_TYPE_PTR_TO_MAP_VALUE_OR_NULL,
EBPF_RETURN_TYPE_INTEGER_OR_NO_RETURN_IF_SUCCEED,
} ebpf_return_type_t;This structure is used to specify the address at which the various helper functions implemented by the extension
reside. If an eBPF program is JIT compiled, then the generated machine code will have call instructions to these
addresses. For interpreted mode, the eBPF Execution Engine will invoke the functions at these addresses. The fields of
this struct should be set as follows:
helper_function_count: Number of helper functions implemented by the extension for the given program type.helper_function_address: Array of addresses (64-bit unsigned integer) for the helper functions. The addresses must be arranged in the array in the same order as the array of helper function prototypes denoted by thehelper_prototypefield inebpf_program_info_tstruct. For the correct execution of eBPF programs, the helper function addresses cannot change while a loaded eBPF program is executing.
There are two sets of helper function addresses that the extension can return. The first are helper functions that are only callable during execution of an eBPF program that matches this program type. The second are helper function implementations that override the global helper function implementations provided by the eBPF runtime.
This optional function is used to build a program type specific context structure that is used when an application
calls bpf_prog_test_run_opts. The application optionally passes in flat buffers representing the data and the context
structure to be passed to the eBPF program. The extension then constructs a context structure to be passed to the eBPF
program. Note: If ebpf_program_context_create_t is present, then ebpf_program_context_destroy_t and required_irql
must be set.
This optional function is used to populate the flat buffers representing the data and context structures that are
returned to the application when the bpf_prog_test_run_opts call completes. In addition, the function frees any
resources allocated in the ebpf_program_context_create_t call.
All the extension data structures are versioned. To maintain backward compatibility with the existing extensions, new fields MUST be added
to the end of a data structure. The constant defining the current size of the modified struct will be updated in ebpf_windows.h. Existing
eBPF extensions will continue to work without requiring recompilation. If an extension is modified to use a newly added field, the length
field must be updated accordingly.
If the change in data structure is such that it is no longer backward compatible (such as changing field type or position), then the version number will be updated. In this case, the product version of eBPF for Windows must be updated to indicate a breaking change as well. Existing eBPF extensions would need to be re-compiled to work with the latest version of eBPF.
For the cases when a new feature is exposed from eBPF to the extensions that is backward compatible, and requires a bit field / flag to be added to one of the extension data structures, there are 2 possible options:
- Add a new variable in the structure corresponding to the new feature. This will result in increasing the size of the struct.
- Utilize a bit field in an (if available) existing
flagsfield. This option does not result in increasing the size of the struct.
Which of the above 2 options to choose will depend if the feature is forward compatible. If the new feature is forward compatible, (i.e. an extension compiled with new eBPF headers can work fine with an older eBPF runtime), then any of the above 2 options can be chosen to add the flag. However, if the feature is not forward compatible, and will cause functional issues or crashes (when a new extension is deployed with older eBPF runtime), then option 1 MUST be chosen. Choosing option 1 results in and increase of the size of the struct, allowing older eBPF runtime to detect an incompatible extension, and reject binding to such an extension.
When native images are generated, bpf2c uses the verifier to ensure that the program is safe to execute and then computes a hash over the invariants used to validate the program. These invariants include the properties of the program information provider and the signature of any helper functions used. The following fields are included in the hash:
- ebpf_program_type_descriptor_t::name
- ebpf_program_type_descriptor_t::context_descriptor
- ebpf_program_type_descriptor_t::program_type
- ebpf_program_type_descriptor_t::bpf_prog_type
- ebpf_program_type_descriptor_t::is_privileged
- Count of helper ids being used (as a unsigned 64bit integer)
- Each helper function being used is then appended to the hash in order of id.
- ebpf_helper_function_prototype_t::helper_id
- ebpf_helper_function_prototype_t::name
- ebpf_helper_function_prototype_t::return_type
- Each element of the ebpf_helper_function_prototype_t::arguments array
- ebpf_helper_function_prototype_t::flags - only if non-default value
Any new fields MUST be added to the end of the hash and the hash MUST include all fields up to and including the last field containing a non-default value. Fields containing default values after the last non-default value MUST NOT be included. This ensures that hashes computed over older versions of the structure remain valid if new functionality is not being used. If a new feature or functionality is being used, the hash value MUST change to ensure that the verification constraints are honored. All new fields that affect verification MUST be included with a non-default value and all fields that do not affect verification MUST NOT be included.
The eBPF Execution Context registers a Program Information NPI client module with the NMR for every eBPF program that
gets loaded. The Execution Context will use the program type GUID of the program as the NPI ID of the client module.
And as a result, upon eBPF program load, the associated Program Information NPI client module will attach with the
corresponding Program Information NPI provider module in the extension. The Program Information NPI does not have any
client or provider dispatch tables. Neither does the client's NpiSpecificCharacteristics have any data. So, no
special processing is required in the client attach and detach callback handler on the provider module. An extension
must not unload until there are no more attached Program Information NPI clients.
When registering itself to the NMR, the Hook NPI provider should have the
NPI_REGISTRATION_INSTANCE
initialized as follows:
NpiId: This should be set toEBPF_HOOK_EXTENSION_IIDdefined inebpf_extension_uuids.h.ModuleId: This should be set to the attach type GUID.NpiSpecificCharacteristics: Pointer to structure of typeebpf_attach_provider_data_t.
This structure is used to specify the attach type supported by the extension for the given Hook NPI provider. It contains the following fields:
supported_program_typebpf_attach_type
The supported_program_type field of the struct should be filled with the ebpf_program_type_t (GUID) of the
supported program type. While attaching an eBPF program to a hook instance, the Execution Context enforces that the
requested attach type is supported by the Hook NPI provider. If not, the eBPF program fails to attach to the hook.
The bpf_attach_type field should contain the equivalent bpf attach type integer. If there is no equivalent bpf attach type, either add a value to the bpf_attach_type enum and assign it here or this field should be set to 0 (BPF_ATTACH_TYPE_UNSPEC).
The eBPF Execution Context registers a Hook NPI client module with the NMR for each program that is attached to a hook. The attach type GUID is used as the NPI of the client module. And as a result, when an eBPF program gets attached to a hook, the associated Hook NPI client module will attach to the corresponding Hook NPI provider module in the extension. The client attach callback function is invoked when the NPI client is being attached. The provider must store the following in a per-client data structure from the passed in parameters:
ClientBindingContext: Client binding context.ClientDispatch: Client dispatch table (see section 2.5 below).NpiSpecificCharacteristics: Obtained fromClientRegistrationInstanceparameter. This contains attach-type specific data that may be used by an extension for attaching an eBPF program. For example, when an eBPF program is being attached to an XDP_TEST hook, the network interface index can be passed via this parameter. This tells the extension to invoke the eBPF program whenever there are any inbound packets on that network interface. The attach parameter can be obtained as follows:
ebpf_extension_data_t* extension_data = (ebpf_extension_data_t*)ClientRegistrationInstance->NpiSpecificCharacteristics;
attach_parameter = extension_data->data;This structure contains the additional data passed from the application to the attach provider. It contains the following fields:
dataAttach type specific data. See documentation for the attach type provider for the format of this data.data_sizeThe length of the attach type specific data.prog_attach_flagsA collection of attach type specific flags passed from the application to the attach provider.
The per-client data structure should be returned as the ProviderBindingContext output parameter.
Upon
client detach callback
the provider must free the per-client context passed in via ProviderBindingContext parameter.
To invoke an eBPF program, the extension uses the dispatch table supplied by the Hook NPI client during attaching. The client dispatch table contains the functions, with the following type prototypes:
/**
* @brief Invoke the eBPF program.
*
* @param[in] extension_client_binding_context The context provided by the extension client when the binding was created.
* @param[in,out] program_context The context for this invocation of the eBPF program.
* @param[out] result The result of the eBPF program.
*
* @retval EBPF_SUCCESS if successful or an appropriate error code.
* @retval EBPF_NO_MEMORY if memory allocation fails.
* @retval EBPF_EXTENSION_FAILED_TO_LOAD if required extension is not loaded.
*/
typedef ebpf_result_t (*ebpf_program_invoke_function_t)(
_In_ const void* extension_client_binding_context, _Inout_ void* program_context, _Out_ uint32_t* result);
/**
* @brief Prepare the eBPF program for batch invocation.
*
* @param[in] state_size The size of the state to be allocated, which should be greater than or equal to
* sizeof(ebpf_execution_context_state_t).
* @param[out] state The state to be used for batch invocation.
*
* @retval EBPF_SUCCESS if successful or an appropriate error code.
* @retval EBPF_NO_MEMORY if memory allocation fails.
* @retval EBPF_EXTENSION_FAILED_TO_LOAD if required extension is not loaded.
*/
typedef ebpf_result_t (*ebpf_program_batch_begin_invoke_function_t)(
size_t state_size, _Out_writes_(state_size) void* state);
/**
* @brief Invoke the eBPF program in batch mode.
*
* @param[in] extension_client_binding_context The context provided by the extension client when the binding was created.
* @param[in,out] program_context The context for this invocation of the eBPF program.
* @param[out] result The result of the eBPF program.
* @param[in] state The state to be used for batch invocation.
*
* @retval EBPF_SUCCESS.
*/
typedef ebpf_result_t (*ebpf_program_batch_invoke_function_t)(
_In_ const void* extension_client_binding_context,
_Inout_ void* program_context,
_Out_ uint32_t* result,
_In_ const void* state);
/**
* @brief Clean up the eBPF program after batch invocation.
*
* @param[in,out] state The state to be used for batch invocation.
*
* @retval EBPF_SUCCESS.
*/
typedef ebpf_result_t (*ebpf_program_batch_end_invoke_function_t)(
_Inout_ void* state);The function pointer can be obtained from the client dispatch table as follows:
invoke_program = (ebpf_program_invoke_function_t)client_dispatch_table->function[0];When an extension invokes this function pointer, then the call flows through the eBPF Execution Context and eventually
invokes the eBPF program. When invoking an eBPF program, the extension must supply the client binding context it
obtained from the Hook NPI client as the client_binding_context parameter. For the second parameter context, it
must pass the program type specific context data structure. Note that the Program Information NPI provider supplies
the context descriptor (using the ebpf_context_descriptor_t type) to the eBPF verifier and JIT-compiler via the NPI
client hosted by the Execution Context. The result output parameter holds the return value from the eBPF program
post execution.
In cases where the same eBPF program will be invoked sequentially with different context data (aka batch invocation), the caller can reduce the overhead by using the batch invocation APIs. Prior to the first invocation, the batch begin API is called, which caches state used by the eBPF program and prevents the program from being unloaded. The caller is responsible for providing storage large enough to store an instance of ebpf_execution_context_state_t and ensuring that it remain valid until calling the batch end API. Between the begin and end calls, the caller may call the batch invoke API multiple times to invoke the BPF program with minimal overhead. Callers must limit the length of time a batch is open and must not change IRQL between calling batch begin and end. Batch end cost may scale with the number of times the program has been invoked, so callers should limit the number of calls within a batch to prevent long delays in batch end.
An extension can provide an implementation of helper functions that can be invoked by the eBPF programs. The helper functions can be of two types:
- Program-Type specific: These helper functions can only be invoked by eBPF programs of a given program type. Usually, an extension may provide implementations for hooks of certain program types and provide helper functions that are associated with those helper functions. The Program Information NPI provider must then provide the prototypes and addresses for those functions. For these type of helpers, the helper function Id must be greater that 65535 (0xFFFF) for program type specific helper functions.
- General: The general helper functions can be invoked by eBPF programs of all types. Examples of this type of helper functions are the eBPF Map helper functions. These helper functions are implemented by the eBPF Execution Context itself. However, if a program type so chooses, it may provide implementations for general helper functions. For that the extension would have to provide another Program Information NPI provider, which does not provide any program context descriptor. Instead, it only supplies the prototypes and addresses of the general helper functions. The NPI ID of this module defined as:
GUID ebpf_general_helper_function_module_id = {/* 8d2a1d3f-9ce6-473d-b48e-17aa5c5581fe */
0x8d2a1d3f,
0x9ce6,
0x473d,
{0xb4, 0x8e, 0x17, 0xaa, 0x5c, 0x55, 0x81, 0xfe}};The helper function ID for a general helper function must be in the range 0 - 65535 and must be globally unique.
The parameter and return types for these helper functions must adhere to the ebpf_argument_type_t and
ebpf_return_type_t enums.
The eBPF Execution Context loads an eBPF program from an ELF file that has program section(s) with section names. The
prefix to these names determines the program type. For example, the section name "xdp_test" implies that the corresponding
program type is EBPF_PROGRAM_TYPE_XDP_TEST.
The Execution Context discovers the program type associated with a section prefix by reading the data from the "eBPF store", which is currently kept in the Windows registry. An extension developer must author a user mode application which will use eBPF store APIs to update the program types it implements along with the associated section prefixes. eBPF store APIs are exported from ebpfapi.dll.
To operate on the eBPF store, the user mode application needs to link with eBPFApi.dll and include the related include\ebpf_store_helper.h header file, both distributed within the eBPF for Windows NuGet package. With these, the application can use the following APIs to register program types, attach types, and helper functions:
-
ebpf_store_update_section_information: updates the section information in the eBPF store, given a pointer to an array of section information (i.e.,_ebpf_program_section_info):ebpf_result_t ebpf_store_update_section_information( _In_reads_(section_info_count) const ebpf_program_section_info_t* section_info, uint32_t section_info_count);
-
ebpf_store_update_program_information_array: updates program information in the eBPF store, given a pointer to an array of program information (i.e.,_ebpf_program_info):ebpf_result_t ebpf_store_update_program_information_array( _In_reads_(program_info_count) const ebpf_program_info_t* program_info, uint32_t program_info_count);
The eBPF for Windows project provides a sample extension driver as an example for how to implement an extension. This simple extension exposes a new program type, and implements a hook for it with a single attach type. It implements simple NPI provider modules for the two NPIs. It also implements three program-type specific helper functions.