From 2c0e84265c85fd849a0426a2e86a000cd2ec2042 Mon Sep 17 00:00:00 2001 From: Bernhard Kaindl Date: Wed, 26 Feb 2025 12:00:00 +0100 Subject: [PATCH] (docs) Extend the xenguest flowcharts for HVM build Signed-off-by: Bernhard Kaindl --- .../lib/xenctrl/xc_domain_node_setaffinity.md | 32 ++++++--- .../xenopsd/walkthroughs/VM.build/xenguest.md | 71 +++++++++++-------- .../VM.build/xenguest/do_hvm_build.md | 62 ++++++++++++++++ .../VM.build/xenguest/mode_vm_build.md | 36 ++++++++++ 4 files changed, 161 insertions(+), 40 deletions(-) create mode 100644 doc/content/xenopsd/walkthroughs/VM.build/xenguest/do_hvm_build.md create mode 100644 doc/content/xenopsd/walkthroughs/VM.build/xenguest/mode_vm_build.md diff --git a/doc/content/lib/xenctrl/xc_domain_node_setaffinity.md b/doc/content/lib/xenctrl/xc_domain_node_setaffinity.md index 82cf2e36f08..d046f5c9f35 100644 --- a/doc/content/lib/xenctrl/xc_domain_node_setaffinity.md +++ b/doc/content/lib/xenctrl/xc_domain_node_setaffinity.md @@ -62,16 +62,30 @@ https://github.com/xen-project/xen/blob/master/xen/common/domain.c#L943-L970" This function implements the functionality of `xc_domain_node_setaffinity` to set the NUMA affinity of a domain as described above. -If the new_affinity does not intersect the `node_online_map`, -it returns `-EINVAL`, otherwise on success `0`. -When the `new_affinity` is a specific set of NUMA nodes, it updates the NUMA -`node_affinity` of the domain to these nodes and disables `auto_node_affinity` -for this domain. It also notifies the Xen scheduler of the change. - -This sets the preference the memory allocator to the new NUMA nodes, -and in theory, it could also alter the behaviour of the scheduler. -This of course depends on the scheduler and its configuration. +- If `new_affinity` does not intersect the `node_online_map`, + it returns `-EINVAL`. Otherwise, the result is a success and it returns `0`. +- When the `new_affinity` is a specific set of NUMA nodes, + it sets `d->node_affinity` of the domain to these nodes + and disables `auto_node_affinity` for this domain. +- If `new_affinity` has all bits set, it re-enables `auto_node_affinity` + for this domain and calls + [domain_update_node_aff()](https://github.com/xen-project/xen/blob/e16acd80/xen/common/sched/core.c#L1809-L1876) + to re-set the domain's `node_affinity` mask to the NUMA nodes of the current + the hard and soft affinity of the domain's online vCPUs. + +The result of changing the domains' node affinity changes the +preference of the memory allocator to the new NUMA nodes. + +Currently, the only scheduling change is that if set before vCPU creation, +the initial pCPU of the new vCPU is the first pCPU of the first NUMA node +in the domain's `node_affinity`. This is if further changed when one of more +`cpupools` are set up. + +When done early, before vCPU creation, domain-related data structures +could be allocated using the domain's `node_affinity` NUMA node mask. +With further changes in Xen, also the vCPU struct could be allocated +using it. ## Notes on future design improvements diff --git a/doc/content/xenopsd/walkthroughs/VM.build/xenguest.md b/doc/content/xenopsd/walkthroughs/VM.build/xenguest.md index 70908d556fb..73e5eceace4 100644 --- a/doc/content/xenopsd/walkthroughs/VM.build/xenguest.md +++ b/doc/content/xenopsd/walkthroughs/VM.build/xenguest.md @@ -2,6 +2,8 @@ title: xenguest description: "Perform building VMs: Allocate and populate the domain's system memory." +mermaid: + force: true --- As part of starting a new domain in VM_build, `xenopsd` calls `xenguest`. When multiple domain build threads run in parallel, @@ -83,38 +85,30 @@ Xenstore[Xenstore platform data] --> xenguest When called to build a domain, `xenguest` reads those and builds the VM accordingly. -## Walkthrough of the xenguest build mode +## Walk-through of the xenguest build mode -```mermaid -flowchart -subgraph xenguest[xenguest #8209;#8209;mode hvm_build domid] -direction LR -stub_xc_hvm_build[stub_xc_hvm_build#40;#41;] --> get_flags[ - get_flags#40;#41; <#8209; Xenstore platform data -] -stub_xc_hvm_build --> configure_vcpus[ - configure_vcpus#40;#41; #8209;> Xen hypercall -] -stub_xc_hvm_build --> setup_mem[ - setup_mem#40;#41; #8209;> Xen hypercalls to setup domain memory -] -end -``` +{{% include "xenguest/mode_vm_build.md" %}} -Based on the given domain type, the `xenguest` program calls dedicated -functions for the build process of the given domain type. +The domain build functions +[stub_xc_hvm_build()](https://github.com/xenserver/xen.pg/blob/65c0438b/patches/xenguest.patch#L2329-L2436) +and stub_xc_pv_build() call these functions: -These are: +1. `get_flags()` to get the platform data from the Xenstore + for filling out the fields of `struct flags` and `struct xc_dom_image`. +2. `configure_vcpus()` which uses the platform data from the Xenstore to configure: + - If `platform/vcpu//affinity` is set, the vCPU affinity. -- `stub_xc_hvm_build()` for HVM, -- `stub_xc_pvh_build()` for PVH, and -- `stub_xc_pv_build()` for PV domains. + By default, this sets the domain's `node_affinity` mask (NUMA nodes) as well. + This configures + [`get_free_buddy()`](https://github.com/xen-project/xen/blob/e16acd80/xen/common/page_alloc.c#L855-L958) + to prefer memory allocations from this NUMA node_affinity mask. + - If `platform/vcpu/weight` is set, the domain's scheduling weight + - If `platform/vcpu/cap` is set, the domain's scheduling cap (%cpu time) +3. The `_build_setup_mem` function for the given domain type. -These domain build functions call these functions: +Call graph of `do_hvm_build()` with emphasis on information flow: -1. `get_flags()` to get the platform data from the Xenstore -2. `configure_vcpus()` which uses the platform data from the Xenstore to configure vCPU affinity and the credit scheduler parameters vCPU weight and vCPU cap (max % pCPU time for throttling) -3. The `setup_mem` function for the given VM type. +{{% include "xenguest/do_hvm_build" %}} ## The function hvm_build_setup_mem() @@ -129,26 +123,41 @@ new domain. It must: 4. Call the `libxenguest` function `xc_dom_boot_mem_init()` (see below) 5. Call `construct_cpuid_policy()` to apply the CPUID `featureset` policy +It starts this by: +- Getting `struct xc_dom_image`, `max_mem_mib`, and `max_start_mib`. +- Calculating start and size of lower ranges of the domain's memory maps + - taking memory holes for I/O into account, e.g. `mmio_size` and `mmio_start`. +- Calculating `lowmem_end` and `highmem_end`. + +It then calls `xc_dom_boot_mem_init()`: + ## The function xc_dom_boot_mem_init() +`hvm_build_setup_mem()` calls +[xc_dom_boot_mem_init()](https://github.com/xen-project/xen/blob/39c45c/tools/libs/guest/xg_dom_boot.c#L110-L126) +to allocate and populate the domain's system memory: + ```mermaid flowchart LR subgraph xenguest hvm_build_setup_mem[hvm_build_setup_mem#40;#41;] end subgraph libxenguest -hvm_build_setup_mem --> xc_dom_boot_mem_init[xc_dom_boot_mem_init#40;#41;] +hvm_build_setup_mem --vmemranges--> xc_dom_boot_mem_init[xc_dom_boot_mem_init#40;#41;] xc_dom_boot_mem_init -->|vmemranges| meminit_hvm[meninit_hvm#40;#41;] click xc_dom_boot_mem_init "https://github.com/xen-project/xen/blob/39c45c/tools/libs/guest/xg_dom_boot.c#L110-L126" _blank click meminit_hvm "https://github.com/xen-project/xen/blob/39c45c/tools/libs/guest/xg_dom_x86.c#L1348-L1648" _blank end ``` -`hvm_build_setup_mem()` calls -[xc_dom_boot_mem_init()](https://github.com/xen-project/xen/blob/39c45c/tools/libs/guest/xg_dom_boot.c#L110-L126) -to allocate and populate the domain's system memory. +Except error handling and tracing, it only is a wrapper to call the +architecture-specific `meminit()` hook for the domain type: + +```c +rc = dom->arch_hooks->meminit(dom); +``` -It calls +For HVM domains, it calls [meminit_hvm()](https://github.com/xen-project/xen/blob/39c45c/tools/libs/guest/xg_dom_x86.c#L1348-L1648) to loop over the `vmemranges` of the domain for mapping the system RAM of the guest from the Xen hypervisor heap. Its goals are: diff --git a/doc/content/xenopsd/walkthroughs/VM.build/xenguest/do_hvm_build.md b/doc/content/xenopsd/walkthroughs/VM.build/xenguest/do_hvm_build.md new file mode 100644 index 00000000000..d3548162a72 --- /dev/null +++ b/doc/content/xenopsd/walkthroughs/VM.build/xenguest/do_hvm_build.md @@ -0,0 +1,62 @@ +--- +title: Call graph of xenguest/do_hvm_build() +description: Call graph of xenguest/do_hvm_build() with emphasis on information flow +--- +```mermaid +flowchart TD +do_hvm_build("do_hvm_build() for HVM") +--> stub_xc_hvm_build +get_flags("get_flags()") --"VM platform_data from XenStore" +--> stub_xc_hvm_build("stub_xc_hvm_build()") +stub_xc_hvm_build --> configure_vcpus(configure_vcpus#40;#41;) +configure_vcpus --"When
platform/ + vcpu/%d/affinity
is set"--> xc_vcpu_setaffinity +configure_vcpus --"When
platform/ + vcpu/cap
or + vcpu/weight
is set"--> xc_sched_credit_domain_set +stub_xc_hvm_build --"struct xc_dom_image, mem_start_mib, mem_max_mib" +--> hvm_build_setup_mem("hvm_build_setup_mem()") +--"struct xc_dom_image + with + optional vmemranges"--> xc_dom_boot_mem_init + subgraph libxenguest + xc_dom_boot_mem_init("xc_dom_boot_mem_init()") + -- "struct xc_dom_image + with + optional vmemranges" --> + meminit_hvm("meminit_hvm()") -- page_size(1GB,2M,4k, memflags: e.g. exact) --> + xc_domain_populate_physmap("xc_domain_populate_physmap()") + end +subgraph set_affinity[XenCtrl Hypercalls] +direction TB + xc_sched_credit_domain_set("xc_sched_credit_domain_set()") + xc_vcpu_setaffinity("xc_vcpu_setaffinity()") + --> vcpu_set_affinity("vcpu_set_affinity()") + --> domain_update_node_aff("domain_update_node_aff()") + -- "if auto_node_affinity + is on (default)"--> auto_node_affinity(Update dom->node_affinity) +end +subgraph Xen hypervisor +xc_domain_populate_physmap("xc_domain_populate_physmap()") +--hypercall: XENMEM_populate_physmap + -->memory_op_populate_physmap("memory_op(populate_physmap)") +end +click vcpu_set_affinity +"https://github.com/xen-project/xen/blob/e16acd806/xen/common/sched/core.c#L1353-L1393" _blank +click domain_update_node_aff +"https://github.com/xen-project/xen/blob/e16acd806/xen/common/sched/core.c#L1809-L1876" _blank +click stub_xc_hvm_build +"https://github.com/xenserver/xen.pg/blob/65c0438b/patches/xenguest.patch#L2329-L2436" _blank +click hvm_build_setup_mem +"https://github.com/xenserver/xen.pg/blob/65c0438b/patches/xenguest.patch#L2002-L2219" _blank +click get_flags +"https://github.com/xenserver/xen.pg/blob/65c0438b/patches/xenguest.patch#L1164-L1288" _blank +click configure_vcpus +"https://github.com/xenserver/xen.pg/blob/65c0438b/patches/xenguest.patch#L1297" _blank +click xc_dom_boot_mem_init +"https://github.com/xen-project/xen/blob/e16acd806/tools/libs/guest/xg_dom_boot.c#L110-L125" +click meminit_hvm +"https://github.com/xen-project/xen/blob/e16acd806/tools/libs/guest/xg_dom_x86.c#L1348-L1648" +click xc_domain_populate_physmap +"../../../../lib/xenctrl/xc_domain_populate_physmap/index.html" _blank +``` diff --git a/doc/content/xenopsd/walkthroughs/VM.build/xenguest/mode_vm_build.md b/doc/content/xenopsd/walkthroughs/VM.build/xenguest/mode_vm_build.md new file mode 100644 index 00000000000..572308833da --- /dev/null +++ b/doc/content/xenopsd/walkthroughs/VM.build/xenguest/mode_vm_build.md @@ -0,0 +1,36 @@ +--- +hidden: true +title: Call graph to the xenguest hvm/pvh/pv build functions +description: Call graph of xenguest for calling the hvm/pvh/pv build functions +--- +```mermaid +flowchart LR +xenguest_main[" + xenguest + --mode hvm_build + / + --mode pvh_build + / + --mode pv_build +"] --> do_hvm_build[" + do_hvm_build() for HVM +"] & do_pvh_build["do_pvh_build() for PVH"] -- "`**Arguments:** + domid + mem_max_mib + mem_start_mib + image + store_port + store_domid + console_port + console_domid`" --> stub_xc_hvm_build["stub_xc_hvm_build()"] +xenguest_main --> do_pv_build[do_pvh_build for PV] --> + stub_xc_pv_build["stub_xc_pv_build()"] +click do_pv_build +"https://github.com/xenserver/xen.pg/blob/65c0438b/patches/xenguest.patch#L575-L594" _blank +click do_hvm_build +"https://github.com/xenserver/xen.pg/blob/65c0438b/patches/xenguest.patch#L596-L615" _blank +click do_pvh_build +"https://github.com/xenserver/xen.pg/blob/65c0438b/patches/xenguest.patch#L617-L640" _blank +click stub_xc_hvm_build +"https://github.com/xenserver/xen.pg/blob/65c0438b/patches/xenguest.patch#L2329-L2436" _blank +```