[svsm-devel] EDK2 CAA Page Fragmented Allocation

[Adam Dunlap]

Hi all,

I've been debugging a boot failure on VMs running under Coconut-SVSM
with 120 or more vCPUs. The guest kernel (on several distros) is
hanging while trying to allocate a 256MiB memblock out of "low" memory
for the SWIOTLB.

While debugging this, I noticed that the memory map reported by EDK2
has a ton of entries. There are the expected reserved regions, but
then there's a section of about 2x #vcpus where there is 1 reserved
and 1 usable page alternating. It turns out the reserved pages are
allocated as CAA pages and then usable pages are allocated (and then
freed) as VMSA pages at this[1] point in EDK2. Note that while these
pages are allocated many times over the various UEFI phases, it's only
the CAA pages from the first allocation that are getting leaked.

I had a few questions that I thought someone here might know the answer to:
1. Is UEFI supposed to keep these CAA pages allocated? I believe that
UEFI is supposed to be able to talk to the TPM post-ExitBootServices
and that would likely require CAA pages, but I might be missing
something.
2. How are the other CAA pages being freed? The commit that adds the
allocation [2] does not add any corresponding frees
3. If everything is working pretty much as-expected, then it'd
probably be a good idea to allocate all the CAA pages in one big block
so that they just take up one big entry in the memory map.

Obviously a large/fragmented memory map *shouldn't* cause the guest
kernel to hang, but we have hit issues in the past related to this,
and blowing up the memory map from having ~10 entries to having
several hundred does not seem good.

Please let me know if this memory map pattern appears on other
platforms (maybe I just messed up the edk2 backports) or if you have
any other thoughts about what might be happening.

[1] https://github.com/coconut-svsm/edk2/blob/69d7c826dde066d389e0ba76112b75f4259584eb/UefiCpuPkg/Library/MpInitLib/X64/AmdSev.c#L144
[2] https://github.com/coconut-svsm/edk2/commit/a010681f74c9110a9e8e1753a98c13743b60f7f7

Thanks!
Adam