How do you create a very large sparse array in Windows?

From What they don’t tell you about demand paging in school they demonstrate how to create a sparse array with mmap. With this code, I can easily create an extremely large sparse array, say 1TB in size, despite only having 32GB of ram available. The advantage here is that only pages I write to will be backed by physical memory, and any other page can safely be read from and will simply return zeros.

Is it possible to achieve this exact same behavior on Windows?

EDIT: First I tried VirtualAlloc, but reserving a large array without committing seems to throw memory access violaiton upon read instead of simply returning zero. If I commit, then the storage gets (or attempts to get) backing as soon as I read from it. Next I tried CreateFileMapping + MapViewOfFile but this is using up lots of temporary space on my filesystem, and I can’t manage to get an array much larger than 50GB on my computer. I’m hoping I’m just not using these functions correctly, or perhaps there’s another option that actually replicates what I have in Linux.

#include <sys/mman.h>
#include <iostream>

const size_t MB = 1024 * 1024;
const size_t GB = MB * 1024;

int main() {
  size_t alloc_size = 512 * MB;
  size_t total_alloc = 100 * GB;
  size_t num_allocs = total_alloc / alloc_size;

  std::cout << "Allocating mem...\n";

  char* base = nullptr;

  // Allocate a ton of memory
  for (size_t i = 0; i < num_allocs; i++) {
    // Unsound alert - assuming allocations are contiguous and grow down.
    base = (char*)mmap(NULL, alloc_size, PROT_READ, MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
    if (base == MAP_FAILED) {
      perror(NULL);
      throw std::runtime_error("Fail");
    }
    std::cout << (void*)base << " " << i << "\n";
  }

  std::cout << "Allocated Virtual Mem (GB): " << total_alloc / GB << "\n";
  std::cout << "Base Addr: " << (void*)base << "\n";
  std::cout << "Press enter to start reading.\n";
  getchar();
  std::cout << "Reading each page...\n";

  // Read the first byte of each page
  for (size_t i = 0; i < total_alloc; i += 0x1000) {
    auto x = base[i];
  }

  std::cout << "Done!\n";
  getchar();
}

You need to sign in to view this answers