How to map large files?

[LCF-AT]

Hi guys,

today I got a error message trying to map a large file  +1 GB and get error code ERROR_NOT_ENOUGH_MEMORY.For smaller files it works.My question now is how to handle large files without to get any problems?

CreateFile
GetFileSize  // till max 4 GB = -1h
CreateFileMapping
MapViewOfFile,handle,FILE_MAP_READ,0,0,0 // = ERROR_NOT_ENOUGH_MEMORY (00000008)

dwNumberOfBytesToMap 
---------------------------
Specifies the number of bytes of the file to map.
If dwNumberOfBytesToMap is zero, the entire file is mapped.

How to handle large files?What is the normal way for that?Should I first check how much virtualmemory the system has (in my case 4 GB - 1 GB for garfic memory) etc?So whats the maximum file size I can map also without to slow down the system etc?

AddOn question: On the other hand I would also like to know how to handle very large files which have maybe 10 GB.So in that case I could also not use GetFileSize API if its limited to max 4 GB so then I should need to use QWORD with any API to get size etc.Does anyone have any small code examples for this?Just wanna know how to handle that if I should need it someday.

Thank you

[HellRaider]

I dont know exact solution, but i stumbled upon Address Windowing Extensions

https://msdn.microsoft.com/en-us/library/windows/desktop/aa366527.aspx

[kao]

Read. The. Fabulous. Manual.

For files that are larger than the address space, you can only map a small portion of the file data at one time. When the first view is complete, you can unmap it and map a new view

Just like you can't do VirtualAlloc of 3GB of RAM, you can't map 3GB file into memory. The actual size you can map depends on number of factors, including all running apps and how much memory they allocated. If you must do something like that, the design of your software is wrong. Go back to drawing board.

2nd argument of GetFileSize() is used for files larger than 4GB.

[null_endian]

@LCF-AT Are you working on a 64-bit or 32-bit Windows system? I think on 64 bit, you should be able to do 4GB with CreateFileMapping() and MapViewOfFile() but also check the AWE as @HellRaider said those are cool. As I recall, they just require a bit more work on your end because you are dealing with physical memory.

[LCF-AT]

Hi guys,

thanks for your answes so far.Ok I see it now with GetFileSize API.Normaly I dont wanna map ultra large files but I just thought about it if you wanna maybe edit any large file.So in this case you must load the file anyhow but if its not doable using MapViewOfFile function in that case.

PS: I am using 32bit OS.

greetz

 

[fearless]

If file is smaller than say 512MB then map entire file

If file is larger than that, then map in chunks of file - but depends on the file your mapping and the requirements.

so essentially you can create a sliding window into the file your require, look at it at a defined position and size (which has to be aligned for granularity) and read required data, then close that mapview when no longer required

so i use something like this function, to calc a specific offset and size, it calcs the mapping offset i will need to read the offset and the size, which can be then passed to a mapview call later on

;-----------------------------------------------------------------------------------------
; CalcLargeFileView - calculate large mapping size, view and offset
;-----------------------------------------------------------------------------------------
CalcLargeFileView PROC USES EBX EDX dwRequiredViewSize:DWORD, dwRequiredViewOffset:DWORD, lpdwMappedViewSize:DWORD, lpdwMappedViewOffset:DWORD
    LOCAL sysinfo:SYSTEM_INFO
    LOCAL dwAllocationGranularity:DWORD
    LOCAL dwAdjustedOffset:DWORD
    
    Invoke GetSystemInfo, Addr sysinfo
    mov eax, sysinfo.dwAllocationGranularity
    mov dwAllocationGranularity, eax
    IFDEF DEBUG32
        PrintDec eax
    ENDIF    

    mov eax, dwRequiredViewSize
    .IF eax < dwAllocationGranularity
        mov ebx, lpdwMappedViewSize
        mov eax, dwAllocationGranularity
        add eax, dwAllocationGranularity
        mov [ebx], eax
    .ELSE
        mov eax, dwRequiredViewSize
        xor edx, edx
        mov ebx, dwAllocationGranularity
        div ebx ; Divides dwRequiredViewSize by dwAllocationGranularity. EAX = quotient and EDX = remainder (modulo).
        .IF edx > 0 ; we have a remainder, so calc to add dwAllocationGranularity to dwRequiredViewSize - remainder
            mov eax, dwRequiredViewSize
            sub eax, edx
            add eax, dwAllocationGranularity
            add eax, dwAllocationGranularity
            mov ebx, lpdwMappedViewSize
            mov [ebx], eax
        .ELSE ; else we have a multiple of dwAllocationGranularity, so just return the dwRequiredViewSize as actualsize
            mov eax, dwRequiredViewSize
            mov ebx, lpdwMappedViewSize
            mov [ebx], eax
        .ENDIF
    .ENDIF

    mov eax, dwRequiredViewOffset
    .IF eax < dwAllocationGranularity
        mov ebx, dwAllocationGranularity
        sub ebx, eax
        mov dwAdjustedOffset, ebx
        mov eax, 0
        mov ebx, lpdwMappedViewOffset
        mov [ebx], eax
    .ELSE
        mov eax, dwRequiredViewOffset
        xor edx, edx
        mov ebx, dwAllocationGranularity
        div ebx ; Divides dwRequiredViewSize by dwAllocationGranularity. EAX = quotient and EDX = remainder (modulo).
        .IF edx > 0 ; we have a remainder, so calc to add dwAllocationGranularity to dwRequiredViewSize - remainder
            mov dwAdjustedOffset, edx
            mov eax, dwRequiredViewOffset
            sub eax, edx
            mov ebx, lpdwMappedViewOffset
            mov [ebx], eax
        .ELSE ; else we have a multiple of dwAllocationGranularity, so just return the dwRequiredViewSize as actualsize
            mov eax, dwRequiredViewOffset
            mov ebx, lpdwMappedViewOffset
            mov [ebx], eax
            mov dwAdjustedOffset, 0
        .ENDIF
    .ENDIF
    
    mov eax, dwAdjustedOffset
    ret

CalcLargeFileView endp

So can be used to map view like so with a function:

;-----------------------------------------------------------------------------------------
; OpenLargeMapView - opens a view in a large mem mapped file to access data pointed to
; by dwRequiredViewOffset. Returns in eax adjusted offset of memory where dwRequiredViewOffset
; can be accesed. Once finished with this view use CloseLargeMapView to close it.
;-----------------------------------------------------------------------------------------
OpenLargeMapView PROC USES EBX dwRequiredViewSize:DWORD, dwRequiredViewOffset:DWORD
    LOCAL LargeMapResourceSize:DWORD
    LOCAL LargeMapResourceOffset:DWORD
    LOCAL LargeMapAdjOffset:DWORD
    LOCAL LargeMapHandle:DWORD
    LOCAL LargeMemMapPtr:DWORD
    LOCAL LargeFileMapping:DWORD

    Invoke CalcLargeFileView, dwRequiredViewSize, dwRequiredViewOffset, Addr LargeMapResourceSize, Addr LargeMapResourceOffset
    mov LargeMapAdjOffset, eax

    Invoke MapViewOfFileEx, LargeMapHandle, FILE_MAP_READ, 0, LargeMapResourceOffset, LargeMapResourceSize, NULL
    .IF eax == NULL ; try again with 0 as no bytes to map - otherwise end of files are a problem, cant alloc size > max file size, let MapViewOfFile handle this with 0 specified as no bytes to map
        Invoke MapViewOfFileEx, LargeMapHandle, FILE_MAP_READ, 0, LargeMapResourceOffset, 0, NULL
        .IF eax == NULL
            mov eax, -1
            ret
        .ENDIF    
    .ENDIF
    mov LargeMemMapPtr, eax
    add eax, LargeMapAdjOffset
    ret
OpenLargeMapView ENDP

Some code stripped from examples, as not relevant and are specific to .bif files etc.