Reading process memory / RAM

RAM

One of the most common questions I get regarding HxD is about obtaining source code to understand and reproduce some of its basic functionality. The most frequent points of interest are how the RAM editor reads / writes memory of other running programs (i.e., processes) and how the disk editor reads from / writes to storage media.

Yesterday, when Mohamed A. Mansour, who will teach an Operating Systems class, suggested this information might be useful for his students, I thought it was a good opportunity to start my blog and write it up in a proper way.

This post is about reading memory that belongs to another running program (aka. process), and will be the first in a series of posts about the implementation and design of HxD.

Memory management basics

So, how do you read memory from another program? Let us first begin by introducing some common memory “layout” and access models. If you know already about the flat memory model, protected memory, and virtual memory you can skip to the next section.

Flat memory model

While memory systems can be complex, it is rare to encounter anything but a flat memory model, which essentially means that all main memory accessible to your program can be represented by a big one-dimensional array of bytes. To address a unit of memory (i.e., a byte) you simply index into the array.

In the illustrative table below, we assume a flat 32-bit memory model where each byte (uint8_t) out of theoretically 4 GiB can be addressed individually.

Array data
Array index 0x0 0x1 0x2 0x3 0xFFFFFFFF

So each address (= array index) must be between 0 and 232-1 (= 4294967295). Represented in hexadecimal numbers, the lowest address is 0x0 and the highest address  is 0xFFFFFFFF. We call this addressable range of memory the address space. Obviously, the actually accessible memory depends on the amount of installed physical memory (RAM).

Simple data types

If all memory can be thought of as being a big array, how do simple data types work?

In programming languages, variables are merely names for addresses that – with some “syntactic sugar” – ease access to memory. So if you want to store the height of a person in a byte-sized variable, you could do the following in C:

Which is effectively just a shorthand for something like this:

You may also want to store a 32-bit wide integer that holds a color in the RGBA color space. For example: 33% red (0x55), 66% green (0xAA), 100% blue (0xFF), 0% transparency/alpha (0x00) could be expressed as hexadecimal number 0x55AAFF00.

In this case memory access would be slightly more complex. You would have to decompose the 32-bit value into 4 bytes, before you can write the value to main memory. Depending on the CPU the order of the bytes differs (see Endianness). In little endian systems such as x86-32 and x86-64 the order starts with the least significant end first.

0x55AAFF00 in little endian order would be this 4 byte sequence:

0x00 0xFF 0xAA 0x55

You can see that the hexadecimal notation lends itself well to split numbers into bytes, which is one reason why it is commonly used in hex editors but also in programming.

To write the 32-bit color value to main memory, each of those 4 bytes is written individually:

In practice CPUs optimize access for commonly used byte multiples such as 4 byte (=32-bit) or 8 byte (=64-bit) data types, but conceptually memory access works as outlined above.

Arrays of non byte-sized elements

An important observation to make here is that addresses are always byte-array indices, even if the array element size is a multiple of bytes, such as for elements of type uint16_t (2 bytes) or double (8 bytes).

For example, assuming the byte array x and the unsigned 16-bit integer array y point to the same block of memory (and therefore their start addresses and lengths in bytes are the same):

Then, it holds that:

  • addr_of(x[0]) == addr_of(y[0])
  • addr_of(x[2]) == addr_of(y[1])
  • addr_of(x[4]) == addr_of(y[2])
  • addr_of(x[6]) == addr_of(y[3])

The memory layout of array x and y is illustrated in the following table, where each element is represented by a cell. Notice how elements of y cover two cells of x.

array x (element size = 1 byte) x[0] x[1] x[2] x[3] x[4] x[5] x[6] x[7]
array y (element size = 2 bytes) y[0] y[1] y[2] y[3]

In general, to get an address of an array-element a[i], you apply this formula:

For example, the address of y[3] would be:

Hints for writing real code

The pseudo code above is meant to exemplify and explain what goes on behind the scenes. In production code you would use a function such as sizeof() in C to get the array_element_size. Also, in C to get an address you use the operator & instead of addr_of(), which is a pseudo code function used for clarity.

Depending on how your compiler performs data alignment, array elements may not be packed tightly together, but padding bytes might be inserted to improve alignment for performance reasons. In this case you have to account for the padding bytes that are usually appended after each element and increase array_element_size accordingly. So be sure to check the alignments and paddings your compiler uses and if sizeof() includes those additional bytes or not. This applies also to languages such as Delphi, not just C.

More safe and simple is to use normal array indexing, such as this in C: &a[5] which gets you the address of the 6th element of the array a. Though, our point is to understand how this addressing works internally, so we are bound to be exposed to some technical details and caveats as mentioned above.

What about pointers?

A pointer is a variable of type integer only holding an index into the big one-dimensional memory array. When you dereference a pointer, you access the data through the array index (= memory address) stored in the pointer. The name pointer refers to the fact that pointer variables do not store the data itself but an index to it, i.e., they point to the data.

Protected memory

On computers that are programmed on the bare metal, such as microcontrollers (think of Arduino) or operating systems that do not protect memory (such as DOS or Win 3.1), all processes share the same address space. That means all running programs (aka. processes) have the same one-dimensional contiguous array available to them, as explained above.

This makes data exchange easy but can also cause erroneous or malicious overwriting of memory that belongs to another process – leading to stability and security problems. Protected memory introduces access barriers to avoid such issues.

Virtual memory

To fully separate processes, address spaces are virtualized, such that it appears to each process as if there is a separate main memory per process, the virtual memory.

That is, each process has its own virtual memory array and can use the entire address range without colliding with another process. Therefore a virtual address is only meaningful inside a process and data cannot be shared (directly). The operating system takes care of mapping virtual addresses to actual physical addresses behind the scenes.

To reinforce the point consider this example: virtual address 0x1234 in process A points to the physical address 0x2000 while the same virtual address 0x1234 in process B points to the physical address 0x4000. So the virtual addresses 0x1234 in A and B do not collide despite having the same value because the operating system maps them to different physical addresses (⇒ different actual memory).

By being the middle man, the operating system can have fine grained control over memory and restrict access or grant it only to processes with special privileges. This makes foreign memory access a deliberate and restricted action and can improve security and stability.

Reading memory from another process under Windows

Given that modern operating systems use virtual protected memory, and therefore processes are totally isolated from each other, how can you access memory belonging to another process? The answer: application programming interfaces (API) provided by the operating system that are primarily designated for debugging purposes.

For Windows the essential functions are ReadProcessMemory, OpenProcess, VirtualQueryEx.

In the following I will present a small C++ program that dumps memory to the command line. For clarity I will only discuss the points directly relevant for accessing foreign memory and omit things such as string processing for pretty printing on the console, detailed error handling, or user interface (console based). You can download a fully working project for VS2010 (and up) that has all those features. Be sure to check its Readme.txt.

The following source code…

… would produce an output similar to this (actual output depends on the read data, here it is the start of a loaded executable file):

The basic principle underlying the source code is simple:

  1. You obtain the process ID from the process whose memory you want to access.
  2. You use  OpenProcess  to get a reading handle for the process.
  3. You use  ReadProcessMemory  to read memory starting at  foreignProcessAddress  and ending at  foreignProcessAddress + bufferSize - 1.
  4. And finally you close the reading handle using CloseHandle.

The data is now in the byte-array  buffer  and can be printed to the screen as a string of hex-pairs using  bytes_to_hex_string  (see project download above for its implementation).

While the buffer is limited to 16 bytes, this is an arbitrary choice to simplify printing on the console. Usually you would read memory in larger chunks, for example using the memory page size (see GetSystemInfo).

What is most puzzling about this code is how to obtain a process ID or find out what memory address to read. Again the example project shows one way how this could be done.

Obtaining process IDs

The task manager can show the PID of each running process in a separate column. If this column is not yet visible, select the tab “processes”, then select “View|Choose columns…”, and tick the checkbox “PID (process-ID)”.

You can also list all running processes using Windows-API functions or get a process ID from a GUI window. I will present some of this in a future blog post.

Determining valid or useful memory addresses

If you know already the address at which the foreign process stores the data you are interested in, this is a no-brainer: just set  foreignProcessAddress  accordingly.

If you want to explore the memory like you can do in HxD, you need to enumerate all accessible memory regions (using VirtualQueryEx ), or simply read through all the virtual memory address space from 0 to end, and just output those  ReadProcessMemory  calls that do not return an error.

Similarly, you can scan for valid memory regions and then perform a text search on the buffer. This is also a candidate for some future blog post.

Other operating systems

As mentioned before, programs running under operating systems that do not protect memory or isolate processes through virtual memory, can just address the entire main memory and read it using pointers. Sometimes addressing can be complicated through a segmented memory model as found in DOS, but this is hardly relevant today.

“Safe” languages such as Java do not allow to specify random memory addresses — by design they only ever allow references to the owned memory — and therefore would still prohibit reading foreign memory even in a low policing OS. Given a special OS API (as presented above) or external library this can be circumvented, but not with native means of the Java language, e.g., arrays or references.

Finally, like most modern operating systems, Linux also uses a virtual protected memory model. However, my experience with reading foreign process memory under Linux is limited and I wouldn’t want to give an untested example.

Closing words

Phew, this first post turned out much longer and formal than I intended it to be, but I thought that some background and clarity was necessary to really understand how to use the functions. I hope this post is useful to you and will give you an idea what’s going on in case of unexpected results.

Do you have an equally functioning example for Linux as the one I provided for Windows? Add a comment below and share your knowledge :).

Also, I welcome comments on my content presentation or writing, I am new to blogging and keen to learn.

 

Image credits:

17 responses on “Reading process memory / RAM

  1. K Hodgkinson

    I like the style – not too formal at all. We wouldn’t be reading if we didn’t want to understand the explanation properly.
    Trivial typo near the beginning – but I thought you’d rather it were correct than not.

    main_memory[address_of_person_height] = 0x00;
    main_memory[address_of_person_height + 1] = 0xFF;
    main_memory[address_of_person_height + 2] = 0xAA;
    main_memory[address_of_person_height + 3] = 0x55;
    should read:
    main_memory[address_of_baloon_color] = 0x00;
    main_memory[address_of_baloon_color + 1] = 0xFF ;
    main_memory[address_of_baloon_color + 2] = 0xAA;
    main_memory[address_of_baloon_color + 3] = 0x55;

    KH
    PS Please feel free to edit out the ‘correction part’ of this comment.

    1. mael Post author

      Fixed. Thanks for notifying me about this mistake. I’ll leave your comment unchanged as reference for people having read the older version of this post.

  2. Andrew

    I think you have an error under “Arrays of non byte-sized elements”,

    For example, the address of y[3] would be:
    address_of_y_3 = &y[0] + i * 2;

    I think it should be = &y[0] + i * 3, or alternatively the address of y[2].

    Great post, thanks!

    1. mael Post author

      Fixed. Thanks for pointing this out. I forgot to replace i with the actual index. 2 is the array element size (for 16-bit array elements).
      So the proper statement would have been
      address_of_y_3 = &y[0] + 3 * 2;

        1. mael Post author

          In general yes, use sizeof(). The point here is however to explicitly demonstrate the computation with examples. sizeof() is an ambiguous name, because size is also used to mean element count sometimes (e.g., size() of vectors).
          array_element_size avoids these issues and seems better suited for pseudo code.

          I edited the post to clarify this.

  3. Andrew

    You need a hex editor that you download on a PC but plug your iOS device into the PC and use the hex editor for the iOS devise

    1. mael Post author

      I am not sure what you want to do exactly? Use a hex editor from Windows to edit files on an iOS device or RAM of the iOS device?
      Both would probably need at least a mini program on the iOS device that communicates with the PC. I don’t think iOS exposes all its files when its plugged into a PC, just your media files, maybe documents, too, but not system files or RAM.

  4. Jakub

    Thank you for the article!
    As was already mentioned, you wrote it in a very clear style.
    I understand it takes some work to write something properly but in case you decide to write more in the future, I will keep checking back! I found it by random and I am glad I did 🙂

  5. Justin

    To anyone reading this in the 64-bit era, please understand a few things so you don’t waste time smashing your face on your keyboard.

    64-bit processors only use 48-bit address spaces. This allows plenty of RAM for consumer systems and is cheaper to implement than the full 64-bit address space (simpler, smaller, and cheaper chips.) If you’re writing a program and see that your max address is 2^48-1, that’s why.

    For Windows, you’ll want to investigate the MEMORY_BASIC_INFO structure. Some of the values in that structure (State, Type, Protect) are enums/constants, and you’ll want to be aware of all of them.

    You’ll also want to be aware of the difference between Base Address and Allocation Base Address and the connection with Region Size. The address you asked for is the Base Address. It is within a memory allocation. The Allocation Base Address tells you where the allocation started. Region size tells you the number of bytes from your base address (not allocation base address) to the point where some property of the RAM changes, i.e. State, Type, Protect. I *believe* that if you have two identical Allocations in a row, the region size reaches all the way past the end of the first allocation to the end of the second (or last back-to-back-and-same-properties) allocation.

    I do not know if Guard regions are triggered when an MBI is acquired for the page, or only when the region is read. If you need to know, you’ll need to build your own sandbox to avoid potential issues.

    1. mael Post author

      Thanks for your comment, some quick additional notes.

      The OS will always reserve a certain portion of the address space, the processor can deal with full 64 bit addresses alright, but it will not work on the memory bus/not be forwarded and result in access errors. The Win-API functions for querying data can behave weirdly as well, but that’s not a processor limitation.

      To be more specific, for 32 bit programs you can iterate over all the addresses from 0 to 2^32-1. For 64 bit programs it makes sense to use GetSystemInfo() and the lpMaximumApplicationAddress field to get the highest valid address an application can have. Hard coding it to a 48-bit value, such as 2^48-1 is not advisable. Be aware that GetSystemInfo() will only return meaningful information if the calling process is 64-bit as well! So your RAM editor would need to be compiled as 64 bit.

      When iterating over these addresses, you have to start at the minimal valid address, and call VirtualQuery/VirtualQueryEx() to get info about accessibility status, if it’s free/allocated, etc. in the form of MEMORY_BASIC_INFO. Then you increase your query address () by MEMORY_BASIC_INFO.RegionSize and call VirtualQuery/VirtualQueryEx again.
      If you repeat this until you reach lpMaximumApplicationAddress you will know the state of the entire address space.

      This may not be a consistent snapshot, though, if the program who’s address space you analyze keeps running.

      Finally, if you are just writing a regular program, you don’t need to worry about this at all. You just use the normal memory allocation functions your programming environment provides, and will always remain within valid addresses, if you stay within the bounds of arrays (length/size). Otherwise you will get protection or access errors like under 32 bit. If you are under a managed environment (e.g., .NET/Java), you wont get those errors either, because you can’t have invalid addresses aside for null and range checks are performed automatically and enforced.

Leave a Reply to mael Cancel reply

Your email address will not be published. Required fields are marked *