Virtual memory wrap up and Binary Formats (by Trek Palmer) ========================================= Page flags ---------- Because the page offset isn't necessary for the page table entries, the bits can be used as flags to encode meta-information into page table entries. Some examples of flags are: resident - the resident flag encodes whether or not the page is actually loaded into memory. If a page is resident, then it's in memory. A non-resident page is actually stored on disk (in the swap file/partition). Using disk to backup cold pages from memory allows modern operating systems to run programs whose total resident memory footprint is larger than physical RAM. Readable - the readable flag is a kind of permission flag. If it is set, then the user program is allowed to read the page. Otherwise, a virtual memory trap is triggered. This allows the operating system to take appropriate action (like killing the program). It may seem odd for a program to not be able to read memory mapped into its own page table, but there are good reasons for disallowing programs to read data. Consider a non-resident page, if the page isn't in memory, but is stored on disk, the program shouldn't be able to access the page (because it isn't currently accessable). But, if the program tries, the OS needs to know so that it can load the page from disk (after the page has become resident, the OS can mark the page readable and resume execution in the program). Writeable - another useful permission flag. Not only useful for trapping writes to non-resident pages, but also to prevent certain kinds of programming errors. Write protected data has all kinds of uses. Executable - a common permission flag (although not supported by intel x86). Data read from an executable page can be assumed to be code and can be executed. The executable flag is useful for typing memory, and it has many security uses. All other kinds of flags are possible, but the basic purpose of this metadata is to allow for the operating system (or the language runtime) to perform more intelligent management of the memory of a program; and to do so transparently to the exeucting program. Note that none of this flag information is presented to the user program. The code just keeps asking for memory locations like it always did, and the metainformation is hidden from it. This is an incredibly powerful abstraction and is one of the reasons why virtual memory is as prevalent as it is. Binary Formats --------------- Up to this point we've been looking a programs by either examining their code, or drawing pictures of how they sit in memory. But when code is assembled, it needs to be stored in a file on disk. First, let us review all the parts of a running program: 1) text - the actual code of the program 2) static data - fixed size global data 3) registers - current program state 4) stack - program function call information 5) heap - dynamic program data Now, the heap is completely dynamic, so it doesn't need to be stored on disk. It is initialized and managed by code. Similarly, the stack is a dynamic structure that depends on the function call sequence of the code. However, you never had to run any special code to initialize the stack. R13 was just magically set to the correct value. For your code, this was done by rolecks when you loaded in your file, in real systems these sorts of initial values are set by a program known as a loader. But, the stack itself has no real static information, so it doesn't need to be stored on disk. The registers are similar, in that they need to be initialized by some loader program (particularly the PC), but have no static data that needs to be permenantly stored. What that leaves is the text and static data. And, for a purely static binary, that's all you really need. Segments and sections ---------------------- Binary files on disk need to be able to distinguish between the different kinds of data stored in them. In binary format nomenclature the different kinds of data are called sections, and sections are grouped together into segments. For our simple static binary example so far, there are really only two sections: .text and .data. The initial '.' is a naming convention passed down to us from computational antiquity (read: the 60s). This explains the previously mysterious keywords you had to adorn your code with (namely, .data and .text). Headers --------- On disk, the binary file begins with a header. This header serves multiple purposes. Initially, the header defines what type of binary file this is. This is important because many systems support several different binary file types. For instance, Linux supports two major formats: a.out and ELF. Linux can also optionally support a bevy of other formats, such as COFF and PE. Because the rest of the file format depends on what kind of file it is, the header must identify which kind of binary it is. Other information in the header includes global parameters, such as the endianness of the file. Remember, these are binary files, so endianness really matters. Many systems may support both endiannesses, but the header itself needs to be byte oriented. Headers will often also contain the offsets for the various sections contained later in the file. The loader ----------- The loader is the program on your system that actually reads in the binary off disk, puts all the sections in their appropriate places in memory, and initializes all the magic values that programs just assume. After the loader has read in the header and decided how to parse the rest of the file, it needs to read in the sections and place them in memory. For a simple static binary, the loader's job is simple. First it reads in the program text, getting enough memory from the operating system to store the program text into. Then it reads in the program data, and stores that in memory as well. Lastly, the loader sets any special register values and then jumps to the starting address of the loaded code. Example: the biny format ------------------------- The format used by rolecks is a simplistic binary format. When you assemble your code, it generates an array of ints that represents the text segment and an array of ints that represents the data segment. It then writes these out to a file with an appropriate header. Header ------- The header begins with the four bytes 'b''i''n''y', the next four bytes are a big-endian 32-bit integer that stores the version number of the format being used. The next four bytes are a big-endian integer that specify what the endianness of the rest of the file is. The next 4 bytes are a big-endian integer giving the size of the text segment (in bytes). The next 4 bytes are an integer giving the size of the data segment (also in bytes). The last 4 bytes specify the size of the symbol table. This is basically a list of the labels and their addresses. This is how the rolecks loader/disassembler knows what your label names were and where they go. Text segment ------------ The text segment begins with a 4 byte integer specifying the starting address for the text segment. The rest of the segment are the 32-bit values that comprise the instructions of your program. Data segment ------------ The data segment starts with a 4-byte integer specifying the starting address of the data segment. The rest of the segment are the bytes that comprise the data segment. Symbol table ------------ The symbol table is something new. You didn't have to specify it when you were writing your code. It is automatically generated by the assembler as it processes your code. The purpose of the symbol table is to make the disassmbled code more readable. Remember that labels are merely a convenience for humans. When the code is assembled, all the labels go away and get turned into numeric values that correspond to offsets from the instruction. Therefore, if we just had the text and the data, the disassembler couldn't reconstruct the label information. So, the assembler builds a little table, that maps a string (the label) to an address. The loader reads this in and the disassembler uses this to write out the labels in the code inspection window when you load up an executable. Dynamic and Relocatable code ----------------------------- So far, we have been talking about static code, where everything is available and known at runtime. This is not always the case. In fact, in systems with dynamic library support (which is most systems now), static binaries are very rare. Most code is dependant upon libraries of other code in order to work. You have used libraries before. In Java, every time you've had to use an import command you've been using a sort of library. In Java, in order to call a library method, you need to know its name. At the machine/assembly level, in order to call a function, you need to know its address. If the code is not immediately available (as in the case with a library), the address cannot be statically generated. Therefore the address needs to be looked up from some kind of data structure built and filled by the loader. In this scheme, rather than directly calling the library routine, your code would call a lookup function with the routine's name as its argument. It is to support this kind of function call indirection that modern binary formats (like ELF) have many segments in addition to text and data.