Experiments in Software

Implications of ECMA-335 and Partition I, 12.1.2 and 12.

2010-04-05T12:23:00.000+02:00

As I’ve been spending quite some time this weekend to refactor parts of the MOSA compiler and fixing things small and large. I’ve stumbled once again over our memory model. I was refactoring our internal representation in order to make load and store operations explicit and broke almost all of our tests at once. Fixing them was pretty easy, except for the smaller types... Section 12.1.2 states:

„Loading from 1- or 2-byte locations (arguments, locals, fields, statics, pointers) expands to 4-byte values.“

Ouch. We’ve gone through a lot of trouble to ensure correct arithmetics on all types and have been badly missing the point: All smaller integral types are handled at 4 bytes in size on the evaluation stack.

Next step was to change the CIL load instructions to correctly reflect this fact and fortunately we already had the appropriate instructions in the IR. So the current state of work is that most of our tests are passing again, but not all yet. Then I started wondering about the floating point specification. Looking at the section for floating point values (12.1.3), it states:

„The supported storage sizes are float32 and float64. Everywhere else (on the evaluation stack, as arguments, as return types, and as local variables) floating-point numbers are represented using an internal floating-point type. In each such instance, the nominal type of the variable or expression is either float32or float64, but its value can be represented internally with additional range and/or precision. The size of the internal floating- point representation is implementation-dependent, can vary, and shall have precision at least as great as that of the variable or expression being represented. An implicit widening conversion to the internal representation from float32 or float64 is performed when those types are loaded from storage. The internal representation is typically the native size for the hardware, or as required for efficient implementation of an operation.“

So for floating point types we have exactly one stack type F, but the implementation is free to choose the precision of its operations as long as it is at least as large as the storage size of the floating point type. Since we’ve spent a great deal of time on single precision arithmetics, I’m inclined to keep the reduced precision operations there. Any opinions?

I’ll continue fixing this in the next couple of days.

Why the Switch Fixture has crashed on I2/U2 and others...

2010-04-04T02:50:00.000+02:00

The tests are run by .NET calling through a function pointer delegate using the stdcall calling convention. This calling convention is similar to cdecl. One of the similarities is that the EBX register must be saved by the callee and restored before it returns to the caller. We didn’t do that and thus corrupted the state of the .NET runtime on Windows. Bug fixed and commit following soon.

The next step: Virtual methods and method tables

2010-03-27T21:05:00.000+01:00

After the static object allocation, I’ve finished the next step for MOSA. The compiler now emits mtable (virtual method tables) records for compiled types and is able to properly call virtual functions. The test to check these is in CallVirtFixture. I’ll add a couple more tests there to check for proper hiding, base class calls and other things - hopefully the current code should handle all of those cases sufficiently well.

In order to accomplish this I’ve had to add a fake System.Object implementation to the existing tests, as those classes wouldn’t compile anymore - the linker couldn’t create the vtable for them due to the 4 virtual methods every object inherits from System.Object: ToString, GetHashCode, Equals and Finalize.

The good thing about this is of course we can now use virtual functions and use overrides to do OO-kernels, the down side of course is: Every kernel has to provide at least a fake implementation of System.Object.

I’ve added a fake System.Object to the existing HelloWorld kernel.

Let’s see what Phil and Simon can come up with in Hello World, now that this is out of the way.

Making good on a promise I've made a long time ago.

2010-03-15T21:55:00.000+01:00

After improving our test situation on the weekend, I’ve started on making good a promise I’ve given a long time ago. The promise was: Allocate static objects at compile time for core kernel services.

The issue with writing a managed operating system or any operating system is memory management and moving to the OS way of working at boot time. The core problem is that there are assumptions about objects, which can’t be met easily as memory management is being initialized on the CPU(s) the OS will run on. MOSA is facing this problem too, with one addition: Writing non-OO code in an OO language feels broken.

Classical operating systems solve this by having a reduced set of services while booting and initializing the OS services later in the OS specific fashion. We could’ve done this too, but why go for common ground if there’s new to explore.

So what does this feature do? This feature detects all dynamic memory allocations happening in static constructors and allocates memory for the allocated objects at compile time in the bss segment of an executable. It replaces the call to new with a load of the address of the data segment location, making the position of the object fixed in memory relative to its load address. This allows core OS services to be written using C# classes right from the start and allows them to be used as such.

There are some limitations though: The allocated object must be fixed in size, it must not have a complex structure and the field used to store the object must have the exact same type as used for the new operator. No casts allowed.

There’re probably further limits to this feature, but I haven’t figured them out yet. It will certainly be interesting to explore our HelloWorld kernel with OO-features using the new MOSA compiler.

Oh and before I forget: The feature must be turned on explicitly on the command line. Use --enable-static-alloc (or the shorter --sa) to enable it.

I’ve update the HelloWorld projects CMOS and Boot classes to take advantage of this mechanism.

Changes I've made to MOSA to support generics, Part I

2010-03-11T20:44:00.000+01:00

Over the past couple of days I’ve made changes to MOSA to support generics. The first and primary change was to support a scheduled compilation model.

Let me explain this: Up to these changes the MOSA compiler would just scan an assembly, locate all types and compile each method contained in them. The only exceptions to this rule were native methods, generic types and generic methods.

The revised compilation scheduler stage

However skipping generics doesn’t work anymore - you want those generic types and methods compiled too. The issue you face is how to compile these methods and types without knowing the usage. So I basically kept the current compilation scheduler, but added the capability for the pipeline to schedule additional types and methods in the scheduler. In order to do this, there’s a new assembly compilation stage: The ICompilationSchedulerStage. This stage performs the type lookups that used to be done by the MethodCompilerBuilderStage.

The compilation scheduler now maintains a schedule of methods and types to compile and executes these in order. Once all scheduled methods and types are compiled, the entire assembly including all of its generic usages has been compiled.

MOSA: Generic method compilation, first test succeeded

2010-03-11T03:01:00.000+01:00

After about 3 hours patching and fixing the last issues I finally completed the first successful test of generics in MOSA. The following C# fragment compiles successfully and passes all tests:

static class Test
{
        private static T GenericMethod<T>(T value)
        {
                return value;
        }

public static bool TestCallGenericMethodWith(int value)
{
                return value == GenericMethod(value);
        }
}

It may not look like much, but this test is the ground for all other generic arguments and has moved forward not only the compiler, but also the assembly loader, the MOSA runtime and some other additions like cleaner error messages from mosacl.

I’ll finish this test case tomorrow and push my changes to Github, I’ll probably need some help with that from fellow MOSA contributors.

Good night.

mosacl is compiling very simple generic types

2010-03-09T02:11:00.000+01:00

Let mosacl speak for itself:

Mini:Bin Michael$ mono --debug mosacl.exe -a x86 -o MQT.exe -f PE Mosa.QuickTest.exe
MOSA AOT Compiler, Version 0.6 'Tanigawa'
Copyright 2009 by the MOSA Project. Licensed under the New BSD License.
Copyright 2008 by Novell. NDesk.Options is released under the MIT/X11 license.

Output file: MQT.exe
Input file(s): /Users/Michael/Projects/MOSA/MOSA-Project/Bin/Mosa.QuickTest.exe
Architecture: Mosa.Platforms.x86.Architecture
Binary format: Portable Executable File Linker
Boot format: Not bootable
Is executable: True

Compiling ...
Scheduling type Mosa.QuickTest.App for compilation.
Compiling type Mosa.QuickTest.App
Compiling method Main
Loaded generic type Mosa.QuickTest.Generic`1
Scheduling type Mosa.QuickTest.Generic`1<System.Int32> for compilation.
Compiling type Mosa.QuickTest.Generic`1<System.Int32>
Compiling method .ctor
Compiling method get_Value

Unresolved symbols.
System.Object..ctor()

Mini:Bin Michael$

Signs of generic types

2010-03-07T10:03:00.000+01:00

After hacking MOSA two days again and trying to figure out what I was thinking way back, here’s one of the first runs with generic types:

        mini:Bin Michael$ mono --debug mosacl.exe -o MQT.exe --Architecture=x86 --format=PE Mosa.QuickTest.exe
        MOSA AOT Compiler, Version 0.6 'Tanigawa'
        Copyright 2009 by the MOSA Project. Licensed under the New BSD License.
        Copyright 2008 by Novell. NDesk.Options is released under the MIT/X11 license.

        Output file: MQT.exe
        Input file(s): /Users/Michael/Projects/MOSA/MOSA-Project/Bin/Mosa.QuickTest.exe
        Architecture: Mosa.Platforms.x86.Architecture
        Binary format: Portable Executable File Linker
        Boot format: Not bootable
        Is executable: True

        Compiling ...
        Scheduling type Mosa.QuickTest.App for compilation.
        Compiling type Mosa.QuickTest.App
        Compiling method Main
        Scheduling type System.Collections.Generic.List`1<System.Int32> for compilation.
        Scheduling type Mosa.QuickTest.Generic`1<System.String> for compilation.
        Compiling type System.Collections.Generic.List`1<System.Int32>
        Compiling method .ctor

Unfortunately this is as far as it gets right now, but I’m sure I’ll find the reason for this soon.

I had to rewrite a whole bunch of stuff to get this far, including the way types are scheduled for compilation by mosacl. Surprising is that generics can be AOT’d under certain circumstances.

Solving the pkg-config and Mono 3.5 Profile mysteries on MacOS X 10.6 (Mono 2.6.1)

2010-03-06T13:26:00.000+01:00

Yesterday I’ve tried to install Mono 2.6.1 on my Mac mini to do some work on my pet project MOSA. Unfortunately things didn’t work out as smoothly as I thought as nant greeted me with the following error message:

        Failed to initialize the 'Mono 3.5 Profile' (mono-3.5) target framework.

Furthermore it stated:

        'pkg-config' failed to start.

I tried all sorts of things from reinstalling, to cleaning, to trying to pull mono from Macports and even compiling Mono myself. Nothing has helped, every attempt was stopped by some error at some point.

The I started to do some digging and tried running pkg-config from Terminal - sure enough it wasn’t there. It was however in my /opt/local/bin folder, so running

        /opt/local/bin/pkg-config --modversion mono

resulted in:

        Package mono was not found in the pkg-config search path.
        Perhaps you should add the directory containing `mono.pc'
        to the PKG_CONFIG_PATH environment variable
        No package 'mono' found

Ok, so again I stumbled - looking further I found another pkg-config installed by Mono itself. So running it with the following two commands in Terminal:

        export PATH=/Library/Frameworks/Mono.framework/Versions/2.6.1/bin:$PATH
        pkg-config --modversion mono

produced:

        Package glib-2.0 was not found in the pkg-config search path.
        Perhaps you should add the directory containing `glib-2.0.pc'
        to the PKG_CONFIG_PATH environment variable
        Package 'glib-2.0', required by 'Mono', not found

Again I stumbled. Looking for glib-2.0.pc I found it in /opt/local/lib. So I added it to the PKG_CONFIG_PATH and ran it again:

        export PKG_CONFIG_PATH=/opt/local/lib/pkgconfig:$PKG_CONFIG_PATH
        pkg-config --modversion mono

Now I finally got the response:

        2.6.1

So I’ve finally got pkg-config to work and running nant worked too - I can now start contributing some code again...

Debug Symbols for MOSA #4 - PDB File Format

2009-01-18T00:53:00.002+01:00

I'll continue my series with this posting about the PDB file format. There are some places on the net, which already describe the file format. However I've found some inaccuracies in most places - probably due to the fact that the format itself is not open and everyone performs some kind of interpretation of the data they see. Again, take this post with a grain of salt. I'm going to talk about the managed PDB files explicitly, so expect some differences compared to native code PDB files.

Test Code

Before we'll dive into the format itself I want to show you the sample code I'm going to use to describe the format with. The following C# source is compiled with debug support in order to generate a PDB file.

<span style="color:#999999;">1</span> <span style="color:#0000ff;">using</span><span style="color:#000000;"> System;<br /></span><span style="color:#999999;">2</span> <span style="color:#000000;"><br /></span><span style="color:#999999;">3</span> <span style="color:#000000;"></span><span style="color:#0000ff;">namespace</span><span style="color:#000000;"> PdbExample<br /></span><span style="color:#999999;">4</span> <span style="color:#000000;">{<br /></span><span style="color:#999999;">5</span> <span style="color:#000000;"></span><span style="color:#0000ff;">class</span><span style="color:#000000;"> Program<br /></span><span style="color:#999999;">6</span> <span style="color:#000000;">{<br /></span><span style="color:#999999;">7</span> <span style="color:#000000;"></span><span style="color:#0000ff;">static</span><span style="color:#000000;"> </span><span style="color:#0000ff;">void</span><span style="color:#000000;"> Main(</span><span style="color:#0000ff;">string</span><span style="color:#000000;">[] args)<br /></span><span style="color:#999999;">8</span> <span style="color:#000000;">{<br /></span><span style="color:#999999;">9</span> <span style="color:#000000;">Console.WriteLine(</span><span style="color:#800000;">"</span><span style="color:#800000;">Hello World!</span><span style="color:#800000;">"</span><span style="color:#000000;">);<br /></span><span style="color:#999999;">10</span> <span style="color:#000000;"></span><span style="color:#0000ff;">for</span><span style="color:#000000;"> (</span><span style="color:#0000ff;">int</span><span style="color:#000000;"> i </span><span style="color:#000000;">=</span><span style="color:#000000;"> </span><span style="color:#800080;">0</span><span style="color:#000000;">; i </span><span style="color:#000000;"><</span><span style="color:#000000;"> </span><span style="color:#800080;">10</span><span style="color:#000000;">; i</span><span style="color:#000000;">++</span><span style="color:#000000;">)<br /></span><span style="color:#999999;">11</span> <span style="color:#000000;">Console.WriteLine(</span><span style="color:#800000;">"</span><span style="color:#800000;">Count: {0}</span><span style="color:#800000;">"</span><span style="color:#000000;">, i);<br /></span><span style="color:#999999;">12</span> <span style="color:#000000;">}<br /></span><span style="color:#999999;">13</span> <span style="color:#000000;">}<br /></span><span style="color:#999999;">14</span> <span style="color:#000000;">}<br /></span><span style="color:#999999;">15</span> <span style="color:#000000;"></span><br />

The examples shown in this series use the C# 3.5 compiler, however I believe the format hasn't changed since C# 1.0.

PDB Header

PDB files start with a pretty large header of 32 bytes, which can be used to identify the file. The following dump shows a header of the resulting PDB file:

<span style="color:#000000;">00000000</span><span style="color:#000000;"> 4D </span><span style="color:#000000;">69</span><span style="color:#000000;"> </span><span style="color:#000000;">63</span><span style="color:#000000;"> </span><span style="color:#000000;">72</span><span style="color:#000000;"> 6F </span><span style="color:#000000;">73</span><span style="color:#000000;"> 6F </span><span style="color:#000000;">66</span><span style="color:#000000;"> </span><span style="color:#000000;">74</span><span style="color:#000000;"> </span><span style="color:#000000;">20</span><span style="color:#000000;"> </span><span style="color:#000000;">43</span><span style="color:#000000;"> 2F </span><span style="color:#000000;">43</span><span style="color:#000000;"> 2B 2B </span><span style="color:#000000;">20</span><span style="color:#000000;"> Microsoft C</span><span style="color:#000000;">/</span><span style="color:#000000;">C</span><span style="color:#000000;">++</span><span style="color:#000000;"><br /></span><span style="color:#000000;">00000010</span><span style="color:#000000;"> 4D </span><span style="color:#000000;">53</span><span style="color:#000000;"> </span><span style="color:#000000;">46</span><span style="color:#000000;"> </span><span style="color:#000000;">20</span><span style="color:#000000;"> </span><span style="color:#000000;">37</span><span style="color:#000000;"> 2E </span><span style="color:#000000;">30</span><span style="color:#000000;"> </span><span style="color:#000000;">30</span><span style="color:#000000;"> 0D 0A 1A </span><span style="color:#000000;">44</span><span style="color:#000000;"> </span><span style="color:#000000;">53</span><span style="color:#000000;"> </span><span style="color:#000000;">00</span><span style="color:#000000;"> </span><span style="color:#000000;">00</span><span style="color:#000000;"> </span><span style="color:#000000;">00</span><span style="color:#000000;"> MSF </span><span style="color:#000000;">7.00</span><span style="color:#000000;">...</span><span style="color:#000000;">DS</span><span style="color:#000000;">...</span><span style="color:#000000;"><br /></span><br />

The header has some interesting properties. First it is an ASCII string, which contains a line break and is zero terminated. This makes it possible to pass a PDB file to the DOS command 'type' and be able to see the version of the PDB file we have:

<span style="color:#000000;">D:</span><span style="color:#000000;">\</span><span style="color:#000000;">My Projects</span><span style="color:#000000;">\</span><span style="color:#000000;">Tests</span><span style="color:#000000;">\</span><span style="color:#000000;">CSharpConsoleBlog</span><span style="color:#000000;">\</span><span style="color:#000000;">CSharpConsoleBlog</span><span style="color:#000000;">></span><span style="color:#0000ff;">type</span><span style="color:#000000;"> Program10</span><span style="color:#000000;">.</span><span style="color:#000000;">pdb<br />Microsoft C</span><span style="color:#000000;">/</span><span style="color:#000000;">C</span><span style="color:#000000;">++</span><span style="color:#000000;"> MSF </span><span style="color:#000000;">7.00</span><span style="color:#000000;"><br /><br />D:</span><span style="color:#000000;">\</span><span style="color:#000000;">My Projects</span><span style="color:#000000;">\</span><span style="color:#000000;">Tests</span><span style="color:#000000;">\</span><span style="color:#000000;">CSharpConsoleBlog</span><span style="color:#000000;">\</span><span style="color:#000000;">CSharpConsoleBlog</span><span style="color:#000000;">></span><br />

You don't get the garbage displayed on the screen but still some valuable information from the file itself. The two letters DS in the header are the initials of Dan Spalding, who owned the linker and much of the PDB code for many years according to Andy Penell.

Next, at byte 0x00000020 starts a structure, which contains a lot of settings and provides a lot of information to a PDB reader:

Field	Size	Meaning
pageSize	4	The size of a page in the file.
bitmapPage	4	The page number of the bitmap page.
filePages	4	The number of pages in the file.
rootBytes	4	The number of bytes in the root stream.
reserved	4	Unused as far as I know.
indexPage	4	The page number of the index page.

Ok, so a PDB file is divided into fixed size pages (size in the pageSize field) and there's a bitmap that specifies if a page is in use or not. Sounds familiar? Well, yes it's the same strategy as used for the FAT file system, OLE compound files and in a lot of other areas.

The filePages field can be used to make sure the PDB file is completely available - multiply pageSize with filePages and you should have the size of the PDB file.

PDB Root Stream

Using the fields we have right now, we still can't unlock the contents of the PDB file. To get there we need to combine the rootBytes and indexPage fields. The indexPage field points to a page, which contains page numbers of the root stream. So it is an array of 4-byte page numbers, which hold the contents of the root stream in order. To determine the number of entries in the index, you must divide rootBytes by pageSize.

If you read all pages in order of the index, you've read the root stream. The root stream tells us, what is contained in the file. The root stream starts with 4 bytes, which tell the total number of streams in the file. An array of stream lengths is located after the stream count, e.g. each entry in this array corresponds to the length of that stream. What follows next is a page index for all streams, e.g. an array of page numbers for stream #1, an array of page numbers for stream #2 etc. The number of entries is again determined by the length of that stream divided by the page size.

The root stream gives us basically an index of all streams available and how they're spread accross the PDB file.

Starting with the next posting, I'll dive into the important streams in order.

Updated my blog style again...

2009-01-17T22:12:00.001+01:00

Ok, this is going to be the last update for some time. I'll add some more categories for postings I've planned, but this will be it - unless I'll find some more bugs somewhere.

Debug Symbols for MOSA #3 - Accessing PDB files

2009-01-16T02:14:00.003+01:00

In the last post, I wrote about the debug symbol formats used by Microsoft in recent years. This post is dedicated to tell you, where to look about accessing these files using official APIs. If the world consisted only of Windows, we could stop here. We wouldn't need to understand the file format itself or be able to read the files without the APIs described below. However the world isn't living in monoculture so I'll keep my goal to describe the PDB format in the next posts.

Essentially Microsoft only makes four APIs available to access debugging symbols:

Image Helper Library (included with Windows, not redistributable)
Debug Help Library (included with Windows, redistributable, updated versions in the Debugging Tools for Windows)
Debug Interface Access SDK (included with Visual Studio)

and the only remaining API is in the .NET System.Diagnostics.SymbolStore namespace in mscorlib.

Of all of these APIs the Image Helper Library provides the most features, followed by the Debug Help Library. The later is used by the Microsoft Debuggers to load symbol information. While both of these libraries are regular Win32 DLLs with WINAPI entry points, the Debug Interface Access SDK provides COM objects to access the contents of symbol files. The library is very easy to work with.

As one can easily see these three options don't work outside the Microsoft world, well they don't except for maybe Wine or ReactOS. My first hope was that using the System.Diagnostics.SymbolStore namespace would be sufficient for our purpose of retrieving the symbol information, but again this quickly makes the code a Windows only option.

From Mike Stall's .NET Debugging Blog, I've taken the following snippet from the sample code of PDB2XML tool - a tool, which uses ISymbolReader to read a PDB file and writes it in an XML file.

        // We demand Unmanaged code permissions because we're reading from the file system and calling out to the Symbol Reader
        // @TODO - make this more specific.
        [System.Security.Permissions.SecurityPermission(System.Security.Permissions.SecurityAction.Demand,
            Flags = System.Security.Permissions.SecurityPermissionFlag.UnmanagedCode)]
        public static ISymbolReader GetSymbolReaderForFile(SymbolBinder binder, string pathModule, string searchPath)
        {
            // Guids for imported metadata interfaces.
            Guid dispenserClassID = new Guid(0xe5cb7a31, 0x7512, 0x11d2, 0x89, 0xce, 0x00, 0x80, 0xc7, 0x92, 0xe5, 0xd8); // CLSID_CorMetaDataDispenser
            Guid dispenserIID = new Guid(0x809c652e, 0x7396, 0x11d2, 0x97, 0x71, 0x00, 0xa0, 0xc9, 0xb4, 0xd5, 0x0c); // IID_IMetaDataDispenser
            Guid importerIID = new Guid(0x7dac8207, 0xd3ae, 0x4c75, 0x9b, 0x67, 0x92, 0x80, 0x1a, 0x49, 0x7d, 0x44); // IID_IMetaDataImport

            // First create the Metadata dispenser.
            object objDispenser;
            NativeMethods.CoCreateInstance(ref dispenserClassID, null, 1, ref dispenserIID, out objDispenser);

            // Now open an Importer on the given filename. We'll end up passing this importer straight
            // through to the Binder.
            object objImporter;
            IMetaDataDispenser dispenser = (IMetaDataDispenser)objDispenser;
            dispenser.OpenScope(pathModule, 0, ref importerIID, out objImporter);

            IntPtr importerPtr = IntPtr.Zero;
            ISymbolReader reader;
            try
            {
                // This will manually AddRef the underlying object, so we need to be very careful to Release it.
                importerPtr = Marshal.GetComInterfaceForObject(objImporter, typeof(IMetadataImport));

                reader = binder.GetReader(importerPtr, pathModule, searchPath);
            }
            finally
            {
                if (importerPtr != IntPtr.Zero)
                {
                    Marshal.Release(importerPtr);
                }
            }
            return reader;
        }

Ouch, that was a lot of code just to get the symbol reader - but wait a minute, what are the Guids doing there and the CoCreateInstance call? This screams trouble for cross platform code... It turns out that ISymbolReader is not useful without an object, which implements the IMetadataImport interface. This is a COM interface implemented by mscoree.dll, the Microsoft .NET Runtime Execution Engine. And you can't get an ISymbolReader without a SymBinder, which is not even defined in the namespace. These GUIDs and the COM classes are defined in ISymWrapper.dll, a COM interop assembly.

But not enough, the ISymbolBinder1 (don't ask, look up ISymbolBinder and ISymbolBinder1 and figure out the reason for ISymbolBinder1) interface uses an IntPtr to access this IMetadataImport interface. Essentially you're passing a COM interface in an unmanaged pointer (or native int in CIL speak) to another unmanaged COM object.

Somehow this is messed up. Really messed up. It looks like symbol information was an afterthought in the development of .NET and hasn't received any priority ever since .NET 1.0 - this mess has been this way since .NET 1.0 was released. I hope that things get better with .NET 4.0, but for some reason I doubt that.

Now we've collected lots of unusable APIs and we still can't read those PDB files anywhere outside of Windows. However this gives us something else: All of those APIs have documented some structures to pass symbol information to the calling application. These structures are very likely to be similar to what is stored on disk - at least these APIs give us some hints that the format is more complex than one might think. And finally the .NET namespace gives us some design guidelines to realize a PDB reader/writer using plain .NET.

New blog design

2009-01-16T01:34:00.001+01:00

Ok, another blog design. The green didn't really look good so now I've been looking on the web for a better blog template and have found some nice ones made by Andreas Viklund. Thank you for sharing these templates.

Debug Symbols for MOSA #2 - Debug Symbol Formats

2009-01-15T14:16:00.007+01:00

This continues the series of Debug Symbols for MOSA, started in Why we need them.

As the title says, there's a multitude of debug symbol formats out there. It usually depends on the operating system, the compiler, compiler switches and linker of choice what kind of symbols are emitted or even worse if any at all. I'm only going to talk about the Microsoft formats, as these are the ones I've actively been working with.

Microsoft has created a multitude of symbol formats in the past, where even PDB files exist in multiple formats. This post sheds some light, which debug symbols are out there.

The general history for Microsoft Symbol Formats is that there are mainly three kinds:

Pre-CodeView
CodeView
PDB

I will not dive into the Pre-CodeView era mainly because I don't have much knowledge about it.

So why does Microsoft change the symbol format all the time? The answer was given by Matt Pietrek. As the compilers and debuggers advanced more information was stored in the file. Some changes were performed due to the 16->32 bit transition, but most changes can be attributed to advances in the debugger. Edit and Continue is an example of this.

CodeView

CodeView is format developed by Microsoft sometime along with the CodeView debugger, which was later integrated into the Microsoft C to become the Visual Studio we know today. There are several revisions of CodeView, which adopt the format to the specific compiler version in use.

There's even a public specification for the CodeView format available various places on the internet.

The CodeView format was stored in various containers (files) over the years, namely the *.dbg files upto Windows 2000 and it is still in use today in the *.pdb files emitted by Visual Studio compilers since around 1997 and more importantly for MOSA: It is also emitted by the .NET compilers.

PDB

PDB files are in use for quite some time now, but even this file format has went through at least three transitions. There's at least one format for managed symbols produced by csc, vbc and the other .NET compilers - yes another format. Again.

CILDB

Microsoft has submitted the Common Language Infrastructure to the ECMA for standardization. The latest standardized edition I'm aware of is ISO/IEC 23271, published on 2006-10-01. Partition V of this standardization defines a Debug Interchange Format, specifically called CILDB. The specification is available for download.

The specification introduction says this:

Portable CILDB files provide a standard way to interchange debugging information between CLI producers and consumers. This partition serves to fill in gaps not covered by metadata, notably the names of local variables and source line correspondences.

Even though Microsoft has pushed this format as part of the specification, no Microsoft tool included with the .NET Framework SDK, the Framework itself or Visual Studio is able to generate or consume these files. So the interchange aspect of this standard is not realized. There are both open and closed source apps, that are able to convert Microsoft PDB files to CILDB - all with a drawback I'll talk about in the next post.

More in the next post. However one last format for managed code remains:

MDB

This is the mono debug format - it is used by mdb and MonoDevelop. There's integrated support in mono using a SymbolWriter/SymbolReader to produce and consume these files from managed code. Talk about fun!

The MDB option is definitely one we should follow to debug applications on MOSA, but it is not one we are able to use for kernel debugging or debugging native code.

Basically this means that mosacl (our ahead of time compiler) must be able to read PDB, MDB and CILDB files in order to map the source code to appropriate places in the native code. Again - more in another post.

Debug Symbols for MOSA #1 - Why we need them

2009-01-14T10:59:00.006+01:00

The MOSA compiler is nearing its 0.1 alpha release and one of the things that has been bugging me since the start was creating debug symbols for compiled assemblies. The MOSA compiler converts CIL assemblies to native code for a specific target architecture. In the process however the mapping of source code to native code gets lost, unless the compiler is able to create new symbol information to map the native code back to the managed source code.

There are various reasons that support is needed for symbols, one of them is that it makes kernel debugging a whole lot easier if the debugger allows stepping in the source code and variable inspection (the Visual Studio experience.) The other point is that various .NET APIs allow creating debug symbols (CodeDOM or Reflection.Emit) or inspecting them using the System.Diagnostics.SymbolStore namespace.

As I'll explain in the posts later this is no easy task, but one that'll raise the productivity of kernel development a lot.

It appears the defense date for my diploma thesis is set

2008-11-10T17:54:00.001+01:00

It's on November 24th 2008 at 9:30 or 10:00 AM. After that I'm finally done with that...

Undocumented CIL Metadata Tables

2008-11-10T17:51:00.001+01:00

For some reason some (IMHO useful) tables in the CIL specification (ISO/IEC 23271, Second edition from 2006-10-01) are not documented. These are mainly the AssemblyOS and AssemblyProcessor tables. As I'm working on finishing the MOSA ahead of time compiler to finally be able to produce bootable images, I think these metadata tables are very useful. What really makes me wonder is that even the constants for these tables seem to be documented. The rdwProcessors field is an array of processor types as defined in winnt.h and the OS version is the same as usually encoded in unmanaged Win32 OSVERSIONINFOEX.

I guess it's time to visit the ReactOS project and to pull these constants out of their repository.

.NET Metadata

2008-11-09T23:32:00.001+01:00

.NET metadata is what powers the platform. Unfortunately the tables in binary files are in typical Microsoft style. Right now I'm trying to rewrite the metadata loader in the mosa runtime to be reusable for our aot compiler. It is just an ugly format to export - at least I can speed up our metadata loading this way too.

MOSA & SharpOS

2008-11-09T23:30:00.001+01:00

As the SharpOS page has gone offline earlier this week/past week (The domain name has expired) I'm wondering how the two projects will relate in the future. MOSA only came to be what it is today by the initial talks tgihil, xfury and I had with other projects, mainly Ensemble and the JNode group.

Admin sein oder nicht?

2004-12-17T20:12:00.000+01:00

Viele Leute installieren sich ein Windows 2000/XP und laufen dann mit dem Nutzerkonto, welches während der Installation angelegt wurde. Was viele jedoch nicht wissen ist, dass dieses Konto (unsinnigerweise) Administrator-Rechte hat.

Viele Viren, Würmer, Trojaner und sonstige Spyware kann sich nur durch diese Rechte installieren. Daher empfehle ich jedem die Rechte der allgemeinen Nutzerkonten auf normale Nutzerrechte zu beschränken.

Falls dies nicht geht, gibts nun doch ein neues Mittel: DropMyRights. Das Programm ermöglicht das starten von anderen Programmen mit anderen Rechten.

Da viele Viren und andere malware sich über den Browser oder den E-Mail client verbreitet, kann man diese mittels DropMyRights nun auch ohne Administratorrechte laufen lassen - obwohl man sonst in einer solchen Umgebung arbeitet.

Weitere Details verrät der (englische) MSDN Artikel: Browsing the Web and Reading E-mail Safely as an Administrator.

Neue Bilder und andere Änderungen

2004-10-10T13:57:00.000+02:00

Ich habe mal wieder Zeit gefunden neue Bilder hochzuladen. Dabei habe ich die Gallerie vom Tim umgestaltet und auch per Passwort geschützt.

Der Passwortschutz dient lediglich der Vorbeugung. Alle Freunde, Verwandte und Familienmitglieder erhalten das Passwort natürlich von Anja oder mir.

Das liebe Essen und unser Tim...

2004-04-27T21:28:00.000+02:00

Diesmal wurde der Tim beim Essen beobachtet. Allerdings war das Vögelchen wahrscheinlich toller und das Essen nur eine leckere Nebensache. ;-)

Hier gibts Bilder von der Raupe mit sieben Köpfen..

Visual C++ Developer Center: Microsoft Visual C++ Toolkit 2003

2004-04-18T18:43:00.000+02:00

Dieser Beitrag hat mal wieder nichts mit dem Tim zu tun ;-).

Microsoft macht mit seinen kostenlosen Entwicklertools weiter: Nachdem im .NET Framework SDK bereits seit mehr als 2 Jahren die Compiler von Visual Basic .NET und C# kostenlos verfügbar sind gibt es nun auch den Microsoft Visual C++ Toolkit kostenlos... Ist das etwa eine neue Strategie um Entwickler von Open Source Plattformen zurückzuholen?

Visual C++ Developer Center: Microsoft Visual C++ Toolkit 2003

Und wenn: Dann sind sie auf dem richtigen Weg.

Tims Lieblingsspielzeug

2004-04-14T21:28:00.000+02:00

Für alle Fans vom Tim gibts heute auch ein Video mit seinem Lieblingsspielzeug - einem kleinen Löwen (eine Rassel) von Fisherprice.

Tim und der Löwe

(Der Ton wurde auf Wunsch des Löwen entfernt. Dem Tier wurde während der Dreharbeiten kein Schaden zugefügt. Video im Microsoft Windows Media Video 9 Format.)

Wieder mal die Fotos...

2004-04-13T02:09:00.000+02:00

Da ich mal wieder im Rückstand bin, habe ich den heutigen (eigentlich schon gestrigen) Tag genutzt neue Fotos hochzuladen. Tim ist im Vergleich zu seinen ersten Tagen schon erstaunlich groß.

Neue Fotos vom Tim