Cci RunAndCollect?

Jan 12, 2011 at 3:43 AM

I'm looking at the HelloAST sample project. I would like to do something similar to this but I need to do it dynamically. Which to me means I need to be able to use the AssemblyBuilderAccess.RunAndCollect flag somehow. How can I make a collectible assembly with Cci?

Coordinator
Jan 12, 2011 at 6:30 AM

You'll have to write a significant amount of code (resuable code that can be contributed back to the project, mind you).

Your best bet would be to clone the CodeModelToIL project. Then change it to first copy the metadata (excluding method bodies) to equivalent System.Reflection.Emit objects (this would be somewhat similar to the code for making a copy of a CCI assembly). Then fill in the methodbodies for the methods using System.Reflection.Emit.ILGenerator.

Of course, it would be magic if we could just load a serialized assembly from a bytestream into the CLR as a collectible assembly, but I'm not aware of that being a possibility. For what it is worth, I asked for this capability back in 2003...

Jan 12, 2011 at 6:35 AM

I was afraid you were going to say that :-S

I proceeded to dig around in Cci and found that it appears to just be writing the complete assembly byte by byte, pretty intense. I was assuming it was just using reflection.emit previously. I'm finding the linq expression API for writing dynamic methods is very limited and buggy and I desperately want to have a single way to write dynamic and saved assemblies. Ugh.

Apr 20, 2011 at 3:12 AM

hi,

just wanted to give you a heads up that i started to work on this.  from a top-down view it looks like this:

using (var host = new PeReader.DefaultHost())
{
  IAssembly assembly = host.LoadUnitFrom("myassembly.dll") as IAssembly;
                
  AssemblyBuilder assemblyBuilder = DynamicAssemblyGenerator.Create(assembly, host);
  if (assemblyBuilder.EntryPoint != null) {
    assemblyBuilder.EntryPoint.Invoke(null, new object[] { args });
  }
}

so far it works very well for very basic assemblies, didn't come to generics or anonymous methods yet.

what i did to make it work was to implement two MetadataTraverser classes. like suggested from hermanv. the first one traverses the metadata excluding method bodies and creates various reflection emit Builder objects, like AssemblyBuilder/ModuleBuilder/etc.
the second one traverses the metadata again including method bodies and uses a big switch statement over each IOperation.OperationCode and emits IL using the Reflection.Emit.ILGenerator class.

i'm pretty sure that a few questions will arise in the next couple of days and therefor wanted to ask where i should put the source code for easier discussions. at first i thought about using git, but maybe it's easier for you to use svn on codeplex? should i open up a new codeplex project for this?

oh and cci is a really wonderful project. thank you for making this open source. like justinc i was using linq expressions at first but it soon came clear that certain things (like defining a generic method) wasn't possible. i'm still getting familiar with the api, but so far i'm really impressed :)

Coordinator
Apr 20, 2011 at 6:37 AM

This sounds really good, but before you sink too much time in it, I had better spill the beans that I have a bunch of code for this sitting on a machine. I've held back on checking it in because I haven't had the time to any testing and I need to write a sample to get people going.

However, in the interest of not having too much duplication of effort, I should probably just check it in so that you can all have a look and perhaps help out.

Apr 20, 2011 at 11:39 AM

yes, check it in please. i'd be eager to help you out with this one.

Coordinator
Apr 20, 2011 at 7:57 PM

Checked in now. It compiles, but I have not yet run a single line of its code.

Apr 21, 2011 at 12:38 AM

Herman,

it looks like you checked in the changes to the solution:
http://cciast.codeplex.com/SourceControl/changeset/changes/61140

But not the new code. I think you need to add the project to the repo still. 

Apr 21, 2011 at 12:39 AM

Nm. Please ignore. I see the changes went into the sister metadata project.

Coordinator
Apr 21, 2011 at 12:40 AM

The code actually lives in ccimetadata.codeplex.com. It should, however, have come down automatically. Have you done a sync?

Apr 21, 2011 at 3:53 AM

great! thank you! did a quick test and ran into a couple of null pointer and not implemented exceptions. i'll see what i can do. i'll keep you updated. with patches hopefully ;)

Apr 21, 2011 at 4:58 AM

azeno, what did you use for the input parameters to dynamicloader? I have created a pdbreader and passed it as both parameters. Also, out of curiosity what's your project?

Apr 21, 2011 at 1:18 PM
Edited Apr 21, 2011 at 1:37 PM

i used null, null. but that didn't seem the problem though. notimplemented exception was thrown in a CreateLabelsForBranchTargets method. kept me wondering though, why even create labels and mark them in the stream. in my implementation i simply used the already computed offset for a branch instruction, like it is done in the PeWriter.

i'm working on vvvv.org, writing on a compiler for a visual programming language.

Apr 23, 2011 at 5:18 PM

So you're saying if I remove that throw statement then it will work? I already fixed the two null ref and stackoverflow errors locally.

Apr 24, 2011 at 3:22 AM

no, sorry, you misunderstood me. it won't work. i'll post a patch to herman's code once i've done some cleanup locally on my machine. i'm struggling with assemblies containing generic types right now. i'd like to get that working before coming up with a patch.

Apr 26, 2011 at 2:42 PM
Edited Apr 26, 2011 at 2:42 PM

i've posted a first patch here:

http://ccimetadata.codeplex.com/workitem/6087

Apr 30, 2011 at 5:20 PM

I'm trying your patch now and getting a StackOverflowException in Mapper.GetType. I'm still investigating but I believe the type that it is having troubles with is IEnumerable<object>.

Apr 30, 2011 at 5:37 PM

This is the repeating callstack:

 Microsoft.Cci.PeReader.DLL!Microsoft.Cci.MetadataReader.ObjectModelImplementation.NamespaceTypeGenericInstanceReference.DispatchAsReference(Microsoft.Cci.IMetadataVisitor visitor) Line 5132 + 0x2a bytes	C#
 Microsoft.Cci.ReflectionEmitter.DLL!Microsoft.Cci.ReflectionEmitter.ReflectionMapper.GetType(Microsoft.Cci.ITypeReference type) Line 268 + 0x37 bytes	C#
 Microsoft.Cci.ReflectionEmitter.DLL!Microsoft.Cci.ReflectionEmitter.ReflectionMapper.GetMethod(Microsoft.Cci.IMethodReference methodReference, bool useCache) Line 170 + 0x69 bytes	C#
 Microsoft.Cci.ReflectionEmitter.DLL!Microsoft.Cci.ReflectionEmitter.MappingVisitorForTypes.Visit(Microsoft.Cci.IGenericMethodParameterReference genericMethodParameterReference) Line 354 + 0x5b bytes	C#
 Microsoft.Cci.PeReader.DLL!Microsoft.Cci.MetadataReader.ObjectModelImplementation.SignatureGenericMethodParameter.DispatchAsReference(Microsoft.Cci.IMetadataVisitor visitor) Line 3318 + 0x2a bytes	C#
 Microsoft.Cci.ReflectionEmitter.DLL!Microsoft.Cci.ReflectionEmitter.ReflectionMapper.GetType(Microsoft.Cci.ITypeReference type) Line 268 + 0x37 bytes	C#
 Microsoft.Cci.ReflectionEmitter.DLL!Microsoft.Cci.ReflectionEmitter.MappingVisitorForTypes.GetConsolidatedTypeArguments(System.Collections.Generic.List consolidatedTypeArguments, Microsoft.Cci.ITypeReference typeReference) Line 373 + 0x3a bytes	C#
 Microsoft.Cci.ReflectionEmitter.DLL!Microsoft.Cci.ReflectionEmitter.MappingVisitorForTypes.Visit(Microsoft.Cci.IGenericTypeInstanceReference genericTypeInstanceReference) Line 364 + 0x1a bytes	C#
 Microsoft.Cci.PeReader.DLL!Microsoft.Cci.MetadataReader.ObjectModelImplementation.NamespaceTypeGenericInstanceReference.DispatchAsReference(Microsoft.Cci.IMetadataVisitor visitor) Line 5132 + 0x2a bytes	C#

I have declared an extension method in an already compiled assembly that looks sort of like this:

public static IEnumerable<T> Cast<T>(this object value) { ... }

In my ast I have a reference to this method that would look like this code:

IEnumerable<object> value = item.Cast<object>();

 

It appears that the stack overflow is happening when trying to resolve the generic parameters for the Cast<> method. 

May 2, 2011 at 12:58 PM

hi justin, i'm not able to reproduce this. maybe my example is too simple? i tried this:

  public static void Main() {
string[] arr = {"a", "b", "c", "d", "e"};
var asEnumerable = arr.Cast<string>();
}

public static class EnumerableExtensions { public static IEnumerable<T> Cast<T>(this object value) { return value as IEnumerable<T>; } }

May 2, 2011 at 1:47 PM
Edited May 2, 2011 at 1:48 PM

try:

 

var obj = new object();
var asEnumerable = obj.Cast<object>();

 

 

But if that doesn't do it then I'll have to keep debugging.