Latest rev (45605) broken?

Apr 27, 2010 at 3:07 PM

I'm working on an assembly modifier, but I'm having issues rewriting my assembly. I'm getting a NullRefException. (See below, after the message)

Then I went ahead and ran RoundtripTests, which failed on test Repro5WithIL. I also ran pereadertests, which fails on ModuleReaderTestClass.MetadataReaderTests.

All my assembly modifier does, is load the assembly, (including mutable CodeModel), and write it back to disk.

 

With kind regards,

Matthijs ter Woord

 

Exception dump of the NRE:

 

Exception occurred while running PostCompiler: [E:\ter Woord\Framework\Source\Foundation\Core\Framework.Core\Framework.Core.csproj]
NullReferenceException: Object reference not set to an instance of an object. [E:\ter Woord\Framework\Source\Foundation\Core\Framework.Core\Framework.Core.csproj]
   at Microsoft.Cci.MutableCodeModel.IteratorClosureInformation.GetReferenceOfFieldUsedByPeers(IFieldDefinition fieldDef) in e:\Libraries\cciast\Sources\MutableCodeModel\Normalizer\Helper.cs:line 69 [E:\ter Woord\Framework\Source\Foundation\Core\Framework.Core\Framework.Core.csproj]
   at Microsoft.Cci.MutableCodeModel.CopyToIteratorClosure.Visit(ThisReference thisReference) in e:\Libraries\cciast\Sources\MutableCodeModel\Normalizer\Duplicator.cs:line 89 [E:\ter Woord\Framework\Source\Foundation\Core\Framework.Core\Framework.Core.csproj]
   at Microsoft.Cci.MutableCodeModel.CodeMutator.CreateMutableType.Visit(IThisReference thisReference) in e:\Libraries\cciast\Sources\MutableCodeModel\Mutator.cs:line 2104 [E:\ter Woord\Framework\Source\Foundation\Core\Framework.Core\Framework.Core.csproj]
   at Microsoft.Cci.MutableCodeModel.ThisReference.Dispatch(ICodeVisitor visitor) in e:\Libraries\cciast\Sources\MutableCodeModel\Expressions.cs:line 2747 [E:\ter Woord\Framework\Source\Foundation\Core\Framework.Core\Framework.Core.csproj]
   at Microsoft.Cci.MutableCodeModel.CodeMutator.Visit(IExpression expression) in e:\Libraries\cciast\Sources\MutableCodeModel\Mutator.cs:line 1263 [E:\ter Woord\Framework\Source\Foundation\Core\Framework.Core\Framework.Core.csproj]
   at Microsoft.Cci.MutableCodeModel.CodeMutator.Visit(MethodCall methodCall) in e:\Libraries\cciast\Sources\MutableCodeModel\Mutator.cs:line 759 [E:\ter Woord\Framework\Source\Foundation\Core\Framework.Core\Framework.Core.csproj]
   at Microsoft.Cci.MutableCodeModel.CodeMutator.CreateMutableType.Visit(IMethodCall methodCall) in e:\Libraries\cciast\Sources\MutableCodeModel\Mutator.cs:line 1874 [E:\ter Woord\Framework\Source\Foundation\Core\Framework.Core\Framework.Core.csproj]
   at Microsoft.Cci.MutableCodeModel.MethodCall.Dispatch(ICodeVisitor visitor) in e:\Libraries\cciast\Sources\MutableCodeModel\Expressions.cs:line 1903 [E:\ter Woord\Framework\Source\Foundation\Core\Framework.Core\Framework.Core.csproj]
   at Microsoft.Cci.MutableCodeModel.CodeMutator.Visit(IExpression expression) in e:\Libraries\cciast\Sources\MutableCodeModel\Mutator.cs:line 1263 [E:\ter Woord\Framework\Source\Foundation\Core\Framework.Core\Framework.Core.csproj]
   at Microsoft.Cci.MutableCodeModel.CodeMutator.Visit(ExpressionStatement expressionStatement) in e:\Libraries\cciast\Sources\MutableCodeModel\Mutator.cs:line 561 [E:\ter Woord\Framework\Source\Foundation\Core\Framework.Core\Framework.Core.csproj]
   at Microsoft.Cci.MutableCodeModel.CodeMutator.CreateMutableType.Visit(IExpressionStatement expressionStatement) in e:\Libraries\cciast\Sources\MutableCodeModel\Mutator.cs:line 1704 [E:\ter Woord\Framework\Source\Foundation\Core\Framework.Core\Framework.Core.csproj]
   at Microsoft.Cci.MutableCodeModel.ExpressionStatement.Dispatch(ICodeVisitor visitor) in e:\Libraries\cciast\Sources\MutableCodeModel\Statements.cs:line 532 [E:\ter Woord\Framework\Source\Foundation\Core\Framework.Core\Framework.Core.csproj]
   at Microsoft.Cci.MutableCodeModel.CodeMutator.Visit(IStatement statement) in e:\Libraries\cciast\Sources\MutableCodeModel\Mutator.cs:line 1273 [E:\ter Woord\Framework\Source\Foundation\Core\Framework.Core\Framework.Core.csproj]
   at Microsoft.Cci.MutableCodeModel.CodeMutator.Visit(List`1 statements) in e:\Libraries\cciast\Sources\MutableCodeModel\Mutator.cs:line 1285 [E:\ter Woord\Framework\Source\Foundation\Core\Framework.Core\Framework.Core.csproj]
   at Microsoft.Cci.MutableCodeModel.CodeMutator.Visit(BlockStatement blockStatement) in e:\Libraries\cciast\Sources\MutableCodeModel\Mutator.cs:line 287 [E:\ter Woord\Framework\Source\Foundation\Core\Framework.Core\Framework.Core.csproj]
   at Microsoft.Cci.MutableCodeModel.CopyToIteratorClosure.Visit(BlockStatement blockStatement) in e:\Libraries\cciast\Sources\MutableCodeModel\Normalizer\Duplicator.cs:line 58 [E:\ter Woord\Framework\Source\Foundation\Core\Framework.Core\Framework.Core.csproj]
   at Microsoft.Cci.MutableCodeModel.CodeMutator.CreateMutableType.Visit(IBlockStatement block) in e:\Libraries\cciast\Sources\MutableCodeModel\Mutator.cs:line 1486 [E:\ter Woord\Framework\Source\Foundation\Core\Framework.Core\Framework.Core.csproj]
   at Microsoft.Cci.MutableCodeModel.BlockStatement.Dispatch(ICodeVisitor visitor) in e:\Libraries\cciast\Sources\MutableCodeModel\Statements.cs:line 133 [E:\ter Woord\Framework\Source\Foundation\Core\Framework.Core\Framework.Core.csproj]
   at Microsoft.Cci.MutableCodeModel.CodeMutator.Visit(IStatement statement) in e:\Libraries\cciast\Sources\MutableCodeModel\Mutator.cs:line 1273 [E:\ter Woord\Framework\Source\Foundation\Core\Framework.Core\Framework.Core.csproj]
   at Microsoft.Cci.MutableCodeModel.CodeMutator.Visit(ConditionalStatement conditionalStatement) in e:\Libraries\cciast\Sources\MutableCodeModel\Mutator.cs:line 413 [E:\ter Woord\Framework\Source\Foundation\Core\Framework.Core\Framework.Core.csproj]
   at Microsoft.Cci.MutableCodeModel.CodeMutator.CreateMutableType.Visit(IConditionalStatement conditionalStatement) in e:\Libraries\cciast\Sources\MutableCodeModel\Mutator.cs:line 1566 [E:\ter Woord\Framework\Source\Foundation\Core\Framework.Core\Framework.Core.csproj]
   at Microsoft.Cci.MutableCodeModel.ConditionalStatement.Dispatch(ICodeVisitor visitor) in e:\Libraries\cciast\Sources\MutableCodeModel\Statements.cs:line 330 [E:\ter Woord\Framework\Source\Foundation\Core\Framework.Core\Framework.Core.csproj]
   at Microsoft.Cci.MutableCodeModel.CodeMutator.Visit(IStatement statement) in e:\Libraries\cciast\Sources\MutableCodeModel\Mutator.cs:line 1273 [E:\ter Woord\Framework\Source\Foundation\Core\Framework.Core\Framework.Core.csproj]
   at Microsoft.Cci.MutableCodeModel.CodeMutator.Visit(List`1 statements) in e:\Libraries\cciast\Sources\MutableCodeModel\Mutator.cs:line 1285 [E:\ter Woord\Framework\Source\Foundation\Core\Framework.Core\Framework.Core.csproj]
   at Microsoft.Cci.MutableCodeModel.CodeMutator.Visit(BlockStatement blockStatement) in e:\Libraries\cciast\Sources\MutableCodeModel\Mutator.cs:line 287 [E:\ter Woord\Framework\Source\Foundation\Core\Framework.Core\Framework.Core.csproj]
   at Microsoft.Cci.MutableCodeModel.CopyToIteratorClosure.Visit(BlockStatement blockStatement) in e:\Libraries\cciast\Sources\MutableCodeModel\Normalizer\Duplicator.cs:line 58 [E:\ter Woord\Framework\Source\Foundation\Core\Framework.Core\Framework.Core.csproj]
   at Microsoft.Cci.MutableCodeModel.IteratorClosureGenerator.TranslateIteratorMethodBodyToMoveNextBody(IteratorClosureInformation iteratorClosure, BlockStatement blockStatement) in e:\Libraries\cciast\Sources\MutableCodeModel\Normalizer\Normalizer.cs:line 1250 [E:\ter Woord\Framework\Source\Foundation\Core\Framework.Core\Framework.Core.csproj]
   at Microsoft.Cci.MutableCodeModel.IteratorClosureGenerator.CreateMoveNextMethod(IteratorClosureInformation iteratorClosure, BlockStatement blockStatement) in e:\Libraries\cciast\Sources\MutableCodeModel\Normalizer\Normalizer.cs:line 1231 [E:\ter Woord\Framework\Source\Foundation\Core\Framework.Core\Framework.Core.csproj]
   at Microsoft.Cci.MutableCodeModel.IteratorClosureGenerator.CreateIteratorClosureMethods(IteratorClosureInformation iteratorClosure, BlockStatement blockStatement) in e:\Libraries\cciast\Sources\MutableCodeModel\Normalizer\Normalizer.cs:line 1196 [E:\ter Woord\Framework\Source\Foundation\Core\Framework.Core\Framework.Core.csproj]
   at Microsoft.Cci.MutableCodeModel.IteratorClosureGenerator.CreateIteratorClosure(BlockStatement blockStatement) in e:\Libraries\cciast\Sources\MutableCodeModel\Normalizer\Normalizer.cs:line 1115 [E:\ter Woord\Framework\Source\Foundation\Core\Framework.Core\Framework.Core.csproj]
   at Microsoft.Cci.MutableCodeModel.IteratorClosureGenerator.CompileIterator(IBlockStatement block) in e:\Libraries\cciast\Sources\MutableCodeModel\Normalizer\Normalizer.cs:line 829 [E:\ter Woord\Framework\Source\Foundation\Core\Framework.Core\Framework.Core.csproj]
   at Microsoft.Cci.MutableCodeModel.MethodBodyNormalizer.GetNormalizedIteratorBody(IBlockStatement body, IMethodDefinition method, IMethodContract methodContract, List`1 privateHelperTypes) in e:\Libraries\cciast\Sources\MutableCodeModel\Normalizer\Normalizer.cs:line 124 [E:\ter Woord\Framework\Source\Foundation\Core\Framework.Core\Framework.Core.csproj]
   at Microsoft.Cci.MutableCodeModel.MethodBodyNormalizer.GetNormalizedSourceMethodBodyFor(IMethodDefinition method, IBlockStatement body) in e:\Libraries\cciast\Sources\MutableCodeModel\Normalizer\Normalizer.cs:line 77 [E:\ter Woord\Framework\Source\Foundation\Core\Framework.Core\Framework.Core.csproj]
   at Microsoft.Cci.MutableCodeModel.SourceMethodBody.GenerateIL() in e:\Libraries\cciast\Sources\MutableCodeModel\Members.cs:line 101 [E:\ter Woord\Framework\Source\Foundation\Core\Framework.Core\Framework.Core.csproj]
   at Microsoft.Cci.MutableCodeModel.SourceMethodBody.Microsoft.Cci.IMethodBody.get_PrivateHelperTypes() in e:\Libraries\cciast\Sources\MutableCodeModel\Members.cs:line 240 [E:\ter Woord\Framework\Source\Foundation\Core\Framework.Core\Framework.Core.csproj]
   at Microsoft.Cci.PeWriter.CreateIndicesFor(ITypeDefinition typeDef) in e:\Libraries\cciast\Metadata\Sources\PeWriter\PeWriter.cs:line 451 [E:\ter Woord\Framework\Source\Foundation\Core\Framework.Core\Framework.Core.csproj]
   at Microsoft.Cci.PeWriter.CreateIndices() in e:\Libraries\cciast\Metadata\Sources\PeWriter\PeWriter.cs:line 405 [E:\ter Woord\Framework\Source\Foundation\Core\Framework.Core\Framework.Core.csproj]
   at Microsoft.Cci.PeWriter.WritePeToStream(IModule module, IMetadataHost host, Stream stream, ISourceLocationProvider sourceLocationProvider, ILocalScopeProvider localScopeProvider, IPdbWriter pdbWriter) in e:\Libraries\cciast\Metadata\Sources\PeWriter\PeWriter.cs:line 228 [E:\ter Woord\Framework\Source\Foundation\Core\Framework.Core\Framework.Core.csproj]
   at TerWoord.PostCompiler.BaseEngine.Execute(String assemblyFile, String outputFile) in E:\ter Woord\PostCompiler\Source\Core\BaseEngine.cs:line 101 [E:\ter Woord\Framework\Source\Foundation\Core\Framework.Core\Framework.Core.csproj]
   at TerWoord.PostCompiler.MSBuild.Tasks.PostCompiler.Execute() in E:\ter Woord\PostCompiler\Source\Frontends\MSBuild.Tasks\PostCompiler.cs:line 76 [E:\ter Woord\Framework\Source\Foundation\Core\Framework.Core\Framework.Core.csproj]
Apr 30, 2010 at 8:22 AM

This issue is related to implementing enumerators as described here: http://msdn.microsoft.com/en-us/library/dscyy5s0%28VS.80%29.aspx

I had 1 enumerator implemented that way. Once I implemented it without yield return statements (ie, with a Enumerable<T> and Enumerator<T> implementation), it works.

 

 

Coordinator
May 5, 2010 at 7:08 PM

The test suite passes for me. If it still fails for you it would be good to try and understand why.

May 6, 2010 at 8:05 AM

First, in response to my initial response (that it works): it doesnt: Trying on a real life assembly gives all kind of errors in peverify (parameter count mismatches, type parameter count mismatches, wrong tokens, etc)

If you want, i can send the assembly in question to you privately (assuming you're only going to use it to test cci stuff....)

 

Now onto the tests.

This is running on a Intel Q9550, with 8GB ram, on a Raid0 setup. WIndows 7 Ultimate x64.

running PEReaderTests.exe:

C:\Data\Sources\Libraries\cci-temp\bin\Debug>PEReaderTests.exe
xUnit.net console runner (xunit.dll version 1.4.9.1416)
Test assembly: C:\Data\Sources\Libraries\cci-temp\bin\Debug\PEReaderTests.exe
F
Total tests: 1, Failures: 1, Skipped: 0, Time: 0.414 seconds

Tests failed:

1) ModuleReaderTests.ModuleReaderTestClass.MetadataReaderTests : System.Reflection.TargetInvocationException : Exception has been thrown by the target of an invocation.
---- System.InvalidCastException : Unable to cast object of type 'Microsoft.Cci.DummyModule' to type 'Microsoft.Cci.IAssembly'.
Stack Trace:
   at System.RuntimeTypeHandle.CreateInstance(RuntimeType type, Boolean publicOnly, Boolean noCheck, Boolean& canBeCached, RuntimeMethodHandle& ctor, Boolean& bNeedSecurityCheck)
   at System.RuntimeType.CreateInstanceSlow(Boolean publicOnly, Boolean fillCache)
   at System.RuntimeType.CreateInstanceImpl(Boolean publicOnly, Boolean skipVisibilityChecks, Boolean fillCache)
   at System.Activator.CreateInstance(Type type, Boolean nonPublic)
   at Xunit.Sdk.LifetimeCommand.Execute(Object testClass)
   at Xunit.Sdk.ExceptionWrapperCommand.<>c__DisplayClass1.b__0()
   at Xunit.Sdk.ExceptionWrapperCommand.RunAndCatch(ExecutionDelegate delegate)
----- Inner Stack Trace -----
   at ModuleReaderTests.ModuleReaderTestClass..ctor() in c:\Data\Sources\Libraries\cci-temp\metadata\Tests\PEReaderTests\ModuleReaderTests.cs:line 393

I tried running roundtriptests, but it seems to be broken (test itself) in how it finds peverify.exe. (could not find peverify.exe under %programfiles%\Microsoft SDKs\Windows\...)

 

I'm trying using ast revision 45605, and metadata version 45604.

Let me know if you need more info from me, as I'm eager to use the cci stuff....

 

 

Coordinator
May 12, 2010 at 3:05 AM

I have found and fixed a problem with Repro5WithIL. I still do not see the problem with PEReaderTests. If you could debug the test and describe why it is failing on your machine, that should be a big help.

As for the problem with your assembly modifier, the problem is probably a bug in the decompiler. Please provide me with a way to reproduce the problem. (The smaller the repro, the more likely I'll be able to respond quickly.)

May 12, 2010 at 1:11 PM
Edited May 12, 2010 at 2:08 PM

Regarding PEReaderTests. Something weird is going on: When doing a "msbuild /p:OutputPath=<dir>" compilation, running PEReaderTests from <dir> gives an error telling that Metadatahelper assembly cant be found. When running from the PEReaderTests\bin\Debug directory, it runs fine. So I think we should somehow blaim this one to "not-so-well-written-tests" ? (no offense to anyone, though..)

I'm preparing a test project, which gives some issues. (already have 1 sample)

 

May 12, 2010 at 2:15 PM
Edited May 12, 2010 at 2:27 PM

For attachment, see: www.terwoord.nl/CCIRepro.zip

 

OK. I prepared some tests in a small test project. Several different situations fail (although I'm not sure if they're related..).

See the attachment.

if you run the project without any changes, it should not show you an exception.

 

Steps to repro all issues. (described incremental, ie, not disabling old steps)

  1. open TestAssembly\DebugHelper.cs, and uncomment the two lines (assignment statement, and Console.WriteLine call)
  2. if you run now, you'll see that it throws an exception. bug #1 ?
  3. Next one: Open up TestAssembly\Iterators.cs, and uncomment the method. run and see an exception (bug #2)
  4. uncomment the code in TestAssembly\LambdaError.cs (#3)
  5. uncomment the line in TestAssembly\SimpleTestClass.cs (#4) . For this issue to showup, you need to run, and then run peverify on the out.dll file in the TestTool\bin\Debug dir.

 

I think #1, #2 and #3 might be related (as they all use instance fields), but #4 seems isolated.

If you have any questions, don't hesitate to ask them.. :-)

 

Regards,

Matthijs ter Woord

 

PS. I found out I missed a few things, but found out incrementally. I hope you didnt get an email for each and every change. If so, please accept my apologies...

Coordinator
May 14, 2010 at 11:41 PM

Your test project does not seem to have a direct bearing on why the check in tests are failing for you, but perhaps I'm missing something.

Looking at your code, something stands out: You are running MethodBodyCodeAndContractMutator over an entire assembly. This mutator assumes that it runs only over a single method body in the context of an existing assembly, so it is not very surprising that you are seeing weird behavior.

The mutator classes have turned out to be very hard to use in practise. We are busy deprecating them. Please have a look at the CodeCopier class in MutableCodeModel and use it to first make a complete (mutable) copy of the assembly that you want to modify. Then use the CodeMutatingVisitor to visit the copy (while modifying it in place).

This way, the mutator code does not have to worry about the mutable model being in an inconsistent state (because it is still being constructed) and it is also much easier to avoid polluting the copy with references to nodes from the original.

May 15, 2010 at 10:07 AM
These great hints help solve most issues, except one. I uploaded a new test project to www.terwoord.nl/CCIRepro.zip

On a side note: probably when deprecating classes, you should be marking them with [Obsolete]

Right now my sample project runs without errors, but the output assembly doesn't validate using peverify. It's adding a ldarg.1 somewhere to a method with only 1 argument.

Please let me know if you are able to reproduce this one..


May 15, 2010 at 11:22 PM
Edited May 15, 2010 at 11:30 PM

Moved to another thread.

May 16, 2010 at 7:40 AM
Perplexed: i think you should start your own thread, so we can focus on the issue I'm having.

But you're trying to mutate something with the Copier. Herman told me the following:

get a Mutable copy of the tree using the copier, and then Mutate it with CodeMutatingVisitor

Herman: I do agree that documentation is very much absent. I'm not even talking about reference docs, even simple Getting Started tutorials would help a lot (ie, make a simple modifier replacing method X with method Y, etc)



On Sun, May 16, 2010 at 1:22 AM, perplexed <notifications@codeplex.com> wrote:

From: perplexed

"The mutator classes have turned out to be very hard to use in practise. We are busy deprecating them"

This is the second time you mention CodeCopier exists.

How should people even know you added them?

 

Anyway, now that I know it exists, I'm trying to use it to complete the pseudo source generator and to merge two method bodies and I don't even understand how I should startt using it.

I mean it: I don't even understand what method should I call to copy something.

I found no examples.

Not to mention the fact that when I read about "cones" I can only imagine what you are talking about but I have no precise idea.

Apparently mterwoord understood how it works right away, so I must be stupid. Could you or somebody help me?

 

Read the full discussion online.

To add a post to this discussion, reply to this email (cciast@discussions.codeplex.com)

To start a new discussion for this project, email cciast@discussions.codeplex.com

You are receiving this email because you subscribed to this discussion on CodePlex. You can unsubscribe on CodePlex.com.

Please note: Images and attachments will be removed from emails. Any posts to this discussion will also be available online at CodePlex.com


May 16, 2010 at 12:16 PM

mterwoord, thanks for your reply. Actually I moved my question to another thread few minutes after I posted it and before you answered. But I see you read/answered via email so you probably received everything with a delay. Thanks again. Sorry for messing this thread up. :)

May 16, 2010 at 2:17 PM
Herman: It also seems that by using the CodeMutatingVisitor, the methods arent decompiled into Source-like representation (MethodCall, etc).

What am I doing wrong this time? :-)


May 16, 2010 at 2:55 PM

Maybe you have to manually replace the metadata method body representation with a code model method representation using ILToCodeModel.

I did that yesterday.

May 16, 2010 at 4:08 PM
Well, problem is not getting the code model representation, but getting that representation into an assembly again (which verifies with peverify.exe)

May 16, 2010 at 4:22 PM

I misunderstood "the methods arent decompiled into Source-like representation"

 

Coordinator
May 22, 2010 at 1:29 AM

This thread has meandered a bit and I'm not quite sure what the issue is any more.