Out of memory exception can be thrown when the process has memory left but there is no segment large enough to store the object in. This happens when the large object heap get fragmented (normally by large string such as XML documents or ViewState strings).
Troubleshooting
1) Start
PerfMon and add the conter ".NET CLR Memory->#Bytes in all heaps"
2) when the high allocation of memory starts take a dump in hang mode
3) wait about 100 Mb of allocated memory and the take dump2 in hang mode
Open the both dumps in
windbg and examine the differences between the large object heaps in both dumps.
To show the size of all heaps
!eeheaps –gc
If the large object heap has allocated most of the allocated memory, examine it
!dumpheap –min 85000
List all objects of the large object heap (because it bigger than 85000 bytes).
To examine what the object includes take the address of one of the objects and dump it
!dumpobj <address>
To see what code that is holding this object use
!gcroot <address>
External links
Large Object Heap Uncovered by Maoni Stephens