T
The Haunt
Guest
I am having some issues writing a large set of data to an XML file. I am using the following class to serialize objects to xml and then write them to disk:
<summary>
Borrowed from http://icanmakethiswork.blogspot.ca/2012/11/xsdxml-schema-generator-xsdexe-taking.html
</summary>
<typeparam name="T"></typeparam>
<remarks></remarks>
Public Class XMLConverter(Of T)
Private Shared serializer As XmlSerializer = Nothing
<summary>
Static constructor that initialises the serializer for this type
</summary>
Shared Sub New()
serializer = New XmlSerializer(GetType(T))
End Sub
<summary>
Write a node to an xmlwriter
</summary>
<param name="writer"></param>
<param name="itemToAppend">the object to be converted and written</param>
<remarks></remarks>
Public Shared Sub AppendToXml(writer As XmlWriter, itemToAppend As T)
Dim strObj As String = ToXML(itemToAppend)
strObj = XMLCleaner.CleanResult(strObj)
writer.WriteRaw(strObj)
writer.Flush()
strObj = Nothing
End Sub
<summary>
Serialize the supplied object into a string of XML
</summary>
<param name="obj"></param>
<returns></returns>
Public Shared Function ToXML(obj As T) As String
Dim strXml As String = ""
Using memoryStream As New MemoryStream()
serializer.Serialize(memoryStream, obj)
memoryStream.Position = 0
Using sr As New StreamReader(memoryStream)
strXml = sr.ReadToEnd()
End Using
End Using
Return strXml
End Function
End Class
Public Class XMLCleaner
This is just for removing junk and slightly modifying the output
Public Shared Function CleanResult(result As String) As String
Dim retVal As String = Regex.Replace(result, "\sxmlns.+?"".*?""", "")
retVal = Regex.Replace(retVal, "SavedSearchRecord", "Record")
retVal = retVal.Replace("<?xml version=""1.0""?>", "")
retVal = Regex.Replace(retVal, vbCrLf, vbCrLf & " ")
Return retVal
End Function
End Class
And am calling this like so:
XMLConverter(Of SavedSearchRecord).AppendToXml(writer, record)
The issue is that memory is quickly being accumulated as I append new records to the file and ultimately results in an out of memory exception.
Ive seen that not caching the serializer can result in this behaviour, but I think Ive sidestepped that issue in my implementation. (Please correct me if I am wrong).
After examining a memory dump:
716821b4 28535 10497120 System.String
71682b74 140213 145562968 System.Char[]
71685670 140258 758802112 System.Byte[]
I can see that I have an enormous number of byte & char arrays getting stuck in memory. The data in the arrays leads me to believe that these are being stranded in memory by the call to the serializer (serializer.Serialize(memoryStream, obj)) as they contain the unmodified serialized object strings.
Given that the memory stream is in a Using block, I cant figure out why these byte arrays are not being collected by the GC. I looked briefly into the XMLSerializer class and I think the char arrays are being created within the serializer (it seems to conver the stream to a writer).
Can anyone tell me how to prevent this code from culminating in out of memory exceptions? Or more importantly explain why the serializer isnt releasing memory as it seems like it should?
FYI code is written using .NET 4.0
Continue reading...
<summary>
Borrowed from http://icanmakethiswork.blogspot.ca/2012/11/xsdxml-schema-generator-xsdexe-taking.html
</summary>
<typeparam name="T"></typeparam>
<remarks></remarks>
Public Class XMLConverter(Of T)
Private Shared serializer As XmlSerializer = Nothing
<summary>
Static constructor that initialises the serializer for this type
</summary>
Shared Sub New()
serializer = New XmlSerializer(GetType(T))
End Sub
<summary>
Write a node to an xmlwriter
</summary>
<param name="writer"></param>
<param name="itemToAppend">the object to be converted and written</param>
<remarks></remarks>
Public Shared Sub AppendToXml(writer As XmlWriter, itemToAppend As T)
Dim strObj As String = ToXML(itemToAppend)
strObj = XMLCleaner.CleanResult(strObj)
writer.WriteRaw(strObj)
writer.Flush()
strObj = Nothing
End Sub
<summary>
Serialize the supplied object into a string of XML
</summary>
<param name="obj"></param>
<returns></returns>
Public Shared Function ToXML(obj As T) As String
Dim strXml As String = ""
Using memoryStream As New MemoryStream()
serializer.Serialize(memoryStream, obj)
memoryStream.Position = 0
Using sr As New StreamReader(memoryStream)
strXml = sr.ReadToEnd()
End Using
End Using
Return strXml
End Function
End Class
Public Class XMLCleaner
This is just for removing junk and slightly modifying the output
Public Shared Function CleanResult(result As String) As String
Dim retVal As String = Regex.Replace(result, "\sxmlns.+?"".*?""", "")
retVal = Regex.Replace(retVal, "SavedSearchRecord", "Record")
retVal = retVal.Replace("<?xml version=""1.0""?>", "")
retVal = Regex.Replace(retVal, vbCrLf, vbCrLf & " ")
Return retVal
End Function
End Class
And am calling this like so:
XMLConverter(Of SavedSearchRecord).AppendToXml(writer, record)
The issue is that memory is quickly being accumulated as I append new records to the file and ultimately results in an out of memory exception.
Ive seen that not caching the serializer can result in this behaviour, but I think Ive sidestepped that issue in my implementation. (Please correct me if I am wrong).
After examining a memory dump:
716821b4 28535 10497120 System.String
71682b74 140213 145562968 System.Char[]
71685670 140258 758802112 System.Byte[]
I can see that I have an enormous number of byte & char arrays getting stuck in memory. The data in the arrays leads me to believe that these are being stranded in memory by the call to the serializer (serializer.Serialize(memoryStream, obj)) as they contain the unmodified serialized object strings.
Given that the memory stream is in a Using block, I cant figure out why these byte arrays are not being collected by the GC. I looked briefly into the XMLSerializer class and I think the char arrays are being created within the serializer (it seems to conver the stream to a writer).
Can anyone tell me how to prevent this code from culminating in out of memory exceptions? Or more importantly explain why the serializer isnt releasing memory as it seems like it should?
FYI code is written using .NET 4.0
Continue reading...