D
dbasnett
Guest
Early in the development of Missouri Revisor of Statutes - Revised Statutes of Missouri, RSMo, Missouri Law, MO Law, Joint Committee on Legislative Research it became clear that we needed some kind of real-time debugging aids due to our inexperience with .ASPX.
What we settled on was an IPC communication strategy between the web server and a console application using NamedPipes which worked well unless the console application wasn't running. Here is a sample of the output from the console application.
14:21:03. 40.77.167.80 msnbot-40-77-167-80.search.msn.com /main/OneSection.aspx?section=210.526
14:21:03. 107.129.78.67 107-129-78-67.lightspeed.spfdmo.sbcglobal.net /main/OneSection.aspx?section=474.120&bid=26270&hl=
14:21:03. 71.10.208.80 071-010-208-080.biz.spectrum.com /main/OneSection.aspx?section=305.150&bid=16296
14:21:03. 173.170.42.170 173-170-42-170.res.bhn.net /main/OneSection.aspx?section=140.260&bid=6942&hl=excess+proceeds%u2044
14:21:04. 71.10.208.80 071-010-208-080.biz.spectrum.com /main/OneSection.aspx?section=305.160&bid=16297
14:21:04. 71.10.208.80 071-010-208-080.biz.spectrum.com /main/OneSection.aspx?section=305.170&bid=16298
14:21:05. 71.10.208.80 071-010-208-080.biz.spectrum.com /main/OneSection.aspx?section=305.180&bid=16299
14:21:05. 71.10.208.80 071-010-208-080.biz.spectrum.com /main/OneSection.aspx?section=305.190&bid=16300
14:21:06. 71.10.208.80 071-010-208-080.biz.spectrum.com /main/OneSection.aspx?section=305.200&bid=16301
14:21:06. 168.166.80.232 * /main/OneTitle.aspx?title=X
14:21:06. 71.10.208.80 071-010-208-080.biz.spectrum.com /main/OneSection.aspx?section=305.205&bid=16302
14:21:07. 71.10.208.80 071-010-208-080.biz.spectrum.com /main/OneSection.aspx?section=305.210&bid=16303
14:21:07. 104.166.238.2 104-166-238-2.client.mchsi.com /main/Home.aspx
14:21:07. 71.10.208.80 071-010-208-080.biz.spectrum.com /main/OneSection.aspx?section=305.220&bid=16304
14:21:08. 99.113.41.25 99-113-41-25.lightspeed.mssnks.sbcglobal.net /main/OneChapter.aspx?chapter=527
14:21:08. 71.10.208.80 071-010-208-080.biz.spectrum.com /main/OneSection.aspx?section=305.230&bid=16305
14:21:08. 71.10.208.80 071-010-208-080.biz.spectrum.com /main/OneSection.aspx?section=305.230&bid=16305
14:21:08. 71.10.208.80 071-010-208-080.biz.spectrum.com /main/OneSection.aspx?section=305.230&bid=16305
...
14:39:41. 207.46.13.28 msnbot-207-46-13-28.search.msn.com /main/Home.aspx
14:39:41. 208.87.158.238 * /main/OneSection.aspx?section=469.010&bid=25857
14:39:41. 104.166.238.2 104-166-238-2.client.mchsi.com /main/OneSection.aspx?section=10.239
14:39:42. 95.84.212.181 * /main/OneSection.aspx?section=1%22+OR+1+%3D+SLEEP%289999999999%29--+
14:39:42. <<< FISH 95.84.212.181 [1" OR 1 = SLEEP(9999999999)--]
14:39:43. 208.87.158.238 * /main/OneSection.aspx?section=461.300&bid=25856
14:39:45. 99.203.93.75 * /main/OneSection.aspx?section=452.410&bid=25044&hl=
14:39:45. 24.207.254.101 024-207-254-101.res.spectrum.com /main/OneSection.aspx?section=168.133
14:39:47. 34.205.37.218 ec2-34-205-37-218.compute-1.amazonaws.com /main/Home.aspx?constit=y
The problem was that if the console application was not running web pages would not load until the IPC timed out.
This led to the development of an IPC based on non-persisted memory mapped files using MemoryMappedViewAccessor. Conceptually it is a server and a client, the distinction being used to control where information is read and written from. The file we created looks like this.
+------------------+
| |
| |
| |
| |
| |
+-->write+--->+ +<---+read<--+
| | | |
| | | |
| | | |
| | | |
| | | |
+ | | +
Server +------------------+ Client
+ | | +
| | | |
| | | |
| | | |
| | | |
+-->read+---->+ +<---+write<-+
| |
| |
| |
| |
| |
| |
+------------------+
| |
| Management |
| |
| |
+------------------+
What follows is some of the lessons learned in no particular order about non-persisted MMF's.
- there is nothing special about the creator of the file. If the server and client are both running and the end that created the file goes away the file is still present. If the end that created the file is then restarted the file is still there. This caused some what do we do and how do we get synchronized... That is where the management area came into being.
- the file is created in 4096 byte increments. Don't make assumptions about where the end is.
- OT. and of course in the middle of all this my work PC goes out of maintenance and I get a brand new PC. Now instead of having 8@3.0GHz cores idling I have 16@3.6GHz. BUT... the new PC crashes (BSOD) every now and then.
- in some of the examples on MSDN they use a mutex to control write/read. In our case it wasn't needed because of the way we wrote and where the writes occurred. We did use the mutex in our connection reset logic.
- and just when the code has been well tested you deploy it to the IIS server and the console application throws errors when trying to access the file. After digging around on the web for some time we found that certain environments are special, e.g. IIS, and after more digging the solution, global access with security. But wait, the solution doesn't work in the development environment because the fix requires administrative privileges. So we ended up with setup code like this which runs in the development environment AND on the IIS machine.
Dim OpenName As String
If Me.UseGlobal Then
OpenName = "Global\" & Me.Name 'Me.Name is the MMF file name
Else
OpenName = Me.Name
End If
'some security in case it is needed
Dim mSec As New MemoryMappedFileSecurity
mSec.AddAccessRule(New AccessRule(Of MemoryMappedFileRights)(New SecurityIdentifier(WellKnownSidType.WorldSid, Nothing),
MemoryMappedFileRights.FullControl, AccessControlType.Allow))
'try to create as is(Global might be prepended)
OK = False
Try
'open existing
Me.ServerMMF = MemoryMappedFile.OpenExisting(OpenName,
MemoryMappedFileRights.ReadWrite,
IO.HandleInheritability.Inheritable)
OK = True
Catch ex As Exception
Try
'create
Me.ServerMMF = MemoryMappedFile.CreateNew(OpenName,
(Me.Length * 2L) + MgmtSZ,
MemoryMappedFileAccess.ReadWrite,
MemoryMappedFileOptions.None,
mSec,
IO.HandleInheritability.Inheritable)
Me.ServerCreator = True
OK = True
Catch iex As Exception
End Try
End Try
If Not OK AndAlso Me.UseGlobal Then
'try without Global.
' Happens if you don't have Administrator privileges, like on development machine
Try
'open existing
Me.ServerMMF = MemoryMappedFile.OpenExisting(Me.Name,
MemoryMappedFileRights.ReadWrite)
OK = True
Catch ex As Exception
Try
'create
Me.ServerMMF = MemoryMappedFile.CreateNew(Me.Name,
(Me.Length * 2L) + MgmtSZ,
MemoryMappedFileAccess.ReadWrite)
Me.ServerCreator = True
OK = True
Catch iex As Exception
'todo
Throw
End Try
End Try
End If
If OK Then
'setup
'Create the ViewAccessor
Me.ServerRDWR = Me.ServerMMF.CreateViewAccessor(Me.ServerWROffs, 0L)
'---------
Overall the goals were achieved, primarily the website doesn't hang up if the IPC channel is down. We see on average over 50K messages a day between the website and console apllication. Hope this helps someone.
Search Documentation
SerialPort Info
Multics - An OS ahead of its time.
"Those who use Application.DoEvents have no idea what it does
and those who know what it does never use it." former MSDN User JohnWein
Continue reading...
What we settled on was an IPC communication strategy between the web server and a console application using NamedPipes which worked well unless the console application wasn't running. Here is a sample of the output from the console application.
14:21:03. 40.77.167.80 msnbot-40-77-167-80.search.msn.com /main/OneSection.aspx?section=210.526
14:21:03. 107.129.78.67 107-129-78-67.lightspeed.spfdmo.sbcglobal.net /main/OneSection.aspx?section=474.120&bid=26270&hl=
14:21:03. 71.10.208.80 071-010-208-080.biz.spectrum.com /main/OneSection.aspx?section=305.150&bid=16296
14:21:03. 173.170.42.170 173-170-42-170.res.bhn.net /main/OneSection.aspx?section=140.260&bid=6942&hl=excess+proceeds%u2044
14:21:04. 71.10.208.80 071-010-208-080.biz.spectrum.com /main/OneSection.aspx?section=305.160&bid=16297
14:21:04. 71.10.208.80 071-010-208-080.biz.spectrum.com /main/OneSection.aspx?section=305.170&bid=16298
14:21:05. 71.10.208.80 071-010-208-080.biz.spectrum.com /main/OneSection.aspx?section=305.180&bid=16299
14:21:05. 71.10.208.80 071-010-208-080.biz.spectrum.com /main/OneSection.aspx?section=305.190&bid=16300
14:21:06. 71.10.208.80 071-010-208-080.biz.spectrum.com /main/OneSection.aspx?section=305.200&bid=16301
14:21:06. 168.166.80.232 * /main/OneTitle.aspx?title=X
14:21:06. 71.10.208.80 071-010-208-080.biz.spectrum.com /main/OneSection.aspx?section=305.205&bid=16302
14:21:07. 71.10.208.80 071-010-208-080.biz.spectrum.com /main/OneSection.aspx?section=305.210&bid=16303
14:21:07. 104.166.238.2 104-166-238-2.client.mchsi.com /main/Home.aspx
14:21:07. 71.10.208.80 071-010-208-080.biz.spectrum.com /main/OneSection.aspx?section=305.220&bid=16304
14:21:08. 99.113.41.25 99-113-41-25.lightspeed.mssnks.sbcglobal.net /main/OneChapter.aspx?chapter=527
14:21:08. 71.10.208.80 071-010-208-080.biz.spectrum.com /main/OneSection.aspx?section=305.230&bid=16305
14:21:08. 71.10.208.80 071-010-208-080.biz.spectrum.com /main/OneSection.aspx?section=305.230&bid=16305
14:21:08. 71.10.208.80 071-010-208-080.biz.spectrum.com /main/OneSection.aspx?section=305.230&bid=16305
...
14:39:41. 207.46.13.28 msnbot-207-46-13-28.search.msn.com /main/Home.aspx
14:39:41. 208.87.158.238 * /main/OneSection.aspx?section=469.010&bid=25857
14:39:41. 104.166.238.2 104-166-238-2.client.mchsi.com /main/OneSection.aspx?section=10.239
14:39:42. 95.84.212.181 * /main/OneSection.aspx?section=1%22+OR+1+%3D+SLEEP%289999999999%29--+
14:39:42. <<< FISH 95.84.212.181 [1" OR 1 = SLEEP(9999999999)--]
14:39:43. 208.87.158.238 * /main/OneSection.aspx?section=461.300&bid=25856
14:39:45. 99.203.93.75 * /main/OneSection.aspx?section=452.410&bid=25044&hl=
14:39:45. 24.207.254.101 024-207-254-101.res.spectrum.com /main/OneSection.aspx?section=168.133
14:39:47. 34.205.37.218 ec2-34-205-37-218.compute-1.amazonaws.com /main/Home.aspx?constit=y
The problem was that if the console application was not running web pages would not load until the IPC timed out.
This led to the development of an IPC based on non-persisted memory mapped files using MemoryMappedViewAccessor. Conceptually it is a server and a client, the distinction being used to control where information is read and written from. The file we created looks like this.
+------------------+
| |
| |
| |
| |
| |
+-->write+--->+ +<---+read<--+
| | | |
| | | |
| | | |
| | | |
| | | |
+ | | +
Server +------------------+ Client
+ | | +
| | | |
| | | |
| | | |
| | | |
+-->read+---->+ +<---+write<-+
| |
| |
| |
| |
| |
| |
+------------------+
| |
| Management |
| |
| |
+------------------+
What follows is some of the lessons learned in no particular order about non-persisted MMF's.
- there is nothing special about the creator of the file. If the server and client are both running and the end that created the file goes away the file is still present. If the end that created the file is then restarted the file is still there. This caused some what do we do and how do we get synchronized... That is where the management area came into being.
- the file is created in 4096 byte increments. Don't make assumptions about where the end is.
- OT. and of course in the middle of all this my work PC goes out of maintenance and I get a brand new PC. Now instead of having 8@3.0GHz cores idling I have 16@3.6GHz. BUT... the new PC crashes (BSOD) every now and then.
- in some of the examples on MSDN they use a mutex to control write/read. In our case it wasn't needed because of the way we wrote and where the writes occurred. We did use the mutex in our connection reset logic.
- and just when the code has been well tested you deploy it to the IIS server and the console application throws errors when trying to access the file. After digging around on the web for some time we found that certain environments are special, e.g. IIS, and after more digging the solution, global access with security. But wait, the solution doesn't work in the development environment because the fix requires administrative privileges. So we ended up with setup code like this which runs in the development environment AND on the IIS machine.
Dim OpenName As String
If Me.UseGlobal Then
OpenName = "Global\" & Me.Name 'Me.Name is the MMF file name
Else
OpenName = Me.Name
End If
'some security in case it is needed
Dim mSec As New MemoryMappedFileSecurity
mSec.AddAccessRule(New AccessRule(Of MemoryMappedFileRights)(New SecurityIdentifier(WellKnownSidType.WorldSid, Nothing),
MemoryMappedFileRights.FullControl, AccessControlType.Allow))
'try to create as is(Global might be prepended)
OK = False
Try
'open existing
Me.ServerMMF = MemoryMappedFile.OpenExisting(OpenName,
MemoryMappedFileRights.ReadWrite,
IO.HandleInheritability.Inheritable)
OK = True
Catch ex As Exception
Try
'create
Me.ServerMMF = MemoryMappedFile.CreateNew(OpenName,
(Me.Length * 2L) + MgmtSZ,
MemoryMappedFileAccess.ReadWrite,
MemoryMappedFileOptions.None,
mSec,
IO.HandleInheritability.Inheritable)
Me.ServerCreator = True
OK = True
Catch iex As Exception
End Try
End Try
If Not OK AndAlso Me.UseGlobal Then
'try without Global.
' Happens if you don't have Administrator privileges, like on development machine
Try
'open existing
Me.ServerMMF = MemoryMappedFile.OpenExisting(Me.Name,
MemoryMappedFileRights.ReadWrite)
OK = True
Catch ex As Exception
Try
'create
Me.ServerMMF = MemoryMappedFile.CreateNew(Me.Name,
(Me.Length * 2L) + MgmtSZ,
MemoryMappedFileAccess.ReadWrite)
Me.ServerCreator = True
OK = True
Catch iex As Exception
'todo
Throw
End Try
End Try
End If
If OK Then
'setup
'Create the ViewAccessor
Me.ServerRDWR = Me.ServerMMF.CreateViewAccessor(Me.ServerWROffs, 0L)
'---------
Overall the goals were achieved, primarily the website doesn't hang up if the IPC channel is down. We see on average over 50K messages a day between the website and console apllication. Hope this helps someone.
Search Documentation
SerialPort Info
Multics - An OS ahead of its time.
"Those who use Application.DoEvents have no idea what it does
and those who know what it does never use it." former MSDN User JohnWein
Continue reading...