K
Kiran_Angadi
Guest
Recently, I worked on a unique scenario with one of our premier customers wherein they were using Application Request Routing (ARR) module of IIS as a load balancer for a web application that is hosted on two backend IIS servers.
Below was Customer’s architecture:
As shown in the diagram, they were using SNI Https Binding for a custom hostname (https://arr.com) on backend servers.
Because of the above scenario, they had added the servers to the web farm on ARR like below:
appcmd.exe set config -section:webFarms /[name='webfarm'].[address='server1.contoso.com'].applicationRequestRouting.hostName:"arr.com" /commit:apphost
appcmd.exe set config -section:webFarms /[name='webfarm'].[address='server2.contoso.com'].applicationRequestRouting.hostName:"arr.com" /commit:apphost
Then set the preserveHostHeader to false for the webFarm, this applies only if we are not using the same hostname which we use for ARR website. If we are using same hostname as ARR's, then we need to set preserveHostHeader to true.
appcmd.exe set config -section:webFarms /[name='webfarm'].applicationRequestRouting.protocol.preserveHostHeader:"False" /commit:apphost
Overall configuration of their web farm looks like below:
Once we test it from client machine by firing a request to https://arr.com from any browser we were getting “502.3 – A security error occurred”
Then we took a network trace to understand what is really happening between ARR server and backend IIS sever with respect to TLS negotiation as the error mostly talks about abnormal behavior in TLS negotiation.
After analyzing the network trace, below is what we found:
74 1:43:44 PM 8/17/2021 0.8680192 ARR Server1 TCP TCP:Flags=......S., SrcPort=50231, DstPort=HTTPS(443), PayloadLen=0, Seq=2882094152, Ack=0, Win=64800 ( Negotiating scale factor 0x8 ) = 64800 {TCP:12, IPv6:3}
78 1:43:44 PM 8/17/2021 0.9006741 Server1 ARR TCP TCP:Flags=...A..S., SrcPort=HTTPS(443), DstPort=50231, PayloadLen=0, Seq=2784624939, Ack=2882094153, Win=65535 ( Negotiated scale factor 0x8 ) = 16776960 {TCP:12, IPv6:3}
79 1:43:44 PM 8/17/2021 0.9007579 ARR Server1 TCP TCP:Flags=...A...., SrcPort=50231, DstPort=HTTPS(443), PayloadLen=0, Seq=2882094153, Ack=2784624940, Win=517 (scale factor 0x8) = 132352 {TCP:12, IPv6:3}
80 1:43:44 PM 8/17/2021 0.9010972 ARR Server1 TLS TLS:TLS Rec Layer-1 HandShake: Client Hello. {TLS:14, SSLVersionSelector:13, TCP:12, IPv6:3}
92 1:43:44 PM 8/17/2021 0.9332598 Server1 ARR TCP TCP:Flags=...A...., SrcPort=HTTPS(443), DstPort=50231, PayloadLen=0, Seq=2784624940, Ack=2882094670, Win=261 (scale factor 0x8) = 66816 {TCP:12, IPv6:3}
99 1:43:44 PM 8/17/2021 0.9922412 Server1 ARR TLS TLS:TLS Rec Layer-1 HandShake: Server Hello.; TLS Rec Layer-2 Cipher Change Spec; TLS Rec Layer-3 SSL Application Data {TLS:14, SSLVersionSelector:13, TCP:12, IPv6:3}
100 1:43:44 PM 8/17/2021 0.9922412 Server1 ARR TCP TCP:[Continuation to #99]Flags=...A...., SrcPort=HTTPS(443), DstPort=50231, PayloadLen=1220, Seq=2784626160 - 2784627380, Ack=2882094670, Win=261 (scale factor 0x8) = 66816 {TCP:12, IPv6:3}
101 1:43:44 PM 8/17/2021 0.9922412 Server1 ARR TCP TCP:[Continuation to #99]Flags=...A...., SrcPort=HTTPS(443), DstPort=50231, PayloadLen=1220, Seq=2784627380 - 2784628600, Ack=2882094670, Win=261 (scale factor 0x8) = 66816 {TCP:12, IPv6:3}
102 1:43:44 PM 8/17/2021 0.9922412 Server1 ARR TCP TCP:[Continuation to #99]Flags=...AP..., SrcPort=HTTPS(443), DstPort=50231, PayloadLen=628, Seq=2784628600 - 2784629228, Ack=2882094670, Win=261 (scale factor 0x8) = 66816 {TCP:12, IPv6:3}
Frame number 80 has the Client Hello and below is how it looks like:
Frame: Number = 80, Captured Frame Length = 641, MediaType = WiFi
+ WiFi: [Unencrypted Data] .T....., (I)
+ LLC: Unnumbered(U) Frame, Command Frame, SSAP = SNAP(Sub-Network Access Protocol), DSAP = SNAP(Sub-Network Access Protocol)
+ Snap: EtherType = IPv6, OrgCode = **************
+ Ipv6: Next Protocol = TCP, Payload Length = 537
+ Tcp: Flags=...AP..., SrcPort=50231, DstPort=HTTPS (443), PayloadLen=517, Seq=2882094153 - 2882094670, Ack=2784624940, Win=517 (scale factor 0x8) = 132352
TLSSSLData: Transport Layer Security (TLS) Payload Data
- TLS: TLS Rec Layer-1 HandShake: Client Hello.
- TlsRecordLayer: TLS Rec Layer-1 HandShake:
ContentType: HandShake:
+ Version: TLS 1.0
Length: 512 (0x200)
- SSLHandshake: SSL HandShake ClientHello(0x01)
HandShakeType: ClientHello(0x01)
Length: 508 (0x1FC)
- ClientHello: TLS 1.2
+ Version: TLS 1.2
+ RandomBytes:
SessionIDLength: 32 (0x20)
SessionID: Binary Large Object (32 Bytes)
CipherSuitesLength: 32
+ TLSCipherSuites: Unknown Cipher
+ TLSCipherSuites: Unknown Cipher
+ TLSCipherSuites: Unknown Cipher
+ TLSCipherSuites: Unknown Cipher
+ TLSCipherSuites: TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256 { 0xC0,0x2B }
+ TLSCipherSuites: TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 { 0xC0,0x2F }
+ TLSCipherSuites: TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384 { 0xC0,0x2C }
+ TLSCipherSuites: TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 { 0xC0,0x30 }
+ TLSCipherSuites: Unknown Cipher
+ TLSCipherSuites: Unknown Cipher
+ TLSCipherSuites: TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA { 0xC0,0x13 }
+ TLSCipherSuites: TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA { 0xC0,0x14 }
+ TLSCipherSuites: TLS_RSA_WITH_AES_128_GCM_SHA256 { 0x00, 0x9C }
+ TLSCipherSuites: TLS_RSA_WITH_AES_256_GCM_SHA384 { 0x00, 0x9D }
+ TLSCipherSuites: TLS_RSA_WITH_AES_128_CBC_SHA { 0x00, 0x2F }
+ TLSCipherSuites: TLS_RSA_WITH_AES_256_CBC_SHA { 0x00, 0x35 }
CompressionMethodsLength: 1 (0x1)
CompressionMethods: 0 (0x0)
ExtensionsLength: 403 (0x193)
- ClientHelloExtension: Unknown Extension Type
ExtensionType: Unknown Extension Type
ExtensionLength: 0 (0x0)
- ClientHelloExtension: Server Name(0x0000)
ExtensionType: Server Name(0x0000)
ExtensionLength: 19 (0x13)
NameListLength: 17 (0x11)
NameType: Host Name (0)
NameLength: 14 (0xE)
ServerName: server1.contoso.com
+ ClientHelloExtension: Unknown Extension Type
+ ClientHelloExtension: Renegotiation Info(0xFF01)
+ ClientHelloExtension: Elliptic Curves(0x000A)
+ ClientHelloExtension: EC Point Formats(0x000B)
+ ClientHelloExtension: SessionTicket TLS(0x0023)
+ ClientHelloExtension: Unknown Extension Type
+ ClientHelloExtension: Status Request(0x0005)
+ ClientHelloExtension: Signature Algorithms(0x000D)
+ ClientHelloExtension: Unknown Extension Type
+ ClientHelloExtension: Unknown Extension Type
+ ClientHelloExtension: Unknown Extension Type
+ ClientHelloExtension: Unknown Extension Type
+ ClientHelloExtension: Unknown Extension Type
+ ClientHelloExtension: Unknown Extension Type
+ ClientHelloExtension: Unknown Extension Type
+ ClientHelloExtension: Unknown Extension Type
Interestingly, the “ServerName” field of ClientHelloExtensions “Server Name” is set to “server1.contoso.com” although we have set an alternate “hostName” for that particular server (server1.contoso.com).
So, this is the field used for SNI to work on IIS. Because there was another binding for this host name (server1.contoso.com) on backend IIS server (server1.contoso.com) on a different site, we were getting different certificate in the Server Hello because of which we were seeing a "502.3 – A security error has occurred". It is the same scenario for server2.contoso.com as well.
We were expecting the TLS serverName in Client Hello to be same as the hostName field that is set for the backend servers added (arr.com).
We checked the ARR source code internally to understand why this behavior and understood that this behavior is by design.
As per the source code, we establish TLS connection with the backend server using the “address” property we provide to the backend server in web farm settings.
Now, logically speaking when we have multiple servers in the backend and when we want to use the same custom hostname, it is not going to be possible for ARR to route the traffic.
We need unique identifier when it comes to addressing the backend server and address value is unique.
It’s not like SNI is not totally supported, it is supported but not in a way current configuration are.
So what’s the solution for this??????
We need to make use of different hostnames for different backend servers, add them to address property and configure SNI for those hostnames with valid certificate in the backend servers to get this to work.
(Otherwise, we will have to move away from SNI and go back to legacy bindings which will restrict the number of bindings on the servers)
Hope this helps! Happy Learning
Continue reading...
Below was Customer’s architecture:
As shown in the diagram, they were using SNI Https Binding for a custom hostname (https://arr.com) on backend servers.
Because of the above scenario, they had added the servers to the web farm on ARR like below:
appcmd.exe set config -section:webFarms /[name='webfarm'].[address='server1.contoso.com'].applicationRequestRouting.hostName:"arr.com" /commit:apphost
appcmd.exe set config -section:webFarms /[name='webfarm'].[address='server2.contoso.com'].applicationRequestRouting.hostName:"arr.com" /commit:apphost
Then set the preserveHostHeader to false for the webFarm, this applies only if we are not using the same hostname which we use for ARR website. If we are using same hostname as ARR's, then we need to set preserveHostHeader to true.
appcmd.exe set config -section:webFarms /[name='webfarm'].applicationRequestRouting.protocol.preserveHostHeader:"False" /commit:apphost
Overall configuration of their web farm looks like below:
Once we test it from client machine by firing a request to https://arr.com from any browser we were getting “502.3 – A security error occurred”
Then we took a network trace to understand what is really happening between ARR server and backend IIS sever with respect to TLS negotiation as the error mostly talks about abnormal behavior in TLS negotiation.
After analyzing the network trace, below is what we found:
74 1:43:44 PM 8/17/2021 0.8680192 ARR Server1 TCP TCP:Flags=......S., SrcPort=50231, DstPort=HTTPS(443), PayloadLen=0, Seq=2882094152, Ack=0, Win=64800 ( Negotiating scale factor 0x8 ) = 64800 {TCP:12, IPv6:3}
78 1:43:44 PM 8/17/2021 0.9006741 Server1 ARR TCP TCP:Flags=...A..S., SrcPort=HTTPS(443), DstPort=50231, PayloadLen=0, Seq=2784624939, Ack=2882094153, Win=65535 ( Negotiated scale factor 0x8 ) = 16776960 {TCP:12, IPv6:3}
79 1:43:44 PM 8/17/2021 0.9007579 ARR Server1 TCP TCP:Flags=...A...., SrcPort=50231, DstPort=HTTPS(443), PayloadLen=0, Seq=2882094153, Ack=2784624940, Win=517 (scale factor 0x8) = 132352 {TCP:12, IPv6:3}
80 1:43:44 PM 8/17/2021 0.9010972 ARR Server1 TLS TLS:TLS Rec Layer-1 HandShake: Client Hello. {TLS:14, SSLVersionSelector:13, TCP:12, IPv6:3}
92 1:43:44 PM 8/17/2021 0.9332598 Server1 ARR TCP TCP:Flags=...A...., SrcPort=HTTPS(443), DstPort=50231, PayloadLen=0, Seq=2784624940, Ack=2882094670, Win=261 (scale factor 0x8) = 66816 {TCP:12, IPv6:3}
99 1:43:44 PM 8/17/2021 0.9922412 Server1 ARR TLS TLS:TLS Rec Layer-1 HandShake: Server Hello.; TLS Rec Layer-2 Cipher Change Spec; TLS Rec Layer-3 SSL Application Data {TLS:14, SSLVersionSelector:13, TCP:12, IPv6:3}
100 1:43:44 PM 8/17/2021 0.9922412 Server1 ARR TCP TCP:[Continuation to #99]Flags=...A...., SrcPort=HTTPS(443), DstPort=50231, PayloadLen=1220, Seq=2784626160 - 2784627380, Ack=2882094670, Win=261 (scale factor 0x8) = 66816 {TCP:12, IPv6:3}
101 1:43:44 PM 8/17/2021 0.9922412 Server1 ARR TCP TCP:[Continuation to #99]Flags=...A...., SrcPort=HTTPS(443), DstPort=50231, PayloadLen=1220, Seq=2784627380 - 2784628600, Ack=2882094670, Win=261 (scale factor 0x8) = 66816 {TCP:12, IPv6:3}
102 1:43:44 PM 8/17/2021 0.9922412 Server1 ARR TCP TCP:[Continuation to #99]Flags=...AP..., SrcPort=HTTPS(443), DstPort=50231, PayloadLen=628, Seq=2784628600 - 2784629228, Ack=2882094670, Win=261 (scale factor 0x8) = 66816 {TCP:12, IPv6:3}
Frame number 80 has the Client Hello and below is how it looks like:
Frame: Number = 80, Captured Frame Length = 641, MediaType = WiFi
+ WiFi: [Unencrypted Data] .T....., (I)
+ LLC: Unnumbered(U) Frame, Command Frame, SSAP = SNAP(Sub-Network Access Protocol), DSAP = SNAP(Sub-Network Access Protocol)
+ Snap: EtherType = IPv6, OrgCode = **************
+ Ipv6: Next Protocol = TCP, Payload Length = 537
+ Tcp: Flags=...AP..., SrcPort=50231, DstPort=HTTPS (443), PayloadLen=517, Seq=2882094153 - 2882094670, Ack=2784624940, Win=517 (scale factor 0x8) = 132352
TLSSSLData: Transport Layer Security (TLS) Payload Data
- TLS: TLS Rec Layer-1 HandShake: Client Hello.
- TlsRecordLayer: TLS Rec Layer-1 HandShake:
ContentType: HandShake:
+ Version: TLS 1.0
Length: 512 (0x200)
- SSLHandshake: SSL HandShake ClientHello(0x01)
HandShakeType: ClientHello(0x01)
Length: 508 (0x1FC)
- ClientHello: TLS 1.2
+ Version: TLS 1.2
+ RandomBytes:
SessionIDLength: 32 (0x20)
SessionID: Binary Large Object (32 Bytes)
CipherSuitesLength: 32
+ TLSCipherSuites: Unknown Cipher
+ TLSCipherSuites: Unknown Cipher
+ TLSCipherSuites: Unknown Cipher
+ TLSCipherSuites: Unknown Cipher
+ TLSCipherSuites: TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256 { 0xC0,0x2B }
+ TLSCipherSuites: TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 { 0xC0,0x2F }
+ TLSCipherSuites: TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384 { 0xC0,0x2C }
+ TLSCipherSuites: TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 { 0xC0,0x30 }
+ TLSCipherSuites: Unknown Cipher
+ TLSCipherSuites: Unknown Cipher
+ TLSCipherSuites: TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA { 0xC0,0x13 }
+ TLSCipherSuites: TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA { 0xC0,0x14 }
+ TLSCipherSuites: TLS_RSA_WITH_AES_128_GCM_SHA256 { 0x00, 0x9C }
+ TLSCipherSuites: TLS_RSA_WITH_AES_256_GCM_SHA384 { 0x00, 0x9D }
+ TLSCipherSuites: TLS_RSA_WITH_AES_128_CBC_SHA { 0x00, 0x2F }
+ TLSCipherSuites: TLS_RSA_WITH_AES_256_CBC_SHA { 0x00, 0x35 }
CompressionMethodsLength: 1 (0x1)
CompressionMethods: 0 (0x0)
ExtensionsLength: 403 (0x193)
- ClientHelloExtension: Unknown Extension Type
ExtensionType: Unknown Extension Type
ExtensionLength: 0 (0x0)
- ClientHelloExtension: Server Name(0x0000)
ExtensionType: Server Name(0x0000)
ExtensionLength: 19 (0x13)
NameListLength: 17 (0x11)
NameType: Host Name (0)
NameLength: 14 (0xE)
ServerName: server1.contoso.com
+ ClientHelloExtension: Unknown Extension Type
+ ClientHelloExtension: Renegotiation Info(0xFF01)
+ ClientHelloExtension: Elliptic Curves(0x000A)
+ ClientHelloExtension: EC Point Formats(0x000B)
+ ClientHelloExtension: SessionTicket TLS(0x0023)
+ ClientHelloExtension: Unknown Extension Type
+ ClientHelloExtension: Status Request(0x0005)
+ ClientHelloExtension: Signature Algorithms(0x000D)
+ ClientHelloExtension: Unknown Extension Type
+ ClientHelloExtension: Unknown Extension Type
+ ClientHelloExtension: Unknown Extension Type
+ ClientHelloExtension: Unknown Extension Type
+ ClientHelloExtension: Unknown Extension Type
+ ClientHelloExtension: Unknown Extension Type
+ ClientHelloExtension: Unknown Extension Type
+ ClientHelloExtension: Unknown Extension Type
Interestingly, the “ServerName” field of ClientHelloExtensions “Server Name” is set to “server1.contoso.com” although we have set an alternate “hostName” for that particular server (server1.contoso.com).
So, this is the field used for SNI to work on IIS. Because there was another binding for this host name (server1.contoso.com) on backend IIS server (server1.contoso.com) on a different site, we were getting different certificate in the Server Hello because of which we were seeing a "502.3 – A security error has occurred". It is the same scenario for server2.contoso.com as well.
We were expecting the TLS serverName in Client Hello to be same as the hostName field that is set for the backend servers added (arr.com).
We checked the ARR source code internally to understand why this behavior and understood that this behavior is by design.
As per the source code, we establish TLS connection with the backend server using the “address” property we provide to the backend server in web farm settings.
Now, logically speaking when we have multiple servers in the backend and when we want to use the same custom hostname, it is not going to be possible for ARR to route the traffic.
We need unique identifier when it comes to addressing the backend server and address value is unique.
It’s not like SNI is not totally supported, it is supported but not in a way current configuration are.
So what’s the solution for this??????
We need to make use of different hostnames for different backend servers, add them to address property and configure SNI for those hostnames with valid certificate in the backend servers to get this to work.
(Otherwise, we will have to move away from SNI and go back to legacy bindings which will restrict the number of bindings on the servers)
Hope this helps! Happy Learning
Continue reading...
