502 Error when using On-premise data Gateway with SharePoint Subscription Edition

(1) Share

Report

Posted on by Steve_25

I get the following when trying to create a connection to my SharePoint Subscription Edition Farms (prod and dev).

It cannot find the site address, but if you put in the site name, and then go to the drop down menu for SharePoint Library Name, you get:

Could not retrieve values. The dynamic invocation request failed with error: Unexpected error occurred when calling the ApiHubsRuntime API: 'Microsoft.Azure.ProcessSimple.Data.Entities.Exceptions.ProcessSimpleDataException: The ApiHubsRuntime API call failed with http status code 'BadGateway' and response content '{ "error": { "code": 502, "source": "msmanaged-na.azure-apim.net", "clientRequestId": "9aacbcbc-dbf9-4c8d-a6e4-6af3b7027420", "message": "BadGateway", "innerError": { "status": 502, "message": "Received error payload from gateway service with ID 140655: <ccon>An error occurred while sending the request.</ccon>.\r\nclientRequestId: 9aacbcbc-dbf9-4c8d-a6e4-6af3b7027420", "source": "https://sharepointonline-eus.azconn-eus-003.p.azurewebsites.net/datasets/<my url here>/tables", "errors": [] } } }' at Microsoft.Azure.ProcessSimple.Data.DataProviders.HttpClientDataProvider.<>c__DisplayClass47_1`2.<<CallService>b__0>d.MoveNext() in C:\__w\1\s\src\processsimple\Roles\ProcessSimple.Data.Shared\DataProviders\Services\HttpClientDataProvider.cs:line 648 --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at Microsoft.WindowsAzure.ResourceStack.Common.Algorithms.AsyncRetry.<Retry>d__3`1.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1.ConfiguredTaskAwaiter.GetResult() at Microsoft.Azure.ProcessSimple.Data.DataProviders.HttpClientDataProvider.<CallService>d__47`2.MoveNext() in C:\__w\1\s\src\processsimple\Roles\ProcessSimple.Data.Shared\DataProviders\Services\HttpClientDataProvider.cs:line 493'.

From the Gateway error logs, we get this message when I put in the URL and get the unable to find the site address:

DM.EnterpriseGateway Error: 0 : 2023-12-12T16:59:15.7086242Z DM.EnterpriseGateway ef3af182-709c-4919-b5b1-ad928c8777cf 00000000-0000-0000-0000-000000000000 MDSR 00000000-0000-0000-0000-000000000000 b086e54a-9b31-41f9-9ecf-f7deb8993c5b 2edee660-1e3e-4ed2-b1cc-28b21a9bd6c8 3A0EA094 [DM.Pipeline.ExceptionUtils] EnsureGatewayPipelineException encountered a non-GatewayPipelineException: System.Net.Http.HttpRequestException: An error occurred while sending the request. ---> System.Net.WebException: The underlying connection was closed: An unexpected error occurred on a send. ---> System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host. ---> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host

Note this is for an existing working flow before we upgraded to Subscription Edition.

I have also:

* Updated the Gateway to the latest version, rebooted systems

* Removed any unnecessary software

* Created NEW connections in Connections in PowerApps

* We are not using a Proxy server

* Switched to HTTP

* Checked name resolution - the gateway can get to the new farms just fine, can connect to the web sites through the browser just fine (so 443 is not blocked)

* Created new Flow, just "When an item is created or modified", and get the same "We are unable to find the site address" error, so it is not existing flows having issues.

Since it is exactly the same sites / same data / same permissions, just migrated to SE, I'm stumped. We are not using proxies and the Gateway can get access to the new farms no problem.

What does work

If I use my old SharePoint 2019 Farms, everything works fine - with the same connectors.

* If I type the DEV farm URL, it works fine with no changes

* If I put the old PROD farm IP's in the HOSTS files of the Gateway, then it works fine again

I even bypassed the VIP and pointed directly to the SharePoint front-end by putting it in the hosts file - and still failed.

Is SharePoint Subscription edition supported for the On-Prem Gateway? Since it all works fine with the 2019 farms, but just the SE farms not working - I am at a loss. As far as I know, there are no settings for this in SharePoint On-Prem at all to make for this to work.

I have the same question (0)

All responses (4)

Answers (0)

Steve_25 6 on at

Like (1)

Report
Re: 502 Error when using On-premise data Gateway with SharePoint Subscription Edition

More information on the 502 Bad Gateway issue:

Upgraded all Gateways to Server 2022 (just to be sure), with all patches
Applied June 18th update for the Gateway
Ran network traces and found that a on the Client Hello message, IIS was resetting the connection when the gateway was trying to connect to the SharePoint Subscription Edition boxes.

Connections were over TLS 1.2. They looked the same to the SharePoint 2019 boxes.

Workaround
=================
Since it is connection related, I thought TLS 1.2 may be the issue. (Maybe SharePoint SE wanted 1.3?)

So I un-selected "Disable Legacy TLS" from the Web Applications in IIS - and it worked from everywhere.

This cannot be done for production since it is security issue. (See attached image of setting). Just allowing TLS 1.0 and 1.1 seems a bad idea.

Added the registry keys to force TLS 1.3, per article and rebooted: https://learn.microsoft.com/en-us/data-integration/gateway/service-gateway-communication But it made no difference.

The gateway was using TLS 1.2 still in failures - and it seems to be the same with the successful traces as well.

Was this reply helpful? Yes No
Suggested answer

CU06081954-3 on at

Like (1)

Report
502 Error when using On-premise data Gateway with SharePoint Subscription Edition

Opened a case with Support, but there does not seem to be a "Gateway" team - so I walked them through the repro a few times.

It was obvious that this was a problem with TLS security. But was getting nowhere with support even after showing them the problem and the workaround of turning off the "Disable Legacy TLS" option.

In fact, there was no connectivity happening at all - so the issue was at the TCP level. This means we could eliminate IIS or SharePoint altogether from troubleshooting.

Root Cause

Looked at a successful trace more closely:

You could see what ciphers the gateway said it could use

You can see the negotiation and the cipher that was picked

So is this cipher, or any of them offered by the gateway, available on the 2022 IIS server by default?

NMAP to the rescue

https://nmap.org

I used "nmap -p 443 -Pn --script-enum-ciphers <url>" to find out that Server 2019 had a LOT more ciphers open (including TLS 1.0 and 1.2) by default, including the one the gateway used.

In fact, unchecking "Disable Legacy TLS" opened the same ciphers as a 2019 IIS server did by default.

So - the gateway was using old ciphers, and did not support ANY the new stricter TLS / cipher options that Server 2022 IIS offered.

This was why we were getting a reset and the root cause of the problem.

Solution

Now we have the root cause - comes the easy part.

But how could we keep our new, shiny enhanced 2022 IIS security - and just add the one weaker cipher that the On-Premise Gateway negotiated successfully? (After turning off the "Disable Legacy TLS" checkbox for the web applications we wanted to be able to use the gateway...)

The steps were:

Identify the exact Ciphers enabled by default by IIS

Find a way to disable TLS 1.0, 1.1 - and add the Cipher for the data gateway to use

Test results to make sure it works

Profit!

Part 1 – Identify the Ciphers (with TLS 1.0 and TLS 1.1 disabled, and some weak TLS 1.2 ciphers)

Run NMAP against the 2022 servers - then we know what ciphers we need enabled to start.

nmap -p 443 -Pn --script ssl-enum-ciphers <url>

Starting Nmap 7.95 ( https://nmap.org ) at 2024-07-24 10:46 Pacific Daylight Time
Nmap scan report for <snip>
Host is up (0.0010s latency).

PORT    STATE SERVICE
443/tcp open https
| ssl-enum-ciphers:
|   TLSv1.2:
|     ciphers:
|       TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (secp384r1) - A
|       TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (ecdh_x25519) - A
|       TLS_DHE_RSA_WITH_AES_256_GCM_SHA384 (dh 2048) - A
|       TLS_DHE_RSA_WITH_AES_128_GCM_SHA256 (dh 2048) - A
|     compressors:
|       NULL
|     cipher preference: server
|     warnings:
|       Key exchange (dh 2048) of lower strength than certificate key
|       Key exchange (ecdh_x25519) of lower strength than certificate key
|   TLSv1.3:
|     ciphers:
|       TLS_AKE_WITH_AES_256_GCM_SHA384 (secp384r1) - A
|       TLS_AKE_WITH_AES_128_GCM_SHA256 (ecdh_x25519) - A
|     cipher preference: server
|_ least strength: A

Part Two – Disable TLS 1.0 and 1.1, and only add the Cipher for the data gateway to use

Now that we know what we need to have enabled, we can set IIS to listen to these.

This can be done in the registry, but luckily there is a utility that can do this in a GUI.

Using IIS Crypto 3.3, I unchecked the box in IIS that disabled all the cipher suites (Disable Legacy TLS) and enabled the ONE extra cipher we knew the gateway was using. I left any ciphers alone that were checked but not in either list.
Nartac Software - IIS Crypto

I also disabled TLS 1.0 and 1.1 altogether under the "Schannel" option:

When I rebooted, the new NMAP showed the old ciphers, plus the TLS 1.2 Cypher we know the Data Gateway could use:

TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384 (secp384r1) - A

Results from NMAP after:

Starting Nmap 7.95 ( https://nmap.org ) at 2024-07-24 13:12 Pacific Daylight Time
Nmap scan report for <snip>
Host is up (0.0020s latency).

PORT    STATE SERVICE
443/tcp open https
| ssl-enum-ciphers:
|   TLSv1.2:
|     ciphers:
|       TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (secp384r1) - A
|       TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (ecdh_x25519) - A
|       TLS_DHE_RSA_WITH_AES_256_GCM_SHA384 (dh 2048) - A
|       TLS_DHE_RSA_WITH_AES_128_GCM_SHA256 (dh 2048) - A
|       TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384 (secp384r1) - A
|     compressors:
|       NULL
|     cipher preference: server
|     warnings:
|       Key exchange (dh 2048) of lower strength than certificate key
|       Key exchange (ecdh_x25519) of lower strength than certificate key
|   TLSv1.3:
|     ciphers:
|       TLS_AKE_WITH_AES_256_GCM_SHA384 (secp384r1) - A
|       TLS_AKE_WITH_AES_128_GCM_SHA256 (ecdh_x25519) - A
|     cipher preference: server
|_ least strength: A

Nmap done: 1 IP address (1 host up) scanned in 1.11 seconds

Part III - Test Results

Once I did this – I could get the gateway to work, and TLS 1.0 and TLS 1.1 was disabled.

I also talked to security, and they said as long as the cipher is TLS 1.2, I can use the weaker cipher. (Yay?)

I rebooted the front-end servers and everything was working using the weaker cipher.

Note: For production, you can disable the TLS 1.0 and TLS 1.1 without rebooting. Enabling the other protocol requires a reboot. I suggest testing using NMAP to be sure to be sure.

As a bonus, we still get an A security rating using an SSL tester:

SSL Server Test: (Powered by Qualys SSL Labs)

IDEALLY – Microsoft will update the gateway to support the more secure Ciphers that come with Server 2022 by default.

Part IV - Profit!

Yeah, well - you can't have everything I suppose.

Was this reply helpful? Yes No
lbendlin 8,372 Super User 2025 Season 2 on at

Like (1)

Report

502 Error when using On-premise data Gateway with SharePoint Subscription Edition

bit of a side question - why use a gateway for cloud sharepoints?

Was this reply helpful? Yes No
Steve_25 6 on at

Like (1)

Report

502 Error when using On-premise data Gateway with SharePoint Subscription Edition

>>>bit of a side question - why use a gateway for cloud sharepoints?

The gateway is only for On-Premise SharePoint. They replace the traditional "Workflows" that use SharePoint Designer.

Workflows use SharePoint designer 2013 - and designer is going away. And the old-style workflows are also going away in the next couple of years for SharePoint Online altogether.

Knowing this, it doesn't make sense to invest any more time in creating NEW workflows in SharePoint designer (and later only in Visual Studio once designer is gone).

The On-Premises Gateway lets you point to your On-Premise SharePoint and create the flows in the cloud.

If you decide to migrate everything to the cloud later - in theory, they should still work with minor modifications. Although there are some things you can only do with On-Premise SharePoint, such as Anonymous access sites. And existing workflows can be converted BEFORE migrating and you do not have to scramble and rewrite them all after you migrate...

Was this reply helpful? Yes No