Answered

Maximum URL length or maximum query param length on request from custom connector

(0) Share

Report

Posted on by dmorland

I have a custom connector for an endpoint that requires a query parameter called 'filters' that is a string of IDs. Each ID is about 17 chars long. The connector is sending the request via an on-premises gateway.

I have found that I get a 500 response as below if I send a long list of IDs in the query parameter. The request never makes it to the endpoint so this error is generated somewhere along the way (I have set up a reverse proxy to monitor the request sent from the custom connector and nothing arrives at the proxy)

"message": "Received error payload from gateway service with ID 761229: <ccon>Error parsing HTTP message header byte 16756 of message System.Byte[].</ccon>.\r\nclientRequestId: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
"error": {
"message": "Received error payload from gateway service with ID 761229: <ccon>Error parsing HTTP message header byte 16756 of message System.Byte[].</ccon>."
},
"source": "gatewayconnector-ase.azconn-ase-001.p.azurewebsites.net"
}

If I reduce the size of the query parameter by removing some IDs, the request makes it to the reverse proxy ok and is processed correctly by the endpoint with a 200 response received.

It seems that there is a limitiation somewhere, probably related to size of URL that is preventing the request making it out of the gateway to the endpoint.

Fails: URL length 508 chars, filter query param length 239 chars

Succeeds: URL length 468 chars, filter query param length 199 chars

Does anyone know what the limitation I am hitting is and is this something that can be configured so that I can send longer query parameters?

Categories:

Using connectors

I have the same question (0)

All responses (4)

Answers (1)

Eder_Ruiz Microsoft Employee on at

Like (0)

Report

@dmorland please refer to What is an on-premises data gateway? | Microsoft Learn

Maybe the issue is in the payload.

Was this reply helpful? Yes No
Matthy79 4,180 Super User 2024 Season 1 on at

Like (0)

Report

Hello @dmorland.

When you create custom connectors, you have the ability to test them. If you use the same payload during testing, does it work? A big problem/bug when using power automate is, that it adds additional (unwanted) headers and that causes a lot of troubles.

The most unwelcome header is "x-arr-clientcert". We remove this header before sending the call to the actual api. Maybe you can check this on your end as well.

The easiest way to check is to do the test in the custom connector. If it works there and when using power automate the same payload doesn't, it is 95% the header size.

Fun fact. When you add a url-param, power automate (custom connector) also adds a header as well. So when you have more url params you also increase the size of the headers.

Was this reply helpful? Yes No
dmorland 53 on at

Like (0)

Report

Yes, there are a huge number of headers added by Power Platform. I remove most of these in the reverse proxy behind the gateway (the endpoint couldnt cope with the size of headers but that is a separate issue). The size of headers does not seem to be the issue in this case. It seems to be the URL length.

I intially worked around this problem by changing the connector definition replacing a query parameter (which was passing a list of IDs) with a header parameter instead. Then behind the gateway, a reverse proxy took the header parameter and added it as a query parameter before forwarding to the endpoint. This seemed to solve the problem but intermittently it has returned.

My custom connector works ok from the test screen but when using the connector from a flow with the same payload it is failing with error parsing header before it even gets out of the gateway.

Microsoft is currently help troubleshoot and indications are that the URL is exceeding the 2048 character limit when sending from the flow (I dont know how it gets to be so long!)

I am going to try changing my connector to replace ALL the query parameters with header parameters and update the reverse proxy to convert them all back to query parameters.

I will update here once I have had a chance to test this.

Was this reply helpful? Yes No
Verified answer

dmorland 53 on at

Like (0)

Report
Matthy79 I did some additional testing and it is pretty clear that the length of the URL is the problem although I can't get any concrete proof since the generated URL is not captured in gateway logs.

The request from the custom connector is passing through the gateway ok but the same request from the flow is not. The failed request shows in the gateway error log but only with the "Error parsing HTTP message header". I can cause the request from the custom connector to fail if I increase the size of one of the query parameter values I send.

I have changed my custom connector so that all (except for one) query parameters are defined as header parameters. The reverse proxy that the requests are directed to looks for the matching base URL and forwards the request to the endpoint after changing the header parameters into query parameters. This works around the problem.

Things I can see that can be done to reduce the size of the URL include:
Changing query parameters to header parameters and converting once through the gateway
Changing the base URL of the connector to shorter value
Changing the name of the connector to shorter value

Was this reply helpful? Yes No