Thank you for the replies and help guys.
Sorry if I explained it in the wrong way.
To make it very clear and add more context for you to understand my problem:
I'm using an entra app with the right context and sufficient rights see screenshot below:

In Power Automate I'm using a connection that is a service principal connection.
Below is an example of my flow failing:
See that during the first child folder deletion is a succes but the second one gives problems:
So, it did work 1 time in the flow, but the second time. It gave an error and didn't delete the child folder.
This is the same for a "root" contactFolder that is originally created by a flow with my custom GraphAPI connector.
Can somebody point me out to where I can check/change some settings so it would not use the retention time?
Since I'm sure its not an rights/delegation issue.
Thank you for your time and help.