Skip to main content

Notifications

Power Apps - AI Builder
Unanswered

Mass Detection of Files, Files getting duplicated Analyzed

(0) ShareShare
ReportReport
Posted on by 182

Hey Folks,

 

in my Project i running an complex Flow, which detects new Files in an Folder (Sharepoint) an analyze them with AI Builder to store the Data in Dataverse.
Also the readout is scriptet in a Dataverse Helplist to get an Export of the whole Files which are getting Analyzed.

The Flow is not running in a big Apply to Each Loop, it gets analyzed File by File.
I Set the Degree of Parallelism in Sharepoint Trigger When a file is created or modified (properties only) at 20 to reduce the Concurency of Files.

 

But when i analyze 500 Files i get 37 Duplicates in it (and i run that several Times with that Amount of Files).
When i anayze 200 Files i get 16 Duplicates in it.
And on 30 Files i get 8 Duplicates in it.

So not only do I have a corrupt database from which I have to throw out the duplicate files, but I have (supposedly) paid for the analysis of 61 additional Files.

How do you deal with this Case, is there an official Document from Microsoft regarding the problem and how to work around it?

Categories:
  • Tjan Profile Picture
    Tjan 1,087 on at
    Re: Mass Detection of Files, Files getting duplicated Analyzed

    @HansenvomDeich  Ah, glad you have a solution via this way! Yeah.. This performance problem will occur if parallelism is set to 1.. If the time doesn't bother too much, you should keep your solution like this! Always better than having double data entires AND double charged AI Builder credits

  • HansenvomDeich Profile Picture
    HansenvomDeich 182 on at
    Re: Mass Detection of Files, Files getting duplicated Analyzed

    Hi @Tjan ,

    i think over, and rebuild the flow (as it should) with an for each loop.

    Now i get the exact Outputs of the Elements which will be readout e.g. 20 Files > 20 Results.

    But the Problem, before i get a readout of 200-500 Files in about 5 to 10 Minutes because of the parallel run from Flow Trigger by 50.

    Now with the For Each, its getting 50 Files in the Trigger Action, but the for Each reduced the Speed of readout to 1 by time.(50 Files in FlowTrigger, reduced by one by one in the For Each Loop).
    So that now i need 10 Minutes for 20 Files 😄

  • HansenvomDeich Profile Picture
    HansenvomDeich 182 on at
    Re: Mass Detection of Files, Files getting duplicated Analyzed

    Now i know why i turned it on:
    With turned that Option on i can use the identifier from file created without for each loop

    HansenvomDeich_0-1718353804108.png

    With turned it off, there i can´t run the flow when i wanna now use the identifier.

    i have to build for each with body/value in for each and identifier in Action

    HansenvomDeich_1-1718353941141.png

     

  • HansenvomDeich Profile Picture
    HansenvomDeich 182 on at
    Re: Mass Detection of Files, Files getting duplicated Analyzed

    @Tjan 
    I think about why i turned that on.
    I thougt i try to test first, build all that things, in one big for each to get start und finish messages, but it doesnt work as i want, or better to say i didnt have enough skills to build that. so i decide to use it as single flow for every file.

    After Teams Collapse because ive got to many messages (start finish msg), i build the help table in dataverse to get feedback about the status of the runnings.
    (the flow was original planned for a usecase with 100 in month not in a minute and i dont got time to rebuild that, because unfortunately i am an employee not an automate,app builder in our company)

    But i think i have tested it, without turned it on, because i read about the function of that nipple and it doesnt work, but i am not sure.
    I try to switch it off and look what happened.

    There are nor for each Loops in the Flow, only multiple conditions to check plausibility of some actions.

    Thanks for your Hints.

  • Tjan Profile Picture
    Tjan 1,087 on at
    Re: Mass Detection of Files, Files getting duplicated Analyzed

    @HansenvomDeich  Ah alright! I also saw that you turned on 'split-on' already. That is also something that would help in your case. Then, one last question: do you have any 'for eaches' in your flow?

  • HansenvomDeich Profile Picture
    HansenvomDeich 182 on at
    Re: Mass Detection of Files, Files getting duplicated Analyzed

    Hi @Tjan,

    thanks for your help at first.

    I know exactly what you mean, the Folder ive created is a Manual Folder, so i drop in there 200 Files, and wait until they got analyzed (after that the file is copied by the flow in target folder and get deleted from that "Auto Trigger" Folder)

    So i can say 100 % that there is no modified Action in that Folder. Only created and only one time per Dataset.

  • Tjan Profile Picture
    Tjan 1,087 on at
    Re: Mass Detection of Files, Files getting duplicated Analyzed

    @HansenvomDeich  Thank you so much for the screenshots. From what I noticed immediately is that you use the trigger 'when an item is created or modified'. This means that when a Sharepoint item changes (no matter which column), this flow will be triggered. This may cause your problem because an item might trigger this flow multiple times. The first time being the creation, the second time being a modification and the third time as well etc. 

     

    My recommendation would be to use another trigger. That is, "When an item is created". With this action you will not have this problem. If you however want to still trigger this flow if an item is modified (but only certain conditions) then try to set up trigger conditions.

  • HansenvomDeich Profile Picture
    HansenvomDeich 182 on at
    Re: Mass Detection of Files, Files getting duplicated Analyzed

    Hi @Tjan , 

    i try to show you

     

    HansenvomDeich_0-1718271373325.png

    HansenvomDeich_1-1718271404888.png

     

    In fact i thinkthese are the Steps which are interested:

    HansenvomDeich_2-1718271556251.pngHansenvomDeich_4-1718271671148.png

     

    HansenvomDeich_3-1718271648407.png

     

  • Tjan Profile Picture
    Tjan 1,087 on at
    Re: Mass Detection of Files, Files getting duplicated Analyzed

    @HansenvomDeich  It might help to share how your flow is set-up! It will help me check what's happening.

Under review

Thank you for your reply! To ensure a great experience for everyone, your content is awaiting approval by our Community Managers. Please check back later.

Helpful resources

Quick Links

Microsoft Kickstarter Events…

Register for Microsoft Kickstarter Events…

Announcing Our 2025 Season 1 Super Users!

A new season of Super Users has arrived, and we are so grateful for the daily…

Announcing Forum Attachment Improvements!

We're excited to announce that attachments for replies in forums and improved…

Leaderboard

#1
WarrenBelz Profile Picture

WarrenBelz 145,475

#2
RandyHayes Profile Picture

RandyHayes 76,287

#3
Pstork1 Profile Picture

Pstork1 64,767

Leaderboard