Answered

Extract Hand Written Text and Train with AI Builder

(0) Share

Report

Posted on by 55552

682

I am starting to use the AI Builder within Power Automate, and have trained a model to recognize fields in hand-written forms. The field recognition works fine, but some of the hand-written answers, such as the person's name, are not providing accurate answers. Is there a way to "train" the model to provide the correct answer? For example, I have a Name field, with only a small number of names that would be entered. When the model sees John Smith in hand-written form, it incorrectly interprets it from form to form. Can I tell it during training that this answer in this form is "John Smith"? So far, although you can see the extracted text during training, I have not found an option to correct the answer. The values are shown during training, but cannot edit.

Categories:

AI Builder

I have the same question (1)

All responses (4)

Answers (1)

Sort by

Verified answer

JoeF-MSFT Microsoft Employee on at

Like (0)

Report
Copy link

Link copied!
Hi @Runner55552, thanks for the question!

Unfortunately, today it's not possible to edit values that are detected during training.

After you train the model and use it, do you get a low confidence score for handwritten text that is wrongly detected? If so, you could build a condition that in case of low confidence score for a field, trigger an alert for a human to review the extracted data and push the right data to the desired destination.

Another option is to use expressions in Power Automate to format the data after extraction. For example, if collector name always come in the format K. yakes, the expression below will convert it to K. Yakes.

concat(split({REPLACE-BY-YOUR-VALUE}, ' ')[0], toUpper(slice({REPLACE-BY-YOUR-VALUE}, indexOf({REPLACE-BY-YOUR-VALUE}, ' '), add(indexOf({REPLACE-BY-YOUR-VALUE}, ' '), 2))), slice({REPLACE-BY-YOUR-VALUE}, add(indexOf({REPLACE-BY-YOUR-VALUE}, ' '), 2)))

We're also constantly improving the quality of text detection technology, including for handwritten text, so over time you should see better and better text recognition.

Was this reply helpful? Yes No
55552 682 on at

Like (0)

Report
Copy link

Link copied!

Thanks. The confidence scores from the training for that field were very high. It appears the confidence is mainly that it found the field in the right location, not so much whether it correctly interpreted the hand writing.

I was thinking along the same lines as you noted above, if there are repetitive errors, use replace and split functions to correct them. It had a harder time with one of the other names, spelling it differently multiple times, but I could test to see if the First name began with G then if so, replace with the full first name and last name. It seemed to recognize the first G quite well.

Was this reply helpful? Yes No
Diallom 4 on at

Like (0)

Report
Copy link

Link copied!

Hello,

How to add a step in case of low confidence score for a field, trigger an alert for a human to review the extracted data? CN you please provide a video on how to add the human review screen with the image form displayed ?

Was this reply helpful? Yes No
SalmanShaik 30 on at

Like (0)

Report
Copy link

Link copied!

Hi @JoeF-MSFT ,
I am extracting 30 fields from a single form can you share me the flow if any of the fields confidence score is less than 90% then trigger an email for manual verification it will be useful to everyone. I have to run this in a loop since i will be having 1000+ documents in a single shot...
This feature is already available by default in other RPA tools only we have to select the confidence score criteria it will be sent to manual verification automatically, human has to do the necessary changes.

Was this reply helpful? Yes No