Introduction

AI Builder is a Power Platform capability that provides you the AI models to take leverage of artificial intelligence to optimize your business as well as automate process. We can either use the already existing prebuilt models that suits our needs to use the Power of AI in Power Apps or Power Automate or create our own custom AI models to suit some specific business needs. You can read more about AI Models in the Official documentation here. The out of the box prebuilt available models in AI Builder are:

• Business card reader    
• Category classification     
• Entity extraction                   
• ID reader
• Invoice Processing          
• Key phrase extraction    
• Language detection              
• Receipt processing            
• Sentiment analysis                
• Text recognition                    
• Text translation      

In earlier days if we wanted to use any of these models, we had to call Power Automate and add the AI Builder actions to leverage the Cognitive abilities. However, more and more AI Builder capabilities are coming natively to Power App so that we can add them as controls to the App and directly interact with the AI Models based on our needs.

In this article we will see how we can make use of the Form Processor within Power App to read forms of a predefined template. Since the forms can vary in templates with different fields , we will also see how to create a Model that can be used to meet our specific needs, train them to identify the data that we need and use it with the Form Processor in Power App.

Scenario

We have a requirement where we are handing over Forms to patients to fill out their details which can be used for Back Office data records keeping. Manually copying over the data can be cumbersome. So, we will automate the reading of the filled out forms by patients and extract the information using a Power App that leverages Form Processor

 

We will save the patient data to SharePoint List with the below columns

 

 

Implementation

To get started with the creation of the Form Processing App, lets create a new Canvas App and add the Form Processor from the AI Builder to the screen. By default, it is not linked to any existing models. In case we do not have a model that has been trained to pick the information form the forms that we are going to use, we will have to create and train a new model.

 

So as to create a new model specific to the forms that we are going to use, lets head over to the AI Builder section in the make.powerapps.com and from Explore tab, select “Extract custom information from documents”

 

Add Fields to Model

This will open a new model builder that will initially ask us to add all the fields from the form that we would like to add to the model so that it can detect corresponding information from the forms.

 

As of now we can use Fields, Checkboxes, and tables with the models. For each Field/Checkbox/Table that we would like to add to the model, click on Add and select the field type and click on Next.

 

 

We will also give the name of the Field that we will be trying to map in the model.

 

Repeat the steps for all the fields. Since Phone Number and Date Of Birth is a collection of multiple sub fields, we will be mapping the sub fields (Day,Month,Year) in case of the Main field(Date of Birth) and combining their values later in the Power App.

 

So, we have mapped Area Code and Mobile Number which will together form the full Phone Number field value.

 

We will do the same for Date of Birth by adding the sub fields Day,Month and Year.

 

The Address field is also mapped using Street Address 1 and Street Address 2

 

We also have one Check Box that will be mapped to the model.

 

 

Let’s specify its name and add it to the model.

 

Thus, we have added all the fields and checkbox in the form to the model that we are going to use.

 

Add Sample Documents

Thus, after adding the fields to the model, lets add a collection of 5 or more documents so that we can use it for mapping the model fields to the fields in the sample documents which will help the model in the future to identify the text from respective fields.

 

 

 

 

 

 

 

 

Click on New collection which will open a right pane that gives the option to add documents.

 

 

Click on Add documents to upload the sample forms. We can select the source from which we can upload the document.

 

Once the documents are browsed and selected, click on Upload to add them to the collection.

 

Thus, we can see that all the selected documents have been uploaded and added to the collection.

 

Click on Next to move to the next phase of Tagging the model fields to the Form Area/Field.

 

Map the Fields to Form Area

 

Select each section of the Form from where we need to extract the data. The model will show a drop down of fields that we had created previously and selecting it will map the model field to the selected form area. Any text in that area will be extracted and mapped to the form field when we use the model in the future.

 

 

 

Since we have split the Date of Birth into 3 fields, Day, Month and Year, we will map them to their respective fields.

 

Thus, we have completed the mapping of all the fields in the current form. Repeat this process for all the forms that we have added to the collection so that it improves the accuracy of the model in extracting the data.

 

Click on Next to move to the next phase of Training the model.

Train the Model

Now based on the mapped model fields, the model will train itself using the uploaded collection and get itself up to speed in identifying text from a similar form that we will use in the future. Click on Train to start the training process.

 

The training will take some time to complete.

 

Test the Model

Once the training is complete, even before publishing, we can do a quick test to see how well the model is able to extract form data.

 

Click on Quick test and upload a new form.

 

The model has identified the data using the mapping we had done previously, and it also give a confidence score indicating the accuracy of the mapping.

 

Now let’s go ahead and Publish the model so that it becomes available for use in our Power App.

 

It can be access from Flow as well as SharePoint Syntex applications. Instead of selecting the New App in Power Apps, I will just head over to the app that we were creating previously.

 

 

 

Add Model to the Form Processor

We can see that the recently created model is now made available for use with the Form Processor control we had added to the screen.

 

 

Let’s select the model which will allow the form processor to use it for analysing the uploaded forms and extract the mapped data for use in the app. We have also added Text Controls to the App for all the fields that will be extracted from the uploaded form. For Address, to show it as a multi-line value, we will use an HTML Text control to better host the extracted data.

 

The Text Input controls get their default values from the Form processor using the expressions. FormProcessor.Results.<FieldsName>.Value . For Full Name field, since we have mapped it inside the model, we will be referring to its value using the below expression in the Default property of the text box.

 

 

FormProcessor.Results.'Full Name'.Value

 

 

 

Same way rest of the fields can be referred using the expressions:

Text Input Control	Expression
Full Name	FormProcessor.Results.'Full Name'.Value
Gender	FormProcessor.Results.Gender.Value
PhoneNumber	Concatenate(FormProcessor.Results.'Area Code'.Value,"-",FormProcessor.Results.'Mobile Number'.Value)
Date of Birth	Concatenate(FormProcessor.Results.Day.Value,"-",FormProcessor.Results.Month.Value,"-",FormProcessor.Results.Year.Value)
Address	Concatenate(FormProcessor.Results.'Street Address Line 1'.Value,"<br/>",FormProcessor.Results.'Street Address Line 2'.Value)
EmailID	FormProcessor.Results.'Email Address'.Value
Visited Previously	FormProcessor.Results.'Visited the Clinic Before ?'.Value

 

Since Date of Birth is a combination of the Day , Month and Year , we will use the Concatenate function to combine them together and show as a single value in the text box.

 

Same way as Address is a combination of Street Address Line 1 and Street Address Line 2, so as to show them as 2 lines, we are using the HTML text control and using the concatenate function with a new line character in the expression so that both addresses would show up in 2 lines.

 

Finally, we will use the Patch function to save the details back to the SharePoint List

 

 

Patch('Patient Details',{'Full Name':TextInput_FullName.Text,Gender:TextInput_Gender.Text,'Phone Number':TextInput_PhoneNumber.Text,'Date of Birth':TextInput_DateofBirth.Text,Address:HtmlText_Address.HtmlText,'Email ID':TextInput_EmailID.Text,'Visited Clinic Previously ?':Checkbox_VisitePreviously.Value})

 

 

 

Test the App

Now let’s test the app by previewing it. We will upload a patient form and the model will quickly extract the content and populate the form fields and on clicking Save, it gets saved to the list back end.

 

 

Summary

Thus, we saw how to Create a Form Processing model, train it and use it with our app to extract data from forms that were created for a specific business need with a predefined layout.