The steps below is the solution that worked for me:
1. Save PowerApps to the disk: myTabletApp.msapp
2. Add file extension .zip : mtTabletApp.msapp.zip. PowerApps file is actually zipped folder

3. Open Properties.json and change the following fields:
"DocumentLayoutWidth":1024,"DocumentLayoutHeight":768,"DocumentLayoutOrientation":"landscape","DocumentLayoutMaintainAspectRatio":true,"DocumentLayoutLockOrientation":true,"OriginatingVersion":"1.160","DocumentAppType":"DesktopOrTablet"
Into:
"DocumentLayoutWidth":640,"DocumentLayoutHeight":1136,"DocumentLayoutOrientation":"portrait","DocumentLayoutMaintainAspectRatio":true,"DocumentLayoutLockOrientation":true,"OriginatingVersion":"1.244","DocumentAppType":"Phone"
4. Remove .zip extension and open file in PowerApps
5. Consequently, you'd need to rearrange and resize the controls to match the mobile phone layout