Through multiple trial and error, I could find a fix -
1. Create a new site in the target environment. My source site is based on a Blank templates, so used the same template -


2. This also creates entry in the Portal Admin Center

3. Next, import the source site using Portal Records Mover -

4. The imported website record in our case is called "Site 1 - site-xrgg5"

5. In the Portal Admin center, switched the website record associated to the site CSO Portal to "Site 1 - site-xrgg5"

6. This also updated the "Website Bindings" in portal management studio.

7. Also, the portal design studio started working and showed the config from the source website.
8. Next, deleted the website record for the site created in Step 1 via the portal management studio.
9. Next deleted the web template records left hanging after deletion of the website record -

10. Lastly, deleted the website binding that was created by the import of the source configuration. This is not needed as the binding on the newly created site was mapped to the site that was imported.
Thanks for making it so easy.