Based on your description, I would say that the new window is not a browser window but a new application, something like an integrated applet within the browser.
In such cases, you could capture an element from that application using the "Add UI element" in the UI elements repository:

In case the captured element is not a web element, you won't be able to use it with actions under the Browser automation group and you may need to use the equivalent action under the UI automation group.