I disagree with the above 'Solution'. I've tested the functionality out by creating an appointment record from a canvas app. The record is created however the Outlook integration doesn't pickup the record (Server Side Syncronisation).
I have then tried creating the record manually without attendees and it appears the owner still recieves an appointment, this is through a Model Driven Power App.
After this, I looked at all fields on both records and there's no difference. This makes me think there's a hidden settitng (somewhere) which we cannot see through the table, however MS taps into it with the Outlook integration.
Right now, it feels like it's not possible. But MS, if you have an answer, I'd love to know.
@Hancha the quick and dirty way I've found is create a seperate field on the appointment, then use a flow to populate the attendee field. Seems to work, but less than ideal if the Outlook integration doesn't then pick it up.
