I'll leave what I wrote below as it might help others with troubleshooting, anyway - I worked out a solution; set the result of your patch to a variable then use that variable in the Remove, so in your end button:
Set(varPatchResult,Patch(Time, ThisItem, {end: Now()}));
Collect(SharePoint, varPatchResult);
Remove(Time, varPatchResult)
(This way also means no need for patching an id value either, which I use below)
Here's a vid: https://youtu.be/eszlcmb1Y5g
---
I think 'ThisItem' is updated to a value that differs to what the next lines 'ThisItem' is trying to remove (in the same function).
I tested this by creating a Time2 collection that used strings instead of datetimes. When the start button is pressed it adds the words "start words" and when the end button is pressed in the gallery item, the words "end words" are added but the item isn't removed (because 'ThisItem' has updated? The "end words" were added to the record). Pressing the end button in the gallery item again, now successfully removes the item, maybe because 'ThisItem' hasn't changed (the end button patched "end words" to the record, but since they were already there, the Remove(Time2,ThisItem) removed the item ok.
As an idea, I added an id to each item that is added to the collection when the start button is pressed, with 'id: Last(Sort(Time2,id)).id+1'. Then I changed the Remove in the gallery's end button to 'RemoveIf(Time2,id=ThisItem.id)'. The end button now patches the "end words" and removes the item all in one go.
I tested this back in the Time collection, with the datetimes; and as shown in your video, clicking the end button in the gallery, even multiple times, just updates the end time and never removes 'ThisItem' (maybe because the 'ThisItem' it's trying to remove is now different because the endtime value is changed by milliseconds/seconds).
Using the same idea of adding an id when the start button is pressed, and updating the gallery's end button to 'RemoveIf(Time,id=ThisItem.id)' works correctly, the end datetime is patched and the item is removed.
Here's a vid: https://youtu.be/eszlcmb1Y5g