Patch is the function you will use to this. The formatting of the patch function is:
Patch(YourListName, Item, {Col1: value, Col2: value, etc...};
For your situation I would try something like:
Patch(SharePointListName, Lookup(SharePointListName, Name = UsersName), {Color: selectedColor});
This will look at your list and find the first record with the same name and then set its color to selectedcolor.
For removing a color you will do the exact same thing except you will set {Color: Blank()} at the end.
If you are wanting to remove the entire entry from your List then you can add an 'X' icon to the gallery item and make its OnSelect property Remove(ThisItem) and it will delete that entire item from the list.
Patch(SharePointListName, Lookup(SharePointListName, Name = UsersName), {Color: Blank()});
I hope this helps!