Updating geoserver did not fix my problem: my layer still had some duplicate columnnames. This might not be such a big problem: everything is drawn correctly, WMS calls work, but WFS calls gave the irritating yet predictable error
ORA-00918: column ambiguously defined. Annoying.
So how does one find column-names for a table in oracle? With a query like:
select * from dba_tab_columns where table_name = 'YOUR_TABLE_NAME';
and all of a sudden I saw the same set of column-names, with some duplication. Apparently my oracle database contains the table twice, in two different schema's. Since my user had the permissions to access the other schema, it seems geoserver does not limit the query to the (specified) schema at all.
The fix then was easy: make the other schema unaccessible. In my case the second schema was for testing purposes, so I could just delete it.