How to go about creating a new transport layer protocol?
Study TCP and UDP for the IP world, and TP0-TP4 transport protocol family for the OSI world. Compare their respective strengths and weaknesses. Find why they were designed the way they were designed, with these strengths and weaknesses. Balance it all, and build your own model depending on the environment it is targeted at.