The chapter is divided between the descendants of Japheth (10:1-5), Ham (10:6-20), and Shem (10:21-32). There is debate among scholars as to the birth order of Noah’s sons.
Some translate verse 21 so that Shem is the older brother of Japheth (NASB), whereas others understand Japheth to be the eldest (NIV, NKJV). There is also debate as to whether Ham was the middle son (he is always listed second) or the youngest (see 9:24).
We probably cannot know for certain, but I’m inclined toward the view of Keil & Delitzsch (Commentary on the Old Testament [Eerdmans], 1:156) that the birth order is Shem, Ham, and Japheth.
In chapter 10, Japheth’s descendants are probably listed first because they were the most remote and thus the least important to Israel (which is Moses’ common pattern in Genesis, to dispose of the least important matters first).
Since the line of Shem will occupy the rest of the book, it comes last.