After Nephi fled from his brethren with those who would follow him from the land of their first inheritance, they settled down in a place they chose to call Nephi. Two things have always stood out in my mind. The fact that Nephi had sisters and the fact that we never hear from Sam. Whether Nephi’s sisters were older and married to Ishmael’s sons, thus making this family the clear choice to bring them out of Jerusalem, or Lehi had at least two more daughters in the wilderness–or perhaps both are the case! It could well be that Nephi had older sisters married into Ishmael’s family and Lehi had at least one more daughter in the wilderness, and thus Nephi had sisters, as in plural.
Anyway, there have been commentaries on this matter for at least twenty years. One more thing that I want to mention is that I have noticed all my life how “silent” Sam is. I can’t help but feel that Sam wasn’t completely normal. He was obviously not deaf, blind or lame. He was strong and thus able to make the journey out of Jerusalem and back twice, once for the plates and once more for Ishmael’s family. Sam was intelligent and able to understand Nephi and believe all his words. His brothers had no problem smiting Nephi and Sam in the cavity of a rock after having fled Laban’s home, although the angel only acknowledged Nephi, at least according to Nephi’s account.
But we never once read about Sam commenting on anything. Sam was able to marry and have children. I, personally, can’t help but feel that Sam might have been unable to speak, as in, he might have been at the very least, mute. This is just speculation, of course but it begs the following questions: If Sam was as faithful as Nephi, then why was Sam not given the opportunity to lead or be a leader? Was he not mighty like Nephi? Why not? The only clue we have, which really isn’t much of a clue, but it does shed light on the fact that Sam was the “least of these.”
2 Nephi 4
10 And it came to pass that when my father had made an end of speaking unto them, behold, he spake unto the sons of Ishmael, yea, and even all his household.
11 And after he had made an end of speaking unto them, he spake unto Sam, saying: Blessed art thou, and thy seed; for thou shalt inherit the land like unto thy brother Nephi. And thy seed shall be numbered with his seed; and thou shalt be even like unto thy brother, and thy seed like unto his seed; and thou shalt be blessed in all thy days.
After Lehi blesses EVERYONE, even his grandchildren, he finally speaks to Sam. Rather than bless Sam according to his rank in the family, Lehi saves him for last. Lehi tells him that he is “like unto [his] brother.” Sam’s posterity becomes Nephi’s posterity. It’s just that simple. And the Book of Mormon history proved it. Nobody ever claimed to be a Samite. Many people, even wicked people claimed to be Zoramites, after Zoram who was pressed out of Jerusalem. So people hundreds and hundreds of years after Lehi, were well aware of their genealogy, but not a single Samite. Ever.
Jacob 1
12 And it came to pass that Nephi died.
13 Now the people which were not Lamanites were Nephites; nevertheless, they were called Nephites, Jacobites, Josephites, Zoramites, Lamanites, Lemuelites, and Ishmaelites.
This brings me back to my first scripture. Jacob said that “the people which were not Lamanites were Nephites; nevertheless, they were called Nephites, Jacobites, Josephites, Zoramites, Lamanites, Lemuelites, and Ishmaelites.” These were the people who went with Nephi. This means that there were indeed one or some children of Laman, maybe even Laman’s wife, as well as one or some children of Lemuel and possibly Lemuel’s wife, and some of Ishmael’s household who abandoned Laman and Lemuel in the land of their first inheritance to settle what they would call the city of Nephi. This may well have included some if not ALL of Nephi’s sisters, any of whom could have been married to Ishmael’s son. This was a very telling verse.
Latter-day Lamanite 
There’s something about Sam you might not have considered yet. When Nephi got the brass plates, he talked about the trouble he had with Laman and Lemuel. When he was returning to the wilderness with Ishmael’s family, he talked about the trouble he had with Laman and Lemuel. When he built the boat, he talked about the trouble he had with his older brothers. Why the difference? Could Sam have been refusing to build the boat, too?
I have never thought that there was anything wrong with Sam, disability-wise. But I have wondered how faithful he was to Nephi. I wonder if he ever got swayed by his older brothers, or just was apathetic. As you mentioned, Zoram earned a place in history, but Sam just sort of faded away.
Nephi clearly tells us that Same believed all his words and the fact that Lehi richly blesses Sam in the end indicates to me that Sam was not a rebellious older brother. When they fled from Laban, both Sam and Nephi were smitten by Laman and Lemuel with a rod in a cavity of a rock. Had Sam been against Nephi, he would probably have participated in the smiting with Laman and Lemuel.
I’ve wondered this same thing. My speculation or thoughts were somewhat more sad though. I’ve wondered if Sam’s posterity, or at least his sons were all killed in an early Nephite war with the Lamanites and for that reason Samites are never mentioned.