There is no record of what happened to those Nephites (if any) who did not follow Mosiah to Zarahemla. When Zeniff's colony returns, the land is filled with Lamanites with no apparent sign of any Nephites left behind. Whether they were destroyed, departed some other way, or assimilated into the Lamanite population, the record is silent.