The term upstate has no relation to the capital city, that was a mistaken assertion. It isn’t used very widely at all, it’s just a local term in a couple of states for their northern portions. Most states don’t have an area that is referred to as upstate.
In NY state it essentially means anywhere north of NYC, the capital city Albany is upstate NY. It doesn't seem to mean north of the capital in any state unless that capital is at the southern part.
It appears so, it was called other things by native people but the Dutch seem to be the first to call it Long Island in the 1600s. Many geographic features in the area have similar sort of names, like Short Hill, East River, West River, Indian Hill, Short Beach, Beaver Swamp, the colonists really weren't very clever with their naming.