Exactly. Gasoline, for example, is remarkably non-toxic, but it will cause instant chemical burns to your throat and lungs, possibly killing you far below the (chemically) lethal dose.
Methanol will turn you blind at a quarter of the listed dose, and those two are just from the top of my head.
Or aspects like arsenic staying in your body a very long time, or the fact that LSD is psychoactive in microgram doses, so you'd need thousands of tabs to die.
I wonder how they came up with the LD50 of all those materials, like THC and LSD. Is this based on theoretical calculation, in vitro tests, or on a (assumably) very small sample of known deaths?
Step 1: Feed/Inject mutliple rat populations with different concentrations
Step 2: See how many die.
Step 3: The concentration which causes 50% of the population to die is the LD50