Do it from the handlebar end.
Back the thumb nut off of the adjuster (Away from the lever).
Screw the adjuster all the way in (toward the lever). Line the groove in the adjuster up with the slot in the perch and the thumb nut and slip the cable end out of the lever.
That will give you enough slack to get it loose on the brake drum end. Loosen the jam nuts on the cable at the brake drum and that will give you the rest.
You'll sometimes need to actuate the brake arm by hand and do a little fidgeting to get the cable end out of the brake arm. If you've got a fresh cable you may even need to remove the brake arm and rotate it to get it out, but it will come out.
The only way you could do it starting at the drum would be to have a really loose cable already or the ability to crush the shoes by actuating the arm with your hand.