Do it from the handlebar end.
Back the thumb nut off of the adjuster (Away from the lever).
Screw the adjuster all the way in (toward the lever). Line the groove in the adjuster up with the slot in the perch and the thumb nut and slip the cable end out of the lever.
That will give you enough slack to get it loose on the brake drum end. Loosen the jam nuts on the cable at the brake drum and that will give you the rest.
You'll sometimes need to actuate the brake arm by hand and do a little fidgeting to get the cable end out of the brake arm. If you've got a fresh cable you may even need to remove the brake arm and rotate it to get it out, but it will come out.
If it turns out that you need to remove the arm be very careful not to use too much force when putting it back on the splines. If you do you'll push the pin out og alignment with the shoes and you'll have to remove the front wheel to fix it.