Grant, I used an aluminum drift and bigass ball pein hammer or single jack. It was hard and fought me most of the way out.
The arbor is to keep the little shim pack in alignment as you drive the idler shaft back in place. Without it you risk the shims moving around and at best not allowing the shaft to pass. At worst you'll munch or bend part of the shim which could cause other problems. I think I'll set the case on it's side and carefully insert the shaft, with the shims on the low side of the assembly. Once the shaft is past them it should be a breeze (a hammer induced breeze).
Chris