Here's a couple of obvious things to try.
As Riley pointed out, are you sure the hardware settings (via jumpers,
dip switches, etc.) are correct on the board for the last two ports?
You can also check to see if there's a UART there at all via the
following:
setserial /dev/cua4 autoconfig -v
and seeing what it says. If it says "Uart: unknown", then it means that
the board is likely not configured to be at the I/O port which you
selected.
If you can find the UART, then the next thing to test is whether or not
you're receiving interrupts on that port. One easy way to test that is
to monitor /proc/interrupts and see if the interrupt corresponding to
the port is working. If not, then it's probably an interrupt problem.
Hope this helps!
- Ted