Fix matrix binary ops for large case where int4 isn't enough

Fix matrix binary ops for large case where int4 isn't enough #608