I believe that -color-matrix should reproduce the following -fx expression. They produce the same results when using values between 0 and 1, but different results when I have values greater than 1 and negative values. I think that the fault is in -color-matrix.
Input:

This works fine -- the results are the same:
rf=0.3
gf=0.6
bf=0.1
convert zelda3.png -separate \
-fx "$rf*u[0]+$gf*u[1]+$bf*u[2]" \
zelda3_gray_mix1a.png

rf=0.3
gf=0.6
bf=0.1
convert zelda3.png -color-matrix "\
$rf 0 0 \
0 $gf 0 \
0 0 $bf \
" -separate -evaluate-sequence add \
zelda3_gray_mix1b.png

But these produce different results and I believe that -color-matrix is the wrong one.
rf=-0.40
gf=1.44
bf=-0.04
convert zelda3.png -separate \
-fx "$rf*u[0]+$gf*u[1]+$bf*u[2]" \
zelda3_gray_mix2a.png

rf=-0.40
gf=1.44
bf=-0.04
convert zelda3.png -color-matrix "\
$rf 0 0 \
0 $gf 0 \
0 0 $bf \
" -separate -evaluate-sequence add \
zelda3_gray_mix2b.png

P.S. I noticed that there was a similar old bug report that might be related. See viewtopic.php?f=3&t=16787