A half answer to try! Not sure what the original audio format is (assuming AC3) but you could just leave the 5.1 channel audio as is and mux this with the H.264 encoded video. Mencoder has an option to retain the original AC3 stream so you can adjust the relevant profile line in the "mencoder.config"...