I was wondering the same thing actually. What i found is that the PSP actually supports full resolution (480x272) h.264 encoded video as of PSP firmware 3.30. Firmware before that did **NOT** support the higher res videos natively. The default profiles in mencodr/MCEBuddy are for spec'd for the older firmware PSP. More info here: http://www.engadget.com/2007/03/28/psp-firmware-hits-3-30-with-bigger-vids-and-better-psone/
So that means if you install the newer firmware on your PSP, you can play higher resolution and better looking video on the PSP. You need to add the following profile to your MCEBuddy "mencoder.conf" file. I built this profile by combining some of the stuff that was already present in another h.264 profile, and info from this link: http://videogeek.shacknet.nu/index.php?entry=entry070917-130603
It works pretty good so far on my firmware "oe 3.5" psp, and should work on any PSP with firmware 3.30 or later. If anyone has any recommendations on how to improve the profile, please post and let me know, i'm a noob at mencoder profiles/MCEbuddy.
[x264-PSP]
profile-desc="H.264 PSP Full Rez (FW3.30) (480:272)"
vf=pullup,softskip,scale=480:-10,harddup,unsharp=l3x3:0.7,expand=480:272
lavdopts=threads=2
ovc=x264=yes
x264encopts=bitrate=800:global_header:partitions=all:trellis=1:vbv_maxrate=768:vbv_bufsize=2000:level_idc=30
oac=faac=yes
faacopts=br=128:raw=yes:mpeg=4:tns=yes:object=2
af=volnorm=2:lavcresample=24000
ofps=30000/1001
of=lavf=yes
lavfopts=format=mp4
#FileExtension=mp4