Yeah, the stock u-boot has that problem. I think it stopped when I hit something, but there was only one key it would stop for. Maybe escape or enter? Ctrl-C? I don’t remember; I started building my own u-boot.
And about “No matter which key I pressi, it proceeds with boot.”
I had the same issue (don’t exactly remember on which board) and resolve it only by installing newer uboot.
You can get bootchain (bootloader_P550.bin) from this repo and flash it to your board