DuoS changing CAM stream framerate

Hi,

I’m using the Milk-V cam on a Duo-S, imaged with SDK 2. I’m using the camera test example with the vision model enabled.

I’m trying to access the stream over a slow connection and there’s a lot of frame losses. There was some loss over lan too, but nearly not as much.

I would rather have a low frame rate of reasonable quality than what I currently manage to squeeze out. Is there a way to change the encoding frame rate for the stream?

(I can see that most cam related questions fall into a black hole, but maybe this one is easy enough…)

made a little table of all the decoding etc. issues I find in the streams, maybe it helps others. There’s some that are just network issues but a good number point at little things with the encoder etc., solving a few should step by step should allow to make large progress

Mplayer component Message Meaning Impact Causes
packet reader? max delay reached. need to consume packet packet loss or not generated
RTP missed <NUM> packets packet loss, not generated
RTP bad cseq <SEQNUM> expected <SEQNUM> packet loss, retransmits, memory corruption in encoder process
RTP dropping old packet received too late packet loss, retransmits
h264 error while decoding MB <NUM> <POS?>
h264 mb_type <ID> in P slice too large at <POS> <POS>
h264 concealing <NYM> DC, <NUM> AC, <NUM> MV errors in P frame
h264 ]out of range intra chroma pred mode

There’s some that are just network issues but a good number point at little things with the encoder etc., solving a few should step by step should allow to make large progress.
The questions are mostly if there are stalls in sending / bufferbloat or similar things that IP stack tuning can already handle and which are bugs in the encoder and whether they are in the hardware processing or the example application.
I think the best approach is

  1. start from a lower framerate,
  2. look at generating only valid stream data
  3. identify stalls, ensure a 10/25/30fps stream is generated constantly with h264 + npu hardware engaged
  4. tune transmission
  5. ensure all out-of-order transmissions are addressed and not caused by processing pipeline side
  6. make sdk patches for 5) since this will be to noone’s disadvantage

thus my question about lowering the fps as a first step.

  • List item

could it be this from the SDK / examples?

Hello,

We’ve been adding camera support for the Duo S to armbian. You can take a look at this PR, compile and play around. This works with the mainline kernel. Using the tools in the linked repository (duos-camera), you can create streams with lower QP etc.

The TPU is added as well so you can run inference: sophgo-sg200x: add TPU support for the Milk-V Duo S (mainline port of the vendor driver) by leafpmpmp · Pull Request #10760 · armbian/build · GitHub

Cheers

1 Like

Awesome, this will save me so much of blindly stabbing around.
Bless you!

q: I see the image is using/running linux-image-edge-sophgo-sg200x-arm64 but I can’t get - or even find via web searches - the headers to go with it as per your instruction to install apt install build-essential linux-headers-<branch>-sophgo-sg200x-<arch> → would be linux-headers-edge-sopho-sg200x-arm64. This is a trixie image (so ‘edge’, right?). also can’t find it on pkgs.org.

I mean I see I’m on a -trunk version, but what I don’t get there was a downloadable SD card image, it boots flawless and has obviously been generated by an armbian build system since it’s there, so the packages ought to be somewhere. I’ve tried to add deb-src entries but no luck so far. so where did your package come from?

q: what kinda console are you using, are there plans to als oadd a usb-cdc or what it’s called network over usb thing?
half-answer: my issue was that it didn’t show up on ethernet, cause was a bad lan cable. 3 hours to figure out how to enable serial ports on an old rpi3b with 1GB ram and from there it was about 30 seconds to get serial console over to the Duo S and another 30 seconds to put in a new cable. I could have just tried that… :slight_smile:

self-answering, as far as i know by now:

  • armbian community image builds are run through a different process than the supported armbian images. the kernel headers aren’t installed in course of the build and there’s no (public constantly available) repo backing them.
  • there might be some nightlys that have packages but it’s dependend on unrelated factors
  • there are unresolved project issues that cause the responsible releng/qa/infra person to be overloaded which hasn’t been solved in a large number of years
  • for that reason there’s no pragmatic changes to solve this common issue since (assumedly) once this is solved there’ll just be another issue that follows it
  • funding/sponsoring needs to be organized to make the community build infra covered cost-wise
  • i’m gonna donate a litte bit because i also just have a little bit
  • if i need the package I can’t rely on the infra that created the successfully working image
  • to get headers I need to run their own build armbian for my board, this uses the repo at armbian-build
  • per the docs at armbian build framework - basic you first install dependencies using the script (that’s as root, change the directories permissions back to being owned by your build user afterwards)
  • resulting command is $ ./compile.sh build BOARD=milkv-duos-arm BRANCH=edge RELEASE=trixie (or ‘riscv’ instead of ‘arm’ or ‘bleedingedge’ instead of ‘edge’, i’ll find out about that later)
  • The default images, and the one one can download is ‘minimal’ and that is likely why we don’t get headers with this image
  • I’m just using ARM because I’m running serial over the ARM RPi3 and it might be helpful in case I mess up libc or something similar. I’ll switch over to RISCV afterwards, in hopes that over the next year there will be more successful uses of the second RISCV core.

I’m running those at home instead of any larger cluster because my homeserver’s fans have been pissing me off ever since I had to turn up the heating and now for one that thing will at least have to earn the priviledge!

I know this is kind of off-topic for the camera itself, but it’ll still be useful to be able to test the camera, so here you go.

Running the non-minimal build off main branch, I got the kernel & headers.

floh@blackbox:/vmdata/build$ find output -name "*deb" 2>/dev/null
output/debs/armbian-firmware_26.11.0-trunk_all__1-SAf7fa-Ba2de-R448a.deb
output/debs/armbian-bsp-cli-milkv-duos-arm-edge_26.11.0-trunk_arm64__1-PC93a8-V6090-Hd475-B3a4b-Ra462.deb
output/debs/linux-image-edge-sophgo-sg200x-arm64_26.11.0-trunk_arm64__7.2.15-S4977-D0000-P38d0-Cb23e-H7e36-HK01ba-Vc222-B0577-R448a.deb
output/debs/linux-u-boot-milkv-duos-arm-edge_26.11.0-trunk_arm64__2021.10-Sbb74-P3083-H1f61-V689c-Bb936-R448a.deb
output/debs/extra/trixie-utils/base-files_26.11.0-trunk-13.8+deb13u7-trixie_arm64__1-trixie-1armbian1-Bced1-U13.8--deb13u7-R55fa.deb
output/debs/armbian-bsp-cli-milkv-duos-arm_26.11.0-trunk_arm64__1-PC93a8-V6090-Hd475-B3a4b-Ra462.deb
output/debs/linux-dtb-edge-sophgo-sg200x-arm64_26.11.0-trunk_arm64__7.2.15-S4977-D0000-P38d0-Cb23e-H7e36-HK01ba-Vc222-B0577-R448a.deb
output/debs/linux-libc-dev-edge-sophgo-sg200x-arm64_26.11.0-trunk_arm64__7.2.15-S4977-D0000-P38d0-Cb23e-H7e36-HK01ba-Vc222-B0577-R448a.deb
output/debs/armbian-zsh_26.11.0-trunk_all__1-SA4b65-Ba2a8-R448a.deb
output/debs/linux-headers-edge-sophgo-sg200x-arm64_26.11.0-trunk_arm64__7.2.15-S4977-D0000-P38d0-Cb23e-H7e36-HK01ba-Vc222-B0577-R448a.deb

slamming them over the running edge image’s packages works:

root@milkv-duos-arm:/install/debs# dpkg -i *deb
Selecting previously unselected package armbian-bsp-cli-milkv-duos-arm.
[...]

dpkg: warning: downgrading armbian-firmware (26.11.0-trunk.108) to (26.11.0-trunk)
Preparing to unpack armbian-firmware_26.11.0-trunk_all__1-SAf7fa-Ba2de-R448a.deb ...
Unpacking armbian-firmware (26.11.0-trunk) over (26.11.0-trunk.108) ...

you see it’s considering a downgrade, so maybe i’ll better do a bleedingedge build as well, but that is just deciding the amount of already ported features, you can make your own choices about that. The build I had fetched was 107 and now i’m using parts of 108 so i think this is the perfect spot to aim for.

it later missed bison and flex for installing the kernel, but those are in the main repos so an apt install -f is enough to fix everything!

sorry, I missed these messages before! Do you need any help still? Does everything work now? I usually build my own debs (./compile.sh BOARD=milkv-duos-arm/-riscv kernel BRANCH=edge), then install them using sudo apt install --reinstall ./linux-*.deb. I’ll try vibe coding the ISP soon.

q: what kinda console are you using, are there plans to als oadd a usb-cdc or what it’s called network over usb thing?

The USB gadget is working. You can enable it by enabling the usb-gadget overlay. I was wondering whether to use the USB host or gadget as the default, decided to use the host mode in the end.

To use H.264 encoding you also need an overlay enabled.

You will get the most recent firmware if you switch to the beta repository (using armbian-config Software etc ). Otherwise I think you’re getting older armbian-firmware.

The packages will be sometimes seen as “downgrades” because of the random number in the hash I think.

As for your “second RISCV core” question - this one is independent, and can be used by the ARM or the RISCV main core. Not fully enabled yet as it needs some work, but you can enable an overlay for it.

Hi,

I got most of it solved/figured out. I went with the ‘bleedingedge’ build, i only lost a lot of time till I understood the community packages don’t have headers.
The advanced section of the armbian-build docs has helped a lot with getting oriented.

At the moment I’m trying to find out how one gets not just the linux…-headers package but the full sources installed as well, since I’m missing stuff from /usr/src/linux-headers-7.3.1-bleedingedge-sophgo-sg200x-arm64/arch/arm64/tools.
edit: still looking for the full kernel sources, and github is rate-limiting my wget attempts - meh.

I also saw you already got a dkms package in the works, so this is purely hobby work to play instead of checking back in a few weeks.

I’m planning to watch this video with some more care: https://www.youtube.com/watch?v=3h7wgR5fYik&t=1066s and learn how to use the edge smoothing etc. things over the next few months.

I got one jetson, one rpi3 and one Duo S with a camera each and plan to put each of them in front of a test image.

I only know about the armbian overlays since 2 days, so i’ll try to figure out the additional parts, hw enc and secondary core.
myself I’m not competent enough to do something good with the 2nd core but I am kinda sure others will break through that wall over the next months as well. It’s so much progress now that it’s just gonna happen.

Today I also looked into crypto offload, unfortunately it seems noone has tried that so far, it seems there’s also no crypto offload examples in the sophgo sdk tree of kernel 5.10, that’s odd.

Other thing:
Maybe you want to make a dedicated armbian post that lists the overlays etc. that make up the ideal setup from your PoV?
I can’t thank you enough for the work you did.
But: Thank you!

as an advice for others: Back then I bought my Duo S batch without eMMC. anyone who is looking into buying some today, I really recommend getting the eMMC one since it gives more versatile options. It’s worth it!

I think the usb gadget mode is the better default
Pro:

  • it matches with the stock firmware
  • at least in the case of these normally headless devices it’s also quite useful
  • there are two USB ports, so there’s no total loss of options if it’s set up this way
  • allows using it without connecting ethernet
  • allows using it without serial cable in reach
    Con:
  • no replacement distro does it by default
  • it doesn’t matter for people who are used to armbian already and set up the config via the sd card (but no chance for eMMC, unless loopback mounting the image with kpartx)

I think in total it could really look in favor of having the gadget mode. I often played with boards like this when i was on a train and had nothing to do. USB-C can do that even if you just got it stuffed in a backpack.

Again, I’ll reply more at legth later but I never usually set up armbian via the SD card, usually I just connect it to the Internet and use ssh. With the latest armbian-imager (2.5.0) you can even create profiles to set up wifi and other stuff automatically, no need for manual config file editing.

1 Like

i got the stream up now :slight_smile: was missing the overlays and hadn’t spotted the compiler flags for some of the tools.

1 Like

OK, I’m not sure about some of the issues here.

  1. linux headers can be installed easily, for example: sudo apt install linux-headers-edge-sophgo-sg200x-riscv64. They can also be installed from armbian-config, please browse its options, the kernel section. If you don’t know a package name, you can use TAB for autocomplete (the most useful key on Linux). Don’t use wget with github, just clone the repository (armbian/build) and use it.
  2. The kernel sources are in build/cache/sources/linux-kernel-worktree/ after you build an image.
  3. The video thumbnail had the guy doing the youtube [open mouth] face so I could not take a look, sorry :smiley:
  4. Overlays have self-explanatory names (like video-codec) if you know what you want to do. If you don’t know them, then you probably don’t need them. There is no “ideal setup”, there are good setups for different purposes. Maybe I’ll make a post though.
  5. Yup, the eMMC version is worth it. You can install the system to the eMMC using sophgo-emmc-install if you wish. Maybe I’ll fold it into armbian-install some day but it’s tricky with this one.
  6. Claude did most of the work, I was just testing and giving orders.
  7. No need to figure out the hw encoder, it’s already done: GitHub - lukaszsobala/duos-camera: GC2083 camera, H.264/JPEG encoders and libcamera on Milk-V Duo S. H.265 WIP · GitHub - with the ISP almost done, encoding uses like 2% CPU cv1800-vi: add the ISP path to YUV by lukaszsobala · Pull Request #1 · leafpmpmp/sg200x-camera-dkms · GitHub

Not sure if I missed anything

1 Like

@darkfader in 26.11 armbian you’ll have the ISP too. Or get a community image. In any case, everything works with mainline now and is open sourced. You can use the ISP.