Using ultraleap on Pico neo 3 and streaming to steam

andyczerwinski 2022-08-01 10:11:29 UTC #1

Wondering if anyone has got examples of using a Pico neo 3 on hmd setup and attempted to use the streaming solution to steamvr? Or would you need to create something akin to sending the info to the pc via some other means?

PLeoncini 2022-10-11 11:05:55 UTC #2

Yes, I did it via VirtualHere USB client-server software, which does as the device would be virtually plugged in to the PC (yet it actually is on the Pico).
But it worked for the Leap Motion, not for an IR-170 unit (specifically, the Hand Tracking module from Pimax).
The former required an adapter cable from the Leap Motion USB port to the Pico Neo 3 USB-C.
The latter only required a very small (20cm) USB-C "extension" cable.
Yet, what do you mean with "the streaming solution"? AFAIK there's no standard mechanism to stream Ultraleap data from a Pico Neo 3 to a PC in such a way to be seen by SteamVR (the VIVE Focus 3 Business Assistant sw does it instead). The one I tried does it, because it is virtually the same as having plugged the Leap Motion into the PC USB port. Even if I don't read Ultraleap data via SteamVR Skeletal Input, or via the OpenXR Hand Tracking extension.

andyczerwinski 2022-10-11 14:13:00 UTC #3

Ah. That's a good solution so the ultraleap thinks it's on the pc. Assume this is available somewhere. What was the latency like?

PLeoncini 2022-10-27 12:14:16 UTC #4

Yep, the PC thinks the Ultraleap is attach to it. Latency is not perceivable, nor the device framerate decreases. Rather, you'll not get a stable framerate, it is actually quite variable yet still high.
I have an update: the IR-170 module also works but with a different cable chain (the shortest, simplest one didn't). Thus, now I'm finding for a 20-25cm USB-C extension cable that works.
You can find the key software pair at virtualhere.com.

fmeinke 2022-11-22 21:32:33 UTC #5

@PLeoncini Do you have any further information regarding your setup? We are currently struggling with streaming the Input of the SIR170 to a Windows10 machine. We are using the Pico Neo 3 Pro and the SIR170. The SIR170 is connected via a Micro-USB to USB-C Cable. We tried using Virtualhere. The Ultraleap Control Panel recognizes the Device but does not receive any images.

PLeoncini 2022-11-22 22:32:51 UTC #6

mmm I don't know other IR-170 module than the Pimax hand tracking one, yet it is USB-C male, not Micro-USB. And, it is strange it were a Micro-USB because IMHO it leverages the full USB 3.0 throughput, or such.

And what about receiving actual data frames? i.e. for hand tracking? Anyway, my setup with the arranged long cable also remotize images.
Is the small circle on the Ultraleap logo green in the tray icon? i.e. the device is actually connected (ok, remotized).
Furthermore, how fast is the WLAN between the computer and the headset? it could be that less of a certain throughput performance, images are not enabled at Ultraleap driver level (Ultraleap people, could it be?)

PLeoncini 2022-11-24 17:06:29 UTC #7

I tried to replicate your setup, with images enabled. Please look at these photographs.

It includes a weird cable-connector chain passing through a USB 3.0 type A to C cable, a USB-C gender on the device side, and a USB type A to C on the Pico side.
On the PC screen, leftmost: the VirtualHere USB Client showing the Pico USB Android hub, and the connected Rigel device (actually a Ultraleap IR-170). Large window: the Ultraleap control/preview panel showing device images fluidly displaying other than the calculated hand skeleton. The WLAN runs at 568Mbps.

That's strange, really. But now you have the counterproof it can work. Maybe it's a question of cable throughput in your setup, i.e. the "electric" connection testifies the device is connected, yet the throughput isn't sufficient. Or, as said previously, the WLAN isn't, in turn.
Anyway, believe me, finding a single working cable it's hard, and for a chain you have to hold on trying all the combinations of the pieces you have, as maybe you can look at on the table in the photo.

fmeinke 2022-11-25 17:40:17 UTC #8

Ah well I see you have a different device. We got it running eventually but now we have high ups and downs in the frame rate using Virutalhere. I will come back to this post later with some additional photos to describe the problems we had, how we fixed them and what we will try next to eliminate the problems with the framerate.

PLeoncini 2022-11-26 10:38:37 UTC #9

About the framerate, also in my setup there's a meaningful variability wrt. physically plugging the device in the computer - but the downs are never to an unusable threshold. I cannot imagine how the variability can damagely affect the application, maybe you could explain a bit your application purpose.
And, curious also how you'll proceed in trying to oercome the framerate problem. Maybe I'd do the same in mine.

PLeoncini 2022-11-26 10:45:28 UTC #10

PS: in my application, the physics runs let's say at 300+Hz. In the past, with a similar yet older setup based on the TPCAST for the VIVE, the hack to its Rasberry/Raspbian in order to run the VirtualHere server, and the old good Leap Motion, the device framerate was strangely quite low, let's say 30-40Hz. I set up an interpolator-extrapolator for the 6-DoF hand data input to the manipulation simulation for the missing data frames, which seemed to work quite the same as having the device plugged in physically.

PLeoncini 2022-11-28 15:24:15 UTC #11

Sorry, I didn't know the IR-170 module for the Pico Neo 3 had the Micro USB-B 2.0 interface socket. The same for the Pimax has USB-C male, so I supposed the full throughput of USB 3.0 was needed.

fmeinke 2022-11-29 10:23:33 UTC #12

Sorry for the delayed response. I am swamped currently and didn't have time to touch the device the last days. I will not forget to post an update.

Thank you for your incredible engagement. You seem like you know your stuff.