How to replicate 2-hand rotation like Tiltbrush?

Question

So im trying to make a two-hand VR rotation mechanic, where the user grabs an object with one hand, and then with the other hand is able to rotate the object around a pivot point. That pivot point is gonna be a weighted point along the line between the two motion controllers closest to the one that is kept most still. The rotation should appear natural. I realize this is a bit hard to explain, so with reference in Google Tiltbrush, i recorded a video showing how i want it to behave: Tiltbrush rotation

Im interrested in the math behind doing such a rotationmechanic so i can replicate it myself in UE4.

I've created a video showing a preliminary result i got, but where i don't have any control over pivot point (which is the center of the object) and the object snaps to a forward direction relative to the 2nd hand grabbing: My current attempt

I wanna also say that im not too strong in math, so i rely highly on the math functions available in UE4. When that's said, im prepared to have to figure out how to convert the matrixtransformations into whatever i need to use in UE4, as long as i understand the logic necessary to get the desired behavior.

So i suppose the problem i want to try solve, is a way to get a lookat-rotation that doesn't suffer from any singularity/gimbal lock related issues at any angles, and rotate around a dynamic pivot point. And rotate relative to it's current location (so it rotates from it's current rotation when the 2nd hand grabs).

Any constructive input will be welcome. Thanks.

Answer 1

I got this working correctly and the trick is that you want to keep applying the most recent rotation delta from each frame to your object's active transformation- NOT from your object's initial transformation at the time of "capture".

The fact that this is done in most two handed VR apps can be gleaned from testing rotations in a given order. Try it in TiltBrush: start with two hands at the same Y some distance horizontally apart, then rotate them around Z 90 deg (as if you are turning a wheel) and then rotate them around X 90deg (as if you are wrapping something around your hands). Your resultant model orientation will be different than had you just rotated the model about Y 90 deg. This shows that the order of transformations matters, even if the final positions of your hands are at the same orientation from one another. This implies that you cannot create a single transformation algorithm that maps from initial positions & orientations to some active position & orientation.

I've found the solution requires applying three transformations:

First: multiply the rotation delta from the previous difference vector between the two hands and the new difference vector (in Unity, you can get this using Quaternion.FromToRotation). This will get you most of the way there.

Then, multiply your new orientation with two additional transformations representing the delta in rotation of each hand's orientation. I've found that this additional transformation should be scaled by half to get a good result (you can use Slerp of 0.5f from the default identity rotation) This gives you a sort of averaging of the influence of each of the two hands' orientations on the overall rotation.

Lastly, because the two hands orientation changes will modify the rotation angle of your model in a way that could "de-attach" the model from your hands, simply use a lookat function (in Unity, you can get this using Quaternion.LookRotation) with the resultant up vector of all these transforms as the up direction to get a final rotation (this essentially forces the rotation component applied from each hand to only influence the rotation about the axis between both hands).

This kind of algorithm will give you a realistic feeling rotation (with realistically changing up direction) and won't ever "jump" to a wildly different rotation (because you're constantly applying a rotation over time).

That said, you'll find that most 2-handed free-rotation algorithms exhibit some kinds of behavior that you may not find suitable with some movements(even in your favorite VR app) so the ideal solution will probably require additional thought to how you finally place your objects (snapping?) and if you want to limit rotation to a specific axis (It is possible -and easier- to create a consistent, completely order-independent transformation algorithm if you lock rotation to a single axis, say, Y).