I think I've found a sign error in the equations for point to plane (and point to point if you don't use the closed form I guess). Specifically in the matrix A.
My point to plane wasn't working right, and on a hunch I tried inverting my rotation matrix R, which worked. So I decided to go though the details of the derivation and I think all the signs on the X columns in A are reversed. I changed them in my code and I didn't need to invert my rotation anymore.
That being said, the signs look correct in that they are equivalent to the base problem. Maybe the signs in the small rotation matrix are wrong?
I think I've found a sign error in the equations for point to plane (and point to point if you don't use the closed form I guess). Specifically in the matrix A.
My point to plane wasn't working right, and on a hunch I tried inverting my rotation matrix R, which worked. So I decided to go though the details of the derivation and I think all the signs on the X columns in A are reversed. I changed them in my code and I didn't need to invert my rotation anymore.
That being said, the signs look correct in that they are equivalent to the base problem. Maybe the signs in the small rotation matrix are wrong?