Answer:
The solution is [tex]X=A^{-1}(BA)^{2}B^{-1}[/tex].
Step-by-step explanation:
Given two matrix [tex]A, B[/tex] are invertible, so they are not commutative. That is, [tex]AB\neq BA[/tex]. Now given,
[tex]AXB=(BA)^{2}[/tex]
[tex]A^{-1}AXB=A^{-1}(BA)^{2}[/tex]
[tex]XB=A^{-1}(BA)^{2}[/tex]
[tex]XBB^{-1}=A^{-1}(BA)^{2}B^{-1}[/tex]
[tex]X=A^{-1}(BA\times BA)^{2}B^{-1}[/tex]
Since invertible matrices are non-commutative, in the next step we cannot write [tex]BA[/tex] as [tex]AB[/tex]. And so the required answer is,
[tex]X=A^{-1}(BA)^{2}B^{-1}[/tex].
