If $\cos ^{-1} \frac{x}{a}+\cos ^{-1} \frac{y}{b}=\alpha$, then $\frac{x^{2}}{a^{2}}-\frac{2 x y}{a b} \cos \alpha+\frac{y^{2}}{b^{2}}=$

Solution:
(a) $\sin ^{2} \alpha$
We know that $\cos ^{-1} x+\cos ^{-1} y=\cos ^{-1}\left(x y-\sqrt{1-x^{2}} \sqrt{1-y^{2}}\right)$.
$\therefore \cos ^{-1} \frac{x}{a}+\cos ^{-1} \frac{y}{b}=\alpha$
$\Rightarrow \cos ^{-1}\left(\frac{x}{a} \frac{y}{b}-\sqrt{1-\frac{x^{2}}{a^{2}}} \sqrt{1-\frac{y^{2}}{b^{2}}}\right)=\alpha$
$\Rightarrow \frac{x y}{a b}-\sqrt{1-\frac{x^{2}}{a^{2}}} \sqrt{1-\frac{y^{2}}{b^{2}}}=\cos \alpha$
$\Rightarrow \sqrt{1-\frac{x^{2}}{a^{2}}} \sqrt{1-\frac{y^{2}}{b^{2}}}=\frac{x y}{a b}-\cos \alpha$
$\Rightarrow\left(1-\frac{x^{2}}{a^{2}}\right)\left(1-\frac{y^{2}}{b^{2}}\right)=\frac{x^{2}}{a^{2}} \frac{y^{2}}{b^{2}}+\cos ^{2} \alpha-\frac{2 x y}{a b} \cos \alpha$ [Squaring both the sides]
$\Rightarrow 1-\frac{x^{2}}{a^{2}}-\frac{y^{2}}{b^{2}}+\frac{x^{2}}{a^{2}} \frac{y^{2}}{b^{2}}=\frac{x^{2}}{a^{2}} \frac{y^{2}}{b^{2}}+\cos ^{2} \alpha-\frac{2 x y}{a b} \cos \alpha$
$\Rightarrow \frac{x^{2}}{a^{2}}+\frac{y^{2}}{b^{2}}-\frac{2 x y}{a b} \cos \alpha=1-\cos ^{2} \alpha=\sin ^{2} \alpha$