Prove the following
Question:

$\tan ^{-1}\left(\frac{a \cos x-b \sin x}{b \cos x+a \sin x}\right),-\frac{\pi}{2}<x<\frac{\pi}{2}$ and $\frac{a}{b} \tan x>-1$

Solution:

Let $y=\tan ^{-1}\left(\frac{a \cos x-b \sin x}{b \cos x+a \sin x}\right)$

$y=\tan ^{-1}\left[\frac{\frac{a \cos x}{b \cos x}-\frac{b \sin x}{b \cos x}}{\frac{b \cos x}{b \cos x}+\frac{a \sin x}{b \cos x}}\right]$

$y=\tan ^{-1}\left[\frac{\frac{a}{b}-\tan x}{1+\frac{a}{b} \tan x}\right]$

$y=\tan ^{-1} \frac{a}{b}-\tan ^{-1}(\tan x)$

$\left[\because \tan ^{-1}\left(\frac{x-y}{1+x y}\right)=\tan ^{-1} x-\tan ^{-1} y\right]$

$\Rightarrow y=\tan ^{-1} \frac{a}{b}-x$

Differentiating both sides with respect to $x$

$\frac{d y}{d x}=\frac{d}{d x}\left(\tan ^{-1} \frac{a}{b}\right)-\frac{d}{d x}(x)=0-1=-1$

Thus, $\frac{d y}{d x}=-1$.