If for non-zero x,

Solution:

Given:

$a f(x)+b f\left(\frac{1}{x}\right)=\frac{1}{x}-5$   ...(i)

$\Rightarrow a f\left(\frac{1}{x}\right)+b f(x)=\frac{1}{\frac{1}{x}}-5$

$\Rightarrow a f\left(\frac{1}{x}\right)+b f(x)=x-5$    ....(2)

On adding equations (i) and (ii), we get:

$a f(x)+b f(x)+b f\left(\frac{1}{x}\right)+a f\left(\frac{1}{x}\right)=\frac{1}{x}-5+x-5$

$\Rightarrow(a+b) f(x)+(a+b) f\left(\frac{1}{x}\right)=\frac{1}{x}+x-10$

$\Rightarrow f(x)+f\left(\frac{1}{x}\right)=\frac{1}{(a+b)}\left[\frac{1}{x}+x-10\right]$    ...(iii)

On subtracting (ii) from (i), we get:

$a f(x)-b f(x)+b f\left(\frac{1}{x}\right)-a f\left(\frac{1}{x}\right)=\frac{1}{x}-5-x+5$

$\Rightarrow(a-b) f(x)-f\left(\frac{1}{x}\right)(a-b)=\frac{1}{x}-x$

$\Rightarrow f(x)-f\left(\frac{1}{x}\right)=\frac{1}{(a-b)}\left[\frac{1}{x}-x\right]$    ....(iv)

On adding equations (iii) and (iv), we get:

$2 f(x)=\frac{1}{a+b}\left[\frac{1}{x}+x-10\right]+\frac{1}{a-b}\left[\frac{1}{x}-x\right]$

$\Rightarrow 2 f(x)=\frac{(a-b)\left[\frac{1}{x}+x-10\right]+(a+b)\left[\frac{1}{x}-x\right]}{(a+b)(a-b)}$

$\Rightarrow 2 f(x)=\frac{\frac{a}{x}+a x-10 a-\frac{b}{x}-b x+10 b+\frac{a}{x}-a x+\frac{b}{x}-b x}{a^{2}-b^{2}}$

$\Rightarrow 2 f(x)=\frac{\frac{2 a}{x}-10 a+10 b-2 b x}{a^{2}-b^{2}}$

$\Rightarrow f(x)=\frac{1}{a^{2}-b^{2}} \times \frac{1}{2}\left[\frac{2 a}{x}-10 a+10 b-2 b x\right]$

$=\frac{1}{a^{2}-b^{2}}\left[\frac{a}{x}-5 a+5 b-b x\right]$

Therefore,

$f(x)=\frac{1}{a^{2}-b^{2}}\left[\frac{a}{x}-b x-5 a+5 b\right]$

$=\frac{1}{a^{2}-b^{2}}\left[\frac{a}{x}-b x\right]-\frac{5(a-b)}{a^{2}-b^{2}}$

$=\frac{1}{a^{2}-b^{2}}\left[\frac{a}{x}-b x\right]-\frac{5(a-b)}{(a-b)(a+b)}$

$=\frac{1}{a^{2}-b^{2}}\left[\frac{a}{x}-b x\right]-\frac{5}{(a+b)}$

Hence,

$f(x)=\frac{1}{a^{2}-b^{2}}\left[\frac{a}{x}-b x\right]-\frac{5}{(a+b)}$