Solve this

Solution:

Given: $f$ is continuous on $(0, \infty)$

$\therefore f$ is continuous at $x=1$ and $\sqrt{2}$

At $x=1$, we have

$\lim _{x \rightarrow 1^{-}} f(x)=\lim _{h \rightarrow 0} f(1-h)=\lim _{h \rightarrow 0}\left[\frac{(1-h)^{2}}{a}\right]=\frac{1}{a}$

$\lim _{x \rightarrow 1^{+}} f(x)=\lim _{h \rightarrow 0} f(1+h)=\lim _{h \rightarrow 0}(a)=a$

Also,

At $x=\sqrt{2}$, we have

$\lim _{x \rightarrow \sqrt{2}^{-}} f(x)=\lim _{h \rightarrow 0} f(\sqrt{2}-h)=\lim _{h \rightarrow 0}(a)=a$

$\lim _{x \rightarrow \sqrt{2}^{+}} f(x)=\lim _{h \rightarrow 0} f(\sqrt{2}+h)=\lim _{h \rightarrow 0}\left[\frac{2 b^{2}-4 b}{(\sqrt{2}+h)^{2}}\right]=\frac{2 b^{2}-4 b}{2}=b^{2}-2 b$

$f$ is continuous at $x=1$ and $\sqrt{2}$

$\therefore \lim _{x \rightarrow 1^{-}} f(x)=\lim _{x \rightarrow 1^{+}} f(x) \quad$ and $\quad \lim _{x \rightarrow \sqrt{2}^{-}} f(x)=\lim _{x \rightarrow \sqrt{2}^{+}} f(x)$

$\Rightarrow \frac{1}{a}=a$ and $b^{2}-2 b=a$

$\Rightarrow a^{2}=1$ and $b^{2}-2 b=a$

$\Rightarrow a=\pm 1$ and $b^{2}-2 b=a$          .....(1)

If $a=1$, then

$b^{2}-2 b=1 \quad[$ From eq. (1) $]$

$\Rightarrow b^{2}-2 b-1=0$

$\Rightarrow b=\frac{2 \pm \sqrt{4+4}}{2}=\frac{2 \pm 2 \sqrt{2}}{2}=1 \pm \sqrt{2}$

If $a=-1$, then

$b^{2}-2 b=-1 \quad$ [From eq. (1)]

$\Rightarrow b^{2}-2 b+1=0$

$\Rightarrow(b-1)^{2}=0$

$\Rightarrow b=1$

Hence, the most suitable values of a and b are

$a=-1, b=1$ or $a=1, b=1 \pm \sqrt{2}$