Let the rectangle of length l and breadth b be inscribed in circle of radius a.

Then, the diagonal of the rectangle passes through the centre and is of length 2a cm.
Now, by applying the Pythagoras Theorem, we have:
${\left(2\mathit{a}\right)}^2$ = ${\mathit{l}}^2+{\mathit{b}}^2$
⇒ ${\mathit{b}}^2$ = $4{\mathit{a}}^2-{\mathit{l}}^2$
⇒ b = $\sqrt{4{\mathit{a}}^2-{\mathit{l}}^2}$
∴ Area of rectangle , A = lb = r $\sqrt{4{\mathit{a}}^2-{\mathit{l}}^2}$
∴ $\frac{\mathit{d}\mathit{A}}{\mathit{d}\mathit{l}}$ = $\sqrt{4{\mathit{a}}^2-{\mathit{l}}^2}$ + l . $\frac{1}{2\sqrt{4{\mathit{a}}^2-{\mathit{l}}^2}}$ (- 2l) = $\sqrt{4{\mathit{a}}^2-{\mathit{l}}^2}$ - $\frac{{\mathit{l}}^2}{\sqrt{4{\mathit{a}}^2-{\mathit{l}}^2}}$
= $\frac{4{\mathit{a}}^2-2{\mathit{l}}^2}{\sqrt{4{\mathit{a}}^2-{\mathit{l}}^2}}$
$\frac{{\mathit{d}}^2\mathit{A}}{\mathit{d}{\mathit{l}}^2}$ =
= $\frac{\left(4{\mathit{a}}^2-{\mathit{l}}^2\right)\left(-4\mathit{l}\right)+\mathit{l}\left(4{\mathit{a}}^2-2{\mathit{l}}^2\right)}{{\left(4{\mathit{a}}^2-{\mathit{l}}^2\right)}^{\frac{3}{2}}}$
= $\frac{-12{\mathit{a}}^2\mathit{l}+2{\mathit{l}}^3}{{\left(4{\mathit{a}}^2-{\mathit{l}}^2\right)}^{\frac{3}{2}}}$ = $\frac{-2\mathit{l}\left(6{\mathit{a}}^2-{\mathit{l}}^2\right)}{{\left(4{\mathit{a}}^2-{\mathit{l}}^2\right)}^{\frac{3}{2}}}$
Now, $\frac{\mathit{d}\mathit{A}}{\mathit{d}\mathit{l}}$ = 0 gives $4{\mathit{a}}^2$ = $2{\mathit{l}}^2$ ⇒ l = $\sqrt{2}$ a
when l = $\sqrt{2}$ a
$\frac{{\mathit{d}}^2\mathit{A}}{\mathit{d}{\mathit{l}}^2}$ = $\frac{-2\left(\sqrt{2}\mathit{a}\right)\left(6{\mathit{a}}^2-2{\mathit{a}}^2\right)}{2\sqrt{2}{\mathit{a}}^3}$ = $\frac{-8\sqrt{2}{\mathit{a}}^3}{2\sqrt{2}{\mathit{a}}^3}$ = - 4 < 0
∴ Thus, from the second derivative test, when l = $\sqrt{2}$ a , the area of the rectangle is maximum.
Since l = b = $\sqrt{2}$ a , the rectangle is a square