Let a rectangle ABCD be inscribed in a circle with radius r.

In ∠DBC = θ
In right ΔBCD ;
$\frac{\mathit{B}\mathit{C}}{\mathit{B}\mathit{D}}$ = cos θ
⇒ BC = BD cos θ = 2r cos θ
$\frac{\mathit{C}\mathit{D}}{\mathit{B}\mathit{D}}$ = sin θ
⇒ CD = BF sin θ = 2r sin θ
Let A be the area of rectangle ABCD.
∴ A = BC × CD
⇒ A = 2r cos θ 2r sin θ = $4{\mathit{r}}^2$ sin θ cos θ
⇒ A = $2{\mathit{r}}^2$ sin 2θ , sin 2θ = 2 sin θ cos θ
∴ $\frac{\mathit{d}\mathit{A}}{\mathit{d}\mathit{\theta }}$ = 2 . $2{\mathit{r}}^2$ cos 2θ = $4{\mathit{r}}^2$ cos 2θ
Now , $\frac{\mathit{d}\mathit{A}}{\mathit{d}\mathit{\theta }}$ = 0
⇒ $4{\mathit{r}}^2$ cos 2θ = 0 ⇒ cos 2θ = 0
⇒ cos 2θ = cos $\frac{\mathit{\pi }}{2}$ ⇒ θ = $\frac{\mathit{\pi }}{4}$
$\frac{{\mathit{d}}^2\mathit{A}}{\mathit{d}{\mathit{\theta }}^2}$ = - 2 . $4{\mathit{r}}^2$ sin 2θ = - $8{\mathit{r}}^2$ sin 2θ
∴ ${\left(\frac{{\mathit{d}}^2\mathit{A}}{\mathit{d}{\mathit{\theta }}^2}\right)}_{\left(\mathit{\theta }=\frac{\mathit{\pi }}{4}\right)}$ = $-8{\mathit{r}}^2\mathit{s}\mathit{i}\mathit{n}\left(2,\frac{\mathit{\pi }}{4}\right)$ = - $8{\mathit{r}}^2$ . 1 = $-8{\mathit{r}}^2$ < 0
Therefore, by the second derivative test, θ = $\frac{\mathit{\pi }}{4}$ is the point of local maxima of A.
So, the area of rectangle ABCD is the maximum at θ = $\frac{\mathit{\pi }}{4}$
Now, θ = $\frac{\mathit{\pi }}{4}$
⇒ $\frac{\mathit{C}\mathit{D}}{\mathit{B}\mathit{C}}$ = tan $\frac{\mathit{\pi }}{4}$
⇒ $\frac{\mathit{C}\mathit{D}}{\mathit{B}\mathit{C}}$ = 1 ⇒ CD = BC
⇒ Rec tangle ABCD is a square
Hence, the rectangle of the maximum area that can be inscribed in a circle is a square.
OR
Let a cylinder be inscribed in a cone of radius R and height h.
Let the radius of the cylinder be r and its height be ${\mathit{h}}_1$ .

It can be easily seen that Δ AGI and Δ ABD are similar.
∴ $\frac{\mathit{A}\mathit{I}}{\mathit{A}\mathit{D}}$ = $\frac{\mathit{G}\mathit{I}}{\mathit{B}\mathit{D}}$
⇒ $\frac{\mathit{h}-{\mathit{h}}_1}{\mathit{h}}$ = $\frac{\mathit{r}}{\mathit{R}}$
⇒ r = $\frac{\mathit{R}}{\mathit{h}}$ (h - ${\mathit{h}}_1$)
Volume (V) of the cylinder = $\mathit{\pi }{\mathit{r}}^2{\mathit{h}}_1$
⇒ V = $\mathit{\pi }\frac{{\mathit{R}}^2}{{\mathit{h}}^2}\mathit{h}-{\mathit{h}}_1^2{\mathit{h}}_1$
⇒ V = $\mathit{\pi }\frac{{\mathit{R}}^2}{{\mathit{h}}^2}{\mathit{h}}^2+{\mathit{h}}_1^2-2\mathit{h}{\mathit{h}}_1{\mathit{h}}_1$
⇒ $\frac{\mathit{d}\mathit{V}}{\mathit{d}{\mathit{h}}_1}$ =
⇒ $\frac{\mathit{d}\mathit{V}}{\mathit{d}{\mathit{h}}_1}$ = $\mathit{\pi }\frac{{\mathit{R}}^2}{{\mathit{h}}^2}{\mathit{h}}^2+3{\mathit{h}}_1^2-4\mathit{h}{\mathit{h}}_1$
Now, $\frac{\mathit{d}\mathit{V}}{\mathit{d}{\mathit{h}}_1}$ = 0
⇒ $\frac{\mathit{\pi }{\mathit{R}}^2}{{\mathit{h}}^2}{\mathit{h}}^2+3{\mathit{h}}_1^2-4\mathit{h}{\mathit{h}}_1$ = 0
⇒ $3{\mathit{h}}_1^2-4\mathit{h}{\mathit{h}}_1+{\mathit{h}}^2$ = 0
⇒ $3{\mathit{h}}_1^2-3\mathit{h}{\mathit{h}}_1-\mathit{h}{\mathit{h}}_1+{\mathit{h}}^2$ = 0
⇒ $3{\mathit{h}}_1{\mathit{h}}_1-\mathit{h}-\mathit{h}{\mathit{h}}_1-\mathit{h}$ = 0
⇒ $\left({\mathit{h}}_1-\mathit{h}\right)$(3h_1-h) = 0
⇒ ${\mathit{h}}_1$ = h , ${\mathit{h}}_1$ = $\frac{\mathit{h}}{3}$
It can be noted that if ${\mathit{h}}_1$ = h, then the cylinder cannot be inscribed in the cone.
∴ ${\mathit{h}}_1$ = $\frac{\mathit{h}}{3}$
Now, $\frac{{\mathit{d}}^2\mathit{V}}{\mathit{d}{\mathit{h}}_1^2}$ = $\frac{\mathit{\pi }{\mathit{R}}^2}{{\mathit{h}}^2}$ (0 + 6${\mathit{h}}_1$ - 4h) = $\frac{\mathit{\pi }{\mathit{R}}^2}{{\mathit{h}}^2}\left(6{\mathit{h}}_1-4\mathit{h}\right)$
∴ $\frac{{\mathit{d}}^2\mathit{V}}{{\mathit{d}{\mathit{h}}_1^2}_{\left({\mathit{h}}_1=\frac{\mathit{h}}{3}\right)}}$ = $\frac{\mathit{\pi }{\mathit{R}}^2}{{\mathit{h}}^2}\left[\frac{6\mathit{h}}{3}-4\mathit{h}\right]$ = $\frac{-2\mathit{\pi }{\mathit{R}}^2}{\mathit{h}}$ < 0
Therefore, by the second derivative test, ${\mathit{h}}_1$ = $\frac{\mathit{h}}{3}$ is the point of local maxima of V.
So, the volume of the cylinder is the maximum when ${\mathit{h}}_1$ = $\frac{\mathit{h}}{3}$
Hence, the height of the cylinder of the maximum volume that can be inscribed in a cone of height h is $\frac{1}{3}$ h