The arctan function has the following property: if x is positive, then
arctan(1/x) + arctan(x) = π/2
This means
[tex]\displaystyle \int_{1/\sqrt3}^1 \arctan\left(\frac1x\right) \, dx = \int_{1/\sqrt3}^1 \left(\frac\pi2 - \arctan(x)\right) \, dx \\\\\\ = \frac{(3-\sqrt3)\pi}6 - \int_{1/\sqrt3}^1 \arctan(x) \, dx[/tex]
For the remaining integral, integrate by parts with
[tex]u = \arctan(x) \implies du = \dfrac{dx}{x^2+1}[/tex]
[tex]dv = dx \implies v = x[/tex]
[tex]\displaystyle \int_a^b \arctan(x) \, dx = uv\bigg|_a^b - \int_a^b v \, du[/tex]
[tex]\implies \displaystyle \int_{1/\sqrt3}^1 \arctan(x) \, dx = x\arctan(x) \bigg|_{1/\sqrt3}^1 - \int_{1/\sqrt3}^1 \frac{x}{x^2+1} \, dx[/tex]
We have arctan(1) = π/4 and arctan(1/√3) = π/6, and
[tex]\displaystyle \int \frac{x}{x^2+1} \, dx = \frac12 \int \frac{d(x^2+1)}{x^2+1} = \frac12\ln(x^2+1) + C[/tex]
which gives
[tex]\displaystyle \int_{1/\sqrt3}^1 \arctan(x) \, dx = \frac\pi4 - \frac\pi{6\sqrt3} - \frac{\ln(2) - \ln\left(\frac43\right)}2 \\\\\\ = \frac{(9-2\sqrt3)\pi}{36} + \frac12\ln\left(\frac23\right)[/tex]
so that
[tex]\displaystyle \int_{1/\sqrt3}^1 \arctan\left(\frac1x\right) \, dx = \frac{(3-\sqrt3)\pi}6 - \frac{(9-2\sqrt3)\pi}{36} - \frac12\ln\left(\frac23\right) \\\\\\ = \boxed{\frac{(9-4\sqrt3)\pi}{36} - \frac12\ln\left(\frac23\right)}[/tex]
Alternatively, you can integrate by parts immediately:
[tex]u = \arctan\left(\dfrac1x\right) \implies du = \dfrac{-\frac1{x^2}}{1+\frac1{x^2}} \, dx = -\dfrac{dx}{x^2+1}[/tex]
[tex]dv = dx \implies v = x[/tex]
so that
[tex]\displaystyle \int_{1/\sqrt3}^1 \arctan\left(\frac1x\right) \, dx = x\arctan\left(\frac1x\right) \bigg|_{1/\sqrt3}^1 + \int_{1/\sqrt3}^1 \frac{x}{x^2+1} \, dx[/tex]
You'll end up with the same result either way.
