A differentiable function [tex]f(x)[/tex] is increasing on an open interval [tex](a,b)[/tex] if [tex]f'(x)>0[/tex] for all [tex]a<x<b[/tex], and decreasing if [tex]f'(x)<0[/tex]. For this problem, you then need to compute the derivative:
[tex]f(x)=x^2\ln x\implies f'(x)=2x\ln x+x=(2\ln x+1)x[/tex]
then solve for [tex]f'(x)=0[/tex]:
[tex](2\ln x+1)x=0\implies x=0\text{ or }x=e^{-1/2}[/tex]
We can ignore [tex]x=0[/tex] because [tex]x^2\ln x[/tex] is defined only for [tex]x>0[/tex]. So we have two intervals to consider, [tex](0,e^{-1/2})[/tex] and [tex](e^{-1/2},\infty)[/tex]. All we need to do is pick any value from either interval and check the sign of the derivative [tex]f'(x)[/tex]. Since [tex]e^{-1/2}\approx0.606[/tex], from the first interval we can take [tex]x=\dfrac12[/tex], and from the second we can pick [tex]x=1[/tex].
[tex]f'\left(\dfrac12\right)\approx-0.193<0[/tex]
[tex]f'(1)=1>0[/tex]
The above indicates that [tex]f(x)[/tex] is decreasing on the first interval [tex](0,e^{-1/2})[/tex], and increasing on the second interval [tex](e^{-1/2},\infty)[/tex].
For part (b), we use the info from above as part of the first derivative test for extrema. We have one critical point at [tex]x=e^{-1/2}[/tex], and we know how [tex]f(x)[/tex] behaves to either side of this point; [tex]f(x)[/tex] decreases to left of it, and increases to the right. This pattern is indicative of a minimum occurring at [tex]x=e^{-1/2}[/tex], and we find that [tex]f(x)[/tex] has the (local) minimum value of [tex]f(e^{-1/2})=-\dfrac1{2e}\approx-0.184[/tex].
