There seems to be one character missing. But I gather that f(x) needs to satisfy
• [tex]x^3[/tex] divides [tex]f(x)[/tex]
• [tex](x-1)^3[/tex] divides [tex]f(x)^2[/tex]
I'll also assume f(x) is monic, meaning the coefficient of the leading term is 1, or
[tex]f(x) = x^5 + \cdots[/tex]
Since [tex]x^3[/tex] divides [tex]f(x)[/tex], and
[tex]f(x) = x^3 p(x)[/tex]
where [tex]p(x)[/tex] is degree-2, and we can write it as
[tex]f(x)=x^3 (x^2+ax+b)[/tex]
Now, we have
[tex]f(x)^2 = \left(x^3p(x)\right)^2 = x^6 p(x)^2[/tex]
so if [tex](x - 1)^3[/tex] divides [tex]f(x)^2[/tex], then [tex]p(x)[/tex] is degree-2, so [tex]p(x)^2[/tex] is degree-4, and we can write
[tex]p(x)^2 = (x-1)^3 q(x)[/tex]
where [tex]q(x)[/tex] is degree-1.
Expanding the left side gives
[tex]p(x)^2 = x^4 + 2ax^3 + (a^2+2b)x^2 + 2abx + b^2[/tex]
and dividing by [tex](x-1)^3[/tex] leaves no remainder. If we actually compute the quotient, we wind up with
[tex]\dfrac{p(x)^2}{(x-1)^3} = \underbrace{x + 2a + 3}_{q(x)} + \dfrac{(a^2+6a+2b+6)x^2 + (2ab-6a-8)x +2a+b^2+3}{(x-1)^3}[/tex]
If the remainder is supposed to be zero, then
[tex]\begin{cases}a^2+6a+2b+6 = 0 \\ 2ab-6a-8 = 0 \\ 2a+b^2+3 = 0\end{cases}[/tex]
Adding these equations together and grouping terms, we get
[tex](a^2+2ab+b^2) + (2a+2b) + (6-8+3) = 0 \\\\ (a+b)^2 + 2(a+b) + 1 = 0 \\\\ (a+b+1)^2 = 0 \implies a+b = -1[/tex]
Then [tex]b=-1-a[/tex], and you can solve for a and b by substituting this into any of the three equations above. For instance,
[tex]2a+(-1-a)^2 + 3 = 0 \\\\ a^2 + 4a + 4 = 0 \\\\ (a+2)^2 = 0 \implies a=-2 \implies b=1[/tex]
So, we end up with
[tex]p(x) = x^2 - 2x + 1 \\\\ \implies f(x) = x^3 (x^2 - 2x + 1) = \boxed{x^5-2x^4+x^3}[/tex]
