I suppose * is supposed to denote a binary operation with the given definition,
a * b = (a - b)²
as opposed to multiplication, so that, for instance, 1 * 1 = (1 - 1)² = 0.
Using the definition of *, you have
2 * 5 = (2 - 5)² = (-3)² = 9
