Even if he believed Simba to be Simba, he was now in a very bad position. You see, he'd made out that Simba had died in the stampede and here Simba is showing up, to which both Scar and Sarabi acknowledge him as Simba.
That forced Scar to try and execute Simba for "causing Mufasa's death", which never really made much sense. Especially as Scar never says how Simba did it. (If you're going to accuse someone a crime that you actually committed, and they have the power to replace you on the throne, at least try and get the support of the public as to their guilt before trying to off them.)
Both Sarabi and Scar originally mistook Simba for Mufasa. However, being a Mufasa-look-alike doesn't mean that it's Simba at all. Nala might have recognized Simba since she and he were old friends and she was able to pin him (just like in the past) and it jarred her memory as to whom she had found.
By claiming that the Simba before them was an imposter and then trying to get rid of him for the act of challenging him, Scar might have reasonably expected far less resistance from the lionesses.