Where do you measure clutter? What waveform is it? The motor is not energized. Did you turn the encoder by hand? If not, it will not happen until the motor is electrified, which means that it is indeed motor interference. If you rotate directly by hand, it means that the encoder quality and your interface part are not handled well. Another point is whether your encoder and motor are connected together? Theoretically, the ground of the encoder should be the same as your detection circuit, but after detection, the PWM signals output to the motor should be isolated, and their ground should not be directly connected. One more thing, I don't know what your circuit is. The encoder is isolated by optical coupler before entering the single chip microcomputer. I don't know if you are measuring the signal before or after entering the optocoupler. If the signal after optical coupling is not your optical coupler, the parameters such as resistance and capacitance are not well selected, resulting in waveform distortion. You can use Schmidt to add a gate for shaping.

In addition, does your encoder have complementary output or open collector output? Complementary differential output, strong anti-interference ability.