The Soviet Union did not agree to invade Poland with the Nazis, it was about spheres of influence and red lines the Nazis should not cross in Poland. Spheres of influence were not agreements to jointly work together. When the USSR went into Poland, it stayed mostly to areas Poland had invaded and annexed a few decades prior. Should the Soviets have let Poland get entirely taken over by the Nazis, standing idle? The West made it clear that they were never going to help anyone against the Nazis until it was their turn to be targeted.

Further, you're ignoring the Munich Agreement, where Czechoslovakia was given to the Nazis by the west. Not only was the west on good terms with the Nazis, signing many non-aggression pacts, they fostered healthy relationships. Finally, the Soviets were reasonably highly suspicious that Britain and Germany would ally. This was not unfounded, both were friendly enough until the outbreak of war. The Soviets desperately needed to make sure that the British and Germans would not ally, which reasonably would have happened had the Soviet Union taken unilateral action against the Nazis, rather than waiting to be attacked first.

Stalin was said to have "obsessed over quantitative and qualitative" improvements in arms in the years leading up to World War II, and had soldiers stationed on high alert, preparing for a German attack. Despite the insistence of many Generals to muster forces on the German border, it was Stalin's insistence that forces not be so concentrated that prevented the Nazis from totally routing the Red Army, enabling the greatest counteroffensive in history.

Hitler himself was quite honest at times, to himself at least:

How can such a primitive people manage such technical achievements in such a short time!

...

The fact that Stalin has raised the Russian standard of living is unquestionable. People don’t go hungry [at the moment when Operation Barbarossa was launched]. In general, it’s necessary to recognize that they have built factories of similar importance to Hermann Goering Reichswerke where two years ago nothing but unknown villages existed. We come across railway lines that aren’t even marked on our maps.

The Soviets expected and planned for it, they were never allies with the Nazis.