test: fix flaky -md and -iter mismatch checks in 20-test_enc.t

The subtests added in f26d632a0e assert that "openssl enc -d" exits
non-zero when the key derivation parameters do not match.  enc has no
MAC, so a wrong key is not detectable as such; the only signal is
whether the garbage happens to form valid PKCS#7 padding, which
openssl-enc(1) documents as passing better than one time in 256.  With
a random salt the tests thus failed about 0.4% of the time.

Assert instead that mismatching parameters do not recover the
plaintext: the decryption must either fail or produce differing output.

Assisted-by: Claude:claude-opus-5[1m]

Signed-off-by: Norbert Pocs <[email protected]>

Reviewed-by: Eugene Syromiatnikov <[email protected]>
Reviewed-by: Saša Nedvědický <[email protected]>
Reviewed-by: Neil Horman <[email protected]>
MergeDate: Mon Jul 27 12:17:21 2026
(Merged from https://github.com/openssl/openssl/pull/32083)

1 files changed, 14 insertions, 4 deletions

diff --git a/test/recipes/20-test_enc.t b/test/recipes/20-test_enc.t
index 2ad9d1f7877..d8173f97610 100644
--- a/test/recipes/20-test_enc.t
+++ b/test/recipes/20-test_enc.t
@@ -154,9 +154,14 @@ plan tests => 10 + (scalar @ciphers)*2;
             "decrypt with -md sha1");
          ok(compare_text($test, "md.clear") == 0,
             "decrypted output matches the original");
+         # A mismatching digest derives a different key.  The decryption
+         # usually fails on the padding check, but with a random salt the
+         # garbage last block occasionally happens to look like valid
+         # padding, in which case it succeeds and produces garbage instead.
          ok(!run(app([$cmd, "enc", "-aes-128-cbc", "-md", "sha256", "-d", "-k", "test",
-                      "-in", "md.cipher", "-out", "md_mismatch.clear"])),
-            "decrypt fails when -md does not match the one used to encrypt");
+                      "-in", "md.cipher", "-out", "md_mismatch.clear"]))
+            || compare_text($test, "md_mismatch.clear") != 0,
+            "decrypt does not recover the plaintext when -md does not match");
      };
 
      subtest "-iter sets the PBKDF2 iteration count" => sub {
@@ -170,8 +175,13 @@ plan tests => 10 + (scalar @ciphers)*2;
             "decrypt with -iter 5");
          ok(compare_text($test, "iter.clear") == 0,
             "decrypted output matches the original");
+         # As with -md above, a mismatching iteration count derives a
+         # different key, so the decryption either fails on the padding check
+         # or, when the garbage happens to look like valid padding, succeeds
+         # and yields garbage.
          ok(!run(app([$cmd, "enc", "-aes-128-cbc", "-iter", "6", "-d", "-k", "test",
-                      "-in", "iter.cipher", "-out", "iter_mismatch.clear"])),
-            "decrypt fails when -iter does not match the one used to encrypt");
+                      "-in", "iter.cipher", "-out", "iter_mismatch.clear"]))
+            || compare_text($test, "iter_mismatch.clear") != 0,
+            "decrypt does not recover the plaintext when -iter does not match");
      };
 }