Spark leading to ClassNotFoundException for corrupt jar files instead of ZipException

 

When Jar passed via --jars that leads to following error -

• zip END header not found

Same Jar passed as application jar file leads to following error -

• ClassNotFoundException

These two errors are not contradictory. They mean Spark is touching the Jar in 2 completely different code paths.

--jars Jars are:

• Downloaded
• Immediately unpacked/ inspected
• Added to executor classpath eagerly.

Thus if Jar is bad. Spark tries to open it as ZIP, which leads to ZipException.

When used as application Jar:

• Spark does minimal Zip validation.
• Only checks enough to start
• Loads manifest + class index
• Attempts to resolve --class

if:

• The Jar opens.
• But the expected class isn't there, which leads to ClassNotFoundException

So:

• Spark never deeply unzips it
• Corruption in unused entries may go unnoticed

Outcomes:

• As app JAR -> Spark reads just enough -> ClassNotFoundException
• As --jars -> Spark fully opens ZIP -> zip END header not found