Skip to content
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.

Commit f8b7cf8

Browse files
authoredOct 31, 2018
Create solution-sample.md
1 parent 86b6c47 commit f8b7cf8

File tree

1 file changed

+35
-0
lines changed

1 file changed

+35
-0
lines changed
 

‎solution-sample.md

+35
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,35 @@
1+
---
2+
title: Azure HDInsight Solutions | Apache Spark | IllegalArgumentException
3+
description: Learn how to resolve an IllegalArgumentException when running Apache Spark jobs using Azure Data Factory.
4+
services: hdinsight
5+
author: confusionblinds
6+
ms.author: sunilkc
7+
ms.service: hdinsight
8+
ms.custom: troubleshooting
9+
ms.topic: conceptual
10+
ms.date: 10/30/2018
11+
---
12+
# Azure HDInsight Solutions | Spark | IllegalArgumentException
13+
14+
## Scenario: Spark activity running in Azure Data Factory fails with IllegalArgumentException
15+
16+
## Issue
17+
18+
You receive the following exception when trying to execute a Spark activity in an Azure Data Factory pipeline:
19+
20+
```java
21+
Exception in thread "main" java.lang.IllegalArgumentException:
22+
Wrong FS: wasbs://additional@xxx.blob.core.windows.net/spark-examples_2.11-2.1.0.jar, expected: wasbs://wasbsrepro-2017-11-07t00-59-42-722z@xxx.blob.core.windows.net
23+
```
24+
25+
## Cause
26+
27+
A Spark job will fail if the application jar file is not located in the Spark cluster’s default/primary storage.
28+
29+
This is a known issue with the Spark open source framework tracked in this bug: [Spark job fails if fs.defaultFS and application jar are different url](https://issues.apache.org/jira/browse/SPARK-22587)
30+
31+
This issue has been resolved in Spark 2.3.0
32+
33+
## Solution
34+
35+
Make sure the application jar is stored on the default/primary storage for the HDInsight cluster. In case of Azure Data Factory make sure the ADF linked service is pointed to the HDInsight default container rather than a secondary container.

0 commit comments

Comments
 (0)
Please sign in to comment.