aspose file tools*
The moose likes Hadoop and the fly likes hdfsExists fails to validate the existence of file Big Moose Saloon
  Search | Java FAQ | Recent Topics | Flagged Topics | Hot Topics | Zero Replies
Register / Login


Win a copy of JavaScript Promises Essentials this week in the JavaScript forum!
JavaRanch » Java Forums » Databases » Hadoop
Bookmark "hdfsExists fails to validate the existence of file " Watch "hdfsExists fails to validate the existence of file " New topic
Author

hdfsExists fails to validate the existence of file

Dayakar Reddy V
Greenhorn

Joined: Jun 06, 2013
Posts: 2
hdfsExists API fail to validate the existence of file. The user(root) has previlages to access the files.

Can you please let me know what could be the reason for hdfsExists API to fail?

Here are the hadoop environment details:

1. Source code

const char* readPath = "/user/testuser/test1.txt";
const char *tuser = "root";

hdfsFS fs = hdfsConnectAsUser("default", 0, tuser);
if(!fs) {
fprintf(stderr, "Oops! Failed to connect to hdfs!\n");
exit(-1);
}

int exists = hdfsExists(fs, readPath);
if (exists) {
fprintf(stderr, "Failed to validate existence of %s\n", readPath);
exit(-1);
}

hdfsFile readFile = hdfsOpenFile(fs, readPath, O_RDONLY, 0, 0, 0);
if (!readFile) {
fprintf(stderr, "Failed to open %s for reading!\n", readPath);
exit(-1);
}

static char buffer[32];

tSize num_read_bytes = hdfsRead(fs, readFile, (void*)buffer, sizeof(buffer));
fprintf(stderr, "Read following %d bytes:\n%s\n",num_read_bytes, buffer);

hdfsCloseFile(fs, readFile);




2. HDFS Directory Listing

[root@localhost /tmp]# hadoop fs -ls /user/testuser/test1.txt
Found 1 items
-rw-r--r-- 1 root supergroup 115 2013-06-02 10:39 /user/testuser/test1.txt

3. CLASSPATH

/usr/lib/hadoop/lib/activation-1.1.jar:/usr/lib/hadoop/lib/asm-3.2.jar:/usr/lib/hadoop/lib/avro-1.7.3.jar:/usr/lib/hadoop/lib/commons-beanutils-1.7.0.jar:
/usr/lib/hadoop/lib/commons-beanutils-core-1.8.0.jar:/usr/lib/hadoop/lib/commons-cli-1.2.jar:/usr/lib/hadoop/lib/commons-codec-1.4.jar:
/usr/lib/hadoop/lib/commons-collections-3.2.1.jar:/usr/lib/hadoop/lib/commons-configuration-1.6.jar:/usr/lib/hadoop/lib/commons-digester-1.8.jar:
/usr/lib/hadoop/lib/commons-el-1.0.jar:/usr/lib/hadoop/lib/commons-httpclient-3.1.jar:/usr/lib/hadoop/lib/commons-io-2.1.jar:/usr/lib/hadoop/lib/commons-lang-2.5.jar:
/usr/lib/hadoop/lib/commons-logging-1.1.1.jar:/usr/lib/hadoop/lib/commons-math-2.1.jar:/usr/lib/hadoop/lib/commons-net-3.1.jar:/usr/lib/hadoop/lib/guava-11.0.2.jar:
/usr/lib/hadoop/lib/hue-plugins-2.2.0-cdh4.2.0.jar:/usr/lib/hadoop/lib/jackson-core-asl-1.8.8.jar:/usr/lib/hadoop/lib/jackson-jaxrs-1.8.8.jar:
/usr/lib/hadoop/lib/jackson-mapper-asl-1.8.8.jar:/usr/lib/hadoop/lib/jackson-xc-1.8.8.jar:/usr/lib/hadoop/lib/jackson-xc-1.8.8.jar:
/usr/lib/hadoop/lib/jasper-compiler-5.5.23.jar:/usr/lib/hadoop/lib/jasper-runtime-5.5.23.jar:/usr/lib/hadoop/lib/jaxb-api-2.2.2.jar:/usr/lib/hadoop/lib/jaxb-impl-2.2.3-1.jar:
/usr/lib/hadoop/lib/jersey-core-1.8.jar:/usr/lib/hadoop/lib/jersey-json-1.8.jar:/usr/lib/hadoop/lib/jersey-server-1.8.jar:/usr/lib/hadoop/lib/jets3t-0.6.1.jar:/usr/lib/hadoop/lib/jettison-1.1.jar:
/usr/lib/hadoop/lib/jetty-6.1.26.cloudera.2.jar: /usr/lib/hadoop/lib/jetty-util-6.1.26.cloudera.2.jar:/usr/lib/hadoop/lib/jline-0.9.94.jar:/usr/lib/hadoop/lib/jsch-0.1.42.jar:
/usr/lib/hadoop/lib/jsp-api-2.1.jar:/usr/lib/hadoop/lib/jsr305-1.3.9.jar:/usr/lib/hadoop/lib/junit-4.8.2.jar:/usr/lib/hadoop/lib/kfs-0.3.jar:/usr/lib/hadoop/lib/log4j-1.2.17.jar:
/usr/lib/hadoop/lib/mockito-all-1.8.5.jar:/usr/lib/hadoop/lib/paranamer-2.3.jar:/usr/lib/hadoop/lib/protobuf-java-2.4.0a.jar:/usr/lib/hadoop/lib/servlet-api-2.5.jar:
/usr/lib/hadoop/lib/slf4j-api-1.6.1.jar:/usr/lib/hadoop/lib/slf4j-log4j12-1.6.1.jar:/usr/lib/hadoop/lib/snappy-java-1.0.4.1.jar:/usr/lib/hadoop/lib/stax-api-1.0.1.jar:
/usr/lib/hadoop/lib/xmlenc-0.52.jar:/usr/lib/hadoop/lib/zookeeper-3.4.5-cdh4.2.0.jar:/usr/lib/hadoop/hadoop-annotations.jar:/usr/lib/hadoop/hadoop-auth-2.0.0-cdh4.2.0.jar:
/usr/lib/hadoop/hadoop-auth.jar:/usr/lib/hadoop/hadoop-common-2.0.0-hdfs_site.xmlcore-site.xmlcdh4.2.0.jar:/usr/lib/hadoop/hadoop-common-2.0.0-cdh4.2.0-tests.jar:/usr/lib/hadoop/hadoop-common.jar:
/usr/lib/hadoop/etc/hadoop/yarn-site.xml:/usr/lib/hadoop/etc/hadoop/core-site.xml:/usr/lib/hadoop/etc/hadoop/hadoop-metrics.properties:/usr/lib/hadoop/etc/hadoop/hdfs-site.xml:
/usr/lib/hadoop/etc/hadoop/mapred-site.xml

4. Please find attached hdfs_site.xml and cross_site.xml

Can you please let me know what is cause for the API failure?
Hussein Baghdadi
clojure forum advocate
Bartender

Joined: Nov 08, 2003
Posts: 3479

What is the error/exception you get?
Dayakar Reddy V
Greenhorn

Joined: Jun 06, 2013
Posts: 2
Eventhough the file(test1.txt) exists in /user/testuser directory, hdfsExists returns me -1 which means that we are not connecting to right server.

Can I know what server and port the HDFS shell connects to retreive the information from HDFS?

 
I agree. Here's the link: http://aspose.com/file-tools
 
subject: hdfsExists fails to validate the existence of file