Spark 1.4 image for Google Cloud? -

-

with bdutil, latest version of tarball can find on spark 1.3.1:

gs://spark-dist/spark-1.3.1-bin-hadoop2.6.tgz

there few new dataframe features in spark 1.4 want use. chance spark 1.4 image available bdutil, or workaround?

update:

following suggestion angus davis, downloaded , pointed spark-1.4.1-bin-hadoop2.6.tgz, deployment went well; however, run error when calling sqlcontext.parquetfile(). cannot explain why exception possible, googlehadoopfilesystem should subclass of org.apache.hadoop.fs.filesystem. continue investigate on this.

caused by: java.lang.classcastexception: com.google.cloud.hadoop.fs.gcs.googlehadoopfilesystem cannot cast org.apache.hadoop.fs.filesystem @ org.apache.hadoop.fs.filesystem.createfilesystem(filesystem.java:2595) @ org.apache.hadoop.fs.filesystem.access$200(filesystem.java:91) @ org.apache.hadoop.fs.filesystem$cache.getinternal(filesystem.java:2630) @ org.apache.hadoop.fs.filesystem$cache.get(filesystem.java:2612) @ org.apache.hadoop.fs.filesystem.get(filesystem.java:370) @ org.apache.hadoop.fs.filesystem.get(filesystem.java:169) @ org.apache.hadoop.fs.filesystem.get(filesystem.java:354) @ org.apache.hadoop.fs.path.getfilesystem(path.java:296) @ org.apache.hadoop.hive.metastore.warehouse.getfs(warehouse.java:112) @ org.apache.hadoop.hive.metastore.warehouse.getdnspath(warehouse.java:144) @ org.apache.hadoop.hive.metastore.warehouse.getwhroot(warehouse.java:159) @ org.apache.hadoop.hive.metastore.warehouse.getdefaultdatabasepath(warehouse.java:177) @ org.apache.hadoop.hive.metastore.hivemetastore$hmshandler.createdefaultdb_core(hivemetastore.java:504) @ org.apache.hadoop.hive.metastore.hivemetastore$hmshandler.createdefaultdb(hivemetastore.java:523) @ org.apache.hadoop.hive.metastore.hivemetastore$hmshandler.init(hivemetastore.java:397) @ org.apache.hadoop.hive.metastore.hivemetastore$hmshandler.<init>(hivemetastore.java:356) @ org.apache.hadoop.hive.metastore.retryinghmshandler.<init>(retryinghmshandler.java:54) @ org.apache.hadoop.hive.metastore.retryinghmshandler.getproxy(retryinghmshandler.java:59) @ org.apache.hadoop.hive.metastore.hivemetastore.newhmshandler(hivemetastore.java:4944) @ org.apache.hadoop.hive.metastore.hivemetastoreclient.<init>(hivemetastoreclient.java:171)

asked separate question exception here

update:

the error turned out spark defect; resolution/workaround provided in above question.

thanks!

haiying

if local workaround acceptable, can copy spark-1.4.1-bin-hadoop2.6.tgz apache mirror bucket control. can edit extensions/spark/spark-env.sh , change spark_hadoop2_tarball_uri='<your copy of spark 1.4.1>' (make service account running vms has permission read tarball).

note haven't done any testing see if spark 1.4.1 works out of box right now, i'd interested in hearing experience if decide give go.


Comments

Post a Comment

Popular posts from this blog

c# - Better 64-bit byte array hash -

-
i need hash algorithm produces 64-bit hash code ( long ) fewer collisions string.gethashcode() , fast (no expensive calls cryptographic functions). here's implementation of fnv still shows 3% of collisions after testing 2 million random strings. need number way lower. void main() { const string chars = "abcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyz!#@$%^&*()_+}{\":?><,./;'[]0123456789\\"; const int n = 2000000; var random = new random(); var hashes = new hashset<long>(); int collisions = 0; for(int = 0; < n; i++) { var len = random.next(chars.length); var str = new char[len]; (int j = 0; j < len; j++) { str[j] = chars[random.next(chars.length)]; } var s = new string(str); if(!hashes.add(get64bithash( s ))) collisions++; } console.writeline("collision percentage after " + n + " random strings: " + ((doubl
Read more

webrtc - Which ICE candidate am I using and why? -

-
[questions in bold below] i have setup kurento media server 5.1.3 in datacenter behind firewall running os ubuntu 14.04. has 2 network cards: 222.222.222.222 (eth0 - private ip) 111.111.111.111 (eth1 - public ip) attached below sdp (setremotedescription) when browser connected kurento media server type: answer, sdp: v=0 o=- 5487318114793304426 0 in ip4 0.0.0.0 s=kurento media server c=in ip4 0.0.0.0 t=0 0 a=group:bundle audio video m=audio 59068 rtp/savpf 111 0 c=in ip4 111.111.111.111 a=rtpmap:111 opus/48000/2 a=rtpmap:0 pcmu/8000 a=sendrecv a=rtcp:59068 in ip4 111.111.111.111 a=rtcp-mux a=ssrc:669011897 cname:user39019747@host-6e83e4c2 a=extmap:3 http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time a=mid:audio b=as:20 a=ice-ufrag:ymdk a=ice-pwd:lylifk5ueqzpwm91ddj37e a=fingerprint:sha-256 ff:0f:81:8c:41:4e:b4:b6:c6:d8:36:f3:d6:5f:09:fd:5f:af:13:b3:9d:fc:12:66:ac:f3:56:d6:5b:0a:73:5d a=candidate:1 1 udp 2013266431 111.111.111.111 55239 typ host a=candidate:2 1 udp
Read more

php - Zend Framework / Skeleton-Application / Composer install issue -

-
i trying create zend framework application using skeleton in netbeans. when run composer install, following error message: "c:\php\php.exe" "c:\composer\composer.phar" "--ansi" "--no-interaction" "update" "--dev" using deprecated option "dev". dev packages installed default now. loading composer repositories package information updating dependencies (including require-dev) requirements not resolved installable set of packages. problem 1 - package requires php >=5.5 php version (5.4.42) not satisfy requirement. problem 2 - installation request zendframework/zendframework 2.5.1 -> satisfiable zendframework/zendframework[2.5.1]. - zendframework/zendframework 2.5.1 requires php >=5.5 -> php version (5.4.42) not satisfy requirement. done. apparently, zend framework only needs php 5.3+ contradicts above. need use php 5.4. my composer.json following: { "name": "
Read more