python - Set up spark using an external virtual machine -

-

i not huge computer person many others on here, majored in math matlab main computer knowledge. have got involved apache spark through excellent edx course offered berkeley.

the method used setting spark provided in great step step guide, involved: downloading oracle vm virtual box ubuntu 32bit vm, through use of vagrant (again i'm not hugely computer-y not 100% sure how worked or is) connect ipython notebook. enabled me have access spark on internet , code in python pyspark, want do.

everything going until second lab exercise, became apparent windows laptop has insufficient free memory (just 3 gb , 4 years old) after continually froze , crashed when trying work large datasets.

it not possible have vm in vm apparently have spent of today looking alternative ways of setting spark no avail; guides aimed @ more computer knowledge have.

my (likely naive) idea rent external machine can interface through windows laptop before virtual machine operates outside of memory of laptop i.e. in cloud (using of ubuntu, windows, etc.). want move oracle vm virtual box outside source rid computer of memory burdens , use ipython notebook before.

how can set virtual machine use computational side of spark in ipython notebook?

or there alternate method simple follow?

don't run vms. instead:

  • download latest spark version. (1.4.1 @ moment.)
  • extract archive.
  • run bin/pyspark.cmd.

it's not ipython notebook, can run python code against local spark instance.

if want beefier instance, same on beefy remote machine. example ec2 m4.2xlarge $0.5 per hour 8 cores , 30 gb of ram.


Comments

Post a Comment

Popular posts from this blog

php - Zend Framework / Skeleton-Application / Composer install issue -

-
i trying create zend framework application using skeleton in netbeans. when run composer install, following error message: "c:\php\php.exe" "c:\composer\composer.phar" "--ansi" "--no-interaction" "update" "--dev" using deprecated option "dev". dev packages installed default now. loading composer repositories package information updating dependencies (including require-dev) requirements not resolved installable set of packages. problem 1 - package requires php >=5.5 php version (5.4.42) not satisfy requirement. problem 2 - installation request zendframework/zendframework 2.5.1 -> satisfiable zendframework/zendframework[2.5.1]. - zendframework/zendframework 2.5.1 requires php >=5.5 -> php version (5.4.42) not satisfy requirement. done. apparently, zend framework only needs php 5.3+ contradicts above. need use php 5.4. my composer.json following: { "name": "
Read more

c# - Better 64-bit byte array hash -

-
i need hash algorithm produces 64-bit hash code ( long ) fewer collisions string.gethashcode() , fast (no expensive calls cryptographic functions). here's implementation of fnv still shows 3% of collisions after testing 2 million random strings. need number way lower. void main() { const string chars = "abcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyz!#@$%^&*()_+}{\":?><,./;'[]0123456789\\"; const int n = 2000000; var random = new random(); var hashes = new hashset<long>(); int collisions = 0; for(int = 0; < n; i++) { var len = random.next(chars.length); var str = new char[len]; (int j = 0; j < len; j++) { str[j] = chars[random.next(chars.length)]; } var s = new string(str); if(!hashes.add(get64bithash( s ))) collisions++; } console.writeline("collision percentage after " + n + " random strings: " + ((doubl
Read more

python - PyCharm Type error Message -

-
i'm newbie trying out ml/dbn in python (3.4) under pycharm (comm.ed 4.5). following definition def make_thetas(xmin,xmax,n): xs = np.linespace(xmin,xmax,n) widths = (xs[1:] - xs[:-1])/2.0 thetas = xs[:-1] + widths return thetas throws error "class 'tuple' not define ' sub ', '-' operator cannot used on instance" on - operator on third line (widths = ....) ideas on how code running under pycharm - works alright in interactive python window. thx. to all, after g'g lot have found workaround seems work within pycharm. workaround because (even python newbie) expected python not need explicit typecasting, not when using datatypes imported lib such numpy. def make_thetas(xmin,xmax,n): xs = np.array(np.linspace(xmin,xmax,n)) widths = (xs[1:] - xs[:-1])/2.0 thetas = xs[:-1]+ widths return thetas using docstrings type-hinting such below did not work def make_thetas(xmin,xmax,n): "&q
Read more