garbage collection - How to pass output of one function to another in Spark -

-

i sending output of 1 function dataframe function.

val df1 = fun1 val df11 = df1.collect val df2 = df11.map(x =fun2( x,df3))

above 2 lines wriiten in main function. df1 large if collect on driver gives outof memory or gc issue. r ways send output of 1 function in spark?

spark can run data processing you. don't need intermediate collect step. should chain of transformations , add action @ end save resulting data out disk.

calling collect() useful debugging small results.

for example, this:

rdd.map(x => fun1(x))    .map(y => fun2(y))    .saveasobjectfile();

this article might helpful explain more this:

http://www.agildata.com/apache-spark-rdd-vs-dataframe-vs-dataset/


Comments

Post a Comment

Popular posts from this blog

c - How to retrieve a variable from the Apache configuration inside the module? -

-
i'm building custom apache module. how retrieve variable apache configuration inside module? see variable "some_var_config_from_apache_or_htaacess" inside apache module: /* source: http://people.apache.org/~humbedooh/mods/examples/mod_example_1.c */ /* include required headers httpd */ #include "httpd.h" #include "http_core.h" #include "http_protocol.h" #include "http_request.h" static void register_hooks(apr_pool_t *pool); static int example_handler(request_rec *r); /* define our module entity , assign function registering hooks */ module ap_module_declare_data example_module = { standard20_module_stuff, null, // per-directory configuration handler null, // merge handler per-directory configurations null, // per-server configuration handler null, // merge handler per-server configurations null, // directives may have httpd register_hoo
Read more

c# - Constructor arguments cannot be passed for interface mocks -

-
when debug code , read line mocklessonplannerafactory creation error: constructor arguments cannot passed interface mocks. var mockschoolclasscodeservice = new mock<ischoolclasscodeservice>(); var mockdateservice = new mock<idateservice>(); var mocklessonplannerafactory = new mock<ilessonplannerafactory>(mockdateservice.object); var mocklessonplannerbfactory = new mock<ilessonplannerbfactory>(mockdateservice.object); var service = new timetableservice(mockunitofwork.object, mocklessonplannerafactory.object, mocklessonplannerbfactory.object, mockschoolclasscodeservice.object); my timetableservice accepts instances of interface type only. mocklessonplannerafactory , bfactory... want in constructor idateservice passed. what wrong code? the clue in error message "constructor arguments cannot passed interface mocks." the mock created interface have default constructor because interfaces don't have constructor. remember mockin
Read more

python - malformed header from script index.py Bad header -

-
i want run python code in apache2(ubuntu 14.04) server. have followed these steps , getting error: step 1: configuration 1: have created directory , file under /var/www/cgi-bin configuration 2 : have edited /etc/apache2/sites-available/000-default.conf alias /cgi-bin /var/www/cgi-bin <directory "/var/www/cgi-bin"> options indexes followsymlinks execcgi addhandler cgi-script .cgi .py allow </directory> <directory /var/www/cgi-bin> options </directory> step 2: and python script is: index.py #!/usr/bin/env python import cgi; import cgitb;cgitb.enable() print "content-type: text/plain\n" print "<b>hello python</b>" step 3: when ran through chrome browser using: url : http://localhost/cgi-bin/index.py step 4: i getting error in error-log malformed header script 'index.py': bad header: hello python you should end header \r\n, must print out yet \r\n signal body coming. (i
Read more