pandas - Ingest data once in python -

-

i have dataframe in python contains of data binary classification. ingest data in 2 iterations - once of data of 1 class , of data of other class. run randomisation of rows. problem have every time rerun script rows data frame recreated , randomised creating unreproducible results.

should run dataframe creation , randomisation external file? there common practices data ingestion in model building?

i haven't tried attempted in regard. wondering if makes sense statistical point of view or common practice ? try such as:

import data_ingest data_ingest.function_data_call()

but again every time run script calls external script forms data , randomises it. not solution looking for.

i can't show example, loading in documents (text files) - document binary classification. structure of dataframe following:

row|           content         | class -------------------------------------- 1  | sky blue           | 0 2  | river runs deep purple| 0 3  | yellow fever              | 0 4  | red strawberries          | 1 5  | black orchids nice    | 1

ingestion code:

for f in [f f in os.listdir(path1) if not f.startswith('.')]:    io.open(path1+f, "r", encoding="utf-8") myfile:      # data1.append(myfile.read().rstrip().replace('-', '').replace('.', '').replace('\n', ''))      tmp1 = myfile.read().rstrip().replace('-', '').replace('\n', '')      data1.append(" ".join(tmp1.split()))  df1 = pd.dataframe(data1, columns=["content"]) df1["class"] = "1"  f in [f f in os.listdir(path1) if not f.startswith('.')]:    io.open(path1+f, "r", encoding="utf-8") myfile:      # data1.append(myfile.read().rstrip().replace('-', '').replace('.', '').replace('\n', ''))      tmp1 = myfile.read().rstrip().replace('-', '').replace('\n', '')      data1.append(" ".join(tmp1.split()))  df1 = pd.dataframe(data1, columns=["content"]) df1["class"] = "1"  f in [f f in os.listdir(path2) if not f.startswith('.')]:    io.open(path2+f, "r", encoding="utf-8") myfile:      # data2.append(myfile.read().rstrip().replace('-', '').replace('.', '').replace('\n', '').replace(' ', ''))      tmp2 = myfile.read().rstrip().replace('-', '').replace('\n', '')      data2.append(" ".join(tmp2.split()))  df2 = pd.dataframe(data2, columns=["content"]) df2["class"] = "0"  ### concatenate 2 dataframe 1 , re-index emails = pd.concat([df1,df2], ignore_index=true)  ## randomize rows  emails = emails.reindex(np.random.permutation(emails.index))

if want reproduce same result after (pseudo-)randomization, can set random seed. each time use same seed, same sequence of random numbers.

secondly, can save intermediate result either file, json or pickle. can check if exists, , if not, recreate it.


Comments

Post a Comment

Popular posts from this blog

c++ - llvm function pass ReplaceInstWithInst malloc -

-
#include "llvm/pass.h" #include "llvm/ir/module.h" #include "llvm/ir/function.h" #include "llvm/support/raw_ostream.h" #include "llvm/ir/legacypassmanager.h" #include "llvm/ir/instrtypes.h" #include "llvm/transforms/ipo/passmanagerbuilder.h" #include "llvm/ir/irbuilder.h" #include "llvm/transforms/utils/basicblockutils.h" using namespace llvm; namespace { struct replacepass : public functionpass { static char id; replacepass() : functionpass(id) {} virtual bool runonfunction(function &f) { allocainst* insttoreplace = ??? basicblock::iterator ii(insttoreplace); replaceinstwithinst(insttoreplace->getparent()->getinstlist(), ii, new allocainst(type::int32ty, 0, insttoreplace)); return true; } }; } char replacepass::id = 0; static void registerreplacepass(const passmanagerbuilder &, legacy::passmanagerbase &pm) { pm.add(new re...
Read more

Cross-Compiling Linux Kernel for Raspberry Pi - ${CCPREFIX}gcc -v does not work -

-
i'm trying follow this guide . i'm running both ubuntu 12.04.5 lts (gnu/linux 3.13.0-74-generic x86_64) on "real" hardware , 14.04.1 via virtualbox on mac. problem don't past step 1: hoffmann@angl99:~$ export ccprefix=/home/hoffmann/tools/arm-bcm2708/gcc-linaro-arm-linux-gnueabihf-raspbian/bin/arm-linux-gnueabihf- hoffmann@angl99:~$ ${ccprefix}gcc -v i'm getting following error: -bash: /home/hoffmann/tools/arm-bcm2708/gcc-linaro-arm-linux-gnueabihf-raspbian/bin/arm-linux-gnueabihf-gcc: no such file or directory however, file i'm told missing there: hoffmann@angl99:~$ less /home/hoffmann/tools/arm-bcm2708/gcc-linaro-arm-linux-gnueabihf-raspbian/bin/arm-linux-gnueabihf-gcc "/home/hoffmann/tools/arm-bcm2708/gcc-linaro-arm-linux-gnueabihf-raspbian/bin/arm-linux-gnueabihf-gcc" may binary file. see anyway? this result of basic error/misconception. suggest solution? thanks! sebastian ok - i've worked out (with of person po...
Read more

java.lang.NoClassDefFoundError When Creating New Android Project -

-
i've tried installing (and re-installing) android studio several times now, , each time, studio isn't able create new android project. when attempt initial android studio window (ie. basic 'create', 'open', 'import' etc options), i'm asked of questions in wizard, brief progress dialog, followed nothing @ all. if open incomplete project process creates choosing 'open' list, can try choosing create new android project menu. this has same issue, error message in output console time... java.lang.noclassdeffounderror: com/sun/xml/internal/bind/v2/model/annotation/annotationreader (wrong name: com/sun/xml/internal/bind/v0/model/qnnotation/annotationreader) note: when installed android studio, installed incompatible jdk (1.8, not 1.7). have had go default project settings change jdk uses 1.8. does know how can avoid this? there place should downloading jxb classes? missing in java paths? this appears have been caused cor...
Read more