duplicates - How to do de-duplication on records from AWS Kinesis Firehose to Redshift? -

-

i read document of official aws kinesis firehose doesn't mention how handle duplicated events. have experience on it? googled use elasticcache filtering, mean need use aws lambda encapsulate such filtering logic? there simple way firehose ingest data redshift , @ same time has "exactly once" semantics? lot!

you can have duplication on both sides of kinesis stream. might put same events twice stream, , might read event twice consumers.

the producers side can happen if try put event kinesis stream, reason not sure if written or not, , decide put again. consumer side can happen if getting batch of events , start processing them, , crash before managed checkpoint location, , next worker picking same batch of events kinesis stream, based on last checkpoint sequence-id.

before start solving problem, should evaluate how have such duplication , business impact of such duplications. not every system handling financial transactions can't tolerate duplication. nevertheless, if decide need have such de-duplication, common way solve use event-id , track if processed event-id already.

elasticcache redis place track event-id. every time pick event processing, check if have in hash table in redis, if find it, skip it, , if don't find it, add table (with ttl based on possible time window such duplication).

if choose use kinesis firehose (instead of kinesis streams), no longer have control on consumer application , can't implement process. therefore, either want run such de-duplication logic on producer side, switch use kinesis streams , run own code in lambda or kcl, or settle de-duplication functions in redshift (see below).

if not sensitive duplication, can use functions in redshift, such count distinct or last_value in window function.


Comments

Post a Comment

Popular posts from this blog

c++ - llvm function pass ReplaceInstWithInst malloc -

-
#include "llvm/pass.h" #include "llvm/ir/module.h" #include "llvm/ir/function.h" #include "llvm/support/raw_ostream.h" #include "llvm/ir/legacypassmanager.h" #include "llvm/ir/instrtypes.h" #include "llvm/transforms/ipo/passmanagerbuilder.h" #include "llvm/ir/irbuilder.h" #include "llvm/transforms/utils/basicblockutils.h" using namespace llvm; namespace { struct replacepass : public functionpass { static char id; replacepass() : functionpass(id) {} virtual bool runonfunction(function &f) { allocainst* insttoreplace = ??? basicblock::iterator ii(insttoreplace); replaceinstwithinst(insttoreplace->getparent()->getinstlist(), ii, new allocainst(type::int32ty, 0, insttoreplace)); return true; } }; } char replacepass::id = 0; static void registerreplacepass(const passmanagerbuilder &, legacy::passmanagerbase &pm) { pm.add(new re
Read more

java.lang.NoClassDefFoundError When Creating New Android Project -

-
i've tried installing (and re-installing) android studio several times now, , each time, studio isn't able create new android project. when attempt initial android studio window (ie. basic 'create', 'open', 'import' etc options), i'm asked of questions in wizard, brief progress dialog, followed nothing @ all. if open incomplete project process creates choosing 'open' list, can try choosing create new android project menu. this has same issue, error message in output console time... java.lang.noclassdeffounderror: com/sun/xml/internal/bind/v2/model/annotation/annotationreader (wrong name: com/sun/xml/internal/bind/v0/model/qnnotation/annotationreader) note: when installed android studio, installed incompatible jdk (1.8, not 1.7). have had go default project settings change jdk uses 1.8. does know how can avoid this? there place should downloading jxb classes? missing in java paths? this appears have been caused cor
Read more

Decoding a Python 2 `tempfile` with python-future -

-
i'm attempting write python 2/3 compatible routine fetch csv file, decode latin_1 unicode , feed csv.dictreader in robust, scalable manner. for python 2/3 support, i'm using python-future including imporing open builtins , , importing unicode_literals consistent behaviour i'm hoping handle exceptionally large files spilling disk, using tempfile.spooledtemporaryfile i'm using io.textiowrapper handle decoding latin_1 encoding before feeding dictreader this works fine under python 3. the problem textiowrapper expects wrap stream conforms bufferediobase . unfortunately under python 2, although have imported python 3-style open , vanilla python 2 tempfile.spooledtemporaryfile still of course returns python 2 cstringio.stringo , instead of python 3 io.bytesio required textiowrapper . i can think of these possible approaches: wrap python 2 cstringio.stringo python 3-style io.bytesio . i'm not sure how approach - need write such wrapper or 1 ex
Read more