Quantcast
Channel: Data Preparation & Blending discussions
Viewing all articles
Browse latest Browse all 4999

Using Advanced Join Macro for Large Data Set

$
0
0

Hi,

 

I am trying to join two tables say, T1 and T2. Table T1 has around 8 million records and T2 has 20k records. Table T1 has two fields enterance_date and exit_date and T2 has field movement_time. All the fields are in DateTime format.

 

I need to perform join on the two tables for the below condition,

 

[entrance_date] <= [movement time] && [exit_date] >= [movement time] 

 

I tried using the Advance Join macro from analytics gallery to join for above condition and it looks like below:

 

Advanced Join.PNG

 

 

The condition has been put as shown below:

 

Join Condition.PNG

 

When i ran the flow, it turned out that the join was behaving extremely slow(possibly because of cross join of huge no of records) and was taking hours to get completed. 

 

Is there a way to improve the performance, for the above problem?

 

Thanks


Viewing all articles
Browse latest Browse all 4999

Trending Articles