== Physical Plan ==
AdaptiveSparkPlan (10)
+- == Final Plan ==
ObjectHashAggregate (6)
+- ShuffleQueryStage (5), Statistics(sizeInBytes=8.4 KiB, rowCount=50)
+- Exchange (4)
+- ObjectHashAggregate (3)
+- * Project (2)
+- * Scan ExistingRDD Delta Table State #0 - hdlfs://7646b954-15f6-4bdc-91a5-2644c1a43a19.files.hdl.prod-eu20.hanacloud.ondemand.com:443/crp-demand-service/internal/insights/10000003977/_delta_log (1)
+- == Initial Plan ==
ObjectHashAggregate (9)
+- Exchange (8)
+- ObjectHashAggregate (7)
+- Project (2)
+- Scan ExistingRDD Delta Table State #0 - hdlfs://7646b954-15f6-4bdc-91a5-2644c1a43a19.files.hdl.prod-eu20.hanacloud.ondemand.com:443/crp-demand-service/internal/insights/10000003977/_delta_log (1)
(1) Scan ExistingRDD Delta Table State #0 - hdlfs://7646b954-15f6-4bdc-91a5-2644c1a43a19.files.hdl.prod-eu20.hanacloud.ondemand.com:443/crp-demand-service/internal/insights/10000003977/_delta_log [codegen id : 1]
Output [10]: [txn#3766, add#3767, remove#3768, metaData#3769, protocol#3770, cdc#3771, checkpointMetadata#3772, sidecar#3773, domainMetadata#3774, commitInfo#3775]
Arguments: [txn#3766, add#3767, remove#3768, metaData#3769, protocol#3770, cdc#3771, checkpointMetadata#3772, sidecar#3773, domainMetadata#3774, commitInfo#3775], Delta Table State #0 - hdlfs://7646b954-15f6-4bdc-91a5-2644c1a43a19.files.hdl.prod-eu20.hanacloud.ondemand.com:443/crp-demand-service/internal/insights/10000003977/_delta_log MapPartitionsRDD[166] at $anonfun$recordDeltaOperationInternal$1 at DatabricksLogging.scala:128, ExistingRDD, UnknownPartitioning(0)
(2) Project [codegen id : 1]
Output [6]: [txn#3766, add#3767, remove#3768, metaData#3769, protocol#3770, domainMetadata#3774]
Input [10]: [txn#3766, add#3767, remove#3768, metaData#3769, protocol#3770, cdc#3771, checkpointMetadata#3772, sidecar#3773, domainMetadata#3774, commitInfo#3775]
(3) ObjectHashAggregate
Input [6]: [txn#3766, add#3767, remove#3768, metaData#3769, protocol#3770, domainMetadata#3774]
Keys: []
Functions [10]: [partial_collect_set(domainMetadata#3774, 0, 0), partial_collect_set(txn#3766, 0, 0), partial_count(protocol#3770), partial_sum(add#3767.size), partial_last(metaData#3769, true), partial_count(metaData#3769), partial_last(protocol#3770, true), partial_count(remove#3768), partial_count(add#3767), partial_count(txn#3766)]
Aggregate Attributes [12]: [buf#3900, buf#3901, count#3902L, sum#3903L, last#3904, valueSet#3905, count#3906L, last#3907, valueSet#3908, count#3909L, count#3910L, count#3911L]
Results [12]: [buf#3912, buf#3913, count#3914L, sum#3915L, last#3916, valueSet#3917, count#3918L, last#3919, valueSet#3920, count#3921L, count#3922L, count#3923L]
(4) Exchange
Input [12]: [buf#3912, buf#3913, count#3914L, sum#3915L, last#3916, valueSet#3917, count#3918L, last#3919, valueSet#3920, count#3921L, count#3922L, count#3923L]
Arguments: SinglePartition, ENSURE_REQUIREMENTS, [plan_id=1661]
(5) ShuffleQueryStage
Output [12]: [buf#3912, buf#3913, count#3914L, sum#3915L, last#3916, valueSet#3917, count#3918L, last#3919, valueSet#3920, count#3921L, count#3922L, count#3923L]
Arguments: 0
(6) ObjectHashAggregate
Input [12]: [buf#3912, buf#3913, count#3914L, sum#3915L, last#3916, valueSet#3917, count#3918L, last#3919, valueSet#3920, count#3921L, count#3922L, count#3923L]
Keys: []
Functions [10]: [collect_set(domainMetadata#3774, 0, 0), collect_set(txn#3766, 0, 0), count(protocol#3770), sum(add#3767.size), last(metaData#3769, true), count(metaData#3769), last(protocol#3770, true), count(remove#3768), count(add#3767), count(txn#3766)]
Aggregate Attributes [10]: [collect_set(domainMetadata#3774, 0, 0)#3824, collect_set(txn#3766, 0, 0)#3823, count(protocol#3770)#3822L, sum(add#3767.size)#3817L, last(metaData#3769) ignore nulls#3825, count(metaData#3769)#3821L, last(protocol#3770) ignore nulls#3826, count(remove#3768)#3820L, count(add#3767)#3819L, count(txn#3766)#3818L]
Results [14]: [collect_set(domainMetadata#3774, 0, 0)#3824 AS domainMetadata#3827, null AS fileSizeHistogram#3828, collect_set(txn#3766, 0, 0)#3823 AS setTransactions#3829, null AS numDeletedRecordsOpt#3830, count(protocol#3770)#3822L AS numOfProtocol#3831L, coalesce(sum(add#3767.size)#3817L, 0) AS sizeInBytes#3832L, last(metaData#3769) ignore nulls#3825 AS metadata#3833, count(metaData#3769)#3821L AS numOfMetadata#3834L, null AS deletedRecordCountsHistogramOpt#3835, null AS numDeletionVectorsOpt#3836, last(protocol#3770) ignore nulls#3826 AS protocol#3837, count(remove#3768)#3820L AS numOfRemoves#3838L, count(add#3767)#3819L AS numOfFiles#3839L, count(txn#3766)#3818L AS numOfSetTransactions#3840L]
(7) ObjectHashAggregate
Input [6]: [txn#3766, add#3767, remove#3768, metaData#3769, protocol#3770, domainMetadata#3774]
Keys: []
Functions [10]: [partial_collect_set(domainMetadata#3774, 0, 0), partial_collect_set(txn#3766, 0, 0), partial_count(protocol#3770), partial_sum(add#3767.size), partial_last(metaData#3769, true), partial_count(metaData#3769), partial_last(protocol#3770, true), partial_count(remove#3768), partial_count(add#3767), partial_count(txn#3766)]
Aggregate Attributes [12]: [buf#3900, buf#3901, count#3902L, sum#3903L, last#3904, valueSet#3905, count#3906L, last#3907, valueSet#3908, count#3909L, count#3910L, count#3911L]
Results [12]: [buf#3912, buf#3913, count#3914L, sum#3915L, last#3916, valueSet#3917, count#3918L, last#3919, valueSet#3920, count#3921L, count#3922L, count#3923L]
(8) Exchange
Input [12]: [buf#3912, buf#3913, count#3914L, sum#3915L, last#3916, valueSet#3917, count#3918L, last#3919, valueSet#3920, count#3921L, count#3922L, count#3923L]
Arguments: SinglePartition, ENSURE_REQUIREMENTS, [plan_id=1651]
(9) ObjectHashAggregate
Input [12]: [buf#3912, buf#3913, count#3914L, sum#3915L, last#3916, valueSet#3917, count#3918L, last#3919, valueSet#3920, count#3921L, count#3922L, count#3923L]
Keys: []
Functions [10]: [collect_set(domainMetadata#3774, 0, 0), collect_set(txn#3766, 0, 0), count(protocol#3770), sum(add#3767.size), last(metaData#3769, true), count(metaData#3769), last(protocol#3770, true), count(remove#3768), count(add#3767), count(txn#3766)]
Aggregate Attributes [10]: [collect_set(domainMetadata#3774, 0, 0)#3824, collect_set(txn#3766, 0, 0)#3823, count(protocol#3770)#3822L, sum(add#3767.size)#3817L, last(metaData#3769) ignore nulls#3825, count(metaData#3769)#3821L, last(protocol#3770) ignore nulls#3826, count(remove#3768)#3820L, count(add#3767)#3819L, count(txn#3766)#3818L]
Results [14]: [collect_set(domainMetadata#3774, 0, 0)#3824 AS domainMetadata#3827, null AS fileSizeHistogram#3828, collect_set(txn#3766, 0, 0)#3823 AS setTransactions#3829, null AS numDeletedRecordsOpt#3830, count(protocol#3770)#3822L AS numOfProtocol#3831L, coalesce(sum(add#3767.size)#3817L, 0) AS sizeInBytes#3832L, last(metaData#3769) ignore nulls#3825 AS metadata#3833, count(metaData#3769)#3821L AS numOfMetadata#3834L, null AS deletedRecordCountsHistogramOpt#3835, null AS numDeletionVectorsOpt#3836, last(protocol#3770) ignore nulls#3826 AS protocol#3837, count(remove#3768)#3820L AS numOfRemoves#3838L, count(add#3767)#3819L AS numOfFiles#3839L, count(txn#3766)#3818L AS numOfSetTransactions#3840L]
(10) AdaptiveSparkPlan
Output [14]: [domainMetadata#3827, fileSizeHistogram#3828, setTransactions#3829, numDeletedRecordsOpt#3830, numOfProtocol#3831L, sizeInBytes#3832L, metadata#3833, numOfMetadata#3834L, deletedRecordCountsHistogramOpt#3835, numDeletionVectorsOpt#3836, protocol#3837, numOfRemoves#3838L, numOfFiles#3839L, numOfSetTransactions#3840L]
Arguments: isFinalPlan=true