== Physical Plan ==
AdaptiveSparkPlan (11)
+- == Final Plan ==
ResultQueryStage (7), Statistics(sizeInBytes=8.0 EiB)
+- ObjectHashAggregate (6)
+- ShuffleQueryStage (5), Statistics(sizeInBytes=10.7 KiB, rowCount=50)
+- Exchange (4)
+- ObjectHashAggregate (3)
+- * Project (2)
+- * Scan ExistingRDD Delta Table State #1 - hdlfs://ced12e6d-bca7-47af-9525-449cba9023d7.files.hdl.prod-eu20.hanacloud.ondemand.com:443/crp-dl-stream-service/cornerstone/sap-cic-product-productplant/_delta_log (1)
+- == Initial Plan ==
ObjectHashAggregate (10)
+- Exchange (9)
+- ObjectHashAggregate (8)
+- Project (2)
+- Scan ExistingRDD Delta Table State #1 - hdlfs://ced12e6d-bca7-47af-9525-449cba9023d7.files.hdl.prod-eu20.hanacloud.ondemand.com:443/crp-dl-stream-service/cornerstone/sap-cic-product-productplant/_delta_log (1)
(1) Scan ExistingRDD Delta Table State #1 - hdlfs://ced12e6d-bca7-47af-9525-449cba9023d7.files.hdl.prod-eu20.hanacloud.ondemand.com:443/crp-dl-stream-service/cornerstone/sap-cic-product-productplant/_delta_log [codegen id : 1]
Output [10]: [txn#13792574, add#13792575, remove#13792576, metaData#13792577, protocol#13792578, cdc#13792579, checkpointMetadata#13792580, sidecar#13792581, domainMetadata#13792582, commitInfo#13792583]
Arguments: [txn#13792574, add#13792575, remove#13792576, metaData#13792577, protocol#13792578, cdc#13792579, checkpointMetadata#13792580, sidecar#13792581, domainMetadata#13792582, commitInfo#13792583], Delta Table State #1 - hdlfs://ced12e6d-bca7-47af-9525-449cba9023d7.files.hdl.prod-eu20.hanacloud.ondemand.com:443/crp-dl-stream-service/cornerstone/sap-cic-product-productplant/_delta_log MapPartitionsRDD[789257] at $anonfun$recordDeltaOperationInternal$1 at DatabricksLogging.scala:128, ExistingRDD, UnknownPartitioning(0)
(2) Project [codegen id : 1]
Output [6]: [txn#13792574, add#13792575, remove#13792576, metaData#13792577, protocol#13792578, domainMetadata#13792582]
Input [10]: [txn#13792574, add#13792575, remove#13792576, metaData#13792577, protocol#13792578, cdc#13792579, checkpointMetadata#13792580, sidecar#13792581, domainMetadata#13792582, commitInfo#13792583]
(3) ObjectHashAggregate
Input [6]: [txn#13792574, add#13792575, remove#13792576, metaData#13792577, protocol#13792578, domainMetadata#13792582]
Keys: []
Functions [10]: [partial_collect_set(domainMetadata#13792582, 0, 0), partial_collect_set(txn#13792574, 0, 0), partial_count(protocol#13792578), partial_sum(add#13792575.size), partial_last(metaData#13792577, true), partial_count(metaData#13792577), partial_last(protocol#13792578, true), partial_count(remove#13792576), partial_count(add#13792575), partial_count(txn#13792574)]
Aggregate Attributes [12]: [buf#13792684, buf#13792685, count#13792686L, sum#13792687L, last#13792688, valueSet#13792689, count#13792690L, last#13792691, valueSet#13792692, count#13792693L, count#13792694L, count#13792695L]
Results [12]: [buf#13792696, buf#13792697, count#13792698L, sum#13792699L, last#13792700, valueSet#13792701, count#13792702L, last#13792703, valueSet#13792704, count#13792705L, count#13792706L, count#13792707L]
(4) Exchange
Input [12]: [buf#13792696, buf#13792697, count#13792698L, sum#13792699L, last#13792700, valueSet#13792701, count#13792702L, last#13792703, valueSet#13792704, count#13792705L, count#13792706L, count#13792707L]
Arguments: SinglePartition, ENSURE_REQUIREMENTS, [plan_id=13505515]
(5) ShuffleQueryStage
Output [12]: [buf#13792696, buf#13792697, count#13792698L, sum#13792699L, last#13792700, valueSet#13792701, count#13792702L, last#13792703, valueSet#13792704, count#13792705L, count#13792706L, count#13792707L]
Arguments: 0
(6) ObjectHashAggregate
Input [12]: [buf#13792696, buf#13792697, count#13792698L, sum#13792699L, last#13792700, valueSet#13792701, count#13792702L, last#13792703, valueSet#13792704, count#13792705L, count#13792706L, count#13792707L]
Keys: []
Functions [10]: [collect_set(domainMetadata#13792582, 0, 0), collect_set(txn#13792574, 0, 0), count(protocol#13792578), sum(add#13792575.size), last(metaData#13792577, true), count(metaData#13792577), last(protocol#13792578, true), count(remove#13792576), count(add#13792575), count(txn#13792574)]
Aggregate Attributes [10]: [collect_set(domainMetadata#13792582, 0, 0)#13792630, collect_set(txn#13792574, 0, 0)#13792631, count(protocol#13792578)#13792632L, sum(add#13792575.size)#13792633L, last(metaData#13792577) ignore nulls#13792634, count(metaData#13792577)#13792635L, last(protocol#13792578) ignore nulls#13792636, count(remove#13792576)#13792637L, count(add#13792575)#13792638L, count(txn#13792574)#13792639L]
Results [14]: [collect_set(domainMetadata#13792582, 0, 0)#13792630 AS domainMetadata#13792615, null AS fileSizeHistogram#13792616, collect_set(txn#13792574, 0, 0)#13792631 AS setTransactions#13792617, null AS numDeletedRecordsOpt#13792618, count(protocol#13792578)#13792632L AS numOfProtocol#13792619L, coalesce(sum(add#13792575.size)#13792633L, 0) AS sizeInBytes#13792620L, last(metaData#13792577) ignore nulls#13792634 AS metadata#13792621, count(metaData#13792577)#13792635L AS numOfMetadata#13792622L, null AS deletedRecordCountsHistogramOpt#13792623, null AS numDeletionVectorsOpt#13792624, last(protocol#13792578) ignore nulls#13792636 AS protocol#13792625, count(remove#13792576)#13792637L AS numOfRemoves#13792626L, count(add#13792575)#13792638L AS numOfFiles#13792627L, count(txn#13792574)#13792639L AS numOfSetTransactions#13792628L]
(7) ResultQueryStage
Output [14]: [domainMetadata#13792615, fileSizeHistogram#13792616, setTransactions#13792617, numDeletedRecordsOpt#13792618, numOfProtocol#13792619L, sizeInBytes#13792620L, metadata#13792621, numOfMetadata#13792622L, deletedRecordCountsHistogramOpt#13792623, numDeletionVectorsOpt#13792624, protocol#13792625, numOfRemoves#13792626L, numOfFiles#13792627L, numOfSetTransactions#13792628L]
Arguments: 1
(8) ObjectHashAggregate
Input [6]: [txn#13792574, add#13792575, remove#13792576, metaData#13792577, protocol#13792578, domainMetadata#13792582]
Keys: []
Functions [10]: [partial_collect_set(domainMetadata#13792582, 0, 0), partial_collect_set(txn#13792574, 0, 0), partial_count(protocol#13792578), partial_sum(add#13792575.size), partial_last(metaData#13792577, true), partial_count(metaData#13792577), partial_last(protocol#13792578, true), partial_count(remove#13792576), partial_count(add#13792575), partial_count(txn#13792574)]
Aggregate Attributes [12]: [buf#13792684, buf#13792685, count#13792686L, sum#13792687L, last#13792688, valueSet#13792689, count#13792690L, last#13792691, valueSet#13792692, count#13792693L, count#13792694L, count#13792695L]
Results [12]: [buf#13792696, buf#13792697, count#13792698L, sum#13792699L, last#13792700, valueSet#13792701, count#13792702L, last#13792703, valueSet#13792704, count#13792705L, count#13792706L, count#13792707L]
(9) Exchange
Input [12]: [buf#13792696, buf#13792697, count#13792698L, sum#13792699L, last#13792700, valueSet#13792701, count#13792702L, last#13792703, valueSet#13792704, count#13792705L, count#13792706L, count#13792707L]
Arguments: SinglePartition, ENSURE_REQUIREMENTS, [plan_id=13505504]
(10) ObjectHashAggregate
Input [12]: [buf#13792696, buf#13792697, count#13792698L, sum#13792699L, last#13792700, valueSet#13792701, count#13792702L, last#13792703, valueSet#13792704, count#13792705L, count#13792706L, count#13792707L]
Keys: []
Functions [10]: [collect_set(domainMetadata#13792582, 0, 0), collect_set(txn#13792574, 0, 0), count(protocol#13792578), sum(add#13792575.size), last(metaData#13792577, true), count(metaData#13792577), last(protocol#13792578, true), count(remove#13792576), count(add#13792575), count(txn#13792574)]
Aggregate Attributes [10]: [collect_set(domainMetadata#13792582, 0, 0)#13792630, collect_set(txn#13792574, 0, 0)#13792631, count(protocol#13792578)#13792632L, sum(add#13792575.size)#13792633L, last(metaData#13792577) ignore nulls#13792634, count(metaData#13792577)#13792635L, last(protocol#13792578) ignore nulls#13792636, count(remove#13792576)#13792637L, count(add#13792575)#13792638L, count(txn#13792574)#13792639L]
Results [14]: [collect_set(domainMetadata#13792582, 0, 0)#13792630 AS domainMetadata#13792615, null AS fileSizeHistogram#13792616, collect_set(txn#13792574, 0, 0)#13792631 AS setTransactions#13792617, null AS numDeletedRecordsOpt#13792618, count(protocol#13792578)#13792632L AS numOfProtocol#13792619L, coalesce(sum(add#13792575.size)#13792633L, 0) AS sizeInBytes#13792620L, last(metaData#13792577) ignore nulls#13792634 AS metadata#13792621, count(metaData#13792577)#13792635L AS numOfMetadata#13792622L, null AS deletedRecordCountsHistogramOpt#13792623, null AS numDeletionVectorsOpt#13792624, last(protocol#13792578) ignore nulls#13792636 AS protocol#13792625, count(remove#13792576)#13792637L AS numOfRemoves#13792626L, count(add#13792575)#13792638L AS numOfFiles#13792627L, count(txn#13792574)#13792639L AS numOfSetTransactions#13792628L]
(11) AdaptiveSparkPlan
Output [14]: [domainMetadata#13792615, fileSizeHistogram#13792616, setTransactions#13792617, numDeletedRecordsOpt#13792618, numOfProtocol#13792619L, sizeInBytes#13792620L, metadata#13792621, numOfMetadata#13792622L, deletedRecordCountsHistogramOpt#13792623, numDeletionVectorsOpt#13792624, protocol#13792625, numOfRemoves#13792626L, numOfFiles#13792627L, numOfSetTransactions#13792628L]
Arguments: isFinalPlan=true