== Physical Plan ==
AdaptiveSparkPlan (11)
+- == Final Plan ==
ResultQueryStage (7), Statistics(sizeInBytes=8.0 EiB)
+- ObjectHashAggregate (6)
+- ShuffleQueryStage (5), Statistics(sizeInBytes=10.7 KiB, rowCount=50)
+- Exchange (4)
+- ObjectHashAggregate (3)
+- * Project (2)
+- * Scan ExistingRDD Delta Table State #1 - hdlfs://ced12e6d-bca7-47af-9525-449cba9023d7.files.hdl.prod-eu20.hanacloud.ondemand.com:443/crp-dl-stream-service/cornerstone/sap-cic-product-productplant/_delta_log (1)
+- == Initial Plan ==
ObjectHashAggregate (10)
+- Exchange (9)
+- ObjectHashAggregate (8)
+- Project (2)
+- Scan ExistingRDD Delta Table State #1 - hdlfs://ced12e6d-bca7-47af-9525-449cba9023d7.files.hdl.prod-eu20.hanacloud.ondemand.com:443/crp-dl-stream-service/cornerstone/sap-cic-product-productplant/_delta_log (1)
(1) Scan ExistingRDD Delta Table State #1 - hdlfs://ced12e6d-bca7-47af-9525-449cba9023d7.files.hdl.prod-eu20.hanacloud.ondemand.com:443/crp-dl-stream-service/cornerstone/sap-cic-product-productplant/_delta_log [codegen id : 1]
Output [10]: [txn#906039, add#906040, remove#906041, metaData#906042, protocol#906043, cdc#906044, checkpointMetadata#906045, sidecar#906046, domainMetadata#906047, commitInfo#906048]
Arguments: [txn#906039, add#906040, remove#906041, metaData#906042, protocol#906043, cdc#906044, checkpointMetadata#906045, sidecar#906046, domainMetadata#906047, commitInfo#906048], Delta Table State #1 - hdlfs://ced12e6d-bca7-47af-9525-449cba9023d7.files.hdl.prod-eu20.hanacloud.ondemand.com:443/crp-dl-stream-service/cornerstone/sap-cic-product-productplant/_delta_log MapPartitionsRDD[45109] at $anonfun$recordDeltaOperationInternal$1 at DatabricksLogging.scala:128, ExistingRDD, UnknownPartitioning(0)
(2) Project [codegen id : 1]
Output [6]: [txn#906039, add#906040, remove#906041, metaData#906042, protocol#906043, domainMetadata#906047]
Input [10]: [txn#906039, add#906040, remove#906041, metaData#906042, protocol#906043, cdc#906044, checkpointMetadata#906045, sidecar#906046, domainMetadata#906047, commitInfo#906048]
(3) ObjectHashAggregate
Input [6]: [txn#906039, add#906040, remove#906041, metaData#906042, protocol#906043, domainMetadata#906047]
Keys: []
Functions [10]: [partial_collect_set(domainMetadata#906047, 0, 0), partial_collect_set(txn#906039, 0, 0), partial_count(protocol#906043), partial_sum(add#906040.size), partial_last(metaData#906042, true), partial_count(metaData#906042), partial_last(protocol#906043, true), partial_count(remove#906041), partial_count(add#906040), partial_count(txn#906039)]
Aggregate Attributes [12]: [buf#906149, buf#906150, count#906151L, sum#906152L, last#906153, valueSet#906154, count#906155L, last#906156, valueSet#906157, count#906158L, count#906159L, count#906160L]
Results [12]: [buf#906161, buf#906162, count#906163L, sum#906164L, last#906165, valueSet#906166, count#906167L, last#906168, valueSet#906169, count#906170L, count#906171L, count#906172L]
(4) Exchange
Input [12]: [buf#906161, buf#906162, count#906163L, sum#906164L, last#906165, valueSet#906166, count#906167L, last#906168, valueSet#906169, count#906170L, count#906171L, count#906172L]
Arguments: SinglePartition, ENSURE_REQUIREMENTS, [plan_id=1395136]
(5) ShuffleQueryStage
Output [12]: [buf#906161, buf#906162, count#906163L, sum#906164L, last#906165, valueSet#906166, count#906167L, last#906168, valueSet#906169, count#906170L, count#906171L, count#906172L]
Arguments: 0
(6) ObjectHashAggregate
Input [12]: [buf#906161, buf#906162, count#906163L, sum#906164L, last#906165, valueSet#906166, count#906167L, last#906168, valueSet#906169, count#906170L, count#906171L, count#906172L]
Keys: []
Functions [10]: [collect_set(domainMetadata#906047, 0, 0), collect_set(txn#906039, 0, 0), count(protocol#906043), sum(add#906040.size), last(metaData#906042, true), count(metaData#906042), last(protocol#906043, true), count(remove#906041), count(add#906040), count(txn#906039)]
Aggregate Attributes [10]: [collect_set(domainMetadata#906047, 0, 0)#906095, collect_set(txn#906039, 0, 0)#906096, count(protocol#906043)#906097L, sum(add#906040.size)#906098L, last(metaData#906042) ignore nulls#906099, count(metaData#906042)#906100L, last(protocol#906043) ignore nulls#906101, count(remove#906041)#906102L, count(add#906040)#906103L, count(txn#906039)#906104L]
Results [14]: [collect_set(domainMetadata#906047, 0, 0)#906095 AS domainMetadata#906080, null AS fileSizeHistogram#906081, collect_set(txn#906039, 0, 0)#906096 AS setTransactions#906082, null AS numDeletedRecordsOpt#906083, count(protocol#906043)#906097L AS numOfProtocol#906084L, coalesce(sum(add#906040.size)#906098L, 0) AS sizeInBytes#906085L, last(metaData#906042) ignore nulls#906099 AS metadata#906086, count(metaData#906042)#906100L AS numOfMetadata#906087L, null AS deletedRecordCountsHistogramOpt#906088, null AS numDeletionVectorsOpt#906089, last(protocol#906043) ignore nulls#906101 AS protocol#906090, count(remove#906041)#906102L AS numOfRemoves#906091L, count(add#906040)#906103L AS numOfFiles#906092L, count(txn#906039)#906104L AS numOfSetTransactions#906093L]
(7) ResultQueryStage
Output [14]: [domainMetadata#906080, fileSizeHistogram#906081, setTransactions#906082, numDeletedRecordsOpt#906083, numOfProtocol#906084L, sizeInBytes#906085L, metadata#906086, numOfMetadata#906087L, deletedRecordCountsHistogramOpt#906088, numDeletionVectorsOpt#906089, protocol#906090, numOfRemoves#906091L, numOfFiles#906092L, numOfSetTransactions#906093L]
Arguments: 1
(8) ObjectHashAggregate
Input [6]: [txn#906039, add#906040, remove#906041, metaData#906042, protocol#906043, domainMetadata#906047]
Keys: []
Functions [10]: [partial_collect_set(domainMetadata#906047, 0, 0), partial_collect_set(txn#906039, 0, 0), partial_count(protocol#906043), partial_sum(add#906040.size), partial_last(metaData#906042, true), partial_count(metaData#906042), partial_last(protocol#906043, true), partial_count(remove#906041), partial_count(add#906040), partial_count(txn#906039)]
Aggregate Attributes [12]: [buf#906149, buf#906150, count#906151L, sum#906152L, last#906153, valueSet#906154, count#906155L, last#906156, valueSet#906157, count#906158L, count#906159L, count#906160L]
Results [12]: [buf#906161, buf#906162, count#906163L, sum#906164L, last#906165, valueSet#906166, count#906167L, last#906168, valueSet#906169, count#906170L, count#906171L, count#906172L]
(9) Exchange
Input [12]: [buf#906161, buf#906162, count#906163L, sum#906164L, last#906165, valueSet#906166, count#906167L, last#906168, valueSet#906169, count#906170L, count#906171L, count#906172L]
Arguments: SinglePartition, ENSURE_REQUIREMENTS, [plan_id=1395125]
(10) ObjectHashAggregate
Input [12]: [buf#906161, buf#906162, count#906163L, sum#906164L, last#906165, valueSet#906166, count#906167L, last#906168, valueSet#906169, count#906170L, count#906171L, count#906172L]
Keys: []
Functions [10]: [collect_set(domainMetadata#906047, 0, 0), collect_set(txn#906039, 0, 0), count(protocol#906043), sum(add#906040.size), last(metaData#906042, true), count(metaData#906042), last(protocol#906043, true), count(remove#906041), count(add#906040), count(txn#906039)]
Aggregate Attributes [10]: [collect_set(domainMetadata#906047, 0, 0)#906095, collect_set(txn#906039, 0, 0)#906096, count(protocol#906043)#906097L, sum(add#906040.size)#906098L, last(metaData#906042) ignore nulls#906099, count(metaData#906042)#906100L, last(protocol#906043) ignore nulls#906101, count(remove#906041)#906102L, count(add#906040)#906103L, count(txn#906039)#906104L]
Results [14]: [collect_set(domainMetadata#906047, 0, 0)#906095 AS domainMetadata#906080, null AS fileSizeHistogram#906081, collect_set(txn#906039, 0, 0)#906096 AS setTransactions#906082, null AS numDeletedRecordsOpt#906083, count(protocol#906043)#906097L AS numOfProtocol#906084L, coalesce(sum(add#906040.size)#906098L, 0) AS sizeInBytes#906085L, last(metaData#906042) ignore nulls#906099 AS metadata#906086, count(metaData#906042)#906100L AS numOfMetadata#906087L, null AS deletedRecordCountsHistogramOpt#906088, null AS numDeletionVectorsOpt#906089, last(protocol#906043) ignore nulls#906101 AS protocol#906090, count(remove#906041)#906102L AS numOfRemoves#906091L, count(add#906040)#906103L AS numOfFiles#906092L, count(txn#906039)#906104L AS numOfSetTransactions#906093L]
(11) AdaptiveSparkPlan
Output [14]: [domainMetadata#906080, fileSizeHistogram#906081, setTransactions#906082, numDeletedRecordsOpt#906083, numOfProtocol#906084L, sizeInBytes#906085L, metadata#906086, numOfMetadata#906087L, deletedRecordCountsHistogramOpt#906088, numDeletionVectorsOpt#906089, protocol#906090, numOfRemoves#906091L, numOfFiles#906092L, numOfSetTransactions#906093L]
Arguments: isFinalPlan=true