The df.write.partitionBy('year','month','day').mode('overwrite') results in what data layout?

• A. A single large file per partition
• B. A flat folder with all partitions mixed
• C. No partitioning effect
• D. A hierarchy of folders for each partition key that can be read in parallel across multiple nodes

Answer: (D)

Partitioning by year, month, and day creates a directory hierarchy where data for each combination of those keys lives in its own folder. You end up with a path structure like root/year=2023/month=06/day=09/ and Spark writes data into these partition folders, often producing multiple files inside each partition due to parallel tasks. This layout allows parts of the data to be read in parallel across different nodes and enables partition pruning when querying by those columns. Using overwrite clears the existing data under the target path before writing the new files. This is different from a flat folder structure or no partitioning at all.