Datasets

[cadams]

Is it just me, or has it taken IBM nearly 20 years to realise hotfiles in impromptu were a good thing and IBM have reincarnated them with a new name called a Dataset ?

[MFGF]

Is it just me, or has it taken IBM nearly 20 years to realise hotfiles in impromptu were a good thing and IBM have reincarnated them with a new name called a Dataset ?

Ha ha - yes, you hit the nail on the head there :-) The difference this time is that data sets are not just local flat files - they are in Parquet (columnar) format for speed and efficiency, and there is a dedicated service written in Spark to make access really zippy. I'd call them hotfiles on steriods :-)

Thanks for making me smile - what a great observation!

Cheers!

MF.