Is it just me, or has it taken IBM nearly 20 years to realise hotfiles in impromptu were a good thing and IBM have reincarnated them with a new name called a Dataset ?
Quote from: cadams on 16 Nov 2021 06:56:59 PM
Is it just me, or has it taken IBM nearly 20 years to realise hotfiles in impromptu were a good thing and IBM have reincarnated them with a new name called a Dataset ?
Ha ha - yes, you hit the nail on the head there :-) The difference this time is that data sets are not just local flat files - they are in Parquet (columnar) format for speed and efficiency, and there is a dedicated service written in Spark to make access really zippy. I'd call them hotfiles on steriods :-)
Thanks for making me smile - what a great observation!
Cheers!
MF.