Lessons learnt building data pipelines — 1

It has been a few years since I embarked on what was then trendily called “data engineering”. That title — and the associated work — was an outgrowth of a tendency for computing systems to accumulate vastly more data than used to be the case before, and more importantly, try to extract business value by sifting through it. Nowadays […]

Notes from Hadoop Summit 2016

This was my first time attending a Hadoop Summit (thanks to my employer Yahoo for sponsoring!). I had a very positive impression of the event. Talks covered a range of topics and speakers were available for questions and hallway discussions. There were many vendors showcasing their wares as well. I mostly attended talks related to […]