This was my first time attending a Hadoop Summit (thanks to my employer Yahoo for sponsoring!). I had a very positive impression of the event. Talks covered a range of topics and speakers were available for questions and hallway discussions. There were many vendors showcasing their wares as well.<\/p>\n
I mostly attended talks related to streaming \/ real time systems as that\u2019s directly related to the kind of work I do.<\/p>\n
Vendor stalls<\/strong><\/p>\n The vendors stalls brought out the heavy competition among data processing tools that can overwhelm anyone new to the space (see Hadoop Buyer\u2019s Guide mentioned above). There was a talk called \u201cThe data ecosystem is too damn big<\/a>\u201d by Andrew Brust of Datameer that I didn\u2019t attend, but heard people passionately discussing the points made therein. This interview at the conference with the CEO of a new data integration company echoes similar sentiments: https:\/\/www.youtube.com\/watch?v=o9R7iQqQDF4<\/a><\/p>\n Interesting talks (Chronological)<\/strong><\/p>\n All the talks are listed here (with a brief description when you click on its name), including links to video and slides: http:\/\/hadoopsummit.org\/san-jose\/agenda\/<\/a> (unfortunately there\u2019s no direct link to each talk).<\/p>\n There is a YouTube playlist as well: https:\/\/www.youtube.com\/playlist?list=PLKnYDs_-dq16K1NH83Bke2dGGUO3YKZ5b<\/a><\/p>\n Day 1<\/strong><\/p>\n Day 2<\/strong><\/p>\n Day 3<\/strong><\/p>\n Miscellaneous thoughts<\/strong><\/p>\n This was my first time attending a Hadoop Summit (thanks to my employer Yahoo for sponsoring!). I had a very positive impression of the event. Talks covered a range of topics and speakers were available for questions and hallway discussions. There were many vendors showcasing their wares as well. I mostly attended talks related to […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"spay_email":""},"categories":[1],"tags":[5,14],"jetpack_featured_media_url":"","_links":{"self":[{"href":"https:\/\/www.pramodb.com\/index.php\/wp-json\/wp\/v2\/posts\/21"}],"collection":[{"href":"https:\/\/www.pramodb.com\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.pramodb.com\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.pramodb.com\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.pramodb.com\/index.php\/wp-json\/wp\/v2\/comments?post=21"}],"version-history":[{"count":0,"href":"https:\/\/www.pramodb.com\/index.php\/wp-json\/wp\/v2\/posts\/21\/revisions"}],"wp:attachment":[{"href":"https:\/\/www.pramodb.com\/index.php\/wp-json\/wp\/v2\/media?parent=21"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.pramodb.com\/index.php\/wp-json\/wp\/v2\/categories?post=21"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.pramodb.com\/index.php\/wp-json\/wp\/v2\/tags?post=21"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}\n
\n
<\/em>Video: https:\/\/www.youtube.com\/watch?v=3NyTcbIIvxE
<\/a>Slides: http:\/\/www.slideshare.net\/HadoopSummit\/end-to-end-processing-of-37-million-telemetry-events-per-second-using-lambda-architecture<\/a>\u00a0
This talk was a run through of a huge<\/em> number of config parameters that these companies tuned to get a system that could process their workload. Apparently it took them 6 months, 20 people from Symantec and many more from Hortonworks to accomplish this. It was a comprehensive ETL pipeline with every framework and querying engine you can think of having a place somewhere in it. Definitely worth going back and looking at again.<\/li>\n
Video: https:\/\/www.youtube.com\/watch?v=tB28rPTvRiI
<\/a>Slides: http:\/\/www.slideshare.net\/HadoopSummit\/file-format-benchmark-avro-json-orc-parquet<\/a>\u00a0
This one was a rundown of a bunch of benchmarks the speaker performed across the 4 file formats. It was interesting for the quick background he provided about how these various formats came into being. Two of the data sets he used are publicly available and worth looking at: the NY city taxi rides data set, and the Github activity log data set.
A brief summary of the results:
– Compression matters. Apparently, JSON is really bad at getting compressed. Avro puts markers after every 16K. This affects compression window for zlib.
– His recommendation was to use ORC with zlib compression by default; Avro is good for complex tables with common strings, when used with Snappy compression.<\/li>\n
Video: https:\/\/www.youtube.com\/watch?v=85sew9OFaYc
<\/a>Slides: http:\/\/www.slideshare.net\/HadoopSummit\/producing-spark-on-yarn-for-etl<\/a>
Netflix has decided to let ML people use Spark for their analysis and thus needed to support it as part of their data platform. This talk documented the struggles to get Spark working with Yarn, S3 and Parquet, since those are the other technologies used inside of Netflix.
– Some basic numbers: They process 700 billion events per day, which comes to 3 Petabytes per day. Their warehouse has 40 Petabytes. Their Yarn cluster is 3000 EC2 nodes on 2 clusters (d2.4xlarge).
\u200a\u2014\u200aThe talk mentioned all the relevant JIRA\u2019s they filed with various projects as they went through their issues. I will mention a few of them: Coalesce() for RDD based on file size, optimization of UnionRDD for listing parent RDD\u2019s, getting HadoopOutputCommitter to work well with S3, poor performance of broadcast variable seed performance, incorrect locality optimization on S3, Fixing Spark\u2019s job history server, switching to G1 GC\u2026 and a few more
– They mentioned that Spark is 2\u20133 times faster than Pig. They also built some IntelliJ IDE plugin and other tooling to help their users work with Spark<\/li>\n<\/ol>\n\n
Video: https:\/\/www.youtube.com\/watch?v=Dvk0cwqGgws
<\/a>Slides: http:\/\/www.slideshare.net\/HadoopSummit\/building-largescale-stream-infrastructures-across-multiple-data-centers-with-apache-kafka<\/a>
A brilliant talk by one of the authors of Kafka. He gave a systematic overview of issues to keep in mind when your data has to be available in more than one datacenter (in a way that is transparent to producers and\/or consumers), and approaches to handle the same.
The 3 techniques he discussed are: Stretched Cluster, Active\/passive and Active\/active. For each of them he suggested what to do when the primary DC goes down and when it comes back up. A main deficit of Kafka is that offsets are not preserved across DC\u2019s when replication happens. He proposed to overcome this by adding timestamp based querying of a Kafka topic to get the matching offset (Kafka Improvement proposal\u200a\u2014\u200a33)<\/li>\n
Video: https:\/\/www.youtube.com\/watch?v=M1yFgYQ4r6I
<\/a>Slides: http:\/\/www.slideshare.net\/HadoopSummit\/realtime-hadoop-the-ideal-messaging-system-for-hadoop<\/a>
An opinionated talk from a long timer in the field. Had a lot of thought provoking points but I wish he had elaborated more. His book \u201cStreaming Architecture\u201d is a good read.
\u200a\u2014\u200aMicroservices go well with streaming applications. It should be trivial to create and setup of streams and associated services.
\u200a\u2014\u200aEach microservice should ship with its own private datastore. He gave an analogy that in the olden days you needed sysadmin approval for your app to create a file on disk, and now that\u2019s no longer the case.
\u200a\u2014\u200aKafka as a centralized message bus and microservices for reading and writing to it, with private datastores as needed, is the best architecture. We can live with lack of strict consistency between these private datastores.
\u200a\u2014\u200aUse self-describing schemas that can evolve over time
\u200a\u2014\u200aHave common API\u2019s that let you talk to multiple systems (for example Kafka Streams). The Linux\/Posix API\u2019s gave you lot of functionality and compatibility. But Hadoop has sacrificed that in the name of scalability.
\u200a\u2014\u200aThis was followed by a pitch for MapR, how it\u2019s compatible with Kafka Streams API. Its C++ implementation helps with perf.
\u200a\u2014\u200aAPI\u2019s matter more than implementation. Building proprietary systems helps you innovate ahead of the community.<\/li>\n
Video: https:\/\/www.youtube.com\/watch?v=_mJYI8A0luA
<\/a>Slides: http:\/\/www.slideshare.net\/HadoopSummit\/the-next-generation-of-data-processing-and-open-source
<\/a> and also (from Day 3)
\u00a0Apache Beam: A Unified Model for Batch and Streaming Data Processing<\/strong> (Davor Bonaci of Google<\/em>)
Video: https:\/\/www.youtube.com\/watch?v=7DZ8ONmeP5A
<\/a>Slides: http:\/\/www.slideshare.net\/HadoopSummit\/apache-beam-a-unified-model-for-batch-and-stream-processing-data<\/a>
Both the talks covered similar ground, namely the Google Cloud Dataflow API which has been open sourced as Apache Beam. This API seeks to unify stream and batch processing application logic, and let you choose the mode you want during execution, along with whichever \u201cexecution engine\u201d you want it to run on.
\u200a\u2014\u200aThis is the result of Google building several generation of internal systems over the last decade, and noticing common patterns of usage across them.
\u200a\u2014\u200aThe API aims to decouple the following 4 concerns that are relevant in any data pipeline:
\u00a0i) What is being computed?
\u00a0ii) Which events in the stream are appropriate for this computation
\u00a0iii) When in processing time is this particular computation over this event set need to be materialized?
\u00a0iv) How do late coming data relate to already performed computations (they call this Refinements<\/em>)
\u200a\u2014\u200aI believe this API and approach has a lot of potential to raise the level of abstraction w.r.t data pipelines. Check out either of these talks or the Google Cloud Dataflow paper. See the article series \u201cThe world beyond batch\u201d published by these same folks.<\/li>\n
Video: https:\/\/www.youtube.com\/watch?v=Rrr_dy-Uauc
<\/a>Slides: http:\/\/www.slideshare.net\/HadoopSummit\/lambdaless-stream-processing-scale-in-linkedin<\/a>
This was a good experience report. It was a self-acknowledged implementation of ideas from the Google Cloud Dataflow model mentioned above. They had to implement this model in the Linkedin Newsfeed and Fraud detection on their Ad Serving engines. The hard issues to tackle were handling late events (how they relate to already processed data), and cross DC replication.<\/li>\n<\/ol>\n\n
Video: https:\/\/www.youtube.com\/watch?v=cUgfor9vgIM
<\/a>Slides: http:\/\/www.slideshare.net\/HadoopSummit\/crossdc-faulttolerant-viewfilesystem-twitter<\/a>
I really liked the idea behind this talk: Abstract away the notion of multiple DC\u2019s by providing a transparent URI scheme that works everywhere, i.e, viewfs:\/\/path\/to\/file will work across all of Twitter clusters.
The implementation required understanding some core concepts of how HDFS works and tweaking them appropriately. Strong consistency on writes was a hard problem (not sure if they solved it). The HDFS config subsystem is too messy and they decided to create a parallel config system or something, to enable their cross DC configs.<\/li>\n
Video: https:\/\/www.youtube.com\/watch?v=odpRIZVNadQ
<\/a>Slides: http:\/\/www.slideshare.net\/HadoopSummit\/the-stream-processor-as-a-database-apache-flink<\/a>
This was another great talk that showed how small tweaks to Apache Flink let it perform queries too and not just act as a data store. The Scala API used to demo this part of Flink had a strong resemblance to Google Cloud Dataflow API again.
\u200a\u2014\u200aTo enable querying, the system has to support storing state associated with a query as it runs (Storm does not have it). They do this in Flink by using a local, embedded RocksDB database in each Flink task manager. He criticized the Yahoo Streaming benchmarks saying they actually showed that the bottleneck was the key-value store and not the querying engines.<\/li>\n
Video: https:\/\/www.youtube.com\/watch?v=_A5ugvLZJNM
<\/a>Slides: http:\/\/www.slideshare.net\/HadoopSummit\/ingest-and-stream-processing-what-will-you-choose<\/a>
This one was Cloudera\u2019s opinionated take on all the various frameworks out there, their pros and cons. It will be a useful reference in case you want to get a bird\u2019s eye view of the space. I\u2019m not adding the specifics here as that will turn out to be just a long list of bullet points, tips and tricks.
\u200a\u2014\u200aThe talk stoked a minor controversy as he mentioned that Cloudera considers Apache Storm to be a legacy technology at this point(!)
\u200a\u2014\u200aHe criticized the Google Cloud Dataflow model saying programmatic abstractions tend not to last (as opposed to query languages and UI abstractions I guess?)
\u200a\u2014\u200aThis set up the last part of the talk, which was a brief demo of StreamSets<\/li>\n<\/ol>\n\n