2016-02-21

译：使用Kafka Connect构建大规模低延迟的数据管道

Announcing Kafka Connect: Building large-scale low-latency data pipelines
http://www.confluent.io/blog/announcing-kafka-connect-building-large-scale-low-latency-data-pipelines

For a long time, a substantial portion of data processing that companies did ran as big batch jobs — CSV files dumped out of databases, log files collected at the end of the day etc. But businesses operate in real-time and the software they run is catching up. Rather than processing data only at the end of the day, why not react to it continuously as the data arrives? This is the emerging world of stream processing. But stream processing only becomes possible when the fundamental data capture is done in a streaming fashion; after all, you can’t process a daily batch of CSV dumps as a stream. This shift towards stream processing has driven the popularity of Apache Kafka. But, even with Kafka, building this type of real-time data pipeline has required some effort.

很长一段时间以来,数据处理的一个重要部分是运行一个大的批处理作业–从数据库中导出CSV文件,在每天结束时收集所有的日志文件等等. 但是实时的商业交易因为底层软件的缺陷而无法做出实时的响应. 为什么非要在每天结束的时候才处理数据,而不是在数据到来时持续地做出响应? 这是流处理最紧迫处理的任务. 但是流处理只有在基础数据的抓取用流式的方式抓取才能变得可能.毕竟你不能把一整天的CSV dumps作为一个流. 这种流处理的变化使得Apache Kafka变得流行起来. 但是即使是Kafka,构建实时数据管道风格也需要一些工作. 现在0.9+版本的kafka已经有了一个全新功能的Kafka Connect,使得构建和管理流式数据管道变得更加容易.