278 lines
12 KiB
HTML
278 lines
12 KiB
HTML
<!--
|
|
Licensed to the Apache Software Foundation (ASF) under one or more
|
|
contributor license agreements. See the NOTICE file distributed with
|
|
this work for additional information regarding copyright ownership.
|
|
The ASF licenses this file to You under the Apache License, Version 2.0
|
|
(the "License"); you may not use this file except in compliance with
|
|
the License. You may obtain a copy of the License at
|
|
|
|
http://www.apache.org/licenses/LICENSE-2.0
|
|
|
|
Unless required by applicable law or agreed to in writing, software
|
|
distributed under the License is distributed on an "AS IS" BASIS,
|
|
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
|
See the License for the specific language governing permissions and
|
|
limitations under the License.
|
|
-->
|
|
|
|
<script>
|
|
<!--#include virtual="js/templateData.js" -->
|
|
</script>
|
|
|
|
<script id="quickstart-template" type="text/x-handlebars-template">
|
|
|
|
<div class="quickstart-step">
|
|
<h4 class="anchor-heading">
|
|
<a class="anchor-link" id="quickstart_download" href="#quickstart_download"></a>
|
|
<a href="#quickstart_download">Step 1: Get Kafka</a>
|
|
</h4>
|
|
|
|
<p>
|
|
<a href="https://www.apache.org/dyn/closer.cgi?path=/kafka/3.1.0/kafka_2.13-3.1.0.tgz">Download</a>
|
|
the latest Kafka release and extract it:
|
|
</p>
|
|
|
|
<pre class="line-numbers"><code class="language-bash">$ tar -xzf kafka_2.13-3.1.0.tgz
|
|
$ cd kafka_2.13-3.1.0</code></pre>
|
|
</div>
|
|
|
|
<div class="quickstart-step">
|
|
<h4 class="anchor-heading">
|
|
<a class="anchor-link" id="quickstart_startserver" href="#quickstart_startserver"></a>
|
|
<a href="#quickstart_startserver">Step 2: Start the Kafka environment</a>
|
|
</h4>
|
|
|
|
<p class="note">
|
|
NOTE: Your local environment must have Java 8+ installed.
|
|
</p>
|
|
|
|
<p>
|
|
Run the following commands in order to start all services in the correct order:
|
|
</p>
|
|
|
|
<pre class="line-numbers"><code class="language-bash"># Start the ZooKeeper service
|
|
# Note: Soon, ZooKeeper will no longer be required by Apache Kafka.
|
|
$ bin/zookeeper-server-start.sh config/zookeeper.properties</code></pre>
|
|
|
|
<p>
|
|
Open another terminal session and run:
|
|
</p>
|
|
|
|
<pre class="line-numbers"><code class="language-bash"># Start the Kafka broker service
|
|
$ bin/kafka-server-start.sh config/server.properties</code></pre>
|
|
|
|
<p>
|
|
Once all services have successfully launched, you will have a basic Kafka environment running and ready to use.
|
|
</p>
|
|
</div>
|
|
|
|
<div class="quickstart-step">
|
|
<h4 class="anchor-heading">
|
|
<a class="anchor-link" id="quickstart_createtopic" href="#quickstart_createtopic"></a>
|
|
<a href="#quickstart_createtopic">Step 3: Create a topic to store your events</a>
|
|
</h4>
|
|
|
|
<p>
|
|
Kafka is a distributed <em>event streaming platform</em> that lets you read, write, store, and process
|
|
<a href="/documentation/#messages"><em>events</em></a> (also called <em>records</em> or
|
|
<em>messages</em> in the documentation)
|
|
across many machines.
|
|
</p>
|
|
|
|
<p>
|
|
Example events are payment transactions, geolocation updates from mobile phones, shipping orders, sensor measurements
|
|
from IoT devices or medical equipment, and much more. These events are organized and stored in
|
|
<a href="/documentation/#intro_concepts_and_terms"><em>topics</em></a>.
|
|
Very simplified, a topic is similar to a folder in a filesystem, and the events are the files in that folder.
|
|
</p>
|
|
|
|
<p>
|
|
So before you can write your first events, you must create a topic. Open another terminal session and run:
|
|
</p>
|
|
|
|
<pre class="line-numbers"><code class="language-bash">$ bin/kafka-topics.sh --create --topic quickstart-events --bootstrap-server localhost:9092</code></pre>
|
|
|
|
<p>
|
|
All of Kafka's command line tools have additional options: run the <code>kafka-topics.sh</code> command without any
|
|
arguments to display usage information. For example, it can also show you
|
|
<a href="/documentation/#intro_concepts_and_terms">details such as the partition count</a>
|
|
of the new topic:
|
|
</p>
|
|
|
|
<pre class="line-numbers"><code class="language-bash">$ bin/kafka-topics.sh --describe --topic quickstart-events --bootstrap-server localhost:9092
|
|
Topic:quickstart-events PartitionCount:1 ReplicationFactor:1 Configs:
|
|
Topic: quickstart-events Partition: 0 Leader: 0 Replicas: 0 Isr: 0</code></pre>
|
|
</div>
|
|
|
|
<div class="quickstart-step">
|
|
<h4 class="anchor-heading">
|
|
<a class="anchor-link" id="quickstart_send" href="#quickstart_send"></a>
|
|
<a href="#quickstart_send">Step 4: Write some events into the topic</a>
|
|
</h4>
|
|
|
|
<p>
|
|
A Kafka client communicates with the Kafka brokers via the network for writing (or reading) events.
|
|
Once received, the brokers will store the events in a durable and fault-tolerant manner for as long as you
|
|
need—even forever.
|
|
</p>
|
|
|
|
<p>
|
|
Run the console producer client to write a few events into your topic.
|
|
By default, each line you enter will result in a separate event being written to the topic.
|
|
</p>
|
|
|
|
<pre class="line-numbers"><code class="language-bash">$ bin/kafka-console-producer.sh --topic quickstart-events --bootstrap-server localhost:9092
|
|
This is my first event
|
|
This is my second event</code></pre>
|
|
|
|
<p>
|
|
You can stop the producer client with <code>Ctrl-C</code> at any time.
|
|
</p>
|
|
</div>
|
|
|
|
<div class="quickstart-step">
|
|
<h4 class="anchor-heading">
|
|
<a class="anchor-link" id="quickstart_consume" href="#quickstart_consume"></a>
|
|
<a href="#quickstart_consume">Step 5: Read the events</a>
|
|
</h4>
|
|
|
|
<p>Open another terminal session and run the console consumer client to read the events you just created:</p>
|
|
|
|
<pre class="line-numbers"><code class="language-bash">$ bin/kafka-console-consumer.sh --topic quickstart-events --from-beginning --bootstrap-server localhost:9092
|
|
This is my first event
|
|
This is my second event</code></pre>
|
|
|
|
<p>You can stop the consumer client with <code>Ctrl-C</code> at any time.</p>
|
|
|
|
<p>Feel free to experiment: for example, switch back to your producer terminal (previous step) to write
|
|
additional events, and see how the events immediately show up in your consumer terminal.</p>
|
|
|
|
<p>Because events are durably stored in Kafka, they can be read as many times and by as many consumers as you want.
|
|
You can easily verify this by opening yet another terminal session and re-running the previous command again.</p>
|
|
</div>
|
|
|
|
<div class="quickstart-step">
|
|
<h4 class="anchor-heading">
|
|
<a class="anchor-link" id="quickstart_kafkaconnect" href="#quickstart_kafkaconnect"></a>
|
|
<a href="#quickstart_kafkaconnect">Step 6: Import/export your data as streams of events with Kafka Connect</a>
|
|
</h4>
|
|
|
|
<p>
|
|
You probably have lots of data in existing systems like relational databases or traditional messaging systems,
|
|
along with many applications that already use these systems.
|
|
<a href="/documentation/#connect">Kafka Connect</a> allows you to continuously ingest
|
|
data from external systems into Kafka, and vice versa. It is thus very easy to integrate existing systems with
|
|
Kafka. To make this process even easier, there are hundreds of such connectors readily available.
|
|
</p>
|
|
|
|
<p>Take a look at the <a href="/documentation/#connect">Kafka Connect section</a>
|
|
to learn more about how to continuously import/export your data into and out of Kafka.</p>
|
|
|
|
</div>
|
|
|
|
<div class="quickstart-step">
|
|
<h4 class="anchor-heading">
|
|
<a class="anchor-link" id="quickstart_kafkastreams" href="#quickstart_kafkastreams"></a>
|
|
<a href="#quickstart_kafkastreams">Step 7: Process your events with Kafka Streams</a>
|
|
</h4>
|
|
|
|
<p>
|
|
Once your data is stored in Kafka as events, you can process the data with the
|
|
<a href="/documentation/streams">Kafka Streams</a> client library for Java/Scala.
|
|
It allows you to implement mission-critical real-time applications and microservices, where the input
|
|
and/or output data is stored in Kafka topics. Kafka Streams combines the simplicity of writing and deploying
|
|
standard Java and Scala applications on the client side with the benefits of Kafka's server-side cluster
|
|
technology to make these applications highly scalable, elastic, fault-tolerant, and distributed. The library
|
|
supports exactly-once processing, stateful operations and aggregations, windowing, joins, processing based
|
|
on event-time, and much more.
|
|
</p>
|
|
|
|
<p>To give you a first taste, here's how one would implement the popular <code>WordCount</code> algorithm:</p>
|
|
|
|
<pre class="line-numbers"><code class="language-bash">KStream<String, String> textLines = builder.stream("quickstart-events");
|
|
|
|
KTable<String, Long> wordCounts = textLines
|
|
.flatMapValues(line -> Arrays.asList(line.toLowerCase().split(" ")))
|
|
.groupBy((keyIgnored, word) -> word)
|
|
.count();
|
|
|
|
wordCounts.toStream().to("output-topic", Produced.with(Serdes.String(), Serdes.Long()));</code></pre>
|
|
|
|
<p>
|
|
The <a href="/25/documentation/streams/quickstart">Kafka Streams demo</a>
|
|
and the <a href="/25/documentation/streams/tutorial">app development tutorial</a>
|
|
demonstrate how to code and run such a streaming application from start to finish.
|
|
</p>
|
|
|
|
</div>
|
|
|
|
<div class="quickstart-step">
|
|
<h4 class="anchor-heading">
|
|
<a class="anchor-link" id="quickstart_kafkaterminate" href="#quickstart_kafkaterminate"></a>
|
|
<a href="#quickstart_kafkaterminate">Step 8: Terminate the Kafka environment</a>
|
|
</h4>
|
|
|
|
<p>
|
|
Now that you reached the end of the quickstart, feel free to tear down the Kafka environment—or
|
|
continue playing around.
|
|
</p>
|
|
|
|
<ol>
|
|
<li>
|
|
Stop the producer and consumer clients with <code>Ctrl-C</code>, if you haven't done so already.
|
|
</li>
|
|
<li>
|
|
Stop the Kafka broker with <code>Ctrl-C</code>.
|
|
</li>
|
|
<li>
|
|
Lastly, stop the ZooKeeper server with <code>Ctrl-C</code>.
|
|
</li>
|
|
</ol>
|
|
|
|
<p>
|
|
If you also want to delete any data of your local Kafka environment including any events you have created
|
|
along the way, run the command:
|
|
</p>
|
|
|
|
<pre class="line-numbers"><code class="language-bash">$ rm -rf /tmp/kafka-logs /tmp/zookeeper</code></pre>
|
|
|
|
</div>
|
|
|
|
<div class="quickstart-step">
|
|
<h4 class="anchor-heading">
|
|
<a class="anchor-link" id="quickstart_kafkacongrats" href="#quickstart_kafkacongrats"></a>
|
|
<a href="#quickstart_kafkacongrats">Congratulations!</a>
|
|
</h4>
|
|
|
|
<p>You have successfully finished the Apache Kafka quickstart.<div>
|
|
|
|
<p>To learn more, we suggest the following next steps:</p>
|
|
|
|
<ul>
|
|
<li>
|
|
Read through the brief <a href="/intro">Introduction</a>
|
|
to learn how Kafka works at a high level, its main concepts, and how it compares to other
|
|
technologies. To understand Kafka in more detail, head over to the
|
|
<a href="/documentation/">Documentation</a>.
|
|
</li>
|
|
<li>
|
|
Browse through the <a href="/powered-by">Use Cases</a> to learn how
|
|
other users in our world-wide community are getting value out of Kafka.
|
|
</li>
|
|
<!--
|
|
<li>
|
|
Learn how _Kafka compares to other technologies_ you might be familiar with.
|
|
[note to design team: this new page is not yet written]
|
|
</li>
|
|
-->
|
|
<li>
|
|
Join a <a href="/events">local Kafka meetup group</a> and
|
|
<a href="https://kafka-summit.org/past-events/">watch talks from Kafka Summit</a>,
|
|
the main conference of the Kafka community.
|
|
</li>
|
|
</ul>
|
|
</div>
|
|
</script>
|
|
|
|
<div class="p-quickstart"></div>
|