如何在Kafka主题

时间:2019-02-04 22:05:37

标签: apache-kafka kafka-consumer-api

如何在Kafka Topic中找到消息的偏移量?偏移量包含多条消息还是一条消息?

2 个答案:

答案 0 :(得分:1)

偏移仅链接到主题的单个分区内的单个消息。

您可以在许多分区和许多主题中使用相同的偏移量,但是除非您明确要求生产者这样做,否则这些值之间几乎没有关联。

没有简单的方法来查找单个消息的偏移量。您需要扫描整个主题(或至少一个分区)

答案 1 :(得分:0)

假设您已经使用\documentclass{article} \usepackage[latin1]{inputenc} \usepackage{tikz} \usetikzlibrary{shapes,arrows} %%%< \usepackage{verbatim} \usepackage[active,tightpage]{preview} \PreviewEnvironment{tikzpicture} \setlength\PreviewBorder{5pt}% %%%> \begin{document} \pagestyle{empty} \tikzset { arrow/.style={ ->,>=stealth} } % Define block styles \tikzstyle{decision} = [diamond, draw, text width=4.5em, text badly centered, inner sep=0pt] \tikzstyle{block} = [rectangle, draw, text width=5em, text centered, rounded corners, minimum height=4em] \tikzstyle{block1} = [rectangle, draw, text width=6em, text centered, rounded corners, minimum height=4em] \tikzstyle{block2} = [rectangle, draw, text width=15em, text centered, rounded corners, minimum height=4em] \tikzstyle{block3} = [rectangle, draw, text width=20em, text centered, rounded corners, minimum height=4em] \tikzstyle{line} = [draw, -latex'] \begin{tikzpicture}[node distance = 2.5cm, auto] % Place nodes \node [block3] (Watermarked Image) {Watermarked Image}; \node[decision, above of =Watermarked Image,node distance=4cm ] (Difference between selected coefficients) {Difference between selected coefficients }; \node [block, left of=Difference between selected coefficients, node distance=3.5cm] (Inverse DCT for each block) {Inverse DCT for each block}; \node [block, right of=Difference between selected coefficients, node distance=3.5cm] (Modify AC coefficients) {Modify AC coefficients and perform Inverse DCT for each block}; \node [block2, above of=Difference between selected coefficients] (Genetic Algorithm) {Genetic Algorithm search DCT coefficients to find best option for insert watermark}; \node [block, above of=Genetic Algorithm] (DCT for each block) {DCT for each block}; \node [block, left of=DCT for each block] (Watermark Message) {Watermark Message}; \node [block, above of=DCT for each block] (8x8 Image Blocking) {8x8 Image Blocking}; \node [block, above of=Watermark Message] (Host Image) {Host Image}; % Draw edges \path [line] (Host Image) -- (8x8 Image Blocking); \path [line] (8x8 Image Blocking) -- (DCT for each block); \path [line] (DCT for each block) -- (Genetic Algorithm); %\path [line] (Watermark Message) -- (Genetic Algorithm); %\path [line] (Modify AC coefficients) -- (Inverse DCT for each block); %\path [line] (Inverse DCT for each block) -- (Watermarked Image); \path [line] (Genetic Algorithm) -- (Difference between selected coefficients); \path [line] (Difference between selected coefficients) -- node {yes} (Inverse DCT for each block); \path [line] (Difference between selected coefficients) -- node {no}(Modify AC coefficients); \coordinate (n2nl) at ([xshift=-2cm]Genetic Algorithm.north); \coordinate (n2nr) at ([xshift=-3cm]Watermarked Image.north); \coordinate (n2ns) at ([xshift=3cm]Watermarked Image.east); \draw[arrow] (Watermark Message.south -| n2nl) -- (n2nl); \draw[arrow] (Inverse DCT for each block.south -| n2nr) -- (n2nr); % \draw[arrow] (Modify AC coefficients.south -| n2ns) -- (n2ns); \draw[arrow] ([xshift=-0.5cm]Modify AC coefficients.south) -| ([xshift=3cm]Watermarked Image.north); \end{tikzpicture} \end{document} consume()阅读了该消息,并且对该消息的详细信息感兴趣,则可以使用以下代码找到相应的分区和偏移量: / p>

poll()

Kafka主题分为多个分区。 Kafka以循环方式在分区之间分配传入消息(除非您已指定要分区的密钥)。因此,对于尚未阅读的随机消息,可能必须扫描所有分区以找到消息,然后扫描相应的偏移量。