-
Notifications
You must be signed in to change notification settings - Fork 0
Expand file tree
/
Copy pathREADME
More file actions
153 lines (124 loc) · 11.8 KB
/
README
File metadata and controls
153 lines (124 loc) · 11.8 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
Apache Kafka & process mining
Data: 12.02.2018
Autor: Tomasz Pawłowski
1. Ogólne
Projekt przedstawia zastosowanie oprogramowania ProM do analizy procesów
na podstawie strumienia logów. Podczas analizy logi, przekazywane przez
system apache kafka, dzielone są na paczki ze względu na daty z których
pochodzą. Każda paczka przekształcana jest do postaci XES, i analizowana
przy pomocy standardowych narzędzi programu ProM. Wyniki w postaci
plików xml i png przedstawiających sieci petriego procesu wysyłane są
do systemu kafka.
2. Składniki
Rozwiązanie składa się z:
- Skryptu kafka_start.sh konfigurującego i uruchamiającego kafkę
- Skryptu kafka_demo_producer.sh generującego przykładowe logi na podstawie
plików z katalogu example-logs
- Katalogu example-logs z przykładowymi logami
- Katalogu Workshop z projektem java'owym dokładającym do ProM-a plugin
obsługujący kafkę.
- Skryptu stream_processor.sh uruchamiającego projekt Workshop.
- Programu kafkapngconsumer zapisującego obrazki z kafki do wybranego katalogu
3. Wymagania
kafka_start.sh nie ma dodatkowych wymagań. Skrypt pobiera na początku działania
odpowiednią wersję kafki i uruchamia ją.
kafka_demo_producer.sh wymaga działającej kafki i zainstalowania programu
gnu-awk użytego do parsowania logów.
Projekt Workshop wymaga zainstalowania ant-a i doinstalowania do niego ivy.
W przypadku macOS wygląda to następująco:
brew install ant
ant_home="$(ant -diagnostics | grep -m1 'ant.home:' | cut -d' ' -f2)"
wget http://ftp.ps.pl/pub/apache//ant/ivy/2.4.0/apache-ivy-2.4.0-bin.tar.gz
tar -xzf apache-ivy-2.4.0-bin.tar.gz
cp apache-ivy-2.4.0/ivy-2.4.0.jar "$ant_home/lib/"
rm -r apache-ivy-2.4.0-bin.tar.gz apache-ivy-2.4.0
Po tej instalacji należy przejść do katalogu Workshop i zbudować projekt
poleceniem 'ant'.
4. Uruchomienie
Uruchomienie powinno odbywać się w kolejności:
- kafka_start.sh
- kafka_demo_producer.sh
- stream_processor.sh
4.1. kafka_start.sh
$ ./kafka_start.sh
Downloading kafka_2.11-1.0.0.tgz from http://www-eu.apache.org/dist/kafka/1.0.0/kafka_2.11-1.0.0.tgz done
Unpacking kafka_2.11-1.0.0.tgz done
Resetting data directory done
Starting ZooKeeper done with pid 21702
Starting Kafka done with pid 21932
Created topic "logs-input".
Created topic "logs-petri".
Created topic "logs-images".
Press Ctrl+C to stop the server
4.2. kafka_demo_producer.sh
Ten skrypt przyjmuje jeden parametr - opóźnienie w sekundach między kolejnymi
logami (wartość 0.2 oznacza opóźnienie 0.2s co linijkę logów, co przekłada się
na generowanie 4-5 linijek na sekundę).
$ ./kafka_demo_producer.sh 0.05
Generating log from file example-logs/L1.txt with date 2018-01-01 done
Generating log from file example-logs/L2.txt with date 2018-01-02 done
Generating log from file example-logs/L3.txt with date 2018-01-03 done
Generating log from file example-logs/L4.txt with date 2018-01-04
Logi generowane przez skrypt można podejrzeć następującą komendą:
$ ./kafka_2.11-1.0.0/bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic logs-input --property print.key=true --property key.separator=": "
Case1.2.0: 1514762783000,a
Case3.1.0: 1514766023000,a
Case2.2.0: 1514766525000,a
Case2.1.0: 1514768372000,a
Case1.1.0: 1514773571000,a
Case3.1.0: 1514775394000,e
Case1.2.0: 1514775923000,b
Case1.3.0: 1514776545000,a
Case1.3.0: 1514779072000,b
Case1.3.0: 1514781383000,c
Case2.1.0: 1514783626000,c
Case2.2.0: 1514783740000,c
Case1.3.0: 1514783795000,d
Case1.2.0: 1514788466000,c
4.3. stream_processor.sh
Działanie stream_processor.sh polega na uruchomieniu skryptu
Workshop/process_stream.txt. W wyniku logi z topiku logs-input
przetwarzane są na sieci w topiku logs-petri i logs-images.
W celach testowych process_stream.txt zapisuje też te pliki
do katalogu /tmp.
$ ./stream_processor.sh
>>> Scanning for plugins took 4.312 seconds
Starting script execution engine...
[-f, process_stream.txt]
initializing all plugins
...
Start plug-in Kafka XES
End plug-in Kafka XES, took 518 milliseconds
Start plug-in Kafka Producer
End plug-in Kafka Producer, took 1 milliseconds
Started collecting log for day 2018-01-01
Returning log for day 2018-01-01 next day started
Started collecting log for day 2018-01-02
Start plug-in Alpha Miner
End plug-in Alpha Miner, took 52 milliseconds
Start plug-in PNML export (Petri net)
End plug-in PNML export (Petri net), took 20 milliseconds
publishing petrinet 2018-01-01 to topic logs-petri
serialized petrinet to xml
Start plug-in Visualize Petri Net (Dot) [local]
End plug-in Visualize Petri Net (Dot) [local], took 1380 milliseconds
publishing image 2018-01-01 to topic logs-images
serialized image to byte array of size 33713
Returning log for day 2018-01-02 next day started
Started collecting log for day 2018-01-03
Wynikowe sieci petriego (w postaci xml/pnml) można podejrzeć komendą:
$ ./kafka_2.11-1.0.0/bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic logs-petri --property print.key=true --property key.separator=": "
2018-03-30: <?xml version="1.0" encoding="ISO-8859-1"?>
<pnml><net id="net1" type="http://www.pnml.org/version-2009/grammar/pnmlcoremodel"><name><text>Petri net (Alpha)</text></name><page id="n0"><name><text/></name><place id="n1"><name><text>([e],[f])</text></name><toolspecific tool="ProM" version="6.4" localNodeID="95db7511-6486-4897-b98c-e596cdc6bd72"/><graphics><position x="11.25" y="11.25"/><dimension x="12.5" y="12.5"/></graphics></place><place id="n2"><name><text>([c],[d])</text></name><toolspecific tool="ProM" version="6.4" localNodeID="46d9f62c-60ca-4db6-b37e-7b22cb110dff"/><graphics><position x="11.25" y="11.25"/><dimension x="12.5" y="12.5"/></graphics></place><place id="n3"><name><text>([a],[e])</text></name><toolspecific tool="ProM" version="6.4" localNodeID="a3f3d1a5-1400-4f80-9727-6174d26c438a"/><graphics><position x="11.25" y="11.25"/><dimension x="12.5" y="12.5"/></graphics></place><place id="n4"><name><text>([b],[c, f])</text></name><toolspecific tool="ProM" version="6.4" localNodeID="3b066aa8-6278-41bf-8b78-a57136d31a3e"/><graphics><position x="11.25" y="11.25"/><dimension x="12.5" y="12.5"/></graphics></place><place id="n5"><name><text>([a, d],[b])</text></name><toolspecific tool="ProM" version="6.4" localNodeID="f5b977e6-23ff-4b30-8b16-2f018a0db9aa"/><graphics><position x="11.25" y="11.25"/><dimension x="12.5" y="12.5"/></graphics></place><place id="n6"><name><text>Start</text></name><toolspecific tool="ProM" version="6.4" localNodeID="2a3a786a-8d88-45e6-9227-85d00557073a"/><graphics><position x="11.25" y="11.25"/><dimension x="12.5" y="12.5"/></graphics><initialMarking><text>1</text></initialMarking></place><place id="n7"><name><text>End</text></name><toolspecific tool="ProM" version="6.4" localNodeID="5469cb91-c9b6-4cd6-b877-7673ed76c1cc"/><graphics><position x="11.25" y="11.25"/><dimension x="12.5" y="12.5"/></graphics></place><transition id="n8"><name><text>a</text></name><toolspecific tool="ProM" version="6.4" activity="a" localNodeID="7e797985-9d26-4251-9d91-34c5d63a3c7c"/><graphics><position x="17.5" y="15.0"/><dimension x="25.0" y="20.0"/><fill color="#FFFFFF"/></graphics></transition><transition id="n9"><name><text>b</text></name><toolspecific tool="ProM" version="6.4" activity="b" localNodeID="cbacb69c-0311-476d-ab5b-5e41e0fe7918"/><graphics><position x="17.5" y="15.0"/><dimension x="25.0" y="20.0"/><fill color="#FFFFFF"/></graphics></transition><transition id="n10"><name><text>c</text></name><toolspecific tool="ProM" version="6.4" activity="c" localNodeID="707fc486-c1fb-4b7a-9ad5-2c00a85907e8"/><graphics><position x="17.5" y="15.0"/><dimension x="25.0" y="20.0"/><fill color="#FFFFFF"/></graphics></transition><transition id="n11"><name><text>d</text></name><toolspecific tool="ProM" version="6.4" activity="d" localNodeID="ab87096c-ad95-4b52-98c3-39be802618d7"/><graphics><position x="17.5" y="15.0"/><dimension x="25.0" y="20.0"/><fill color="#FFFFFF"/></graphics></transition><transition id="n12"><name><text>e</text></name><toolspecific tool="ProM" version="6.4" activity="e" localNodeID="373de05f-08ea-4486-b653-81ae7df97026"/><graphics><position x="17.5" y="15.0"/><dimension x="25.0" y="20.0"/><fill color="#FFFFFF"/></graphics></transition><transition id="n13"><name><text>f</text></name><toolspecific tool="ProM" version="6.4" activity="f" localNodeID="61260366-1564-4273-8726-631b83d3cade"/><graphics><position x="17.5" y="15.0"/><dimension x="25.0" y="20.0"/><fill color="#FFFFFF"/></graphics></transition><arc id="arc14" source="n9" target="n4"><name><text>1</text></name><toolspecific tool="ProM" version="6.4" localNodeID="484c44ed-5298-45bb-af63-fcbb819be817"/><arctype><text>normal</text></arctype></arc><arc id="arc15" source="n5" target="n9"><name><text>1</text></name><toolspecific tool="ProM" version="6.4" localNodeID="168fc027-8700-4cf6-81bb-39ffea027cf0"/><arctype><text>normal</text></arctype></arc><arc id="arc16" source="n4" target="n10"><name><text>1</text></name><toolspecific tool="ProM" version="6.4" localNodeID="93fb2f70-c4d6-4208-bddf-f84554b94740"/><arctype><text>normal</text></arctype></arc><arc id="arc17" source="n1" target="n13"><name><text>1</text></name><toolspecific tool="ProM" version="6.4" localNodeID="29c352d9-c97e-4a49-aeb9-553071b83bbc"/><arctype><text>normal</text></arctype></arc><arc id="arc18" source="n6" target="n8"><name><text>1</text></name><toolspecific tool="ProM" version="6.4" localNodeID="53b2d022-4d2c-4d57-ae63-4c59e2fbf3a9"/><arctype><text>normal</text></arctype></arc><arc id="arc19" source="n12" target="n1"><name><text>1</text></name><toolspecific tool="ProM" version="6.4" localNodeID="8d07cbef-7bc5-4f19-b258-d7bc7fe85098"/><arctype><text>normal</text></arctype></arc><arc id="arc20" source="n2" target="n11"><name><text>1</text></name><toolspecific tool="ProM" version="6.4" localNodeID="659b98a5-f714-4667-a66b-bcc8fdade6a2"/><arctype><text>normal</text></arctype></arc><arc id="arc21" source="n4" target="n13"><name><text>1</text></name><toolspecific tool="ProM" version="6.4" localNodeID="ca74c643-fc84-4c8a-8ce8-200b47b80c4c"/><arctype><text>normal</text></arctype></arc><arc id="arc22" source="n3" target="n12"><name><text>1</text></name><toolspecific tool="ProM" version="6.4" localNodeID="83edca12-ffdc-4200-9840-7a94398d13f0"/><arctype><text>normal</text></arctype></arc><arc id="arc23" source="n13" target="n7"><name><text>1</text></name><toolspecific tool="ProM" version="6.4" localNodeID="5c1e8b45-290b-488d-a2b4-35b25f178b9e"/><arctype><text>normal</text></arctype></arc><arc id="arc24" source="n8" target="n3"><name><text>1</text></name><toolspecific tool="ProM" version="6.4" localNodeID="dcfb8f5d-cb6b-45b6-a769-ba318349adca"/><arctype><text>normal</text></arctype></arc><arc id="arc25" source="n10" target="n2"><name><text>1</text></name><toolspecific tool="ProM" version="6.4" localNodeID="bc8f5a9a-d3dc-4822-80ea-0463f715f086"/><arctype><text>normal</text></arctype></arc><arc id="arc26" source="n11" target="n5"><name><text>1</text></name><toolspecific tool="ProM" version="6.4" localNodeID="2eed9d78-90a0-4da0-849e-accd277990d2"/><arctype><text>normal</text></arctype></arc><arc id="arc27" source="n8" target="n5"><name><text>1</text></name><toolspecific tool="ProM" version="6.4" localNodeID="02bf16c6-17ca-441a-aba7-b062cb3816da"/><arctype><text>normal</text></arctype></arc></page><finalmarkings/></net></pnml>
4.4. KafkaPngConsumer
Jest to prosty program którego jedynym celem jest nasłuchiwanie na topiku
i zapisywanie danych napływających danych w wybranym katalogu. Każda wiadomość
zachowywana jest w oddzielnym pliku w formacie KEY.png.
Kompilacja jest standardowa:
$ mvn clean compile assembly:single
Uruchomienie:
$ java -cp target/kafka-png-consumer-0.1-jar-with-dependencies.jar com.example.KafkaPngConsumer logs-images out 60000
Subscribing to topic logs-images
Wrote png image to file out/2018-01-09.png
Wrote png image to file out/2018-01-10.png
Wrote png image to file out/2018-01-11.png