-
Notifications
You must be signed in to change notification settings - Fork 0
Expand file tree
/
Copy pathindex.html
More file actions
375 lines (369 loc) · 21.3 KB
/
index.html
File metadata and controls
375 lines (369 loc) · 21.3 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
<!doctype html>
<html class="default no-js">
<head>
<meta charset="utf-8">
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<title>@imqueue/core</title>
<meta name="description" content="">
<meta name="viewport" content="width=device-width, initial-scale=1">
<link rel="stylesheet" href="assets/css/main.css">
</head>
<body>
<header>
<div class="tsd-page-toolbar">
<div class="container">
<div class="table-wrap">
<div class="table-cell" id="tsd-search" data-index="assets/js/search.js" data-base=".">
<div class="field">
<label for="tsd-search-field" class="tsd-widget search no-caption">Search</label>
<input id="tsd-search-field" type="text" />
</div>
<ul class="results">
<li class="state loading">Preparing search index...</li>
<li class="state failure">The search index is not available</li>
</ul>
<a href="index.html" class="title">@imqueue/core</a>
</div>
<div class="table-cell" id="tsd-widgets">
<div id="tsd-filter">
<a href="#" class="tsd-widget options no-caption" data-toggle="options">Options</a>
<div class="tsd-filter-group">
<div class="tsd-select" id="tsd-filter-visibility">
<span class="tsd-select-label">All</span>
<ul class="tsd-select-list">
<li data-value="public">Public</li>
<li data-value="protected">Public/Protected</li>
<li data-value="private" class="selected">All</li>
</ul>
</div>
<input type="checkbox" id="tsd-filter-inherited" checked />
<label class="tsd-widget" for="tsd-filter-inherited">Inherited</label>
<input type="checkbox" id="tsd-filter-only-exported" />
<label class="tsd-widget" for="tsd-filter-only-exported">Only exported</label>
</div>
</div>
<a href="#" class="tsd-widget menu no-caption" data-toggle="menu">Menu</a>
</div>
</div>
</div>
</div>
<div class="tsd-page-title">
<div class="container">
<ul class="tsd-breadcrumb">
<li>
<a href="globals.html">Globals</a>
</li>
</ul>
<h1> @imqueue/core</h1>
</div>
</div>
</header>
<div class="container container-main">
<div class="row">
<div class="col-8 col-content">
<div class="tsd-panel tsd-typography">
<h1 id="i-message-queue-imqueue-core-">I Message Queue (@imqueue/core)</h1>
<p><a href="https://travis-ci.org/imqueue/core"><img src="https://travis-ci.org/imqueue/core.svg?branch=master" alt="Build Status"></a>
<a href="https://codebeat.co/projects/github-com-imqueue-core-master"><img src="https://codebeat.co/badges/b7685cb5-b290-47de-80e1-bde3e0582355" alt="codebeat badge"></a>
<a href="https://coveralls.io/github/imqueue/core?branch=master"><img src="https://coveralls.io/repos/github/imqueue/core/badge.svg?branch=master" alt="Coverage Status"></a>
<a href="https://david-dm.org/imqueue/core"><img src="https://img.shields.io/david/imqueue/core.svg" alt="David"></a>
<a href="https://david-dm.org/imqueue/core?type=dev"><img src="https://img.shields.io/david/dev/imqueue/core.svg" alt="David"></a>
<a href="https://snyk.io/test/github/imqueue/core?targetFile=package.json"><img src="https://snyk.io/test/github/imqueue/core/badge.svg?targetFile=package.json" alt="Known Vulnerabilities"></a>
<a href="https://rawgit.com/imqueue/core/master/LICENSE"><img src="https://img.shields.io/badge/license-ISC-blue.svg" alt="License"></a></p>
<p>Simple JSON-based messaging queue for inter service communication</p>
<p><strong>Related packages:</strong></p>
<ul>
<li><a href="https://github.com/imqueue/rpc">@imqueue/rpc</a> - RPC-like client/service
implementation over @imqueue/core.</li>
<li><a href="https://github.com/imqueue/cli">@imqueue/cli</a> - Command Line Interface
for imqueue.</li>
</ul>
<h1 id="features">Features</h1>
<p>With current implementation on RedisQueue:</p>
<ul>
<li><strong>Fast unwarranted</strong> message delivery (if consumer grab the message and dies
message will be lost). Up to ~35-40k of 1Kb messages per second on i7 core by
benchmarks.</li>
<li><strong>Fast warranted</strong> message delivery (only 1.5-2 times slower than
unwarranted). If consumer grab a message and dies it will be re-scheduled
in queue. Up to ~20-25K of 1Kb messages per second on i7 core by benchmarks.</li>
<li><strong>No timers or constant redis polling</strong> used for implementation, as result -
no delays in delivery and low CPU usage on application workers. When idling
it does not consume resources!</li>
<li><strong>Supports gzip compression for messages</strong> (decrease traffic usage, but
slower).</li>
<li><strong>Concurrent workers model supported</strong>, the same queue can have multiple
consumers.</li>
<li><strong>Delayed messages supported</strong>, fast as ~10K of 1Kb messages per second on i7
core by benchmarks.</li>
<li><strong>Safe predictable scaling of queues</strong>. Scaling number of workers does not
influence traffic usage.</li>
<li><strong>Round-robin message balancing between several redis instances</strong>. This
allows easy messaging queue redis horizontal scaling.</li>
<li><strong>TypeScript included!</strong></li>
</ul>
<h1 id="requirements">Requirements</h1>
<p>Currently this module have only one available adapter which is Redis server
related. So redis-server > 3.8+ is required.</p>
<p>If config command is disabled on redis it will be required to turn on manually
keyspace notification events (actual on use with ElasticCache on AWS), like:</p>
<pre><code><span class="hljs-attribute">notify-keyspace-events Ex</span></code></pre><p>Further, more adapters will be added... if needed.</p>
<h1 id="install">Install</h1>
<pre><code class="language-bash">npm i --save @imqueue/core</code></pre>
<h1 id="usage">Usage</h1>
<pre><code class="language-typescript"><span class="hljs-keyword">import</span> IMQ, { IMessageQueue, IJson } <span class="hljs-keyword">from</span> <span class="hljs-string">'@imqueue/core'</span>;
<span class="hljs-function">(<span class="hljs-params"><span class="hljs-keyword">async</span> (<span class="hljs-params"></span>) => {
<span class="hljs-keyword">const</span> queueOne: IMessageQueue = IMQ.create(<span class="hljs-params">'QueueOne'</span>);
<span class="hljs-keyword">const</span> queueTwo: IMessageQueue = IMQ.create(<span class="hljs-params">'QueueTwo'</span>);
<span class="hljs-comment">// start queues</span>
<span class="hljs-keyword">await</span> queueOne.start(<span class="hljs-params"></span>);
<span class="hljs-keyword">await</span> queueTwo.start(<span class="hljs-params"></span>);
<span class="hljs-comment">// handle queue messages</span>
queueOne.on(<span class="hljs-params">'message', (<span class="hljs-params">message: IJson, id: <span class="hljs-built_in">string</span>, fromQueue: <span class="hljs-built_in">string</span></span>) => {
<span class="hljs-built_in">console</span>.log(<span class="hljs-params">'queueOne message received:', message, id, fromQueue</span>);
<span class="hljs-keyword">if</span> (<span class="hljs-params">message.delay</span>) {
queueOne.destroy(<span class="hljs-params"></span>);
queueTwo.destroy(<span class="hljs-params"></span>);
}
}</span>);
queueTwo.on(<span class="hljs-params">'message', (<span class="hljs-params">message: IJson, id: <span class="hljs-built_in">string</span>, fromQueue: <span class="hljs-built_in">string</span></span>) => {
<span class="hljs-built_in">console</span>.log(<span class="hljs-params">'queueTwo message received:', message, id, fromQueue</span>);
}</span>);
<span class="hljs-comment">// sending queue messages</span>
<span class="hljs-keyword">await</span> queueOne.send(<span class="hljs-params">'QueueTwo', { hello: 'two' }</span>);
<span class="hljs-keyword">await</span> queueTwo.send(<span class="hljs-params">'QueueOne', { hello: 'one' }</span>);
<span class="hljs-comment">// sending delayed messages</span>
<span class="hljs-keyword">const</span> delay = 1000;
<span class="hljs-keyword">await</span> queueOne.send(<span class="hljs-params">'QueueOne', { delay }, delay</span>);
}</span>)<span class="hljs-params">()</span>;</span></code></pre>
<h1 id="benchmarking">Benchmarking</h1>
<p>First of all make sure redis-server is running on the localhost. Current
version of benchmark supposed to have redis running localhost because
it is going to measure it's CPU usage stats.</p>
<p>All workers during benchmark test will have their dedicated CPU affinity
to make sure collected stats as accurate as possible.</p>
<pre><code class="language-bash">git <span class="hljs-built_in">clone</span> git@github.com:Mikhus/core.git
<span class="hljs-built_in">cd</span> imq
node benchmark -c 4 -m 10000</code></pre>
<p>Other possible benchmark options:</p>
<pre><code>node benchmark -h
Options:
--version Show version number [boolean]
-h, --help Show help [boolean]
-c, --children Number of children test process <span class="hljs-keyword">to</span> fork
-d, --delay Number of milliseconds <span class="hljs-keyword">to</span> delay message delivery
<span class="hljs-keyword">for</span> delayed messages. By<span class="hljs-built_in"> default </span>delayed
messages is of <span class="hljs-keyword">and</span> this argument is equal <span class="hljs-keyword">to</span> 0.
-m, --messages Number of messages <span class="hljs-keyword">to</span> be sent by a child process
during test execution.
-z, --gzip Use gzip <span class="hljs-keyword">for</span> message encoding/decoding.[boolean]
-s, --safe Use safe (guaranteed) message delivery
algorithm. [boolean]
-e, --example-message Path <span class="hljs-keyword">to</span> a file containing JSON of example
message <span class="hljs-keyword">to</span> use during the tests.
-p, --port Redis<span class="hljs-built_in"> server port </span><span class="hljs-keyword">to</span> connect <span class="hljs-keyword">to</span>.
-t, --message-multiply-times Increase sample message data given number of
times.</code></pre><p>Number of child workers running message queues are limited to a max number
of CPU in the system -2. First one, which is CPU0 is reserved for OS tasks and
for stats collector process. Second, which is CPU1 is dedicated to redis
process running on a local machine, All others are safe to run queue workers.</p>
<p>For example, if there is 8 cores on a machine it is safe to run up to 6 workers.
For 4-core machine this number is limited to 2.
If there is less cores results will not give good visibility of load.</p>
<p><em>NOTE: paragraphs above this note are Linux-only related! On MacOS
there is no good way to set process affinity to a CPU core, Windows support is
not tested and is missing for the moment in benchmarking. I does not mean
benchmark will not work on Mac & Win but the results won't be accurate and
predictable.</em></p>
<h2 id="running-unit-tests">Running Unit Tests</h2>
<pre><code class="language-bash">git <span class="hljs-built_in">clone</span> git@github.com:imqueue/core.git
<span class="hljs-built_in">cd</span> imq
npm <span class="hljs-built_in">test</span></code></pre>
<h2 id="license">License</h2>
<p><a href="https://rawgit.com/imqueue/core/master/LICENSE">ISC</a></p>
</div>
</div>
<div class="col-4 col-menu menu-sticky-wrap menu-highlight">
<nav class="tsd-navigation primary">
<ul>
<li class="globals ">
<a href="globals.html"><em>Globals</em></a>
</li>
</ul>
</nav>
<nav class="tsd-navigation secondary menu-sticky">
<ul class="before-current">
<li class=" tsd-kind-class">
<a href="classes/clusteredredisqueue.html" class="tsd-kind-icon">Clustered<wbr>Redis<wbr>Queue</a>
</li>
<li class=" tsd-kind-class">
<a href="classes/imq.html" class="tsd-kind-icon">IMQ</a>
</li>
<li class=" tsd-kind-class">
<a href="classes/redisqueue.html" class="tsd-kind-icon">Redis<wbr>Queue</a>
</li>
<li class=" tsd-kind-interface">
<a href="interfaces/debuginfooptions.html" class="tsd-kind-icon">Debug<wbr>Info<wbr>Options</a>
</li>
<li class=" tsd-kind-interface">
<a href="interfaces/ijson.html" class="tsd-kind-icon">IJson</a>
</li>
<li class=" tsd-kind-interface">
<a href="interfaces/ilogger.html" class="tsd-kind-icon">ILogger</a>
</li>
<li class=" tsd-kind-interface">
<a href="interfaces/imqoptions.html" class="tsd-kind-icon">IMQOptions</a>
</li>
<li class=" tsd-kind-interface">
<a href="interfaces/imessage.html" class="tsd-kind-icon">IMessage</a>
</li>
<li class=" tsd-kind-interface">
<a href="interfaces/imessagequeue.html" class="tsd-kind-icon">IMessage<wbr>Queue</a>
</li>
<li class=" tsd-kind-interface">
<a href="interfaces/imessagequeueconstructor.html" class="tsd-kind-icon">IMessage<wbr>Queue<wbr>Constructor</a>
</li>
<li class=" tsd-kind-interface">
<a href="interfaces/imulti.html" class="tsd-kind-icon">IMulti</a>
</li>
<li class=" tsd-kind-interface">
<a href="interfaces/iredisclient.html" class="tsd-kind-icon">IRedis<wbr>Client</a>
</li>
<li class=" tsd-kind-type-alias">
<a href="globals.html#allowedtimeformat" class="tsd-kind-icon">Allowed<wbr>Time<wbr>Format</a>
</li>
<li class=" tsd-kind-variable">
<a href="globals.html#imq_log_args" class="tsd-kind-icon">IMQ_<wbr>LOG_<wbr>ARGS</a>
</li>
<li class=" tsd-kind-variable">
<a href="globals.html#imq_log_time" class="tsd-kind-icon">IMQ_<wbr>LOG_<wbr>TIME</a>
</li>
<li class=" tsd-kind-variable">
<a href="globals.html#imq_log_time_format" class="tsd-kind-icon">IMQ_<wbr>LOG_<wbr>TIME_<wbr>FORMAT</a>
</li>
<li class=" tsd-kind-variable tsd-is-not-exported">
<a href="globals.html#rx_client_clean" class="tsd-kind-icon">RX_<wbr>CLIENT_<wbr>CLEAN</a>
</li>
<li class=" tsd-kind-variable tsd-is-not-exported">
<a href="globals.html#rx_client_name" class="tsd-kind-icon">RX_<wbr>CLIENT_<wbr>NAME</a>
</li>
<li class=" tsd-kind-variable tsd-is-not-exported">
<a href="globals.html#rx_client_test" class="tsd-kind-icon">RX_<wbr>CLIENT_<wbr>TEST</a>
</li>
<li class=" tsd-kind-variable tsd-is-not-exported">
<a href="globals.html#commands" class="tsd-kind-icon">commands</a>
</li>
<li class=" tsd-kind-variable tsd-is-not-exported">
<a href="globals.html#lookuptable" class="tsd-kind-icon">lookup<wbr>Table</a>
</li>
<li class=" tsd-kind-variable tsd-is-not-exported">
<a href="globals.html#rand" class="tsd-kind-icon">rand</a>
</li>
<li class=" tsd-kind-function tsd-has-type-parameter">
<a href="globals.html#buildoptions" class="tsd-kind-icon">build<wbr>Options</a>
</li>
<li class=" tsd-kind-function">
<a href="globals.html#intrand" class="tsd-kind-icon">intrand</a>
</li>
<li class=" tsd-kind-function">
<a href="globals.html#logdebuginfo" class="tsd-kind-icon">log<wbr>Debug<wbr>Info</a>
</li>
<li class=" tsd-kind-function tsd-is-not-exported">
<a href="globals.html#makecallback" class="tsd-kind-icon">make<wbr>Callback</a>
</li>
<li class=" tsd-kind-function tsd-is-not-exported">
<a href="globals.html#makepromised" class="tsd-kind-icon">make<wbr>Promised</a>
</li>
<li class=" tsd-kind-function">
<a href="globals.html#pack" class="tsd-kind-icon">pack</a>
</li>
<li class=" tsd-kind-function">
<a href="globals.html#profile" class="tsd-kind-icon">profile</a>
</li>
<li class=" tsd-kind-function">
<a href="globals.html#promisify" class="tsd-kind-icon">promisify</a>
</li>
<li class=" tsd-kind-function">
<a href="globals.html#propertiesof" class="tsd-kind-icon">properties<wbr>Of</a>
</li>
<li class=" tsd-kind-function">
<a href="globals.html#sha1" class="tsd-kind-icon">sha1</a>
</li>
<li class=" tsd-kind-function">
<a href="globals.html#unpack" class="tsd-kind-icon">unpack</a>
</li>
<li class=" tsd-kind-function">
<a href="globals.html#uuid" class="tsd-kind-icon">uuid</a>
</li>
<li class=" tsd-kind-object-literal">
<a href="globals.html#default_imq_options" class="tsd-kind-icon">DEFAULT_<wbr>IMQ_<wbr>OPTIONS</a>
</li>
</ul>
</nav>
</div>
</div>
</div>
<footer>
<div class="container">
<h2>Legend</h2>
<div class="tsd-legend-group">
<ul class="tsd-legend">
<li class="tsd-kind-module"><span class="tsd-kind-icon">Module</span></li>
<li class="tsd-kind-object-literal"><span class="tsd-kind-icon">Object literal</span></li>
<li class="tsd-kind-variable"><span class="tsd-kind-icon">Variable</span></li>
<li class="tsd-kind-function"><span class="tsd-kind-icon">Function</span></li>
<li class="tsd-kind-function tsd-has-type-parameter"><span class="tsd-kind-icon">Function with type parameter</span></li>
<li class="tsd-kind-index-signature"><span class="tsd-kind-icon">Index signature</span></li>
<li class="tsd-kind-type-alias"><span class="tsd-kind-icon">Type alias</span></li>
</ul>
<ul class="tsd-legend">
<li class="tsd-kind-enum"><span class="tsd-kind-icon">Enumeration</span></li>
<li class="tsd-kind-enum-member"><span class="tsd-kind-icon">Enumeration member</span></li>
<li class="tsd-kind-property tsd-parent-kind-enum"><span class="tsd-kind-icon">Property</span></li>
<li class="tsd-kind-method tsd-parent-kind-enum"><span class="tsd-kind-icon">Method</span></li>
</ul>
<ul class="tsd-legend">
<li class="tsd-kind-interface"><span class="tsd-kind-icon">Interface</span></li>
<li class="tsd-kind-interface tsd-has-type-parameter"><span class="tsd-kind-icon">Interface with type parameter</span></li>
<li class="tsd-kind-constructor tsd-parent-kind-interface"><span class="tsd-kind-icon">Constructor</span></li>
<li class="tsd-kind-property tsd-parent-kind-interface"><span class="tsd-kind-icon">Property</span></li>
<li class="tsd-kind-method tsd-parent-kind-interface"><span class="tsd-kind-icon">Method</span></li>
<li class="tsd-kind-index-signature tsd-parent-kind-interface"><span class="tsd-kind-icon">Index signature</span></li>
</ul>
<ul class="tsd-legend">
<li class="tsd-kind-class"><span class="tsd-kind-icon">Class</span></li>
<li class="tsd-kind-class tsd-has-type-parameter"><span class="tsd-kind-icon">Class with type parameter</span></li>
<li class="tsd-kind-constructor tsd-parent-kind-class"><span class="tsd-kind-icon">Constructor</span></li>
<li class="tsd-kind-property tsd-parent-kind-class"><span class="tsd-kind-icon">Property</span></li>
<li class="tsd-kind-method tsd-parent-kind-class"><span class="tsd-kind-icon">Method</span></li>
<li class="tsd-kind-accessor tsd-parent-kind-class"><span class="tsd-kind-icon">Accessor</span></li>
<li class="tsd-kind-index-signature tsd-parent-kind-class"><span class="tsd-kind-icon">Index signature</span></li>
</ul>
<ul class="tsd-legend">
<li class="tsd-kind-constructor tsd-parent-kind-class tsd-is-inherited"><span class="tsd-kind-icon">Inherited constructor</span></li>
<li class="tsd-kind-property tsd-parent-kind-class tsd-is-inherited"><span class="tsd-kind-icon">Inherited property</span></li>
<li class="tsd-kind-method tsd-parent-kind-class tsd-is-inherited"><span class="tsd-kind-icon">Inherited method</span></li>
<li class="tsd-kind-accessor tsd-parent-kind-class tsd-is-inherited"><span class="tsd-kind-icon">Inherited accessor</span></li>
</ul>
<ul class="tsd-legend">
<li class="tsd-kind-property tsd-parent-kind-class tsd-is-protected"><span class="tsd-kind-icon">Protected property</span></li>
<li class="tsd-kind-method tsd-parent-kind-class tsd-is-protected"><span class="tsd-kind-icon">Protected method</span></li>
<li class="tsd-kind-accessor tsd-parent-kind-class tsd-is-protected"><span class="tsd-kind-icon">Protected accessor</span></li>
</ul>
<ul class="tsd-legend">
<li class="tsd-kind-property tsd-parent-kind-class tsd-is-private"><span class="tsd-kind-icon">Private property</span></li>
<li class="tsd-kind-method tsd-parent-kind-class tsd-is-private"><span class="tsd-kind-icon">Private method</span></li>
<li class="tsd-kind-accessor tsd-parent-kind-class tsd-is-private"><span class="tsd-kind-icon">Private accessor</span></li>
</ul>
<ul class="tsd-legend">
<li class="tsd-kind-property tsd-parent-kind-class tsd-is-static"><span class="tsd-kind-icon">Static property</span></li>
<li class="tsd-kind-call-signature tsd-parent-kind-class tsd-is-static"><span class="tsd-kind-icon">Static method</span></li>
</ul>
</div>
</div>
</footer>
<div class="overlay"></div>
<script src="assets/js/main.js"></script>
<script>if (location.protocol == 'file:') document.write('<script src="assets/js/search.js"><' + '/script>');</script>
</body>
</html>