-
Notifications
You must be signed in to change notification settings - Fork 1
Expand file tree
/
Copy pathindex.html
More file actions
1092 lines (1006 loc) · 48.6 KB
/
index.html
File metadata and controls
1092 lines (1006 loc) · 48.6 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
<!DOCTYPE html>
<html lang="en-US">
<head>
<meta charset="UTF-8" />
<meta name="title" property="og:title" content="BridgeAPI" />
<meta
name="description"
property="og:description"
content="BridgeAPI is a serverless integration platform that
lets you connect your apps through event-driven workflows"
/>
<meta name="type" property="og:type" content="website" />
<meta
name="url"
property="og:url"
content="https://github.com/williampj/bridgeapi.js"
/>
<meta name="viewport" content="width=device-width, initial-scale=1" />
<meta
name="author"
content="Andi Crotwell, William Jackson, Angel Ruiz-Bates"
/>
<title>BridgeAPI Case Study</title>
<link
href="https://fonts.googleapis.com/css?family=Roboto+Mono&display=swap"
rel="stylesheet"
/>
<link
href="https://fonts.googleapis.com/css?family=Roboto:400,500&display=swap"
rel="stylesheet"
/>
<link
href="https://fonts.googleapis.com/css?family=Lato&display=swap"
rel="stylesheet"
/>
<!-- <style></style> -->
<link rel="stylesheet" href="stylesheets/bootstrap.min.css" />
<link rel="stylesheet" href="stylesheets/main.css" />
<!-- <script></script> -->
<script src="javascripts/main.js"></script>
</head>
<body class="d-flex">
<div class="logo-links">
<p id="burger-menu">
<img src="./images/icons/menu.svg" width="35" />
</p>
<a href="https://github.com/bridgeapi-dev" target="_blank">
<img
src="./images/logos/github_black.png"
alt="github logo"
id="github-logo"
/>
</a>
</div>
<!-- MAIN NAV -->
<nav id="site-navigation" style="background-color: white; display: none">
<ul class="reset-ul">
<div class="top-nav">
<li class="no-bullet">
<a href="https://bridgeapi-dev.github.io" class="main-nav-link"
>HOME</a
>
</li>
<li class="no-bullet">
<a href="#team" class="main-nav-link">TEAM</a>
</li>
<li class="no-bullet">
<a
href="https://bridgeapi.dev"
class="main-nav-link"
target="_blank"
>TRY NOW</a
>
</li>
</div>
<li id="mobile-nav-li" class="no-bullet">
<nav id="case-study-mobile">
<ul>
<li class="mb-2">
<a href="#introduction" class="case-study-anchor">
INTRODUCTION
</a>
</li>
<li class="mb-2">
<a href="#user-experience" class="case-study-anchor">
USER EXPERIENCE
</a>
</li>
<li class="mb-2">
<a href="#frontend-architecture" class="case-study-anchor">
FRONTEND ARCHITECTURE
</a>
</li>
<li class="mb-2">
<a href="#system-architecture" class="case-study-anchor">
SYSTEM ARCHITECTURE
</a>
</li>
<li class="mb-2">
<a href="#cloud-hosting" class="case-study-anchor">
CLOUD HOSTING
</a>
</li>
<li class="mb-2">
<a href="#future-work" class="case-study-anchor">
FUTURE WORK
</a>
</li>
<li class="mb-2">
<a href="#team" class="case-study-anchor"> THE TEAM </a>
</li>
<li class="mb-2">
<a href="#references" class="case-study-anchor"> REFERENCES </a>
</li>
</ul>
</nav>
</li>
</ul>
</nav>
<!-- END OF MAIN NAV -->
<div class="h-100">
<!-- Cover -->
<section
class="d-flex w-100 mx-auto flex-column text-center text-white bg-dark"
>
<div class="wrapper">
<div
id="cover"
class="d-flex flex-column justify-content-center page-main"
>
<div class="cover-logo">
<img src="./images/logos/logo.svg" class="mw-100" />
</div>
<p class="lead d-flex align-self-center mt-5">
BridgeAPI is a serverless integration platform that lets you
connect your apps through event-driven workflows.
</p>
</div>
</div>
</section>
<!-- End of Cover -->
<!-- Section something -->
<section class="text-center" style="color: #184d47">
<div style="background-color: white">
<div class="row py-2" style="max-width: 100%">
<div class="col-md-6 col-sm-12 p-5">
<img
src="./images/first.png"
class="img-fluid border-radius-10"
/>
</div>
<div class="col-md-6 col-sm-12 px-5 pt-1 pb-5">
<h1 class="my-4">Connect Nearly Any App</h1>
<p class="mx-3" style="font-size: 1.5rem">
BridgeAPI empowers you to easily bridge two incompatible apps -
without the need for a separate server
</p>
</div>
</div>
</div>
</section>
<!-- End of Section something -->
<!-- Section something -->
<section class="text-center text-white">
<div style="background-color: #96bb7c">
<div class="row py-2" style="max-width: 100%; --bs-gutter-x: 0">
<div class="col-md-6 col-sm-12 p-5 py-2">
<h1 class="my-4">Easy Setup. Unlimited Possibilities</h1>
<p style="font-size: 1.5rem">
Behind most services is an API. Having the ability to
effortlessly bridge them unlocks unlimited possibilities
</p>
</div>
<div class="col-md-6 col-sm-12 p-4">
<img
src="./images/create bridge.gif"
class="img-fluid border-radius-10"
/>
</div>
</div>
</div>
</section>
<!-- End of Section something -->
<!-- Main Body -->
<section id="main">
<div class="container mt-4" style="margin-left: 0; margin-right: 0">
<div class="row">
<!-- TODO: Mobile size -->
<div class="col-md-3">
<nav
id="case-study-nav"
class="sticky-top top-25"
style="opacity: 0"
>
<ul>
<li class="mb-2">
<a href="#introduction" class="case-study-anchor">
INTRODUCTION
</a>
</li>
<li class="mb-2">
<a href="#user-experience" class="case-study-anchor">
USER EXPERIENCE
</a>
</li>
<li class="mb-2">
<a href="#frontend-architecture" class="case-study-anchor">
FRONTEND ARCHITECTURE
</a>
</li>
<li class="mb-2">
<a href="#system-architecture" class="case-study-anchor">
SYSTEM ARCHITECTURE
</a>
</li>
<li class="mb-2">
<a href="#cloud-hosting" class="case-study-anchor">
CLOUD HOSTING
</a>
</li>
<li class="mb-2">
<a href="#future-work" class="case-study-anchor">
FUTURE WORK
</a>
</li>
<li class="mb-2">
<a href="#team" class="case-study-anchor"> THE TEAM </a>
</li>
<li class="mb-2">
<a href="#references" class="case-study-anchor">
REFERENCES
</a>
</li>
</ul>
<div style="padding-left: 2rem; padding-top: 2rem">
<a
href="https://bridgeapi.dev"
class="try-now"
target="_blank"
>Try BridgeAPI</a
>
</div>
</nav>
</div>
<!-- CASE STUDY -->
<div id="main-case-study" class="col-md-9 col-sm-12">
<h1 id="case-study">Case Study</h1>
<!-- 1. INTRODUCTION -->
<section id="introduction">
<h1 class="case-study-h1">1. Introduction</h1>
<h2 class="case-study-h2">1.1 BridgeAPI</h2>
<p>
BridgeAPI is a free, open-source web developer tool that
receives, filters, and forwards HTTP requests. This
combination allows developers to set up event-driven workflows
in a few simple steps with BridgeAPI as the bridge between
apps.
</p>
<div class="mx-auto w-75">
<img
src="./images/event-driven workflow.svg"
class="img-fluid border-radius-10 my-4"
/>
</div>
<h2 class="case-study-h2">1.2 The API Economy</h2>
<p>
Propelled by the rise in microservices, cloud computing, the
Internet of Things, and the general benefits of integrating
diverse business applications, the API space has grown
exponentially since the mid-2000s. ProgrammableWeb, a
directory that lists public web-based APIs, documented growth
of 2000 APIs yearly to more than 23,000 by 2019.
<sup>[1]</sup>
Postman, the most used developer API tool, registered a
ten-fold rise in API request folders, from 3.1 to 34.9 million
between 2017 and 2020.
<sup>[2]</sup>
The integral role API requests play in the developing API
economy demands developer tools to send, receive, monitor, and
chain API requests.
</p>
<h2 class="case-study-h2">1.3 Current Tools</h2>
<p>
Among developer tools used to <i>send</i> API requests,
Postman is by far the most utilized. It lets users select the
request type and endpoint and a host of optional settings,
including parameters, headers, and body. It also allows for
pre-request scripts, whereby a request can be delayed, or more
advanced features like regularly scheduled requests. The
response is subsequently rendered with corresponding status,
time, size, and body, among other data. Postman is very
intuitive to use and practically covers every conceivable
aspect of an API request. Yet, it is limited in so far that
any workflow involving Postman must originate with a Postman
request - making it more useful as a testing tool for an API
interface. Any need to integrate workflows triggered by
webhooks will, therefore, require other tools.
</p>
<p>
There are also practical tools to <i>receive</i> API requests.
Hookbin.com is one such tool that provides a URL endpoint to
receive webhooks, which can then be inspected upon reception.
Users can configure an automatic response that can be
hardcoded or include data from the incoming request in the
response.
</p>
<p>
Lastly, some apps provide for workflows by providing URL
endpoints that, when pinged, trigger a user-defined workflow.
Zapier, IFTTT, and Pipedream are examples within this
category. These fully-fledged sites are built to integrate
large numbers of predefined apps, which are selected with a
click and the granting of permission to access the external
app from the workflow site. With this access granted, any
number of events (not merely webhooks) within the external app
can trigger the user-defined workflow. These sites are
therefore popular with retail users since the intricacies of
API requests are abstracted away. Furthermore, these sites are
proprietary, involving a limited free tier and scaled paid
tiers.
</p>
<img
src="./images/comparison table.svg"
class="img-fluid border-radius-10 my-4"
/>
<h2 class="case-study-h2">1.4 BridgeAPI Niche</h2>
<p>
Within this mapping of the landscape of online API tools, a
gap emerges between the pure developer tools and the retail
workflow tools. We view BridgeAPI as an attempt to occupy this
space as an open-source developer tool to establish workflows.
It maintains the simplicity of the purely sending or receiving
API developer tools while combining the functionality of both.
Similar to the developer tools, BridgeAPI requires no
permissions but entrusts the user with configuring the
endpoint, headers, and body of an API request. As a
functioning workflow tool, BridgeAPI can act as a switchboard
to monitor all API-driven single-step workflows between two
apps. This ambition framed our user interface as we sought to
create an intuitive and simple tool, in accordance with the
KISS principle, that nonetheless contains the features
necessary to fulfill a developer’s workflow needs, all while
being as unintrusive as possible.
</p>
</section>
<!-- End of 1. INTRODUCTION -->
<!-- 2. USER EXPERIENCE -->
<section id="user-experience">
<h1 class="case-study-h1">2. User Experience</h1>
<h2 class="case-study-h2">2.1 Headers and Settings</h2>
<p>
The most demanding aspect of the app from the user’s
perspective is the workflow configuration, which we refer to
as a <i>bridge</i>. The ideal we strived towards was that of a
self-explanatory interface, where the headings of each form
and field sufficiently guide the user through the setup.
</p>
<img
src="./images/url_and_features.png"
class="mx-auto img-fluid mb-3 border-radius-10"
/>
<p>
In addition to providing a BridgeAPI endpoint, the outbound
(forwarding) URL endpoint (1) and method dropdown (2) fields
constitute the minimum viable settings for the workflow to
function. While any HTTP request can be selected, we set the
default to POST as we envision forwarding data in a request
body as the most frequent function of a bridge.
</p>
<p>
The user can set any number of headers (3), but BridgeAPI is
opinionated regarding the content type, which is hardcoded to
application/json. The following chart from
<i>ProgrammableWeb</i> between March 2018 and March 2020 shows
the preponderance of JSON and query strings as the standard
data formats for API requests, each accounting for more than
ten times the number of APIs accepting XML as the data format.
Therefore, we chose not to add XML support for the first
iteration of BridgeAPI, given the added complexity required
for what would likely be limited use cases.
</p>
<img
src="./images/api_stat.png"
class="mb-3 w-50 mx-auto d-block border-radius-10"
/>
<p>
Delays (4) and retries (5) are the two optional ‘extras’ in
the bridge configuration. A bridge allows for 0, 1, 3, and 5
retries, where a retry is triggered when a response to an
outbound request is unsuccessful. We define an unsuccessful
request as any request that is either incomplete due to an
error or that completed but with a response code equal to or
greater than 300. While redirect responses (300s) could be
defined as successful, we chose to handle them as unsuccessful
to account for temporary redirects, whereby a later retry
could conceivably generate a response code in the 200s. The
reason we included the retry feature was to let the user set
the importance of completing the outbound request so that five
retries could be set for consequential bridges or in cases
where the receiving API is less reliable.
</p>
<p>
The following code snippet represents the formula for
exponential backoff between each retry:
<code>
(retry_count ** 4) + 15 + (rand(30) * (retry_count + 1))
</code>
<br />
With 5 retries, the attempts will be spread out over nearly 11
minutes.
<img
src="./images/sidekiq delay.svg"
class="img-fluid border-radius-10"
/>
</p>
<p>
In addition, users can decide between an instant forwarding of
the request or manually setting a delay of 15 minutes, 30
minutes, 1 hour, or 1 day before the first request attempt. We
decided it would be logical for certain workflows to include a
delay between links in a workflow chain. For example, if a
user wanted to snooze all forwarding requests of a bridge for
a day, the 1 day delay option would ensure that all incoming
requests are received and held until the following day.
Another advantage is that it gives the user time to abort or
deactivate the bridge if circumstances during the delay period
call for it.
</p>
<p>
In order to include sensitive information in the body or
headers of an outbound request, we allow the user to store
environmental variables.
</p>
<img
src="./images/env_variables.png"
class="mx-auto img-fluid mb-3 border-radius-10"
/>
<p>
Environmental variables are scoped to the bridge rather than
at the user level for a couple of reasons. Firstly, there is a
high likelihood of variable name collision at the user level
for repeating keys such as “password.” Secondly, by defining
and inspecting environmental variables on the same form where
they are applied, we can better adhere to our straightforward
design philosophy and goal of user-friendliness by alleviating
the need to navigate to a configuration page to set or
recollect them. While it could be slightly inconvenient to
duplicate environmental variables for different bridges, we
accept this tradeoff as they’re only expected to be set
infrequently, typically only once when defining a bridge.
</p>
<img
src="./images/headers.png"
class="mx-auto img-fluid mb-3 border-radius-10"
/>
<p>
To best protect the data, an environmental variable’s values
will be encrypted and filtered for the UI upon any save
action. They cannot be viewed thereafter, leaving users with
the option of either referencing, deleting, or reassigning
them. For reasons of brevity and familiarity, we selected
<code>$env</code> as the name for referencing the
environmental variable namespace, which users can do either
from the header fields or in the payload. If a “JWT-secret”
key were stored as an environmental variable, it would thus be
accessible using the <code>$env.JWT-secret</code> syntax.
</p>
<img
src="./images/encrypt.gif"
class="img-fluid border-radius-10 my-4"
/>
<h2 class="case-study-h2">2.2 Payload</h2>
<p>
GET requests will be useful when users are more interested in
assessing responses than in forwarding data. Yet, most bridges
are expected to be POST requests and thus require a defined
payload in the request body.
</p>
<img
src="./images/payload.png"
class="mx-auto img-fluid mb-3 border-radius-10"
/>
<p>
To aid the user, we set a default payload to be edited or
tested as is, along with a JSON linter. The
<code>$payload</code> syntax accesses the inbound request
payload, so in the above example,
<code>$payload.message</code> accesses a
<code>message</code> key at the highest level of the inbound
request payload, while
<code>$payload.nested.message</code> plucks the value
referenced by the <code>message</code> key of the
<code>nested</code> object. As in the header fields,
<code>$env</code> syntax could likewise be applied to
reference environmental variables within the payload.
</p>
<h2 class="case-study-h2">2.3 Actions</h2>
<p>
In order to operate as a switchboard for event-driven
workflows, users need the option of configuring bridges after
saving and activating them. Defining and saving a bridge
automatically activates it, whereafter it listens for inbound
requests to its endpoint.
</p>
<img
src="./images/actions.png"
class="mx-auto img-fluid mb-3 border-radius-10"
/>
<p>
Actions to the bridge are all collected within an “Actions”
modal one click away. We provide the option of deleting the
bridge entirely, toggling the active status (activate and
deactivate), or aborting current ongoing requests in the jobs
queue - which is made up of requests being delayed, retried,
or that still need to be executed. This feature set is simple,
yet paired with its adjacent save button for editing, it
exhausts all essential configuration needs of a defined
bridge.
</p>
<p>
With this single page, all the necessary fields are provided
to define a workflow between two apps. The interface is
simple, intuitive, and focused on the essential features, with
a couple of add-ons (delays and retries) that provide useful
options to the user without adding significant complexity.
</p>
</section>
<!-- End of 2. USER EXPERIENCE -->
<!-- 3. FRONTEND ARCHITECTURE -->
<section id="frontend-architecture">
<h1 class="case-study-h1">3. Frontend Architecture</h1>
<h2 class="case-study-h2">3.1 NextJS</h2>
<p>
We selected NextJS as the optimal framework for building out
our React frontend because it leverages the benefits of
client-side rendered (CSR) apps, such as responsiveness to
user activity and decreased server load, as well as benefits
of server-side rendered (SSR) apps such as search engine
optimization (SEO) and fast initial page loads. Applying
NextJS, our initial page load is server-side rendered while
assets and pre-rendered HTML for linked pages are loaded on
the client, offering users the experience of a single-page app
(SPA).
</p>
<h2 class="case-study-h2">3.2 Payload Editor</h2>
<p>
Even though the editor only requires JSON support, we decided
to use an editor that offers multi-language support to ease
future extensions to other languages. We chose CodeMirror, a
text editor written in JavaScript for the browser, because it
is lighter weight than alternative options such as Ace and
Monaco, which makes it a better fit for our use cases of
rendering multiple code editors on a single page with
different purposes.
</p>
<p>
The versatility of CodeMirror furthermore lets us configure it
as read-only for inspecting past events yet editable for users
configuring a bridge, as well as interfacing it with a form
component when needed. Note that CodeMirror does not offer
support for Server-Side Rendering (SSR), but as we employ
Client Side Rendering (CSR), we can retrieve editor state
asynchronously using React’s UseEffect hook in order to avoid
slowing down page loads. To improve the user experience, we
added a loading spinner displayed in place of each editor
until that editor is populated.
</p>
<img
src="./images/editor.png"
class="mx-auto img-fluid mb-3 border-radius-10"
/>
<p>
The result is an intuitive editor containing aids that users
have come to expect from editors like VSCode, including a
full-screen mode for expansive payload editing, syntax
highlighting, JSON linting, auto-indent, and active line
highlighter.
</p>
</section>
<!-- End of 3. FRONTEND ARCHITECTURE -->
<!-- 4. SYSTEM ARCHITECTURE -->
<section id="system-architecture">
<h1 class="case-study-h1">4. System Architecture</h1>
<h2 class="case-study-h2">4.1 Background process</h2>
<p>
We needed a way to process events concurrently without holding
up the single-threaded server, and for this, a background
processor was required. While Rails ships with ActiveJob, we
chose not to use this built-in processor since it enqueues
jobs in RAM, causing jobs to be cleared if the process is
interrupted or if the server is reset.
</p>
<p>
We're using Sidekiq instead, which runs concurrently and uses
threads to handle jobs simultaneously in the same process.
Another reason for choosing Sidekiq is performance, estimated
to be up to twenty times faster than alternative background
processors. In addition, Sidekiq ships with a monitoring
dashboard that lets us view jobs in its queue, ongoing jobs,
and other information useful to debugging - a feature that
allowed us to develop our business logic quicker than if we
were without the dashboard or had to implement one ourselves.
The final reason for using Sidekiq is its focus on distributed
systems. A single Redis cache - the cache used by Sidekiq -
can issue jobs to numerous Sidekiq server instances.
</p>
<img
src="./images/architecture/event_diagram.svg"
class="mx-auto img-fluid"
/>
<p>
This diagram illustrates the processing of an event. Once an
inbound request arrives, the EventsController inspects that
both a valid JSON web token is attached and that the endpoint
parameter matches an existing bridge endpoint, in which case
it returns an HTTP response with the status code of 202. It
then delegates all further business logic to the Sidekiq
background process, which promptly pushes a new job to the
Redis jobs queue with the id of the newly created Event
object. As each job is shifted from the jobs queue, an HTTP
request object is generated using Ruby’s built-in Net::HTTP
library and then sent to the designated outbound URL endpoint.
Lastly, it handles the response and saves it to the database,
while, if the response was unsuccessful, it reassigns the job
to the jobs queue if further retries are configured.
</p>
<p>
The result is a job queue system that runs in tandem with our
main Rails API server that can easily scale horizontally and
vertically.
</p>
<h2 class="case-study-h2">4.2 Database Schema</h2>
<p>
We needed a schema for the database that would let us rapidly
retrieve all data related to workflow events yet without
overloading the database with queries. Our initial database
design envisioned separate tables for most data elements of an
event:
</p>
<img
src="./images/architecture/complex_erd.svg"
class="mx-auto d-block img-fluid h-75 w-75"
/>
<p>
An implication of this would have been long join-queries to
fetch all relevant data of an event.
</p>
<p>
Such queries would not only be cumbersome to work with, but
also slow to execute, and tax the database needlessly. A
better solution was to make use of PostgreSQL’s jsonb data
type that stores JSON data. Hstore, which stores data as
key-value string pairs, would have been a viable solution if
we needed to manipulate data after saving it to the database,
since this would not require deserialization. Yet, this
approach would require serialization before sending it, which
increases memory usage and costs. Because events in our
implementation are never edited and their individual parts
never queried, we opted to pool all event data into a single
JSON data structure to always be pulled from the database and
sent to the frontend as is.
</p>
<img
src="./images/architecture/simple_erd.svg"
class="mx-auto d-block img-fluid h-75 w-75"
/>
<p>
The metric below shows metric scores for loading of the event
page in production. First Contentful Paint (FCP) records how
soon the first text or images is visible to the user.
According to statistics from the HTTP Archive, “sites
performing in the ninety-ninth percentile render FCP in about
1.5 seconds”<sup>[4][5]</sup>, which means that our FCP load
of 600 ms makes us faster than 99 percent of websites. While
there are numerous factors in FCP performance, this
performance can significantly be attributed to the decision to
use jsonb.
</p>
<img
src="./images/lighthouse-results.png"
class="mx-auto img-fluid border-radius-10 mb-3"
/>
</section>
<!-- End of 4. SYSTEM ARCHITECTURE -->
<!-- 5. CLOUD HOSTING -->
<section id="cloud-hosting">
<h1 class="case-study-h1">5. Cloud Hosting</h1>
<h2 class="case-study-h2">5.1 Scaling and costs</h2>
<p>
A challenge in production was to find a hosting solution that
eases future vertical and horizontal scaling while maintaining
strict cost control. Scaling vertically, such as adding CPU or
memory, would be beneficial if the user-base growth is
concentrated in a specific region. In contrast, horizontal
scaling through added server instances in other regional data
centers could help the app remain performant (through lower
latency) to all users if user activity is more globally
dispersed. As BridgeAPI is currently maintained by three
developers without corporate backing, we also require the app
to scale in a cost-efficient manner.
</p>
<h2 class="case-study-h2">5.2 AWS Elastic Beanstalk</h2>
<p>
We chose a Platform as a Service (PaaS) in the form of AWS
Elastic Beanstalk (EB) to host the API server. With EB it’s
possible to scale both vertically and horizontally with only a
few clicks. At the same time, monitoring, reporting, health
check, and load balancer for distributed systems make it
easier for a small team such as ours to manage it. Also, it
has transparent, and linear pricing rather than exponential
price jumps for common features as reported for other PaaS
such as Heroku.<sup>[6]</sup> Elastic Beanstalk manages our
load balancer and autoscales our EC2 instances, leaving a
redis server and a postgres server to complete our backend
architecture.
</p>
<img
src="./images/architecture/backend.svg"
class="mx-auto d-block img-fluid h-25 w-75"
/>
<h2 class="case-study-h2">5.3 Redis server</h2>
<p>
While we tried to apply a serverless solution to the greatest
extent possible for cost reasons, our reliance on Sidekiq as a
background job processor necessitates a running Redis server
in our AWS setup. AWS does offer a serverless key-value cache,
DynamoDB, which is similar to Redis, but unfortunately,
Sidekiq does not support it. As a result, 60 percent of our
costs are estimated to be related to the Redis server since
AWS does not offer a serverless Redis cache. Still, we chose
to accept this steep proportion given the relatively modest
total expenses. Nonetheless, future cost-cutting would likely
entail researching replacements for Sidekiq and Redis.
</p>
<h2 class="case-study-h2">5.4 AWS Aurora</h2>
<p>
AWS is more accommodating with regards to the database. While
we used PostgreSQL in development and had planned to use it in
production, we chose AWS Aurora instead as it’s a
PostgreSQL-compatible database that supports serverless
architectures, allowing us to follow a pay-per-use pricing
model. Moreover, using the AWS-native database lets us rely on
AWS to handle database scaling and monitoring for us.
</p>
<h2 class="case-study-h2">5.5 AWS Lambda</h2>
<img
src="./images/architecture/frontend.png"
class="mx-auto img-fluid"
/>
<p>
Finally, we are able to leverage a serverless service in the
form of AWS Lambda for the NextJS frontend, whereby we’re
freed from needing to manage servers as AWS Lambda provisions
these and tears them down as needed. Our server assets
(images, stylesheets, and scripts) are stored in an S3 bucket
integrated with AWS Cloudfront CDN, ensuring that our assets
can be cached and made available locally from the edge
location nearest the user.
</p>
<h2 class="case-study-h2">5.6 Results</h2>
<p>
The result has been universally fast loads. Our total cost is
estimated to be 20 dollars per month, of which Redis accounts
for 12 dollars. We’re also well equipped to handle an increase
in load from anywhere as the API server can be replicated to
any region within minutes.
</p>
<img
src="./images/architecture/full.png"
class="mx-auto img-fluid"
/>
</section>
<!-- End of 5. CLOUD HOSTING -->
<!-- FUTURE WORK -->
<section id="future-work">
<h1 class="case-study-h1">Future Work</h1>
<p>
One avenue of improvement to the user experience is to include
multiple data formats for receiving and forwarding requests.
Currently, it’s only JSON-compatible, but it could be extended
to XML, although it would deviate slightly from the minimalist
aesthetic of the bridge page.
</p>
<p>
Another idea, drawing from popular sites such as Zapier and
Pipedream, is to allow multiple-step workflows instead of the
current single-step limit. This would add significant
complexity to the app and cause it to depart from the niche
position it currently occupies between the single-direction
tools and multi-step workflow tools. Still, it could be
justified by user demand. In its current iteration, a
workaround is to define multiple bridges as endpoints for the
same webhook that could thereby trigger numerous actions with
the same request - a feasible solution as long as each request
is independent of the outcome of other requests.
</p>
</section>
<!-- End of FUTURE WORK -->
<!-- Section Team -->
<section id="team">
<h1 class="case-study-h1 text-center mb-5">Our Team</h1>
<div class="row mb-5">
<div class="mx-auto col-md-8 col-sm-10">
<p class="team-message text-center">
We are looking forward to new opportunities. Please
contact us if this project interested you, or if you have
any questions!
</p>
</div>
</div>
<div class="row mx-auto">
<!-- Andi -->
<div class="col-md-4 col-sm-12">
<img
src="images/team/andrew.png"
class="img-fluid avatar mb-2"
/>
<h4 class="text-center mb-3">Andi Crotwell</h4>
<p class="text-center">Asheville, NC</p>
<div class="row text-center mb-4">
<div class="col">
<a href="https://awcrotwell.com">
<img
src="images/web-icon.png"
class="img-fluid person-icon"
/>
</a>
</div>
<div class="col">
<a href="https://www.linkedin.com/in/awcrotwell/">
<img
src="images/linkedin-icon.png"
class="img-fluid person-icon"
/>
</a>
</div>
<div class="col">
<a href="mailto:awcrotwell@gmail.com">
<img
src="images/mail-icon.png"
class="img-fluid person-icon"
/>
</a>
</div>
</div>
</div>
<!-- William -->
<div class="col-md-4 col-sm-12">
<img
src="images/team/william.jpg"
class="img-fluid avatar mb-2"
/>
<h4 class="text-center mb-3">William Jackson</h4>
<p class="text-center">Chicago, IL</p>
<div class="row text-center mb-4">
<div class="col">
<a href="https://williampj.github.io/">
<img
src="images/web-icon.png"
class="img-fluid person-icon"
/>
</a>
</div>
<div class="col">
<a
href="https://www.linkedin.com/in/william-jackson-62514b5/"
>
<img
src="images/linkedin-icon.png"
class="img-fluid person-icon"
/>
</a>
</div>
<div class="col">
<a href="mailto:williampj@gmail.com">
<img
src="images/mail-icon.png"
class="img-fluid person-icon"
/>
</a>
</div>
</div>
</div>
<!-- Angel -->
<div class="col-md-4 col-sm-12">
<img
src="images/team/angel.jpg"
class="img-fluid avatar mb-2"
/>
<h4 class="text-center mb-3">Angel Ruiz-Bates</h4>
<p class="text-center">Rincón, PR</p>
<div class="row text-center mb-4">
<div class="col">
<a href="https://angelruizbates.com/">
<img
src="images/web-icon.png"
class="img-fluid person-icon"
/>
</a>
</div>
<div class="col">
<a
href="https://www.linkedin.com/in/angel-ruiz-bates-1b68a2142/"
>
<img
src="images/linkedin-icon.png"
class="img-fluid person-icon"
/>
</a>
</div>
<div class="col">
<a href="mailto:angelbates5@yahoo.com">
<img
src="images/mail-icon.png"
class="img-fluid person-icon"
/>