Open-source, Self-hosting on Raspberry-Pi and the Gizmo -"n8n.io"

I once asked a CxO of a Fortune-500 company what were his views about "Open-source". His answer made me realize that I had a long way to go in every walk of life. He said, "Open-source is like Potluck, everyone wants to be there and enjoy the dishes, but not many want to cook and bring something for others".

After more than a year and a half after that, as I experiment with n8n.io, I do realize that he was right. There is a lot of great work happening around the world, especially in the open-source community. But, we might want to ask ourselves are we doing enough to support it? Even if a lot is happening in the world of Open-source, are we giving enough traction to the good that's happening? And are we all contributing enough towards it? Are we unconditionally embracing open-source and giving a shoulder to the fraternity? Are we unyielding in implementing open standards? And, most important of all, are we helping the open-source project grow? Are we giving enough voice to the good work and jumping to fix the bugs on these products for the larger audience?

These questions are very relevant for many reasons. To begin with, many great products enjoy a large pool of fandom. While I was working on Apache Airflow, I realized that the community and the ecosystem while still growing, it was already mature enough and that many enterprises have come in sheer support of the product. But, is that enough? Are we all pulling the heavyweight of an open-source project that can benefit the masses of the tech world?

Certainly, there is no answer to such questions. While about 3.5 years in the software industry might seem too little for any individual to talk about it, a hands-on experience with many era-defining products and technologies gave me exposure to some of the many faces of this industry. There comes but rarely comes a technology so significant that we can vividly see the industry gyrate about it and pace further. By grace, I had the opportunity to work on many such technologies.

While professionally we might be finding it at ease to move ahead with implementing and experimenting with new tech, self-hosting on a home server throws a different problem to solve. It is because we cannot keep up with maintenance and licensing costs. And if technology is going to be a prerogative of the elite-erudite few, how can we empower everybody? Also, if everyone had to build their own tools from scratch, or if everyone does not have the same failsafe tools used for the enterprises, how can we all be equally equipped for building a better future?

Just when I was trying to observe these thoughts, I came across a Reddit comment which introduced me to something new. Something that gave me a peep into the future of no-code!

I have been an Integration Engineer for quite some time now, but, very recently have started working on Dell Boomi. It is undeniably one of the most polished iPaaS solutions probably. There is a well-thought interface, a more structured interface, a granular version control, a keen logging mechanism, and above all, a very well-developed set of connectors and much more. The design principles are more or less timeless ideals realized through years of changes and improvements.

There are many more qualified engineers and professionals working on Dell Boomi than I am. I have fallen in love with the marvel Dell Boomi is. BUT, Dell Boomi does come at a cost - a cost of licensing. The pricing models are probably a big burden on the small pockets. Just to be clear, there is no comparison between Dell Boomi and the latter part of this article. Boomi is a sneak peek into the future.

What if I told you, there was a similar product, with a large fanbase and community, bundled a wider range of connector support and all that is free! Yes, such a thing exists, and it is n8n.io.

Self-hosting is interesting, but, it is also very tiring, at least for me. Because I use a Raspberry Pi for all my hosting needs, and that I am sort of a beginner here, it is very painstaking to set up things. Some things that are tiring include-

Maintaining your domain registrar and DNS provider records (even if it is one-time-setup)
Managing the CDN level settings like Security, Captcha, bot-blocking, etc.
Router and router-level security
Devices and network configurations on the intranet
Raspberry-pi's and power supply
OS'es and kernel versions
Installations and software configurations
Applications and data-volumes
Application configurations and backups
Managing the hosted containers/applications

The list goes on. There is no shame in admitting the friction on the joyous journey of self-hosting. And yet, regardless of friction we do things anyway.

Although I was suffering from all these challenges, I did script and automate many tasks. I scheduled Crons and watchdogs to restart and take care of many services I was hosting. I also started some tools like Portainer, Nginx-Proxy-Manager, and other cool dashboards to manage the many containers and services. But, there was always some friction. There was always a tall wall of uneasiness that loomed between me and my server. And, I could not easily scale it.

I wanted to be able to automate and control things with more ease, with much greater agility and force. That is when I wanted to be able to find an open-source tool that could get me going fast! Something like Dell-Boomi. Something that could get me going with faster development and implementation. A bridge over the tall wall so that I can trek and slide to the other side, or jump over the tricky tall discomfort.

One quick thought was to use something like IFTTT, Twilio, or Zapier. These services were amazing. I was stunned by what I could do from outside the network. I was amazed by the number of integrations and features. I quickly jumped to integrate my GhostCMS blog with Medium using Zapier. It was only a few clicks away! As soon as I published something on Ghost, I could see it come up on Medium as a new story and a new tweet for each new post. This was an overwhelming amount of automation for me because it was only a few clicks and no scripting!

There was no way to be not contented with what anybody can do with Zapier/IFTTT. But, certainly, this still costed a fair bit. Even if the pricing plans were fair, for an enthusiast like me, this cost was still heavy as I continue to experiment with tech. I scraped through the internet and forums, probably through every page that spoke about alternatives to Zapier/IFTTT. In some corner of the internet, in one of the infinite pools of comments, somewhere in between the post archives and tiny fonts, I saw something that changed my entire outlook on self-hosting it is "n8n".

What is N8N?

I will put it in simpler terms, it is not just a workflow automation tool, it is a gizmo, a boon to the automation-freaks, a UI based automation tool that can be extended and probably has no limits on what we can do (except for its licensing and our imagination).

It has about 200 different and out-of-box integrations. Triggers and connectors make it a suitable and easy choice for the home-server automation I was looking for!

Some quick features:

Docker ready!
has an encrypted credential store
Import/Export of workflows and credentials are supported
Basic authentication can be enabled
Logging and tracking executions
Support for webhooks!
Batteries included!

Now, the bigger question, is, what can you do with it? Probably anything and everything. Here are a few workflows I have built -

Ghost CMS to Medium - every article published on GHOST is automatically published on Medium via a cron workflow.
Ghost to Twitter - A new tweet is sent each time a post is published on Ghost
Continuous monitoring of ghost blog
A view counter based on JS
A backup of n8n configs to Gmail and Gdrive

While there were many other integrations I made on n8n, the King of the workflows was a Telegram bot! that I integrated with n8n! This changed everything for me. It turned my home theatre into a smart speaker system. A simple telegram chat into an email delivery workflow. And above all, it made my Raspberry-pi into an automation powerhouse!

What can my bot do!

Play a YouTube video on the speakers connected to Raspberry-Pi
Mail delivery with SendGrid
Update a blog with just a telegram message
A URL shortener
A smart-light control
A backup system away from the phone!
Execute Python scripts
Run shell commands
A password update with Bitwarden!
Server uptime and health monitoring
Send Newsletters (I tried this with a limited audience, and it works)

All this just in a matter of few hours!

It is exciting to be able to think of a certain use case and be able to get it working in just a few clicks. What excites me further is how it rightly fits into the space left vacant between Apache Airflow and Dell Boomi, a gap that kept me wondering and pushing for exploration. This solved most of my self-hosting problems.

This tool is indeed a gizmo, anybody looking for open-source and self-hosted alternatives for Zapier/IFTTT. It is free and ready for experimentation. But, in my view, it is also true that n8n is not all-encompassing from an enterprise perspective because it lacks some of the most fundamental features. Just to name a few -

Granular version control for each shape and configuration
Workflow level version control
Logging and easy monitoring of jobs

I also noticed some intermittent App freezes when experimenting with multiple web-hooks, but, a container restart was all it needed to be fixed.

If there are enough supporters ready to adopt a technological marvel like N8N, projects like this can grow. I do know and understand that the license which comes with N8N prohibits commercialization. So, it is not completely open-source in some ways. But, what is it like if a project like this gets enough traction? What if projects like this come together? What if the community embraces the risks, and works on fixing them? What if we can add more robustness to this marvel? What if we can improve on the shortcomings and make N8N a strong alternative for the masses? What if it solves the problems of many others like me who are beginners in self-hosting!

With solutions like n8n, the future seems more interesting, and more promising. Above all, it is very comforting to be able to join the dots and gaps that are evident!

In future series, I will be elucidating more on what n8n is, how to deploy and develop with n8n! and the limitless possibilities it brings!