When it comes to streaming, writing your own data generator is often times the easiest way forward. I’ve written a simple Python script providing me with realistically looking web proxy logs. Apart from other features (volumes, frequency, DoS attack simulation etc.), it allows to stream traffic to a remote HTTP endpoint. This post Read more…
One of my previous posts about NiFi dealt with POSTing dynamically built JSON. Looking at it in retrospect, I realised there was space for improvement and ended up writing a custom processor. The result is not as feature rich as NiFi’s out-of-the-box counterparts InvokeHTTP and PostHTTP. However, my implementation incorporates request body definition Read more…
In my previous post I looked at a basic use of plain HTTP in a NiFi ingest pipeline. In practice however, an encrypted communication channel is an imperative. In an ideal world, switching to HTTPS is easy, but in reality we frequently face SSL errors of various kinds. This post shows how to go Read more…
Apache NiFi does a great job when it comes to data flow automation. It excels in many regards – ease of use, built-in processors of all kinds, straightforward debugging or data provenance, to name but a few. Despite of a fairly moderate learning curve, there surely are new concepts to grasp. It took Read more…
Our team of five won at #TADHackLDN with a simple, yet compelling hack called Moodlight. As the name suggests the project is based on sentiment analysis. The mood intercepted on a Slack channel illuminates a LED bulb (Philips Hue smart light). Setting up a team Slack channel was an obvious first step. Next, rather than delving Read more…
I’ve tried setting up Hadoop from scratch couple of times, but in the end there were too many subtleties for me to handle. I quickly realised that adding components to Hadoop ecosystem takes time (configuration, compatibility issues). For example, this is a great post on how to install Hadoop on Ubuntu, but Read more…