Blog Migration Complete!

Now I finally have my very own domain name! The old Blogger site is still available if anyone wants to see it but for all future intents and purposes, this will be the place where new blog posts will be put up. While I could configure all traffic to the Blogger site to redirect to this one, the old site has grown on me so much over the years that I feel that it would be a shame to hide it that way.

I had to migrate from Blogger because it is more than evident that Blogger has been getting less love from Google than it deserves. Take for instance, the Blogger web-based post editor - a shining example of an undeveloped relic of Google’s products. Posts never autosaved or had any manual version control and a few misplaced keypresses could cause you to lose all your writing progress. Inserting images is a chore and putting up more complicated parts of text such as code blocks and sub-headings is even worse.

Of course as with most things in software development, there are workarounds for everything. The work-around which I had used for the last few blog posts was to write directly in Markdown and then convert it to HTML with pandoc to copy paste in the Blogger editor. Even so, this is clearly sub-optimal and I was spending more time wrestling with the Blogger editor to make my pages look good than I was spending writing actual content. The Blogger theme which I was using appealled to me as 13 year old (heck, it still appeals to me), but it was growing long in the tooth and only had basic support for responsive design and mobile devices. With all these things in mind, I started looking for a new home for my blog somewhere in the middle of last December on a break from college.

I first looked at WordPress. At first, it seemed like my search for a new blogging platform would stop here. Open-sourced, a great browser editor, a cohesive Android app, direct editing in Markdown, local installation for testing, and support for plugins made WordPress everything Blogger wasn’t. I loved the number of themes and customisation WordPress provided. But the love was short lived, and it ended when I started researching hosting options for WordPress. Most solutions required me to rent a web-server on a monthly basis and I had no idea what tier of server to get as my blog had only recently seen a huge surge in pageviews. Not to mention, the cost of maintaining a website with such a setup was by no means cheap. This is when I started asking to my geeky friends about how they maintained their own personal websites.

The talks were very helpful, it made me realise that a dynamic blogging solution such as WordPress was overkill for a humble blog of less than 30 published posts like this one. Having a static site made so much more sense. I could write directly in Markdown, in the text editor of my choice. Not to mention, it made the workflow of writing a blog post just like I wrote code, make, git commit, and git push straight to GitHub Pages. GitHub Pages, by the way, offers completely free hosting for static sites. This means that the only thing I would need to pay for would be the custom domain name, a nominal ₹700 ($10) a year, an amount half of what I would be paying for a month of paid hosting. Plus, it would let me get my hands dirty with a bit of web development, something which I had pushed back for a long time.

I decided to start this - what I knew was going to be painful task at about 11pm on a night I just knew I wouldn’t be able to sleep.

The first step was to convert all my Blogger posts to Markdown. There were some tools online but all of them messed up the conversion pretty badly. After some more digging, I ended up using Aaron Swartz’s html2text Python library which did a better job than other solutions in generating some useful Markdown. I still needed to edit every generated Markdown file by hand to make it something I would be happy with using on my site. I then had to export all the images I had on my Blogger site. This lead to a few more laborious hours of saving each image on the site by hand (Right Click -> View as Image -> Save As). It did cross my mind to automate everything with a script, but it was going to take more time to automate everything and check if the automation was working than it was to do the grunt work of pulling the images. With all the resources safely on my laptop and backed on my Dropbox, I took the next step of looking at static site generators to convert my lovingly handmade Markdown files to HTML.

GitHub Pages seemed to heavily advocate Jekyll so I went with it first. With some tinkering to get the Ruby dependencies installed and posts adapted for Jekyll with the Front Matter content, I managed to get a pretty presentable blog running on localhost:4000/ at 5am on that day. With a quick push to my github.io site, I decided to call it a night and slept off a sleep-deprived session of hacking.

The next few days I played with some more Jekyll themes and found that there were many things I didn’t like about it. For one thing, it was written in Ruby which I have no experience with. Themes didn’t look easy to work with and there was no native support for tags (there is a workaround for this, but due to my lack of Ruby-fu, it all looked terribly arcane to me). I then put the blog migration on the back burner for a while to work on projects at my college’s Automation & Robotics Club.

A few weeks later, I took a look at the blog project with some new perspective. I started by poking around for alternatives to Jekyll. There was one such alternative which ticked all my boxes - a static site generator called Pelican. As WordPress looked inherently superior to Blogger, Pelican looked inherently superior to Jekyll for what I wanted to do with it. For example, it had built-in support for tags, had a theming engine, supported Markdown and reStructuredText, and had several easy to install plugins. Above all, Pelican is written in Python which made it so much easier for me to mess around with it. There were some more modifications to make to the Markdown files (particularly with the post metadata), but it was so trivial that it didn’t pain one bit to modify all the files. Not too long after I settled on Pelican, I found a theme which made my blog look exactly how I wanted it to look. The Pure Single theme also has nifty support for custom sidebar images.

There was some initial trouble with setting a blog subfolder in the site and getting images to work on some auto-generated sites (such as the Tags and Categories pages). It later turned out that it was some problem with localhost/ not finding the paths correctly to the images and the site was totally fine when published to the GitHub Pages site. After only three days of using Pelican, I had something which I was willing to show off. The next step was much more straightforward for a change - registering a domain name. I looked into a few options such as GoDaddy, Hover, and Namecheap. Namecheap had positive reviews (unlike GoDaddy) and was the cheapest of the lot. The site configuration to serve pages from GitHub’s servers was not more than a 10 minute procedure, and I finally had the site you are reading this article on right now.

There will be a lot more changes coming up on this blog, some of them aesthetic and some functional. I’m also probably going to change the name of this blog sometime soon, to something which is more reflective of my current sensibilities.

What I've been upto

This semester has gone a little too fast for my liking. Though there were plenty of good memories and takeaways from this semester, it has been by far my most stressful time in this college yet. Between taking on a number of technical projects and trying to rescue a sharply falling SG due to a poor T1, there has been very little free time this sem. Above all, this sem has shown me just how limited of a resource brain power is. Thinking about solving problems wears me out much more easily than it used to.

That’s not to say what I’ve been doing is all drudgery. Some interesting projects that I’ve been working on:

GSoC 2016 Project - kio-stash

Yup, this project has been in the pipeline for months. While it (mostly) works on a clean install of KDE, it has some bugs with copying with mtp:/ device slaves and isn’t very well integrated with Dolphin yet. It is in my best interest to have this shipped with KDE Frameworks as soon as possible, so I’m looking into patching Dolphin with better, more specific action support for my project.

IEEE Signal Processing Cup 2017 - Real-Time Beat Tracking Challenge

The premise of this project doesn’t seem very complicated - a team just needs to submit an algorithm which can successfully detect the beats in a song in real-time on an embedded device. Heck, I had already implemented the same on an Arduino with a microphone for rather fancy lighting in my hostel room. Unfortunately, as I’ve learned the hard way, anything that looks simple is deceptive and this project is a perfect example. For one thing, a song by itself in the amplitude-time domain is nearly impossible to extract any data from, so it requires a (Fast) Fourier Transform to extract frequency bin amplitudes to get anything useful from it. Next, obtaining beat onset and detection requires generating a tempogram to find the BPM of a song to find the approximate beat positions. There are many interesting approaches and different algorithms to solve this problem. Fortunately, I’m not doing this alone and the seniors I’m working with have a better understanding of the more difficult mathematical components of this project, leaving me free to code in C++ and deploy it on actual hardware.

foss@bphc

This project is a society on campus I started with threeprogrammers far more competent than myself. It all started in September this year when we had the two-day long Wikimedia Hackathon on campus. Though my contribution to Wikimedia was small, I got a lot out of the hackathon by meeting the people from Amrita University who came for organising the event. Listening to what Tony and Srijan had to say about how much working for foss@amrita had benefited them made me realise how much room for improvement there was in BPHC. As far as this society goes, things are still in their infancy, yet I feel there is just enough critical mass of talent and enthusiasm in open source development to push this society forward. We even have our own Constitution!

Making a personal website?

While Blogspot has been a faithful home for this blog for over half a decade, it is time to move to a modern solution for hosting. Plus setting up a website will teach me a bit about web development too, and I’ve needed an excuse to learn it for a while. Surely it can’t be all that hard, right?

Burnout

While the slew of all these technical projects seemed like a good idea at the time, it has pushed me closer to burning out. It is the same feeling as what I felt the same time last year, albeit for different reasons.

Some of it has carried over from studying harder at the end of this sem than I had done so before. To my surprise, studying under pressure was actually more effective than it was without. In some ways, it was a throwback to studying for the number of exams nearly a couple of years ago. It has made me question whether any of my education was as much for the knowledge as it was for the desperation of avoiding failure. I still haven’t figured it out, but at least I feel like I’m not as adverse to putting in the hours for getting a decent CGPA as I had been last semester.

There have been compromises in other spheres of college life as well, investing so much time in studying or doing projects has made me cut down my participation in other clubs and I’ve quit nearly all the non-technical clubs I joined last year. There’s more than that too - I feel it’s much harder to be fully immersed in anything anymore.

Maybe it’s just something else which requires more soul searching.

GSoC Report - Wrapping up GSoC 2016

That’s it. After a combined total of 217 git commits, 6,202 lines of code added, and 4,167 lines of code deleted, GSoC 2016 is finally over.

Screenshot_20160830_014801

These twelve weeks of programming have been a very enriching experience for me and making this project has taught me a lot about production quality software development. Little did I know that a small project I had put together in a 6 hour session of messing around with Qt would lead to something as big as this!

There have been many memorable moments throughout my coding period for the GSoC - such as the first time I got an ioslave to install correctly, to writing its “Hello World” equivalent, and getting a basic implementation of the project up and running by doing a series of dirty hacks with Dolphin’s code. There were also times when I was so frustrated with debugging for this project, that I wanted to do nothing but smash my laptop’s progressively failing display panel with the largest hammer I could find. The great mentorship from my GSoC mentor and the premise of the GSoC program itself kept me going. This also taught me an important lesson with regards to software development - no one starting out gets it right on their first try. It feels like after a long run of not quite getting the results I wanted, the GSoC is the thing which worked out for me as everything just fell into place.

There’s a technical digression here, which you can feel free to skip through if you don’t want to get into the details of the project.

Following up from the previous blog post, with the core features of the application complete, I had moved on to unit testing my project. For this project, unit testing involved writing test cases for each and every component of the application to find and fix any bugs. Despite the innocuous name, unit testing this project was a much bigger challenge than I expected. As for one thing, the ioslave in my project is merely a controller in the MVC system of the virtual Stash File System, the Dolphin file manager, and the KIO Slave itself. Besides, most of the ioslave’s functions have a void return type, so feeding the slave’s functions’ arguments to get an output for checking was not an option either.

This led me to use an approach, which my mentor aptly called “black box testing”.

In this approach, one writes unit tests testing for a specific action and then checking for whether the effects of the said action are as expected. In this case, the ioslave was tested by giving it a test file and then apply some of the ioslave’s functions such as copy, rename, delete, and stat. From there, through a bunch of QVERIFY calls is to check whether the ioslave has completed the operation successfully. Needless to say, this approach is far more convoluted to write unit tests for as it required checking each and every test file for its properties in every test case. Fortunately, the QTestLib API is pretty well documented so it wasn’t difficult to get started with writing unit tests. I also had a template of what a good test suite should look like thanks to David Faure’s excellent work on implementing automated unit testing for the Trash ioslave. With these two tools in hand, I started off with writing unit tests shortly before the second year of college started.

As expected, writing black box unit tests was a PITA in its own right. The first time I ran my unit test I came up with a dismal score of 6 unit tests passed out of the 17 I had written. This lead me to go back and check whether my unit tests were testing correctly at all. It turned out that I had made so many mistakes with writing the unit tests that an entire rewrite of the test suite wasn’t unwarranted.

With a rewrite of the test cases completed, I ran the test suite again. The results were a bit better - 13 out of the 17 test cases passed, but 4 failed test cases - enough reason for the project to be unshippable. Looking into the issue a bit deeper, I found out that all the D-Bus calls to my ioslave for copy and move operations were not working correctly! Given that I had spent so much time on making sure the ioslave was robust enough, this was a mixed surprise. Finally, after a week of rewriting and to an extent, refactoring the rename and copy functions of the ioslave, I got the best terminal output I ever wanted from this project.

Screenshot_20160829_184812

Definitely the highest point of the GSoC for me. From there on out, it was a matter of putting the code on the slow burner for cleaning up any leftover debug statements and for writing documentation for other obscure sections. With a net total of nearly 2000 lines of code, it far surpasses any other project I’ve done in terms of size and quality of code written.

At some points in the project, I felt that the stipend was far too generous, for many people working on KDE voluntarily produce projects much larger thann mine. In the end, I feel the best way to repay the generosity is to continue with open source development - just as the GSoC intended. Prior to the GSoC, open source was simply an interesting concept to me, but contributing a couple of thousands of lines of code to an open source codebase has made me realise just how powerful open source is. There were no restrictions on when I had to work (usually my productivity was at its peak at outlandish late night hours), on the machine I used for coding (a trusty IdeaPad, replaced with a much nicer ThinkPad), or on the place where I felt most comfortable coding from (a toss up between my much used study table or the living room). In many ways, working from home was probably the best environment I could ask for when it came to working on this project. Hacking on an open source project gave me a sense of gratification solving a problem in competitive programming never could have.

The Google Summer of Code may be over, but my journey with open source development has just begun. Here’s to even bigger and better projects in the future!

GSoC Update: Tinkering with KIO

I’m a lot closer to finishing the project now. Thanks to some great support from my GSoC mentor, my project has turned out better than what I had written about in my proposal! Working together, we’ve made a lot of changes to the project.

For starters, we’ve changed the name of the ioslave from “File Tray” to “staging” to “stash”. I wasn’t a big fan of the name change, but I see the utility in shaving off a couple of characters in the name of what I hope will be a widely used feature.

Secondly, the ioslave is now completely independent from Dolphin, or any KIO application for that matter. This means it works exactly the same way across the entire suite of KIO apps. Given that at one point we were planning to make the ioslave fully functional only with Dolphin, this is a major plus point for the project.

Next, the backend for storing stashed files and folders has undergone a complete overhaul. The first iteration of the project stored files and folders by saving the URLs of stashed items in a QList in a custom “stash” daemon running on top of kded5. Although this was a neat little solution which worked well for most intents and purposes, it had some disadvantages. For one, you couldn’t delete and move files around on the ioslave without affecting the source because they were all linked to their original directories. Moreover, with the way ‘mkdir’ works in KIO, this solution would never work without each application being specially configured to use the ioslave which would entail a lot of groundwork laying out QDBus calls to the stash daemon. With these problems looming large, somewhere around the midterm evaluation week, I got a message from my mentor about ramping up the project using a “StashFileSystem”, a virtual file system in Qt that he had written just for this project.

The virtual file system is a clever way to approach this - as it solved both of the problems with the previous approach right off the bat - mkdir could be mapped to virtual directory and now making volatile edits to folders is possible without touching the source directory. It did have its drawbacks too - as it needed to stage every file in the source directory, it would require a lot more memory than the previous approach. Plus, it would still be at the whims of kded5 if a contained process went bad and crashed the daemon.

Nevertheless, the benefits in this case far outweighed the potential cons and I got to implementing it in my ioslave and stash daemon. Using this virtual file system also meant remapping all the SlaveBase functions to corresponding calls to the stash daemon which was a complete rewrite of my code. For instance, my GitHub log for the week of implementing the virtual file system showed a sombre 449++/419–. This isn’t to say it wasn’t productive though - to my surprise the virtual file system actually worked better than I hoped it would! Memory utilisation is low at a nominal ~300 bytes per stashed file and the performance in my manual testing has been looking pretty good.

With the ioslave and other modules of the application largely completed, the current phase of the project involves integrating the feature neatly with Dolphin and for writing a couple of unit tests along the way. I’m looking forward to a good finish with this project.

You can find the source for it here: https://github.com/KDE/kio-stash (did I mention it’s now hosted on a KDE repo? ;) )

GSoC Update(?): Writing a KIO slave 101!

This project has been going well. Though it was expectedly difficult in the beginning, I feel like I am on the other side of the learning curve now. I will probably make a proper update post sometime later this month. My repo for this project can be found here: https://github.com/shortstheory/kio-stash

For now, this is a small tutorial for writing KDE I/O slaves (KIO slaves) which can be used for a variety of KDE applications. KIO slaves are a great way for accessing files from different filesystems and protocols in a neat, uniform way across many KDE applications. Their versatility makes them integral to the KIO library. KIO slaves have changed in their structure the transition to KF5 and this tutorial highlights some of these differences from preceding iterations of it.

Project Structure

For the purpose of this tutorial, your project source directory needs to have the following files.

  • kio_hello.h
  • kio_hello.cpp
  • hello.json
  • CMakeLists.txt

If you don’t feel like creating these yourself, just clone it from here: https://github.com/shortstheory/kioslave-tutorial

hello.json

The .json file replaces the .protocol files used in KIO slaves pre KF5. The .json file for the KIO slave specifies the properties the KIO slave will have such as the executable path to the KIO slave on installation. The .json file also includes properties of the slave such as being able to read from, write to, delete from, among many others. Fields in this .json file are specified from the KProtocolManager class. For creating a KIO slave capable of showing a directory in a file manager such as Dolphin, the listing property must be set to true. As an example, the .json file for the Hello KIO slave described in this tutorial looks like this:

{  
    "KDE-KIO-Protocols" : {   
        "hello": {   
            "Class": ":local",   
            "X-DocPath": "kioslave5/kio_hello.html",   
            "exec": "kf5/kio/hello",   
            "input": "none",   
            "output": "filesystem",   
            "protocol": "hello",   
            "reading": true   
        }   
    }   
}  

As for the CMakeLists.txt, you will need to link your KIO slave module with KF5::KIOCore. This can be seen in the project directory.

kio_hello.h

#ifndef HELLO_H  
#define HELLO_H  

#include <kio/slavebase.h>  

/**  
  This class implements a Hello World kioslave  
 */   
class Hello : public QObject, public KIO::SlaveBase  
{  
    Q_OBJECT   
public:  
    Hello(const QByteArray &pool, const QByteArray &app);   
    void get(const QUrl &url) Q_DECL_OVERRIDE;   
};  

#endif  

The Hello KIO slave is derived from KIO::SlaveBase. The SlaveBase class has some basic functions already implemented for the KIO slave. This can be found in the documentation. However, most of the functions of SlaveBase are virtual functions and have to be re-implemented for the KIO slave. In this case, we are re-implementing the get function to print a QString when it is called by kioclient5.

In case you don’t need special handling of the KIO slave’s functions, you can derive your KIO slave class directly from KIO::ForwardingSlaveBase. Here, you would only need to re-implement the rewriteUrl function to get your KIO slave working.

kio_hello.cpp

#include "hello.h"  
#include <QDebug>  

class KIOPluginForMetaData : public QObject  
{  
    Q_OBJECT   
    Q_PLUGIN_METADATA(IID "org.kde.kio.slave.hello" FILE "hello.json")   
};  

extern "C"  
{  
    int Q_DECL_EXPORT kdemain(int argc, char **argv)   
    {   
        qDebug() << "Launching KIO slave.";   
        if (argc != 4) {   
            fprintf(stderr, "Usage: kio_hello protocol domain-socket1 domain-socket2\n");   
            exit(-1);   
        }   
        Hello slave(argv[2], argv[3]);   
        slave.dispatchLoop();   
        return 0;   
    }   
}  

void Hello::get(const QUrl &url)  
{  
    qDebug() << "Entering function.";   
    mimeType("text/plain");   
    QByteArray str("Hello world!\n");   
    data(str);   
    finished();   
    qDebug() << "Leaving function";   
}  

Hello::Hello(const QByteArray &pool, const QByteArray &app)  
    : SlaveBase("hello", pool, app) {}   

#include "hello.moc"  

The .moc file is, of course, auto-generated at compilation time.

As mentioned earlier, the KIO Slave’s .cpp file will also require a new KIOPluginForMetaData class to add the .json file. The following is used for the hello KIO slave and can be used as an example:

class KIOPluginForMetaData : public QObject  
{  
    Q_OBJECT   
    Q_PLUGIN_METADATA(IID "org.kde.kio.slave.hello" FILE "hello.json")   
};  

CMakeLists.txt

cmake_minimum_required(VERSION 3.5)  
set(QT_MIN_VERSION "5.4.0")  
set(KF5_MIN_VERSION "5.16.0")  

find_package(ECM ${KF5_MIN_VERSION} REQUIRED NO_MODULE)  
set(  
    CMAKE_MODULE_PATH   
        ${CMAKE_MODULE_PATH}   
        ${ECM_MODULE_PATH}   
        ${ECM_KDE_MODULE_DIR}   
)  

include(KDEInstallDirs)  
include(KDECMakeSettings)  
include(KDECompilerSettings NO_POLICY_SCOPE)  
include(ECMSetupVersion)  
include(FeatureSummary)  
add_library(kio_hello MODULE hello.cpp)  
find_package(KF5 ${KF5_MIN_VERSION} REQUIRED KIO)  
target_link_libraries(kio_hello KF5::KIOCore)  
set_target_properties(kio_hello PROPERTIES OUTPUT_NAME "hello")  

install(TARGETS kio_hello DESTINATION ${KDE_INSTALL_PLUGINDIR}/kf5/kio )  

Installation

Simply run the following commands in the source folder:

mkdir build  
cd build  
cmake -DCMAKE_INSTALL_PREFIX=/usr -DKDE_INSTALL_USE_QT_SYS_PATHS=TRUE ..  
make  
sudo make install  
kdeinit5  

As shown above, we have to run kdeinit5 again so the new KIO slave is discovered by KLauncher and can be loaded when we run a command through an application such as kioclient5.

Run:

kioclient5 'cat' 'hello:/'  

And the output should be:

Hello_world