Wednesday, December 6, 2017

The urge to do

Programming and debugging are  activities where we should think a lot and do little. Commonly what happens is the reverse. We are driven by the urge to do something.
It's particularly true for debugging. We see a  problem, the first instinct is to print something, build and rerun. And then print some more. In the end when we discover the problem we go: Ah, that wasobvious. I could have gotten there faster instead of these x iterations. And on complex systems these iterations take a long long time.
It often helps to resist the urge to do  and think what could be going off here. It saves a ton of time.
So also for programming.
A lot of programming time is spent in the edit build debug cycle. We make numerous iterations through this cycle to get to our goal. How could we optimize this? What if we are allowed only 1 iteration through this?
To ensure that I put enough thought in the process, recently I created this game for myself. After breaking the problem into subproblems I take it as a challenge that every submodule that is built will run completely in the first iteration itself. I get to decide the size of the submodule, but my code should do exactly what I intended in the first shot. Over time I have found that this leads to a lot of efficiency by forcing me to think of all the dependencies and side effects upfront.
Of course not all problems can be addressed with this method. Especially if it's a new technology or domain you learn much faster through each iteration. But for a lot of cases this approach does magic. You should try it out...

Thursday, October 26, 2017

Defining Objects

Earlier in my career, I loved browsing through the Linux kernel's source code. Mostly to satisfy my curiosity about how certain things are internally implemented. As I read through more and more files of code, I started noticing one common pattern. For the most part, the code was organised around  abstract concepts (spin-lock, work-queue, kthreads etc). And every abstract concept had

  • An object representation, a C structure that holds details about that structure
  • An init/alloc call, that initialises the object
  • The init/alloc call also returns a handle to this object
  • A number of operations that accept this handle as a parameter and then operate on that object
  • And finally a deinit/free call that destroys that object and also makes the handle invalid for any further use

Oh yes, you say, we had learnt it in our classes, no big deal! No, no, no. I mean, I had also learnt about it in my classes, but I had never ever used it practically in my code, apart from the stack example in my class assignment.

This was a good way to see it in practice in the Linux kernel, and that too used at multiple places. We even use it in our everyday programming lives. For example,

  •  File IO:
    • fd = open(...);
    • read(fd, ...);
    • write(fd, ...);
    • close(fd);
  • Socket IO:
    • socket_fd = socket(...);
    • bind(socket_fd, ...);
    • connect(socket_fd, ...);
    • send(socket_fd, ...);
    • close(socket_fd);
  • Pthreads: (pthread_create)
  • OpenSSL: (SSL_new)
  • Semaphores: (sem_init)
  • and numerous others

So what is really my point here? Thinking in this way, helps you focus on defining these objects. And thinking about these objects helps you have a more modular and cleaner approach to software development. Design your data structures first, and the rest automatically follows. 

Next time you add a global int/char variable, think if there is a relationship among those data structures. Do they belong into a structure together? Can you make things easier by defining an abstract concept/object and operations around that object?

Here is what Linus Torvalds (the founder of the Linux kernel and git) has to say about this:

I'm a huge proponent of designing your code around the data,
rather than the other way around, and I think it's one of the
reasons git has been fairly successful (*).
(*) I will, in fact, claim that the difference between a bad
programmer and a good one is whether he considers his code or
his data structures more important. Bad programmers worry
about the code. Good programmers worry about data structures
and their relationships.

Saturday, October 7, 2017


Let's do an experiment. I assume you have been working on a project (academic or professional) for some time now. Today, setup some realistic goal as to what you want to achieve at the end of this week. And in the morning, every day of this week, setup a sub-goal as to what you wish to finish at the end of that day. At the end of the week, evaluate whether you were more productive than the weeks before this. Repeat this exercise for another week and check if you were more productive.

To me, I find myself more productive when I do this. Moreover, I find that things just keep sliding when I do not do this. It just goes on and on, with no concrete outcome.

For one, setting milestones help you pace yourself through the day. Without it, its just work-in-work-out, a chore. With a target at the end of the day, your whole approach is different. You are focused on results, and achieving them is so satisfying.

Secondly, setting milestones helps you focus on the prioritisation and planning activity, which is particularly overlooked early in your career. Nothing fancy, just a small enough time frame to experiment and improve week over week.

It has so often happened to me, that I go: "Hmm, if I hadn't given it this thought, I would have continued pushing down a path that would waste so much time"

So, try it out, and let me know how it goes...

Thursday, October 5, 2017

In Pursuit of Efficiency

The first few years of your career you learn a huge amount. Make the best out of it, and develop an attitude to observe and learn from your own experiences. After every task, take a moment. Look back at what you did. How you did it. Feel good about it. And importantly, think how could you could have done it better and faster.

This look-back ends up becoming very critical, and helps you evolve much faster with each iteration. And act on, what you think, could have been improved upon.

Typical things you should think of:

  • Can I optimise the time spent switching windows
  • Could I save the build time by running multiple threads 
  • Could I overlap this activity with that
  • If a tool keeps taking most of my time, I should be a power-user of that tool
  • I got these code review comments, what was the rationale behind points 2 and 5? Search or ask the reviewer
  • Could I have done this faster if I had approached the problem differently?
  • Are there any recurring patterns across the tasks that I have done, and the deductions I have drawn so far?

What is important is having that perspective, almost like zooming out of your body and looking at yourself, trying to identify where could you improve your efficiency.

Thursday, September 14, 2017

Programming with Maturity

As a computer engineering student, you learn multiple languages in your college.

You understand the syntax and concepts of this language.

You work on a few assignments to understand some unique features of this language.

Most of the focus in these assignments and studies is on getting it done. Implementing quick sort an sorting an array, or  traversing a tree in pre/post/in-order.

But what about the qualities of software?

Beyond the customary paragraph on each of the software qualities, how do you learn about those? How do you acquire the right taste in software? Because at the end of the day, its not about whether you can implement it. Of course, you can, any so-called software developer can. But its about whether you can implement it with the best qualities. That is what sets your deliverable apart. Once you join an organisation this will happen through your mentors at the organisation. But what can you do while you are still a student?

One interesting way to develop this taste is to track open-source projects in your domain of interest. These are treasure troves of knowledge, from some of the most experienced folks in the world. Imagine being able to read the source code of the Linux Kernel which powers almost everything, from the OS in your phone, to large-scale servers.

And what do I mean by track an open-source project?

  • In its simplest form, try to understand a small enough module from that project. And document it for the world (this also acts as a good milestone to work against). You will have to read multiple files and cross-references in the process. And when you finish one module, try a more complex one. Not only this exercise make you good at reading code, but also the best practices and the development philosophy of that project starts rubbing on you.

  • As a next step, subscribe to the project's development mailing list, or their git repositories pull/merge requests. Read about the various proposals contributors make, the arguments against these proposals. You might not understand everything, but you start understanding what members of that project care about beyond functionality.

  • Best yet, pick a small portion of the TODO list of that project and send out a patch for inclusion. While reading has its benefits, doing it yourself exposes you to a world of other questions, that you hadn't considered before. And the comments on your work, that's golden. 

If you are interested in System Programming, track the Linux Kernel. Or if its too large for you, track Zephyr.