Are You Bored At Work?

• Do you find your current engineering job repetitive?

• Are you bored propagating yet another field from the database onto the UI?

• How often are you checking your favourite social network?

• Are you reading this handbook while at work? Yeah, you're bored at work.

At Infobip... That doesn't happen often

It might be the scale of work: we do have over 40 data centers, process millions of messages, etc.

Or it might be the way work is organised: we try to keep it agile, adapting our product to quickly changing needs of our clients. It's not uncommon to be given a new technical challenge, like scale one service 10 times, and a sprint or two to solve it. There is little opportunity to be bored around here.

There will always be some repetitive tasks in any software engineer's work. We try to automate it. It takes manual work to change external config for each instance of your service? Write a config server to manage all configuration changes in one central place. Config server is now a single point of failure? Refactor it to support active-active redundancy. Thousands of services are now continually pulling data from the config server and congesting the network? Write a client side library for integrating incremental updates and push them only when config actually changes...

You get the picture. There's no time to be bored at Infobip.

It helps that our requirements come from our clients and their real world needs.

For example, a client wants to run a contest over SMS:

They'll send a personalised introduction text, wait for a response and, upon receiving it, send an acknowledgement. They will engage 10% of their country's population. And they will limit the voting to one hour. Let's do some back of the napkin math: that's 2 outbound + 1 inbound message for every participant. Let's say there are around 7 million of them. All this needs to go down in one hour, or 3600 seconds. That's average of ~4000 outbound and ~2000 inbound messages per second, which all need to be properly paired up and processed individually. They all need to go through the billing process, all need to be routed to the most reliable telecom operators and their delivery needs to be tracked and reported back to the client. And all this needs to happen without interfering with existing traffic from other clients.

We try to ask good questions:

• How would you go about scaling existing infrastructure to take on this extra load?

• Would you spin up dedicated infrastructure for this particular client, or would you scale existing, multi-tenant services?

• What kind of networking throughput would you need for this? What kind of algorithmic optimizations can you suggest?

• How would you go about pinpointing performance bottlenecks that need improving?

How Bored Are You?

There are challenges that emerge less directly.

Some of our clients operate in regulated industries and are required by law to deliver notifications to their end-users. If we fail to send even a few of their messages they may get in trouble with regulators. That's exactly why we are always chasing after high availability of our services on individual level, and the platform as a whole. When the conversation moves to 9s of availability you need to find an out-of-the-box solution.

Much has been written on topics of distributed systems, redundancy etc. But theory is one and practice brings its own set of messy problems. How do you achieve data redundancy across AWS, Azure and on-premise datacenters? How do you improve HTTP API availability if clients refuse to implement retries on their end? Theory jumps in with helpful suggestions: separate command requests (like sending SMS) from query requests (like fetching delivery reports). Then we can enqueue messages and cache delivery reports. But now we need to set up and maintain those queues and caches. How boring do you find a prospect of becoming a Redis expert?

In addition to relieving you from boredom, all these technical challenges will quickly help you hone in your problem-solving skills.

Solving hard, real-world backend problems is not an extraordinary occurrence in Infobip, it's what we do on a daily basis.

And practice makes perfect. As you grow personally you'll have an opportunity and an obligation to help your teammates grow and develop alongside you. It's this network of approachable and highly skilled professionals that make all of this problem solving within our engineering department work - we'd love for you to be part of our team!