A Hands-on Guide to Backend Performance Optimization of a SaaS platform for scalable AI workloads 

This article contains a more detailed treatment of backend performance optimisation considerations in platforms targeted to handle scalable AI workloads. A brief introduction of the same can be found in our blog on Use of Celery in Serverless systems for scaling AI workloads.  The platform referenced is Ignitarium’s TYQ-i® SaaS Platform (TSP), a serverless architecture that is used for delivering high performance anomaly detections for civil infrastructure and manufacturing lines. The following were some of the key platform imperatives: 

  • Scalability – easy to scale; as the number of algorithms and components that are deployed simultaneously increases, should not degrade the guaranteed frames per second (fps) performance 
  • Flexibility in terms of infrastructure – the platform should be capable of running on-premises or in standard public cloud offerings 
  • High performance – real-time or batch processing capabilities (mandated by the specific deployment scenario trading off latency vs throughput) 
  • Plug and play architecture – well defined protocols to incorporate new components from internal developers or partners 

1. Simplified TSP architecture

Fig 1. shows the simplified TSP architecture; it consists of a sequence of microservices triggered by user requests from the browser. These requests are served by a set of nodes (called Workers) that are sequenced by an Orchestrator block. This article is focused on the implementation and optimization of these Workers.

Fig 1: Simplified TSP architecture

An important aspect of the platform being scalable is the choice of distributed computing. Each application has a sequence of tasks to complete which could include pre-processing (filtering, transformation, image registration, stitching etc.), AI inferencing jobs (segmentation, classification, detection) and post processing jobs. The platform should be able to support a responsive User Interface (a batch upload or viewing the result set), while tasks belonging to multiple instances of different applications are executed concurrently (asynchronously). This implies that the compute backend should have the ability to: 

  1. fetch from an input/result queue and task queue 
  2. push the result back to input/result queue (the output of one task may be an input to another task) 
  3. flag the status of a task 
  4. use standard Python functions as tasks  

2. Distributed task queue

With the need for multiple tasks to execute asynchronously, we needed a distributed task queue and execution method. Our first choice of distributed task queue was Celery, though we tried alternatives as proof of concepts. Celery is a task queue implementation intended for use with web applications to offload heavy duty compute from webservers. Compared to alternatives, Celery ticked many boxes, including its compatibility with different message queues, cluster/container tools and a large user/expert community.  

3. Task pipeline

Celery provides a feature named “Canvas” to develop workflows. This feature enables pre- sequencing of tasks in a pipeline. Celery does not just execute the tasks in the specified order but will also take care of passing results from one task to its successor, without needing any special coding effort. This feature can be leveraged to run a static pipeline of tasks. We used this extensively during the initial phase of platform development, being fully aware that it cannot handle decision-based execution of tasks. To illustrate the simplicity of Canvas, please refer Fig 2. In this example, tasks T1, T2a, T2b and T3 are connected. The output of T1 is fed to T2a and T2b which are parallel tasks, and their respective outputs need to be fed to T3. Using Canvas, this workflow can be built with a single statement: 

chain(T1(), group(T2a(), T2b()), T3())

Fig 2: Workflow representation using Canvas

4. Branch executors

Canvas worked great until we faced situations where the output of a task determined the next task to run. Canvas does not link jobs as successor or predecessor dynamically. We worked around this limitation by introducing a wrapper task named ‘Branch Executor’ that will call the function or execute the task depending on the output from the previous task. Without branch executor, the execution logic is embedded in the successor branch. That is, each of the possible output branch will check if the output matches with the condition in which it should run and if not, it will return. This meant that Celery must pick up all the potential successor branches, check if the predecessor’s result matches with the condition in which the current task must execute. Such testing in all possible successor tasks will result in inefficiency and avoidable delays.  

To illustrate this point, let us extend the previous example. Consider that the task T1() produces ‘a’ or ‘b’ as output and accordingly either T2a() or T2b() must be executed. We can still use Canvas in the way it was explained in our previous example (Fig 2). But the tasks T2a or T2b, (referred to as T2X) should have a logic to check if output from T1() is equal to X and if not return immediately. Apart from this, T3() will have to be ready to accept an empty response from all T2X() tasks except the one which has result from T1() matching its condition.

Fig 3: Task execution using Branch Executor

Branch executors replaced execution of all possible task options and executed only the task that should be run (Fig 3). This not only improved the performance, but also allowed us to keep the workflow simple:  

chain(T1(), BranchExecutor(), T3()) 

However, the combination of Branch Executor and Celery Canvas too had its limitations as the number of decision boxes in the workflow increases. 

5. Workflow/pipeline scheduler

Branch Executors worked just fine with Celery Canvas when the control of execution can be handed back to a main thread after execution of a single task. But, if the path chosen by the branch executor itself had to execute a series of tasks in each branch, or worse, if there were branches within branches, pre-sequencing using Canvas becomes cumbersome. This meant that it was time to move on from the ‘Canvas’ based sequencing solution.  

We had to introduce a dynamic workflow author and scheduler into the platform architecture. From the set of options, we chose Airflow because of its versatility, ease of use and pluggable architecture with executors of various kinds; it employed the Celery executor, which allowed a smooth integration into our infrastructure.  

Soon, we realised that though Airflow’s workflow authoring, scheduling and monitoring capabilities were excellent, task scheduling latency exceeded one second. For batch jobs, this would not have been a challenge. But, in real-time inferencing, and with sub second throughput time of each task, waiting for more than one second between tasks was certainly unaffordable. We had to find an alternative, which is simple but fast enough for our needs.  

6. Custom scheduler

None of the alternatives of Airflow matched our needs, so we decided to build a custom scheduler from scratch. We arrived at a few constraints that we wanted to impose on ourselves, while developing our own replacement for Airflow. It should be 

  1. separate from the algorithms and part of the platform/infrastructure components 
  2. implemented without adding too many parameters 
  3. quick and with minimal processing overhead 
  4. easy to maintain 

A light-weight scheduler was thus developed that could be plugged into the platform. When a new algorithm gets on-boarded, the task configuration (yaml file) was made available to the platform. This included the parameters for each task (algorithm specific) and infrastructure needs like GPU requirement etc. (platform specific). Two new fields – predecessor and successor for the jobs were added to the configuration file. The scheduler used this information to build the graph in a single pass over the task configuration file. The scheduler builds a list of dependents (successors) for each node. As each node completes, the status is updated, and the scheduler picks up the successor tasks of the completed task and adds them to the execution queue.  

Fig 4. shows the task configuration file with non-relevant details removed and the corresponding graph created by the scheduler. Note that successor and predecessor information, though redundant, allows the construction of a dependency graph in a single scan of the yaml file.

Fig 4: Scheduler task configuration file

To indicate that a task is a potential starting node of a subchain, a third field was added (subchain =True/False). Complex sub chains belonging to multiple paths were stored separately and connected to the main graph on the fly. This avoided the need to traverse through all the nodes for each image as they enter the task lifecycle. Another Boolean field named ‘condition’ is used to identify nodes after which the branching will start. To demonstrate this, let us consider this example where result from T1 would determine one of the following paths from (T2a, T2a1, T2a2) or (T2b, T2b1). The yaml file and the graph constructed from the initial pass are shown in Fig 5.

Fig 5: Initial graph constructed on first pass

Fig 6. shows the graph modified during run time, with the result from T1 causing the Orchestrator to pick the (T2b, T2b1) path. The scheduler having access to the result from T1 will rewire the graph and the tasks get executed as per the rewired graph. Rewiring is accomplished by overriding the predecessor and successor fields of just 4 tasks (marked in red circles) – i.e., condition node (T1), subchain node (T2b), the last node in the subchain (T2b1) and the original child node of condition node (T3).

Fig 6: Graph re-wired at run-time

7. Performance optimization

The platform orchestrator invokes the custom scheduler as a separate thread for each image, which further reduced the impact of graph construction time on the performance requirement and isolating it from impacting other task pipelines. The custom scheduler needed about 100 lines of code and just a few additional lines in the configuration of each node (task) – the predecessor, successor, subchain (optional) and condition (optional). Further improvements to performance, abstraction and code readability were achieved by introducing the networkx library for implementation of Directed Acyclic Graphs (DAG). 

We distributed the tasks in separate queues to take advantage of differing execution times. For example, smaller duration jobs were consolidated in a single queue and longer ones were split into multiple queues. Similarly, CPU intensive jobs were given a dedicated queue, while GPU queues were dedicated based on history of their graphic core needs.   

We also separated workflow of projects into multiple small workflows when the execution order of a video frame mattered. This separation allowed us to get further improvement on latency to the tune of 20 to 30%.  

8. Summary

Performance tuning of highly scalable AI platforms is a complex exercise and requires careful upfront architecture, numerous trade-offs and extensive PoCs. With the right mix of off-the-shelf and custom-built components, we were able to induct high performance workflow orchestration methodologies into Ignitarium’s TYQ-i SaaS Platform to meet the demanding requirements of Infrastructure analytics use cases.  

Scroll to Top

Human Pose Detection & Classification

Some Buildings in a city


  • Suitable for real time detection on edge devices
  • Detects human pose / key points and recognizes movement / behavior
  • Light weight deep learning models with good accuracy and performance

Target Markets:

  • Patient Monitoring in Hospitals
  • Surveillance
  • Sports/Exercise Pose Estimation
  • Retail Analytics

OCR / Pattern Recognition

Some Buildings in a city

Use cases :

  • Analog dial reading
  • Digital meter reading
  • Label recognition
  • Document OCR

Highlights :

  • Configurable for text or pattern recognition
  • Simultaneous Analog and Digital Dial reading
  • Lightweight implementation

Behavior Monitoring

Some Buildings in a city

Use cases :

  • Fall Detection
  • Social Distancing

Highlights :

  • Can define region of interest to monitor
  • Multi-subject monitoring
  • Multi-camera monitoring
  • Alarm triggers

Attire & PPE Detection

Some Buildings in a city

Use cases :

  • PPE Checks
  • Disallowed attire checks

Use cases :

  • Non-intrusive adherence checks
  • Customizable attire checks
  • Post-deployment trainable


Request for Video

    Real Time Color Detection​

    Use cases :

    • Machine vision applications such as color sorter or food defect detection

    Highlights :

    • Color detection algorithm with real time performance
    • Detects as close to human vison as possible including color shade discrimination
    • GPGPU based algorithm on NVIDIA CUDA and Snapdragon Adreno GPU
    • Extremely low latency (a few 10s of milliseconds) for detection
    • Portable onto different hardware platforms

    Missing Artifact Detection

    Use cases :

    • Detection of missing components during various stages of manufacturing of industrial parts
    • Examples include : missing nuts and bolts, missing ridges, missing grooves on plastic and metal blocks

    Highlights :

    • Custom neural network and algorithms to achieve high accuracy and inference speed
    • Single-pass detection of many categories of missing artifacts
    • In-field trainable neural networks with dynamic addition of new artifact categories
    • Implementation using low cost cameras and not expensive machine-vision cameras
    • Learning via the use of minimal training sets
    • Options to implement the neural network on GPU or CPU based systems

    Real Time Manufacturing Line Inspection

    Use cases :

    • Detection of defects on the surface of manufactured goods (metal, plastic, glass, food, etc.)
    • Can be integrated into the overall automated QA infrastructure on an assembly line.

    Highlights :

    • Custom neural network and algorithms to achieve high accuracy and inference speed
    • Use of consumer or industrial grade cameras
    • Requires only a few hundred images during the training phase
    • Supports incremental training of the neural network with data augmentation
    • Allows implementation on low cost GPU or CPU based platforms

    Ground Based Infrastructure analytics

    Some Buildings in a city

    Use cases :

    • Rail tracks (public transport, mining, etc.)
    • Highways
    • Tunnels

    Highlights :

    • Analysis of video and images from 2D & 3D RGB camera sensors
    • Multi sensor support (X-ray, thermal, radar, etc.)
    • Detection of anomalies in peripheral areas of core infrastructure (Ex: vegetation or stones near rail tracks)

    Aerial Analytics

    Use cases :

    • Rail track defect detection
    • Tower defect detection: Structural analysis of Power
      transmission towers
    • infrastructure mapping

    Highlights :

    • Defect detection from a distance
    • Non-intrusive
    • Automatic video capture with perfectly centered ROI
    • No manual intervention is required by a pilot for
      camera positioning


    Co-founder & CEO


    Founder and Managing director of Ignitarium, Sanjay has been responsible for defining Ignitarium’s core values, which encompass the organisation’s approach towards clients, partners, and all internal stakeholders, and in establishing an innovation and value-driven organisational culture.


    Prior to founding Ignitarium in 2012, Sanjay spent the initial 22 years of his career with the VLSI and Systems Business unit at Wipro Technologies. In his formative years, Sanjay worked in diverse engineering roles in Electronic hardware design, ASIC design, and custom library development. Sanjay later handled a flagship – multi-million dollar, 600-engineer strong – Semiconductor & Embedded account owning complete Delivery and Business responsibility.


    Sanjay graduated in Electronics and Communication Engineering from College of Engineering, Trivandrum, and has a Postgraduate degree in Microelectronics from BITS Pilani.


    Request Free Demo

      RAMESH EMANI Board Member


      Board Member

      Ramesh was the Founder and CEO of Insta Health Solutions, a software products company focused on providing complete hospital and clinic management solutions for hospitals and clinics in India, the Middle East, Southeast Asia, and Africa. He raised Series A funds from Inventus Capital and then subsequently sold the company to Practo Technologies, India. Post-sale, he held the role of SVP and Head of the Insta BU for 4 years. He has now retired from full-time employment and is working as a consultant and board member.


      Prior to Insta, Ramesh had a 25-year-long career at Wipro Technologies where he was the President of the $1B Telecom and Product Engineering Solutions business heading a team of 19,000 people with a truly global operations footprint. Among his other key roles at Wipro, he was a member of Wipro's Corporate Executive Council and was Chief Technology Officer.


      Ramesh is also an Independent Board Member of eMIDs Technologies, a $100M IT services company focused on the healthcare vertical with market presence in the US and India.


      Ramesh holds an M-Tech in Computer Science from IIT-Kanpur.


      General Manager - Marketing

      A professional with a 14-year track record in technology marketing, Malavika heads marketing in Ignitarium. Responsible for all branding, positioning and promotional initiatives in the company, she has collaborated with technical and business teams to further strengthen Ignitarium's positioning as a key E R&D services player in the ecosystem.

      Prior to Ignitarium, Malavika has worked in with multiple global tech startups and IT consulting companies as a marketing consultant. Earlier, she headed marketing for the Semiconductor & Systems BU at Wipro Technologies and worked at IBM in their application software division.

      Malavika completed her MBA in Marketing from SCMHRD, Pune, and holds a B.E. degree in Telecommunications from RVCE, Bengaluru.



      VP - Operations

      Pradeep comes with an overall experience of 26 years across IT services and Academia. In his previous role at Virtusa, he played the role of Delivery Leader for the Middle East geography. He has handled complex delivery projects including the transition of large engagements, account management, and setting up new delivery centers.

      Pradeep graduated in Industrial Engineering and Management, went on to secure an MBA from CUSAT, and cleared UGN Net in Management. He also had teaching stints at his alma mater, CUSAT, and other management institutes like DCSMAT. A certified P3O (Portfolio, Program & Project Management) from the Office of Government Commerce, UK, Pradeep has been recognized for key contributions in the Management domain, at his previous organizations, Wipro & Virtusa.

      In his role as the Head of Operations at Ignitarium, Pradeep leads and manages operational functions such as Resource Management, Procurement, Facilities, IT Infrastructure, and Program Management office.


      SONA MATHEW Director – Human Resources


      AVP – Human Resources

      Sona heads Human Resource functions - Employee Engagement, HR Operations and Learning & Development – at Ignitarium. Her expertise include deep and broad experience in strategic people initiatives, performance management, talent transformation, talent acquisition, people engagement & compliance in the Information Technology & Services industry.


      Prior to Ignitarium, Sona has had held diverse HR responsibilities at Litmus7, Cognizant and Wipro.


      Sona graduated in Commerce from St. Xaviers College and did her MBA in HR from PSG College of Technology.



      Vice President - Sales

      As VP of Sales, Ashwin is responsible for Ignitarium’s go-to-market strategy, business, client relationships, and customer success in the Americas. He brings in over a couple of decades of experience, mainly in the product engineering space with customers from a wide spectrum of industries, especially in the Hi-Tech/semiconductor and telecom verticals.


      Ashwin has worked with the likes of Wipro, GlobalLogic, and Mastek, wherein unconventional and creative business models were used to bring in non-linear revenue. He has strategically diversified, de-risked, and grown his portfolios during his sales career.


      Ashwin strongly believes in the customer-first approach and works to add value and enhance the experiences of our customers.


      AZIF SALY Director – Sales


      Vice President – Sales & Business Development

      Azif is responsible for go-to-market strategy, business development and sales at Ignitarium. Azif has over 14 years of cross-functional experience in the semiconductor product & service spaces and has held senior positions in global client management, strategic account management and business development. An IIM-K alumnus, he has been associated with Wipro, Nokia and Sankalp in the past.


      Azif handled key accounts and sales process initiatives at Sankalp Semiconductors. Azif has pursued entrepreneurial interests in the past and was associated with multiple start-ups in various executive roles. His start-up was successful in raising seed funds from Nokia, India. During his tenure at Nokia, he played a key role in driving product evangelism and customer success functions for the multimedia division.


      At Wipro, he was involved in customer engagement with global customers in APAC and US.


      RAJU KUNNATH Vice President – Enterprise & Mobility


      Distinguished Engineer – Digital

      At Ignitarium, Raju's charter is to architect world class Digital solutions at the confluence of Edge, Cloud and Analytics. Raju has over 25 years of experience in the field of Telecom, Mobility and Cloud. Prior to Ignitarium, he worked at Nokia India Pvt. Ltd. and Sasken Communication Technologies in various leadership positions and was responsible for the delivery of various developer platforms and products.


      Raju graduated in Electronics Engineering from Model Engineering College, Cochin and has an Executive Post Graduate Program (EPGP) in Strategy and Finance from IIM Kozhikode.


      PRADEEP SUKUMARAN Vice President – Business Strategy & Marketing


      Vice President - Software Engineering

      Pradeep heads the Software Engineering division, with a charter to build and grow a world-beating delivery team. He is responsible for all the software functions, which includes embedded & automotive software, multimedia, and AI & Digital services

      At Ignitarium, he was previously part of the sales and marketing team with a special focus on generating a sales pipeline for Vision Intelligence products and services, working with worldwide field sales & partner ecosystems in the U.S  Europe, and APAC.

      Prior to joining Ignitarium in 2017, Pradeep was Senior Solutions Architect at Open-Silicon, an ASIC design house. At Open-Silicon, where he spent a good five years, Pradeep was responsible for Front-end, FPGA, and embedded SW business development, marketing & technical sales and also drove the IoT R&D roadmap. Pradeep started his professional career in 2000 at Sasken, where he worked for 11 years, primarily as an embedded multimedia expert, and then went on to lead the Multimedia software IP team.

      Pradeep is a graduate in Electronics & Communication from RVCE, Bangalore.


      SUJEET SREENIVASAN Vice President – Embedded


      Vice President – Automotive Technology


      Sujeet is responsible for driving innovation in Automotive software, identifying Automotive technology trends and advancements, evaluating their potential impact, and development of solutions to meet the needs of our Automotive customers.

      At Ignitarium, he was previously responsible for the growth and P&L of the Embedded Business unit focusing on Multimedia, Automotive, and Platform software.

      Prior to joining Ignitarium in 2016, Sujeet has had a career spanning more than 16 years at Wipro. During this stint, he has played diverse roles from Solution Architect to Presales Lead covering various domains. His technical expertise lies in the areas of Telecom, Embedded Systems, Wireless, Networking, SoC modeling, and Automotive. He has been honored as a Distinguished Member of the Technical Staff at Wipro and has multiple patents granted in the areas of Networking and IoT Security.

      Sujeet holds a degree in Computer Science from Government Engineering College, Thrissur.


      RAJIN RAVIMONY Distinguished Engineer


      Distinguished Engineer


      At Ignitarium, Rajin plays the role of Distinguished Engineer for complex SoCs and systems. He's an expert in ARM-based designs having architected more than a dozen SoCs and played hands-on design roles in several tens more. His core areas of specialization include security and functional safety architecture (IEC61508 and ISO26262) of automotive systems, RTL implementation of math intensive signal processing blocks as well as design of video processing and related multimedia blocks.


      Prior to Ignitarium, Rajin worked at Wipro Technologies for 14 years where he held roles of architect and consultant for several VLSI designs in the automotive and consumer domains.


      Rajin holds an MS in Micro-electronics from BITS Pilani.


      SIBY ABRAHAM Executive Vice President, Strategy


      Executive Vice President, Strategy


      As EVP, of Strategy at Ignitarium, Siby anchors multiple functions spanning investor community relations, business growth, technology initiatives as well and operational excellence.


      Siby has over 31 years of experience in the semiconductor industry. In his last role at Wipro Technologies, he headed the Semiconductor Industry Practice Group where he was responsible for business growth and engineering delivery for all of Wipro’s semiconductor customers. Prior to that, he held a vast array of crucial roles at Wipro including Chief Technologist & Vice President, CTO Office, Global Delivery Head for Product Engineering Services, Business Head of Semiconductor & Consumer Electronics, and Head of Unified Competency Framework. He was instrumental in growing Wipro’s semiconductor business to over $100 million within 5 years and turning around its Consumer Electronics business in less than 2 years. In addition, he was the Engineering Manager for Enthink Inc., a semiconductor IP-focused subsidiary of Wipro. Prior to that, Siby was the Technical Lead for several of the most prestigious system engineering projects executed by Wipro R&D.


      Siby has held a host of deeply impactful positions, which included representing Wipro in various World Economic Forum working groups on Industrial IOT and as a member of IEEE’s IOT Steering Committee.


      He completed his MTech. in Electrical Engineering (Information and Control) from IIT, Kanpur and his BTech. from NIT, Calicut


      SUJEETH JOSEPH Chief Product Officer


      Chief Technology Officer


      As CTO, Sujeeth is responsible for defining the technology roadmap, driving IP & solution development, and transitioning these technology components into practically deployable product engineering use cases.


      With a career spanning over 30+ years, Sujeeth Joseph is a semiconductor industry veteran in the SoC, System and Product architecture space. At SanDisk India, he was Director of Architecture for the USD $2B Removable Products Group. Simultaneously, he also headed the SanDisk India Patenting function, the Retail Competitive Analysis Group and drove academic research programs with premier Indian academic Institutes. Prior to SanDisk, he was Chief Architect of the Semiconductor & Systems BU (SnS) of Wipro Technologies. Over a 19-year career at Wipro, he has played hands-on and leadership roles across all phases of the ASIC and System design flow.


      He graduated in Electronics Engineering from Bombay University in 1991.


      SUJITH MATHEW IYPE Co-founder & CTO


      Co-founder & COO


      As Ignitarium's Co-founder and COO, Sujith is responsible for driving the operational efficiency and streamlining process across the organization. He is also responsible for the growth and P&L of the Semiconductor Business Unit.


      Apart from establishing a compelling story in VLSI, Sujith was responsible for Ignitarium's foray into nascent technology areas like AI, ML, Computer Vision, and IoT, nurturing them in our R&D Lab - "The Crucible".


      Prior to founding Ignitarium, Sujith played the role of a VLSI architect at Wipro Technologies for 13 years. In true hands-on mode, he has built ASICs and FPGAs for the Multimedia, Telecommunication, and Healthcare domains and has provided technical leadership for many flagship projects executed by Wipro.


      Sujith graduated from NIT - Calicut in the year 2000 in Electronics and Communications Engineering and thereafter he has successfully completed a one-year executive program in Business Management from IIM Calcutta.


      RAMESH SHANMUGHAM Co-founder & COO


      Co-founder & CRO

      As Co-founder and Chief Revenue Officer of Ignitarium, Ramesh has been responsible for global business and marketing as well as building trusted customer relationships upholding the company's core values.

      Ramesh has over 25 years of experience in the Semiconductor Industry covering all aspects of IC design. Prior to Ignitarium, Ramesh was a key member of the senior management team of the semiconductor division at Wipro Technologies. Ramesh has played key roles in Semiconductor Delivery and Pre-sales at a global level.

      Ramesh graduated in Electronics Engineering from Model Engineering College, Cochin, and has a Postgraduate degree in Microelectronics from BITS Pilani.