<
Neerav Mehta
Founder & CEO
The integration enables the app platform to: - Pull out content courses from the development partner. - Filter and display courses based on users' skills, goals, and preferences. - Allow employees to click on the course link from the app platform and navigate directly to the development - partner's course detail page.
The course lists need to be fetched using two API endpoints (i.e., recommended courses and content filter query). The recommended courses should be shown on priority, followed by content courses.
To get the final result for getting the recommended course list, we made 3 API calls. The first API call was to get the list. Then, the second API call was made for getting skills per course from the first API call list. Finally, the final API call was made for fetching details of filtered courses only. However, for fetching both lists, the app took over 20 seconds. That kind of processing time is way too long. Ideally, the app platform should process a batch of content concurrently to minimize the overall processing time.
When you need to make concurrent API calls for content, you wouldn't want to wait for each call to finish. Therefore, the ideal methodology is to process a batch of content concurrently. For example, details of 30 courses are called in one go instead of individually so that the overall processing time is substantially reduced.
We can use concurrent future ThreadPool Python library to run both the API calls (recommended courses and regular content courses) paralleling to reduce the process timing. Using this method, we can decrease the total time to about 12 seconds— that's a 40% improvement.
However, the app takes longer than expected, even after adding concurrent futures. As a resolution, debugging the code further can help find the root cause of the problem.
During debugging, we checked which functions were taking longer to perform. We found out an API call was being made in a loop to fetch skills per course. To reduce the execution time for the loop, we added the ThreadPool method. As a
Sample code of the method:
with concurrent.futures.ThreadPoolExecutor(max_workers=recommended_num) as executor: futures = [executor.submit(matching_schema, course) for course in filtered_matched_course_list] for future in concurrent.futures.as_completed(futures): course = future.result() matched_course_list.append(course)
The below recommended course flow chart shows the flow of the function. These actions are performed to get the recommended course list.
The below flow chart shows the flow of the function to fetch the list based on filter query.
The below flow chart shows the flow of the function to get detailed information about a course.
Conclusion
The ThreadPool method can be used for functions and loops. Yet, it is more useful for loops. As a result, we can substantially reduce the app execution time using the method. Consequently, regardless of increasing the limit of the courses fetched, it takes approximately the same time to fetch all recommended and content courses.
Neerav Mehta
Neerav Mehta is the Founder & CEO of Red Crackle. With sterling qualities, Neerav’s technological acumen is firing a generation of progressive companies on the digital path. With an undergraduate degree in Electrical Engineering from India's most prestigious institution IIT Bombay and having spent seven years developing and contributing to the launch of AMD's innovative line of computer products, Neerav founded Red Crackle where he is lauded for his dynamic and innovative genius.
Let’s get you started!