Discover how migrating data pipelines with MS Copilot streamlines complex transitions, turning challenges into opportunities for optimization and enhanced documentation. This blog explores key insights and strategies from a senior data and analytics consultant at NTT DATA Business Solutions.
Migrating Data Pipelines from one platform to another with MS Copilot

Background
One of the initiatives I undertook in my capacity as a senior data and analytics consultant at NTT DATA Business Solutions was the migration of data pipelines from one platform to another. While the migration project could be challenging, at the same time it’s an opportunity to enhance, optimise, and document the migration.
My use case was to migrate the data pipelines that were created over a period using a graphical and tool-based ETL tool to PostgreSQL blocks using GitHub (version control and collaboration), S3 (code repository), and integration tools (Orchestrating/CI/CD ETL Jobs). The reason for this migration was to make the data environment more portable and technology-agnostic, as well as to minimise the effort to maintain the environment.
Pipeline migration from one platform to another
The following were my primary areas responsibilities on this migration journey
- Review Understand the complexity of the current logic of the ETL/Data Pipelines.
- Migration and Conversion Optimised Code Generation and Code Refactoring
- Optimisation Optimise the performance of the current pipeline.
- Documentation Automated Documentation Generation
Getting help from Copilot
The Practice Director of the D&A team at NTT DATA challenged me and a few others to see if GenAI tools like MS Copilot (an AI-powered assistant designed to enhance productivity, streamline tasks, and enhance the quality of deliverables) could help improve the output quality and time to complete this complex assignment.
In this blog post, I will explain how MS Copilot’s assistance saved me time in understanding, optimising, and converting the artefacts that can be used in the target environment.
- Review (MS Copilot to understand the complexity of the current pipeline) I used the MS Copilot to understand the complexity of the current graphical data pipeline. The below image shows how an image is used to explain the different steps involved in the data pipeline:
Using MS Copilot to explain the purpose and complexity of a graphical data pipeline
This helped me understand the pipeline’s components and what the job is doing at a high level. This approach also helped me define the pipeline’s complexity and estimate the effort based on the different components used in the pipeline.
- Migration and Conversion
Converting and migrating the pipelines from one environment to another is a tedious and error-prone process. By using MS Copilot, I was able to migrate and convert the pipeline at an accelerated pace.
MS Copilot helped me to convert the SQL, Python, and other source components while keeping their semantics, behaviour, and performance intact. My target environment for ETL was a PostgreSQL block.
I even went one step further i.e. getting the details of each source component, which helped migrate to the target environment. This makes it easier to adjust the code to a new environment, simplifying the migration process, improving code quality, and ensuring robustness and reliability.
Highlighting a scenario where I have to convert an ETL component ‘Table Iterator’ from my current environment into the PostgreSQL block for my migration:
Using MS Copilot to convert individual components of the source data pipeline to the target
Refactoring code is an important task, while migration is also a manual activity for the developer. I used MS CoPilot to refactor the code, which again would be time-consuming if it were to be done manually. The main goals of refactoring are to improve the code’s readability, reduce complexity, enhance maintainability, and sometimes improve performance. One of the examples below:
Using MS Copilot to refactor code – adding comments automatically is a real help
This general AI assistance helped me to focus on a broader architecture rather than line-by-line conversion, removing the friction and building multi-language and cross-platform applications across tech stacks.
- Optimisation
MS CoPilot is a significant help for optimising the code, specifically reviewing the code. In my use case, I uploaded the code from the source environment before migration and carefully reviewed the code suggestions and improvements provided by Copilot. As an output, it provides optimised code snippets and best practices to improve code’s performance and efficiency.
I incorporated the recommended suggestions by Copilot after thoroughly testing the code to ensure performance goals are met and coding standards are not compromised.
Highlighting below one of the scenarios for code optimisation and improvement whilst migrating the pipeline
Using MS Copilot to improve code performance
Code optimisation is, again, a time-consuming task, and you should be well versed with the objective of the code and understand the most critical parts of your code that will benefit the most from optimisation.
The steps that I followed to optimize my code:
- Identify the Query: Start by selecting the SQL query to be optimised. This could be a complex query that is running slowly or one that retrieves a large dataset.
- Comments for Context: Add comments in SQL code to provide context. For example, specify what the code is meant to achieve, or any specific performance issues being faced. This helps Copilot generate more relevant suggestions. Use comments like Ask Copilot for optimisation suggestions. You can do this by typing a comment like — Optimize this query above your SQL code.
- Analyse Suggestions: Review the suggestions provided by Copilot. It might recommend changes like remove nesting of code, rewrite joins, might recommend using latest functions available, adding indexes.
- Implement Suggestions and Optimisations: Apply the suggested optimisations to your SQL code. Make sure to test the new query to see if it performs better and analyses the generated query plan.
- Benchmark Performance: Run the optimised query and compare its performance to the original one. Expect improvements in execution time and resource utilisation.
If further optimisation is required, repeat the process. You can continue refining your query based on new insights or changes in your data structure.
Key benefits for me using CoPilot to optimise my code:
Speed: Copilot can suggest optimised algorithms quickly, saving you time.
Efficiency: It helps you find more efficient solutions, improving the performance of your code.
Learning: Learn new techniques and best practices by following the suggestions Copilot provides.
Scalability: This approach can handle much larger datasets without running into memory issues.
- Documentation
This could be one of any project’s most time-consuming but crucial activities. A well-documented code makes the updating and maintaining process easier and facilitates better collaboration among team members by providing clear and comprehensive documentation.
I use CoPilot in my project to create documentation for the following processes:
Data Pipelines: to document the pipeline clearly, highlighting the objective of the pipeline as a summary
Using MS Copilot to generate an English summary of a data pipeline
Code: To document the code explaining the functionality of various components of the code with comments
The image highlights the code snippet where CoPilot provided the detailed commentary and explanation:
Using MS Copilot to add commentary to code
KPI definition: CoPilot was used to convert complex code logic into a business-centric definition using natural language, which is easier to understand.
One of the scenarios explained below where CoPilot helped to summarize the complex calculation to an extent that even business stakeholders can easily understand:
Using MS Copilot to explain complex KPIs in English
Conclusion
Migrating code can be a daunting task, but with the right tools and strategies, it becomes a manageable and even rewarding process. Throughout this project, I leveraged Microsoft Copilot to streamline and enhance our migration efforts. Copilot’s intelligent code suggestions and real-time assistance significantly reduced the time and effort required to refactor and optimize our codebase.
By integrating Copilot into the migration journey, I not only improved the quality and consistency of my code but also empowered our team to focus on more complex and creative aspects of the project. This use case has set a solid foundation for future development, ensuring our code is more maintainable, scalable, and aligned with modern standards.