Job Specification

Purpose of Position :

To provide data leadership on projects which will help ODS clients achieve efficiencies in delivery of programs/services, expedite business processes, achieve business goals/objectives and management of processes in an Agile product development and DevOps environment.
To provide data leadership on agile product teams and ensure compliance to data architecture, standards, deployment process, security and infrastructure stability.
To provide data expertise on projects which will help ODS clients achieve business goals/objectives.
To develop, maintain, and improve data engineering services to support the sharing of data across government.
To lead, design, architect, build, deploy and operate data pipelines and infrastructure for enterprise consumption in support of advanced analytics, visualization, Artificial Intelligence (AI)/ Machine Learning (ML) products, and system interoperability.
To lead and deliver Data Warehouses, Data Lake, Self-Service Tooling, Data Models, APIs, Business Intelligence platforms and Data Solutions for multiple functional areas using cloud technologies.
To mentor staff in new data technology, tools, automation, scripting, databases, container-based virtualization tech, processes, and handling production issues.

Duties / Responsibilities :

1. Leads and coaches data engineers to translate user requirements, handle complex extract, transform, and load (ETL) architecture and data models into effective, scalable, robust ETL solutions that meet build standards and regulations.
2. Identifies re-use opportunities, lead development of reusable services / components and ensure that they are effectively implemented.
3. Translates business rules and requirements into data objects, produce associated data models and source to target mappings and write abstracted, reusable code components accordingly.
4. Builds accessible data for analysis, supporting the development and underlying data infrastructure of tools that improve our understanding of data, powering insights across products.
5. Reviews the data quality and data definitions, and perform data cleansing, validation and data management tasks to maintain data integrity and data availability.
6. Manages the continuous automated/manual testing and development of data pipeline, new data sources and analytic dashboards.
7. Leads the adoption of new data technologies and tools by providing technology orientation sessions, defining migration techniques and developing guidelines / best practices in areas of design, build, integration, deployment and operational safeguards.
8. Provides advice for the selection of reusable services / components, their creation or procurement and on-going support.
9. Participates in the creation of data standards by providing technical expertise for cloud environments, information, and security architecture. Develops templates and automation tools that will improve productivity.
10. Liaises with IT colleagues as technical expert and provides guidance and coaching, conducting reviews, and resolving operational problems, and supporting business partners in effective use of solutions.
11. Develops processes to improve the efficiency of teams. Supports team success and resolves conflicts by facilitating the expression of diverse points of view to enhance teamwork and attain a beneficial resolution.
12. Applies tools/libraries such as TensorFlow to statistical modelling, pattern recognition, supervised and unsupervised learning, data mining, predictive analysis and natural language processing.
13. Oversees delivery of data solutions in accordance with agreed organizational standards that ensure services are resilient, scalable and future-proof.
14. Advocates for data quality, data governance, compliance and data product thinking.
15. Participates in code reviews and provides technical expertise and guidance to ensure team follows engineering best practices towards ensuring performance, reliability, and scalability of the data platform.
16. Researches emerging data technologies, leads the development of prototypes, proof of concepts or feasibility studies and provides design evaluations and recommendations of appropriate technologies and tools pertinent to current and future Data standards.

Staffing and Licensing :

N/A

Knowledge :

Job requires knowledge of Data Modelling, ETL Development, and Data Warehousing.
Job requires knowledge of Enterprise Data Warehouse, Data Marts and storage, including data ingestion,
transformation, orchestration tools and ETL patterns using modern data pipelines across multiple structured and unstructured data sources.
Job requires knowledge to undertake data profiling and source system analysis and can present clear insights to colleagues to support the end use of the data.
Job requires extensive knowledge of methodologies, technologies and tools used to build enterprise data solutions and infrastructure; platform integration technologies; awareness of technology-specific standards, policies and practices for tools.
Job requires knowledge of Data Architecture patterns and how patterns are implemented in actual solutions; expert knowledge of large-scale systems analysis, design and development, application and data modelling using industry standards for modelling languages and tools, programming languages and frameworks.
Job requires knowledge of ODS programs and services, and systems that support them; information technology, application and security architecture standards; planning and development methodology.
Monitors industry and other organizations for innovations in application delivery approaches. Leads selection and implementation of new and emerging development and delivery methodologies. Monitors ODS' development environment; contributes to best practices and policies. Reviews and measures the history and evolution of application delivery processes and components.
Job requires knowledge of all major IT departments, roles, responsibilities and inter-dependencies. Advanced knowledge of existing data platforms and infrastructure across all major functions.
Job requires hands-on knowledge of common authentication technologies, security controls, and standard application security tools.
Job requires knowledge of deploying and scaling ML solutions using open-source frameworks.
Job requires knowledge of programming languages like SQL, R, MATLAB, and Python (NumPy, SciPy, Panda).
Job requires knowledge of data lineage and governance for different data sources.
Job requires knowledge of principles of infrastructure as code (e.g. Terraform), software development best practices, Continuous Integration/Continuous Delivery (CI/CD) such as Jenkins, GitHub, Puppet, Ansible, Chef, etc.) and containerisation.
Job requires in-depth knowledge of both SQL and NoSQL databases, including performance tuning and troubleshooting (e.g. ElasticSearch & mongdb)
Job requires knowledge of how to create repeatable and reusable products.
Job requires knowledge of Linux, shell scripting, building full-stack applications, logging framework such as ELK and NLP or NLTK

Skills :

Job requires hands-on experience with development and the application of elevated software design, testing, documentation, and support practices within the data engineering discipline. Mentoring and lead a team that specializes in data engineering.
Job requires technical ownership for data and data pipelines to ensure compliance with data standards, architectural standards and achievement of business requirements and Industry best practices. Hands-on experience with designing, building and testing data products based on feeds from multiple systems using a range of different storage technologies and/or access methods within the data engineering discipline.
Job requires experience working with a variety of databases, APIs, and formats to extract and transform data, including use of data collection and analysis techniques such as data mining and machine learning. Experience with implementing security in data domain, role-based access control, and data lineage.
Job requires collaborating closely with development, infrastructure and architecture teams to ensure our data engineering systems are well structured for scale, performance and cost.
Job requires written and oral communication skills to conduct team meetings, write/edit systems documentation, prepare and present written reports on findings/alternate solutions, develop Data guidelines / best practices.
Job requires interpersonal skills to explain and discuss the advantages and disadvantages of various approaches and supervise team members.
Job requires analytical skills to assess applicability of new technologies and impacts of their adoption; design and develop software compatible with client's needs; make a solid business case by summarizing findings and offering recommendations to buy or build software components; investigate and resolve complex technical Issues.
Create a mechanism to monitor effectiveness of existing evaluation processes and criteria for new technologies.
Job requires design skills that can be applied for enterprise wide solutions which are interoperable with existing and future solutions; designed, implemented and maintained DevOps continuous delivery pipelines to manage from build to deployment and cloud infrastructure using infrastructure as code practices; good understanding of platform integration technologies. Advanced expertise in database platforms, structures, migration and connectivity.
Job requires comprehensive experience in the development of complex, cross-functional environments/applications using expertise in continuous delivery to automate testing and deployment pipelines to deliver fast solutions. Job requires assessment of client information technology needs and objectives, monitoring emerging technologies and provision of consultation, information, advice and strategic recommendations to clients on acquisition/development of technology to meet their business objectives, and improve operational performance
while remaining consistent with overall ODS technology strategy.
Job requires planning and coordinating design and development work, including giving work assignments, accepting results, performing quality assurance activities, directing and mentoring team members. Ability to apply capacity planning approaches, tools and techniques for application design, deployment, and maintenance.
Advanced analytical skills to describe key issues and considerations for cross-platform interoperability.
Job requires project management skills for planning, estimating, staffing, organizing and managing multiple projects. Ability to articulate methods used to make build/buy decisions on project components. Ability to identify common IT risks and address them with contingency plans and back-out options. Creates and sustains a link between business requirements and technology production.
Holds regular and ad-hoc project reviews with product team, sponsors and clients. Articulates requirements and control expectations of both business and technical clients.
Job requires problem-solving skills to resolve complex, multi-component failures. Ability to distinguish hardware, software and data communications problems. Able to troubleshoot non-recurring and performance-related problems. Ability to discuss alternative troubleshooting approaches and their applicability to types of problems.
Organizes teams of own and vendor experts to resolve major problems. Participates in defining and monitoring problem logs, resolutions and lessons learned.
Job requires monitoring and assessing the performance of applications in cloud environment to ensure the services are available anytime, anywhere. Create, test and implement operational safeguards that maintain data integrity and protection against unauthorized access and malicious attacks.
Job requires experience with open source frameworks such as search engines, analytics including set up, configuration, integration, watcher and alerts. Fluency in scripting languages and build tools for continuous integration and provide infrastructure support for multiple cloud environments.
Job requires integration experience using and building APIs and Databases (relational and non-relational).
Job requires expertise in security standards and securing environments using offensive and defensive approaches.
Job requires experience with building dashboards, data visualizations and various messaging systems such as KAFKA, or RabbitMQ, Data Factory, Azure Data Lake, Azure SQL Server, PowerBI).
Job requires experience with Git workflows, continuous integration, testing and automated build pipelines.
Job requires senior level experience in operationalizing data platforms with Azure data solutions and using Data Factory, Azure.
Job requires experience with innovating with Cloud technologies like Docker, concurrency, scalability and fault tolerant designs in a distributed data environment.

Freedom of Action :

Job requires working within; ODS and government directives for information technology development and acquisition; and within overall ODS objectives and strategies for use of information technology.
Job oversees research of new technologies with vendors and makes authoritative recommendations to management in line with ODS decisions. Job provides senior technical leadership on application development projects.
Job requires decision-making by: defining application architecture; identifying reuse opportunities and recommending to buy or build software components; setting design and development guidelines/best practices; addressing integration and technology transition issues.
Job exercises latitude in decision-making by recommending new technologies and tools; cloud development strategies and initiatives; process and productivity improvements.
Work on significant technical issues is regularly discussed with management, vendors and other technical staff.
Job requires referring to manager problems/issues such as: conflicting client requirements, need for additional resources, performance issues and non-conformance to Data architecture standards.