Data Engineer with Python
6 months ago
Develop and implement data pipelines for ingesting and collecting data from various sources into a centralized data platform.
- Develop and maintain ETL jobs using AWS Glue services to process and transform data at scale.
- Optimize and troubleshoot AWS Glue jobs for performance and reliability.
- Utilize Python and PySpark to efficiently handle large volumes of data during the ingestion process.
- Design and implement scalable data processing solutions using PySpark to transform raw data into a structured and usable format.
- Create and maintain ETL processes using Python and PySpark to move and transform data between different systems.
- Optimize ETL workflows for performance and efficiency.
- Collaborate with data architects to design and implement data models that support business requirements.
- Ensure data structures are optimized for analytics and reporting.
- Work with distributed computing frameworks, such as Apache Spark, to process and analyze large-scale datasets.
- Manage and optimize databases, both SQL and NoSQL, to support data storage and retrieval needs.
- Implement indexing, partitioning, and other database optimization techniques.
- Collaborate with cross-functional teams, including data scientists, analysts, and business stakeholders, to understand data requirements and deliver effective solutions.
- Implement monitoring solutions to track data pipeline performance and proactively identify and address issues
- Ensure compliance with data privacy regulations and company policies.
- Stay abreast of industry trends and advancements in data engineering, Python, and PySpark.
**Requirements**:
- Proficiency in Python and PySpark.
- Strong knowledge of data engineering concepts and best practices.
- Hands-on experience with AWS Glue and other AWS services.
- Experience with big data technologies and distributed computing.
- Familiarity with database management systems (SQL and NoSQL).
- Understanding of ETL processes and data modeling.
- Excellent problem-solving and analytical skills.
- Strong communication and collaboration skills.
- Bachelor's degree in Computer Science, Information Technology, or a related field.
-
Python Backend Developer
1 month ago
Jakarta, Jakarta, Indonesia IlmuOne Data Full timeBackend Engineer at IlmuOne Data IlmuOne Data is seeking a talented and experienced Backend Engineer to join our growing team. As a key member of our engineering department, you will play a vital role in designing, developing, and maintaining the robust server-side infrastructure that powers our data-driven applications. Responsibilities: Design, develop,...
-
Data Engineer
5 months ago
Jakarta, Indonesia IlmuOne Data Full timeResearch, build and maintain the infrastructure required for optimal extraction, transformation, and loading of data from a wide variety of data sources using JavaScript or other languages - Contributes to team effort by accomplishing related results as needed. - Explore/learn new technologies that can complement or replace our current stack to improve it. -...
-
Data Engineer
5 months ago
Jakarta, Indonesia IlmuOne Data Full timeJob Description:Research, build and maintain the infrastructure required for optimal extraction, transformation, and loading of data from a wide variety of data sources using JavaScript or other languagesContributes to team effort by accomplishing related results as needed.Explore/learn new technologies that can complement or replace our current stack to...
-
Python Data Scientist
5 months ago
Jakarta, Indonesia Nucleus Health Full time**Responsibilities**: - Write effective, high-quality Python code to solve problems - Clearly explain the reasoning behind the code in Jupyter notebooks - Debug code and produce thorough documentation - Using free datasets (such as Kaggle, the UN, the US government, etc.), use your data analytic skills to formulate and respond to important business issues -...
-
Data Architect
3 weeks ago
Jakarta, Jakarta, Indonesia IlmuOne Data Full timeJob Title: Data EngineerAt IlmuOne Data, we are seeking a highly skilled Data Engineer to join our team. The successful candidate will be responsible for designing, building, and maintaining the infrastructure required for optimal data extraction, transformation, and loading from a wide variety of data sources using JavaScript or other languages.Key...
-
Data Architect
1 month ago
Jakarta, Jakarta, Indonesia IlmuOne Data Full timeJob Title: Data EngineerWe are seeking a highly skilled Data Engineer to join our team at IlmuOne Data. As a Data Engineer, you will be responsible for designing, building, and maintaining the infrastructure required for optimal data extraction, transformation, and loading from various data sources.Key Responsibilities:Design and implement data pipelines...
-
Data Architect
2 months ago
Jakarta, Jakarta, Indonesia IlmuOne Data Full timeJob DescriptionWe are seeking a highly skilled Data Engineer to join our team at IlmuOne Data. As a Data Engineer, you will be responsible for designing, building, and maintaining the infrastructure required for optimal data extraction, transformation, and loading from various data sources using JavaScript and other languages.Key ResponsibilitiesDesign and...
-
IT Data Engineer
5 months ago
Jakarta, Indonesia PGI Data Full timeDevelop, construct, test and maintain cloud architectures to nurture data-focused products and internal tools Align said architecture with business requirements Acquire and integrate data from external sources Bachelor’s degree in any major (preferably in Informatics, Computer Science, Software Engineer) 2-4 years experience Good knowledge in Data...
-
Data Engineer
5 months ago
Jakarta, Indonesia Group Hijra (Hijra Bank, ALAMI P2PL) Full timeHijra (previously known as the ALAMI Group) is a financial technology company that follows sharia principles. It was founded in 2018 by Dima Djani, Harza Sandityo, and Bembi Juniar. The company offers a range of services, including a peer-to-peer lending platform, mobile banking app, and mortgage all of which are based on sharia principles. Hijra has a...
-
Data Engineer
6 months ago
Jakarta, Indonesia sribu.com Full timeThe Data Engineer will support our software developers, database architects, data analysts and data scientists on data initiatives and will ensure optimal data delivery architecture is consistent throughout ongoing projects. They must be self-directed and comfortable supporting the data needs of multiple teams, systems and products. **Qualification**: -...
-
Python Data Scientist/analyst
5 months ago
Jakarta, Indonesia Aera Technology & Consulting Full time**Responsibilities**: - Write effective Python code to tackle complex issues, but also use your business sense and analytical abilities to glean valuable insights from public databases - Communicate clearly with researchers and help the organization in realizing its objectives - Clearly express the reasoning and logic when writing code in Jupyter notebooks,...
-
Data Architect
2 months ago
Jakarta, Jakarta, Indonesia IlmuOne Data Full timeAbout the RoleWe are seeking a highly skilled Data Engineer to join our team at IlmuOne Data. As a Data Engineer, you will play a critical role in designing, building, and maintaining our data infrastructure.Key ResponsibilitiesData Infrastructure Development: Design, build, and maintain the infrastructure required for optimal extraction, transformation, and...
-
Python Engineer for Machine Learning
5 months ago
Jakarta, Indonesia Permana Solutions Full timeCraft great code, empathize with our users, review pull requests, share your knowledge, and learn together with other team members - Build, deploy and monitor scalable, stable Python services - Develop and maintain robust data pipelines to collect, manipulate, and analyze large datasets. - Implement and optimize machine learning algorithms in Python. -...
-
Data Architect
2 months ago
Jakarta, Jakarta, Indonesia IlmuOne Data Full timeAbout the RoleWe are seeking a highly skilled Data Engineer to join our team at IlmuOne Data. As a Data Engineer, you will play a critical role in designing, building, and maintaining the infrastructure required for optimal data extraction, transformation, and loading from various data sources.Key ResponsibilitiesData Infrastructure Development: Design,...
-
Python Engineer for Cyber Security Product
5 months ago
Jakarta, Indonesia Autobahn Security GmbH Full timeAbout Autobahn Security Autobahn Security is a technology start-up with a vision to make cybersecurity and vulnerability management accessible to all businesses. Our SaaS product has been developed by some of the world’s finest cybersecurity researchers and experts. Our SaaS tool allows both cybersecurity experts and non-experts to embark on a journey to...
-
Python Test Engineer
5 months ago
Jakarta, Indonesia ATI Business Group Full timeWe are excited to invite a talented Python Test Engineer to join our dynamic team and contribute to our mission of delivering top-notch software solutions. As a Python Test Engineer, you will play a crucial role in ensuring the quality and reliability of our software through the implementation of various automated testing methodologies. If you have a passion...
-
Cloud Data Engineer
5 months ago
Jakarta, Indonesia IlmuOne Data Full timeWe are seeking a skilled Cloud Data Engineer to join our team. In this role, you will be responsible for designing, building, and maintaining cloud-based data infrastructure and services that enable our organization to process, store, and analyze large volumes of data. You will work closely with data scientists, data analysts, and other stakeholders to...
-
Cloud Data Engineer
5 months ago
Jakarta, Indonesia IlmuOne Data Full timeWe are seeking a skilled Cloud Data Engineer to join our team. In this role, you will be responsible for designing, building, and maintaining cloud-based data infrastructure and services that enable our organization to process, store, and analyze large volumes of data. You will work closely with data scientists, data analysts, and other stakeholders to...
-
Senior GIS Data Analyst
3 weeks ago
Jakarta, Jakarta, Indonesia IlmuOne Data Full timeJob Title: Senior GIS Data AnalystThe Senior GIS Data Analyst will play a key role in developing and integrating machine learning models with geographic information systems (GIS) for the analysis, visualization, and processing of spatial data. This role requires collaboration with cross-functional teams to deliver data-driven solutions, automate geospatial...
-
Junior Gis Ml Engineer
2 months ago
Jakarta, Indonesia IlmuOne Data Full timeThe Junior GIS ML Engineer will assist in developing and integrating machine learning models with geographic information systems (GIS) for the analysis, visualization, and processing of spatial data. This role requires collaboration with cross-functional teams to deliver data-driven solutions, automate geospatial tasks, and provide insights from spatial...