logo

Ideal House APP

Your Go-To Interior Design Hub
LogoIDEAL HOUSE
Home
Home
Board
Board
Explore
Explore
Log in
pricingPricingAPI
HomeHome
BoardBoard
ExploreExplore
Logo
O
Operation Sports
1/8
O
Operation Sports

Architect fault-tolerant applications with instance fleets on Amazon EMR on EC2

The article addresses strategies for optimizing capacity and architecting fault-tolerant applications on Amazon EMR on EC2. Organizations frequently utilize Amazon EMR for large-scale data processing with frameworks such as Apache Spark, Apache Hive, and Trino. However, fluctuating compute demands, often spurred by events like marketing campaigns, necessitate efficient capacity planning to avert workload failures or limits. A common challenge arises when daily Spark jobs use consistent Amazon EC2 instance types, as auto-scaling during demand spikes can lead to InsufficientCapacityExceptions (ICE) if the specific on-demand instance pool lacks the required capacity. This article provides solutions to mitigate such issues by analyzing EMR workloads and implementing tailored capacity optimization strategies. It introduces instance fleets in Amazon EMR as a flexible and robust solution for managing EC2 instances. This feature allows users to define target capacities for On-Demand and Spot Instances, select up to five (or thirty via AWS CLI/API) EC2 instance types per fleet, and utilize multiple subnets across various Availability Zones. A key capability is the support for Amazon EC2 On-Demand Capacity Reservations (ODCRs), enabling EMR clusters to align with pre-purchased EC2 capacity, thereby guaranteeing capacity for predictable workloads. The article categorizes EMR workload patterns into stable and variable (spiky) and outlines optimization techniques for each. Stable workloads, characterized by predictable resource utilization (e.g., daily processing of large datasets in pharmaceutical research), benefit from reserving baseline capacity using ODCRs. The process involves estimating baseline usage via AWS Cost and Usage Reports (AWS CUR), creating and tagging an ODCR, and configuring Amazon EMR to use these targeted ODCRs. For stable workloads, the article recommends activating the `aws:elasticmapreduce:job-flow-id` cost allocation tag in AWS CUR to track compute resources. A provided SQL query for Amazon Athena helps analyze historical instance usage to establish a baseline. It further details the creation of targeted ODCRs using the AWS CLI, illustrating how instances can be launched against these reservations. The importance of associating ODCRs with resource groups for better management and cleanup is also highlighted, along with the necessary IAM permissions for the EMR service role to utilize capacity reservations. The article explains how to configure the EMR cluster to use ODCRs with instance fleets, emphasizing the decision-making process of Amazon EMR when prioritizing targeted capacity reservations during cluster provisioning. Spiky workloads, marked by unpredictable and significant fluctuations in processing demands (e.g., real-time processing of user location data during peak hours), are addressed through a combination of instance flexibility, intelligent subnet selection, and managed scaling. Running spiky workloads with limited instance type or Availability Zone flexibility can lead to ICE errors during scaling. To counter this, the article suggests using diverse instance types, prioritized allocation strategies, and Availability Zone flexibility to enhance availability and balance price-performance. It explains how Amazon EMR instance fleets support up to 30 instance types with weighted capacities and spot bid prices, allowing for more cost-effective and faster capacity acquisition. The prioritized allocation strategy, introduced in August 2024, enables EMR to allocate capacity to higher-priority instances first, improving cost savings and reducing launch times. The article also recommends combining new-generation instances for price-performance with previous-generation instances for broader availability and suggests fixing instance sizes for latency-sensitive workloads. To ensure resilience, the article advises specifying multiple EC2 subnets across different Availability Zones during cluster creation, enabling Amazon EMR to select a subnet with adequate IP addresses. Managed scaling is presented as a mechanism to automatically adjust the number of instances based on workload demands, scaling up during high demand and down during idle periods to optimize costs and performance. The article concludes by recommending a hybrid approach for managing both baseline and spiky workloads in Amazon EMR: using ODCRs for consistent baseline capacity and configuring instance fleets with a strategic mix of ODCR, On-Demand, and Spot Instances, prioritizing ODCR usage. It stresses the importance of continuous monitoring and configuration adjustments based on specific workload patterns and business requirements. #AmazonEMR #EC2 #FaultTolerance #InstanceFleets #OnDemandCapacityReservations #CapacityPlanning #BigData #CostOptimization #CloudComputing #AWSCLI #AmazonEMR #EC2 #FaultTolerance #InstanceFleets #OnDemandCapacityReservations #CapacityPlanning #BigData #CostOptimization #CloudComputing #AWSCLI
7 months ago
Read Full Article  
No comments yet
You may also like
Let’s Architect! Getting started with containers
Let’s Architect! Getting started with containers
O
Operation Sports
Efficient and cost-effective rendering pipelines with Blender and AWS Batch
Efficient and cost-effective rendering pipelines with Blender and AWS Batch
F
flatlandkc.org
Let’s Architect! Designing serverless solutions
Let’s Architect! Designing serverless solutions
O
Operation Sports
Let’s Architect! Security in software architectures
Let’s Architect! Security in software architectures
O
Operation Sports
Let’s Architect! Resiliency in architectures
Let’s Architect! Resiliency in architectures
O
Operation Sports
Let’s Architect! Designing microservices architectures
Let’s Architect! Designing microservices architectures
O
Operation Sports
Let’s Architect! Serverless developer experience in AWS | AWS Architecture Blog
Let’s Architect! Serverless developer experience in AWS | AWS Architecture Blog
O
Operation Sports
Let’s Architect! Getting started with containers
This blog post, authored by Luca Mezzalira, Federica Ciuffo, Laura Hyatt, Vittorio Denti, and Zamira Jaupaj, focuses on the benefits and resources for using containers on Amazon Web Services (AWS) for cloud-native application development and modernization. It highlights that most AWS customers building or modernizing applications choose containers for microservices to accelerate innovation, reduce time to market, and lower total cost of ownership. The post emphasizes that using containers with AWS services provides enhanced portability, scalability, flexibility, load balancing, auto-scaling, and service discovery, simplifying application deployment and management at scale. The article introduces several key resources to help users get started with containers on AWS. The first resource is the Container Build Lens, a whitepaper that is part of the AWS Well-Architected Framework. This document offers general design principles, best practices, and implementation guidance for containers, structured around the Six Pillars of the Well-Architected Framework, to help customers review and improve their cloud architectures and understand the business impact of their design choices. Another significant resource presented is the EKS Workshop, designed to familiarize users with Amazon Elastic Kubernetes Service (Amazon EKS). This workshop provides practical, real-world use cases and is organized into high-level learning modules such as Networking, Security, and DevOps Automation. These modules are further divided into standalone labs focusing on specific features, tools, or use cases. For building environments, the post recommends Amazon EKS Blueprints, a collection of Infrastructure as Code (IaC) modules available in Terraform and CDK, which facilitate configuring and deploying consistent, best-practice Amazon EKS clusters across various accounts and regions. Additionally, the blog post highlights an article titled "Architecting for resiliency on AWS App Runner." This resource demonstrates how to design highly available and resilient applications using AWS App Runner. It explains that App Runner simplifies the complexities of running containerized applications by abstracting away the underlying cloud resources, managing aspects like load balancers, TLS certificates, auto-scaling, logs, and metrics. This allows developers to concentrate on implementing business logic within a scalable and elastic environment. Finally, the article addresses the crucial aspect of security with a video session titled "Securing Kubernetes: How to address Kubernetes attack vectors." This session covers the fundamentals of Kubernetes architecture and common attack vectors. It also details the security controls offered by Amazon EKS and provides strategies to mitigate risks for Kubernetes-based workloads. The authors, all AWS experts, contributed their knowledge in areas ranging from solution architecture and container services to machine learning and enterprise solutions, ensuring a comprehensive overview of container technologies on AWS. #AWS #Containers #Kubernetes #AmazonEKS #AWSWellArchitected #CloudNative #Microservices #DevOps #CloudSecurity
O
Operation Sports
Let’s Architect! Designing serverless solutions
This blog post, "Let’s Architect! Designing serverless solutions," explores the architectural considerations and benefits of leveraging AWS serverless services for building event-driven architectures. The article highlights Werner Vogels' re:Invent 2022 keynote, where he emphasized the asynchronous nature of modern workloads and how serverless technologies can address these challenges. The authors, Luca Mezzalira, Federica Ciuffo, Laura Hyatt, Vittorio Denti, and Zamira Jaupaj, delve into the in-depth architecture of serverless AWS services like AWS Lambda, EventBridge, and AWS Step Functions. The post emphasizes that AWS serverless services enable users to focus on the asynchronous aspects of their workloads, facilitating the implementation of event-driven architectures and effective integration patterns for communication both within and beyond defined boundaries. It introduces a new workshop dedicated to design patterns that utilize serverless AWS services, providing practical insights into building event-driven systems. The article includes several key resources and examples to illustrate these concepts. One such resource is a video offering a detailed look at how AWS Lambda functions operate internally. This video is presented as a valuable companion for those aiming to master Lambda architecture, enabling them to design optimized function code for efficient performance and resource utilization. Understanding Lambda functions is crucial for identifying and applying the most suitable approaches to cloud workloads, leading to agile and robust cloud infrastructure tailored to specific project requirements. Another example cited is an event-driven serverless story generation application that integrates ChatGPT and DALL-E. This application showcases the potential of combining AWS serverless services—including Amazon EventBridge, Amazon DynamoDB, AWS Lambda, Amazon S3, and AI services like Amazon Polly—with advanced AI technologies. The architecture allows for scheduled content generation, with EventBridge triggering a Lambda function nightly to create new stories. The compatibility of AI services with the serverless model demonstrates efficient and dynamic content creation, highlighting the innovative possibilities when combining event-driven serverless architectures with cutting-edge AI. Furthermore, the AWS Workshop Studio: Serverless Patterns is presented as a comprehensive learning experience for understanding architectural patterns in serverless projects. This workshop covers various synchronous and asynchronous implementation patterns, offering production-ready knowledge, including testing serverless workloads and establishing automation pipelines. The post also refers to Serverlesspresso, an event-driven serverless workload utilizing EventBridge and AWS Step Functions to manage microservice events and process thousands of orders daily. This example provides insights into design considerations, development processes, and lessons learned from building a production-ready solution, emphasizing practical patterns and extensibility for robust, scalable, and cost-effective applications. The combination of EventBridge and Step Functions is shown to be effective in addressing complex architectural challenges in large-scale applications. In summary, the article advocates for the adoption of serverless solutions on AWS, detailing how these services support event-driven architectures, optimize performance, and integrate with advanced AI tools. It provides practical learning opportunities through workshops and real-world examples, reinforcing the importance of asynchronous design and efficient resource management in modern cloud environments. #Serverless #AWSLambda #EventDrivenArchitecture #AWSStepFunctions #AmazonEventBridge #CloudArchitecture #AIIntegration #Microservices
O
Operation Sports
Let’s Architect! Resiliency in architectures
This blog post, "Let’s Architect! Resiliency in architectures," authored by Luca Mezzalira, Federica Ciuffo, Vittorio Denti, and Zamira Jaupaj, focuses on the importance of building resilient architectures, particularly within the context of the AWS Cloud platform. The article emphasizes that resilience is crucial for ensuring the continuous availability and performance of digital services, and it aligns with the AWS Well-Architected Framework's definition of resilience as the ability to recover from stress, attacks, and component failures. The authors aim to provide best practices for constructing resilient architectures to maintain uninterrupted service operations. The blog post highlights five distinct resilience patterns, each possessing unique strengths and trade-offs. Architects are encouraged to understand these patterns to tailor their resilience strategies to the specific requirements of their applications and business objectives. By thoughtfully applying these patterns, organizations can develop cloud architectures that offer high availability and efficient recovery capabilities, thereby minimizing the impact of potential disruptions. This approach ensures that systems can withstand various challenges and continue functioning as expected. Marc Broker's insights on timeouts, retries, and backoff with jitter are also discussed as essential tools for building system resilience. Failures are acknowledged as an inevitable part of system operation, and the article stresses the importance of designing systems to effectively cope with them. Timeouts prevent processes from hanging indefinitely, retries allow systems to reattempt operations that might have failed due to transient issues, and backoff mechanisms, especially with jitter, help to prevent overwhelming systems with repeated requests during recovery. These techniques are fundamental for distributing traffic, preventing system overloads, and maintaining high availability. The AWS Resilience Hub is presented as a key platform for protecting mission-critical applications from downtime risks. This centralized platform enables users to define, validate, and track the resilience of their applications. The article refers to an AWS Online Tech Talk that demonstrates how the Resilience Hub can be used to implement strategies for avoiding disruptions caused by software, infrastructure, or operational issues. The Resilience Hub is highlighted as a valuable resource for monitoring and implementing robust architectural resilience. Data resiliency is further explored through a re:Invent 2022 session, which discusses its significance for customers and how it can be integrated into application architecture. This session provides a comprehensive overview of data resiliency, outlining its core components and its vital role in modern application development. It covers application data resiliency and protection designs, extending from the native data resiliency features of AWS storage to disaster recovery solutions using AWS Elastic Disaster Recovery. The discussion underscores the necessity of robust data protection strategies to ensure business continuity and data integrity. The article concludes by inviting readers to explore further resources on architecture resiliency and anticipating a future discussion on security on AWS. #Architecture #AWSWellArchitectedFramework #Resilience #ResilienceHub #Timeouts #Retries #Backoff #Jitter #DataResiliency
O
Operation Sports
Let’s Architect! Designing microservices architectures
This blog post, a continuation of the "Let's Architect!" series, focuses on the complexities and best practices of designing microservices architectures. It expands upon a previous post that explored microservices with containers, delving deeper into the specific challenges faced by software architects and engineers when developing large-scale distributed systems composed of independent services. The authors, Luca Mezzalira, Federica Ciuffo, Laura Hyatt, Vittorio Denti, and Zamira Jaupaj, highlight several key areas critical to successful microservice implementation. The article begins by addressing the need for effective methods to authorize, route, and monitor network traffic between these specialized services, as well as the ability to identify the root cause of issues at various levels. Amazon VPC Lattice is introduced as a solution that offers a consistent way to connect, secure, and monitor communication across instances, containers, and serverless functions. This service enables the definition of policies for traffic management, advanced routing, and network access, facilitating discoverability and real-time monitoring of traffic flow within complex applications. This capability is essential for maintaining control and visibility in a distributed microservices environment. Next, the post explores application integration patterns, emphasizing the importance of loosely coupled integration for designing independent and highly available systems. The discussion highlights asynchronous communication as a particularly suitable paradigm for microservices, contributing to the overall reliability and resilience of the system landscape. While various integration approaches exist, the article stresses the significance of selecting the most appropriate method for specific scenarios. It encourages the use of cloud-native and serverless services in real-world use cases, providing guidance on choosing suitable integration technologies. The article further examines design patterns for achieving success in serverless microservices. It notes that well-established patterns, such as CQRS (Command Query Responsibility Segregation) and Event Sourcing, are crucial for decoupling and scaling systems effectively. Event Sourcing, for instance, stores data as a series of events rather than direct data store updates, allowing microservices to replay events to compute their data stores' state. The post also touches upon how data access patterns can influence polyglot persistence, which involves using multiple data storage technologies within an application based on their suitability for different types of data and access patterns. Practical demonstrations and examples from the AWS console are provided to illustrate these concepts. Finally, the blog post addresses the critical issue of avoiding overload in distributed systems, particularly when a smaller service is subjected to a large influx of requests from a larger client service, such as a data plane overwhelming a control plane. Drawing insights from the Amazon Builder’s Library, the authors discuss the risks associated with disproportionate service scaling and offer mental models and design strategies to prevent system overload. One suggested approach involves the control plane polling an Amazon S3 bucket, where data plane servers periodically write their operational state, to stay updated and prevent overwhelming the smaller service. These strategies are beneficial for teams working on microservices architectures to ensure system stability and performance. #MicroservicesArchitecture #DistributedSystems #AmazonVPCLattice #AsynchronousCommunication #ServerlessMicroservices #EventSourcing #DesignPatterns #AWSCloud #SystemScalability
O
Operation Sports
+2
Let’s Architect! Serverless developer experience in AWS | AWS Architecture Blog
This article provides a comprehensive overview of various tools and techniques designed to enhance the serverless developer experience on AWS. It caters to both new and experienced developers, aiming to accelerate the feedback loop from code development to production deployment. The discussion highlights several key AWS offerings and open-source projects that streamline the creation, testing, debugging, and optimization of serverless applications, particularly those utilizing AWS Lambda and Amazon DynamoDB. The AWS Toolkit is presented as a significant enhancement for AWS Lambda developers, offering an integrated development environment (IDE) experience. This toolkit allows developers to create, test, and debug Lambda functions locally, eliminating the need for constant cloud deployments during the initial development phases. Key features include local invocation of Lambda functions, real-time testing and debugging capabilities, and intelligent code completion with inline documentation for AWS SDK calls. These improvements contribute to faster iteration cycles, deeper insights into function behavior, and a more intuitive and productive environment for building cloud-native solutions. Another innovative tool featured is Amazon Q Developer, an AI-powered assistant that significantly streamlines various aspects of the software development lifecycle. It assists with tasks ranging from code writing and documentation to generating unit tests, thereby reducing the time developers spend on routine activities. The article specifically demonstrates how Amazon Q Developer can facilitate Test Driven Development (TDD) by generating relevant tests, boosting developer productivity, and enabling a greater focus on creative problem-solving and code quality. For optimizing Lambda function performance and cost efficiency, AWS Lambda Power Tuning is introduced as a crucial tool. This solution addresses the common issue of developers relying on guesswork when allocating memory to Lambda functions. Lambda Power Tuning automates the process of testing functions across different memory configurations, helping identify the optimal balance between performance and cost. This data-driven approach ensures functions run at peak efficiency, potentially lowering operational costs and improving response times. The article emphasizes the importance of regular power tuning to adapt to evolving application requirements and usage patterns. NoSQL Workbench for Amazon DynamoDB is highlighted as an essential tool for local development with DynamoDB. This graphical interface simplifies the interaction with DynamoDB tables, allowing developers to visually design, create, and modify table structures without direct interaction with the AWS Console. It includes features for data modeling, quick data insertion for testing, and real-time visualization of query results, thereby accelerating the development and debugging processes for DynamoDB-backed applications. Finally, AWS Lambda Powertools is presented as an open-source project that significantly enhances observability and other aspects of AWS Lambda functions. Available in multiple programming languages, Powertools facilitates structured logging, custom metric creation, and distributed tracing with minimal configuration. Beyond observability, it offers utilities for parameter store and secrets management, idempotency helpers for reliable execution, and event handler functions that reduce boilerplate code. This suite of tools enables developers to build production-ready Lambda functions more efficiently, allowing them to concentrate on core business logic while ensuring robust, performant, and maintainable serverless applications. The AWS Serverless Developer Experience workshop is also mentioned as a resource that integrates these tools and techniques into a hands-on guide for skill development. #ServerlessDevelopment #AWSLambda #AmazonDynamoDB #DeveloperTools #AIPoweredAssistance #CloudComputing #CodeOptimization #Observability #TestDrivenDevelopment
O
Operation Sports
Efficient and cost-effective rendering pipelines with Blender and AWS Batch
This blog post explores how visual effects rendering, a computationally intensive task, can be performed efficiently and cost-effectively using AWS Batch and AWS Step Functions. Rendering, the process of generating 2D or 3D images from models, is widely used in video game design, visual effects, and animated movie compositing. The article details a method for parallelizing rendering workloads and producing animations, leveraging AWS Batch for managing rendering jobs on Amazon EC2, and AWS Step Functions for coordinating the workflow's dependencies. A significant advantage highlighted is the use of Amazon EC2 Spot Instances, which can reduce compute costs by up to 90% compared to On-Demand prices. The core principle of the proposed rendering pipeline relies on the parallelizable nature of rendering: each frame of an animation can be processed independently. This characteristic makes it ideal for AWS Batch, which provides fully managed batch processing and dynamically provisions compute resources. Blender, an open-source 3D creation suite, is presented as a suitable tool for this pipeline, although AWS Thinkbox Deadline is mentioned for professional VFX teams. By rendering individual frames as separate tasks, AWS Batch can launch numerous jobs concurrently, significantly speeding up the rendering process. The workflow comprises three main stages: extracting the number of frames from a Blender file using an AWS Lambda function and the Blender Python library, submitting a rendering array job where each child job renders a single frame, and finally, stitching these rendered frames into a complete MP4 video. The independence of frame rendering jobs also allows for efficient retry mechanisms in case of failures. The article provides detailed guidance on configuring compute environments in AWS Batch. It differentiates between On-Demand and Spot Compute Environments (CEs), explaining their optimal use cases. On-Demand CEs, with a `BEST_FIT_PROGRESSIVE` allocation strategy, are recommended for critical, serial tasks like video stitching, where uninterrupted processing is crucial. Spot CEs, utilizing `SPOT_CAPACITY_OPTIMIZED` strategy, are advocated for the parallelizable rendering tasks due to their cost-effectiveness, despite the possibility of interruptions. The configuration examples using AWS Command Line Interface (CLI) demonstrate how to set up these environments, including specifying instance types, vCPU limits, and subnets. The post also explains how to combine these different CEs into a single job queue with varying priorities, allowing for a balanced approach where high-priority tasks use On-Demand instances and less critical, but more numerous, tasks leverage the cost savings of Spot Instances. AWS Step Functions is introduced as the orchestration tool for the rendering pipeline. Its low-code, visual workflow service integrates seamlessly with AWS Batch through optimized integrations. The state machine defined in AWS Step Functions manages the entire process, from extracting frame counts to submitting rendering jobs and initiating the final stitching. A key feature discussed is the ability to configure job retries within AWS Step Functions and AWS Batch. This retry mechanism is particularly beneficial when using Spot Instances, as it allows jobs to automatically restart if an instance is interrupted, ensuring the overall workflow's completion without manual intervention. The post concludes by emphasizing the benefits of this integrated approach in achieving efficient and cost-effective rendering pipelines and refers readers to a workshop for hands-on experience. #AWSBatch #AWSStepFunctions #BlenderRendering #CostEffectiveCloud #HPC #SpotInstances #VFXPipeline #CloudComputing #WorkflowOrchestration
F
flatlandkc.org
Let’s Architect! Security in software architectures
This blog post explores fundamental security practices for cloud workloads, focusing on network and application security within the Amazon Web Services (AWS) ecosystem. It highlights architectural practices and services offered by AWS to enhance security. The discussion begins by emphasizing the critical role of security in all product and service development, whether for backend systems, data, or machine learning components. The article refers to a previous post, "Let’s Architect! Architecting for Security," and aims to delve deeper into practical security measures. The post outlines strategies for protecting workloads against common attacks, introducing the Zero Trust principle and its implementation within AWS. A re:Invent session from 2022 is referenced, providing insights into the most prevalent threat vectors and vulnerabilities encountered by AWS customers. This session details how these threats are executed by attackers, the weaknesses they exploit, and the AWS-provided solutions to mitigate such security issues. Adopting suitable services and following architectural practices are deemed essential for fundamental security. The Zero Trust security model is presented as a superior alternative to traditional network perimeter models, leading to higher security outcomes. An AWS re:Invent 2022 session defines the Zero Trust models, explains their practical implementation, and showcases how AWS internally applies this principle. A case study featuring Delphix demonstrates the successful production deployment of Zero Trust, illustrating how architectural designs can integrate these security pillars effectively. Container security is also a significant topic, given its importance in modern distributed systems. The article points to an AWS re:Inforce 2023 session that provides a deep dive into container security on AWS. It explains how services like Amazon Elastic Kubernetes Service (EKS) and Amazon Elastic Container Service (ECS) leverage Linux security-layer protections, adhering strictly to the principle of least privilege. This approach significantly reduces the attack surface by restricting permissions and privileges for processes, thereby maintaining system integrity. Finally, the blog post addresses the secure management of secrets, which are crucial for accessing confidential systems and resources. It identifies common anti-patterns in organizational secrets management, such as sharing plaintext secrets through insecure channels or neglecting regular rotation. Detailed guidance is provided on discovering, classifying, implementing, and migrating secrets to AWS Secrets Manager, ensuring a secure and consistent approach to secrets management. The article concludes by inviting readers to explore further content on cost optimization, as part of the "Let’s Architect!" series on the AWS Architecture Blog. #SoftwareArchitecture #CloudSecurity #ZeroTrust #ContainerSecurity #AWSSecretsManager #CybersecurityBestPractices #AWSArchitecture #CloudWorkloads #DataSecurity
O
Operation Sports