# Brian Pfeil > Personal blog and technical notes by Brian Pfeil. Topics include cloud architecture, AWS, serverless, JavaScript, Go, and software engineering. https://brianpfeil.com/ ## About Hello, I’m Brian Pfeil. I’m passionate about technology and building things. This is my personal blog where I write about those and other interests. I currently work at Merck as a Principal Cloud Architect. I’m an AWS Community Builder (Serverless) Instagram LinkedIn X (Twitter) Facebook GitHub Stack Overflow ## Posts ### Apache Kafka - URL: https://brianpfeil.com/post/apache-kafka/ - Date: 2024-02-20 - Tags: streaming code for article  pfeilbr/apache-kafka-playground learn APACHE KAFKA Kafka three key capabilities: To publish (write) and subscribe to (read) streams of events, including continuous import/export of your data from other systems. To store streams of events durably and reliably for as long as you want. To process streams of events as they occur or retrospectively. concepts KAFKA CONNECT tool for scalably and reliably streaming data between Apache Kafka and other systems. It makes it simple to quickly define connectors that move large collections of data into and out of Kafka. KAFKA STREAMS client library for building real-time applications, where the input and/or output data is stored in Kafka clusters. Kafka Streams combines the simplicity of writing and deploying standard Java and Scala applications on the client side with the benefits of Kafka’s server-side cluster technology to make these applications highly scalable, elastic, fault-tolerant, distributed, and much more https://github.com/apache/kafka/blob/3.6/streams/examples/src/main/java/org/apache/kafka/streams/examples/wordcount/WordCountDemo.java resources kafka introduction APACHE KAFKA QUICKSTART --- ### Apache Iceberg - URL: https://brianpfeil.com/post/apache-iceberg/ - Date: 2024-02-13 - Tags: db, spark, glue, emr code for article  pfeilbr/apache-iceberg-playground learn Apache Iceberg Iceberg is a high-performance format for huge analytic tables. Iceberg brings the reliability and simplicity of SQL tables to big data, while making it possible for engines like Spark, Trino, Flink, Presto, Hive and Impala to safely work with the same tables, at the same time. concepts data lakehouse - data architecture that blends a data lake and data warehouse together data layer stores the actual data of the table and is primarily made up of the data files themselves, though also included are delete files and puffin files. Iceberg is file-format agnostic and currently supports Apache Parquet, Apache ORC, and Apache Avro parquet is the most common underlying file format delete files - track which records in the dataset have been deleted can either be a copy of the old file with the changes reflected in a new copy of it (called copy-on-write) or it can be a new file that only has the changes written, which then engines reading the data coalesce (called merge-on-read). two ways to identify a given row that needs to be removed from the logical dataset when an engine reads the dataset: either identify the row by its exact position in the dataset or identify the row by the values of one or more fields of the row. positional delete files - denote what rows have been logically deleted equality delete files - denote what rows have been logically deleted puffin file format - stores statistics and indexes about the data in the table that improve the performance of an even broader range of queries than the statistics stored in the data files and metadata files. metadata layer tree structure that tracks the data files and metadata about them as well as the operations that made them. made up of three file types, all of which are stored in data lake storage: manifest files, manifest lists, and metadata files enabling core features like time travel and schema evolution. Manifest files - keep track of files in the data layer (i.e., data files, delete files, and puffin files) as well as additional details and statistics about each file. manifest file keeps track of a subset of the data files. They contain information such as details about partition membership, record count, and lower and upper bounds of columns, that is used to improve efficiency and performance while reading the data from these data files. catalog central place where you go to find the current location of the current metadata pointer is the Iceberg catalog Within the catalog, there is a reference or pointer for each table to that table’s current metadata file. Diagrams resources Apache Iceberg Apache Iceberg: The Definitive Guide --- ### Pkl - URL: https://brianpfeil.com/post/pkl/ - Date: 2024-02-04 - Tags: config, language code for article  pfeilbr/pkl-playground learn Pkl , configuration that is programmable, scalable, and safe demo install the cli for your platform # verify install pkl --version FILE=config.pkl pkl eval $FILE pkl eval --format json $FILE pkl eval --format yaml $FILE resources Pkl Pkl // Docs Truffle Language Implementation Framework Pkl was built using the GraalVM Truffle framework --- ### Mountpoint for S3 - URL: https://brianpfeil.com/post/mountpoint-for-s3/ - Date: 2024-02-01 - Tags: s3, aws code for article  pfeilbr/mountpoint-for-s3-playground learn Mountpoint for Amazon S3 support of posix file system apis: create, read, and delete files. does not support updates to existing files. demo # install Mountpoint for Amazon S3 client # for amzn linux, centos, redhat wget https://s3.amazonaws.com/mountpoint-s3-release/latest/x86_64/mount-s3.rpm sudo yum install -y ./mount-s3.rpm # or for debian wget https://s3.amazonaws.com/mountpoint-s3-release/latest/x86_64/mount-s3.deb sudo apt-get install -y ./mount-s3.deb # set credentials for `mount-s3` client to use export $(printf "AWS_ACCESS_KEY_ID=%s AWS_SECRET_ACCESS_KEY=%s AWS_SESSION_TOKEN=%s" \ $(aws --profile "PROFILE_NAME" sts assume-role \ --role-arn arn:aws:iam::xxxxxxxxxx:role/s3-full-access \ --role-session-name MySessionName \ --query "Credentials.[AccessKeyId,SecretAccessKey,SessionToken]" \ --output text)) # show all options mount-s3 --help # mount with no delete mount-s3 <bucket name> ~/mnt/s3 # mount with delete mount-s3 --allow-delete "com.brianpfeil.scratch01" ~/mnt/s3 # unmount sudo umount ~/mnt/s3 # `--allow-delete` - the corresponding object is immediately deleted from your S3 bucket. # `--read-only` - forbid all mutating actions on your S3 bucket resources https://github.com/awslabs/mountpoint-s3 Configuring Mountpoint for Amazon S3 --- ### Instructor - URL: https://brianpfeil.com/post/instructor/ - Date: 2024-01-14 - Tags: ai, llm, python code for article  pfeilbr/instructor-playground learn Instructor Structured extraction in Python, powered by llms, designed for simplicity, transparency, and control. see main.ipynb demo # set local version of python for *this* shell pyenv local 3.12 # create virtual environment with local version of python python -m venv .venv # activate environment source ./.venv/bin/activate # upgrade pip pip install --upgrade pip # install requirements pip install -r requirements.txt # freeze requirements pip freeze > requirements.txt # open main.ipynb in vscode code main.ipynb # OR open main.ipynb in `juypter lab` juypter lab # open http://localhost:8888/lab in browser resources https://jxnl.github.io/instructor/ https://github.com/jxnl/instructor --- ### VectorDB - URL: https://brianpfeil.com/post/vectordb/ - Date: 2023-11-26 - Tags: db, vectordb, ai, llm code for article  pfeilbr/vectordb-playground learn VectorDB demo see main.ipynb # install dependencies pipenv install pipenv run jupyter lab # OR pipenv shell jupyter lab resources https://vectordb.com/ https://github.com/kagisearch/vectordb --- ### Mojo - URL: https://brianpfeil.com/post/mojo/ - Date: 2023-11-24 - Tags: machine-learning, python code for article  pfeilbr/mojo-playground learn and experiment with Modular MojoπŸ”₯ resources -https://docs.modular.com/ --- ### Next Amplify Gen2 - URL: https://brianpfeil.com/post/next-amplify-gen2/ - Date: 2023-11-23 - Tags: amplify, aws, next code for article  pfeilbr/next-amplify-gen2-playground demo via https://docs.amplify.aws/gen2/start/quickstart/ resources https://docs.amplify.aws/gen2/start/quickstart/ This is a Next.js project bootstrapped with create-next-app . Getting Started First, run the development server: npm run dev # or yarn dev # or pnpm dev # or bun dev Open http://localhost:3000 with your browser to see the result. You can start editing the page by modifying pages/index.tsx. The page auto-updates as you edit the file. API routes can be accessed on http://localhost:3000/api/hello . This endpoint can be edited in pages/api/hello.ts. The pages/api directory is mapped to /api/*. Files in this directory are treated as API routes instead of React pages. This project uses next/font to automatically optimize and load Inter, a custom Google Font. Learn More To learn more about Next.js, take a look at the following resources: Next.js Documentation - learn about Next.js features and API. Learn Next.js - an interactive Next.js tutorial. You can check out the Next.js GitHub repository - your feedback and contributions are welcome! Deploy on Vercel The easiest way to deploy your Next.js app is to use the Vercel Platform from the creators of Next.js. Check out our Next.js deployment documentation for more details. --- ### Amazon Bedrock - URL: https://brianpfeil.com/post/amazon-bedrock/ - Date: 2023-10-19 - Tags: aws, ai, genai, llm code for article  pfeilbr/amazon-bedrock-playground learn Amazon Bedrock demo pipenv install # login for boto3 credentials aws sso login --profile root-AWSAdministratorAccess visit main.ipynb in vscode select .venv kernel in top right run jupyter lab UI pipenv run jupyter lab # open http://localhost:8888/lab Concepts Foundation Models Custom models Continued Pre-training model allows you to train a model with new unlabeled data. Use continued pre-training to teach a model new domain knowledge that’s not already present in the base models. You can train a model with private data, such as business documents, that are not publically available for training large language models. Additionally, you can continue to improve the model by retraining the model with more unlabeled data as it becomes available. Fine-Tuning model allows you to improve a model’s performance on specific tasks. By providing a training dataset of labeled examples related to a specific task, you help the model learn the task it’s supposed to carry out. Model customization job A model customization job is a process that takes a base model and trains it with your data to create a custom model. You can use a model customization job to create a custom model from a base model that’s provided by Amazon Bedrock or a custom model that you’ve already created. upload a training dataset and, optionally, a validation dataset to Amazon S3 and provide the Amazon S3 bucket path to the model customization job After you complete a model customization job, you can purchase Provisioned Throughput (see Provisioned Throughput) for the customized model so that you can use the model for inference using the InvokeModel or InvokeModelWithResponseStream API operations. Model Provisioned Throughput When you configure Provisioned Throughput for a model, you receive a level of throughput at a fixed cost. You can use Provisioned Throughput with Amazon and third-party base models, and with customized models. Multimodal Embeddings model e.g. Amazon Titan Multimodal Embeddings model searching images by text, image, or a combination of text and image. Designed for high accuracy and fast responses, this model is an ideal choice for search and recommendations use cases. see Multimodal RAG example with code at 1_multimodal_rag.ipynb Knowledge base for Amazon Bedrock managed Retrieval Augmented Generation (RAG) Steps create data source (e.g. s3 bucket with pdf docs) select embedding model (e.g. Amazon Titan G1 Embeddings - Text) create and select a vector database (e.g. OpenSearch) ingest data sources into knowledge base test knowledge base - send queries and see the responses Agents Build a Foundation Model (FM) powered customer service bot with agents for Amazon Bedrock - good example TL;DR model that uses OpenAPI schema backed by Lambda function that defines the business logic for the action that your agent will carry out Steps select model (e.g. amazon titan) define action groups add knowledge base (optional) Action Groups Action groups define the tasks that you want your agent to help customers carry out Action groups are mapped to an AWS Lambda function and related API schema to perform API calls. Knowledge bases provide a repository of information that the agent can query to answer customer queries and improve its generated responses. Action group consists of the following components that you set up OpenAPI schema that define the APIs that your action group should call. Your agent uses the API schema to determine the fields it needs to elicit from the customer to populate for the API request. A Lambda function that defines the business logic for the action that your agent will carry out. Resources https://docs.aws.amazon.com/bedrock/latest/userguide Amazon Bedrock Base model IDs (on-demand throughput) Amazon Bedrock base model IDs (provisioned throughput) aws-samples/amazon-bedrock-samples Build a Foundation Model (FM) powered customer service bot with agents for Amazon Bedrock generative-ai-amazon-bedrock-langchain-agent-example --- ### Cloudscape Design System - URL: https://brianpfeil.com/post/cloudscape-design-system/ - Date: 2023-09-19 - Tags: aws, design, react, ui code for article  pfeilbr/cloudscape-design-system-playground learn cloudscape-design-system . created and used by AWS. demo see src/App.js npm install @cloudscape-design/global-styles # add to src/App.js import "@cloudscape-design/global-styles/index.css" npm install @cloudscape-design/components npm run start resources cloudscape-design-system --- ### AWS CloudWatch Synthetics - URL: https://brianpfeil.com/post/aws-cloudwatch-synthetics/ - Date: 2023-09-06 - Tags: aws, cloudwatch code for article  pfeilbr/aws-cloudwatch-synthetics-playground learn CloudWatch Synthetics You can use Amazon CloudWatch Synthetics to create canaries, configurable scripts that run on a schedule, to monitor your endpoints and APIs. Canaries follow the same routes and perform the same actions as a customer, which makes it possible for you to continually verify your customer experience even when you don’t have any customer traffic on your applications. notes supports monitoring your REST APIs, URLs, and website content every minute, 24x7, and alerts you when your application endpoints don’t behave as expected. Node.js or python based. bundles in Puppeteer + Chromium to the runtime trigger types - cron (1 min smallest freq), run once can run in VPC can also used in any workloads requiring general browser automation creates several CloudWatch metrics in CloudWatchSynthetics namespace EventBridge support. See monitoring canary events with Amazon EventBridge CloudFormation support via AWS::Synthetics::Canary if your handler has an error (e.g. throws Error(..)) that is not caught it counts as a failed canary e.g. Canary: Type: 'AWS::Synthetics::Canary' e.g. CanaryAlarm: Type: AWS::CloudWatch::Alarm e.g. CanaryRole: Type: 'AWS::IAM::Role' resources CloudWatch Synthetics CloudWatch metrics published by canaries AWS::Synthetics::Canary aws-reinvent-trivia-game/canaries https://github.com/awslabs/aws-well-architected-labs/blob/main/static/Reliability/300_Fault_Isolation_with_Shuffle_Sharding/Code/regular.yaml#L575 - example cloudformation template --- ### Amazon MemoryDB - URL: https://brianpfeil.com/post/amazon-memorydb/ - Date: 2023-09-05 - Tags: aws, db code for article  pfeilbr/amazon-memorydb-playground learn Amazon MemoryDB for Redis Demo export AWS_PROFILE="hub01-admin" export STACK_NAME=amazon-memorydb-playground export REGION=us-east-1 export AWS_ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text) sam build --use-container sam deploy --guided sam deploy aws cloudformation describe-stacks \ --stack-name $STACK_NAME \ --region $REGION \ --query 'Stacks[0].Outputs' \ > stack-outputs.json sam delete --no-prompts resources Amazon MemoryDB for Redis Amazon MemoryDB for Redis Documentation serverless-patterns/apigw-lambda-memorydb AWSinAction/code3/chapter11/memorydb-minimal.yaml Configure ACLs with the ACL command --- ### Momento - URL: https://brianpfeil.com/post/momento/ - Date: 2023-08-24 - Tags: cache, db code for article  pfeilbr/momento-playground learn and experiment with Momento general currently support cache and topic resource types simple pay-as-you-go pricing model by data trasfered gRPC for efficient communication with server demo see index.js cp .env.sample .env # edit .env to include youe `MOMENTO_AUTH_TOKEN` npm install npm start resources Momento https://docs.momentohq.com/ Service Limits --- ### DuckDB - URL: https://brianpfeil.com/post/duckdb/ - Date: 2023-06-15 - Tags: db code for article  pfeilbr/duckdb-playground learn duckdb , an in-process SQL OLAP database management system demo pipenv install pipenv shell jupyter lab # visit main.ipynb resources duckdb --- ### AWS Boto3 - URL: https://brianpfeil.com/post/aws-boto3/ - Date: 2023-06-11 - Tags: aws, python, sdk code for article  pfeilbr/aws-boto3-playground learn aws boto3 demo pipenv install pipenv shell jupyter lab # visit main.ipynb resources boto3 Boto3 documentation --- ### Streamlit - URL: https://brianpfeil.com/post/streamlit/ - Date: 2023-06-11 - Tags: ui, python code for article  pfeilbr/streamlit-playground learn streamlit demo pipenv install pipenv shell jupyter lab # visit main.ipynb resources streamlit --- ### PyTorch - URL: https://brianpfeil.com/post/pytorch/ - Date: 2023-06-03 - Tags: machine-learning, python, ai code for article  pfeilbr/pytorch-playground --- ### ChromaDB - URL: https://brianpfeil.com/post/chromadb/ - Date: 2023-06-01 - Tags: db, vectordb code for article  pfeilbr/chromadb-playground learn chromadb , the AI-native open-source embedding/vector database demo pipenv install pipenv shell jupyter lab # visit main.ipynb resources https://github.com/chroma-core/chroma https://docs.trychroma.com/ --- ### LangChain Python - URL: https://brianpfeil.com/post/langchain-python/ - Date: 2023-05-25 - Tags: python, llm, genai code for article  pfeilbr/langchain-python-playground learn LangChain (python) concepts Agents - use an LLM to determine which actions to take and in what order. An action can either be using a tool and observing its output, or returning to the user. the llm is used to select the tool to use langchain/agents/agent.py - _call method has main logic langchain/agents/types.py langchain/agents/mrkl/prompt.py langchain/agents/**/*prompt.py - these contain all the prompts for different agents demo pipenv shell export OPENAI_API_KEY=... export SERPAPI_API_KEY=... jupyter lab # visit `main.ipynb` resources langchain-python LangChain AI Handbook - great resource --- ### OpenAI Nodejs - URL: https://brianpfeil.com/post/openai-nodejs/ - Date: 2023-05-18 - Tags: nodejs, llm, ai code for article  pfeilbr/openai-nodejs-playground concepts tiktoken - a fast BPE (Byte pair encoding) tokenizer for use with OpenAI’s models demo resources https://platform.openai.com/ OpenAI API OpenAI Playground https://github.com/openai/openai-cookbook How to count tokens with tiktoken Question answering using embeddings-based search https://github.com/openai/tiktoken How to stream completions --- ### Observability - URL: https://brianpfeil.com/post/observability/ - Date: 2023-03-03 code for article  pfeilbr/observability-playground observability (o11y for short) definition of β€œobservability” for software systems is a measure of how well you can understand and explain any state your system can get into If you can understand any bizarre or novel state without needing to ship new code, you have observability. observability is more relevant in the age of micro services and solutions being composed of many disparate services. not as important in the age of the monolith, where everything happened in a single process + options DB and all the information was available in a single log. Observability lets you easily deal with unknown unknowns β€œYou can understand the inner workings of a system […] by asking questions from the outside […], without having to ship new code every time. It’s easy to ship new code to answer a specific question that you found that you need to ask. But instrumenting so that you can ask any question and understand any answer is both an art and a science, and your system is observable when you can ask any question of your system and understand the results without having to SSH into a machine.” concepts telemetry - refers to data emitted from a system OTLP (OpenTelemetry Protocol) monitoring - is inherently reactive. traditional monitoring tools work by checking system conditions against known thresholds that indicate whether previously known error conditions are present metrics - aggregations over a period of time of numeric data about your infrastructure or application. e.g. system error rate, CPU utilization, request rate for a given service. metics require you to define them upfront. if there is an issue that you can’t answer with your pre-defined metrics, you need to add the metric, and try to re-create the problem to populate the new metric. Metrics require foresight into what’s going to happen later on. SLA (service level agreement) - commitment between a service provider and a customer SLI (Service Level Indicator) - actual measurement of a service’s behavior. A good SLI measures your service from the perspective of your users. e.g. the speed at which a web page loads. SLO (Service Level Objective) - means by which reliability is communicated to an organization/other teams. This is accomplished by attaching one or more SLIs to business value. An SLI (service level indicator) measures compliance with an SLO (service level objective). So, for example, if your SLA specifies that your systems will be available 99.95% of the time, your SLO is likely 99.95% uptime and your SLI is the actual measurement of your uptime. Maybe it’s 99.96%. Maybe 99.99%. trace (aka distributed traces) - made up of a tree of spans, starting with a root span, which encapsulates the end-to-end time that it takes to accomplish a task span - represents a unit of work APM - Application Performance Monitoring. monitoring and management of the performance and availability of software applications. APM strives to detect and diagnose complex application performance problems to maintain an expected level of service. OpenTelemetry OpenTelemetry (OTel for short) API -The specification portion of OTel libraries that allows developers to add instrumentation to their code without concern for the underlying implementation. SDK - The concrete implementation component of OTel that tracks state and batches data for transmission. Tracer - A component within the SDK that is responsible for tracking which span is currently active in your process. It also allows you to access and modify the current span to perform operations like adding attributes, events, or finishing it when the work it tracks is complete. Meter - A component within the SDK that is responsible for tracking which metrics are available to report on in your process. It also allows you to access and modify the current metrics to perform operations like adding values, or retrieving those values at periodic intervals. Exporter - plug-in for the SDK that translates OTel in-memory objects into the appropriate format for delivery to a specific destination Collector - standalone binary process that can be run as a proxy or sidecar that receives telemetry data (by default in OTLP format), processes it, and tees it to one or more configured destinations. Agent: A Collector instance running with the application or on the same host as the application (e.g. binary, sidecar, or daemonset). -Gateway: One or more Collector instances running as a standalone service (e.g. container or deployment) typically per cluster, data center or region. Grafana web ui for dashboarding process - select data sources, write queries against them in promql, influx query Prometheus typically used to monitor containers, micro services, and traditional servers originally built at SoundCloud in 2012 exporters typically run on the monitored host to export local metrics prometheus server pulls exported via http://hostaddress/metrics by default there are client libraries that allow you to expose your own /metrics endpoint with your custom metrics for custom applications prometheus.yml - specifies the targets to scrape and what interval Prometheus data store to centralize and store the metrics time series are built through a pull model: the Prometheus server queries a list of data sources (sometimes called exporters) at a specific polling frequency. Alertmanager to trigger alerts based on those metrics. Configuration for alerts can be specified in Prometheus which specifies a condition that needs to be maintained for a specific duration in order for an alert to trigger. When alerts trigger, they are forwarded to the Alertmanager service. Alertmanager can include logic to silence alerts and also to forward them to email, Slack, or notification services such as PagerDuty PromQL is the query language used to create dashboards and alerts. Prometheus server exposes HTTP interface for PromQL queries Prometheus Web UI run PromQL queries and see results simple visualizations (use graphana for more complex) Prometheus Server Architecture Prometheus metric format A metric is composed by several fields: Metric name Any number of labels (can be 0), represented as a key-value array Current metric value Optional metric timestamp Metric output is typically preceded with # HELP and # TYPE metadata lines. The HELP string identifies the metric name and a brief description of it. The TYPE string identifies the type of metric. example metric # HELP metric_name Description of the metric # TYPE metric_name type # Comment that's not parsed by prometheus http_requests_total{method="post",code="400"} 3 1395066363000 Prometheus metrics / OpenMetrics types Counter - represents a cumulative metric that only increases over time, like the number of requests to an endpoint. Gauge - Gauges are instantaneous measurements of a value. They can be arbitrary values which will be recorded. Gauges represent a random value that can increase and decrease randomly such as the load of your system. Histogram - samples observations (usually things like request durations or response sizes) and counts them in configurable buckets. It also provides a sum of all observed values. observability platforms aws x-ray honeycomb lightstep grafana splunk dynatrace resources http://opentelemetry.io/ OpenTelemetry | Documentation OTLP (OpenTelemetry Protocol) Youtube: How Prometheus Monitoring works | Prometheus Architecture explained --- ### Helm - URL: https://brianpfeil.com/post/helm/ - Date: 2023-02-17 - Tags: kubernetes code for article  pfeilbr/helm-playground learn Helm - The package manager for Kubernetes concepts similar to apt, yum, homebrew, but for k8s Helm is the only package manager specifically for Kubernetes applications. You can package your applications as a Helm Chart, and then install, update, and roll back your Kubernetes applications using Helm. a vanilla k8s app deployment consists of several yaml files to describe the k8s artifacts in a declarative manner. Helm is a tool that allows you to create deployment templates. This template is called a helm chart, which consists of the yaml files that look more or less like a regular k8s yaml file but have some variables in them. Those variables can be defined through an additional values.yaml file. helm chart dir structure Chart.yaml - top-level chart metadata. e.g. name, version, etc. values.yaml - declare variables here that are used in the templates/*.yaml files $ helm create demo Creating demo $ tree demo demo β”œβ”€β”€ Chart.yaml β”œβ”€β”€ charts β”œβ”€β”€ templates β”‚ β”œβ”€β”€ NOTES.txt β”‚ β”œβ”€β”€ _helpers.tpl β”‚ β”œβ”€β”€ deployment.yaml β”‚ β”œβ”€β”€ hpa.yaml β”‚ β”œβ”€β”€ ingress.yaml β”‚ β”œβ”€β”€ service.yaml β”‚ β”œβ”€β”€ serviceaccount.yaml β”‚ └── tests β”‚ └── test-connection.yaml └── values.yaml resources Helm: Quickstart Guide Kubernetes and Helm in a Nutshell Helm Charts: Kubernetes Tutorial and Examples ArtifactHUB - Find, install and publish Kubernetes packages --- ### Amazon EventBridge Pipes - URL: https://brianpfeil.com/post/amazon-eventbridge-pipes/ - Date: 2023-02-12 - Tags: eventbridge code for article  pfeilbr/amazon-eventbridge-pipes-playground learn Amazon EventBridge Pipes concepts create point-to-point integrations between event producers and consumers with optional transform, filter and enrich steps removes the need for “glue” lambdas high-level concepts: sources source event filter transform and enrich targets resources New β€” Create Point-to-Point Integrations Between Event Producers and Consumers with Amazon EventBridge Pipes Amazon EventBridge Pipes AWS::Pipes::Pipe CloudFormation examples sourcegraph: lang:yaml ‘AWS::Pipes::Pipe’ --- ### Amazon OpenSearch Serverless - URL: https://brianpfeil.com/post/amazon-opensearch-serverless/ - Date: 2023-01-27 - Tags: serverless, opensearch code for article  pfeilbr/amazon-opensearch-serverless-playground notes removes need to configure, manage, or scale OpenSearch clusters collection - group of indices representing a specific workload or use case. two collection types - time series and search capacity is managed for you. you create a collection, then you query and index data using the same OpenSearch APIs as before Serverless compute capacity is measured in OpenSearch Compute Units (OCUs). Each OCU is a combination of 6 GiB of memory and corresponding virtual CPU (vCPU), as well as data transfer to S3 decouples compute and storage. separates the indexing (ingest) components from the search (query) components, with S3 as the primary data storage for indexes. can scale search and index functions independently of each other and independently of the indexed data in S3. costs - charged for the following components: data ingestion compute search and query compute storage retained in S3 billed for a minimum of 4 OCUs for the first collection in your account Amazon OpenSearch Serverless architecture. Source: docs.aws.amazon.com demo creates opensearch serverless collection, network policy, access policy, and iam user template.yaml - based on Using AWS CloudFormation to create Amazon OpenSearch Serverless collections sam deploy screenshots resources Amazon OpenSearch Serverless CloudFormation | Amazon OpenSearch Serverless Build a search application with Amazon OpenSearch Serverless Using AWS CloudFormation to create Amazon OpenSearch Serverless collections Amazon OpenSearch goes Serverless! --- ### Amazon EventBridge Scheduler - URL: https://brianpfeil.com/post/amazon-eventbridge-scheduler/ - Date: 2022-11-11 - Tags: eventbridge, aws code for article  pfeilbr/amazon-eventbridge-scheduler-playground learn Introducing Amazon EventBridge Scheduler General new service name scheduler (actions scheduler:*) Key features of EventBridge Scheduler Templated targets – EventBridge Scheduler supports templated targets to perform common API operations using Amazon SQS, Amazon SNS, Lambda, and EventBridge. With predefined targets, you can configure your schedules quickly using the EventBridge Scheduler console, the EventBridge Scheduler SDK, or the AWS CLI. Universal targets – EventBridge Scheduler provides a universal target parameter (UTP) that you can use to create customized triggers that target more than 270 AWS services and over 6,000 API operations on a schedule. With UTP, you can configure your customized triggers using the EventBridge Scheduler console, the EventBridge Scheduler SDK, or the AWS CLI. Flexible time windows – EventBridge Scheduler supports flexible time windows, allowing you to disperse your schedules and improve the reliability of your triggers for use cases that do not require precise scheduled invocation of targets. Retries – EventBridge Scheduler provides at-least-once event delivery to targets, meaning that at least one delivery succeeds with a response from the target. EventBridge Scheduler allows you to set the number of retries for your schedule for a failed task. EventBridge Scheduler retries failed tasks with delayed attempts to improve the reliability of your schedule and ensure targets are available. Demo aws scheduler create-schedule --name SendEmailOnce \ --schedule-expression "at(2022-11-01T11:00:00)" \ --schedule-expression-timezone "Europe/Helsinki" \ --flexible-time-window "{\"Mode\": \"OFF\"}" \ --target "{\"Arn\": \"arn:aws:sns:us-east-1:xxx:test-chronos-send-email\", \"RoleArn\": \" arn:aws:iam::xxxx:role/sam_scheduler_role\" }" Bash there is a SAM event type of Type: ScheduleV2 that can trigger lambda or state machine MyFunction: Type: AWS::Serverless::Function Properties: Events: CWSchedule: Type: ScheduleV2 Properties: Schedule: 'rate(1 minute)' Name: TestScheduleV2Function Description: Test schedule event MyStateMachine: Type: AWS::Serverless::StateMachine Properties: Events: CWSchedule: Type: ScheduleV2 Properties: Schedule: 'rate(1 minute)' Name: TestScheduleV2StateMachine Description: Test schedule event Differences between EventBridge Scheduler and EventBridge rules Resources Introducing Amazon EventBridge Scheduler Amazon EventBridge Scheduler - Launch Announcement (twitter) Scheduler User Guide Scheduling events with EventBridge Scheduler Amazon EventBridge Scheduler resource type reference AWS::Scheduler::Schedule AWS::Scheduler::ScheduleGroup --- ### Control Plane - URL: https://brianpfeil.com/post/control-plane/ - Date: 2022-11-11 code for article  pfeilbr/control-plane-playground learn Control Plane General multi-cloud (aws, azure, gcp) costs - compute, memory and egress are passed along to customer move off of platfrom team. checking latency 10xs per sec health check - multi-cloud for everything. resilient. dns is 2 major providers - rt 53, constellex graphana under the covers control plane can use oidc connect for aws identity (Creating OpenID Connect (OIDC) identity providers ) reduced complexity for eng teams agent (ec2 instance) needs to run in vpc creates a wiregaurd connection to hub.controlplane.com agent can run on a local server secrets management is built in cli and tf provider (verified) Concepts Org - strictly isolated bounded context that encapsulates all the resources managed by Control Plane Global Virtual Cloud (GVC) - defines a set of cloud providers and their locations. Workloads - you specify a container image. this is the unit of deployment. Policy - governs resource access within an org to a set of principals. It enables fine-grained authorization rules to define the minimum amount of permissions required when accessing resources of the platform. Principal Types supports 4 principal types. Users Service Accounts Groups Identity Accessing Cloud Resources Erik - director of cust success IAM Roles { "Version": "2012-10-17", "Statement": [ { "Sid": "CreateRoles", "Effect": "Allow", "Action": [ "iam:CreatePolicy", "iam:UpdateAssumeRolePolicy", "iam:DetachRolePolicy", "iam:TagRole", "iam:UpdateRoleDescription", "iam:DeletePolicy", "iam:CreateRole", "iam:DeleteRole", "iam:AttachRolePolicy", "iam:UpdateRole", "iam:PutRolePolicy", "iam:TagPolicy" ], "Resource": "*" } ] } trust policy { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "AWS": [ "arn:aws:iam::957753459089:user/controlplane-driver", "arn:aws:iam::957753459089:role/controlplane-driver" ] }, "Action": "sts:AssumeRole", "Condition": { "StringEquals": { "sts:ExternalId": "org-dev-01" } } } ] } Demo # cli usage only for the deployment of resources Questions recommendation for secrets storage (api keys, db creds, etc.) “secrets” concept is there an iac solution for provisioning workload? what is used for “repeatability” Resources Control Plane https://docs.controlplane.com/ https://registry.terraform.io/providers/controlplane-com/cpln/latest --- ### Alfred Utilities Workflow - URL: https://brianpfeil.com/post/alfred-utilities-workflow/ - Date: 2022-09-08 code for article  pfeilbr/alfred-utilities-workflow general set of utility commands that can be invoked via alfred. u is the alfred keyword Install git clone https://github.com/pfeilbr/alfred-utilities-workflow.git` cd alfred-utilities-workflow npm install # follow [alfy#usage instructions](https://github.com/sindresorhus/alfy#usage) # enable "Alfred filters results" checkbox # symlink workflow directory to here # example cd /Users/pfeilbr/Dropbox/Alfred/Alfred.alfredpreferences/workflows ln -s ~/projects/alfred-utilities-workflow user.workflow.3AFB139F-7BE0-4430-9EE4-36FCD4B2376D Usage In Alfred, type u, Enter, and your query. Select a command to run press Enter Resources sindresorhus/alfy Script Filter JSON Format --- ### Amazon Redshift Serverless - URL: https://brianpfeil.com/post/amazon-redshift-serverless/ - Date: 2022-08-02 - Tags: serverless, aws, db, redshift code for article  pfeilbr/amazon-redshift-serverless-playground learn Amazon Redshift Serverless Amazon Redshift Serverless makes it convenient for you to run and scale analytics without having to provision and manage data warehouse clusters. Amazon Redshift automatically provisions and scales data warehouse capacity to deliver fast performance for demanding and unpredictable workloads. You pay only for the capacity that you use. Concepts Namespace - collection of database objects and users. The storage-related namespace groups together schemas, tables, users, or AWS Key Management Service keys for encrypting data. Workgroup - a collection of compute resources. The compute-related workgroup groups together compute resources like RPUs, VPC subnet groups, and security groups. Other resources that are grouped under workgroups include access and usage limits. Demo psql -h workgroup01.529276214230.us-east-1.redshift-serverless.amazonaws.com -d sample_data_dev -p 5439 -U uadmin # create user CREATE USER user01 with password 'Password01' createdb connection limit 30; # redshift serverless costs per day in USD select trunc(start_time) "Day", sum(compute_seconds)/60/60 * 0.375 from sys_serverless_usage group by trunc(start_time) order by 1 # login as `user01` psql -h workgroup01.529276214230.us-east-1.redshift-serverless.amazonaws.com -d sample_data_dev -p 5439 -U user01 # list databases \l # use database \c sample_data_dev # run query select * from tpcds.customer limit 10; Screenshots Serverless Dashboard Namespace Workgroup Workgroup Limits Data backup Snapshot Recovery point Resources Amazon Redshift Serverless Amazon Redshift Serverless – Now Generally Available with New Capabilities --- ### AWS EFS - URL: https://brianpfeil.com/post/aws-efs/ - Date: 2022-07-31 - Tags: aws code for article  pfeilbr/aws-efs-playground learn Amazon Elastic File System (EFS) Notes the client mounting the EFS FS (e.g. ec2, lambda, etc.) must be in the same VPC and subnet of the AWS::EFS::MountTarget for EC2 - ec2-user uid=1000, gid=1000. this can be used for AWS::EFS::AccessPoint.PosixUser and AWS::EFS::AccessPoint.RootDirectory properties for lambda - uid=1001, gid=1001 Demo # deploy # note if `MyFunctionWithEfs` fails because efs mount point not created, re-run deploy and ensure # `disable_rollback = true` in `samconfig.toml` # note the outputs and use to populate the env vars below sam deploy export FILESYSTEM_ID="fs-09a044741cf632443" export MOUNT_TARGET_DNS="${FILESYSTEM_ID}.efs.us-east-1.amazonaws.com" export MOUNT_TARGET_IP="172.30.4.9" export MOUNT_POINT="/home/ec2-user/efs-mount-point" export ACCESS_POINT_ID="fsap-01514965758e980b3 " export ACCESS_POINT_MOUNT_POINT="/home/ec2-user/efs-access-point-mount-point-01" export ACCESS_POINT_ARN="arn:aws:elasticfilesystem:us-east-1:529276214230:access-point/fsap-01514965758e980b3" # ssh into ec2 instance within the *same VPC and subnet* of EFS mount target(s) ssh ec2-user@dev01.brianpfeil.com # install EFS mount helper sudo yum install amazon-efs-utils # *** mount and use efs mount target example *** # make mount dir mkdir "${MOUNT_POINT}" # mount with any of the following # by efs dns hostname sudo mount -t nfs -o nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2,noresvport $MOUNT_TARGET_DNS:/ "${MOUNT_POINT}" # by efs ip address sudo mount -t nfs -o nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2,noresvport $MOUNT_TARGET_IP:/Β "${MOUNT_POINT}" # by using `amazon-efs-utils` package sudo mount -t efs "${FILESYSTEM_ID}" "${MOUNT_POINT}/" cd "${MOUNT_POINT}" # change perms so others can add files sudo chmod go+rw . # create file echo hello > a.txt # move to parent directory cd .. # unmount sudo umount "${MOUNT_POINT}" # re-mount sudo mount -t nfs -o nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2,noresvport $MOUNT_TARGET_DNS:/ "${MOUNT_POINT}" # view files created ls "${MOUNT_POINT}" # *** Mounting a file system using an access point example *** mkdir "${ACCESS_POINT_MOUNT_POINT}" sudo mount -t efs -o tls,accesspoint=${ACCESS_POINT_ID} ${FILESYSTEM_ID}: "${ACCESS_POINT_MOUNT_POINT}" cd "${ACCESS_POINT_MOUNT_POINT}" echo hello > a.txt cd .. sudo umount "${ACCESS_POINT_MOUNT_POINT}" # re-mount filesystem (not the access point. this is the root) sudo mount -t nfs -o nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2,noresvport $MOUNT_TARGET_DNS:/ "${MOUNT_POINT}" # list the file created within the access point (the /myefs directory) ls "${MOUNT_POINT}/myefs" # *** lambda mount and write to efs example *** # trigger lambda which will write file to `/mnt/efs/a.txt` which maps to `myefs-lambda/a.txt` in efs curl https://gwx70rb0q1.execute-api.us-east-1.amazonaws.com/Prod/ # output: {"output": "/mnt/efs contents\n\n['a.txt']"} # re-mount filesystem in ec2 (not the access point. this is the root) sudo mount -t nfs -o nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2,noresvport $MOUNT_TARGET_DNS:/ "${MOUNT_POINT}" # view contents of file written by lambda cat efs-mount-point/myefs-lambda/a.txt # output: hello from lambda # clean up sam delete --no-prompts Resources Amazon Elastic File System (EFS) --- ### AWS Batch - URL: https://brianpfeil.com/post/aws-batch/ - Date: 2022-07-29 - Tags: aws, containers code for article  pfeilbr/aws-batch-playground Examples aws-batch-fargate-example example of creating fargate batch compute environment, queue, definition and submitting a job Resources --- ### AWS SSO - URL: https://brianpfeil.com/post/aws-sso/ - Date: 2022-07-06 - Tags: aws, sso code for article  pfeilbr/aws-sso-playground learn AWS SSO SSO Access Token The aws sso CLI commands require the --access-token parameter. First login via sso (e.g. aws sso login --profile root-AWSAdministratorAccess), then run the following to get. # get cached aws sso accessToken function aws-access-token() { cat $(ls -1d ~/.aws/sso/cache/* | grep -v botocore) | jq -r "{accessToken} | .[]" } Demo list account assignments (AWS::SSO::Assignment ) aws sso-admin list-account-assignments \ --instance-arn 'arn:aws:sso:::instance/ssoins-72234101455cbc87' \ --account-id '529276214230' \ --permission-set-arn 'arn:aws:sso:::permissionSet/ssoins-72234101455cbc87/ps-51eacb02632f0b26' { "AccountAssignments": [ { "AccountId": "529276214230", "PermissionSetArn": "arn:aws:sso:::permissionSet/ssoins-72234101455cbc87/ps-51eacb02632f0b26", "PrincipalType": "USER", "PrincipalId": "906770ec60-e34082a0-033a-4dd2-90cb-9107804545e9" }, { "AccountId": "529276214230", "PermissionSetArn": "arn:aws:sso:::permissionSet/ssoins-72234101455cbc87/ps-51eacb02632f0b26", "PrincipalType": "USER", "PrincipalId": "906770ec60-9d6f0b65-701c-4650-b95c-7dab0f6046d7" } ] } Resources AWS Single Sign-On Documentation aws sso-admin - cli benkehoe/aws-sso-util AWS::SSO::PermissionSet AWS::SSO::Assignment How can I get temporary credentials for an AWS Single Sign-On user using the AWS CLI? --- ### R and RStudio - URL: https://brianpfeil.com/post/r-and-rstudio/ - Date: 2022-07-06 - Tags: machine-learning code for article  pfeilbr/r-and-rstudio-playground learn R and RStudio General R - cli environment Rscript - used to run R scripts from the command line . E.g.Rscript -e "date()" -e "format(Sys.time(), \"%a %b %d %X %Y\")" RStudio - desktop app RStudio Server - provide a browser based interface to a version of R running on a remote Linux server, bringing the power and productivity of the RStudio IDE to server-based deployments of R. RStudio Workbench The ability to develop in RStudio and Jupyter Load balancing Tutorial API Data connectivity and RStudio Professional Drivers Collaboration and project sharing Scale with Kubernetes and SLURM Authentication, access, & security Run multiple concurrent R and Python sessions Remote execution with Launcher Auditing and monitoring Advanced R and Python session management RStudio Connect is a publishing platform for the work your teams create in R and Python. RStudio Connect allows you to share the following, and more, in one convenient place: Shiny applications. R Markdown reports. Plumber APIs. SageMaker RStudio browser based rstudio workbench launch rstudio app win sm domain launch rstudio session select ec2 instance type session has user home directory backed by s3 installed r pkgs are preserved in home dir every session uses same home dir shiny apps plumber apis r reticulate package offers integration with python for working w sm easily from R import py package and ref from r variable. proxies calls from r to py coming soon - rstudio connect managed by sm RStudio Connect is a publishing platform for the work your teams create in R and Python. RStudio Connect allows you to share the following, and more, in one convenient place: Shiny applications. R Markdown reports. Plumber APIs. Running RStudio in Docker # run docker run --rm -p 8787:8787 -e PASSWORD=password01 rocker/rstudio # open IDE in browser open https://localhost:8787 # username: rstudio, password: password01 Resources R CRAN (The Comprehensive R Archive Network) Rocker Project - Docker Containers for the R Environment https://docs.rstudio.com/ pfeilbr/rstudio-on-sagemaker-playground --- ### AWS ALB to Lambda - URL: https://brianpfeil.com/post/aws-alb-to-lambda/ - Date: 2022-07-01 - Tags: aws, lambda code for article  pfeilbr/aws-alb-to-lambda-playground learn ALB to Lambda integration Notes AWS provided DNSName for the ALB is of the format xxxxxxx.us-east-1.elb.amazonaws.com the ALB DNSName does not provide SSL termination by default you cannot request a certificate for the ALB DNSName (xxxxxxx.us-east-1.elb.amazonaws.com) via ACM if you own the domain, you can request a certificate (e.g. alb01.allthecloudbits.com) via ACM and have it automatically validated via route 53 CNAME record (see DomainValidationOptions ) In order for a AWS::CertificateManager::Certificate to be provisioned and validated in CloudFormation automatically, the DomainName property needs to be identical to one of the DomainName property supplied in DomainValidationOptions, if the ValidationMethod is DNS. Failing to keep them like-for-like will result in failure to create the domain validation records in Route53. you can specify a single certificate for your AWS::ElasticLoadBalancingV2::Listener via the AWS::ElasticLoadBalancingV2::Listener Certificate property. If you need multipe certificates, use AWS::ElasticLoadBalancingV2::ListenerCertificate Demo based on alexcasalboni/template.yml - AWS ALB - AWS Lambda integration with CloudFormation (YAML) includes a custom domain name CNAME’d to the ALB hostname with ACM cert for https lambda function code is inlined in template.yaml # deploy stack sam deploy # check our ALB andpoint curl http://aws-a-myLoa-MFYJ9QBS4CCL-673155384.us-east-1.elb.amazonaws.com # output: # <h1>Hello from Lambda via ALB</h1><pre><code>{ # "requestContext": { # "elb": { # "targetGroupArn": "arn:aws:elasticloadbalancing:us-east-1:529276214230:targetgroup/aws-a-myTar-1OF41QMSUR19N/c81f17d663a5ef3a" # } # }, # "httpMethod": "GET", # "path": "/", # "queryStringParameters": {}, # "headers": { # "accept": "*/*", # "host": "aws-a-myLoa-MFYJ9QBS4CCL-673155384.us-east-1.elb.amazonaws.com", # "user-agent": "curl/7.79.1", # "x-amzn-trace-id": "Root=1-62bf13a2-7ec6c217009df4474336906d", # "x-forwarded-for": "100.11.104.251", # "x-forwarded-port": "80", # "x-forwarded-proto": "http" # }, # "body": "", # "isBase64Encoded": false # }</code></pre> # check path based listener (`/api/*`) curl http://aws-a-myLoa-MFYJ9QBS4CCL-673155384.us-east-1.elb.amazonaws.com/api/hello # output: # <h1>Hello from Lambda via ALB</h1><pre><code>{ # "requestContext": { # "elb": { # "targetGroupArn": "arn:aws:elasticloadbalancing:us-east-1:529276214230:targetgroup/aws-a-myTar-1OF41QMSUR19N/c81f17d663a5ef3a" # } # }, # "httpMethod": "GET", # "path": "/api/hello", # "queryStringParameters": {}, # "headers": { # "accept": "*/*", # "host": "aws-a-myLoa-MFYJ9QBS4CCL-673155384.us-east-1.elb.amazonaws.com", # "user-agent": "curl/7.79.1", # "x-amzn-trace-id": "Root=1-62bf27ec-5b6861564ac9b30938dd1e67", # "x-forwarded-for": "100.11.104.251", # "x-forwarded-port": "80", # "x-forwarded-proto": "http" # }, # "body": "", # "isBase64Encoded": false # }</code></pre> # check our custom domain curl http://alb01.allthecloudbits.com # check our custom domain https endpoint curl https://alb01.allthecloudbits.com # clean up sam delete --no-prompts Resources alexcasalboni/template.yml - AWS ALB - AWS Lambda integration with CloudFormation (YAML) Lambda functions as targets for Application Load Balancers Application Load Balancer can now Invoke Lambda Functions to Serve HTTP(S) Requests One Load Balancer to rule them all How do I associate multiple ACM SSL or TLS certificates with Application Load Balancer using CloudFormation? AWS::ElasticLoadBalancingV2::LoadBalancer AWS::ElasticLoadBalancingV2::Listener AWS::ElasticLoadBalancingV2::ListenerCertificate AWS::ElasticLoadBalancingV2::ListenerRule AWS::ElasticLoadBalancingV2::TargetGroup --- ### RStudio On SageMaker - URL: https://brianpfeil.com/post/rstudio-on-sagemaker/ - Date: 2022-07-01 - Tags: sagemaker, machine-learning code for article  pfeilbr/rstudio-on-sagemaker-playground learn RStudio on Amazon SageMaker Notes fully managed and self-serve RStudio solution. elimiinates administrative overhead. access to scalable AWS resources provides R users with native environment withing SageMaker direct access to all tools SageMaker offers (via reticulate ) fully integrated with SageMaker ecosystem. e.g. Access to SageMaker Built-In Algorithms browser based RStudio workbench launch RStudio app w/in SageMaker Domain launch RStudio session select ec2 instance type session has user home directory backed by s3 installed R pkgs are preserved in home dir every session uses same home dir develop shiny apps, etc. plumber apis R reticulate package offers integration with python for working with SM easily from R import py package and ref from r variable. proxies calls from r to py coming soon custom images for RStudio Workbench RStudio Connect managed by SageMaker RStudio Connect is a publishing platform for the work your teams create in R and Python. RStudio Connect allows you to share the following, and more, in one convenient place: Shiny applications. R Markdown reports. Plumber APIs. Screenshots Launch RStudio from SM Domain Creating new session in Studio Workbench Workbench UI Shiny App Example Plumber API Example R Integration with Boto3 SageMaker package via R reticulate package Running SageMaker Training Job from R Deploying SageMaker Real-time Predication API Endpoint Resources Announcing Fully Managed RStudio on Amazon SageMaker for Data Scientists Use RStudio on Amazon SageMaker Using RStudio on Amazon SageMaker: Questions from the Community James Blair | Using RStudio on Amazon SageMaker | RStudio - good video with demo blairj09-talks/rstudio-sagemaker-webinar - slides from James Blair | Using RStudio on Amazon SageMaker | RStudio video. direct link slides.pdf aws/amazon-sagemaker-examples/r_examples/ --- ### SageMaker Ground Truth - URL: https://brianpfeil.com/post/sagemaker-ground-truth/ - Date: 2022-06-16 - Tags: sagemaker, machine-learning code for article  pfeilbr/sagemaker-ground-truth-playground learn SageMaker Ground Truth (GT) [Plus] Concepts Label Data - workforce - the group of workers that you have selected to label your dataset. You can choose either the Amazon Mechanical Turk workforce, a vendor-managed workforce, or you can create your own private workforce to label or review your dataset Each AWS account has access to a single private workforce per region, and the owner has the ability to create multiple private work teams within that workforce. A single private work team is used to complete a labeling job or human review task, or a job. You can assign each work team to a separate job or use a single team for multiple jobs. A single worker can be in more than one work team. SageMaker Ground Truth Plus With SageMaker Ground Truth Plus, you receive a custom quote that is tailored to your specific use case and requirements. To get your customized quote, fill out the project requirement form. SageMaker Ground Truth Plus is priced on a per label basis, which can be a bounding box, cuboid, key-value pair, etc. Ground Truth Plus can be used for a variety of use cases, including computer vision, natural language processing, and speech recognition. Amazon SageMaker Ground Truth Plus allows you to easily create high-quality training datasets without having to build labeling applications or manage labeling workforces on your own. Once you provide data along with labeling requirements, SageMaker Ground Truth Plus handles setting up the data labeling workflows and managing them on your behalf, in accordance with your requirements. From there, an expert workforce that is trained on a variety of machine learning (ML) tasks does data labeling. Ground Truth Plus uses ML techniques, including active-learning, pre-labeling, and machine validation. This increases the quality of the output dataset and decreases the data labeling costs. Ground Truth Plus provides transparency into your data labeling operations and quality management. With it, you can review the progress of training datasets across multiple projects, track project metrics, such as daily throughput, inspect labels for quality, and provide feedback on the labeled data. Ground Truth Plus can be used for a variety of use cases, including computer vision, natural language processing, and speech recognition. Can the datasets be shared with amazon provided workforce Ground Truth Plus does not support PHI, PCI or FedRAMP certified data, and you should not provide this data to Ground Truth Plus. request a pilot share data by creating bucket and applying bucket policy allowing GT aws account the bucket actions s3://your-bucket-name/ground-truth-plus/input/project-name/batch-name/.. create project team - cognito - up to 50 email addresses Your team members receive an email inviting them to join the Ground Truth Plus project team as shown in the following image. Open the Project Portal view status on batches The portal allows your project team members and you to review a small sample set of the labeled objects for each batch. Accept or Reject Batches * Text Classification (Multi-label) Use a multi-label text classification task when you want workers to group a piece of text into one or more classes by choosing one or more labels. When you create a multi-label text classification job, you can provide up to 50 labels for workers to choose from when classifying text. Workers must select at least one label, and can select as many labels as you provide. For example, if you provide 10 labels, the worker can select up to 10 labels for an object. cannot provision any GT resources via CloudFormation Create Labeling Job Python (Boto3) response = client.create_labeling_job( LabelingJobName="example-labeling-job", LabelAttributeName="label", InputConfig={ 'DataSource': { 'S3DataSource': { 'ManifestS3Uri': "s3://bucket/path/manifest-with-input-data.json" } }, 'DataAttributes': { 'ContentClassifiers': [ "FreeOfPersonallyIdentifiableInformation"|"FreeOfAdultContent", ] } }, OutputConfig={ 'S3OutputPath': "s3://bucket/path/file-to-store-output-data", 'KmsKeyId': "string" }, RoleArn="arn:aws:iam::*:role/*", LabelCategoryConfigS3Uri="s3://bucket/path/label-categories.json", StoppingConditions={ 'MaxHumanLabeledObjectCount': 123, 'MaxPercentageOfInputDatasetLabeled': 123 }, HumanTaskConfig={ 'WorkteamArn': "arn:aws:sagemaker:region:*:workteam/private-crowd/*", 'UiConfig': { 'UiTemplateS3Uri': "s3://bucket/path/custom-worker-task-template.html" }, 'PreHumanTaskLambdaArn': "arn:aws:lambda:us-east-1:432418664414:function:PRE-tasktype", 'TaskKeywords': [ "Images", "Classification", "Multi-label" ], 'TaskTitle': "Multi-label image classification task", 'TaskDescription': "Select all labels that apply to the images shown", 'NumberOfHumanWorkersPerDataObject': 1, 'TaskTimeLimitInSeconds': 3600, 'TaskAvailabilityLifetimeInSeconds': 21600, 'MaxConcurrentTaskCount': 1000, 'AnnotationConsolidationConfig': { 'AnnotationConsolidationLambdaArn': "arn:aws:lambda:us-east-1:432418664414:function:ACS-" }, Tags=[ { 'Key': "string", 'Value': "string" }, ] ) Screenshots Resources Use Amazon SageMaker Ground Truth to Label Data Use Amazon SageMaker Ground Truth Plus to Label Data Getting Started with Amazon SageMaker Ground Truth Plus. Build a custom data labeling workflow with Amazon SageMaker Ground Truth Crowd HTML Elements Reference aws-samples/amazon-sagemaker-ground-truth-task-uis nitinaws/gt-custom-workflow aws-samples/sagemaker-ground-truth-label-training-data Create a Labeling Job (API) - Examples - python and aws cli examples --- ### EMR Serverless - URL: https://brianpfeil.com/post/emr-serverless/ - Date: 2022-06-01 - Tags: emr, serverless code for article  pfeilbr/emr-serverless-playground learn EMR Serverless provides a serverless runtime environment that simplifies the operation of analytics applications that use the latest open source frameworks, such as Apache Spark and Apache Hive. With EMR Serverless, you don’t have to configure, optimize, secure, or operate clusters to run applications with these frameworks. general flow create application start application your application is setup to start with pre-initialized capacity of 1 Spark driver and 1 Spark executor. Your application is by default configured to start when jobs are submitted and stop when the application is idle for more than 15 minutes. submit application jobs. e.g. spark or hive Demo # deploy template.yaml sam deploy --guided aws emr-serverless get-application \ --application-id 00f1du6284vksm09 { "application": { "applicationId": "00f1du6284vksm09", "name": "spark-3.2", "arn": "arn:aws:emr-serverless:us-east-1:529276214230:/applications/00f1du6284vksm09", "releaseLabel": "emr-6.6.0", "type": "Spark", "state": "CREATED", "stateDetails": "", "createdAt": "2022-06-02T20:09:49.041000+00:00", "updatedAt": "2022-06-02T20:09:49.667000+00:00", "tags": { "tag-on-create-key": "tag-on-create-value" }, "autoStartConfiguration": { "enabled": true }, "autoStopConfiguration": { "enabled": true, "idleTimeoutMinutes": 15 } } } aws s3 cp s3://us-east-1.elasticmapreduce/emr-containers/samples/wordcount/scripts/wordcount.py s3://emr-serverless-playground-scriptbucket-1fl6snd6viqk6/scripts/ APPLICATION_ID="00f1du6284vksm09" EXECUTION_ROLE_ARN="arn:aws:iam::529276214230:role/EMRServerless_Job_Execution_Role" BUCKET_NAME="emr-serverless-playground-scriptbucket-1fl6snd6viqk6" aws emr-serverless start-job-run \ --application-id $APPLICATION_ID \ --execution-role-arn $EXECUTION_ROLE_ARN \ --job-driver '{ "sparkSubmit": { "entryPoint": "s3://emr-serverless-playground-scriptbucket-1fl6snd6viqk6/scripts/wordcount.py", "entryPointArguments": ["s3://emr-serverless-playground-scriptbucket-1fl6snd6viqk6/emr-serverless-spark/output"], "sparkSubmitParameters": "--conf spark.executor.cores=1 --conf spark.executor.memory=4g --conf spark.driver.cores=1 --conf spark.driver.memory=4g --conf spark.executor.instances=1" } }' # output { "applicationId": "00f1du6284vksm09", "jobRunId": "00f1e0i4fr44ra01", "arn": "arn:aws:emr-serverless:us-east-1:529276214230:/applications/00f1du6284vksm09/jobruns/00f1e0i4fr44ra01" } JOB_RUN_ID="00f1e0i4fr44ra01" aws emr-serverless get-job-run \ --application-id $APPLICATION_ID \ --job-run-id $JOB_RUN_ID # output { "jobRun": { "applicationId": "00f1du6284vksm09", "jobRunId": "00f1e0i4fr44ra01", "arn": "arn:aws:emr-serverless:us-east-1:529276214230:/applications/00f1du6284vksm09/jobruns/00f1e0i4fr44ra01", "createdBy": "arn:aws:sts::529276214230:assumed-role/AWSReservedSSO_AWSAdministratorAccess_af1bdd24238208b5/pfeilbr", "createdAt": "2022-06-02T22:22:36.221000+00:00", "updatedAt": "2022-06-02T22:28:06.614000+00:00", "executionRole": "arn:aws:iam::529276214230:role/EMRServerless_Job_Execution_Role", "state": "SUCCESS", "stateDetails": "", "releaseLabel": "emr-6.6.0", "jobDriver": { "sparkSubmit": { "entryPoint": "s3://emr-serverless-playground-scriptbucket-1fl6snd6viqk6/scripts/wordcount.py", "entryPointArguments": [ "s3://emr-serverless-playground-scriptbucket-1fl6snd6viqk6/emr-serverless-spark/output" ], "sparkSubmitParameters": "--conf spark.executor.cores=1 --conf spark.executor.memory=4g --conf spark.driver.cores=1 --conf spark.driver.memory=4g --conf spark.executor.instances=1" } }, "tags": {}, "totalResourceUtilization": { "vCPUHour": 0.066, "memoryGBHour": 0.331, "storageGBHour": 1.322 } } } JOB_RUN_LOGS="s3://$BUCKET_NAME/emr-serverless-spark/logs/applications/$APPLICATION_ID/jobs/$JOB_RUN_ID" Notes unlike EMR the EC2 instances backing EMR-Serverless applications/jobs are not visible to the customer / fully managed (e.g. not visible in the customer EC2 console) Type - Spark or HIVE ReleaseLabel - EMR release version associated with the application. see Release versions EMR Serverless is a Regional service Job run states Spark examples Hive job examples Using Python libraries with EMR Serverless Submitting jobs from Airflow Resource Types application - arn:${Partition}:emr-serverless:${Region}:${Account}:/applications/${ApplicationId} jobRun arn:${Partition}:emr-serverless:${Region}:${Account}:/applications/${ApplicationId}/jobruns/${JobRunId} AWS CLI Support # create application aws emr-serverless create-application \ --type HIVE \ --name serverless-demo \ --release-label "emr-6.6.0" \ --initial-capacity '{ "DRIVER": { "workerCount": 1, "resourceConfiguration": { "cpu": "2vCPU", "memory": "4GB", "disk": "30gb" } }, "TEZ_TASK": { "workerCount": 10, "resourceConfiguration": { "cpu": "4vCPU", "memory": "8GB", "disk": "30gb" } } }' \ --maximum-capacity '{ "cpu": "400vCPU", "memory": "1024GB", "disk": "1000GB" }' # start application # your application is setup to start with pre-initialized capacity of 1 Spark driver and 1 Spark executor. Your application is by default configured to start when jobs are submitted and stop when the application is idle for more than 15 minutes. aws emr-serverless start-application \ --application-id $APPLICATION_ID # submit hive job aws emr-serverless start-job-run \ --application-id $APPLICATION_ID \ --execution-role-arn $JOB_ROLE_ARN \ --job-driver '{ "hive": { "initQueryFile": "s3://'${S3_BUCKET}'/code/hive/create_table.sql", "query": "s3://'${S3_BUCKET}'/code/hive/extreme_weather.sql", "parameters": "--hiveconf hive.exec.scratchdir=s3://'${S3_BUCKET}'/hive/scratch --hiveconf hive.metastore.warehouse.dir=s3://'${S3_BUCKET}'/hive/warehouse" } }' \ --configuration-overrides '{ "applicationConfiguration": [ { "classification": "hive-site", "properties": { "hive.driver.cores": "2", "hive.driver.memory": "4g", "hive.tez.container.size": "8192", "hive.tez.cpu.vcores": "4" } } ], "monitoringConfiguration": { "s3MonitoringConfiguration": { "logUri": "s3://'${S3_BUCKET}'/hive-logs/" } } }' CloudFormation Support AWS::EMRServerless::Application example AWS::EMRServerless::Application usage in cfn https://github.com/aws-samples/emr-serverless-samples/blob/main/cloudformation/emr_serverless_spark_app.yaml#L6 https://github.com/aws-samples/emr-serverless-samples/blob/main/cloudformation/emr_serverless_full_deployment.yaml#L61 AWS Console Screenshots Resources EMR Serverless Amazon EMR Serverless Now Generally Available – Run Big Data Applications without Managing Servers aws-samples/emr-serverless-samples AWS::EMRServerless::Application Actions, resources, and condition keys for Amazon EMR Serverless --- ### EC2 Image Builder - URL: https://brianpfeil.com/post/ec2-image-builder/ - Date: 2022-05-25 - Tags: aws, ec2 code for article  pfeilbr/ec2-image-builder-playground learn EC2 Image Builder EC2 Image Builder is a fully managed AWS service that makes it easier to automate the creation, management, and deployment of customized, secure, and up-to-date server images that are pre-installed and pre-configured with software and settings to meet specific IT standards. AWS Systems Manager Agent (SSM Agent) is Amazon software that runs on Amazon Elastic Compute Cloud (Amazon EC2) instances, edge devices, and on-premises servers and virtual machines (VMs). SSM Agent makes it possible for Systems Manager to update, manage, and configure these resources. The agent processes requests from the Systems Manager service in the AWS Cloud, and then runs them as specified in the request. SSM Agent then sends status and execution information back to the Systems Manager service by using the Amazon Message Delivery Service (service prefix: ec2messages). Process to Build AMI select base AMI (e.g. amazon linux 2, a custom AMI). must have SSM agent installed. select IAM role that is used as instance profile for EC2 build process and test process select amazon provided build components or specify your own build components amazon provided e.g. correto custom - defined via yaml with steps. e.g. step types - bash, executable select amazon provided tests or specify own custom tests specify build schedule - manual, schedule builder, CRON specify which regions to publish the resulting ami to can be shared with account, organization ou, whole organization Process to Build Docker Image select base container image. e.g. aws maanged images, an image in ECR, docker hub image select IAM role that is used as instance profile for EC2 build process and test process select amazon provided build components or specify your own build components amazon provided e.g. correto custom - defined via yaml with steps. e.g. step types - bash, executable select amazon provided tests or specify own custom tests specify build schedule - manual, schedule builder, CRON specify Dockerfile template the target ECR repository to store the image AWS Systems Manager Agent (SSM Agent) AWS Systems Manager Agent (SSM Agent) is Amazon software that runs on Amazon Elastic Compute Cloud (Amazon EC2) instances, edge devices, and on-premises servers and virtual machines (VMs). SSM Agent makes it possible for Systems Manager to update, manage, and configure these resources. The agent processes requests from the Systems Manager service in the AWS Cloud, and then runs them as specified in the request. SSM Agent then sends status and execution information back to the Systems Manager service by using the Amazon Message Delivery Service (service prefix: ec2messages). AWS Task Orchestrator and Executor (AWSTOE) component manager application to orchestrate complex workflows, modify system configurations, and test your systems without writing code. This application uses a declarative document schema. Because it is a standalone application, it does not require additional server setup. It can run on any cloud infrastructure and on premises. single golang binary that can validate and run AWSTOE components documents (yaml) hello-world-linux.yml example AWSTOE components document name: Hello World description: This is hello world testing document for Linux. schemaVersion: 1.0 phases: - name: build steps: - name: HelloWorldStep action: ExecuteBash inputs: commands: - echo 'Hello World from the build phase.' - name: validate steps: - name: HelloWorldStep action: ExecuteBash inputs: commands: - echo 'Hello World from the validate phase.' - name: test steps: - name: HelloWorldStep action: ExecuteBash inputs: commands: - echo 'Hello World from the test phase.' see Get started with AWSTOE # validate awstoe validate --documents hello-world-linux.yml # run all phases # a local directory named `TOE*` is created for the run and includes detailed logs awstoe run --documents hello-world-linux.yml # run single phase awstoe run --documents hello-world-linux.yml --phases build a local directory named TOE* is created for the run and includes detailed logs see TOE_2022-05-25_14-01-45_UTC-0_3699d6f5-dc33-11ec-9989-00224820200b Resources EC2 Image Builder Youtube - AWS re:Invent 2019: NEW LAUNCH! EC2 Image Builder: Virtual machine images made easy (CMP214-R1) AWS Task Orchestrator and Executor component manager - runs image builder and codebuild yaml files Get started with AWSTOE --- ### AWS Parallel Cluster - URL: https://brianpfeil.com/post/aws-parallel-cluster/ - Date: 2022-05-22 - Tags: aws, hpc code for article  pfeilbr/aws-parallel-cluster-playground learn AWS ParallelCluster . formerly CfnCluster (β€œcloud formation cluster”) framework that deploys and maintains high performance computing clusters on Amazon Web Services (AWS). Developed by AWS, CfnCluster facilitates both quick start proof of concepts (POCs) and production deployments. CfnCluster supports many different types of clustered applications and can easily be extended to support different frameworks. Notes distributed as a Python package and is installed using the Python pip package manager. supports Slurm and AWS Batch schedulers generates CloudFormation to create the cluster resources CDK is used to generate templates head node - responsible for submitting and scheduling jobs compute node(s) - runs jobs AMI for head and compute nodes (ec2 instances) AMI Name: aws-parallelcluster-3.1.4-amzn2-hvm-x86_64-202205121006 2022-05-12T10-09-45.467Z AMI Locaiton: amazon/aws-parallelcluster-3.1.4-amzn2-hvm-x86_64-202205121006 2022-05-12T10-09-45.467Z When you use the awsbatch scheduler, the AWS ParallelCluster CLI commands for AWS Batch are automatically installed in the AWS ParallelCluster head node CLI commands for AWS Batch - e.g. awsbsub, awsbqueues, etc. Demo install, configure cluster, create cluster, submit job to cluster python3 -m pip install "aws-parallelcluster" --upgrade --user # verify install pcluster version # configure cluster. prompts for scheduler type, region, etc. # when done `hello-world.yaml` is created # and it creates the networking / vpc resources via a cfn stack (e.g. `parallelclusternetworking-pubpriv-20220522231401` stack) # <https://github.com/aws/aws-parallelcluster/tree/release-3.0/cli/tests/pcluster/example_configs> for example # cluster configuration files # <https://docs.aws.amazon.com/parallelcluster/latest/ug/cluster-configuration-file-v3.html> - configuration files spec pcluster configure --config hello-world.yaml # create / provision the cluster (the networking/vpc resources already exist from previous `configure` command) pcluster create-cluster --cluster-name hello-world --cluster-configuration hello-world.yaml # login to cluster head node pcluster ssh --cluster-name hello-world -i /path/to/keyfile.pem # run the command sinfo to verify that your compute nodes are set up and configured. sinfo # create job to run (hello-job.sh) cat << EOF > hello-job.sh #!/bin/bash sleep 30 echo "Hello World from $(hostname)" EOF # submit job # this will create and ec2 compute instance on the fly sbatch hello-job.sh # view job in job queue squeue # once no job in queue a `.out` file will be created with results (STDOUT) cat slurm-1.out # clean up # delete cluster compute nodes (this doesn't delete cluster head node) pcluster delete-cluster-instances --cluster-name hello-world # delete the cluster itself # this terminates the head node and delete the cfn stack used to create the cluster pcluster delete-cluster --cluster-name hello-world Screenshots EC2 instances Resources AWS ParallelCluster aws/aws-parallelcluster cfncluster.readthedocs.io AWS services used by AWS ParallelCluster AWS Services used in CfnCluster Running your first job on AWS ParallelCluster Cluster configuration file aws-parallelcluster/cli/tests/pcluster/example_configs/ - example cluster configuration files AWS ParallelCluster CLI commands Slurm Workload Manager - Documentation --- ### AWS Web Content - URL: https://brianpfeil.com/post/aws-web-content/ - Date: 2022-05-14 - Tags: aws code for article  pfeilbr/aws-web-content App URL: https://brianpfeil.com/apps/awswebcontent/build/index.html fetch all aws directory api metadata (arch diagrams, products, blog posts, builders library articles, etc.) based on public data fetched via https://aws.amazon.com/api/dirs/items/search?item.directoryId=${directoryId}&item.locale=en_US index.js - entrypoint data/ - directory items Updating AWS Web Content Data the downloads new items in each directory and updates the following files data/${directoryId}.flat.json data/${directoryId}.display.json npm run fetch-new-items Frontend under construction # development cd frontend npm start # publishing - copies to `personal-website` cd frontend npm run publish Data Shapes general response shape { "items": [], "metadata": { "count": 0, "totalHits": 299 }, "fieldTypes": { "updateDate": "Date", "imageSrcUrl": "URL", "featureFlag": "Text", "description": "LongText", "sortDate": "Date", "docTitle": "Text", "primaryURL": "URL", "datePublished": "Date", "publishedText": "Text", "footerInfoSubtext": "Text", "subHeadline": "Text", "enableShare": "Boolean", "category": "Text", "contentType": "Text" } } example item (items[0].item) { "item": { "id": "whitepapers#image-moderation-chatbot", "locale": "en_US", "directoryId": "whitepapers", "name": "image-moderation-chatbot", "author": "julicoll", "createdBy": "julicoll", "lastUpdatedBy": "julicoll", "numImpressions": 0, "score": 0, "dateCreated": "2019-06-25T17:21:57+0000", "dateUpdated": "2021-07-29T17:01:46+0000", "additionalFields": { "datePublished": "2018-12-05", "publishedText": "December 2018", "description": "Shows you how to build a serverless chatbot on AWS that monitors your chat channels and removes images containing suggestive or explicit content.<p><a href=\"https://github.com/awslabs/lambda-refarch-imagemoderationchatbot?did=wp_card&trk=wp_card\" target=\"_blank\" rel=\"noopener\">Code</a></p><p class=\"m-subheadline\">Media Services | Serverless</p>", "docTitle": "Image Moderation Chatbot", "sortDate": "2018-12-05", "enableShare": "1", "contentType": "Reference Architecture Diagram", "primaryURL": "https://github.com/awslabs/lambda-refarch-imagemoderationchatbot?did=wp_card&trk=wp_card" } }, "tags": [ { "id": "GLOBAL#content-type#reference-arch-diagram", "locale": "en_US", "tagNamespaceId": "GLOBAL#content-type", "name": "Reference Architecture Diagram", "description": "Reference Architecture Diagram", "createdBy": "jenbar", "lastUpdatedBy": "jenbar", "dateCreated": "2020-04-29T05:19:31+0000", "dateUpdated": "2022-02-03T03:31:09+0000" }, { "id": "GLOBAL#methodology#serverless", "locale": "en_US", "tagNamespaceId": "GLOBAL#methodology", "name": "Serverless", "description": "Serverless", "createdBy": "jenbar", "lastUpdatedBy": "jenbar", "dateCreated": "2020-06-05T07:06:34+0000", "dateUpdated": "2022-02-03T03:32:11+0000" }, { "id": "GLOBAL#tech-category#media-services", "locale": "en_US", "tagNamespaceId": "GLOBAL#tech-category", "name": "Media Services", "description": "Media Services", "createdBy": "jarfaa", "lastUpdatedBy": "jenbar", "dateCreated": "2020-07-17T03:06:10+0000", "dateUpdated": "2022-02-03T03:35:28+0000" } ] } TODO add “AWS Quick Starts” directory https://aws.amazon.com/api/dirs/items/search?item.directoryId=alias%23solutions-experience&sort_by=item.additionalFields.sortDate&sort_order=desc&size=15&item.locale=en_US&tags.id=!GLOBAL%23flag%23archived stubbed and commented out in metadata.js step fn processing logic - need to figure out how to not download everything each run. way to download only new or changed items since last run define work by getting metadata for number results via https://…?item.directoryId=${directoryId}&item.locale=en_US&page=0. generate singe sqs message for each unique URL use returned metadata.count for size query string parameter &sort_by=item.[dateCreated|dateUpdated]&sort_order=desc lambda subscription to SQS. process sequencially. set batch size to >1 initially to see if throttling. can always set batch size to 1 use https://lunrjs.com/ for searching. see https://lunrjs.com/guides/getting_started.html for pre-creating index all blogs (https://aws.amazon.com/blogs/ ) template URL - “https://aws.amazon.com/api/dirs/items/search?item.directoryId=blog-posts&sort_by=item.additionalFields.createdDate&sort_order=desc&size=10&item.locale=en_US&page=1" all events content (https://aws.amazon.com/events/events-content ) page=0…N template URL “https://aws.amazon.com/api/dirs/items/search?item.directoryId=event-content&sort_by=item.dateCreated&sort_order=desc&size=12&item.locale=en_US&tags.id=GLOBAL%23language%23english&page=1" Builders Library - https://aws.amazon.com/api/dirs/items/search?item.directoryId=amazon-redwood&sort_by=item.additionalFields.customSort&sort_order=asc&size=24&item.locale=en_US Scratch # download data from api node index.js download # download a specified directory node index.js download --directoryId "whats-new" # flatten nested item arrays into single array of items node index.js flatten-data # generate data for frontend which contains only a subset of the full data for each item node index.js create-data-for-frontend # index data into lunr indexes node index.js index # search against an index node index.js search --directoryId "whats-new" --query "name:*lambda*" BUCKET_NAME="aws-web-content-s3bucket-1rmdk8t0ols56" BUCKET_DOMAIN_NAME="${BUCKET_NAME}.s3.amazonaws.com" aws s3 cp data s3://$BUCKET_NAME/data/ --recursive aws s3 cp index s3://$BUCKET_NAME/index/ --recursive aws lambda invoke \ --cli-binary-format raw-in-base64-out \ --function-name aws-web-content-UpdateAwsWebContent-xzzE4avkIwN6 \ --payload '{}' \ tmp/response.json One-liner to retrieve a list of all AWS products # source: <https://gist.github.com/garystafford/37442d8fd8dde388f50856c6a2900b0d> # One-liner to retrieve a list of all AWS products from aws.amazon.com/products sorted by product category (requires jq). Worked as of 2022-01-03. Page format tends to change a lot... curl --silent --compressed \ 'https://aws.amazon.com/api/dirs/items/search?item.directoryId=aws-products&sort_by=item.additionalFields.productCategory&sort_order=asc&size=500&item.locale=en_US' \ | jq -r '.items[].item | .additionalFields.productCategory + " | " + .additionalFields.productName' \ | sort Resources https://github.com/tycarac/aws-documents - good refernce project that “Downloads AWS documents, currently whitepapers, from AWS documentation website.” https://github.com/nragusa/aws-newrelease-slack - An AWS CDK application that sends AWS new service and feature release announcements to a Slack channel of your choice --- ### Lambda Function Urls - URL: https://brianpfeil.com/post/lambda-function-urls/ - Date: 2022-04-09 - Tags: lambda, aws, http code for article  pfeilbr/lambda-function-urls-playground learn Lambda function URLs Notes Function URLs are best for use cases where you must implement a single-function microservice with a public endpoint that doesn’t require the advanced functionality of API Gateway, such as request validation, throttling, custom authorizers, custom domain names, usage plans, or caching. Function URL format https://<url-id>.lambda-url.<region>.on.aws removes the 29s API Gateway limitation. your fn can take 15 min to respond and there is no timeout unexpected function URL caching can cause issues You can throttle the rate of requests that your Lambda function processes through a function URL by configuring reserved concurrency. Reserved concurrency limits the number of maximum concurrent invocations for your function. Your function’s maximum request rate per second (RPS) is equivalent to 10 times the configured reserved concurrency. For example, if you configure your function with a reserved concurrency of 100, then the maximum RPS is 1,000. can limit invocations via lambda condiation keys AWS Lambda - Permissions - Policy conditions Condition keys for AWS Lambda aws:SourceIp - can limit lambda invocation by source ip address Demo see template.yaml # deploy sam deploy --guided # access lambda url curl https://klm4pgp5qacvq52o4zptf6avei0cuczb.lambda-url.us-east-1.on.aws/ Resources Lambda function URLs Announcing AWS Lambda Function URLs: Built-in HTTPS Endpoints for Single-Function Microservices Adding a function URL to a CloudFormation template SCP to prevent people from creating open Lambda URLs use case discussion --- ### Plotly Dash API Gateway Lambda - URL: https://brianpfeil.com/post/plotly-dash-api-gateway-lambda/ - Date: 2022-04-04 - Tags: lambda, dash, aws code for article  pfeilbr/plotly-dash-api-gateway-lambda-playground Example of Dash application deployed to AWS Lambda + AWS API Gateway. based on ratajczak/dash-lambda Demo sam build --use-container sam deploy --guided open https://zzzzzzzzzz.execute-api.us-east-1.amazonaws.com/Prod/dash Resources ratajczak/dash-lambda - Example of Dash application deployed to AWS Lambda Uncaught ReferenceError: DashRenderer is not defined original README.md below Example of Dash application deployed to AWS Lambda Deployment The Serverless Application Model Command Line Interface (SAM CLI) is an extension of the AWS CLI that adds functionality for building and testing Lambda applications. It uses Docker to run your functions in an Amazon Linux environment that matches Lambda. It can also emulate your application’s build environment and API. To use the SAM CLI, you need the following tools. SAM CLI - Install the SAM CLI Python 3 installed Docker - Install Docker community edition To build and deploy your application for the first time, run the following in your shell: sam build --use-container sam deploy --guided The first command will build the source of your application. The second command will package and deploy your application to AWS, with a series of prompts: Stack Name: The name of the stack to deploy to CloudFormation. This should be unique to your account and region, and a good starting point would be something matching your project name. AWS Region: The AWS region you want to deploy your app to. Confirm changes before deploy: If set to yes, any change sets will be shown to you before execution for manual review. If set to no, the AWS SAM CLI will automatically deploy application changes. Allow SAM CLI IAM role creation: Many AWS SAM templates, including this example, create AWS IAM roles required for the AWS Lambda function(s) included to access AWS services. By default, these are scoped down to minimum required permissions. To deploy an AWS CloudFormation stack which creates or modified IAM roles, the CAPABILITY_IAM value for capabilities must be provided. If permission isn’t provided through this prompt, to deploy this example you must explicitly pass --capabilities CAPABILITY_IAM to the sam deploy command. Save arguments to samconfig.toml: If set to yes, your choices will be saved to a configuration file inside the project, so that in the future you can just re-run sam deploy without parameters to deploy changes to your application. You can find your API Gateway Endpoint URL in the output values displayed after deployment. Use the SAM CLI to build and test locally Build your application with the sam build --use-container command. dash-app$ sam build --use-container The SAM CLI installs dependencies defined in requirements.txt, creates a deployment package, and saves it in the .aws-sam/build folder. Test a single function by invoking it directly with a test event. An event is a JSON document that represents the input that the function receives from the event source. Test events are included in the events folder in this project. Run functions locally and invoke them with the sam local invoke command. dash-app$ sam local invoke Dash --event events/event.json The SAM CLI can also emulate your application’s API. Use the sam local start-api to run the API locally on port 3000. dash-app$ sam local start-api dash-app$ curl http://localhost:3000/ The SAM CLI reads the application template to determine the API’s routes and the functions that they invoke. The Events property on each function’s definition includes the route and method for each path. Events: HelloWorld: Type: Api Properties: Path: /hello Method: get Add a resource to your application The application template uses AWS Serverless Application Model (AWS SAM) to define application resources. AWS SAM is an extension of AWS CloudFormation with a simpler syntax for configuring common serverless application resources such as functions, triggers, and APIs. For resources not included in the SAM specification , you can use standard AWS CloudFormation resource types. Fetch, tail, and filter Lambda function logs To simplify troubleshooting, SAM CLI has a command called sam logs. sam logs lets you fetch logs generated by your deployed Lambda function from the command line. In addition to printing the logs on the terminal, this command has several nifty features to help you quickly find the bug. NOTE: This command works for all AWS Lambda functions; not just the ones you deploy using SAM. dash-app$ sam logs -n Dash --stack-name sam-dash --tail You can find more information and examples about filtering Lambda function logs in the SAM CLI Documentation . Cleanup To delete the sample application that you created, use the AWS CLI. Assuming you used your project name for the stack name, you can run the following: aws cloudformation delete-stack --stack-name sam-dash Resources See the AWS SAM developer guide for an introduction to SAM specification, the SAM CLI, and serverless application concepts. Next, you can use AWS Serverless Application Repository to deploy ready to use Apps that go beyond hello world samples and learn how authors developed their applications: AWS Serverless Application Repository main page --- ### AWS RDS - URL: https://brianpfeil.com/post/aws-rds/ - Date: 2022-01-23 - Tags: aws, rds code for article  pfeilbr/aws-rds-playground explore AWS RDS Demo see template.yaml # initial deploy with SAM sam deploy --guided # subsequent deploys with SAM (no prompt) sam deploy Resources Amazon Relational Database Service Documentation --- ### AWS Vscode Dev Container - URL: https://brianpfeil.com/post/aws-vscode-dev-container/ - Date: 2022-01-17 - Tags: aws code for article  pfeilbr/aws-vscode-dev-container a vscode devcontainer for aws development Visual Studio Code Remote - Containers extension lets you use a Docker container as a full-featured development environment. It allows you to open any folder inside (or mounted into) a container and take advantage of Visual Studio Code’s full feature set. Usage To use this repo as a template for an AWS project gh repo clone pfeilbr/aws-vscode-dev-container my-aws-project cd my-aws-project sam init Install into existing repo mkdir -p .devcontainer pushd .devcontainer curl -O https://raw.githubusercontent.com/pfeilbr/aws-vscode-dev-container/master/.devcontainer/Dockerfile curl -O https://raw.githubusercontent.com/pfeilbr/aws-vscode-dev-container/master/.devcontainer/devcontainer.json curl -O https://raw.githubusercontent.com/pfeilbr/aws-vscode-dev-container/master/.devcontainer/postCreateCommand.sh chmod +x postCreateCommand.sh popd Login with AWS SSO see https://gist.github.com/pahud/ba133985e1cf3531c09b5ea553a72739 mkdir ~/.bin && cd $_ wget https://raw.githubusercontent.com/pahud/vscode/main/.devcontainer/bin/aws-sso-credential-process && \ chmod +x aws-sso-credential-process aws configure set credential_process ${HOME}/.bin/aws-sso-credential-process touch ~/.aws/credentials && chmod 600 $_ aws configure sso --profile default Configuration # aws credentials are mounted into the guest container via `~/.aws`s # source=/home/ec2-user/.aws,target=/root/.aws `` --- ## TODO * ... --- ## Files * [`.devcontainer/devcontainer.json`](https://github.com/pfeilbr/aws-vscode-dev-container/blob/master/.devcontainer/devcontainer.json) * [`.devcontainer/Dockerfile`](https://github.com/pfeilbr/aws-vscode-dev-container/blob/master/.devcontainer/Dockerfile) --- ## Resources * [Developing inside a Container](https://code.visualstudio.com/docs/remote/containers) --- ### AWS Assume Role Using Custom OIDC Provider - URL: https://brianpfeil.com/post/aws-assume-role-using-custom-oidc-provider/ - Date: 2022-01-07 - Tags: aws, iam, oidc code for article  pfeilbr/aws-assume-role-using-custom-oidc-provider-playground Goal is to assume an AWS role via a custom OIDC provider. It is based on a trust relationship between the OIDC provider and AWS account. Allows assume role without credentials. Jenkins OIDC Provider Enable assume Role directly from Jenkin using a Jenkins OIDC provider in the target AWS account (no IAM User credentials / access key/secret). provide similar service for Jenkins as the following for gh actions aws-actions/configure-aws-credentials see also AWS federation comes to GitHub Actions create jenkins OIDC provider server. see pfeilbr/aws-custom-nodejs-oidc-provider-server-on-heroku create a jenkins OIDC provider in AWS account (AWS::IAM::OIDCProvider) jenkins job at start creates token (guid) and persists it - e.g. JENKINS_ID_TOKEN_REQUEST_TOKEN (github actions is called ACTIONS_ID_TOKEN_REQUEST_TOKEN) e.g could be persisted to service at https://vstoken.jenkins.company.com (https://vstoken.jenkins.company.com/.well-known/openid-configuration ) jenkins job issues assume role passing in JENKINS_ID_TOKEN_REQUEST_TOKEN - see https://github.com/aws-actions/configure-aws-credentials/blob/master/index.js#L93 https://github.com/aws-actions/configure-aws-credentials/blob/master/index.js https://github.com/aws-actions/configure-aws-credentials/blob/master/index.js#L268 - where ACTIONS_ID_TOKEN_REQUEST_TOKEN is used Secure Communication Between Actions and App JenkinsOidc: Type: AWS::IAM::OIDCProvider Condition: CreateOIDCProvider Properties: Url: https://vstoken.jenkins.company.com ClientIdList: - sts.amazonaws.com ThumbprintList: - a031c46782e6e6c662c2c87c76da9aa62ccabd8e “sub” (Subject) Claim The “sub” (subject) claim identifies the principal that is the subject of the JWT e.g. for github actions it’s repo:aidansteele/aws-federation-github-actions:ref:refs/heads/main “aud” (Audience) Claim The “aud” (audience) claim identifies the recipients that the JWT is intended for. e.g. for github actions it’s the repo url like https://github.com/aidansteele Resources aws-actions/configure-aws-credentials AWS federation comes to GitHub Actions pfeilbr/aws-custom-nodejs-oidc-provider-server-on-heroku Secure Communication Between Actions and App Deploy on AWS using Bitbucket Pipelines OpenID Connect Bitbucket / Ben Kehoe Twitter Thread --- ### Promise Pool - URL: https://brianpfeil.com/post/promise-pool/ - Date: 2021-12-11 - Tags: nodejs, concurrency code for article  pfeilbr/promise-pool-playground learn supercharge/promise-pool : Map-like, concurrent promise processing see index.js Resources supercharge/promise-pool Supercharge β€” an open-source Node.js framework, not just a web framework --- ### AWS SAM - URL: https://brianpfeil.com/post/aws-sam/ - Date: 2021-12-05 - Tags: aws, sam code for article  pfeilbr/aws-sam-playground playground for all things AWS SAM Demo cd sam-app sam sync --watch --stack-name sam-app sam logs --tail # run tests in `__tests__/` AWS_SAM_STACK_NAME=sam-app npm run test aws lambda invoke \ --cli-binary-format "raw-in-base64-out" \ --function-name "sam-app-helloFromLambdaFunction-SvAJyAV6zcDN" \ --payload '{"msg": "hello"}' \ output.log; cat output.log; rm output.log --- ### Iamlive - URL: https://brianpfeil.com/post/iamlive/ - Date: 2021-12-02 - Tags: aws, iam code for article  pfeilbr/iamlive-playground learn iann0036/iamlive - cli tool to Generate an IAM policy from AWS calls using client-side monitoring (CSM) or embedded proxy Demo CSM Mode (Client-side monitoring mode) Example with AWS CLI # run in CSM mode example and generate policy from aws cli # can add `--output-file iamlive-policy-output.json` to save generated policy in file iamlive --set-ini # run the following in another shell aws s3 ls s3://com.brianpfeil.scratch01 Proxy Mode Example with AWS CLI # run in proxy mode example and generate policy from aws cli iamlive --set-ini --mode proxy # run the following in another shell export AWS_CA_BUNDLE=~/.iamlive/ca.pem export HTTP_PROXY=http://127.0.0.1:10080 export HTTPS_PROXY=http://127.0.0.1:10080 CSM Mode (Client-side monitoring mode) Example with AWS SDK for JavaScript v2 see index.js # run in CSM mode example and generate policy from aws cli iamlive --set-ini # run the following in another shell # install node deps npm install # run nodejs script that uses `aws-sdk` node index.js Resources iann0036/iamlive --- ### Pre Commit - URL: https://brianpfeil.com/post/pre-commit/ - Date: 2021-11-24 - Tags: git code for article  pfeilbr/pre-commit-playground learn pre-commit Demo # install brew install pre-commit # generate config pre-commit sample-config > .pre-commit-config.yaml # setup git hook scrips pre-commit install # run against all the files pre-commit run --all-files Resources pre-commit --- ### Sceptre - URL: https://brianpfeil.com/post/sceptre/ - Date: 2021-11-06 - Tags: cloudformation, infrastructure-as-code code for article  pfeilbr/sceptre-playground learn sceptre , a tool to drive cloudformation StackGroup Config - config/dev/config.yaml - project_code, region, profile, etc. Stack Config - config/dev/template.yaml - config for specific stack. e.g. stack params, template_path, dependencies, hooks, etc. Stack - templates/template.yaml Demo via Get Started β€” Sceptre 2.6.3 documentation # install cli pipx install sceptre # create new project sceptre new project my-sceptre-project cd my-sceptre-project # create stack sceptre create dev/template.yaml # show stack outputs sceptre --ignore-dependencies list outputs dev/template.yaml # update stack sceptre update dev/template.yaml # delete stack sceptre delete dev/template.yaml Resources sceptre GitHub - Sceptre/sceptre: Build better AWS infrastructure Introduction to Sceptre: An AWS Cloudformation Orchestration Tool --- ### AWS Step Functions AWS SDK Integration - URL: https://brianpfeil.com/post/aws-step-functions-aws-sdk-integration/ - Date: 2021-10-15 - Tags: aws, sdk, step-functions code for article  pfeilbr/aws-step-functions-aws-sdk-integration-playground learn AWS Step Functions AWS SDK Integrations . Provides direct integration with 200+ AWS services Demo Based on Gather Amazon S3 bucket info using AWS SDK service integrations - AWS Step Functions . Deploy example01.yaml stack NOTE: ResultSelector added to ListBuckets to hard code an array containing a single bucket to reduce amount of processing. PROFILE="admin" REGION="us-east-1" STACK_NAME="aws-step-functions-aws-sdk-integration-playground" aws cloudformation deploy \ --profile "${PROFILE}" \ --region "${REGION}" \ --stack-name "${STACK_NAME}" \ --template-file example01.yaml \ --capabilities "CAPABILITY_IAM" "CAPABILITY_NAMED_IAM" "CAPABILITY_AUTO_EXPAND" # execute step fn aws stepfunctions start-execution \ --state-machine-arn "arn:aws:states:us-east-1:529276214230:stateMachine:Gather-S3-Bucket-Info-Standard" \ --input "{}" Example Execution Resources AWS Step Functions adds support for over 200 AWS Services with AWS SDK Integration Now β€” AWS Step Functions Supports 200 AWS Services To Enable Easier Workflow Automation | Amazon Web Services AWS SDK service integrations - AWS Step Functions --- ### GitHub Actions Configure AWS Credentials - URL: https://brianpfeil.com/post/github-actions-configure-aws-credentials/ - Date: 2021-10-06 - Tags: aws, github, iam code for article  pfeilbr/github-actions-configure-aws-credentials-playground learn aws-actions/configure-aws-credentials Demo PROFILE=admin REGION=us-east-1 STACK_NAME="configure-aws-credentials-playground" aws cloudformation deploy \ --profile "${PROFILE}" \ --region "${REGION}" \ --stack-name "${STACK_NAME}" \ --template-file sample-role.yaml \ --capabilities "CAPABILITY_IAM" "CAPABILITY_NAMED_IAM" \ --parameter-overrides GitHubOrg=pfeilbr \ RepositoryName=github-actions-configure-aws-credentials-playground \ # OIDCProviderArn="arn:aws:iam::529276214230:oidc-provider/vstoken.actions.githubusercontent.com" # run workflow manually gh workflow run default.yml # list runs gh run list --workflow=default.yml # show run details gh run view <run-id> # delete stack aws cloudformation delete-stack \ --profile "${PROFILE}" \ --region "${REGION}" \ --stack-name "${STACK_NAME}" successful workflow run with aws sts get-caller-identity Resources aws-actions/configure-aws-credentials AWS federation comes to GitHub Actions benkehoe/github-actions-boto3-demo: Demonstrate how GitHub OIDC token getting should be included in boto3 --- ### GitHub Actions - URL: https://brianpfeil.com/post/github-actions/ - Date: 2021-09-20 code for article  pfeilbr/github-actions-playground learn github actions Resources GitHub Actions Documentation - GitHub Docs --- ### Ngrok - URL: https://brianpfeil.com/post/ngrok/ - Date: 2021-09-20 code for article  pfeilbr/ngrok-playground learn ngrok npm package see index.js Demo expose local express.js web app npm install node index.js screenshots Resources GitHub - bubenshchykov/ngrok: Expose your localhost to the web. Node wrapper for ngrok. --- ### AWS Healthlake - URL: https://brianpfeil.com/post/aws-healthlake/ - Date: 2021-09-16 - Tags: aws code for article  pfeilbr/aws-healthlake-playground learn Health Data Lake and Healthcare Analytics - Amazon HealthLake - Amazon Web Services Concepts DataStore data is encrypted at rest Customers can choose an AWS owned KMS key or a Customer-managed KMS key when creating a Data Store. encrypted in transit Amazon HealthLake uses TLS 1.2 to encrypt data in transit through the public endpoint and through backend services. import data (line delimited json record - .ldjson) from S3 bucket where each line consists of a valid FHIR resource query data using Create, Read, Update, Delete functions via provided REST endpoint for a given DataStore use FHIR search After the data is ingested, it is indexed using Amazon ES, which makes the data searchable. Integrated medical natural language processing (NLP) the NLP results are automatically included in the results for a returned DocumentReference resource type. No separate actions/etc. required. Amazon HealthLake automatically integrates with natural language processing (NLP) for the DocumentReference resource type. The integrated medical NLP output is provided as an extension to the existing DocumentReference resource. The integration involves reading the text data within the resource, and then calling the following integrated medical NLP operations: DetectEntities-V2, InferICD10-CM, and InferRxNorm. export data to S3 Amazon HealthLake enables you to bulk export your files to an Amazon S3 bucket. Use either the console or start-fhir-export-job to begin an export job. Afterwards, you can use describe-fhir-export-job to monitor the status of the job and discover its properties. After the export job is complete, the data can then be visualized using AWS Quicksight or accessed by other AWS services. VPC endpoint (PrivateLink) support You can establish a private connection between your VPC and Amazon HealthLake by creating an interface VPC endpoint. Interface VPC endpoints are powered by AWSPrivateLink Examples # https://docs.aws.amazon.com/healthlake/latest/devguide/healthlake-examples-cli.html ​aws healthlake create-fhir-datastore \ --datastore-type-version R4 \ --preload-data-config PreloadDataType="SYNTHEA" \ --datastore-name "FhirTestDatastore" Resources Paging Doctor Cloud! Amazon HealthLake Is Now Generally Available | Amazon Web Services Amazon HealthLake Workshop https://www.hl7.org/fhir/patient-examples.html# - FHIR examples --- ### Hugging Face - URL: https://brianpfeil.com/post/hugging-face/ - Date: 2021-09-13 - Tags: machine-learning, nlp, python code for article  pfeilbr/hugging-face-playground learn hugging face transformers (transfer learning) based on Hugging Face Transformers Package – What Is It and How To Use It - KDnuggets Transformers is an opinionated library built for: NLP researchers and educators seeking to use/study/extend large-scale transformers models hands-on practitioners who want to fine-tune those models and/or serve them in production engineers who just want to download a pretrained model and use it to solve a given NLP task. Demo pipenv install pipenv install transformers pipenv install tensorflow # test install python -c "from transformers import pipeline; print(pipeline('sentiment-analysis')('we love you'))" # note first run will download the [pipeline] model (potentially time consuming) on first run pipenv run python main.py Resourcces Hugging Face – The AI community building the future. Hugging Face Transformers Package – What Is It and How To Use It - KDnuggets Hosting Hugging Face models on AWS Lambda for serverless inference | Amazon Web Services --- ### CDKTF - URL: https://brianpfeil.com/post/cdktf/ - Date: 2021-09-03 - Tags: aws, cdk, terraform code for article  pfeilbr/cdktf-playground learn cdktf. based on Build AWS Infrastructure with TypeScript Notes cdktf generates tf hcl json cdktf synth creates cdktf.out can use tf apply within a cdktf.out/stacks/* directory to deploy or let cdktf do both the synth and tf deploy Demo cdktf init --template=typescript --local # add aws provider to `cdktf.json` // "hashicorp/aws@~> 3.42" # creates `.gen` folder which is codegend .ts cdktf get # edit `main.ts` # deploy cdktf deploy # clean up cdktf destroy Directory Structure Deploy Output Destroy Output Resources Write CDK for Terraform configurations --- ### Azure - URL: https://brianpfeil.com/post/azure/ - Date: 2021-06-15 - Tags: azure azure Management IAM Service Principals Roles Role Assignments Resources Azure Policy Azure Resource Manager (ARM) Resources Azure Blueprints Blob Storage Resources Storage Queues Service Bus Queues Service Bus Topics Resources Event Grid Resources Event Hub Cosmos DB SQL Database SQL Managed Instance Static Web Apps Functions Resources Function Proxies Resources API Management (APIM) Resources SignalR Resources Key Vault CDN App Configuration Data Factory Synapse Analytics HDInsight Databricks Data Lake Analytics Logic Apps Resources Application Insights Cognitive Search Introducing Authentication Azure AD Authentication Options for Users Azure Cost Management + Billing documentation Management Account -> [Management Groups] -> Subscriptions -> Resources Groups -> Resources e.g. hierarchy in URI for Key Vault Resource /subscriptions/8a1f586d-1032-4471-803a-25126ef17c42/resourceGroups/resource-group-01/providers/Microsoft.KeyVault/vaults/pfeilkeyvault01 Management Groups (enable enterprise governance) are optional but allow you to group subscriptions and apply policies at management group level that are inherited by all contained subscriptions. (e.g. only create storage accounts in a given geography) Billing done at Subscription level Resource groups - scope for applying role-based access control (RBAC) permissions Regions Availability Zones Each Azure region is always paired with another region within the same geography. provide reliable services and data redundancy IAM Azure AD - stories identities in directory and governs access to azure resources. Identity - the fact of being something or someone. e.g. Users, Applications, Servers. Authentication - process of verification of identity Authorization - process of ensuring that only authenticated identities get access to the resources for which they have been granted access. Access Management - process of controlling, verifying, tracking and managing access to authorized users and applications. RBAC - role based access control Azure AD Roles - roles that allow admin access to global (tenant-level) settings and services. e.g. user and group admin, domain names, adding/removing user licenses, etc. Azure Roles - roles that define permissions to azure resources. Privileged Identity Management (PIM) - just in time access to elevated roles. Eligible roles are assigned to a user. That user can go into PIM to request access. Duration and reason can be set when requesting and assuming the elevated role. Service Principals An Azure service principal is a security identity used by user-created apps, services, and automation tools to access specific Azure resources. Think of it as a ‘user identity’ (login and password or certificate) with a specific role, and tightly controlled permissions to access your resources. It only needs to be able to do specific things, unlike a general user identity. It improves security if you only grant it the minimum permissions level needed to perform its management tasks. Similar to AWS IAM User create service principal in portal via Azure AD | App registrations # create service principal az ad sp create-for-rbac --name "http://service-principal-01" # assign role az role assignment create --assignee "00133c8e-a08e-490e-ae7c-872ea2debf1e" --role Contributor # login az login --service-principal -u <appid> --password {password-or-path-to-cert} --tenant {tenant} Roles permissions - read blob (Microsoft.Storage/storageAccounts/blobServices/containers/blobs/read). read storage queue (Microsoft.Storage/storageAccounts/queueServices/queues/messages/read) scopes - subscriptions or resource groups) where this role will be available for assignment. Azure RBAC scope covers management groups, subscriptions, resource groups, and resources Portal | Subscription | Access control (IAM) - assign / create / delete roles. Role Assignments process of attaching a role definition to a user, group, service principal, or managed identity at a particular scope for the purpose of granting access Role Definition (permissions) + Resource + Scope Can view / managed in portal via “Access control (IAM)” blade Resources Role-Based Access Control | AZ-900 Episode 28 - Cheat Sheet Use Azure service principals with Azure CLI Create an Azure AD app & service principal in the portal - Microsoft identity platform Securing Azure Services with Managed Identities Azure AD consent framework - describes process of grant permissions to an application and the user consent flow Azure Policy policies focus on resource properties e.g. only create resources where location is US. Policy Definitions - include ALLOW and DENY for singular check and singular effect Initiative - grouping of multiple policies can assign policy at the following scope levels: management group, subscription, resource group can add an exclusion scope that specifies where not to apply the policy show which resources are compliant and non-compliant with policies there are built-in policies and you can create you own custom policies summary azure portal view policy blocking resource creation Azure Resource Manager (ARM) infrastructure-as-code with json parameters, variables, resources, outputs nested templates external templates. reference via URL Resources Azure/azure-quickstart-templates Azure Blueprints similar to AWS Service Catalog helps with ensuring security and compliance Package of various Azure components (artifacts). Resource Groups, ARM Templates, Policy Assignments, Role Assignments central repository for pre-approved patterns and solutions Blueprint Definitions - container for artifacts Blueprint Assignments - assign to resource group and it deploys artifacts to it supports versioning. Blob Storage Resources Static website hosting in Azure Storage Storage Queues good for async worker processes 64 kb size limit on messages messages stored up to 7 days Service Bus Queues FIFO 256 KB - 1 MB message size duplicate detection (idempotent) supports in order and at-most-once delivery dead-letter queues message expiration Service Bus Topics topic and subscriptions support duplicate detection (idempotent) message TTL dead-letter queues Resources Service Bus Explorer on the Azure portal is now available in preview Event Grid AWS EventBridge Publishers - built-in events from azure services. custom events from own applications Event Sources - where the event happens. e.g. Azure Storage Topics - endpoint where the source sends events System topics - built-in topics provided by Azure services such as Azure Storage, Azure Event Hubs, and Azure Service Bus Custom topics - application and third-party topics Subscriptions - which events on a topic you’re interested in receiving. When creating the subscription, you provide an endpoint for handling the event Resources Azure Resource Manager template samples - Event Grid - Azure Event Grid Receive events from Azure Event Grid to an HTTP endpoint - Azure Event Grid Event Hub AWS Kinesis streaming partitions consumer groups, consumers listening streaming applications stream to blob storage (similar to aws firehose -> S3) as .avro files namespace -> hub Cosmos DB NoSQL multi-model / multiple interfaces - SQL, MongoDB, Cassandra, Tables, and Gremlin similar to AWS DynamoDB SQL Database PaaS Database Engine (not fully managed) handles upgrading, patching, backups, and monitoring, without user involvement similar to AWS RDS SQL Managed Instance … Static Web Apps allows you to build modern web applications that automatically publish from changes made in GitHub. Functions triggers and bindings Azure Durable Functions - lets you write stateful functions in a serverless compute environment. Similar to AWS Step Functions, but implemented as language level library custom handlers via binary that runs local http server deploy function via zip file. ensure correct nodejs folder structure az functionapp deployment source config-zip -g <resource_group> -n \ <app_name> --src <zip_file_path> When http triggered function is configured with Azure AD authentication , the identity information is in the req.headers Example "x-ms-client-principal-name": "brian.pfeil@gmail.com", "x-ms-client-principal-id": "38d35c72-5a26-464c-bbb3-c4487a1d4779", "x-ms-client-principal-idp": "aad", Resources mspnp/serverless-reference-implementation - great full-stack serverless reference Zip push deployment for Azure Functions JavaScript developer reference for Azure Functions Function Proxies Proxies are defined inside a proxies.json configuration file at the root of the project. see following for details https://docs.microsoft.com/en-us/azure/azure-functions/functions-proxies#advanced-configuration allow us to define a single API surface for multiple function apps map a path (/something) to backend URL. e.g. /hello -> https://durablefnsplayground01.azurewebsites.net/api/orchestrators/{functionName}?code=UD0/aY27zc3FPpjNjaAqMU73yVJuLB1tva3U9TaaWDAMzwEzg3qeVA== this way you don’t need to expose code query string parameter you can map a path and hard code a response for mocking Resources Work with proxies in Azure Functions Microsoft Azure Function Proxies in 5 Minutes API Management (APIM) AWS API Gateway import OpenAPI /Swagger spec can import a function app as an API in API Management APIM product - contains one or more APIs as well as a usage quota and the terms of use. Once a product is published, developers can subscribe to the product and begin to use the product’s APIs Subscriptions - developers who need to consume the published APIs must include a valid subscription key in HTTP requests when they make calls to those APIs can be scoped to product, all APIs, or an individual API need to provide ocp-apim-subscription-key header when calling Policies - allow the publisher to change the behavior of the API through configuration. specified in XML Allow for request and response transformations. Rate limiting. Similar to AWS APIG request/response mapping templates. as of 2021-02-12 takes about 30 min to create API Management instance Resources Import an Azure Function App as an API in API Management - Azure API Management Policies in Azure API Management SignalR real-time application websocket, SSE (EventSource), HTTP long polling Resources Enable automatic updates in a web application using Azure Functions and SignalR Service - Learn MicrosoftDocs/mslearn-advocates.azure-functions-and-signalr Key Vault AWS Secrets Manager functions can have Key Vault references which make the secrets available in environment variables. The secrets are also caches per function instance on startup Example Secret CDN CDN profile - collection of CDN endpoints App Configuration centralized service for management of application settings and feature flags supports key vault integration via references to the secrets stored in key vault integrates with event grid. emits events when App Configuration changes that you can subscribe and respond to e.g. A/B testing with percentages e.g. multiple apps use same settings like storage key Data Factory managed ETL concepts Pipelines - outer container Connections (Linked Services) - connection information for source(s) and sink(s) with credentials / keys Dataset - file formats and schema Activities - copy data, transform, etc. Triggers - on-demand/manual (REST API), scheduled, tumbling windows, event based via event grid for blob storage for a pipeline run Parameters - can be passed into pipeline and used with the various types. Synapse Analytics … HDInsight Apache Spark , Apache Hadoop , Apache Kafka , Apache HBase , Apache Storm , and Machine Learning Services like AWS EMR Databricks spark, hadoop, and friends … Data Lake Analytics … Logic Apps schedule, automate, and orchestrate tasks, business processes, and workflows when you need to integrate apps, data, systems, and services across enterprises or organizations Guidance The following flowchart summarizes the key questions to ask when you’re considering using Logic Apps. Resources Securing Logic App with Azure AD authentication using API Management Workflow Definition Language schema reference - Azure Logic Apps Application Insights author queries using Kusto Query Language Cognitive Search indexing push model - upload json docs for indexing pull model - point indexers where the content lives. e.g. azure blob storage querying - can include relevance tuning, autocomplete, synonym matching, fuzzy matching, pattern matching, filter, and sort. spelling mistakes, geospatial queries, paging, highlighting. Skillset - content type indexer for unstructured and structured content (docx, pdf, ). predefined ones from Microsoft, or custom skills that you create. e.g. text split skill, sentiment detection skill document cracking - understand the contents of the document. text, images, metadata, etc. portions of underlying tech is based on apache lucene Introducing Authentication OAuth 2.0 is specifically about authorization and permissions Open ID Connect (OIDC) is focused on authentication and built on OAuth 2.0. Has strict JWT format that part of the spec. identity token typically JWT user claims - e.g. username, email access token short lived (e.g. 1 hr) typically JWT scope refresh token long life (e.g. 14 days) can be used to request an access token without the user needing to login again Azure AD Authentication Options for Users Azure AD Connect - replicate objects in AD to Azure AD. AD is the source of truth. Azure Cost Management + Billing documentation https://docs.microsoft.com/en-us/azure/cost-management-billing/ --- ### AWS API Gateway to S3 Static Site - URL: https://brianpfeil.com/post/aws-api-gateway-to-s3-static-site/ - Date: 2021-05-21 - Tags: aws, s3, api-gateway code for article  pfeilbr/aws-api-gateway-to-s3-static-site-playground WARNING: you probably shouldn’t do this / there’s a better way just because you can do it doesn’t mean you should :) was curious if it could be done #awswishlist private s3 bucket websites with TLS support example of hosting static site (create-react-app) with api gateway -> s3 static site content at my-react-app Example Deployment Steps STAGE="dev" STACK_NAME="aws-api-gateway-to-s3-static-site-playground" # deploy resoruces sam deploy --guided # build react app pushd my-react-app # need to set root to subpath because of api gateway stage in path limitation PUBLIC_URL="/${STAGE}" npm run build popd # deploy site aws s3 sync my-react-app/build/ s3://aws-api-gateway-to-s3-static-sit-staticsitebucket-2k3gk0lgl72u # visit site open https://a4kesxi0gg.execute-api.us-east-1.amazonaws.com/dev/index.html screenshots Resources API-Gateway S3 Proxy - followed this console guide to get it to work. export as openapi from console and embed in template.yaml Tutorial: Create a REST API as an Amazon S3 proxy in API Gateway - Amazon API Gateway API Gateway S3 Integration Signature does not match with blank folder name Functionless S3 Integration inside a Serverless Casserole-Part 1 SAM Template for AWS Api Gateway Integration with S3 as AWS Service --- ### AWS App Runner - URL: https://brianpfeil.com/post/aws-app-runner/ - Date: 2021-05-19 - Tags: aws, containers code for article  pfeilbr/aws-app-runner-playground learn AWS App Runner Comments good service to get users “closer” to serverless on eases the onboarding to AWS for solutions less knobs/options to turn/get wrong it’s cheap , but not as cheap as lambda. public only endpoints for now similar goals and roadmap as GCP Cloud Run (AWS is playing catch-up) More choices for containers and potentially confusing for customers (container options menu is getting pretty big) Notes prefer App Runner AWS managed container runtimes (e.g. Node and Python ). Similar to lambda, ElasticBeanstalk, CloudFoundry, Heroku, etc. App Runner product manager said the managed container runtimes as based on elastic beanstalk. (source video - AWS App Runner Workshop ) aws takes care of / not your responsibility load balanced autoscale no clusters App Runner configuration file (apprunner.yaml) - connect directly to github config options health check config cpu, memory instance role see Create a source code repository service for all options custom domain support Node and Python managed container runtimes to start. Planned to support and follow order of lambda supported runtimes . public endpoints only (as of 2021-05-18) - see roadmap #1 , #2 for private endpoints and access to VPC resources. Pricing You are charged for the compute and memory resources used by your application. In addition, if you automate your deployments, you will pay a set monthly fee for each application that covers all automated deployments for that month. If you opt to deploy from source code, you will pay a build fee for the amount of time it takes App Runner to build a container from your source code. Demo # setup mkdir hello-app-runner touch apprunner.yaml # install venv and deps pipenv install # run locally pipenv run python app.py # deploy SERVICE_NAME="hello-app-runner-01" aws apprunner create-service --service-name ${SERVICE_NAME} \ --source-configuration file://apprunner.yaml Resources Introducing AWS App Runner Hello AWS App Runner - youtube AWS App Runner Workshop - youtube. good details on roadmap. aws tech pm is a part of it. Documentation | AWS App Runner aws-containers/apprunnerworkshop apprunner-roadmap - vote up issues important to your needs aws-containers/hello-app-runner AWS App Runner API Reference Pricing --- ### Thinking Architecturally - URL: https://brianpfeil.com/post/thinking-architecturally/ - Date: 2021-05-13 - Tags: architecture Source Resources Thinking Architecturally - Nathaniel Schutta - youtube Thinking Architecturally: Lead Technical Change Within Your Engineering Team - oreilly book Infrastructure & Ops Superstream Series: CI/CD - oreilly video Building Evolutionary Architectures - oreilly book Notes (Thinking Architecturally - Nathaniel Schutta youtube video) AaaS - Architecture-as-a-Service many competing agendas technology change is a constant nobody wants to be on legacy platform java foundational tech for aws, google, twitter, netflix, etc. how to evaluate technology chasing the new things computer science - lifetime learning education doesn’t end. lifetime of learning. developers have strong opinions new and old tech https://youtu.be/d5bNZX8tpiI?t=681 new - unfinished, unbuggy, unproven old - refined, stable, tested predictable hype cycle how do we know where not to use tech. takes trial and error. asked to build something never built before, using tools we got a few weeks ago, shocked we don’t know how long it takes developer get bored quickly learning keeps it fresh commit at some point. can’t constantly experiment bleeding edge … means you will bleed pioneers … ones with arrows in back hope is not a strategy need to be deliberate when it comes to technology choices challenge is how to keep up (“technology merry go round”) make it part of routine to learn “morning coffee” - peruse the tech news attention is most precious resource you have attention is a resource. it doesn’t scale. don’t waste your attention. have to be selective. if its a big enough deal, you will hear about it. how do we know where to invest our time? be aware of bias skating to where the puck was thoughtworks technology radar 20% projects are good space, but fallen out of favor in some circles innovation fridays architectural briefing - one person does research, presents back to team. short ~45min briefing pans out -> workshop/invest more in it greenfield always works. need to use in target env to understand constraints start with low risk capability / not company core security - want to stay safe, need to go fast. time feeds black hats policy needed. measure and enforce, hard part pros and cons - always trade-offs “it depends” means -> tell me more. not a stopper. ack the negatives. the most insight is learned from this question “You haven’t mastered a tool until you understand when it should not be used” - Kelsey what is the appropriate scale for evaluation. e.g. 1-5 harvey balls. images to remove the “hard number” focus effect nobody tells you which criteria to use, up to you, need to make with incomplete information architects spread thin, need to scale with principles, north stars, etc. arch agile backlash. chose tech that can evolve - evolutionary arch create arch with expectation of change. Γ  la werner “everything fails, all the time” “evolutionary arch supports guided, incremental change across multiple dimensions” [arch]fitness functions 0: make arch change, 1: eval result, 2: closer to goal? yes -> keep, no -> trash, 3: -> goto 0 (basic regression iterations as in stats/ML) set of tests we exec to validate arch. (arch unit tests) reminds everyone on the team what’s important for your arch informs discussions around tradeoffs need to be visible to team need to be reviewed on regular basis just because you can measure it doesn’t mean it matters avoid resume driven design --- ### Hashicorp Vault - URL: https://brianpfeil.com/post/hashicorp-vault/ - Date: 2021-05-04 - Tags: secrets-management, security code for article  pfeilbr/hashicorp-vault-playground learn HashiCorp Vault Notes iam and ec2 auth methods iam uses sts:GetCallerIdentity under the hood vault server receives request with attributes to construct sigv4 and issues the request to AWS STS AWS STS API endpoint is wide open / available to anyone. No auth required to issue request to it. vault roles map/bind to aws roles - vault roles add additional capabilities (e.g. leases, finer grain policies, etc.). new layer to manage Demo # install macOS (single golang binary) brew tap hashicorp/tap brew install hashicorp/tap/vault # start in-memory server vault server -dev # set env vars export VAULT_ADDR='http://127.0.0.1:8200' export VAULT_TOKEN="s.uObLGyypsT6LWnxNaKNurYaV" # ok to expose for demo. ephemeral for in-memoery dev # confirm connectivity / env vars / server running vault status # put secret vault kv put secret/hello foo=bar # get secret (json output) vault kv get -format=json secret/hello # delete secret vault kv delete secret/hello # list secrets engines vault secrets list -format=json # enable aws secrets engine vault secrets enable -path=aws aws # make aws keys available to env export AWS_ACCESS_KEY_ID=<aws_access_key_id> export AWS_SECRET_ACCESS_KEY=<aws_secret_key> # configure AWS secrets engine vault write aws/config/root \ access_key=$AWS_ACCESS_KEY_ID \ secret_key=$AWS_SECRET_ACCESS_KEY \ region=us-east-1 # create role vault write aws/roles/my-role \ credential_type=iam_user \ policy_document=-<<EOF { "Version": "2012-10-17", "Statement": [ { "Sid": "Stmt1426528957000", "Effect": "Allow", "Action": [ "ec2:*" ], "Resource": [ "*" ] } ] } EOF # generate access key pair for role # creates an IAM user with the policy inlined. # e.g. arn:aws:iam::529276214230:user/vault-root-my-role-1620146368-5005 vault read aws/creds/my-role Screenshots Example IAM User Created by Vault Resources Vault Tutorials - HashiCorp Learn Getting Started | Vault - HashiCorp Learn HTTP API: Libraries | Vault by HashiCorp AWS - Auth Methods | Vault by HashiCorp hashicorp/vault-lambda-extension - utilizes the AWS Lambda Extensions API to help your Lambda function read secrets from your Vault deployment https://registry.terraform.io/modules/hashicorp/vault/aws/latest/examples/vault-iam-auth Serverless lambda with vault β€” Avoiding committing your passwords --- ### Multi-Account AWS Environment - URL: https://brianpfeil.com/post/multi-account-aws-environment/ - Date: 2021-04-29 - Tags: aws, governance, management Multi-Account AWS Environment options. AWS Landing Zone In short, it a collection of AWS accounts and other mechanisms to establish guardrails within those accounts. Created as part of AWS Solutions Library . AWS Landing Zone is currently in Long-term Support and will not receive any additional features Should now use AWS Control Tower zip file with CloudFormation templates builds accounts types enables AWS Config, apply config rules enables CloudTrail enables SSO and AD connector Builds Account Vending Machine - self-service portal using Service Catalog to allow devs to request accounts, which are then created via CodePipeline Core Accounts The following landing zone managed accounts are created as part of landing zone Log Archive - consolidated log files Security - creates auditor (read-only) and administrator (full-access) cross-account roles from a Security account to all AWS Landing Zone managed accounts. Also master Amazon GuardDuty account Shared services - AD, DNS, LDAP Sandbox Accounts - devs can experiment / PoCs/ etc. Business Unit accounts - dev, test, prod workloads AWS Control Tower Better packaged and managed AWS Landing Zone. AWS Landing Zone as a first class AWS service. Functions as a pre-baked layer of abstraction on top of AWS Organizations, AWS Config, CloudTrail, CloudFormation, and a few other services All in AWS Console. No IaC for it. Control Tower Landing Zone Creating Service Catalog | Create Control Tower Manged Account superwerker Opinionated layer on top of Control Tower. Automates the setup of an AWS Cloud environment with prescriptive best practices. AWS Organization Formation Infrastructure as Code (IaC) tool for AWS Organizations. Implemented using CloudFormation Custom Resource Types (providers) which then orchestrate and call the relevant AWS APIs Resources AWS Control Tower Customizations for AWS Control Tower | Implementations | AWS Solutions AWS Organizations superwerker AWS Organization Formation AWS CONTROL TOWER - How to Automate Landing Zone deployment with AWS Control Tower | Detailed DEMO --- ### AWS Chalice - URL: https://brianpfeil.com/post/aws-chalice/ - Date: 2021-04-28 - Tags: aws, serverless, python, lambda code for article  pfeilbr/aws-chalice-playground learn aws/chalice framework for writing serverless apps in python. It allows you to quickly create and deploy applications that use AWS Lambda. Notes focus is on building REST APIs (API Gateway + lambda) event sources support - cron, EventBridge, S3, SNS, SQS, Kinesis, DynamoDB compiles down to SAM template + zip containing python code + dependencies can combine with CDK to get all CDK features + chalice easy workflow/deploy benefits Demo # install in isolated venv via pipx pipx install chalice # create new project chalice new-project helloworld # deploy cd helloworld chalice deploy # see `helloworld/.chalice` for package artifacts # test api gateway endpoint curl https://zqzmirc025.execute-api.us-east-1.amazonaws.com/api/ # output: {"hello":"world"} # generate package chalice package packaged/ # SAM template created at `helloworld/packaged/sam.json` # bundled python code + deps at `helloworld/packaged/deployment.zip` # tear down chalice delete contents of helloworld/packaged/deployment.zip Resources aws/chalice --- ### AWS CDK Step Functions - URL: https://brianpfeil.com/post/aws-cdk-step-functions/ - Date: 2021-04-26 - Tags: aws, cdk, step-functions code for article  pfeilbr/aws-cdk-step-functions-playground CDK app providing API Gateway endpoint to provision a CloudFormation stack. Endpoint is backed by step function that initiates the create-stack and polls (describe-stack) on an interval for completion. This can be used as a backend to provision a AWS CloudFormation Custom Resource Type that itself is backed by a set of AWS services. see CDK stack components at lib/aws-cdk-step-functions-playground-stack.ts Architecture Dependencies CDK awscurl Demo # install deps npm install # deploy cdk deploy --force --require-approval never # test with IAM auth (aws sigv4 request) awscurl --service execute-api -X POST https://7t0zeiul1l.execute-api.us-east-1.amazonaws.com/prod/ -d '{"foo": "bar"}' Welcome to your CDK TypeScript project! This is a blank project for TypeScript development with CDK. The cdk.json file tells the CDK Toolkit how to execute your app. Useful commands npm run build compile typescript to js npm run watch watch for changes and compile npm run test perform the jest unit tests cdk deploy deploy this stack to your default AWS account/region cdk diff compare deployed stack with current state cdk synth emits the synthesized CloudFormation template --- ### AWS CloudFormation Custom Resource Type - URL: https://brianpfeil.com/post/aws-cloudformation-custom-resource-type/ - Date: 2021-04-23 - Tags: aws, cloudformation code for article  pfeilbr/aws-cloudformation-custom-resource-type-playground examples for creating CloudFormation extensions and specifically CloudFormation Custom Resource Type An extension is an artifact, registered in the CloudFormation Registry, which augments the functionality of CloudFormation in a native manner You can use the CloudFormation CLI to register extensionsβ€”both those you create yourself, as well as ones shared with youβ€”with the CloudFormation registry. This enables you to use CloudFormation capabilities to create, provision, and manage these custom types in a safe and repeatable manner, just as you would any AWS resource There are the following four types of CloudFormation extension mechanisms: CloudFormation Custom Resources CloudFormation Module CloudFormation Custom Resource Types CloudFormation Macros Example Custom Resource Types python-custom-resource-type-example typescript-custom-resource-type-example Notes Custom resources can be backed by lambda or SNS topic TODO Org::Service::WAFCDN -> CDK app containing API Gateway (custom auth token | or cloudformation security context identity) -> Step Fn -> create stack -> wait stack -> loop custom resource lambda handler is just a pass through to APIG provisioning the resources. store API Key in secrets manager and reference from cfn custom resource type property - Sid: DeleteAppRoles Effect: Allow Action: - wafcdn:* Resource: !Sub "arn:aws:cloudformation:${AWS::Region}:${AWS::AccountId}:type/resource/MyOrg-MyService-MyResource/*" Resources User Guide for Extension Development Use Python to manage third-party resources in AWS CloudFormation | Amazon Web Services Learn Best Practices for Implementing AWS Lambda-backed Custom Resources with AWS CloudFormation Writing an AWS CloudFormation Resource Provider in Python: Step by Step - Cloudar CloudFormation Resource Providers - A Chicken and Egg Problem Resolve the "Resource timed out waiting for creation of physical resource" error in AWS CloudFormation Deploying CloudFormation resource type --- ### AWS CDK Pipeline - URL: https://brianpfeil.com/post/aws-cdk-pipeline/ - Date: 2021-04-15 - Tags: aws, cdk, codepipeline, cicd code for article  pfeilbr/aws-cdk-pipeline-playground learn AWS CDK Pipelines Example CI/CD pipeline and solution code exist in single CDK project / repo in all authored in same language (typescript). Any commits to main automatically trigger a CodePipeline to run and deploy changes to production. The changes can be to the solution or the pipeline itself. Summary Simple API Gateway -> Lambda solution to exercise CDK pipeline capabilities. via AWS Solutions Constructs | aws-apigateway-lambda Creation of PreProd enviroment (stack) with e2e integration tests. Creation of Prod environment (stack) Automated creation of metrics, alarms, and notification (sns email) for API Gateway and Lambda resources via awslabs/cdk-watchful Automated notification (sns email) of CodePipeline state change events (STARTED, SUCCEEDED, FAILED) via EventBridge -> SNS rule. High-level Development Workflow Create solution stack lib/aws-cdk-pipeline-playground-stack.ts (API gateway -> Lambda). Create stage lib/aws-cdk-pipeline-demo-stage.ts (CdkpipelinesDemoStage) that wraps the solution stack (AwsCdkPipelinePlaygroundStack) for CodePipeline Create pipeline stack lib/aws-cdk-demo-pipeline-stack.ts and add CdkpipelinesDemoStage stage to it Steps npx cdk init --language=typescript CDK_VERSION="1.97.0" # install pipeline deps npm install \ @aws-cdk/aws-codepipeline@$CDK_VERSION \ @aws-cdk/aws-codepipeline-actions@$CDK_VERSION \ @aws-cdk/pipelines@$CDK_VERSION # leverages `@aws-solutions-constructs/aws-apigateway-lambda` # see https://docs.aws.amazon.com/solutions/latest/constructs/aws-apigateway-lambda.html # note CDK version must match `aws-solutions-constructs` version (e.g. $CDK_VERSION) npm i \ @aws-cdk/aws-apigateway@$CDK_VERSION \ @aws-cdk/aws-lambda@$CDK_VERSION \ @aws-solutions-constructs/aws-apigateway-lambda@$CDK_VERSION # build locally npm run build # test npm run test # create *this* repo in github and do initial push to ensure it exists for # CodePipeline to find # provision pipeline # [optional] if not already ran. bootstrap for each target account+region combination cdk bootstrap aws://AWS_ACCOUNT_NUMBER/us-east-1 cdk bootstrap aws://AWS_ACCOUNT_NUMBER/us-west-1 # ensure `cdk-pipeline-01-github-token` exists in Secrets Manager # one-time operation, deploy the pipeline stack from local machine cdk deploy --force --require-approval never # add stage to pipeline. this is the api gateway -> lambda stack code lib/aws-cdk-demo-pipeline-stack.ts # edit # push changes and pipeline will run and deploy PreProd stage npm run build git commit -am 'Add PreProd stage' git push # the pipeline automatically reconfigures itself to add the new stage and # deploy to it # modify solution and/or pipeline, commit, and iterate Notes Each pipeline stage is compiled into it’s own cloud assembly as follows: cdk.out/assembly-Dev cdk.out/assembly-CdkpipelinesDemoPipelineStack-PreProd cdk.out/assembly-CdkpipelinesDemoPipelineStack-Prod You can deploy and individual stage (cfn stack) by itself. For example, for dev. npm run build # If necessary, to recompile the Lambda sources cdk synth cdk -a cdk.out/assembly-Dev deploy --force --require-approval never # ensure dev account is bootstrapped first npx cdk bootstrap \ --cloudformation-execution-policies arn:aws:iam::aws:policy/AdministratorAccess \ aws://DEVELOPER_ACCOUNT/us-east-1 see cdk-deploy-to.sh for handy deployment script for different account+region combinations. Remember to bootstrap target account+region with cdk bootstrap aws://account/region had to downgrade npm 7 to 6 to get around types error when running npm run build on codebuild. see self mutating cdk pipeline fails after updating cdk version from 1.85.0 to 1.92.0 Β· Issue #13541 Β· aws/aws-cdk # steps npm install -g npm@6 rm package-lock.json npm i Screenshots AWS Console | CodePipeline SNS Email Notifications CodePipeline State Change COMPLETED API Gateway 5XX Count Threshold Hit Lambda Error Count Threshold Hit Resources CDK Pipelines: Continuous delivery for AWS CDK applications | Amazon Web Services cdkworkshop.com | CDK PIPELINES @aws-cdk/pipelines module Β· AWS CDK original cdk generated README.md below Welcome to your CDK TypeScript project! This is a blank project for TypeScript development with CDK. The cdk.json file tells the CDK Toolkit how to execute your app. Useful commands npm run build compile typescript to js npm run watch watch for changes and compile npm run test perform the jest unit tests cdk deploy deploy this stack to your default AWS account/region cdk diff compare deployed stack with current state cdk synth emits the synthesized CloudFormation template --- ### ReactCDK RFC - URL: https://brianpfeil.com/post/reactcdk-rfc/ - Date: 2021-04-11 - Tags: cdk, react, rfc original RFC proposal date 2020-09-27 Issue | ReactCDK: Add JSX/TSX Support PR | “RFC 256: ReactCDK - Add JSX/TSX Support” Rendered Proposal (feature: react-cdk-add-jsx-tsx-support) Summary Support the expression of cloud infrastructure resources via React JSX / TSX. Cloud solutions are composed of many services. Being able to efficiently and naturally express the composition is a developer ergonomics/DX concern. README Create CDK applications using React. Hello ReactCDK Example ReactCDK application const { ReactCDK, App, Stack } = require('@aws-cdk/react') const Bucket = require('@aws-cdk/react/s3') const { CloudFrontDistribution, Origins, DefaultOrigin } = require('@aws-cdk/react/cloudfront') const { Api, Resource, Integration } = require('@aws-cdk/react/apigateway') const Lambda = require('@aws-cdk/react/lambda') const EchoLambda = ( <Lambda> { async ({event, context}) => ({ "statusCode": 200, "headers": { "Content-Type": "application/json" } "body: JSON.stringify(event) }) } </Lambda> ) const EchoApi = ( <Api> <Resource path="*"> <Integration type="lambda"> <EchoLambda /> </Integration> </Resource> </Api> ) const WebsiteBucket = () => ( <Bucket src="./public" /> ) const MyCloudFrontDistribution = ( <CloudFrontDistribution> <Origins> <DefaultOrigin oai="true"> <WebsiteBucket /> </DefaultOrigin> <Origin path="/api/*"> <EchoApi /> </Origin> </Origins> </CloudFrontDistribution> ) const MyApp = ( <App> <Stack> <MyCloudFrontDistribution /> </Stack> </App> ) ReactCDK.render(MyApp, {region: 'us-east-1'}) Motivation Why are we doing this? Developer accessibility. Expanding and making the AWS platform accessible to a larger group of people. Library / framework support is a natural evolution on top of already supported CDK languages. React is familiar to a large group of frontend developers. It’s a generalized tech that allows developers to naturally express composition, which is the core tenet of serverless / serviceful solutions. This eases developers into more advanced AWS usage and therefore enabling them to get more value from the platform. Library/framework support on top of already supported language (ts) via jsii What use cases does it support? The large pool of existing React developers who have experience with React and it’s component based composition idioms. Allows them to reuse and leverage their existing knowledge to build full stack cloud solutions. What is the expected outcome? More people leveraging CDK suite -> more innovation and solutions -> advance humanity. Design Summary Code generate a set of ReactCDK components that has feature parity with all TS components from cfn constructs to solutions constructs. Detailed Design Codegen ReactCDK components A set of ReactCDK components that has feature parity with all TS components from cfn constructs to solutions constructs. mapping from react concepts of props, state, children, render props to typescript CDK constructors, params, hierarchical nesting (parent), etc. code generated based on the jsii manifests of CDK modules via @eladb custom react renderer / reconciler where cloud (cfn) is it’s render target getting there via CDK language layer. ReactCDKBaseComponent (class or functional component/hooks friendly) ReactCDKPatternComponent - pre-composed infra for common use cases. leverage React ecosystem adjacencies (e.g. react storybook, etc.) wrap these ReactCDK components for Amplify β€œBackend” library special cases of lambda function code being webpack + babel + etc being transpiled, bundled, etc. and handed over to CDK provisioning layer. this is a bit more involved and also needs research. this is the blending of app code and infra resources it interacts with. This is somewhat similar to Meteor , but instead of only mongodb, it’s every cfn supported AWS service. Need to not make same mistakes as meteor. step function special cases where the components that make it up are customized for developer ergonomics/dx. this is additional alternative to amplify add [backend] (e.g. api, auth, etc.) Drawbacks React is not universally liked. If it becomes associated with CDK there is potential brand/product damage. CDK is associated and tied to AWS brand. AWS tends to support all “popular” frameworks equally. Favoring one might not be good for brand. Might draw demand / more work for Vue.js, Angular, Svelete, etc. support Frontend libraries and frameworks come and go much more frequently than programming languages. React itself has gone through major “idiomatic” changes throughout it’s short lifetime. For example, classes, functional components, hooks. JSX has remained stable, but there could be “another” evolution step that makes JSX a moving target. A new layer and choice is introduced to the end user. Should I use a supported programming language or go up a layer and use a supported higher-level technology. Rationale and Alternatives leverage developers existing knowledge around technologies that easy the use of component oriented solutions. there are no technical negative impacts in not doing this. Purely additive. There is only the “potential” lost opportunity cost. Adoption Strategy Leverage existing Amplify brand platform/position with frontend developer community. This is an additive “personality” / choice for a target CDK user type. Does not impact existing code bases. Unresolved questions mapping from react concepts of props, state, children, render props to typescript CDK constructors, params, hierarchical nesting (parent), etc. Future Possibilities There currently is a “locality of code” issue with CDK, SAM, CFN, etc. The infrastructure I define is not close to the application code that leverages it (e.g. lambda code living in separate file, referencing it via s3://.zip). Explicit IAM permissions (or what could be codegened) live only in infra code. Application code events expressed in code seamlessly leveraging EventBridge for example without the developer needing to do any additional work. Would be a step towards a “cloud native programming language” where the programming language control constructs naturally scale and map to corresponding services (e.g. control flow -> step fn, events -> EventBridge, lambdas (lang level) -> lambdas :)) Implementation Plan Resources original “concept” conversation on twitter Hello ReactCDK app concept gist Part 1/3 - Beginners guide to Custom React Renderers. How to build your own renderer from scratch? --- ### AWS Well-Architected - URL: https://brianpfeil.com/post/aws-well-architected/ - Date: 2021-04-09 - Tags: aws, aws-well-architected, architecture code for article  pfeilbr/aws-well-architected-playground deep dive on all things AWS Well-Architected Key Points consistent pre-launch review process against AWS best practices helps you understand the pros and cons of decisions you make while building systems review process is a conversation and not an audit. working together to improve. practical advice. goal is not to have “perfect” architecture from the start. identify areas for improvement and choose a couple that delivery the most value AWS does not provide prescriptive guidance on how to perform the review. WA tool is the closest. concepts: Pillars -> Design Principles -> Questions enables: learn -> measure -> improve iterative cycle input: answer questions, output: improvement plan (PDF reports) learning / education - can be used as standalone tool solely for learning what the best practices are milestone - record the state of a workload for given point in time. e.g. original design, design review, v1, v2 Use Cases learning best practices for the cloud technology governance portfolio management - inventory of workloads, historical decisions made, risks, highlights where to invest Well-Architected Framework The AWS Well-Architected Framework helps you to design and operate a reliable, secure, efficient, and cost-efficient systems on AWS. It also helps you constantly measure your architecture against best practices and provides you an opportunity to improve your architecture. 5 Pillars Operational Excellence Security Reliability Performance Efficiency Cost Optimization Design Principles General design principles Stop guessing your capacity needs: If you make a poor capacity decision when deploying a workload, you might end up sitting on expensive idle resources or dealing with the performance implications of limited capacity. With cloud computing, these problems can go away. You can use as much or as little capacity as you need, and scale up and down automatically. Test systems at production scale: In the cloud, you can create a production-scale test environment on demand, complete your testing, and then decommission the resources. Because you only pay for the test environment when it’s running, you can simulate your live environment for a fraction of the cost of testing on premises. Automate to make architectural experimentation easier: Automation allows you to create and replicate your workloads at low cost and avoid the expense of manual effort. You can track changes to your automation, audit the impact, and revert to previous parameters when necessary. Allow for evolutionary architectures: In a traditional environment, architectural decisions are often implemented as static, onetime events, with a few major versions of a system during its lifetime. As a business and its context continue to evolve, these initial decisions might hinder the system’s ability to deliver changing business requirements. In the cloud, the capability to automate and test on demand lowers the risk of impact from design changes. This allows systems to evolve over time so that businesses can take advantage of innovations as a standard practice. Drive architectures using data: In the cloud, you can collect data on how your architectural choices affect the behavior of your workload. This lets you make fact-based decisions on how to improve your workload. Your cloud infrastructure is code, so you can use that data to inform your architecture choices and improvements over time. Improve through game days: Test how your architecture and processes perform by regularly scheduling game days to simulate events in production. This will help you understand where improvements can be made and can help develop organizational experience in dealing with events. Operational excellence Perform operations as code: In the cloud, you can apply the same engineering discipline that you use for application code to your entire environment. You can define your entire workload (applications, infrastructure) as code and update it with code. You can implement your operations procedures as code and automate their execution by triggering them in response to events. By performing operations as code, you limit human error and enable consistent responses to events. Make frequent, small, reversible changes: Design workloads to allow components to be updated regularly. Make changes in small increments that can be reversed if they fail (without affecting customers when possible). Refine operations procedures frequently: As you use operations procedures, look for opportunities to improve them. As you evolve your workload, evolve your procedures appropriately. Set up regular game days to review and validate that all procedures are effective and that teams are familiar with them. Anticipate failure: Perform β€œpre-mortem” exercises to identify potential sources of failure so that they can be removed or mitigated. Test your failure scenarios and validate your understanding of their impact. Test your response procedures to ensure that they are effective, and that teams are familiar with their execution. Set up regular game days to test workloads and team responses to simulated events. Learn from all operational failures: Drive improvement through lessons learned from all operational events and failures. Share what is learned across teams and through the entire organization. Security Implement a strong identity foundation: Implement the principle of least privilege and enforce separation of duties with appropriate authorization for each interaction with your AWS resources. Centralize identity management, and aim to eliminate reliance on long-term static credentials. Enable traceability: Monitor, alert, and audit actions and changes to your environment in real time. Integrate log and metric collection with systems to automatically investigate and take action. Apply security at all layers: Apply a defense in depth approach with multiple security controls. Apply to all layers (for example, edge of network, VPC, load balancing, every instance and compute service, operating system, application, and code). Automate security best practices: Automated software-based security mechanisms improve your ability to securely scale more rapidly and cost-effectively. Create secure architectures, including the implementation of controls that are defined and managed as code in version-controlled templates. Protect data in transit and at rest: Classify your data into sensitivity levels and use mechanisms, such as encryption, tokenization, and access control where appropriate. Keep people away from data: Use mechanisms and tools to reduce or eliminate the need for direct access or manual processing of data. This reduces the risk of mishandling or modification and human error when handling sensitive data. Prepare for security events: Prepare for an incident by having incident management and investigation policy and processes that align to your organizational requirements. Run incident response simulations and use tools with automation to increase your speed for detection, investigation, and recovery. Reliability Automatically recover from failure: By monitoring a workload for key performance indicators (KPIs), you can trigger automation when a threshold is breached. These KPIs should be a measure of business value, not of the technical aspects of the operation of the service. This allows for automatic notification and tracking of failures, and for automated recovery processes that work around or repair the failure. With more sophisticated automation, it’s possible to anticipate and remediate failures before they occur. Test recovery procedures: In an on-premises environment, testing is often conducted to prove that the workload works in a particular scenario. Testing is not typically used to validate recovery strategies. In the cloud, you can test how your workload fails, and you can validate your recovery procedures. You can use automation to simulate different failures or to recreate scenarios that led to failures before. This approach exposes failure pathways that you can test and fix before a real failure scenario occurs, thus reducing risk. Scale horizontally to increase aggregate workload availability: Replace one large resource with multiple small resources to reduce the impact of a single failure on the overall workload. Distribute requests across multiple, smaller resources to ensure that they don’t share a common point of failure. Stop guessing capacity: A common cause of failure in on-premises workloads is resource saturation, when the demands placed on a workload exceed the capacity of that workload (this is often the objective of denial of service attacks). In the cloud, you can monitor demand and workload utilization, and automate the addition or removal of resources to maintain the optimal level to satisfy demand without over- or under-provisioning. There are still limits, but some quotas can be controlled and others can be managed (see Manage Service Quotas and Constraints). Manage change in automation: Changes to your infrastructure should be made using automation. The changes that need to be managed include changes to the automation, which then can be tracked and reviewed. Performance efficiency Democratize advanced technologies: Make advanced technology implementation easier for your team by delegating complex tasks to your cloud vendor. Rather than asking your IT team to learn about hosting and running a new technology, consider consuming the technology as a service. For example, NoSQL databases, media transcoding, and machine learning are all technologies that require specialized expertise. In the cloud, these technologies become services that your team can consume, allowing your team to focus on product development rather than resource provisioning and management. Go global in minutes: Deploying your workload in multiple AWS Regions around the world allows you to provide lower latency and a better experience for your customers at minimal cost. Use serverless architectures: Serverless architectures remove the need for you to run and maintain physical servers for traditional compute activities. For example, serverless storage services can act as static websites (removing the need for web servers) and event services can host code. This removes the operational burden of managing physical servers, and can lower transactional costs because managed services operate at cloud scale. Experiment more often: With virtual and automatable resources, you can quickly carry out comparative testing using different types of instances, storage, or configurations. Consider mechanical sympathy: Understand how cloud services are consumed and always use the technology approach that aligns best with your workload goals. For example, consider data access patterns when you select database or storage approaches. Cost optimization Implement Cloud Financial Management: To achieve financial success and accelerate business value realization in the cloud, you need to invest in Cloud Financial Management /Cost Optimization. Your organization needs to dedicate time and resources to build capability in this new domain of technology and usage management. Similar to your Security or Operational Excellence capability, you need to build capability through knowledge building, programs, resources, and processes to become a cost-efficient organization. Adopt a consumption model: Pay only for the computing resources that you require and increase or decrease usage depending on business requirements, not by using elaborate forecasting. For example, development and test environments are typically only used for eight hours a day during the work week. You can stop these resources when they are not in use for a potential cost savings of 75% (40 hours versus 168 hours). Measure overall efficiency: Measure the business output of the workload and the costs associated with delivering it. Use this measure to know the gains you make from increasing output and reducing costs. Stop spending money on undifferentiated heavy lifting: AWS does the heavy lifting of data center operations like racking, stacking, and powering servers. It also removes the operational burden of managing operating systems and applications with managed services. This allows you to focus on your customers and business projects rather than on IT infrastructure. Analyze and attribute expenditure: The cloud makes it easier to accurately identify the usage and cost of systems, which then allows transparent attribution of IT costs to individual workload owners. This helps measure return on investment (ROI) and gives workload owners an opportunity to optimize their resources and reduce costs. Review Process The review process describes in high-level terms, how the assessment of the principles should be done. For AWS, this should be a lightweight process, which is taking rather hours, instead of days and it should be repeated multiple times across the architecture lifecycle. AWS states that it is important to have a conversation (not an audit) and a β€œblame-free approach” during the review and it is also important to involve the right people. The results of the conversations should be a list of issues that can then be prioritized based on the business context and that can be formulated into a set of actions that help to improve the overall customer experience of the architecture. Well-Architected Tool AWS Console Tool that steps a user through the Well-Architected Review Process Well-Architected Alternate Renditions Consuming the WA PDFs or web content can be a bit challenging with navigation. The WA questions, resources, etc. are available as json via the AWS WA Console Tool with some scraping. This creates alternate renditions based on this data. Run aws-well-architected-reformat/main.js to generate a reformatted and condensed version of WA. cd aws-well-architected-reformat node main.js > ../aws-well-architected-condensed.md see alternate rendition at aws-well-architected-condensed.md See aws-well-architected-reformat/data for the question data as json for each lens. Notes on pulling data via AWS Console # aws console request format POST https://console.aws.amazon.com/wellarchitected/api/apiservice {"method":"GET","path":"/workloads/4cfa14fe4a9d351afc9975cfdcb434af/lensReviews/wellarchitected/answers","region":"us-east-1","headers":{"Content-Type":"application/json","Accept":"application/json"},"params":{"PillarId":"operationalExcellence","MaxResults":50,"Locale":"en"}} # helpful Resources on right sidebar example https://wa.aws.amazon.com/TypeII/en/foundationaltechnicalreview/foundationaltechnicalreview.sec_q1.helpful-resources.en.html Feature Request Custom lenses have been added. See Custom lenses One area where there is a gap for an enterprises are all the company specific policies, standards, and best practices that are additive and need to be addressed on top of AWS. These types of questions and guidance would need to happen outside of WA Tool. A feature to define custom lenses - a customer defined lens. This way the single WA Tool could be the method for review facilitation, improvement reporting and maintaining history. Custom Lens template JSON file for creating custom lens { "schemaVersion": "2021-11-01", "name": "Replace with lens name", "description": "Replace with your description", "pillars": [ { "id": "pillar_id_1", "name": "Pillar 1", "questions": [ { "id": "pillar_1_q1", "title": "My first question", "description": "Description isn't a necessary property here for a question, but it might help your lens users.", "choices": [ { "id": "choice1", "title": "Best practice #1", "helpfulResource": { "displayText": "It's recommended that you include a helpful resource text and URL for each choice for your users.", "url": "https://aws.amazon.com" }, "improvementPlan": { "displayText": "You must have improvement plans per choice. It's optional whether or not to have a corresponding URL." } }, { "id": "choice2", "title": "Best practice #2", "helpfulResource": { "displayText": "It's recommended that you include a helpful resource text and URL for each choice for your users.", "url": "https://aws.amazon.com" }, "improvementPlan": { "displayText": "You must have improvement plans per choice. It's optional whether or not to have a corresponding URL." } } ], "riskRules": [ { "condition": "choice1 && choice2", "risk": "NO_RISK" }, { "condition": "choice1 && !choice2", "risk": "MEDIUM_RISK" }, { "condition": "default", "risk": "HIGH_RISK" } ] } ] } ] } Key Visuals Resources AWS Well-Architected Documentation | AWS Well-Architected Framework The Review Process - AWS Well-Architected Framework AWS Well Architected framework: A Complete Checklist AWS Well-Architected Framework Cheatsheet | Cloud Noon AWS Well-Architected Tool: Custom lenses Announcing AWS Well-Architected Custom Lenses: Extend the Well-Architected Framework with Your Internal Best Practices Customize Well-Architected Reviews using Custom Lenses and the AWS Well-Architected Tool Custom lenses --- ### API Gateway Private Endpoints - URL: https://brianpfeil.com/post/api-gateway-private-endpoints/ - Date: 2021-04-06 - Tags: aws, api-gateway, vpc, network code for article  pfeilbr/api-gateway-private-endpoints-playground learn API Gateway private endpoints Setup Steps configure API Gateway private endpoint that leverages HTTP proxy integration. Test via lambda with VPC config that issues HTTP GET request to vpc endpoint. Notes supported on REST API endpoints (not HTTP API at the time 2021-04-05) be sure to re-deploy API for policy changes check security groups check VPC endpoint policy check lambda VPC config and security groups API Gateway resource policy { "Version": "2012-10-17", "Statement": [ { "Effect": "Deny", "Principal": "*", "Action": "execute-api:Invoke", "Resource": "arn:aws:execute-api:us-east-1:529276214230:scheqe4ymi/*", "Condition": { "StringNotEquals": { "aws:sourceVpce": "vpce-02b487f2d021986fb" } } }, { "Effect": "Allow", "Principal": "*", "Action": "execute-api:Invoke", "Resource": "arn:aws:execute-api:us-east-1:529276214230:scheqe4ymi/*" } ] } note the Deny + Allow combination* Lambda test code const https = require('https'); exports.handler = (event, context, callback) => { // https://scheqe4ymi.execute-api.us-east-1.amazonaws.com/prod var options = { host: 'vpce-02b487f2d021986fb-yl0hav78.execute-api.us-east-1.vpce.amazonaws.com', path: '/prod/index.html', method: 'GET', port: 443, headers: { 'Host':'scheqe4ymi.execute-api.us-east-1.amazonaws.com' } }; const cb = function(response) { let str = ''; response.on('data', function (chunk) { str += chunk; }); response.on('end', function () { console.log(str); const response = { statusCode: 200, body: JSON.stringify(str), }; callback(null, response) return response; }); } https.request(options, cb).end(); } note the Host header that allows for the request to be routed Screenshots Resources Introducing Amazon API Gateway Private Endpoints | Amazon Web Services How do I troubleshoot issues connecting to an API Gateway private API endpoint? --- ### CFN Diagram - URL: https://brianpfeil.com/post/cfn-diagram/ - Date: 2021-04-05 - Tags: cloudformation, diagram, tools code for article  pfeilbr/cfn-diagram-playground learn cfn-diagram CLI tool to visualise CloudFormation/SAM/CDK templates as diagrams. Usage # install npm i -g @mhlabs/cfn-diagram # draw.io render cfn-dia draw.io --template-file CloudFrontS3WebsiteCdkStack.template.yaml \ --output-file CloudFrontS3WebsiteCdkStack.drawio # html render cfn-dia html --template-file CloudFrontS3WebsiteCdkStack.template.yaml \ --output-path CloudFrontS3WebsiteCdkStack-html Screenshots Notes good starting point. generate draw.io to get all resources and icons, then modify from there. allows for filtering by resource types to eliminate resources that may not be needed. works by importing cfn template into CDK. CDK allows to treat as graph for traversal and inspection Resources mhlabs/cfn-diagram How (and Why) You Need To Start Generating Your Serverless Infrastructure Diagrams | Ready, Set, Cloud! --- ### API Gateway Service Integrations for HTTP APIs - URL: https://brianpfeil.com/post/api-gateway-service-integrations-for-http-apis/ - Date: 2021-04-02 - Tags: aws, api-gateway, eventbridge, cloudwatch, integration code for article  pfeilbr/api-gateway-service-integrations-for-http-apis-playground Example API Gateway service integration with EventBridge with SAM API Gateway -> EventBridge -> CloudWatch Log Group Running cd eventbridge # deploy sam deploy --guided # run following to given eventbridge rule permissions to write to log group # can not create resource-based policy for Log Group via cfn # see https://stackoverflow.com/questions/49242010/how-to-define-resource-policy-for-cloudwatch-logs-with-cloudformation aws logs put-resource-policy \ --region us-east-1 \ --policy-name eventbridge-rule-to-logs \ --policy-document '{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "Service": "events.amazonaws.com" }, "Action":[ "logs:CreateLogStream", "logs:PutLogEvents" ], "Resource" : "arn:aws:logs:*:*:log-group:*" } ] }' # send event to api gateway endpoint curl --header "Content-Type: application/json" \ --request POST \ --data '{"detail":"{\"action\": \"withdrawal\", \"amount\": 100}"}' \ https://8djn9e18ej.execute-api.us-east-1.amazonaws.com # response: {"Entries":[{"EventId":"11555ee7-774f-e922-1022-46401bbe5f9b"}],"FailedEntryCount":0} # view event in log group aws logs tail "apigateway-http-api-service-integration/EventBus" # response # 2021-04-02T23:47:46+00:00 ca4b84fb-60f2-302d-b6ce-90338469d249 {"version":"0","id":"11555ee7-774f-e922-1022-46401bbe5f9b","detail-type":"MyDetailType","source":"demo","account":"529276214230","time":"2021-04-02T23:47:46Z","region":"us-east-1","resources":[],"detail":{"action":"withdrawal","amount":100}} # cleanup STACK_NAME="apig-svc-int-eb-cwl-01" aws cloudformation delete-stack --stack-name $STACK_NAME aws cloudformation wait stack-delete-complete --stack-name $STACK_NAME Screenshots Resources Working with AWS service integrations for HTTP APIs - Amazon API Gateway --- ### AWS SES - URL: https://brianpfeil.com/post/aws-ses/ - Date: 2021-03-30 - Tags: aws, ses, email code for article  pfeilbr/aws-ses-playground learn Simple Email Service (Amazon SES) Notes send or receive emails verify domain (DNS txt) and/or email addresses (confirmation email) - verify that you own the email address or domain that you plan to send from understand Service quotas max message size - 10 MB per message (after base64 encoding). sending identity - domain or an email address send emails via SMTP or API (AWS CLI, AWS SDK ) connect to a URL that provides an endpoint for the Amazon SES API or SMTP interface (e.g. email-smtp.us-east-1.amazonaws.com:587) DKIM support - DKIM works by adding a digital signature to the headers of an email message. This signature can then be validated against a public cryptographic key that is located in the organization’s DNS record SPF support - SPF establishes a method for receiving mail servers to verify that incoming email from a domain was sent from a host authorized by that domain’s administrators. TXT record "v=spf1 include:amazonses.com ~all" IAM to control user access to email sending (e.g. ses:SendEmail) Configuration sets - groups of rules that you can apply to the emails you send using Amazon SES. can publish email sending events to CWL, Firehose, SNS Event types - Send, Reject, Delivery, Bounce, Complaint, Click Open Rendering Failure store inbound emails in S3 trigger lambdas based on inbound emails publish your email sending events to CWLs or kinesis firehose Sending personalized email via email templates. templates contain placeholder values. based on Handlebars template system list management customers can manage their own mailing lists, known as contact lists. can create topics, associate topic preferences to a contact and specify OPT_[IN|OUT] for the topic. Global Suppression List includes a global suppression list. When any Amazon SES customer sends an email that results in a hard bounce, Amazon SES adds the email address that produced the bounce to a global suppression list. The global suppression list is global in the sense that it applies to all Amazon SES customers. In other words, if a different customer attempts to send an email to an address that’s on the global suppression list, Amazon SES accepts the message, but doesn’t send it, because the email address is suppressed. enabled by default for all Amazon SES accounts. You can’t disable it. reputation dashboard to track bounce and complaint rates Dedicated IP Addresses IP pool management – If you lease dedicated IP addresses to use with Amazon SES, you can create groups of these addresses, called dedicated IP pools. You can then associate these dedicated IP pools with configuration sets SES sandbox - all new accounts. only send mail to verified email addresses and domains only send mail from verified email addresses and domains send a maximum of 200 messages per 24-hour period send a maximum of 1 message per second need to request production access to move out of sandbox VPC endpoint support - see New – Amazon Simple Email Service (SES) for VPC Endpoints Send Email Using API create destination.json and message.json files # send email # NOTE: cli security principal must have `ses:SendEmail` permission aws ses send-email \ --from 'brian.pfeil@allthecloudbits.com' \ --destination file://destination.json \ --message file://message.json # output/response # { # "MessageId": "010001788372bd22-b3d9950b-8310-43bf-bef3-256917951097-000000" # } Using SMTP interface Create SMTP Credentials Creates IAM user arn:aws:iam::529276214230:user/ses-smtp-user.20210330-092121 with following policy { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": "ses:SendRawEmail", "Resource": "*" } ] } Send Email using SMTP # Testing your connection to the Amazon SES SMTP interface openssl s_client -crlf -quiet -starttls smtp -connect email-smtp.us-east-1.amazonaws.com:587 # base64 encode creds SMTP_USERNAME=$(echo -n "SMTPUsername" | openssl enc -base64) SMTP_PASSWORD=$(echo -n "SMTPPassword" | openssl enc -base64) # create SMTP email # see `smtp-email-example.txt` # send email openssl s_client -crlf -quiet -starttls smtp -connect email-smtp.us-east-1.amazonaws.com:587 < smtp-email-example.txt Resources Documentation | Amazon Simple Email Service --- ### AWS Copilot - URL: https://brianpfeil.com/post/aws-copilot/ - Date: 2021-03-21 - Tags: aws, ecs, copilot code for article  pfeilbr/aws-copilot-playground learn AWS Copilot CLI Workload Types load balanced web service backend service - isolated to private VPC that is accessible to frontend web app (load balanced web service) scheduled job Concepts Application - a group of related services, environments, and pipelines Environments - environment can have its own version of a service running allowing you to create a “test” and “production” environment Service - your code and all of the supporting infrastructure needed to get it up and running on AWS Jobs - ephemeral ECS tasks that are triggered by an event. uses CloudWatch’s rate expressions. Scheduled Jobs are composed of an AmazonECS Task Definition, Task Role, Task Execution Role, a Step Function State Machine for retrying on failures, and finally an Event Rule to trigger the state machine. Pipelines - release pipeline that deploys your service whenever you push to your git repository. (At this time, Copilot supports GitHub, Bitbucket, and CodeCommit repositories.) When a push is detected, your pipeline will build your service, push the image to ECR, and deploy to your environments. Notes copilot cli automatically creates thw following stacks aws-copilot-playground-infrastructure-roles stack StackSet-aws-copilot-playground-infrastructure-777bf4f7-f4a1-469e-bbff-5113f346b47e similar dev workflow to Elastic Beanstalk (eb), Cloud Foundry (cf), heroku. Files and Directories copilot/ - copilot working directory copilot/aws-copilot-playground-nginx/manifest.yml - manifest file generated by copilot Running # copilot does not allow use of root account. # copilot will use `AWS_PROFILE` setting export AWS_PROFILE=admin copilot init copilot deploy copilot app show copilot svc ls # create new service (web, backend, scheduled) copilot svc init copilot svc show copilot svc logs # delete all resources / stack(s) copilot app delete Resources AWS Copilot CLI - AWS Copilot CLI ECSworkshop.com --- ### Amazon Managed Workflows for Apache Airflow - URL: https://brianpfeil.com/post/amazon-managed-workflows-for-apache-airflow/ - Date: 2021-03-13 code for article  pfeilbr/amazon-managed-workflows-for-apache-airflow-playground learn Amazon Managed Workflows for Apache Airflow Install Steps create S3 bucket with versioning enabled create VPC - this is where RDS Postgres for airflow data lives, the airflow web UI runs in a container here create aiflow environment Notes you create named MWAA environments and they each have own configuration airflow ui exposed publically or private within VPC login to airflow ui with IAM user can create web login token via AWS CLI upload DAG .py files to dag path in S3 run DAGs via airflow cli, REST endpoint, boto3 invoke DAG via lambda install additional deps by providing requirements.txt auto scales up and down via Apache Celery Executor by adding / removing worker containers as needed scale via number of workers, and environment class Airflow Plugins Hooks: Hooks are basically python modules that enables tasks to access external platform, like AWS, AZURE, GCP and many more. Sensors: Sensors are python modules which are used to create watcher tasks(in the most basic sense), for example s3Sensor is used to create s3 file watcher task. A sensor stays in running state till a specific state appears. Operators: Operators are typically execution engines. Operators are used to create task that execute some process, based on the type of Operator. For example: PythonOperator can be used create a task that will run a specific python method. XCom share small bits of data between tasks stored in airflow postgres db Example Environment Attributes { "Environment": { "AirflowConfigurationOptions": {}, "AirflowVersion": "1.10.12", "Arn": "arn:aws:airflow:eu-west-1:xxxxxxxxxxxx:environment/airflow-blogpost-dublin", "CreatedAt": 1610632127.0, "DagS3Path": "dags", "EnvironmentClass": "mw1.medium", "ExecutionRoleArn": "arn:aws:iam:: xxxxxxxxxxxx:role/airflow-demo-mwaa-eks-iamrole", "LastUpdate": { "CreatedAt": 1611137820.0, "Status": "SUCCESS" }, "LoggingConfiguration": { "DagProcessingLogs": { "CloudWatchLogGroupArn": "arn:aws:logs:: xxxxxxxxxxxx:log-group:airflow-ricsue-dublin-DAGProcessing", "Enabled": true, "LogLevel": "INFO" }, "SchedulerLogs": { "CloudWatchLogGroupArn": "arn:aws:logs:: xxxxxxxxxxxx:log-group:airflow-ricsue-dublin-Scheduler", "Enabled": true, "LogLevel": "INFO" }, "TaskLogs": { "CloudWatchLogGroupArn": "arn:aws:logs:: xxxxxxxxxxxx:log-group:airflow-ricsue-dublin-Task", "Enabled": true, "LogLevel": "INFO" }, "WebserverLogs": { "CloudWatchLogGroupArn": "arn:aws:logs:: xxxxxxxxxxxx:log-group:airflow-ricsue-dublin-WebServer", "Enabled": true, "LogLevel": "INFO" }, "WorkerLogs": { "CloudWatchLogGroupArn": "arn:aws:logs:: xxxxxxxxxxxx:log-group:airflow-ricsue-dublin-Worker", "Enabled": true, "LogLevel": "INFO" } }, "MaxWorkers": 5, "Name": "ricsue-dublin", "NetworkConfiguration": { "SecurityGroupIds": [ "sg-0c88ef4755c295zzz" ], "SubnetIds": [ "subnet-0493dffd0282f4xxx", "subnet-08f416023356ffyyy" ] }, "RequirementsS3Path": "requirements/requirements.txt", "ServiceRoleArn": "arn:aws:iam:: xxxxxxxxxxxx:role/aws-service-role/airflow.amazonaws.com/AWSServiceRoleForAmazonMWAA", "SourceBucketArn": "arn:aws:s3:::airflow-mybucket", "Status": "AVAILABLE", "Tags": {}, "WebserverAccessMode": "PUBLIC_ONLY", "WebserverUrl": "aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee.c5.eu-west-1.airflow.amazonaws.com", "WeeklyMaintenanceWindowStart": "SUN:14:00" } } Resources Amazon Managed Workflows for Apache Airflow Interacting with Amazon Managed Workflows for Apache Airflow via the command line AWS CLI Command Reference | mwaa Airflow β€” Custom Plugins Airflow XCOM : The Ultimate Guide --- ### AWS Glue Pyspark Fetch Databases and Tables Metadata - URL: https://brianpfeil.com/post/aws-glue-pyspark-fetch-databases-and-tables-metadata/ - Date: 2021-02-19 - Tags: aws, glue, spark, python code for article  pfeilbr/aws-glue-pyspark-fetch-databases-and-tables-metadata-playground example AWS Glue pyspark job script that fetches all the catalog databases and tables metadata. see main.py first method uses spark sql second method uses python boto3 Glue client to interact with Glue API directly Notes ensure --enable-glue-datacatalog is enabled for glue job to allow spark sql to access metadata catalog Glue Console Script View --- ### Azure Durable Functions - URL: https://brianpfeil.com/post/azure-durable-functions/ - Date: 2021-02-05 - Tags: azure code for article  pfeilbr/azure-durable-functions-playground learn Azure Durable Functions Running LOCATION=eastus BASE_NAME=durable-fns-playground-01 RESOURCE_GROUP="${BASE_NAME}" STORAGE_ACCOUNT=$(echo $BASE_NAME | sed -e 's/-//g') FUNCTION_APP=$(echo $BASE_NAME | sed -e 's/-//g') # create resource group az group create \ --name "${RESOURCE_GROUP}" \ --location "${LOCATION}" # create storage account az storage account create \ --name "${STORAGE_ACCOUNT}" \ --location "${LOCATION}" \ --resource-group "${RESOURCE_GROUP}" \ --sku "Standard_LRS" # create function app in azure az functionapp create \ --resource-group "${RESOURCE_GROUP}" \ --consumption-plan-location "${LOCATION}" \ --runtime "node" \ --runtime-version 12 \ --functions-version 3 \ --name "${FUNCTION_APP}" \ --storage-account "${STORAGE_ACCOUNT}" # create functions # DurableFunctionsHTTPstarter is HTTP endpoint that invokes DurableFunctionsOrchestrator # which then calls DurableFunctionsActivity multiple time with different parameters func new -t "Durable Functions HTTP starter" -l javascript -n DurableFunctionsHTTPstarter func new -t "Durable Functions orchestrator" -l javascript -n DurableFunctionsOrchestrator func new -t "Durable Functions activity" -l javascript -n DurableFunctionsActivity # deploy func azure functionapp publish "${FUNCTION_APP}" # get function all base URL FUNCTION_APP_URL="https://$(az functionapp show -g ${RESOURCE_GROUP} -n ${FUNCTION_APP} --query defaultHostName -o tsv)/api" # pull down app settings for running locally func azure functionapp fetch-app-settings "${FUNCTION_APP}" # run locally func start # trigger durable function curl 'http://localhost:7071/api/orchestrators/DurableFunctionsOrchestrator' # async execution - output # # { # "id": "20f2ddc1e34543379efc49a09962dfc5", # "statusQueryGetUri": "http://localhost:7071/runtime/webhooks/durabletask/instances/20f2ddc1e34543379efc49a09962dfc5?taskHub=DurableFunctionsHub&connection=Storage&code=CODE", # "sendEventPostUri": "http://localhost:7071/runtime/webhooks/durabletask/instances/20f2ddc1e34543379efc49a09962dfc5/raiseEvent/{eventName}?taskHub=DurableFunctionsHub&connection=Storage&code=CODE", # "terminatePostUri": "http://localhost:7071/runtime/webhooks/durabletask/instances/20f2ddc1e34543379efc49a09962dfc5/terminate?reason={text}&taskHub=DurableFunctionsHub&connection=Storage&code=CODE", # "rewindPostUri": "http://localhost:7071/runtime/webhooks/durabletask/instances/20f2ddc1e34543379efc49a09962dfc5/rewind?reason={text}&taskHub=DurableFunctionsHub&connection=Storage&code=CODE", # "purgeHistoryDeleteUri": "http://localhost:7071/runtime/webhooks/durabletask/instances/20f2ddc1e34543379efc49a09962dfc5?taskHub=DurableFunctionsHub&connection=Storage&code=CODE" # } # fetch `statusQueryGetUri` curl 'http://localhost:7071/runtime/webhooks/durabletask/instances/20f2ddc1e34543379efc49a09962dfc5?taskHub=DurableFunctionsHub&connection=Storage&code=CODE' # { # "name": "DurableFunctionsOrchestrator", # "instanceId": "20f2ddc1e34543379efc49a09962dfc5", # "runtimeStatus": "Completed", # "input": null, # "customStatus": null, # "output": [ # "Hello Tokyo!", # "Hello Seattle!", # "Hello London!" # ], # "createdTime": "2021-02-04T21:33:29Z", # "lastUpdatedTime": "2021-02-04T21:33:30Z" # } # --- # start in own terminal # Show interactive streaming logs for an Azure-hosted Function App func azure functionapp logstream "${FUNCTION_APP}" # trigger durable function curl "${FUNCTION_APP_URL}/orchestrators/DurableFunctionsOrchestrator?code=CODE" # async execution - output # # { # "id": "7501819801f942ebaed9289e87420cbd", # "statusQueryGetUri": "https://durablefnsplayground01.azurewebsites.net/runtime/webhooks/durabletask/instances/7501819801f942ebaed9289e87420cbd?taskHub=DurableFunctionsHub&connection=Storage&code=CODE", # "sendEventPostUri": "https://durablefnsplayground01.azurewebsites.net/runtime/webhooks/durabletask/instances/7501819801f942ebaed9289e87420cbd/raiseEvent/{eventName}?taskHub=DurableFunctionsHub&connection=Storage&code=CODE", # "terminatePostUri": "https://durablefnsplayground01.azurewebsites.net/runtime/webhooks/durabletask/instances/7501819801f942ebaed9289e87420cbd/terminate?reason={text}&taskHub=DurableFunctionsHub&connection=Storage&code=CODE", # "rewindPostUri": "https://durablefnsplayground01.azurewebsites.net/runtime/webhooks/durabletask/instances/7501819801f942ebaed9289e87420cbd/rewind?reason={text}&taskHub=DurableFunctionsHub&connection=Storage&code=CODE", # "purgeHistoryDeleteUri": "https://durablefnsplayground01.azurewebsites.net/runtime/webhooks/durabletask/instances/7501819801f942ebaed9289e87420cbd?taskHub=DurableFunctionsHub&connection=Storage&code=CODE" # } # fetch `statusQueryGetUri` curl "https://durablefnsplayground01.azurewebsites.net/runtime/webhooks/durabletask/instances/7501819801f942ebaed9289e87420cbd?taskHub=DurableFunctionsHub&connection=Storage&code=CODE" # { # "name": "DurableFunctionsOrchestrator", # "instanceId": "7501819801f942ebaed9289e87420cbd", # "runtimeStatus": "Completed", # "input": null, # "customStatus": null, # "output": [ # "Hello Tokyo!", # "Hello Seattle!", # "Hello London!" # ], # "createdTime": "2021-02-04T20:55:25Z", # "lastUpdatedTime": "2021-02-04T20:55:26Z" # } # query function logs read -r -d '' QUERY << EOM traces | project timestamp, message | where timestamp > ago(24h) | limit 10 EOM az monitor app-insights query \ --app 'cd03d419-0bba-410e-ac3f-d7e934c027f1' \ --analytics-query "${QUERY}" # output # # { # "tables": [ # { # "columns": [ # { # "name": "timestamp", # "type": "datetime" # }, # { # "name": "message", # "type": "string" # } # ], # "name": "PrimaryResult", # "rows": [ # [ # "2021-02-04T20:56:04.3256227Z", # "Host Status: {\r\n \"id\": \"durablefnsplayground01\",\r\n \"state\": \"Running\",\r\n \"version\": \"3.0.15193.0\",\r\n \"versionDetails\": \"3.0.15193 Commit hash: 75da1ebed23b08b39c8c3c20b3ea687813c0acdf\",\r\n \"platformVersion\": \"91.0.7.116\",\r\n \"instanceId\": \"00d386aeb51a3d5098651c58f61ffb53a9b1a656ea9e9b0dd570b9834cdc3d4f\",\r\n \"computerName\": \"DW0-HR0-3-17\",\r\n \"processUptime\": 145740,\r\n \"extensionBundle\": {\r\n \"id\": \"Microsoft.Azure.Functions.ExtensionBundle\",\r\n \"version\": \"1.5.0\"\r\n }\r\n}" # ], # [ # "2021-02-04T20:56:05.4099623Z", # "Host Status: {\r\n \"id\": \"durablefnsplayground01\",\r\n \"state\": \"Running\",\r\n \"version\": \"3.0.15193.0\",\r\n \"versionDetails\": \"3.0.15193 Commit hash: 75da1ebed23b08b39c8c3c20b3ea687813c0acdf\",\r\n \"platformVersion\": \"91.0.7.116\",\r\n \"instanceId\": \"00d386aeb51a3d5098651c58f61ffb53a9b1a656ea9e9b0dd570b9834cdc3d4f\",\r\n \"computerName\": \"DW0-HR0-3-17\",\r\n \"processUptime\": 146826,\r\n \"extensionBundle\": {\r\n \"id\": \"Microsoft.Azure.Functions.ExtensionBundle\",\r\n \"version\": \"1.5.0\"\r\n }\r\n}" # ], # ... Resources Azure Durable Functions documentation --- ### Azure ARM - URL: https://brianpfeil.com/post/azure-arm/ - Date: 2021-01-29 - Tags: azure code for article  pfeilbr/azure-arm-playground RESOURCE_GROUP_NAME="armplayground01" # create resource group az group create --name "${RESOURCE_GROUP_NAME}" --location eastus # deploy arm outputs example az deployment group create \ --name "my-deployment-01" --resource-group "${RESOURCE_GROUP_NAME}" \ --template-file "templates/outputs.azuredeploy.json" \ --parameters @templates/outputs.parameters.json --- ### Azure Functions - URL: https://brianpfeil.com/post/azure-functions/ - Date: 2021-01-25 - Tags: azure, azure-functions, serverless code for article  pfeilbr/azure-functions-playground learn azure functions . Following is based on Create a JavaScript function from the command line - Azure Functions # create nodejs function project func init LocalFunctionProj --javascript cd LocalFunctionProj # create new function trigger via http request func new --name HttpExample --template "HTTP trigger" --authlevel "anonymous" # start locally func start # make request against local endpoint curl http://localhost:7071/api/HttpExample?name=Brian # create resource group az group create --name "AzureFunctionsQuickstart-rg" --location "eastus" # create storage account az storage account create --name "brianpfeilmystorage01" --location "eastus" --resource-group "AzureFunctionsQuickstart-rg" --sku "Standard_LRS" # create function app in azure az functionapp create --resource-group "AzureFunctionsQuickstart-rg" --consumption-plan-location "eastus" --runtime "node" --runtime-version 12 --functions-version 3 --name "brianpfeilmyfn01" --storage-account "brianpfeilmystorage01" # deploy func azure functionapp publish "brianpfeilmyfn01" # invoke function on azure curl "https://brianpfeilmyfn01.azurewebsites.net/api/httpexample?name=Brianv2" # view near real-time streaming logs func azure functionapp logstream "brianpfeilmyfn01" # fetch the app settings. this populates them in `local.settings.json` # credentials for the storage account are stored in the `AzureWebJobsStorage` property # this is needed to store message in storage queue func azure functionapp fetch-app-settings "brianpfeilmyfn01" # Show settings for a function app. (autogenerated) az functionapp config appsettings list --name "brianpfeilmyfn01" --resource-group "AzureFunctionsQuickstart-rg" # Update a function app's settings. (autogenerated) az functionapp config appsettings set --name "brianpfeilmyfn01" --resource-group "AzureFunctionsQuickstart-rg" --settings "AzureWebJobsStorage=$storageConnectionString" # list available function templates for javascript func templates list -l javascript # create new function using "Durable Functions HTTP starter" template func new -t "Durable Functions HTTP starter" -l javascript -n DurableFunctionsHTTPstarter Screenshots message(s) in the azure storage queue Managed Identity for Function in Azure Console Notes app settings can be access via environment variables. See Configure function app settings in Azure Functions Use Key Vault references to store secrets stored in Key Vault. They are automatically fetched and provided as environment variables to your function. see Secure App Settings variables in Azure Functions e.g. App Settings Value KeyVault Reference @Microsoft.KeyVault(SecretUri=https://pfeilkeyvault01.vault.azure.net/secrets/secret01/e0fca4271fb243178a0a861d8e6fbc59) Functions have in/out bindings defined in function.json To access other azure resources/services from a function, you configure a managed identity on the function app and provide access to Azure resources for that identity using Azure role-based access control. See Azure Services that support managed identities - Azure AD . system-assigned managed identity - identity tied to and managed by a specific service. when that service instance is deleted, the identity is deleted with it. user-assigned managed identity - not owned by a specific service. lifecycle is fully managed by you. can be assigned to multiple services. Azure Durable Functions documentation - lets you write stateful functions in a serverless compute environment. Similar to AWS Step Functions, but implemented as language level library. shared access signature (SAS) URLs for granting limited access. Similar to S3 signed URLs. See Grant limited access to data with shared access signatures (SAS) - Azure Storage A shared access signature is a signed URI that points to one or more storage resources. The URI includes a token that contains a special set of query parameters. The token indicates how the resources may be accessed by the client. One of the query parameters, the signature, is constructed from the SAS parameters and signed with the key that was used to create the SAS. This signature is used by Azure Storage to authorize access to the storage resource. Resources Triggers and bindings in Azure Functions Create a JavaScript function from the command line - Azure Functions Azure Functions scale and hosting - covers limits and constraints Using Managed Identity between Azure Functions and Azure Storage - Code Samples How to use managed identities for App Service and Azure Functions Configure function app settings in Azure Functions Secure App Settings variables in Azure Functions Source Application Settings from Key Vault --- ### SAM Lambda Layers - URL: https://brianpfeil.com/post/sam-lambda-layers/ - Date: 2021-01-08 - Tags: sam, lambda, aws code for article  pfeilbr/sam-lambda-layers-playground learn SAM lambda layers Running Example cd lambda-layer-python sam build # run lambda with layer locally # `--force-image-build` to clear out layer cache sam local invoke --force-image-build # start local api endpoint sam local start-api --force-image-build curl http://127.0.0.1:3000/ # deploy sam deploy --guided # run lambda behind api gateway endpoint curl https://hl6u5bubr4.execute-api.us-east-1.amazonaws.com/Prod/ Resources Working with AWS Lambda and Lambda Layers in AWS SAM --- ### Click CLI - URL: https://brianpfeil.com/post/click-cli/ - Date: 2021-01-07 - Tags: cli code for article  pfeilbr/click-cli-playground learn Click a python package for creating beautiful command line interfaces in a composable way with as little code as necessary see main.py Running pipenv install pipenv shell python main.py Resources Click --- ### AWS Data Wrangler - URL: https://brianpfeil.com/post/aws-data-wrangler/ - Date: 2021-01-06 - Tags: aws, data, python code for article  pfeilbr/aws-data-wrangler-playground learn AWS Data Wrangler python initiative that extends the power of Pandas library to AWS connecting DataFrames and AWS data related services (Amazon Redshift, AWS Glue, Amazon Athena, Amazon Timestream, Amazon EMR, Amazon QuickSight, etc). Running Examples # install venv and deps pipenv install # activate venv pipenv shell # run examples python main.py See https://github.com/awslabs/aws-data-wrangler/tree/master/tutorials for example usages. Lambda Layer Can be used via a lambda layer . Resources https://aws-data-wrangler.readthedocs.io/ awslabs/aws-data-wrangler AWS Data Wrangler| API Reference https://github.com/awslabs/aws-data-wrangler/tree/master/tutorials - notebooks with examples --- ### Architecture Notes and Resources - URL: https://brianpfeil.com/post/architecture-notes-and-resources/ - Date: 2020-12-11 - Tags: architecture Definition(s) Key Questions Organization Considerations Quality Attributes Patterns event-sourcing Resources Hexagonal Resources Topics / Concepts / Terms Database Shuffle Sharding Constant Work Canary Resources Books (oreilly.com) Websites Decks Videos let business value guide all architecture decisions evolutionary architectures are needed to support rapid rate of business and technology change Definition(s) β€œArchitecture is about the important stuff. Whatever that is” the decisions you wish you could get right early in a project the shared understanding that the expert developers have of the system design how the components are assembled and organized. This will be done in a way that meets the quality attributes. significant design decisions that shape a system, where significant is measured by cost of change - Grady Booch Key Questions who are the users what devices and form factors will be used what is the context of their usage scale and growth who are the main actors in the system (domain objects - e.g. orders, products, etc.) data classifications (pii) data types and sizes (relation records, documents, media files, etc.) what is the time frame for delivery is there an existing product / SaaS / open-source / etc. that provides the solution or a portion / components of it Capacity estimation & Constraints Functional Requirements Non Functional Requirements - Latency, Consistency, Availability, High Throughput, etc. Out of scope organization and teams structure see https://medium.com/partha-pratim-sanyal/system-design-doordash-a-prepared-food-delivery-service-bf44093388e2 for good reference Organization Considerations engineering (application & platform) operations (application & platform) Quality Attributes reliability - ability to continue to operate under predefined conditions availability - ratio of the available system time to the total working time scalability - ability of the system to handle load increases without decreasing performance efficiency performance security cost interoperability correctness maintainability readability extensibility testability Patterns event-sourcing Capture all changes to an application state as a sequence of events. Core Design Decisions Domain Entities and Events popular method is via Event Storming Event Content each event stores delta state each event stores full state idempotent is easy to solve for duplicate events Total Ordering (ordered stream of events - ledger) ensure all event are processed in order. this is needed for causal relationships. e.g. ordering matters for two messages related to the same entity Resources Scaling Event Sourcing for Netflix Downloads, Episode 1 Scaling Event Sourcing for Netflix Downloads, Episode 2 InfoQ | Scaling Event Sourcing for Netflix Downloads | Video + Presentation - shows in detail how they implemented event sourcing backed by cassandra matrinfowler.com | Event Sourcing Pattern: Event sourcing EventBridge Storming β€” How to build state-of-the-art Event-Driven Serverless Architectures - approach to defining the Events, Boundaries and Entities in your business domain Decomposing the Monolith with Event Storming Hexagonal Allow an application to equally be driven by users, programs, automated test or batch scripts, and to be developed and tested in isolation from its eventual run-time devices and databases. Resources Hexagonal Architecture: three principles and an implementation example Hexagonal architecture Topics / Concepts / Terms Database CAP theorem Consistency: Every read receives the most recent write or an error Availability: Every request receives a (non-error) response, without the guarantee that it contains the most recent write Partition tolerance: The system continues to operate despite an arbitrary number of messages being dropped (or delayed) by the network between nodes Serializability Snapshot isolation Multiversion concurrency control Things I Wished More Developers Knew About Databases Shuffle Sharding limits / isolates tenants in a multi-tenant system so they don’t negatively impact other tenants. method of assigning tenant to resources. Resources Workload isolation using shuffle-sharding AWS Well-Architected Labs | Fault isolation with shuffle sharding Constant Work overprovision resources to the point where it would operate correctly even if an availability zone were to be unavailable If AZ becomes unavailable, no new resources need to be provisioned, just a quick re-routing. you are essentially always operating the infrastructure for failure mode (active-active) Resources Static stability using Availability Zones Reliability, constant work, and a good cup of coffee Canary A canary release is a technique to reduce the risk from deploying a new version of software into production. A new version of software, referred to as the canary, is deployed to a small subset of users alongside the stable running version. Traffic is split between these two versions such that a portion of incoming requests are diverted to the canary. This approach can quickly uncover any problems with the new version without impacting the majority of users. Resources Automated Canary Analysis at Netflix with Kayenta Resources Books (oreilly.com) The Software Architect Elevator Fundamentals of Software Architecture Clean Architecture: A Craftsman’s Guide to Software Structure and Design, First Edition Software Architecture Patterns Building Evolutionary Architectures Clean Architecture: A Craftsman’s Guide to Software Structure and Design, First Edition Domain-Driven Design: Tackling Complexity in the Heart of Software Microservices Patterns Patterns of Enterprise Application Architecture Refactoring: Improving the Design of Existing Code Design Patterns: Elements of Reusable Object-Oriented Software Designing Distributed Systems Designing Distributed Control Systems: A Pattern Language Approach (Wiley Software Patterns Series) Distributed Tracing in Practice Making Sense of Stream Processing I Heart Logs Streaming Systems Organization Architecture Accelerate: Building and Scaling High Performing Technology Organizations The Phoenix Project Websites The System Design Primer - great real life architecture and design examples martinfowler.com martinfowler.com | Software Architecture Guide AWS Architecture Center AWS Architecture Blog Amazon Builders’ Library Azure Architecture Center medium | software architecture C4 model for visualizing software architecture Decks Scalability, Availability & Stability Patterns Videos The elephant in the architecture - Martin Fowler --- ### Python - URL: https://brianpfeil.com/post/python/ - Date: 2020-11-20 - Tags: python code for article  pfeilbr/python-playground learn and experiment with python, it’s libraries, and packages add test_*.py file to tests/unit directory with code to test, learn, experiment with language, library, package, etc. # clone and cd into this repo # activate venv pipenv shell # install deps (e.g. nodemon) npm install # run tests npm run test # run tests and watch for changes to re-run npm run test:watch --- ### Serverless Flask - URL: https://brianpfeil.com/post/serverless-flask/ - Date: 2020-11-13 - Tags: serverless, flask, aws, python, lambda, api-gateway code for article  pfeilbr/serverless-flask-playground Example of running python Flask app on Lambda + API Gateway via serverless framework . Based on Build a Python REST API with Serverless, Lambda, and DynamoDB . Architecture Prerequisites serverless framework Docker Desktop Python 3.8 pipenv Running # ensure python venv loaded in shell pipenv shell # install npm deps for serverless npm install # run flash app locally. server reloads on file change. still need to refresh page in browser npm run dev # deploy. (docker must be running to build python packages for linux target / lambda) npm run deploy Resources The Official Guide to Serverless Flask Build a Python REST API with Serverless, Lambda, and DynamoDB --- ### DynamoDB Export to S3 and Query with Athena - URL: https://brianpfeil.com/post/dynamodb-export-to-s3-and-query-with-athena/ - Date: 2020-11-11 - Tags: dynamodb, s3, aws, athena code for article  pfeilbr/dynamodb-export-to-s3-and-query-with-athena-playground example exporting dynamodb table to S3 and then querying via athena Files template.yaml main.sh example-export/ - example contents of export (copied from S3) Running sam deploy --guided # note: seed data is generated as part of deploy via cfn custom resource `Custom::SeedData` # which triggers a lambda which populates the dynamodb table # update `STACK_NAME` variable in ./main.sh # run export table to s3 script ./main.sh aws dynamodb list-exports # in progress { "ExportSummaries": [ { "ExportArn": "arn:aws:dynamodb:us-east-1:529276214230:table/dynamodb-export-to-s3-and-query-with-athena-playground-v2-MyTable-1WGSJ3W2WJWPK/export/01605130432834-8a918b87", "ExportStatus": "IN_PROGRESS" } ] } aws dynamodb list-exports # completed { "ExportSummaries": [ { "ExportArn": "arn:aws:dynamodb:us-east-1:529276214230:table/dynamodb-export-to-s3-and-query-with-athena-playground-v2-MyTable-1WGSJ3W2WJWPK/export/01605130432834-8a918b87", "ExportStatus": "COMPLETED" } ] } Example export file contents (line delimited json items) gzcat example-export/demo_prefix/AWSDynamoDB/01605130432834-8a918b87/data/vajn3deidy4svdja3fgej2ynay.json.gz {"Item":{"pk":{"S":"pk001-1605121017583"},"sk":{"S":"sk001"}}} {"Item":{"pk":{"S":"pk002-1605121017583"},"sk":{"S":"sk002"}}} Access Exported Data in S3 via Athena -- Create external table in athena pointing to exported S3 data CREATE EXTERNAL TABLE IF NOT EXISTS ddb_exported_table ( Item struct <pk:struct<S:string>, sk:struct<S:string>> ) ROW FORMAT SERDE 'org.openx.data.jsonserde.JsonSerDe' LOCATION 's3://dynamodb-export-to-s3-and-query-with-exportbucket-103kukqwmab7n/demo_prefix/AWSDynamoDB/01605130432834-8a918b87/data/' TBLPROPERTIES ( 'has_encrypted_data'='true'); -- issues SELECT query SELECT Item.pk.S as pk, Item.sk.S as sk FROM ddb_exported_table Screenshot from Athena Console Resources New – Export Amazon DynamoDB Table Data to Your Data Lake in Amazon S3, No Code Writing Required Now you can export your Amazon DynamoDB table data to your data lake in Amazon S3 to perform analytics at any scale Exporting DynamoDB table data to Amazon S3 AWS CloudFormation custom resource creation with Python, AWS Lambda, and crhelper --- ### AWS Database Migration Service - URL: https://brianpfeil.com/post/aws-database-migration-service/ - Date: 2020-11-01 - Tags: aws code for article  pfeilbr/aws-database-migration-service-playground Concepts AWS DMS connects to the source data store, reads the source data, and formats the data for consumption by the target data store. It then loads the data into the target data store At a high level, when using AWS DMS you do the following: Create a replication server. Create source and target endpoints that have connection information about your data stores. Create one or more migration tasks to migrate data between the source and target data stores. A task can consist of three major phases: The full load of existing data The application of cached changes (change data capture, CDC) Ongoing replication Pricing - Pay for ec2 replication instances and log storage Resources AWS Database Migration Service AWS Schema Conversion Tool --- ### AWS CloudWatch Logs Insights - URL: https://brianpfeil.com/post/aws-cloudwatch-logs-insights/ - Date: 2020-10-30 - Tags: aws, cloudwatch code for article  pfeilbr/aws-cloudwatch-logs-insights-playground learn CloudWatch Logs Insights Running Example Query via AWS CLI see main.sh # run script containing query ./main.sh # OR # re-run on change make dev Example Queries fields @timestamp, detail.eventSource, detail.eventName, @message | sort @timestamp desc | limit 100 fields @timestamp, detail.eventSource, detail.eventName, @message | filter detail.eventSource = "logs.amazonaws.com" | sort @timestamp desc | limit 100 fields @timestamp, detail.requestParameters.bucketName, detail.eventSource, detail.eventName, @message | filter detail.eventSource like /s3.amazonaws.com/ | sort @timestamp desc | limit 100 # sts assume role CloudTrail events fields @timestamp, source, `detail.eventName`, detail.requestParameters.roleArn, detail.userIdentity.userName, @message | filter detail.eventSource = 'sts.amazonaws.com' | sort @timestamp desc # CodePipeline pipeline and stage change events fields @timestamp, `detail-type`, detail.pipeline, detail.stage, detail.state, @message | filter source = 'aws.codepipeline' | sort @timestamp desc Resources CloudWatch Logs Insights Query Syntax Sample Queries - Amazon CloudWatch Logs --- ### Dynamic Nodejs Code Execution - URL: https://brianpfeil.com/post/dynamic-nodejs-code-execution/ - Date: 2020-10-28 - Tags: nodejs code for article  pfeilbr/dynamic-nodejs-code-execution-playground example(s) of dynamically executing nodejs code. see index.js Running node index.js --- ### AWS Kendra - URL: https://brianpfeil.com/post/aws-kendra/ - Date: 2020-10-27 - Tags: aws, kendra, search, machine-learning code for article  pfeilbr/aws-kendra-playground learn Amazon Kendra , intelligent search service powered by machine learning Concepts Index - two types. document (unstructured) and FAQ (structured) You can add documents directly to an index using the BatchPutDocument operation You can add questions and answers (FAQs) directly to your index using the console or the CreateFaq operation Data Source - s3, salesforce documents, sharepoint, custom Documents - html, ppt, doc, pdf, txt custom fields - can define custom fields of types: Date, Number, String, String List Queries - natural language (NLP), keyword queries, Factoid questions β€” Simple who, what, when, or where questions. Facets are scoped views of a set of search results By default, Query returns all search results. To filter responses, you can perform logical operations on the document attributes. Tags - can assign tags to indexes, data sources, and FAQs default encryption at rest with customer of AWS KMS keys General Steps create index add data sources for index synchronize the data source query/search the index Editions Developer Enterprise You specify Edition: DEVELOPER_EDITION | ENTERPRISE_EDITION when you create an index Creating Example Index see main.py sam deploy --guided # copy outputs into `.env` python main.py Resources Amazon Kendra Docs Getting started (AWS SDK for Python (Boto3)) - Amazon Kendra - shows create index, create data source for index, sync data source, all while waiting for each step to complete in between Deploying Amazon Kendra - pre-built react frontend for kendra. provides <search /> react component https://kendrasamples.s3.amazonaws.com/kendrasamples.zip aws-samples/enterprise-search-with-amazon-kendra-workshop Kendra | Using a custom data source --- ### AWS JavaScript SDK V3 - URL: https://brianpfeil.com/post/aws-javascript-sdk-v3/ - Date: 2020-10-26 - Tags: aws, sdk, javascript code for article  pfeilbr/aws-javascript-sdk-v3-playground learn AWS JavaScript SDK v3 index.js index.test.js New Features v3 Commands commands for each AWS Service package to enable you to perform operations for that AWS Service. After you install an AWS Service, you can browse the available commands in your project’s node-modules/@aws-sdk/client-PACKAGE_NAME/commands folder middleware stack can use a new middleware stack to control the lifecycle of an operation call Each middleware stage in the stack calls the next middleware stage after making any changes to the request object. This also makes debugging issues in the stack much easier, because you can see exactly which middleware stages were called leading up to the error. Use Cases debugging logging request updates instrumentation Example middleware args DynamoDB ListTablesCommand { middlewareStack: { add: [Function: add], addRelativeTo: [Function: addRelativeTo], clone: [Function: clone], use: [Function: use], remove: [Function: remove], removeByTag: [Function: removeByTag], concat: [Function: concat], applyToStack: [Function: cloneTo], resolve: [Function: resolve] }, input: {}, request: HttpRequest { method: 'POST', hostname: 'dynamodb.us-east-1.amazonaws.com', port: undefined, query: {}, headers: { 'Content-Type': 'application/x-amz-json-1.0', 'X-Amz-Target': 'DynamoDB_20120810.ListTables', 'user-agent': 'aws-sdk-nodejs-v3-@aws-sdk/client-dynamodb/1.0.0-rc.2 darwin/v12.16.1', 'content-length': '2' }, body: '{}', protocol: 'https:', path: '/' } } Running # run tests that exercise new features npm run test -- --watch Resources aws/aws-sdk-js-v3 AWS SDK for JavaScript | Developer Guide for SDK Version 3 Modular AWS SDK for JavaScript – Release Candidate --- ### AWS Transfer Family - URL: https://brianpfeil.com/post/aws-transfer-family/ - Date: 2020-10-23 - Tags: aws, aws-transfer-family, sftp, ftps, ftp code for article  pfeilbr/aws-transfer-family-playground learn AWS Transfer Family , file transfer to Amazon S3 using SFTP, FTPS, and FTP Resources aws-samples/transfer-for-sftp-logical-directories - complete cfn provisioning example. sftp server, custom IdP via API Gateway, generates sample data, logical directory to s3 path mappings, etc. AWS Transfer Family User Guide AWS Transfer Family resource type reference - AWS CloudFormation Working with custom identity providers github advanced search ‘“AWS::Transfer::Server” language:YAML’ - find cfn examples in github Jerry Hargrove - Cloud Diagrams & Notes | AWS Transfer Family Jerry Hargrove - Cloud Diagrams & Notes | AWS Transfer for SFTP --- ### AWS AppConfig - URL: https://brianpfeil.com/post/aws-appconfig/ - Date: 2020-10-20 - Tags: aws, appconfig, configuration, deployment, lambda code for article  pfeilbr/aws-appconfig-playground learn AWS AppConfig create, manage, and quickly deploy application configurations. AWS AppConfig supports controlled deployments to applications of any size and includes built-in validation checks and monitoring. You can use AWS AppConfig with applications hosted on EC2 instances, AWS Lambda, containers, mobile applications, or IoT devices. Concepts Domain Application (myapp) Environment (dev, test, demo, prod) Configuration Profile - versioned configuration data. text, json, yaml, S3 object, SSM Document, SSM Parameter, CodePipeline Deployment - types: AllAtOnce, Linear50PercentEvery30Seconds, Canary10Percent20Minutes Functional deploy configuration changes from a central location rules to validate your configuration. configurations that aren’t valid can’t be deployed. validation types: JSON schema, Lambda. see About validators Example(s) get configuration via cli aws appconfig get-configuration \ --application 'app01' \ --environment 'dev' \ --configuration 'config-profile-01' \ --client-id 'test' \ appconfig.json # view results in specified output file `appconfig.json` cat appconfig.json output AWS Console Screenshots Configuration profile source types Example JSON configuration profile Deployment Strategy Types Start Deployment Deployment Resources AWS AppConfig Documentation Deploying application configuration to serverless: introducing the AWS AppConfig Lambda extension - blog post on how to use AppConfig Lambda extension AWS AppConfig integration with Lambda extensions - sessions-with-aws-sam/appconfig-lambda-extensions/README.md - good complete reference implementation on how to effectively use with lambda sthulb/appconfig-demo --- ### Bash Shell Script - URL: https://brianpfeil.com/post/bash-shell-script/ - Date: 2020-10-13 - Tags: bash, shell code for article  pfeilbr/bash-shell-script-playground learn and experiment with bash shell scripts Development Workflow main.sh # re-run on changes to main.sh. run in own shell / blocks make dev # edit main.sh --- ### PowerShell - URL: https://brianpfeil.com/post/powershell/ - Date: 2020-10-12 - Tags: powershell code for article  pfeilbr/powershell-playground # connects to Azure with an authenticated account for use with cmdlets from the # Az PowerShell modules Connect-AzAccount Get-AzADUser -ObjectId "user@example.com"| fl Resources Installing PowerShell on macOS Azure PowerShell documentation Yes you can! Use PowerShell in MacOS to connect to Microsoft Azure Microsoft Graph PowerShell Preview - Now on PowerShell Gallery Scratch --- ### AWS Cloud9 - URL: https://brianpfeil.com/post/aws-cloud9/ - Date: 2020-10-05 - Tags: aws, cloud9 code for article  pfeilbr/aws-cloud9-playground learn AWS Cloud9 setup remote development environment Enabling SSH Connectivity add SSH inbound rule to Security Group add ~/.ssh/id_rsa.pub to /home/ec2-user/.ssh/authorized_keys. do this via Cloud9 IDE shell ssh -i ~/.ssh/id_rsa ubuntu@dev01.brianpfeil.com Troubleshooting may need to connect via Cloud9 IDE to “wake up”/start instance due to “auto stop” of EC2 instance after X minutes. May need to remote remote host in ~/.ssh/known_hosts on client if the following error @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ @ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @ @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY! Someone could be eavesdropping on you right now (man-in-the-middle attack)! It is also possible that a host key has just been changed. The fingerprint for the ECDSA key sent by the remote host is SHA256:<deleted> Please contact your system administrator. Add correct host key in ~/.ssh/known_hosts to get rid of this message. Offending ECDSA key in ~/.ssh/known_hosts:37 ECDSA host key for dev01.brianpfeil.com has changed and you have requested strict checking. Host key verification failed. VS Code Remote Development on Cloud9 perform “Enabling SSH Connectivity” steps see vscode | Remote development over SSH vscode remotely SSH connected screenshot Resources VS Code Remote Development vscode | Remote development over SSH --- ### Boto3 - URL: https://brianpfeil.com/post/boto3/ - Date: 2020-10-05 - Tags: boto, python code for article  pfeilbr/boto3-playground learn boto3 , the Amazon Web Services (AWS) SDK for Python Concepts Clients provide a low-level interface to AWS whose methods map close to 1:1 with service APIs. All service operations are supported by clients. Clients are generated from a JSON service definition file. Resources represent an object-oriented interface to Amazon Web Services (AWS). They provide a higher-level abstraction than the raw, low-level calls made by service clients. To use resources, you invoke the resource() method of a Session and pass in a service name: Session manages state about a particular configuration. By default, a session is created for you when needed. However, it’s possible and recommended that in some scenarios you maintain your own session. Sessions typically store the following: Credentials AWS Region Other configurations related to your profile Collection provides an iterable interface to a group of resources. A collection seamlessly handles pagination for you, making it possible to easily iterate over all items from all pages of data. Paginators Some AWS operations return results that are incomplete and require subsequent requests in order to attain the entire result set. The process of sending subsequent requests to continue where a previous request left off is called pagination. Paginators are a feature of boto3 that act as an abstraction over the process of iterating over an entire result set of a truncated API operation. Developing # install virtual env and dependencies pipenv install # (optional) install additional pip package pipenv install <package> # activate python virtual env (optional) pipenv shell # run on change make dev # --- running via jupyter notebook --- # in vscode # click on `main.ipynb`. this will automatically start jupyter notebook and connect # ctrl+enter to run cell # ctrl+space for intellisense # see [How to use Pipenv with Jupyter and VSCode](https://towardsdatascience.com/how-to-use-pipenv-with-jupyter-and-vscode-ae0e970df486) # manually run jupyter notebook pipenv run jupyter notebook # access via browser manually (auto opens via above command) open http://localhost:8888/tree # convert notebook to python. generates `main_notebook.py` jupyter nbconvert --to script main.ipynb --output main_notebook Notes for vscode, install Pylance - Visual Studio Marketplace extension Resources Boto3 documentation boto/boto3 boto3-stubs - for code completion Working with Jupyter Notebooks in Visual Studio Code How to use Pipenv with Jupyter and VSCode Big Upgrades are coming to VSCode Jupyter Notebooks --- ### AWS Systems Manager Automation - URL: https://brianpfeil.com/post/aws-systems-manager-automation/ - Date: 2020-10-02 - Tags: aws code for article  pfeilbr/aws-systems-manager-automation-playground learn AWS Systems Manager Automation SSM Automation document contains one or more steps that run in sequential order can specify parameters Each step is built around a single action Output from one step can be used as input in a later step json or yaml documents can run python or powershell scripts via aws:executeScript (max execution time is 10 min) invoke aws:invokeLambdaFunction can specify execution role via AutomationAssumeRole parameter. if not specified, uses the security context of calling principal can trigger based on EventBridge rule can reference parameters in Parameter Store within an SSM doc via {{ssm:parameter-name}} ssm document types (yaml or json) automation (renamed to runbooks) - command - remotely and securely manage the configuration of your managed instances (ec2 or on-prem) aws:executeScript stdout/stderr can be sent to CloudWatch logs. see https://docs.aws.amazon.com/systems-manager/latest/userguide/automation-action-logging.html Examples # create an ssm document aws ssm create-document \ --content file://path/to/file/documentContent.json \ --name "document-name" \ --document-type "Command" \ --tags "Key=tag-key,Value=tag-value" # run an ssm document (max execution time is 10 min) aws ssm start-automation-execution \ --document-name "AWS-UpdateLinuxAmi" \ --parameters "AutomationAssumeRole=arn:aws:iam::123456789012:role/SSMAutomationRole,SourceAmiId=ami-EXAMPLE,IamInstanceProfileName=EC2InstanceRole" # run `command` type document (runs on EC2 instances) aws ssm send-command \ --instance-ids "instance-ID" \ --document-name "AWS-RunShellScript" \ --comment "IP config" \ --parameters commands=ifconfig \ --output text Resources AWS Systems Manager Automation AWS Systems Manager documents - describes the various document types (e.g. command, automation, etc.) Custom Automation document samples AWS Management and Governance Tools Workshop | AWS SYSTEMS MANAGER awslabs/aws-systems-manager - examples Use the power of script steps in your Systems Manager Automation runbooks | Amazon Web Services Automation actions reference - actions that you can specify in an AWS Systems Manager Automation document aws:executeScript – Run a script - runtimes: python3.6 | python3.7 | PowerShell Core 6.0, as of 2020-10-02 max duration of 10 min (600 secs) aws:executeAwsApi – Call and run AWS API actions Walkthrough: Using Automation with Jenkins ceshihao/ssm-public-documents - example ssm automation documents How To Create AWS SSM Automation Workflow | CloudAffaire - goo walkthrough how to create automation doc and run via aws cli --- ### GitHub CLI - URL: https://brianpfeil.com/post/github-cli/ - Date: 2020-10-02 - Tags: cli, git, github code for article  pfeilbr/github-cli-playground learn GitHub CLI Resources GitHub CLI GitHub CLI | Manual --- ### Diagrams As Code - URL: https://brianpfeil.com/post/diagrams-as-code/ - Date: 2020-10-01 - Tags: diagrams, visualize, architecture code for article  pfeilbr/diagrams-as-code-playground learn Diagrams (Diagram as Code), lets you draw the cloud system architecture in Python code Diagram Authoring Workflow pipenv install # install deps pipenv shell # correct python env make dev # re-runs on file changes / live reload # split vscode window. put .py on left and *.png in right # edit and save .py Screenshots --- ### S3 Access Points - URL: https://brianpfeil.com/post/s3-access-points/ - Date: 2020-10-01 - Tags: s3, aws code for article  pfeilbr/s3-access-points-playground learn S3 Access Points Be sure to review Access points restrictions and limitations before considering for a use case. Feature Overview Access points are unique hostnames enforce distinct permissions and network controls for any request made through the access point. Customers with shared data sets scale access for hundreds of applications by creating individualized access points with names and permissions customized for each application. access point can be restricted to a VPC to firewall S3 data access within customers’ private networks AWS Service Control Policies can be used to ensure all access points are VPC restricted. no longer have to manage a single, complex bucket policy with hundreds of different permission rules that need to be written, read, tracked, and audited. With S3 Access Points, you can now create application-specific access points permitting access to shared data sets with policies tailored to the specific application. Example Policy limit access to partner-01 role { "Version": "2012-10-17", "Statement": [ { "Sid": "AllObjectActions", "Effect": "Allow", "Principal": { "AWS": "arn:aws:iam::529276214230:role/partner-01" }, "Action": "s3:*Object", "Resource": ["arn:aws:s3:us-east-1:529276214230:accesspoint/partner-01-read-write/object/*"] } ] } Access Point Hostname https://partner-01-read-write-529276214230.s3-accesspoint.us-east-1.amazonaws.com Example Usage # get `a.c` using `partner-01` role aws --profile partner-01 s3api get-object --key a.c --bucket 'arn:aws:s3:us-east-1:529276214230:accesspoint/partner-01-read-write' - Resources S3 | Access Points Easily Manage Shared Data Sets with Amazon S3 Access Points Access points restrictions and limitations Amazon S3 Access Points - Tutorials Dojo Screenshots get via access point example access point details access point policy partner-01 role (requires S3 read permissions) Scratch cd ~/tmp aws s3 cp a.c s3://s3-access-points-playground/ # verify assume role works aws --profile partner-01 sts get-caller-identity aws --profile partner-01 s3api get-object --key a.c --bucket 'arn:aws:s3:us-east-1:529276214230:accesspoint/partner-01-read-write' - aws --profile partner-01 sts get-caller-identity --- ### MDX Deck - URL: https://brianpfeil.com/post/mdx-deck/ - Date: 2020-09-28 - Tags: react, markdown, presentation code for article  pfeilbr/mdx-deck-playground learn jxnblk/mdx-deck examples/example01/deck.mdx examples/example01/package.json:scripts build/serve static site, export to PDF Export to PDF # run in own shell. blocks npm start # run in different shell npm run pdf Resources mdx-deck | Theming mdx-deck | Exporting --- ### AWS Serverless Feedback - URL: https://brianpfeil.com/post/aws-serverless-feedback/ - Date: 2020-09-11 - Tags: aws, serverless Feedback on serverless usage on AWS. Based on working with and observations of the teams that leverage the set of AWS “serverless” services. Feedback format is Challenge and Suggestion for a given topic, concept, service, or feature. Table of Contents Serverless at AWS: General Challenge: Skill Set / Developer Accessibility Suggestion: Skill Set / Developer Accessibility Challenge: Developer Experience (DX) Suggestion: Developer Experience (DX) Challenge: Cost Forecasting Suggestion: Cost Forecasting Challenge: Non-AWS Core Enterprise Tools Suggestion: Non-AWS Core Enterprise Tools IAM Challenge: Skill Set / Developer Accessibility Suggestion: Skill Set / Developer Accessibility CloudFormation Challenge: Deployment Time Suggestion: Deployment Time Lambda Challenge: Lambda or Container Decision Suggestion: FaaS or Container Decision Challenge: Cost vs. Performance Tuning Suggestion: Cost vs. Performance Tuning DynamoDB Challenge: Learning Curve Suggestion: Learning Curve Amplify CLI Challenge: Enterprise Support Suggestion: Enterprise Support Challenge: Uncomfortable Level of Magic Suggestion: Uncomfortable Level of Magic AppSync Challenge: GraphQL Centered Development Learning Curve Suggestion: GraphQL Centered Development Learning Curve API Gateway Challenge: Limited Service Support for HTTP API -> AWS Service Integrations Suggestion: Limited Service Support for HTTP API -> AWS Service Integrations Challenge: VTL Mapping Templates Suggestion: VTL Mapping Templates EventBridge Challenge: Security Model Suggestion: Security Model Challenge: Observability Suggestion: Observability Challenge: Real-time and High Volume Event Rates Suggestion: Real-time and High Volume Event Rates X-Ray Challenge: Effort to Enable Suggestion: Effort to Enable AWS Serverless Application Model (SAM) Challenge: Infrastructure Resource to Code Locality Suggestion: Infrastructure Resource to Code Locality Serverless at AWS: General Challenge: Skill Set / Developer Accessibility Serverless skill set has less to do with coding and more knowing the capabilities of each service and how they can be integrated and composed together. The role of the architect becomes central to building effective serverless solutions. This is increasingly so, because the integrations between services are trending more towards configuration over code. “Traditional” application developers see the next step as containers because it’s familiar / closer to what they’re used to (long-lived processes running on an OS vs. a transient run-time environment). It’s hard to make the leap directly to serverless and bypass the containers and k8s space. Suggestion: Skill Set / Developer Accessibility This is where education / resources / etc. are the main tools. Continue and accelerate the creation of content on the AWS blogs, twitch channel, youtube channels, podcasts, Developer Advocates on social media, etc. Amplify suite of tools and services is helping in this area for easing the on-boarding those new to serverless development on AWS. Challenge: Developer Experience (DX) The path to “serverless” adoption has shifted from the foundational technologies being in place to improving developer experience. Everyone is sold on serverless being the right thing, now it’s enabling it’s usage and making accessible to the huge pool of developers who only have experience with “traditional” application development. Suggestion: Developer Experience (DX) SAM and Amplify tooling are quickly addressing the gaps. Accelerate what’s already being done. Challenge: Cost Forecasting Currently difficult to estimate cost with serverless solutions because usually composed of many services. It’s a good deal of effort to understand all the cost contributing dimensions across all the used services. Then assemble. Suggestion: Cost Forecasting An “Cost Profiler” feature by exercising a solution while “recording” it, then extrapolating to give a cost per “user” transaction / time (for sustained transactions) / etc.. “Cost Profiler” similar to a performance profiler. This has similarities and maybe could leverage X-Ray distributed traces + the already captured default CloudWatch metrics per service. Challenge: Non-AWS Core Enterprise Tools Building an effective CI/CD pipeline at a large enterprise involves leveraging non-AWS already established tooling. For example, Bitbucket, Jenkins, and Artifactory. This leads to “lowest common denominator” pipelines that don’t meet the vision of CI/CD. They are dependant on the availability of adapters/integrations/etc. for specific clouds that the cloud agnostic tools provide. AWS has a corresponding solution for each of these that eases building a CI/CD pipeline when the target is AWS. Integrated security via IAM threaded through all the service interactions is the biggest win as security is usually the sticking point. We lose “potential” value as a result of using these non-AWS equivalents over the flexibility it provides the company to support it’s current multi-cloud (aws,azure,gcp, etc.) position. Suggestion: Non-AWS Core Enterprise Tools This is driven by the “non all-in” stance of many large enterprises for multi-cloud. More education in this area about explicitly acknowledging a single cloud approach for given enterprise areas, and understanding and documenting the cost to switch to a different cloud is enough to mitigate the “perceived” risk. IAM Challenge: Skill Set / Developer Accessibility Security is inherently hard and involves many complex entities and their relationships. To be effective and successful on AWS, a customer needs to know this. This is table stakes. IAM continues to be the largest barrier to entry I see with AWS users. It is necessary and robust, but a new concept to “traditional” developers. The closest thing to it that many devs have experience with is RDBMS permissions. Suggestion: Skill Set / Developer Accessibility AWS is leveraging Amplify suite as an on-boarding mechanism for those new to AWS. It currently “handles” all the IAM concerns via automation / codegen / etc. Maybe helpful to add features that allow developers to take more ownership in this area. Similar to what SAM CLI does with AWS SAM Policy Templates CloudFormation Challenge: Deployment Time A challenge I’ve observed with serverless development, is slowed down iterative workflow due to the length of time it takes for a CloudFormation deployment to run. Developers are used to live reload level of speed to see dev changes reflected back to them. Suggestion: Deployment Time Faster deployment times :) Lambda Challenge: Lambda or Container Decision Lambda has grown to have a lot of features and configuration options. Lambda feature set is converging towards Fargate and vice versa. There is already a decision to be made on which to choose. This convergence could make it even more unclear which to pick. Or if one goes away and they are merged, the migration path. Suggestion: Lambda or Container Decision More tools similar to Well-Architected to help decide. Converge Lambda and Container functionality into single service Challenge: Cost vs. Performance Tuning The external community has solutions such as lambda power tuning to automate finding the right compute/memory configuration for a given applications priority balance between performance and cost. Suggestion: Cost vs. Performance Tuning AWS has all the information and metrics about the container environment through the duration of the run. Would be nice to have this as a service. Maybe “Lambda tuning suggestions” that is purely informational and someone has to take action to apply configuration changes. This helps customers, but AWS also with it’s own efficiency goals via underlying resource scheduling (e.g. scheduling, workload placement, container packing, etc.) DynamoDB Challenge: Learning Curve the default serverless database. NoSQL does not have standards like the RDBMS and extended SQL family. You are learning an AWS specific DB and how to interact with it via an API. * There is no smoothing / accessibility layer like SQL. This is a DB where to use effectively, you need to learn it. I’ve observed devs try to lean on language/library/framework ORMs, but that only gets them so far. In order to dev, debug, deploy, operationalize, and troubleshoot it, you need to learn it. Suggestion: Learning Curve Continue with tools like NoSQL Workbench for DynamoDB GUI Client and education resources. There is a lot of content on single table vs. multi-table design for a single application. I’ve observed developers go down the “expert” path of single table design, but once it was operational and had to be supported, the support team has to learn the “advanced” single table concepts. They made errors, etc., and we eventually split into multi-table design because it was more supportable. Amplify CLI Challenge: Enterprise Support Amplify in it’s current form poses some challenges for use in enterprises. This is due to the amount of automation around the creation of IAM resource types on behalf of the developer. You need a large set of permissions to use including those to create new IAM resource types, which becomes an application partitioning/firewalling issue in shared accounts in addition to the increased security concerns around IAM. Suggestion: Enterprise Support Add various IAM role/policy/etc. parameters to configure. It’s aim to stay simple for the target audience may make this extra/advanced usage an product impedance mismatch, and therefore out of scope for the product. Challenge: Uncomfortable Level of Magic Amplify CLI is making AWS more accessible to “traditional” developers. Makes it really easy to get started by automating all of the resource provisioning pieces. Once you need to go outside the “happy path” / debug / troubleshoot / deal with IAM, etc., devs need to understand these concepts. The amount of infrastructure code and source code it generates for the developer has the “feel” of too much magic. All those extra artifacts that get checked into the repo makes it “feel” prone to breakage as the CLI evolves. Suggestion: Uncomfortable Level of Magic To address the users that want to break out of “beginner” mode on rails, and take more control of resource provisioning, allow the one-way “ejection” from Amplify CLI to SAM CLI. Similar to what create-react-app | npm run eject . This enables a natural learning and skills transition for devs to take on and understand more AWS concepts, and therefore more effective us of the AWS platform. AppSync Challenge: GraphQL Centered Development Learning Curve Great service and there are clear benefits to graphql centric development. There is a necessary steep learning curve inherent to graphql. Many “new” concepts for devs to learn. API, Schema, DataSource, Resolver, Resolver Function, Resolver Pipeline. New graphql schema language to learn, and to use effectively, AWS specific schema directives. Feels vendor lockin-y, but traded for the auto provisioned DB, configurable security (e.g. Cognito UserPools), generated client side code. Suggestion: GraphQL Centered Development Learning Curve Looking at other tech of the past and their evolution, community standardization around GraphQL managed services would be the next likely step. The value AppSync provides is much more than graphql itself. It builds on the benefit of the schema via tooling like Amplify CLI for strong typing, auto provision storage backend (AWS graphql directives), client-side code generation, etc. All these things are AWS specific and don’t yet have standards or a generalized community name. Maybe they never will, and it’s similar to DynamoDB as a unique only at AWS cloud differentiating super-service, but if not, AWS could be a part of that future body and drive standardization along with other cloud vendors. API Gateway Challenge: Limited Service Support for HTTP API -> AWS Service Integrations HTTP API -> AWS Service Integrations are great for simplified and removing lambda as a simple glue layer. There is currently a limited set of services supported. Suggestion: Limited Service Support for HTTP API -> AWS Service Integrations Keep the momentum with adding service support. Challenge: VTL Mapping Templates I have observed with developers I’ve worked with that VTL mapping templates cause a lot of confusion. Should they become a expert at VTL and put the logic to transform a payload in it? Another legacy language to learn that has limited returns on investment. Should they just do it in lambda with a language they are familiar with? It “feels” like it’s something that should not be exposed to customer and only used internally. The Java-ness heritage of verbosity, and yet another language to learn and know, is an implementation detail/concern being surfaced to the customer. Suggestion: VTL Mapping Templates Phase out / deprecate usage. Introduce a way to express via simplified, declarative only (no control constructs trying to be a general purpose programming language), yaml/json. EventBridge EventBridge seems like it’s on the way to being a centerpiece service of serverless solutions. As core to serverless as lambda is. Challenge: Security Model The security model around events “feels” a bit off at the moment. It almost feels like it’s bypassing security. If I can subscribe to events on a given bus, especially “default” account bus, I can access information coming from a whole host of services. It similar to the sensitivity at which CloudTrail requires. This sensitivity means more concerns around security, which limits adoption. Suggestion: Security Model It is covered by IAM Policy Conditions , but it doesn’t feel like first class support. Less than ideal DX. IAM policy conditions is an already overloaded mechanism where concepts that don’t cleanly fit into the Resources and Actions entity concepts are put. The naive answer would be a new IAM concept, like “Events” and corresponding “Producers” and “Consumers” (Actions, Resources, Events, Producers, Consumers and Condition Keys for AWS Services). Similar to Resources, Events could benefit from first class IAM entity concepts like Events, Producers, Consumers to align with developer expectations around event based systems. Challenge: Observability Events in general are challenging when it comes to observability. EventBridge is no different. Suggestion: Observability event retention per bus. similar to kinesis 7 day max. native event replay / event time travel to help with dev debugging, app state transitions, event schema evolution. Challenge: Real-time and High Volume Event Rates Also some hook into the trend of client-side event based state solutions feels natural for EventBridge (e.g. React Redux ). Maybe it’s in the form of a library that fits into Amplify since it’s aligned to client-side web and mobile tech. Given events in many systems can be high volume and high rates, exposing a native websocket API for the publishing and consumption of events would be helpful. There are already projects in the community doing this, which validates the need for it. Suggestion: Real-time and High Volume Event Rates native websocket api for high volume and real-time without having the added complexity of AppSync or API Gateway. X-Ray Challenge: Effort to Enable Observability and distributed tracing is required for almost any serverless solution since there are many services involved. This is the “serverless” version of a “traditional” stack trace and/or local application log. These are essential tools for developers. The following are friction points with the service: Instrumenting code with the x-ray sdk configuring to turn on/off in different way for each service that supports it having to understand and make decisions around sampling rates learning each services equivalent of a header to put a correlation/trace id in that doesn’t interfere with the “core” payload, etc. Suggestion: Effort to Enable Ideally it would be on by default across all services, baked into each services pricing, and the customer would not have the choice to turn off. These are plumbing level, distributed systems, “undifferentiated heavy lifting” types of concerns. The ones AWS is best at handling. The view is that aws runs all the services, and I as the customer shouldn’t need to do anything, but use the services to get this visibility. AWS Serverless Application Model (SAM) Challenge: Infrastructure Resource to Code Locality The linkage from a SAM template to local code via a directory path/s3 location/etc. suffers the locality of code problem. Infrastructure and code are clearly separated (you can do inline code up to char limit), when it should be converging and feel natural. A future state being infrastructure usage as part of a solution feels like a first class concept of the language or a library where you offload language level single process control constructs to infrastructure that is distributed, scalable, and elastic (e.g. step fn, EventBridge, SNS, SQS, etc.). To the point where I’m not thinking about the infrastructure at all. I only express it through code. Has a feeling of “shifting gears”. Suggestion: Infrastructure Resource to Code Locality With the introduction of the CDK , the lines between application code and infrastructure provisioning code is merging. This “feels” like the more natural path for SAM to adopt the CDK style. If you extend CDK to the next level of evolution in abstraction, it’s a cloud specific programming language. Infrastructure as a whole is “undifferentiated heavy lifting” and not of concern to an end user of a solution. A current means to an end. Something like “CloudScript”?, but already taken by MS :) --- ### AWS AppFlow Veeva Vault to S3 - URL: https://brianpfeil.com/post/aws-appflow-veeva-vault-to-s3/ - Date: 2020-09-08 - Tags: aws, appflow, s3 code for article  pfeilbr/aws-appflow-veeva-vault-to-s3-playground AWS AppFlow example of Veeva Vault data -> AppFlow -> S3. Notes no events from Vault. only poll / query existing data manually/on demand or schedule. at the time (2020-09-08) EventBridge was not available as a destination. available destinations: redshift, s3, salesforce Screenshot Tour veeva vault data source general information veeva vault org veeva vault connection source -> destination mapping created run history data landed in s3 Resources AppFlow Saas Applications | Veeva Vault https://developer.veevavault.com/ --- ### Product Development - URL: https://brianpfeil.com/post/product-development/ - Date: 2020-09-03 code for article  pfeilbr/product-development-playground product, product development, strategy Wardley Mapping A Wardley Map is a representation of the landscape in which a business operates. It consists of a value chain (activities needed to fulfill user needs) graphed against evolution (how individual activities change over time under supply and demand competition). A Wardley Map represents the situational awareness and shared assumptions being made about a context and hints at what strategic options are available. Concise summary of process at Understand context and diminish risk: How to build your first Wardley Map with Miro Key Points use mapping as a way to quickly validate solutions and avoid costly mistakes – to identify what to build, buy and outsource map minimizes the chance that I’ll miss something important a more evolved component can be outsourced and treated like a building block, while a less evolved component usually needs to be built from scratch or given other special consideration (money, time and energy). The real danger is in mistreating a more evolved component by building it from scratch. Outsourcing highly evolved components is a critical aspect of building cost-effective solutions. Since you have limited resources, it makes sense to save them for the components that actually need special treatment. Map Y-axis - value chain. invisible (bottom) to visible (top). user at top X-axis - genesis (new, uncertain, failure-prone), custom, product, commodity (old, boring, reliable) Example(s) Video Showing the Process https://www.youtube.com/embed/84vV8fj4Ljg Resources wardley-map-powerpoint-template.potx wardley-maps-community.github.io/awesome-wardley-maps˝ medium.com/wardleymaps Wardley map - Wikipedia Intro to Wardley Mapping Understand context and diminish risk: How to build your first Wardley Map with Miro https://twitter.com/swardley https://twitter.com/dvassallo --- ### Ant Design Pro - URL: https://brianpfeil.com/post/ant-design-pro/ - Date: 2020-09-02 code for article  pfeilbr/ant-design-pro-playground learn And Design Pro Ant Design Pro is a production-ready solution for admin interfaces. Built on the design principles developed by Ant Design, this project introduces higher level components; we have developed templates, components, and a corresponding design kit to improve the user and development experience for admin interfaces. see myApp based on Ant Design Pro | Getting Started --- ### Antd React - URL: https://brianpfeil.com/post/antd-react/ - Date: 2020-09-01 - Tags: react, design, ui, framework, ui-components code for article  pfeilbr/antd-react-playground learn Ant Design (antd) for React Files and Directories src/App.js src/App.css Resources Use in create-react-app - Ant Design This project was bootstrapped with Create React App . Available Scripts In the project directory, you can run: yarn start Runs the app in the development mode. Open http://localhost:3000 to view it in the browser. The page will reload if you make edits. You will also see any lint errors in the console. yarn test Launches the test runner in the interactive watch mode. See the section about running tests for more information. yarn build Builds the app for production to the build folder. It correctly bundles React in production mode and optimizes the build for the best performance. The build is minified and the filenames include the hashes. Your app is ready to be deployed! See the section about deployment for more information. yarn eject Note: this is a one-way operation. Once you eject, you can’t go back! If you aren’t satisfied with the build tool and configuration choices, you can eject at any time. This command will remove the single build dependency from your project. Instead, it will copy all the configuration files and the transitive dependencies (webpack, Babel, ESLint, etc) right into your project so you have full control over them. All of the commands except eject will still work, but they will point to the copied scripts so you can tweak them. At this point you’re on your own. You don’t have to ever use eject. The curated feature set is suitable for small and middle deployments, and you shouldn’t feel obligated to use this feature. However we understand that this tool wouldn’t be useful if you couldn’t customize it when you are ready for it. Learn More You can learn more in the Create React App documentation . To learn React, check out the React documentation . Code Splitting This section has moved here: https://facebook.github.io/create-react-app/docs/code-splitting Analyzing the Bundle Size This section has moved here: https://facebook.github.io/create-react-app/docs/analyzing-the-bundle-size Making a Progressive Web App This section has moved here: https://facebook.github.io/create-react-app/docs/making-a-progressive-web-app Advanced Configuration This section has moved here: https://facebook.github.io/create-react-app/docs/advanced-configuration Deployment This section has moved here: https://facebook.github.io/create-react-app/docs/deployment yarn build fails to minify This section has moved here: https://facebook.github.io/create-react-app/docs/troubleshooting#npm-run-build-fails-to-minify --- ### AWS AppFlow Salesforce EventBridge - URL: https://brianpfeil.com/post/aws-appflow-salesforce-eventbridge/ - Date: 2020-08-27 - Tags: aws, appflow, salesforce, eventbridge code for article  pfeilbr/aws-appflow-salesforce-eventbridge-playground AWS AppFlow example of salesforce event -> AppFlow -> EventBridge -> CloudWatch Logs. Followed steps in Building Salesforce integrations with Amazon EventBridge and Amazon AppFlow | Amazon Web Services post. Notes observed low latency - AppFlow flow runs within 2 seconds of event being generated in salesforce. only fields specified in mapping are passed to EventBridge as properties of detail. Private Amazon AppFlow flows is available for salesforce that uses AWS PrivateLink to route data over AWS infrastructure without exposing it to the public internet Screenshot Tour Turn on Change Data Capture for the relevant objects in salesforce Error when CDC is not turned on in salesforce for the object AppFlow leverages already installed/available “Amazon AppFlow Embedded Login App” Connected App in salesforce Run History Event in CloudWatch Logs AppFlow Mapping Resources Building Salesforce integrations with Amazon EventBridge and Amazon AppFlow | Amazon Web Services --- ### AWS API Gateway AWS Service Integration - URL: https://brianpfeil.com/post/aws-api-gateway-aws-service-integration/ - Date: 2020-08-20 - Tags: aws, api-gateway, eventbridge, api code for article  pfeilbr/aws-api-gateway-aws-service-integration-playground shows how an API Gateway endpoint can directly invoke an aws service like EventBridge, DynamoDB, etc. No lambda in the middle (reduces latency and costs). HTTP REST APIs are the most common integration pattern. API Gateway is AWS managed service for creating REST APIs. It allows us to apply security, throttling, logging, etc. to our APIs. In some cases an API is simply a passthrough to a backend AWS service. The typical approach is to have API Gateway invoke a lambda, which then contains the code to call th target AWS service. If you don’t need to perform any additional logic to the request prior to calling the backend AWS service, you may be able to use an API Gateway AWS service integration, which can directly invoke the AWS service. This simplifies the architecture, reduces request latency and reduces costs. Demo Prerequisites An AWS account AWS Command Line Interface SAM CLI Running # clone repo git clone https://github.com/pfeilbr/aws-api-gateway-aws-service-integration-playground cd aws-api-gateway-aws-service-integration-playground # deploy make deploy # test make test example output { "Entries": [ { "EventId": "58c635b4-21aa-09d8-0ad3-0874b7dc0a39" } ], "FailedEntryCount": 0 } # teardown make teardown Details invoking the EventBridge PutEvents action with the request payload using the APIGatewayEventBridgeAccessRole role. Integration: Type: AWS Credentials: !GetAtt APIGatewayEventBridgeAccessRole.Arn IntegrationHttpMethod: POST Uri: !Sub "arn:aws:apigateway:${AWS::Region}:events:action/PutEvents" setting the required headers to invoke the PutEvents action RequestTemplates: application/json: | $input.json("$") #set($context.requestOverride.header.X-Amz-Target ="AWSEvents.PutEvents") #set($context.requestOverride.header.Content-Type ="application/x-amz-json-1.1") Resources Use API Gateway as a Proxy for Another AWS Service AWS::ApiGateway::Method.Integration to dynamodb Query example PutEvents - Amazon EventBridge CloudFormation Resources Generated By SAM API Gateway integration problem --- ### AWS Comprehend - URL: https://brianpfeil.com/post/aws-comprehend/ - Date: 2020-08-18 - Tags: aws, machine-learning, javascript code for article  pfeilbr/aws-comprehend-playground learn Amazon Comprehend [Medical] Files and Directories of Interest src/index.js src/index.test.js src/fixtures src/example-responses Running # while developing npm test -- --watch Resources Amazon Comprehend - Natural Language Processing (NLP) and Machine Learning (ML) Comprehend - Amazon Comprehend Amazon Comprehend Medical - Amazon Comprehend --- ### Graphviz - URL: https://brianpfeil.com/post/graphviz/ - Date: 2020-08-18 - Tags: tools, diagram, architecture code for article  pfeilbr/graphviz-playground learn Graphviz - Graph Visualization Software example01.dot Developing # vscode "Graphviz Interactive Preview" extension was not working with images # use the following to create .png image on .dot file change # open vscode in split view with .dot in one and .png in the other. # .png view will refresh when the file is updated. fswatch -o example01.dot | xargs -n1 -I{} dot -Tpng example01.dot -oexample01.png Screenshots vscode example with “Graphviz Interactive Preview” Resources Graphviz Interactive Preview - Visual Studio Marketplace Create diagrams with code using Graphviz --- ### Mage - URL: https://brianpfeil.com/post/mage/ - Date: 2020-08-18 - Tags: golang, build, tools code for article  pfeilbr/mage-playground learn mage , a make/rake-like build tool using Go. You write plain-old go functions, and Mage automatically uses them as Makefile-like runnable targets. Install git clone https://github.com/magefile/mage cd mage go run bootstrap.go Demo git clone https://github.com/pfeilbr/mage-playground cd mage-playground go mod init mage-playground # NOTE: `go.mod` must exist in same directory as `magefile.go` # show targets mage # run "hello" target mage hello --- ### AWS SAM Step Functions - URL: https://brianpfeil.com/post/aws-sam-step-functions/ - Date: 2020-08-14 - Tags: aws, sam, step-functions, infrastructure-as-code code for article  pfeilbr/aws-sam-step-functions-playground learn and experiment with using AWS SAM to define and deploy AWS Step Functions . see template.yaml and data/event-bus-events.json Running export STACK_NAME="aws-sam-step-functions-playground" # deploy sam deploy --guided # trigger step fn via EventBridge rule aws events put-events --cli-input-json file://data/event-bus-events.json # e.g. output # { # "FailedEntryCount": 0, # "Entries": [ # { # "EventId": "369fc438-8a99-bc45-7d79-46788420dbf8" # } # ] # } # trigger via API Gateway. starts step fn then returns (does not wait for step fn to complete) curl https://17xns8g41h.execute-api.us-east-1.amazonaws.com/Prod/start/ # e.g. output # { # "executionArn": "arn:aws:states:us-east-1:529276214230:execution:SimpleStateMachine-zIFFWgUF6O6D:53313d15-1005-44d0-84a0-ea57b66d1ac3", # "startDate": 1.597426336318E9 # } # teardown aws cloudformation delete-stack --stack-name "aws-sam-step-functions-playground" --region "us-east-1" Resources Simplifying application orchestration with AWS Step Functions and AWS SAM [aws | events | put-events]](https://docs.aws.amazon.com/cli/latest/reference/events/put-events.html#examples ) --- ### AWS SAM Golang - URL: https://brianpfeil.com/post/aws-sam-golang/ - Date: 2020-07-30 - Tags: aws, sam, golang, iam, testing code for article  pfeilbr/aws-sam-golang-playground An example API and Worker written in Golang using the Amazon Serverless Application Model (AWS SAM). modified version of cpliakas/aws-sam-golang-example . switched to use go modules, added the use of local environment variables, and run sam local assuming lambda function role. Overview Go is arguably one of the easiest languages in which to write a RESTful API. With the addition of Go support for AWS Lambda coupled with the maturity of tooling around the AWS Serverless Application Model , deploying Golang-based APIs to serverless infrastructure is becoming much more straightforward, too. Thanks to the APEX Gateway , you can even write APIs in a familiar manner without changing how the code is structured. The purpose of this project is to give a slightly more complicated example than the “hello world” ones provided by Amazon with a toolchain that supports both local development and deployment to AWS as well as design patterns that facilitate unit testing. Prerequisites An AWS account Golang dep Docker Node.js AWS Command Line Interface SAM CLI jq (optional) Installation With a correctly configured Go toolchain: git clone https://github.com/pfeilbr/aws-sam-golang-example Usage Run the API Locally :warning: Make sure to install all the Prerequisites . On Mac OSX and Windows, ensure that the Docker VM is running. Build the API and run it locally: GOARCH=amd64 GOOS=linux go build -o api ./service/api sam local start-api or … make run You can now consume the API using your tool of choice. HTTPie is pretty awesome. http localhost:3000/ HTTP/1.1 200 OK Content-Length: 28 Content-Type: application/json; charset=utf8 Date: Sat, 03 Feb 2018 20:12:07 GMT { "message": "Hello, world!" } Deploy to AWS First, set the following environment variables replacing <MY-BUCKET-NAME> and <MY-STACK-NAME> as appropriate: export S3_BUCKET="<MY-BUCKET-NAME>" export STACK_NAME="<MY-STACK-NAME>" Now build, package, and deploy the application: GOOS=linux GOARCH=amd64 go build -o api ./service/api GOOS=linux GOARCH=amd64 go build -o error ./service/error GOOS=linux GOARCH=amd64 go build -o worker ./service/worker sam package --template-file template.yaml --s3-bucket $S3_BUCKET --output-template-file packaged.yaml sam deploy --stack-name $STACK_NAME --template-file packaged.yaml --capabilities CAPABILITY_IAM or … make deploy Consume the Endpoint The API endpoint is captured in the CloudFormation stack’s Endpoint output key. Either view the output value via the AWS Management Console, or run the following command assuming the jq tool is installed: aws cloudformation describe-stacks --stack-name $STACK_NAME | jq -r '.Stacks[0].Outputs[0].OutputValue' Again, HTTPie is a pretty awesome tool. View AWS Logs Run the following command to get the CloudWatch logs for the API. sam logs -n Api --stack-name $STACK_NAME Replace Api with Worker or Error to get logs for the Lambda functions in those resources as well. :warning: The sam tool will throw a nasty stack trace if you try to view the logs before the Lambda function has been invoked. Only run this command after you have made requests to the corresponding handlers. Session example development session export S3_BUCKET="${S3_SAM_DEPLOY_BUCKET}" export STACK_NAME="$(basename $(pwd))" # test make test # build make build # build make deploy # start SAM local API sam local start-api --profile my-lambda-role --env-vars env-vars.json # GET curl -X POST http://127.0.0.1:3000 # POST a job curl -X POST http://127.0.0.1:3000/job -d '{"name": "my job"}' # tail the logs for the worker that processes SQS messages lumigo-cli tail-cloudwatch-logs --namePrefix "/aws/lambda/aws-sam-golang-example-Worker" --region "us-east-1" --- ### AWS SAM Local Invoke with Lambda Role - URL: https://brianpfeil.com/post/aws-sam-local-invoke-with-lambda-role/ - Date: 2020-07-30 - Tags: aws, sam, lambda code for article  pfeilbr/aws-sam-golang-playground AWS Serverless Application Model (SAM) allows you to develop and test your lambda backed API Gateway endpoints locally via sam local start-api . By default, your function is invoked with the default credentials you have configured for the AWS CLI . If your function accesses other AWS services, it may encounter permission issues. It’s ideal to have your lambda run under as close to the same security context locally as it would when deployed. In the example below, lambda sends a message to a SQS worker queue. The permissions for the queue are configured to allow the lambda role to send a message to it. The following details how to achieve this. Allowing Our Lambda Role to be Assumed In our scenario, we have a local user profile named admin stored in ~/.aws/credentials [admin] aws_access_key_id = YOUR_ACCESS_KEY aws_secret_access_key = YOUR_SECRET_ACCESS_KEY This user needs to be able to assume our my-lambda-role role. We define the following Role Trust policy to enable: { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "Service": "lambda.amazonaws.com" }, "Action": "sts:AssumeRole" }, { "Effect": "Allow", "Principal": { "AWS": "arn:aws:iam::xxxxxxxxxxxx:user/admin" }, "Action": "sts:AssumeRole" } ] Configure Assume Role via AWS CLI Next we need to configure the AWS CLI to assume a role. We can do so by adding the following to ~/.aws/config # ~/.aws/config [profile my-lambda-role] role_arn = arn:aws:iam::xxxxxxxxxxxx:role/my-lambda-role source_profile = admin output = json region = us-east-1 Note the source_profile = admin line. This identifies the profile in ~/.aws/credentials that will be used to assume the role. See How do I assume an IAM role using the AWS CLI? for full details on options. We can test with the following: aws --profile my-lambda-role sts get-caller-identity If successful, the response will look like. { "UserId": "AROAXWO2SDPLLBS55Q345:botocore-session-1596129185", "Account": "xxxxxxxxxxxx", "Arn": "arn:aws:sts::529276214230:assumed-role/my-lambda-role/botocore-" } Running SAM Local The SQS queue URL is provided to the lambda via an environment variable. This is defined in the SAM template template.yaml Environment: Variables: QUEUE_URL: !Ref Queue We need QUEUE_URL to be available to our function running locally. To do that we can create an environment variable .json file and pass as a parameter for sam local to use. env-vars.json { "Parameters": { "QUEUE_URL": "https://sqs.us-east-1.amazonaws.com/xxxxxxxxxxxx/aws-sam-golang-example-Queue-Q12J860AETTS" } } We can now start the local SAM API server and test our endpoint sam local start-api --profile my-lambda-role --env-vars env-vars.json curl -X POST http://127.0.0.1:3000/job -d '{"name": "my job"}' Conclusion We’ve seen how to run a lambda locally with the same security context as when deployed. SAM provides a great development workflow to allow quick iterations. The local environment provided via docker tries to be as true to the AWS environment, but you should test with the real services as early as possible. One point to note, you must have the AWS resources your lambda is interacting with provisioned. For example, the SQS queue must exist. SAM does not provide a locally running SQS service. The source for this example is written in Go, which has a great workflow. Our focus wasn’t on the code, but if you want more Go + SAM + Lambda, be sure to check out the AWS GO SERVERLESS! workshop by AWS Serverless Developer Advocate Rob Sutter . It covers the details and workflow of using Go with SAM. --- ### Serverless Architecture: Key Service Considerations - URL: https://brianpfeil.com/post/serverless-architecture-key-considerations-per-service/ - Date: 2020-07-20 - Tags: architecture, aws, serverless A serverless architecture is “typically” composed of many services. The following covers the key considerations and configuration options for the most common AWS services leveraged for serverless architectures. Relevant Patterns Lambda SNS SQS Kinesis EventBridge DynamoDB Step Functions API Gateway CloudFront Route53 Global Accelerator WAF Relevant Patterns common cloud native patterns to consider in the context of serverless architectures of scale event sourcing circuit breaker - trip circuit to prevent downstream systems overload load shedding - prevent backlog buildup handle poison messages - prevent kinesis and dynamodb streams from progressing prevent distributed transactions. e.g. lambda send job to SQS and stores status in dynamodb. break it up. lambda put job status in dynamo -> dynamo stream -> lambda send job to SQS Lambda synchronous vs asynchronous vs poll based (poll based is sync) - impacts automatic retries, stuck messages due to poison message, etc. see Understanding the Different Ways to Invoke Lambda Functions if lambda is strictly a glue passthrough for API Gateway to call a backend AWS service, look to use API Gateway Service Proxies to remove lambda. simpler/cheaper/etc. memory DLQ lambda destinations (only for async invokes) reserved concurrency - concurrency allocated for a specific function. e.g. i always want fn X to be able to run 10 lambda invokes concurrently provisioned concurrency - pre-warmed lambda instances / no cold starts. good for latency sensitive needs can optionally use auto scaling to adjust on based on metrics and/or schedule. will spill over to on-demand scaling (lambda default) Provisioned Concurrency comes out of your regional concurrency limit concurrent executions (throttles) - 1000 per account timeout - 15min set code timeouts based on remaining invocation time provided in context burst concurrency - 500 - 3000 burst - 500 new instances / min poll based options (kinesis, dynamodb, SQS) on-failure destination (SNS or SQS) retry attempts max age of record - use to implement load shedding (prioritize newer messages) split batch on error concurrent batches per shard SNS fan out to address scale KMS to encrypt payloads SQS batch size - batch fails as unit visibility timeout - set to 6x lambda timeout message retention period delivery delay - max 15min types - standard vs FIFO standard - at least once delivery. need to ensure idempotent alarm on queue depth KMS Kinesis partition key - choose wisely as order is guaranteed per shard and pk determines the shard the message lands on poison messages (retry until success - can cause backlog) KMS to encrypt payloads enhanced fan-out via AWS::Kinesis::StreamConsumer. each consumer gets 2 MiB per second for every shard you subscribe to. can subscribe a max of 5 consumers per stream. EventBridge put events - 2400 requests per second per region invocation quota - 4500 requests per second per region (invocation is an event matching a rule and being sent on to the rule’s targets) DynamoDB global tables - for resilient active-active architectures throttles streams - 24hr data retention. poison messages (retry until success - can cause backlog) partition key - distribute data among nodes to minimize hot partitions TTL - can the data be removed automatically Step Functions Standard Workflows vs Express Workflows saga pattern for rollback parallel map opportunities - run tasks in parallel API Gateway REST API vs HTTP API (cheaper ) caching - fixed cost based on time / no pay per use throttles timeout - 29s auth - cognito, JWT, IAM (aws sigv4), custom lambda auth OpenAPI specs for payload validation service proxies - no need for lambda glue in middle custom domains websockets CloudFront origin access identity to force traffic through CloudFront and removes direct access to S3 website domain URL signed URLs or cookies lambda@edge - headers only requests, rewrite URLs, server-side rendering (SSR), auth, etc. cache invalidations non GET HTTP methods support. must explicitly turn on support for PUT, POST, PATCH, etc. WAF in front Route53 Geoproximity routing for global solutions serving multiple regions Global Accelerator uses the AWS global network to optimize the path from your users to your applications, improving the performance of your traffic by as much as 60% WAF can put in front of API Gateway or CloudFront API Gateway provides overlapping functionality with WAF. Need to determine the appropriate service to use. --- ### Cloud Native Software Architecture - URL: https://brianpfeil.com/post/cloud-native-software-architecture/ - Date: 2020-07-18 - Tags: architecture, cloud A personal cheatsheet/reference for cloud native software architecture. architecture how the components are assembled and organized. This will be done in a way that meets the quality attributes. Key Questions Organization Considerations Quality Attributes (*ities and friends) Patterns event-sourcing Core Design Decisions Resources Hexagonal Resources Topics / Concepts / Terms Database Shuffle Sharding Constant Work Canary Resources Books (oreilly.com) Websites Key Questions who are the users? what devices and form factors will be used? what is the context of their usage? scale and growth? who are the main actors in the system (domain objects - e.g. orders, products, etc.)? data classifications (PII)? data types and sizes (relation records, documents, media files, etc.)? what is the time frame for delivery? is there an existing product / SaaS / open-source / etc. that provides the solution or a portion / components of it capacity estimation & constraints? functional requirements? Non Functional Requirements - Latency, Consistency, Availability, High Throughput, etc. what is explicitly out of scope organization and teams structure see System Design: DoorDash β€” a prepared food delivery service for good reference Organization Considerations engineering (application & platform) operations (application & platform) Quality Attributes (*ities and friends) reliability - ability to continue to operate under predefined conditions availability - ratio of the available system time to the total working time scalability - ability of the system to handle load increases without decreasing performance efficiency performance security cost interoperability correctness maintainability readability extensibility testability Patterns modern cloud native architecture patterns as of July 2020 event-sourcing Capture all changes to an application state as a sequence of events. Core Design Decisions Domain Entities and Events popular method is via Event Storming Event Content each event stores delta state each event stores full state idempotent is easy to solve for duplicate events Total Ordering (ordered stream of events - ledger) ensure all event are processed in order. this is needed for causal relationships. e.g. ordering matters for two messages related to the same entity Resources Scaling Event Sourcing for Netflix Downloads, Episode 1 Scaling Event Sourcing for Netflix Downloads, Episode 2 InfoQ | Scaling Event Sourcing for Netflix Downloads | Video + Presentation - shows in detail how they implemented event sourcing backed by cassandra matrinfowler.com | Event Sourcing Pattern: Event sourcing EventBridge Storming β€” How to build state-of-the-art Event-Driven Serverless Architectures - approach to defining the Events, Boundaries and Entities in your business domain Decomposing the Monolith with Event Storming Hexagonal the ports and adapters architecture. decouples core domain logic from specific storage, database, protocol, etc. Resources Hexagonal Architecture: three principles and an implementation example Topics / Concepts / Terms Database CAP theorem Consistency: Every read receives the most recent write or an error Availability: Every request receives a (non-error) response, without the guarantee that it contains the most recent write Partition tolerance: The system continues to operate despite an arbitrary number of messages being dropped (or delayed) by the network between nodes Serializability Snapshot isolation Multiversion concurrency control Things I Wished More Developers Knew About Databases Shuffle Sharding limits / isolates tenants in a multi-tenant system so they don’t negatively impact other tenants. method of assigning tenant to resources. Resources Workload isolation using shuffle-sharding Constant Work overprovision resources to the point where it would operate correctly even if an availability zone were to be unavailable if AZ becomes unavailable, no new resources need to be provisioned, just a quick re-routing. you are essentially always operating the infrastructure for failure mode (active-active) Resources Static stability using Availability Zones Canary A canary release is a technique to reduce the risk from deploying a new version of software into production. A new version of software, referred to as the canary, is deployed to a small subset of users alongside the stable running version. Traffic is split between these two versions such that a portion of incoming requests are diverted to the canary. This approach can quickly uncover any problems with the new version without impacting the majority of users. Resources Automated Canary Analysis at Netflix with Kayenta Resources Books (oreilly.com ) Fundamentals of Software Architecture Clean Architecture: A Craftsman’s Guide to Software Structure and Design, First Edition Software Architecture Patterns Building Evolutionary Architectures Clean Architecture: A Craftsman’s Guide to Software Structure and Design, First Edition Domain-Driven Design: Tackling Complexity in the Heart of Software Microservices Patterns Patterns of Enterprise Application Architecture Refactoring: Improving the Design of Existing Code Design Patterns: Elements of Reusable Object-Oriented Software Designing Distributed Systems Designing Distributed Control Systems: A Pattern Language Approach (Wiley Software Patterns Series) Websites martinfowler.com AWS Architecture Center AWS Architecture Blog Amazon Builders’ Library Azure Architecture Center medium | articles tagged with “software architecture” C4 model for visualizing software architecture --- ### Lumigo CLI - URL: https://brianpfeil.com/post/lumigo-cli/ - Date: 2020-07-17 - Tags: cli code for article  pfeilbr/lumigo-cli-playground lumigo-CLI is a command line tool that contains a collection of helpful commands for working with AWS Lambda and the serverless AWS services it integrates with. It’s an invaluable tool for serverless development on AWS. It contains commands that let you work with individual services line sending messages to SNS, SQS, and Kinesis. Tailing changes to CloudWatch Logs, DynamoDB, Kinesis, EventBridge, SNS, SQS. Automatically tune lambda for the optimal memory configuration, measure lambda cold starts. Easy to install via global npm package (npm install lumigo-cli -g). Example Usage # display contents of gzipped cloudfront log file to stdout # see S3 SQL reference @ https://docs.aws.amazon.com/AmazonS3/latest/dev/s3-glacier-select-sql-reference-select.html lumigo-cli s3-select-batch \ --region="us-east-1" \ --bucket="com.brianpfeil.cloudfront.logs" \ --prefix="brianpfeil.com/E2IL5HY5XTHLNW.2019-09-17-21.66bbda6b.gz" \ --expression="SELECT * FROM S3Object s" \ --fileType="CSV" \ --compressionType="GZIP" # list Lambda functions in ALL regions lumigo-cli list-lambda --- ### AWS Interactive Video Service - URL: https://brianpfeil.com/post/aws-interactive-video-service/ - Date: 2020-07-16 - Tags: aws code for article  pfeilbr/aws-interactive-video-service-playground learn Amazon Interactive Video Service based on Getting Started with Amazon Interactive Video Service Running # create channel via https://docs.aws.amazon.com/ivs/latest/userguide/GSIVS.html # streaming an existing video via ffmpeg VIDEO_FILEPATH="/Users/pfeilbr/Downloads/tailer.mp4" # fortnite game trailer STREAM_KEY="sk_us-east-1_LvBpPyJZzkix_yMQ6LPLZ7PDr6dYsI0zqM4H2oZTL31" INGEST_ENDPOINT="0f426742eaf2.global-contribute.live-video.net" ffmpeg -re -stream_loop -1 -i $VIDEO_FILEPATH -r 30 -c:v libx264 -pix_fmt yuv420p -profile:v main -preset veryfast -x264opts "nal-hrd=cbr:no-scenecut" -minrate 3000 -maxrate 3000 -g 60 -c:a aac -b:a 160k -ac 2 -ar 44100 -f flv rtmps://$INGEST_ENDPOINT/app/$STREAM_KEY # view in AWS Console | Amazon IVS | Live Channels Screenshots Resources Introducing Amazon Interactive Video Service (Amazon IVS) Amazon Interactive Video Service – Add Live Video to Your Apps and Websites Amazon Interactive Video | Docs --- ### Bref PHP Lambda - URL: https://brianpfeil.com/post/bref-php-lambda/ - Date: 2020-07-13 - Tags: php, lambda code for article  pfeilbr/bref-php-lambda-playground learn bref , deploy and run serverless PHP applications Prerequisites PHP serverless framework Running # install composer require bref/bref # init vendor/bin/bref init # edit `index.php` # deploy serverless deploy # test / invoke serverless invoke -f "function" # test / invoke with event data serverless invoke --function "function" --data '{"name": "foo"}' Resources bref | Installation --- ### Heroku Node Worker - URL: https://brianpfeil.com/post/heroku-node-worker/ - Date: 2020-07-13 - Tags: heroku code for article  pfeilbr/heroku-node-worker-playground example of running a long lived worker process on heroku that is not a web app/server Prerequisites heroku account Heroku CLI Running # clone git clone https://github.com/pfeilbr/heroku-node-worker-playground.git # change to root directory cd heroku-node-worker-playground # create heroku app heroku create # >>> make changes to `worker.js`, install packages, etc. # test locally npm run worker # commit changes git commit -a -m "my awesome changes" # deploy / push changes to heroku git push heroku master # if worker is already running, this push will kill it and start the new one # run worker only / stop web dyno heroku ps:scale web=0 worker=1 # verify running heroku ps # view logs heroku logs -t # stop worker heroku ps:scale worker=0 # start worker heroku ps:scale worker=1 Resources Background Jobs in Node.js with Redis Run Non-web Java Dynos on Heroku --- ### Strapi - URL: https://brianpfeil.com/post/strapi/ - Date: 2020-07-10 - Tags: cms, javascript code for article  pfeilbr/strapi-playground learn strapi , the open-source headless CMS Prerequisites docker desktop Session # create app npx create-strapi-app my-project --quickstart # local docker install / run via https://strapi.io/documentation/v3.x/installation/docker.html # pull docker-compose pull # run docker-compose up -d # open web ui open http://localhost:1337/admin Screenshots Resources Strapi Documentation --- ### PHP Lambda Layer - URL: https://brianpfeil.com/post/php-lambda-layer/ - Date: 2020-07-02 - Tags: php, lambda code for article  pfeilbr/php-lambda-layer-playground learn stackery/php-lambda-layer for running PHP on lambda Prerequisites php 7.x composer installed globally (used by makefile) Key Files src/php/index.php src/php/php.ini - for enabled extensions template.yaml packaged.yaml - generated via sam Session see Makefile for details and make changes to variables as necessary. # deploy stack make deploy # test api gateway endpoint URL make test # delete stack make delete Resources Introducing the new Serverless LAMP stack Introducing the serverless LAMP stack – part 2 relational databases aws-samples/php-examples-for-aws-lambda - github repo for article. AWS Lambda Custom Runtime for PHP: A Practical Example - older article, but covers low level details AWS SDK for PHP --- ### Serverless Golang - URL: https://brianpfeil.com/post/serverless-golang/ - Date: 2020-06-17 - Tags: serverless, golang code for article  pfeilbr/serverless-golang-playground see myservice/README.md for build and deploy steps NOTE this is based on serverless framework from Jan 2018 (just adding as repo now June 2020) see golang.serverlessworkshops.io for more modern example (go modules ) using AWS SAM --- ### AWS EventBridge - URL: https://brianpfeil.com/post/aws-eventbridge/ - Date: 2020-05-19 - Tags: aws, eventbridge code for article  pfeilbr/aws-eventbridge-playground learn AWS EventBridge # put events on default event bus aws events put-events --entries file://sample-events/my-custom-app-events.json Resources https://github.com/jbesw/s3-to-lambda/blob/master/eventbridge/README.md - examples using AWS::Events::Rule in SAM to map event sources to targets (e.g. lambda, kinesis, etc.) --- ### GoReleaser - URL: https://brianpfeil.com/post/goreleaser/ - Date: 2020-04-16 - Tags: golang, continuous-delivery code for article  pfeilbr/goreleaser-playground learn GoReleaser release automation tool for Go projects. see https://goreleaser.com/quick-start/ Session goreleaser --snapshot --skip-publish --rm-dist export GITHUB_TOKEN='GITHUB_TOKEN' # tag. release name is based on it git tag -a v0.1.0 -m "First release" git push origin v0.1.0 # run locally without publishing to github goreleaser --snapshot --rm-dist # build and publish goreleaser --- ### AWS Services - URL: https://brianpfeil.com/post/aws-services/ - Date: 2020-03-29 - Tags: aws AWS Services The following services are commonly used for AWS solutions. Each service specifies key considerations and features per service for architecture and design. Serverless Services Serverless Benefits Relevant Patterns Deployment Types Networking and Content Delivery VPC ELB/ALB PrivateLink / VPC Endpoint Route 53 Cloud Map Global Accelerator CloudFront API Gateway AppSync Compute EC2 ECS ECR Fargate Batch EKS App Runner LightSail Elastic Beanstalk Lambda Lambda@Edge EC2 Image Builder Storage S3 Glacier EFS (Elastic File System) FSx FSx for Windows File Server FSx for Lustre FSx for ONTAP Storage Gateway Amazon File Cache EBS AWS Transfer AWS Backup Database DynamoDB DocumentDB (MongoDB compatibility) RDS Amazon Aurora Aurora Serverless Redshift Redshift Serverless ElastiCache ElasticSearch / OpenSearch Amazon OpenSearch Serverless Neptune Timestream Cloud Directory SSM Parameter Store Application Integration AppFlow Step Functions SNS SQS SES EventBridge Kinesis AWS Application Composer Analytics Kinesis Data Analytics Pinpoint EMR Data Pipelines Glue Glue DataBrew Athena QuickSight Lake Formation Management & Governance Well-Architected Framework Control Tower Organizations CloudFormation Serverless Application Repository (SAR) Service Catalog Config AppConfig CloudWatch Logs CloudWatch Events (see EventBridge) CloudWatch Insights CloudWatch Metrics CloudWatch Dashboards CloudWatch Alarms CloudWatch Synthetics (Canaries) CloudTrail Proton Developer Tools AWS Application Composer Cloud9 CodeCommit CodeBuild CodeDeploy CodePipeline CodeArtifact X-Ray AWS CLI Amplify SAM (Serverless Application Model) CDK AWS SDKs Migration & Transfer AWS DataSync AWS DMS (Database Migration Service) Machine Learning SageMaker Comprehend Polly Rekognition Textract Translate Transcribe Forecast Personalize Lex Kendra Security, Identity, and Compliance IAM Cognito Secrets Manager WAF Certificate Manager (ACM) KMS Directory Service Media Services Amazon Interactive Video Service Serverless Services Route53 Global Accelerator WAF Cognito CloudFront API Gateway AppSync Amplify Lambda DynamoDB S3 SNS SQS SES Kinesis EventBridge Glue Step Functions Athena SSM Parameter Store Secrets Manager AppConfig AWS Config CloudWatch Synthetics (Canaries) CloudWatch Metrics and Alarms CloudWatch Logs CloudFormation Serverless Application Repository (SAR) SAM (Serverless Application Model) CDK X-Ray Serverless Benefits less things to own less/no ops costs - pay-per-use elastic / limits scaling concerns deliver value quicker scale teams / org fit durability/resiliency - services built-in replication across AZs or regions every service has soft limits for protection Relevant Patterns event sourcing circuit breaker - trip circuit to prevent downstream systems overload load shedding - prevent backlog buildup handle poison messages - prevent kinesis and dynamo streams from progressing prevent distributed transactions. e.g. lambda send job to SQS and stores status in dynamodb. break it up. lambda put job status in dynamo -> dynamo stream -> lambda send job to SQS strangler - migrate from monolith to serverless. e.g. DB - run RDS and dynamodb in parallel and update both for a period of time Deployment Types all-at-once blue/green canary - traffic shift percentages with metrics linear - changing the amount of traffic split to the new version incrementally according to a percentage that is provided when configured. Networking and Content Delivery VPC virtual private cloud Subnets, route tables, internet gateways, elastic ips, nat gateways, network ACLs, security groups, prefix lists ELB/ALB Elastic Load Balancing (TCP) ALB application load balancer β€” Layer 7 (HTTP/HTTPS traffic), Flexible NLB network load balancer β€” Layer 4 (TLS/TCP/UDP traffic), Static IPs CLB classic load balancer β€” Layer 4/7 (HTTP/TCP/SSL traffic), Legacy, Avoid The NLB forwards requests whereas the ALB examines the contents of the HTTP request header to determine where to route the request. So, the ALB is performing content based routing. Resource Type Hierarchy AWS::ElasticLoadBalancingV2::LoadBalancer -> AWS::ElasticLoadBalancingV2::Listener -> [AWS::ElasticLoadBalancingV2::ListenerRule] (e.g. path based) -> AWS::ElasticLoadBalancingV2::TargetGroup -> [Instance | IP Address | Lambda | ALB] PrivateLink / VPC Endpoint PrivateLink allows access to AWS services and customer provided services hosted by other AWS customers, while maintaining network traffic within the AWS network. connect to AWS services from VPC without going through internet enables you to privately connect your VPC to supported AWS services and VPC endpoint services powered by PrivateLink without requiring an internet gateway, NAT device, VPN connection, or AWS Direct Connect connection gateway endpoint - S3 and DynamoDB. via VPC route table. gateway that you specify as a target for a route in your route table for traffic destined to a supported AWS service interface endpoint - all other services. via DNS resolver for VPC/subnets an elastic network interface with a private IP address from the IP address range of your subnet that serves as an entry point for traffic destined to a supported service specify subnets, security groups, IAM policy doc, enable private DNS VPC endpoint services - create your own application in your VPC and configure it as an AWS PrivateLink-powered service (referred to as an endpoint service). Other AWS principals can create a connection from their VPC to your endpoint service using an interface VPC endpoint. You are the service provider, and the AWS principals that create connections to your service are service consumers. Route 53 managed DNS main service for reliability / DR public and private domains (hosted zones) route to aws services - CloudFront, API Gateway, ELB, RDS, S3 bucket, EC2, VPC Interface Endpoint record sets, TTL health checks - public endpoints or via CloudWatch metrics (e.g. for private endpoints. see AlarmIdentifier ) can associate a health check with recordset. e.g. Route 53 failover recordset. load balancing via DNS routing policies - latency Based Routing, Geo DNS, Geoproximity, and Weighted Round Robin domain registration geo routing, geoproximity routing alias record type (aws specific. used for root/bare/naked domains) AWS Route53 β€” Cheat Sheet(In 2 Minutes) Cloud Map name and discover your cloud resources via API or DNS Global Accelerator uses the highly available and congestion-free AWS global network to direct internet traffic from your users to your applications on AWS fixed entry point to your applications through static IP addresses allocates static Anycast IP addresses that are globally unique for your application and do not change CloudFront CDN POP / edge servers - traffic over AWS global infrastructure price classes ACM for TLS/SSL certs cache policies. cookie, headers, querystring, TTLs configs origin access identity origin custom headers signed URLs or cookies origin groups - primary origin and a second origin to failover to custom error responses - http error codes mapped to response page paths georestrictions lambda@edge - headers only requests, rewrite URLs, server-side rendering (SSR), auth, etc. cloudfront functions - run lightweight JavaScript code no network and file system access. max run time - run less than 1 ms. Where a Lambda@Edge is deployed to one of the 13 regional edge locations, Cloudfront Functions are deployed even further down and closer to the viewer at one of the 280+ edge locations CloudFront KeyValueStore - a low-latency, in-memory data store that can be used to store and retrieve data from within a CloudFront Function cache invalidations non GET HTTP methods support. must explicitly turn on support for PUT, POST, PATCH, etc. WAF association can point to Object Lambda Access Point API Gateway REST API vs HTTP API (cost). see Choosing between HTTP APIs and REST APIs edge (cloudfront) and regional endpoints caching (memcached) (fixed cost based on time / no pay-per-use) API Keys Usage Plans / quotas client certificates - ensure requests to backend are from APIG throttles with WAF in front, you can set up rate-based rules to specify the number of web requests that are allowed by each client IP in a trailing, continuously updated, 5-minute period. no API Key required for this timeout - 29s request (POST) payload limits (10 MB). no response size limits. (tested with proxy integration for 200 MB video file download) auth - cognito, JWT, IAM (aws sigv4), custom lambda auth OpenAPI / Swagger specs for payload validation service integrations - no need for lambda glue in middle. e.g. apig -> step fns velocity templates (vtl) - request/response mapping custom domain names custom domain names for REST APIs custom domain names for HTTP APIs private REST API - endpoints that are accessible only from within your VPC see diagram API Gateway private integrations - expose your HTTP/HTTPS resources within a VPC for access by clients outside of the VPC VPC links for REST APIs - apig endpoint -> (NLB within VPC) VPC links for HTTP APIs - apig endpoint -> ([LB within VPC) see Understanding VPC links in Amazon API Gateway private integrations see diagram websockets lambda integration. point to lambda alias for deployments. stages mock integrations / responses AppSync GraphQL managed service integrates with Amazon DynamoDB, Amazon Elasticsearch, and Amazon Lambda resolvers resolver mapping templates via velocty (vtl) Real-time subscriptions aws specific graphql schema @directives for model (ddb), auth (cognito), GraphiQL javascript for vtl coming (2021-03-28) Compute EC2 AMI elastic IPs ASGs (launch templates) UserData - script to run on instance start EC2 metadata service ECS container management service Fargate and EC2 launch types prefer ECS on Fargate over ECS on EC2. No managing ec2 cluster with fargate if using ECS on EC2 use Bottlerocket Using Bottlerocket with Amazon ECS Amazon ECS-optimized AMI variant of the Bottlerocket operating system is provided as an AMI task definitions Service - maintain a specified number of instances of a task definition Service load balancing - distribute traffic evenly across the tasks in your service Service auto scaling - via Application Auto Scaling service. CPU/memory utilization CW metrics EFS or EBS for persistent storage EFS is recommended. Can be mounted by multiple ECS tasks for parallel access EBS can be used but is tied to a hosting EC2 instance. Not supported on fargate. ECR container registry concepts Registry - can create image repositories in your registry and store images in them Authorization token - client must authenticate to Amazon ECR registries as an AWS user before it can push and pull images Repository - contains your Docker images, Open Container Initiative (OCI) images, and OCI compatible artifacts Repository policy - can control access to your repositories and the images within them with repository policies Image - push and pull container images to your repositories can use them in ECS task definitions and EKS pod specifications ECR public resource-based permissions using AWS IAM Announcing Pull Through Cache Repositories for Amazon Elastic Container Registry Fargate Docs | Amazon ECS on AWS Fargate containers task definitions run containers without having to manage servers or clusters removes the need to choose server types, decide when to scale your clusters, or optimize cluster packing. When you run your tasks and services with the Fargate launch type, you package your application in containers, specify the CPU and memory requirements, define networking and IAM policies, and launch the application. Amazon ECS tasks for Fargate can authenticate with private image registries, including Docker Hub, using basic authentication. When you enable private registry authentication, you can use private Docker images in your task definitions. Fargate Spot - you can run interruption tolerant Amazon ECS tasks at a discounted rate compared to the Fargate price. Fargate Spot runs tasks on spare compute capacity. When AWS needs the capacity back, your tasks will be interrupted with a two-minute warning. ECS Service - desired task count. allows you to run and maintain a specified number of instances of a task definition simultaneously in an Amazon ECS cluster. If any of your tasks should fail or stop for any reason, the Amazon ECS service scheduler launches another instance of your task definition to replace it in order to maintain the desired number of tasks in the service. optionally front with ALB Service autoscaling is done using the Application Auto Scaling service. See How can I configure Amazon ECS Service Auto Scaling on Fargate? Multi AZ by default - Fargate will look to spread Task placement across all available Availability Zones Deployment types - rolling update, blue/green (via CodeDeploy) Batch run batch computing jobs using containers concepts: Compute Environments - set of managed or unmanaged compute resources that are used to run jobs. Fargate or EC2. specify the minimum, desired, and maximum number of vCPUs for the environment. Job Queues - backed by 1 or more compute envs, assign priority Job Definitions - cpu and memory requirements, iam role for access to other aws resources Jobs - things that run on fargate or ec2. shell script, a Linux executable, or a Docker container image Array Jobs - run parallel jobs such as Monte Carlo simulations, parametric sweeps, or large rendering jobs. AWS_BATCH_JOB_ARRAY_INDEX is passed to each job as an env var to represent the current index Automated Job Retries Job Dependencies EKS managed Kubernetes automates the deployment, scaling, and management of containerized applications App Runner Service Types Image-based service - container image (docker) Code-based service - source code and a supported runtime (managed platforms . e.g. python, node.js, java, .net, php, ruby, go) When you create an AWS App Runner service using a source code repository, AWS App Runner requires information about building and starting your service. You can set service options by using a configuration file (apprunner.yaml ). service source: source code (python, node, java, etc. provided managed runtimes) and source image (container/docker image). compute configurations - 0.25 vCPU (512 MB memory) and up to 4 vCPU (12 GB memory) public and private (VPC) support for service endpoints Repository provider - github, bitbucket and ECR are supported private service (endpoint) - enables access to App Runner services from within a VPC. Enabling Private endpoint for incoming traffic AWS::AppRunner::VpcIngressConnection - associate your App Runner service to an Amazon VPC endpoint AWS::AppRunner::VpcIngressConnection.DomainName . build URL with !Sub https://${AppRunnerService1VpcIngressConnection.DomainName} create AWS::EC2::VPCEndpoint with ServiceName: !Sub "com.amazonaws.${AWS::Region}.apprunner.requests" see KarlDeux/arps/template.yaml for full example. service access VPC resources - e.g. app access to RDS database running in VPC VPC Connector - enables associate your service with a VPC by creating a VPC endpoint Enabling VPC access for outgoing traffic AWS::AppRunner::VpcConnector LightSail Virtual servers, storage, databases, and networking for a low, predictable price. backed by EC2, but easier to use similar to DigitalOcean Elastic Beanstalk PaaS with language runtime + docker containers heroku-like App Runner is the recommended service for new applications Lambda synchronous, asynchronous, poll based/stream processing (poll based is sync. via event-source mappings), respoonse streaming memory - single knob for memory and CPU (between 128 MB and 10,240 MB, and up to 6 vCPUs) Lambda function URLs DLQ lambda destinations (only for async invokes) reserved concurrency - concurrency allocated for a specific function. e.g. i always want fn X to be able to run 10 lambda invokes concurrently provisioned concurrency - pre-warmed lambda instances / no cold starts. good for latency sensitive needs can optionally use auto scaling to adjust on based on metrics and/or schedule. will spill over to on-demand scaling (lambda default) Provisioned Concurrency comes out of your regional concurrency limit concurrent executions (throttles) - 1000 per account timeout - 15min set code timeouts based on remaining invocation time provided in context burst concurrency - 500 - 3000 burst - 500 new instances / min poll based options (kinesis, dynamodb, SQS) on-failure destination (SNS or SQS) retry attempts max age of record - use to implement load shedding (prioritize newer messages) split batch on error concurrent batches per shard APIG -> lambda ALB -> lambda service integrations - Using AWS Lambda with other services - AWS Lambda lambda private endpoints - access lambda from VPC without going over internet Lambda Extensions - executables in /opt/extensions that conform to the Lambda Extensions API Container Images Runtime interface clients runtime interface client in your container image manages the interaction between Lambda and your function code Lambda provides an open source runtime interface client for each of the supported Lambda runtimes. e.g. node.js, python, etc. Lambda Runtime Interface Emulator allows customers to locally test their Lambda function packaged as a container image web-server that converts HTTP requests to JSON events and maintains functional parity with the Lambda Runtime API max image size: 10 GB Lambda Wrapper Scripts | Modifying the runtime environment - customize the runtime startup behavior of your Lambda function. e.g. set env vars, add/update parameters. code example aws-samples/aws-lambda-environmental-variables-from-aws-secrets-manager AWS Lambda Operator Guide Lambda runtime management controls visibility into which patch version of a runtime your function is using and when runtime updates are applied can optionally synchronize runtime updates with function deployments roll back your function to an earlier runtime version. example - lambda CloudWatch log line Runtime Version: python:3.9.v14 Runtime Version ARN: arn:aws:lambda:eu-south-1::runtime:7b620fc2e66107a1046b140b9d320295811af3ad5d4c6a011fad1fa65127e9e6I Introducing AWS Lambda response streaming send responses larger than Lambda’s 6 MB response payload limit up to a soft limit of 20 MB. Shared responsibility model for AWS Lambda Lambda@Edge feature of Amazon CloudFront that lets you run code closer to users of your application, which improves performance and reduces latency EC2 Image Builder automate the creation, management, and deployment of customized, secure, and up-to-date β€œgolden” server images that are pre-installed and pre-configured with software and settings to meet specific IT standards. automate the creation of EC2 AMI and docker images Storage S3 object/blob storage with strong read-after-write consistency versioned buckets presigned URLs for private content (download or upload) S3 batch operations S3 select Query via Athena batch operations - perform operation on list of objects specified in manifest. e.g. lambda, copy, etc. storage classes - Standard, Standard-IA, Intelligent-Tiering, One Zone-IA, Glacier Instant Retrieval, Glacier Flexible Retrieval, Glacier Deep Archive lifecycle rules - moving between storage tiers for cost savings replication - cross-region, same-region access points - managing data access at scale, access points are unique hostnames, enforce distinct permissions and network controls for any request made through the access point, scale to many applications accessing bucket with own set of permissions. S3 Object Lock - write-once-read-many (WORM) model. retention periods and legal holds. static website hosting Multi-Region Access Points addresses pain point- Managing access to this shared bucket requires a single bucket policy that controls access for dozens to hundreds of applications with different permission levels Multi-Region access points S3 event notifications - notification destinations are SNS, SQS, lambda Use Amazon S3 Event Notifications with Amazon EventBridge S3 object lambda - process data retrieved from S3 with lambda before returning it to an application. lambda calls writeGetObjectResponse to send modified object contents back to GET request. Create S3 Access Point, then Object Lambda Access Point. Mountpoint for Amazon S3 Glacier low cost/long-term object/blob storage EFS (Elastic File System) elastic file system for Linux-based workloads for use with AWS Cloud services and on-premises resources. can mount as NFS v4 e.g. shared file system. many EC2 instances can mount same efs file system. can mount to lambda local filesystem FSx File System X, where X means your choice + any workload NetApp ONTAP, Windows File Server, Lustre, ZFS FSx for Windows File Server fully managed native windows file system SMB, NTFS, AD integration FSx for Lustre fully managed native lustre file system - the open-source, parallel file system designed to support any Linux workload and is POSIX-compliant sub-millisecond latencies, millions of IOPS, and throughput of as much as hundreds of gigabytes per second. workload types - HPC, ML, media processing and transcoding, big data deployment options Scratch File Systems - shorter-term processing of data and for temporary storage. In the event that a file server fails, data does not persist and is not replicated. Persistent File Systems - file servers remain highly available and data is automatically replicated in the same Availability Zone (AZ) as the file system. The data volumes connected to the file servers are replicated separately from the file system they are attached to mount lustre file system to S3 bucket Choosing an Amazon FSx File System FSx for ONTAP NFS, SMB and iSCSI storage powered by NetApp HA across multi-AZs automatic data tiering to S3, and instant data cloning Storage Gateway NFS or SMB interface to S3, FSx, volume/tape gateways compute runs on once of following: ec2, kvm, VMware, Hyper-V, or appliance if SMB, need to Configuring Microsoft Active Directory access VPC support. network traffic between compute and AWS Service goes over VPC endpoint VPC endpoint enabled, all VPC endpoint communication from your gateway to AWS services occurs through the public service endpoint using your VPC in AWS Creating a VPC endpoint for Storage Gateway Amazon File Cache temporary, high-performance storage for data in on-premises file systems, or in file systems or object stores on AWS link the cache to multiple NFS file systemsβ€”including on-premises and in-cloud file systemsβ€”or Amazon Simple Storage Service (S3) buckets, providing a unified view of and fast access to your data spanning on-premises and multiple AWS Regions create cache by specifying: Cache name, Cache storage capacity (Set this to a value of 1.2 TiB, 2.4 TiB, or increments of 2.4 TiB), Throughput capacity, VPC, Security Groups, Subnets create Data repository associations (DRAs) - linking your cache to Amazon S3 or NFS data repositories Data repository path (e.g. s3://my-bucket/my-prefix/), cache path (e.g. /ns1/subdir1) mount NFSv3 or S3. sudo mount -t lustre -o relatime,flock cache_dns_name@tcp:/mountname /mnt supports ec2, ecs, eks write to cache support. need to manually export changes to sources To export changes from the cache, use HSM commands. When you export a file or directory using HSM commands, your cache exports only data files and metadata that were created or modified since the last export. see https://docs.aws.amazon.com/fsx/latest/FileCacheGuide/exporting-files-hsm.html sudo lfs hsm_archive path/to/export/file sudo lfs hsm_state path/to/export/file access your cache from a Linux instance, first install the open-source Lustre client. EBS block level storage volumes for use with EC2 instances. EBS volumes behave like raw, unformatted block devices AWS Transfer SFTP to S3 enables the transfer of files directly into and out of S3 using SFTP public facing and w/in VPC File Transfer Workflows (AWS::Transfer::Workflow ) – MFTW is a fully managed, serverless File Transfer Workflow service that makes it easy to set up, run, automate, and monitor the processing of uploaded files. AWS Backup centralize and automate data protection across AWS services configure backup policies and monitor activity for your AWS resources in one place backup and restore for: EC2 instances, S3 data, EBS volumes, DynamoDB tables, RDS database instances, Aurora clusters, EFS and FSx FSs, Storage Gateway volumes, DocumentDB clusters, Neptune clusters concepts Backup Vault Backup Plans Backup Selection - e.g. backup resources having tag(s) Database DynamoDB concepts - tables, items, queries, scans, indexes global tables - for resilient active-active architectures DAX - DynamoDB Accelerator - in memory cache in front GSI (Global Secondary Indexes), LSI (Local Secondary Indexes) transactions throttles point-in-time recovery (PITR) streams - 24hr data retention. poison messages (retry until success - can cause backlog) partition key - distribute data among nodes to minimize hot partitions TTL - can the data be removed automatically parallelization factor for DDB streams processed by lambda single table designs fine grained item (dynamodb:LeadingKeys) and attribute level IAM (dynamodb:Attributes). enables multi-tenant isolation. Amazon DynamoDB Encryption Client PartiQL - A SQL-Compatible Query Language for Amazon DynamoDB - Amazon DynamoDB dynamodb table export to s3 dynamodb table import from s3 DynamoDB Standard-IA table class you will save up to 60 percent in storage costs as compared to using the DynamoDB Standard table class. However, DynamoDB reads and writes for this new table class are priced higher than the Standard tables DynamoDB zero-ETL integration with Amazon OpenSearch Amazon DynamoDB zero-ETL integration with Amazon Redshift DocumentDB (MongoDB compatibility) RDS Aurora, PostgreSQL, MySql, MariaDB, Oracle, SQL Sever DB Instance (contains 1 or more dbs), Instance Classes (compute+memory), Instance Storage HA Multi-AZ Database Activity Streams - supported for Oracle and SQL Server Amazon Aurora fully managed relational database engine that’s compatible with MySQL and PostgreSQL compute and storage are separate. underlying storage grows automatically as needed. An Aurora cluster volume can grow to a maximum size of 128 tebibytes (TiB) HA - data remains safe even if some or all of the DB instances in the cluster become unavailable Aurora DB cluster consists of one or more DB instances and a cluster volume that manages the data for those DB instances When data is written to the primary DB instance, Aurora synchronously replicates the data across Availability Zones to six storage nodes associated with your cluster volume. Doing so provides data redundancy, eliminates I/O freezes, and minimizes latency spikes during system backups. cluster endpoint - a connection string that stays the same even when a failover promotes a new primary instance. cluster endpoint always represents the current primary instance in the cluster. DB Instance Types Primary DB instance - Supports read and write operations, and performs all of the data modifications to the cluster volume. Each Aurora DB cluster has one primary DB instance. Aurora Replica - Connects to the same storage volume as the primary DB instance and supports only read operations. Each Aurora DB cluster can have up to 15 Aurora Replicas in addition to the primary DB instance. Amazon Aurora connection management primary instance handles all data definition language (DDL) and data manipulation language (DML) statements. Up to 15 Aurora Replicas handle read-only query traffic Types of Aurora endpoints Cluster endpoint - read/write/DDL/DML (e.g. mydbcluster.cluster-123456789012.us-east-1.rds.amazonaws.com:3306) Reader endpoint - provides load-balancing support for read-only connections to the DB cluster. (e.g. mydbcluster.cluster-ro-123456789012.us-east-1.rds.amazonaws.com:3306) Custom endpoint - represents a set of DB instances that you choose. When you connect to the endpoint, Aurora performs load balancing and chooses one of the instances in the group to handle the connection. (e.g. myendpoint.cluster-custom-123456789012.us-east-1.rds.amazonaws.com:3306) Instance endpoint - connects to a specific DB instance within an Aurora cluster. (e.g. mydbinstance.123456789012.us-east-1.rds.amazonaws.com:3306) Amazon Aurora global databases Aurora Serverless Using the Data API for Aurora Serverless v1 - http API, no long lived connections see Class: AWS.RDSDataService Redshift managed data warehouse service. postgres foundation RA3 instances - Scale compute and storage independently for fast query performance and lower costs UDFs - lambda backed, Redshift Data API - http based (no JDBC or ODBC). async so can retrieve results later. query results stored for 24 hrs redshift spectrum - SQL queries on data stored in S3 Kinesis Data Streaming ingestion - eliminates the need to stage data in S3 before ingesting it into Redshift zero-ETL integrations - RDS -> Redshift Amazon DynamoDB zero-ETL integration with Amazon Redshift Redshift Serverless automatically provisions data warehouse capacity and intelligently scales the underlying resources concepts Namespace is a collection of database objects and users Workgroup is a collection of compute resources RPUs - Redshift Processing Units (RPUs). RPUs are resources used to handle workloads. costs - billed RPU hours on a per-second basis. no queries are running, not billed for compute capacity. charged for Redshift managed storage, based on the amount of data stored. ElastiCache managed redis and memcached ElasticSearch / OpenSearch cluster kibana - integrated with IAM IAM for granular es api operations automated time-based data deletion via index lifecycle policies Amazon OpenSearch Serverless removes need to configure, manage, or scale OpenSearch clusters (called provisioned OpenSearch domains, which you manually manage capacity for) collection - group of indices representing a specific workload or use case. two collection types - time series and search capacity is managed for you. you create a collection, then you query and index data using the same OpenSearch APIs as before Serverless compute capacity is measured in OpenSearch Compute Units (OCUs). Each OCU is a combination of 6 GiB of memory and corresponding virtual CPU (vCPU), as well as data transfer to S3 decouples compute and storage. separates the indexing (ingest) components from the search (query) components, with S3 as the primary data storage for indexes. can scale search and index functions independently of each other and independently of the indexed data in S3. costs - charged for the following components: data ingestion compute search and query compute storage retained in S3 billed for a minimum of 4 OCUs for the first collection in your account CloudFormation Resource Types | OpenSearch Serverless Neptune graph database. query languages Apache TinkerPop Gremlin and SPARQL (RDF) Timestream time series database use SQL to query Concepts Time series - A sequence of one or more data points (or records) recorded over a time interval. Record - A single data point in a time series Multi-measure records - store multiple measures in a single table row, instead of storing one measure per table row Dimension - An attribute that describes the meta-data of a time series. A dimension consists of a dimension name and a dimension value. Measure - The actual value being measured by the record. Examples are the stock price, the CPU or memory utilization, and the temperature or humidity reading Timestamp - Indicates when a measure was collected for a given record Table - A container for a set of related time series. Database - A top level container for tables. Scheduled Query - Timestream periodically and automatically runs these queries and reliably writes the query results into a separate table. serverless and fully managed Query string - query to run Schedule expression - when your scheduled query instances are run. using a cron expression Target configuration - map the result of a scheduled query into the destination table where the results of this scheduled query will be stored. Notification configuration - receive a notification for every such query run on an SNS topic that you configure when you create a scheduled query. fully decoupled data ingestion, storage, and query architecture where each component can scale independent of other components storage tiering, with a memory store for recent data and a magnetic store for historical data InfluxDB, Prometheus, Riak Cloud Directory cloud-native directory that can store hundreds of millions of application-specific objects with multiple relationships and schemas SSM Parameter Store Systems Manager Parameter Store provides secure, hierarchical storage for configuration data management and secrets management Application Integration AppFlow managed integration (ETL) service securely transfer data between SaaS applications (Salesforce, Marketo, Slack, etc.), and AWS services (S3, Redshift, EventBridge, etc.) concepts - flow, source, destination, flow trigger (on demand, event, schedule), map fields from source to destination (formula transforms, value validations), filters (determine records to transfer) Flow notifications - flow start|complete|deactivated events sent to CloudWatch Events/EventBridge ("source": "aws.appflow") security encryption at rest - connection data stored in secrets manger using AWS managed or Customer managed CMK Encryption in Transit (TLS 1.2) - choose either an AWS managed CMK or a customer managed CMK. When executing a flow, Amazon AppFlow stores data temporarily in an intermediate S3 bucket and encrypts it using this key. This intermediate bucket is deleted after 24 hours, using a bucket lifecycle policy. Actions defined by Amazon AppFlow AppFlow integrations - built-in integrations Custom Connector SDK (python or java) - source data from any system Step Functions Standard Workflows (max duration: 1 yr) exactly-once model Express Workflows (max duration: 5 min) need to be idempotent / at-least-once model, where an execution could potentially run more than once States - Task, Choice, Wait, Pass, Parallel, Map, Succeed, Fail Activities - enables you to have a task in your state machine where the work is performed by a worker that can be hosted anywhere. start other [sub] workflows error handling and Retrying after an error Input/Output Processing InputPath to select a portion of the state input Parameters field to create a collection of key-value pairs that are passed as input ResultSelector field to manipulate a state’s result before ResultPath is applied OutputPath enables you to select a portion of the state output to pass to the next state orchestration with many built-in integrations to aws services Step Functions AWS SDK Service Integrations full execution event history for a given execution Intrinsic functions - use these instead of creating lambdas to do logic. Logging using CloudWatch Logs Logging Step Functions Using AWS CloudTrail CloudWatch Metrics EventBridge Events - Execution started, succeeded, failed, timed out, aborted X-Ray tracing support saga pattern for rollback parallel map opportunities - run tasks in parallel service integrations - request/response, run a job (.sync), callback with task token (.waitForTaskToken) JSONPath expressions Data flow simulator Step Functions Workflow Studio - visual designer for state machines VPC Endpoint support Step Functions Local - downloadable version of Step Functions that lets you develop and test applications using a version of Step Functions running in your own development environment. SNS if not doing “sizable” fan-out, see if EventBridge meets the need (more features around archiving, replay, retries) pub/sub message filtering with subscription push notifications standard topic at least once delivery best effort ordering - ensure downstream consumers are idempotent FIFO topic strict ordering Strict deduplication: Duplicate messages aren’t delivered. Deduplication happens within a 5-minute interval, from the message publish time. FIFO Topic Message Archiving and Replay - e.g. replay topic messages to SQS fan out subscription filters destination types SQS lambda http/s mobile push notifications SMS messages email DLQ configuration KMS encryption SQS managed message queuing service batch size - batch fails as unit visibility timeout - set to 6x lambda timeout message retention period delivery delay - max 15min types - standard vs FIFO standard - at least once delivery. need to ensure idempotent FIFO - strict ordering. exactly-once processing alarm on queue depth KMS encryption DLQ for redrive for messages that can’t be delivered to target SQS queue partial batch response - Reporting batch item failures when used with lamdba SES send or receive emails verify domain (DNS txt) and/or email addresses (confirmation email) - verify that you own the email address or domain that you plan to send from understand Service quotas max message size - 10 MB per message (after base64 encoding). sending identity - domain or an email address send emails via SMTP or API (AWS CLI, AWS SDK ) connect to a URL that provides an endpoint for the Amazon SES API or SMTP interface (e.g. email-smtp.us-east-1.amazonaws.com:587) DKIM support - DKIM works by adding a digital signature to the headers of an email message. This signature can then be validated against a public cryptographic key that is located in the organization’s DNS record SPF support - SPF establishes a method for receiving mail servers to verify that incoming email from a domain was sent from a host authorized by that domain’s administrators IAM to control user access to email sending (e.g. ses:SendEmail) Configuration sets - groups of rules that you can apply to the emails you send using Amazon SES. can publish email sending events to CWL, Firehose, SNS Event types - Send, Reject, Delivery, Bounce, Complaint, Click Open Rendering Failure store inbound emails in S3 trigger lambdas based on inbound emails publish your email sending events to CWLs or kinesis firehose Sending personalized email via email templates. templates contain placeholder values. based on Handlebars template system list management customers can manage their own mailing lists, known as contact lists. can create topics, associate topic preferences to a contact and specify OPT_[IN|OUT] for the topic. Global Suppression List includes a global suppression list. When any Amazon SES customer sends an email that results in a hard bounce, Amazon SES adds the email address that produced the bounce to a global suppression list. The global suppression list is global in the sense that it applies to all Amazon SES customers. In other words, if a different customer attempts to send an email to an address that’s on the global suppression list, Amazon SES accepts the message, but doesn’t send it, because the email address is suppressed. enabled by default for all Amazon SES accounts. You can’t disable it. reputation dashboard to track bounce and complaint rates Dedicated IP Addresses IP pool management – If you lease dedicated IP addresses to use with Amazon SES, you can create groups of these addresses, called dedicated IP pools. You can then associate these dedicated IP pools with configuration sets SES sandbox - all new accounts. only send mail to verified email addresses and domains only send mail from verified email addresses and domains send a maximum of 200 messages per 24-hour period send a maximum of 1 message per second need to request production access to move out of sandbox VPC endpoint support - see New – Amazon Simple Email Service (SES) for VPC Endpoints EventBridge pub/sub with many built-in integrations integrate with external SaaS or any custom application bus-to-bus routing within same account + region, x-account, and x-region. dlq for eb rules. if fails to deliver to target, goes in sqs queue e.g. can log all events in account including CloudTrail to CloudWatch Log Group put events - 2400 requests per second per region AWS service rule targets at-least-once event delivery to targets (ensure idempotent behavior) no ordering guarantees schema registry - helps with managing and versioning event schemas for evolution. Codegen code for handling events in various languages. can auto discover schemas by observing events on the bus. based on json schema invocation quota - 4500 requests per second per region (invocation is an event matching a rule and being sent on to the rule’s targets) DLQ EventBridge resource policies archive and replay events IAM - resource-based and identity-based policies. owner of EB resources (bus, rules, etc.) is an AWS root account. supports sending and recieving events across accounts API destinations - integrate with services outside of AWS using REST API calls. e.g. send events to a third-party service that doesn’t have a built-in integration with EventBridge SaaS Partner Integrations X-Ray Tracing support EventBridge Scheduler - can schedule one-time or recurrently tens of millions of tasks across many AWS services Amazon EventBridge Pipes create point-to-point integrations between event producers and consumers with optional transform, filter and enrich steps high-level concepts: sources , source event filter , transform and enrich , targets Kinesis collect, process, and analyze real-time, streaming data kafka alternative partition key shard count Kinesis Data Streams On-Demand Mode - charged per gigabyte of data written, read, and stored in the stream, in a pay-per-throughput fashion ordering guaranteed for messages per shard dynamic partitioning - continuously partition streaming data in Kinesis Data Firehose using keys within data like β€œcustomer_id” or β€œtransaction_id” and deliver data grouped by these keys into corresponding Amazon Simple Storage Service (Amazon S3) prefixes lambda - lambda polls per shard batch size batch window parallelization factor Concurrent batches per shard – Process multiple batches from the same shard concurrently. enhanced fan-out via AWS::Kinesis::StreamConsumer. each consumer gets 2 MiB per second for every shard you subscribe to. can subscribe a max of 5 consumers per stream. Starting position - Latest, Trim horizon, At timestamp On-failure destination Retry attempts Maximum age of record – The maximum age of a record that Lambda sends to your function. Split batch on error poison messages (retry until success - can cause backlog) KMS aggregate multiple records into one while staying under size limits to increase throughput. see https://github.com/awslabs/kinesis-aggregation no autoscaling around shards. requires management/ops. consider SQS first as there’s less to manage and see if it can meet the need AWS Application Composer Analytics Kinesis Data Analytics Process and analyze streaming data using SQL or Java. two application types are supported SQL Applications - AWS specific concepts - application, input steam -> application code (SQL statements) -> output stream time based windows. tumbling windows. pump Apache Flink Applications - OSS standard for streaming you can use Java, Scala, or SQL to process and analyze streaming data use DataStream API and Table API Pinpoint usage, customer, and engagement analytics EMR hadoop, spark, and friends Amazon EMR Serverless part of the job specification, you can provide the minimum and maximum number of concurrent workers, and the vCPU, memory, and storage for each worker charged for aggregate vCPU, memory, and storage resources used from the time workers start executing till the time they terminate, rounded up to the nearest second with a one-minute minimum. Data Pipelines data processing workloads AWS Data Pipeline, you can regularly access your data where it’s stored, transform and process it at scale, and efficiently transfer the results to AWS services such as Amazon S3, Amazon RDS, Amazon DynamoDB, and Amazon EMR. Glue catalog / metadata (hive metadata catalog) crawlers autodiscover schema data sources - S3, RDS, JDBC, dynamodb, mongodb, documentdb data targets - S3, RDS, JDBC, mongodb, documentdb jobs job types - Spark, Streaming ETL (kinesis, kafka via spark structured streaming (micro batches)), and Python shell python shell job start-up time - 7-30 secs (based on usage observations) languages - [py]spark and scala concepts - Data Catalog, Classifier, Connection, Crawler, Database, Table, Dynamic Frame (extend spark RDD), Job, Transform, Trigger (time based or event) Auto Scaling for AWS Glue AWS Glue interactive sessions - run locally via jupyter notebook. replaces Glue Dev endpoint. serverless with no infrastructure to manage. You can start interactive sessions very quickly. Interactive sessions have a 1-minute billing minimum with cost-control features. This reduces the cost of developing data preparation applications. Developing using a Docker image - AWS Glue hosts Docker images on Docker Hub to set up your development environment with additional utilities. glue notebook (Jupyter/Zeppelin) - interactive development and testing of your ETL scripts on a development endpoint partitions AWS Data Wrangler - excellent integration library to use with glue via python shell jobs Starting an AWS Glue Workflow with an Amazon EventBridge Event - trigger a workflow/job via EventBridge Glue DataBrew visual data preparation service extract, clean, normalize, transform, combine, data at scale target audience: non-technical Data Analyst serverless. pay for what you use concepts datasets recipes - steps to apply/take on dataset job - recipe + dataset run project - visual workspace for working with data interactively. can apply changes and visually see the results in UI. you specify a sampling of the data to work with. Athena serverless querying of S3 data federated query - run SQL queries across data stored in relational, non-relational, object, and custom data sources. CTAS - create table as select query S3 data in place. pay per query / data accessed. integrated with glue catalog Presto is underlying tech Connecting to Amazon Athena with ODBC and JDBC Drivers QuickSight Integration QuickSight Lake Formation Management & Governance Well-Architected Framework describes the key concepts, design principles, and architectural best practices for designing and running workloads in the cloud 6 pillars - Operational Excellence, Security, Reliability, Performance, Cost Optimization, Sustainability Well-Architected Tool - provides guidance by answer questions Lens - serverless, analytics, ML, SaaS, etc. Control Tower set up and govern a new, secure multi-account AWS environment. builders can provision new AWS accounts in a few clicks, while you have peace of mind knowing your accounts conform to your company-wide policies Organizations account management service that lets you consolidate multiple AWS accounts into an organization that you create and centrally manage. CloudFormation declarative provisioning of AWS infrastructure/resource parameters, mappings, conditionals intrinsic functions change sets nested stacks stack drift stacksets - deploy stack to multiple regions. provide the accounts and regions to deploy to. For DR, active-active, etc. max resources declared in stack (500) custom resources - backed by lambda macros - lambda performs the template processing / transform modules - package resource configurations for inclusion across stack templates, in a transparent, manageable, and repeatable way CloudFormation Registry IaC generator - generate a CloudFormation template using resources provisioned in your account that are not already managed by CloudFormation Serverless Application Repository (SAR) enables teams, organizations, and individual developers to find, deploy, publish, share, store, and easily assemble serverless architectures any cfn can be used for SAR app iam for access to SAR app Service Catalog create and manage catalogs of IT services that are approved for use on AWS concepts: products are cloudformation templates portfolio is collection of products access to portfolios is via IAM users, groups, roles IT administrator creates products and portfolios and grants access End user accesses products and deploys them approved self-service products from Solution Factory e.g. Oracle RDS DB with all security, tags, etc. in place e.g. static web site. S3 + CloudFormation + WAF + ACM (certificate) + Route 53 (hosted zone, domain) Service Actions - enable end users to perform operational tasks, troubleshoot issues, run approved commands, or request permissions in AWS Service Catalog via SSM docs. Config monitor, notify, quarantine, remediate based on resource changes. RDK - rule development kit. Config triggers lambda on resource changes. AWS Config is a service that enables you to assess, audit, and evaluate the configurations of your AWS resources. Config continuously monitors and records your AWS resource configurations and allows you to automate the evaluation of recorded configurations against desired configurations. define rules that get evaluated when any change is made (e.g. resource provisioned) conformance - collection of Config rules and remediation actions. portable. can be applied across multiple accounts and regions there are aws managed rules that are part of the service and you can define custom ones via lambda AppConfig feature flags, Update applications without interruptions, Control deployment of changes across your application a capability of AWS Systems Manager, to create, manage, and quickly deploy application configurations. AppConfig supports controlled deployments to applications of any size and includes built-in validation checks and monitoring. You can use AppConfig with applications hosted on EC2 instances, AWS Lambda, containers, mobile applications, or IoT devices. JSON Schema Validators - ensure that new or updated configuration settings conform to the best practices required by your application e.g. a JSON doc with application configuration can be sourced from S3, parameter store CloudWatch Logs centralize the logs from all of your systems, applications, and AWS services that you use, in a single, highly scalable service concepts - log groups, log streams subscriptions - real-time feed of log events from CloudWatch Logs (Kinesis [stream|firehose], elasticsearch, Lambda) Creating Metrics From Log Events Using Filters encrypt with KMS CloudWatch Events (see EventBridge ) cron triggers CloudWatch Insights query log groups CWL Query Syntax CloudWatch Metrics time series data metric - time-ordered set of data points that are published to CloudWatch concepts - namespace, dimensions (name/value pairs), units (Bytes, Seconds, Count, and Percent), time stamp, resolution (granularity) statistics - sum, max, min, average, sample count, percentile (pNN) (metric data aggregations over specified periods of time) metrics retention Creating Metrics From Log Events Using Filters Embedded Metric Format - generate metrics from structured (json) log messages CloudWatch Metric Streams - metrics are delivered to a Kinesis Data Firehose stream CloudWatch Dashboards CloudWatch Dashboards - for Visualizations Dashboard Custom Widgets - dashboard widget that can call any AWS Lambda function with custom parameters. It then displays the returned HTML or JSON. CloudWatch Dashboard Sharing - Share a single dashboard and designate specific email addresses, Share a single dashboard publicly, so that anyone who has the link can view the dashboard, Share all the CloudWatch dashboards in your account and specify a third-party single sign-on (SSO) CloudWatch Alarms notify via email, SNS topics create a CloudWatch alarm that watches a single CloudWatch metric or the result of a math expression based on CloudWatch metrics An alarm watches a single metric over a specified time period, and performs one or more specified actions, based on the value of the metric relative to a threshold over time. The action is a notification sent to an Amazon SNS topic or an Auto Scaling policy. You can also add alarms to dashboards. composite alarms Alarm States - OK, ALARM, INSUFFICIENT_DATA (missing data points) EventBridge integration - CloudWatch sends events to Amazon EventBridge whenever a CloudWatch alarm changes alarm state. CloudWatch Synthetics (Canaries) supports monitoring your REST APIs, URLs, and website content every minute, 24x7, and alerts you when your application endpoints don’t behave as expected. Node.js or python based. bundles in Puppeteer + Chromium to the runtime trigger types - cron (1 min smallest freq), run once can run in VPC can also used in any workloads requiring general browser automation creates several CloudWatch metrics in CloudWatchSynthetics namespace EventBridge support. See monitoring canary events with Amazon EventBridge CloudFormation support via AWS::Synthetics::Canary create canaries, configurable scripts that run on a schedule, to monitor your endpoints and APIs. Canaries follow the same routes and perform the same actions as a customer, which makes it possible for you to continually verify your customer experience even when you don’t have any customer traffic on your applications. By using canaries, you can discover issues before your customers do. CloudTrail logs all recording AWS API and Management Console actions to S3 can query via Athena Proton self-serve for platform enabling teams enables the standardization of cross cutting concerns for microservices based solutions (composition of many microservices). e.g. the following is needed for each microservice and should be consistent/aligned with standards and best practices: compute, DNS, load balancing, code deployment pipeline, monitoring and alarms similar goals as Netflix’s Spinnaker k8s ecosystem Developer Tools AWS Application Composer visual designer that you can use to build your serverless applications from multiple AWS services Cloud9 cloud/browser based compute environment and IDE. dev machine (ec2 amzn linux) in the cloud with browser based IDE and terminal CodeCommit fully-managed source control service that hosts secure Git-based repositories CodeBuild managed build service provides prepackaged build environments continuous integration service that compiles source code, runs tests, and produces software packages like Jenkins, Travis, CircleCI concepts - build project - environment (linux/windows, container image to use, etc.), buildspec.yml - phases, env vars, artifacts can be used to run ad-hoc workloads over lambda when need to run longer than 15 min CodeDeploy automates software deployments to a variety of compute services such as Amazon EC2, AWS Fargate, AWS Lambda, and your on-premises servers CodePipeline continuous delivery service that helps you automate your release pipelines orchestrates CodeBuild and CodeDeploy sources: github, CodeCommit, S3 CodeArtifact fully managed software artifact repository service that makes it easy for organizations of any size to securely store, publish, and share packages used in their software development process artifactory competitor X-Ray distributed tracing instrument code similar to zipkin, jaeger AWS CLI ~/.aws/[config|credentials] --generate-cli-skeleton - e.g. aws codebuild start-build --generate-cli-skeleton > build.json -> aws codebuild start-build --cli-input-json file://start-build.json Amplify CLI to provision resources (Auth (cogntio), API (API Gateway), GraphQL (AppSync), Storage (S3, DynamoDB)) client-side javascript/typescript, iOS, Android libraries and UI components Amplify Console. CI/CD static site hosting. SAM (Serverless Application Model) higher-level cfn resource types (transformed via cfn macro on backend) SAM CLI local development features via docker (apig endpoint CDK express resources using general purpose programming languages (ts/js/python/java/C#) constructs - cfn (L1), CDK (L2), pattern/solution (L3) synth to cfn cloud assemblies - cfn + source code, docker images, assets (s3) aspects - ability to visit each node/resource in stack and apply changes Application -> Stacks -> Constructs Runtime context [tf|k8s] CDKsnc jsii - core/foundational tech for multi-language/polyglot support. bind any language to underlying typescript implementation. CDK pipelines for CI/CD CDK Migrate - cloudformation to CDK AWS SDKs built in retries, timeouts can configure timeouts (e.g. AWS.config.update({maxRetries: 2, httpOptions: { timeout: 2 * 1000, connectTimeout: 3 * 1000, },})) Migration & Transfer AWS DataSync copy data between NFS, SMB, S3, EFS, FSx, HDFS concepts transfer types Data transfer between self-managed storage and AWS need to install an agent that can access self-managed storage Data transfer between AWS storage services no need to install an agent agent VM that runs the sync software. can run on EC2 or hypervisors (VMware ESXi, KVM, and Microsoft Hyper-V hypervisors) scheduled transfers Data transfer between self-managed storage and AWS Data transfer between AWS storage services AWS DMS (Database Migration Service) migrate RDBS, data warehouses, nosql dbs, etc. in cloud, between combos of cloud and on-prem it’s a server (EC2) in the cloud that runs replication software (replication engine). DMS replication instance types create source and target connections schedule task on server to move data pay-as-you-go model data at rest is encrypted SSL / TLS encrypts data in -flight HA with multi-AZ deployment can provision DMS resources using CloudFormation. migration types: one-time, ongoing replication (CDC) AWS DMS doesn’t perform schema or code conversion you can use the AWS Schema Conversion Tool (AWS SCT) create endpoints to access source or target data store. endpoint properties Endpoint type – Source or target. Engine type – Type of database engine, such as Oracle or PostgreSQL.. Server name – Server name or IP address that AWS DMS can reach. Port – Port number used for database server connections. Encryption – Secure Socket Layer (SSL) mode, if SSL is used to encrypt the connection. Credentials – User name and password for an account with the required access rights. At a high level, when using AWS DMS you do the following: Create a replication server. Create source and target endpoints that have connection information about your data stores. Create one or more migration tasks to migrate data between the source and target data stores. A replication task can consist of three major phases: The full load of existing data The application of cached changes Ongoing replication At the start of the ongoing replication phase, a backlog of transactions generally causes some lag between the source and target databases. The migration eventually reaches a steady state after working through this backlog of transactions. If your migration is heterogeneous (between two databases that use different engine types), you can use the AWS Schema Conversion Tool (AWS SCT) to generate a complete target schema for you. Depending on the Amazon EC2 instance class you select, your replication instance comes with either 50 GB or 100 GB of data storage public and private replication instances You use a private instance when both source and target databases are in the same network that is connected to the replication instance’s VPC. The network can be connected to the VPC by using a VPN, AWS Direct Connect, or VPC peering. DMS Replication Process Replication Machine Learning SageMaker build, train, and deploy machine learning models provides a suite of built-in algorithms (via docker containers). provides common machine learning algorithms that are optimized to run efficiently against extremely large data in a distributed environment prebuilt containers for common machine learning frameworksβ€”such as Tensorflow, Pytorch, and MxNet customer can bring own custom containers (bring-your-own-algorithms and frameworks) To train a model in SageMaker, you create a training job that includes S3 location with training data Specify ML Compute instances managed by SageMaker S3 location to store output of job ECR path to where training code is model serving endpoints (inference) (SageMaker Hosting Services ). your endpoint config specifys instance type, number of instances, etc. jupyter notebooks SageMaker notebook instance is a machine learning (ML) compute instance running the Jupyter Notebook App Amazon SageMaker Python SDK Serverless Inference - ideal for workloads which have idle periods between traffic spurts and can tolerate cold starts. Serverless endpoints automatically launch compute resources and scale them in and out depending on traffic, eliminating the need to choose instance types or manage scaling policies Asynchronous inference - ideal for requests with large payload sizes (up to 1GB), long processing times (up to 15 minutes), and near real-time latency requirements. Asynchronous Inference enables you to save on costs by autoscaling the instance count to zero when there are no requests to process, so you only pay when your endpoint is processing requests. SageMaker Model Building Pipelines - build pipelines using the SageMaker Python SDK can run locally via SageMaker Pipelines local mode Comprehend NLP (natural language processing) By utilizing NLP, you can extract important phrases, sentiment, syntax, key entities such as brand, date, location, person, etc., and the language of the text find insights and relationships in text use case e.g.: gauge whether customer sentiment is positive, neutral, negative, or mixed based on the feedback you receive via support calls, emails, social media, and other online channels Polly text-to-speech (TTS) supports MP3, Vorbis, and raw PCM audio stream formats Neural Text-to-Speech (NTTS) voices Rekognition API to analyze any image or video file identify the objects, people, text, scenes, and activities, as well as detect any inappropriate content. Textract extracts text and data from scanned documents supports PNG, JPEG, and PDF formats. For synchronous APIs, you can submit images either as an S3 object or as a byte array. For asynchronous APIs, you can submit S3 objects Translate neural machine translation service for translating text to and from English across a breadth of supported languages Transcribe audio to text transcription services for your audio files. It uses advanced machine learning technologies to recognize spoken words and transcribe them into text. Forecast managed deep learning service for time-series forecasting. By providing Amazon Forecast with historical time-series data, you can predict future points in the series. Personalize create individualized recommendations for customers using their applications e.g. use cases Personalized recommendations Similar items Personalized re-ranking i.e. re-rank a list of items for a user Personalized promotions/notifications Lex conversational interfaces into any application using voice and text. Amazon Lex provides the advanced deep learning functionalities of automatic speech recognition (ASR) for converting speech to text, and natural language understanding (NLU) to recognize the intent of the text chat bots Kendra intelligent search service (ML powered) concepts - index, documents (html, pdf, word, ppt, txt), data sources (S3, confluence, OneDrive, etc.), query point indexer to files in S3 pre-built faceted search UI component (web based) supports custom data sources . e.g. salesforce attachments data source can create custom document attributes developer and enterprise editions Security, Identity, and Compliance IAM terms - Resources, Identities, Entities, Principals (person or application), Actions authentication, authorization actions and operations on resources policy docs - AWS managed policies, Customer managed policies, Inline policies Policy Types Identity-based (e.g. users, groups, roles) Resource-based policies (e.g. bucket policy, IAM role trust policy, lambda permissions, sns topic policy, sqs queue policy, kms key policy, vpc endpoint policy) Permissions boundaries Organizations SCPs Session policies - via assume role or federated user. Session policies limit permissions for a created session, but do not grant permissions. STS - temp security credentials assume role identity providers/federation - Federated users and roles (via OIDC, SAML2) Attribute-based access control (ABAC) - defines permissions based on attributes (tags) permission boundaries sigv4 requests account root user MFA IAM Access Analyzer - validate policies, generate policies (based on CT logs, role or user, and timeframe) Cognito UserPool IdentityPool - exchange UserPool.Identity for temporary IAM credentials unauthenticated and authenticated roles Built-in IdP Providers - amazon, google, twitter, facebook. Federation - OIDC, SAML API Gateway authorizer provided login UIs Secrets Manager WAF web application firewall associate with followng resource types: ALB, CloudFront Distribution, API Gateway REST API, AppSync GraphQL API concepts Web ACLs - contain rules and/or rule groups Rules - contains a statement that defines the inspection criteria, and an action to take if a web request meets the criteria Rules groups - group of rules for easier management AWS::SecretsManager::SecretTargetAttachment - link between a Secrets Manager secret and the associated database by adding the database connection information to the secret JSON Certificate Manager (ACM) provision, manage, and deploy public and private Secure Sockets Layer/Transport Layer Security (SSL/TLS) certificates for use with AWS services and your internal connected resources. KMS key management service - AWS KMS concepts symmetric, asymmetric (pki) kye types: Customer managed key (CMK), AWS managed key, AWS owned key key policy determines principals that can use (identity and service) envelope encryption - key to gen data key, data key used to encrypt plaintext, key to encrypt data key, encrypted data + encrypted data key stored aws encryption sdk and cli Directory Service provides multiple ways to set up and run Amazon Cloud Directory, Amazon Cognito, and Microsoft AD with other AWS services. Amazon Cloud Directory provides a highly scalable directory store for your application’s multihierarchical data. Amazon Cognito helps you create a directory store that authenticates your users either through your own user pools or through federated identity providers. AWS Directory Service for Microsoft Active Directory (Enterprise Edition), also known as Microsoft AD, enables your directory-aware workloads and AWS resources to use a managed Active Directory in the AWS Cloud. Media Services Amazon Interactive Video Service Amazon Interactive Video Service (Amazon IVS) is a managed live streaming solution that is quick and easy to set up, and ideal for creating interactive video experiences. Send your live streams to Amazon IVS using standard streaming software like Open Broadcaster Software (OBS) and the service does everything you need to make low-latency live video available to any viewer around the world, letting you focus on building interactive experiences alongside the live video. --- ### HTTP Live Streaming HLS - URL: https://brianpfeil.com/post/http-live-streaming-hls/ - Date: 2020-03-23 - Tags: streaming, http code for article  pfeilbr/http-live-streaming-hls-playground example of creating an HTTP Live Stream video stream from an h.264 video file. Prerequisites download and install “HTTP Live Streaming Tools” from https://developer.apple.com/download/more/?=HLS (binaries are installed to /usr/local/bin) need to login with apple developer account Running # create segments for HTTP Live Streaming from media file mediafilesegmenter \ -f ./public \ ./assets/video/SampleVideo_1280x720_10mb.mp4 cd public python -m SimpleHTTPServer 8000 open http://localhost:8000/ public/index.html - include js code for video playback assets/video/SampleVideo_1280x720_10mb.mp4 - sample video Resources How can I play a m3u8 (file) video using the HTML5 element? sample-videos.com HTTP Live Streaming --- ### React Async - URL: https://brianpfeil.com/post/react-async/ - Date: 2020-03-19 - Tags: react code for article  pfeilbr/react-async-playground learn React Async , a React component and hook for declarative promise resolution and data fetching see src/App.js This project was bootstrapped with Create React App . Available Scripts In the project directory, you can run: yarn start Runs the app in the development mode. Open http://localhost:3000 to view it in the browser. The page will reload if you make edits. You will also see any lint errors in the console. yarn test Launches the test runner in the interactive watch mode. See the section about running tests for more information. yarn build Builds the app for production to the build folder. It correctly bundles React in production mode and optimizes the build for the best performance. The build is minified and the filenames include the hashes. Your app is ready to be deployed! See the section about deployment for more information. yarn eject Note: this is a one-way operation. Once you eject, you can’t go back! If you aren’t satisfied with the build tool and configuration choices, you can eject at any time. This command will remove the single build dependency from your project. Instead, it will copy all the configuration files and the transitive dependencies (webpack, Babel, ESLint, etc) right into your project so you have full control over them. All of the commands except eject will still work, but they will point to the copied scripts so you can tweak them. At this point you’re on your own. You don’t have to ever use eject. The curated feature set is suitable for small and middle deployments, and you shouldn’t feel obligated to use this feature. However we understand that this tool wouldn’t be useful if you couldn’t customize it when you are ready for it. Learn More You can learn more in the Create React App documentation . To learn React, check out the React documentation . Code Splitting This section has moved here: https://facebook.github.io/create-react-app/docs/code-splitting Analyzing the Bundle Size This section has moved here: https://facebook.github.io/create-react-app/docs/analyzing-the-bundle-size Making a Progressive Web App This section has moved here: https://facebook.github.io/create-react-app/docs/making-a-progressive-web-app Advanced Configuration This section has moved here: https://facebook.github.io/create-react-app/docs/advanced-configuration Deployment This section has moved here: https://facebook.github.io/create-react-app/docs/deployment yarn build fails to minify This section has moved here: https://facebook.github.io/create-react-app/docs/troubleshooting#npm-run-build-fails-to-minify --- ### Azure Pipelines - URL: https://brianpfeil.com/post/azure-pipelines/ - Date: 2020-02-26 - Tags: azure code for article  pfeilbr/azure-pipelines-playground learn azure pipelines azure-pipelines-playground Description Infrastructure Provisioning Steps Website Content Publishing Steps Deprovisioning Architecture Key Files and Directories Screenshots TODO Completed / Cancelled Notes Scratch Description Pipeline performs an atomic deploy of static content from a github repo to a static site (Rout53 + ACM + WAF CloudFront + S3 when) a tag (release) is applied to the repo Infrastructure Provisioning Steps create route 53 hosted zone for your domain name (e.g. mydomain.com) update DOMAIN_NAME parameter in scripts/stack.sh with the hosted zone name provision aws resources ./scripts/stack.sh create Check ACM to confirm Certificate validation via DNS validation has completed. May need to add DNS validation records to route53 hosted zone. update pipeline variables REGION - default is us-east-1 STACK_NAME - defined in scripts/stack.sh AWS_ACCESS_KEY_ID - AccessKey output in ./tmp/${STACK_NAME}-outputs.json AWS_SECRET_ACCESS_KEY - SecretKey output in ./tmp/${STACK_NAME}-outputs.json Website Content Publishing Steps ensure you have develop branch checked out (this corresponds to staging environment) update website content in public directory and push to github. (optional) update redirect rules in routing-rules/routing-rules.txt push your commit(s) to remote (github) publish will run. can take up to 20 minutes to complete due CloudFront distribution update. verify updated content by visiting https://staging.mydomain.com to publish staging to production, checkout master branch and merge in develop push your commit(s) to remote (github) verify updated content by visiting https://mydomain.com and https://www.mydomain.com Deprovisioning deprovision aws resources ./scripts/stack.sh delete (optional) manually delete S3 website and CloudFront logs buckets. these are not deleted because they still contain objects (optional) run ./scripts/stack.sh delete again to permanently delete stack Architecture Key Files and Directories cfn-templates/resources.yaml - CloudFormation stack for provisioning AWS resources. S3 bucket(s) for static content (staging + production) CloudFront distribution(s) (staging + production) lambda@edge function for basic auth for staging site WAF Web ACL for CloudFront distribution(s) aws secrets manager secret (json doc) to store basic auth users S3 bucket for CloudFront access logs SSL Certificate (ACM) route53 root domain ALIAS record to CloudFront distribution route53 staging and www CNAME records to CloudFront distribution IAM user for CI/CD automation used by the azure pipeline public - static web content scripts/stack.sh - provisions AWS resources tmp/automation-outputs.json - stack outputs stored here. file gets created when stack is provisioned. scripts/tag-and-trigger-publish.sh - tags and pushes the tag to github to trigger the publish pipeline scripts/publish.sh - publishes a new version of the static site based on git tag. this is used by pipeline azure-pipelines.yml - pipeline definition that get triggered on tag to publish to site Screenshots Pipeline Variables TODO route53 CNAME record to point directly to S3 bucket website domain. used to troubleshoot/bypass cache issues. e.g. https://bucket.mysite.com -> https://bucket.s3-website-us-east-1.amazonaws.com research basic auth options ok not to have basic auth? Completed / Cancelled add WAF ACL and associate to CF dist(s) see Web ACL created om WAF V2 not accessible on CloudFront add API origin to CF distribution for requests to /api/* path precedence to 1, forward query strings, cookies, all all HTTP methods this is IMPORTANT You can indeed put CF dist in front of APIG, the trick is to force HTTPS only “Viewer Protocol Policy” AND to NOT forward the HOST header because APIG needs SNI. see How do you add CloudFront in front of API Gateway How to use API Gateway with CloudFront add staging CloudFront distribution options separate bucket s3://stage s3://prod single bucket with prefix s3://bucket/stage/* s3://bucket/prod/* update s3 redirect/routing rules for deploy version prefix e.g. domain.com/oldlink would point to /v0.0.1/newlink in the bucket. the /v0.0.1 prefix need to be updated in all redirect rules on deploy see https://docs.aws.amazon.com/cli/latest/reference/s3api/put-bucket-website.html more reliable method of picking the origin path to update since there are two origins create IAM policy and role for resource provisioning look at CloudFormation | Stack | Resources view for resource types specify resource name prefix and suffix as variable to allow for change specify role-arn for cloudformation cli redirects options via lambda@edge check if WAF supports S3 bucket routing rules (AWS::S3::Bucket RoutingRule) s3 object metadata header. see (Optional) Configuring a Webpage Redirect and x-amz-website-redirect-location If the bucket is configured as a website, redirects requests for this object to another object in the same bucket or to an external URL. Amazon S3 stores the value of this header in the object metadata. trailing slashes: see the following on how to handle Re: S3 make a non-trailing slash URL send a 301 instead of a 302 basic auth on staging cloudfront dist options lambda@edge WAF rule for Authorization header update scripts/publish.sh with proper cache control for index.html (no-cache) aws s3 sync --cache-control 'max-age=604800' --exclude index.html build/ s3://mywebsitebucket/ aws s3 sync --cache-control 'no-cache' build/ s3://mywebsitebucket/ deny requests directly to s3. must use domain. remove OAI and add this. this will allows redirects in S3 to work. see How do I use CloudFront to serve a static website hosted on Amazon S3? for details. TLDR; the referer is set on the CloudFront distribution and is a secret. the S3 bucket policy only allows requests from this referer I’m using an S3 REST API endpoint as the origin of my CloudFront distribution. Why am I getting 403 Access Denied errors? I’m using an S3 website endpoint as the origin of my CloudFront distribution. Why am I getting 403 Access Denied errors? { "Version": "2012-10-17", "Id": "http referer policy ${DomainName}", "Statement": [ { "Sid": "Allow get requests referred by ${DomainName}", "Effect": "Allow", "Principal": "*", "Action": "s3:GetObject", "Resource": "arn:aws:s3:::${BUCKET}/*", "Condition": { "StringLike": { "aws:Referer": [ "http://${DomainName}/*", "https://${DomainName}/*" ] } } }, { "Sid": "Explicit deny to ensure requests are allowed only from specific referer.", "Effect": "Deny", "Principal": "*", "Action": "s3:GetObject", "Resource": "arn:aws:s3:::${BUCKET}/*", "Condition": { "StringNotLike": { "aws:Referer": [ "http://${DomainName}/*", "https://${DomainName}/*" ] } } } ] } Notes pipeline is running in azure DevOps tied to personal gmail account Scratch # delete all remote tags git tag -l | xargs -n 1 git push --delete origin # delete all local tags git tag | xargs git tag -d #REDIRECT_LOCATION="https://allthecloudbits.com/products/product02/" REGION="us-east-1" STACK_NAME="dev-agency-website" BUCKET=$(aws cloudformation describe-stacks --region "${REGION}" --stack-name "${STACK_NAME}" --query "Stacks[0].Outputs[?OutputKey=='WebsiteBucketName'].OutputValue" --output text) PREFIX="v0.0.1" TARGET="${PREFIX}/about" REDIRECT_LOCATION="/about/" aws --profile automation-user s3api put-object \ --bucket "${BUCKET}" \ --key "${TARGET}" \ --website-redirect-location "${REDIRECT_LOCATION}" \ --content-length "0" aws --profile automation-user s3api head-object \ --bucket "${BUCKET}" \ --key "${TARGET}" aws --profile automation-user s3api delete-object \ --bucket "${BUCKET}" \ --key "${TARGET}" aws --profile automation-user s3api list-objects \ --bucket "${BUCKET}" policy components { "Effect": "Allow", "Principal": "arn:aws:iam::529276214230:user/admin", "Action": "s3:*", "Resource": "arn:aws:s3:::dev-agency-website-s3bucketforwebsitecontent-11u56g1n9u9oo/*", }, { "Effect": "Allow", "Principal": "*", "Action": "s3:GetObject", "Resource": "arn:aws:s3:::dev-agency-website-s3bucketforwebsitecontent-11u56g1n9u9oo/*", "Condition": { "StringLike": { "aws:Referer": "79011a81-c048-4877-84f4-efe9577d7250" } } }, { "Effect": "Deny", "Principal": "*", "Action": "s3:GetObject", "Resource": "arn:aws:s3:::dev-agency-website-s3bucketforwebsitecontent-11u56g1n9u9oo/*", "Condition": { "StringNotLike": { "aws:Referer": "79011a81-c048-4877-84f4-efe9577d7250" } } } create_routing_rule() { bucket=$1 prefix=$2 target=$3 redirect_location=$4 aws s3api put-object \ --bucket "${bucket}" \ --key "${prefix}${target}" \ --website-redirect-location "${redirect_location}" \ --content-length "0" } create_routing_rules() { bucket=$1 prefix=$2 IFS=$'\r\n' GLOBIGNORE='*' rules=($(cat routing-rules/routing-rules.txt)) for rule in "${rules[@]}" do components=($(echo $rule | tr " " "\r\n")) target="${components[1]}" redirect_location="${components[2]}" # echo "target=${target}, redirect_location=${redirect_location}" create_routing_rule "${bucket}" "${prefix}" "${target}" "${redirect_location}" done } create_routing_rules "dev-agency-website-s3bucketforwebsitecontent-1fbv8htrn7nna" "v0.0.1" # on tag BUILD_SOURCEBRANCHNAME=v0.0.1 BUILD_SOURCEBRANCH=refs/tags/v0.0.1 BUILD_SOURCEVERSION=f302ed7e007e57c118a8835f378ddd04f63e105c BUILD_SOURCEVERSIONMESSAGE=output env variables # on develop branch BUILD_SOURCEBRANCHNAME=develop BUILD_SOURCEBRANCH=refs/heads/develop BUILD_SOURCEVERSION=e59a3dfea5bd88be3808f46b48da7ddd83e8b809 BUILD_SOURCEVERSIONMESSAGE=output env variables # pipeline environment variables LEIN_HOME=/usr/local/lib/lein BUILD_QUEUEDBY=GitHub AGENT_HOMEDIRECTORY=/home/vsts/agents/2.165.0 M2_HOME=/usr/share/apache-maven-3.6.3 BOOST_ROOT=/usr/local/share/boost/1.69.0 SYSTEM_STAGEDISPLAYNAME=__default AGENT_VERSION=2.165.0 SYSTEM_JOBATTEMPT=1 GOROOT_1_11_X64=/usr/local/go1.11 ANDROID_HOME=/usr/local/lib/android/sdk JAVA_HOME_11_X64=/usr/lib/jvm/zulu-11-azure-amd64 SYSTEM_TEAMFOUNDATIONSERVERURI=https://dev.azure.com/brianpfeil/ ImageVersion=20200301.1 AGENT_TOOLSDIRECTORY=/opt/hostedtoolcache SYSTEM_DEFINITIONID=1 AGENT_DISABLELOGPLUGIN_TESTFILEPUBLISHERPLUGIN=true LANG=C.UTF-8 AGENT_WORKFOLDER=/home/vsts/work AZURE_EXTENSION_DIR=/opt/az/azcliextensions SYSTEM_DEFINITIONNAME=pfeilbr.azure-pipelines-playground AGENT_TEMPDIRECTORY=/home/vsts/work/_temp INVOCATION_ID=71a915879cc04391a20e7d2867d21806 JAVA_HOME_12_X64=/usr/lib/jvm/zulu-12-azure-amd64 AWS_SECRET_ACCESS_KEY=*** BUILD_REQUESTEDFOR=Brian Pfeil SYSTEM_PHASENAME=Job ANDROID_SDK_ROOT=/usr/local/lib/android/sdk SYSTEM_JOBIDENTIFIER=Job.__default SYSTEM_PULLREQUEST_ISFORK=False JAVA_HOME=/usr/lib/jvm/zulu-8-azure-amd64 SYSTEM_JOBPARALLELISMTAG=Private DOTNET_SKIP_FIRST_TIME_EXPERIENCE=1 USER=vsts BUILD_REASON=IndividualCI AGENT_OS=Linux SYSTEM_ISSCHEDULED=False BUILD_SOURCEVERSION=f302ed7e007e57c118a8835f378ddd04f63e105c ENDPOINT_URL_SYSTEMVSSCONNECTION=https://dev.azure.com/brianpfeil/ BUILD_SOURCEBRANCH=refs/tags/v0.0.1 GRADLE_HOME=/usr/share/gradle SYSTEM_WORKFOLDER=/home/vsts/work BUILD_QUEUEDBYID=a399e9b2-ba17-47ed-9005-8c263530afdf AGENT_DISABLELOGPLUGIN_TESTRESULTLOGPLUGIN=true AGENT_ROOTDIRECTORY=/home/vsts/work PWD=/home/vsts/work/1/s ImageOS=ubuntu18 HOME=/home/vsts AGENT_ID=8 GOROOT=/usr/local/go1.12 JOURNAL_STREAM=9:31906 SYSTEM_TOTALJOBSINPHASE=1 SYSTEM_COLLECTIONURI=https://dev.azure.com/brianpfeil/ RUNNER_TOOLSDIRECTORY=/opt/hostedtoolcache JAVA_HOME_8_X64=/usr/lib/jvm/zulu-8-azure-amd64 AGENT_OSARCHITECTURE=X64 SYSTEM_CULTURE=en-US SYSTEM_TEAMPROJECTID=398b5300-9203-4d2c-957c-82eedb6aea92 VSTS_AGENT_PERFLOG=/home/vsts/perflog CONDA=/usr/share/miniconda PIPELINE_WORKSPACE=/home/vsts/work/1 GOROOT_1_13_X64=/usr/local/go1.13 BOOST_ROOT_1_69_0=/usr/local/share/boost/1.69.0 SYSTEM_JOBPOSITIONINPHASE=1 DEBIAN_FRONTEND=noninteractive SYSTEM_JOBID=12f1170f-54f2-53f3-20dd-22fc7dff55f9 AGENT_JOBNAME=Job AGENT_ACCEPTTEEEULA=True SYSTEM_STAGEATTEMPT=1 SYSTEM_PIPELINESTARTTIME=2020-03-05 22:20:12+00:00 STACK_NAME=dev-agency-website BUILD_REPOSITORY_URI=https://github.com/pfeilbr/azure-pipelines-playground AGENT_READONLYVARIABLES=true BUILD_REPOSITORY_PROVIDER=GitHub SYSTEM_TASKINSTANCENAME=CmdLine GOROOT_1_12_X64=/usr/local/go1.12 GECKOWEBDRIVER=/usr/local/share/gecko_driver SYSTEM_PLANID=b5cbb7c3-6a1c-4c2f-85da-7cd89b56bfcc BUILD_DEFINITIONVERSION=13 TASK_DISPLAYNAME=deploy SYSTEM_HOSTTYPE=build CHROMEWEBDRIVER=/usr/local/share/chrome_driver BUILD_STAGINGDIRECTORY=/home/vsts/work/1/a MSDEPLOY_HTTP_USER_AGENT=VSTS_1cc55890-46d6-40c1-9714-8220ddd17bd5_build_1_0 BUILD_REQUESTEDFOREMAIL=brian.pfeil@gmail.com TF_BUILD=True AZURE_HTTP_USER_AGENT=VSTS_1cc55890-46d6-40c1-9714-8220ddd17bd5_build_1_0 BUILD_REPOSITORY_LOCALPATH=/home/vsts/work/1/s SYSTEM_PHASEDISPLAYNAME=Job BUILD_REPOSITORY_NAME=pfeilbr/azure-pipelines-playground BUILD_ARTIFACTSTAGINGDIRECTORY=/home/vsts/work/1/a SYSTEM_TASKDEFINITIONSURI=https://dev.azure.com/brianpfeil/ BUILD_REPOSITORY_GIT_SUBMODULECHECKOUT=False BUILD_SOURCEVERSIONAUTHOR=Brian Pfeil SYSTEM_TASKDISPLAYNAME=deploy SYSTEM_STAGENAME=__default VSTS_PROCESS_LOOKUP_ID=vsts_f104e61b-5db9-493c-8204-4ffeb8973a65 BUILD_REPOSITORY_ID=pfeilbr/azure-pipelines-playground BUILD_SOURCEVERSIONMESSAGE=output env variables SYSTEM=build SYSTEM_PHASEATTEMPT=1 VCPKG_INSTALLATION_ROOT=/usr/local/share/vcpkg SYSTEM_JOBNAME=__default REGION=us-east-1 SYSTEM_PHASEID=3a3a2a60-14c7-570b-14a4-fa42ad92f52a AGENT_MACHINENAME=fv-az755 COMMON_TESTRESULTSDIRECTORY=/home/vsts/work/1/TestResults agent.jobstatus=Succeeded JAVA_HOME_7_X64=/usr/lib/jvm/zulu-7-azure-amd64 BUILD_DEFINITIONNAME=pfeilbr.azure-pipelines-playground AWS_ACCESS_KEY_ID=AKIAXWO2SDPLDE43XM4W SYSTEM_ARTIFACTSDIRECTORY=/home/vsts/work/1/a SHLVL=2 BUILD_REQUESTEDFORID=5d5ba760-cdcf-6c15-8d80-c37de4cf7631 AGENT_NAME=Hosted Agent BUILD_BUILDNUMBER=20200305.5 BUILD_SOURCEBRANCHNAME=v0.0.1 SYSTEM_JOBDISPLAYNAME=Job BUILD_SOURCESDIRECTORY=/home/vsts/work/1/s LEIN_JAR=/usr/local/lib/lein/self-installs/leiningen-2.9.2-standalone.jar BUILD_BUILDURI=vstfs:///Build/Build/53 SYSTEM_SERVERTYPE=Hosted SYSTEM_TIMELINEID=b5cbb7c3-6a1c-4c2f-85da-7cd89b56bfcc AGENT_RETAINDEFAULTENCODING=false SYSTEM_TASKINSTANCEID=9c939e41-62c2-5605-5e05-fc3554afc9f5 SYSTEM_DEFAULTWORKINGDIRECTORY=/home/vsts/work/1/s AGENT_JOBSTATUS=Succeeded ANT_HOME=/usr/share/ant BUILD_REPOSITORY_CLEAN=False SYSTEM_TEAMPROJECT=project01 PATH=/usr/share/rust/.cargo/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin SELENIUM_JAR_PATH=/usr/share/java/selenium-server-standalone.jar CHROME_BIN=/usr/bin/google-chrome SYSTEM_COLLECTIONID=1cc55890-46d6-40c1-9714-8220ddd17bd5 BUILD_CONTAINERID=5246219 SYSTEM_TEAMFOUNDATIONCOLLECTIONURI=https://dev.azure.com/brianpfeil/ GIT_TERMINAL_PROMPT=0 AGENT_BUILDDIRECTORY=/home/vsts/work/1 SYSTEM_STAGEID=96ac2280-8cb4-5df5-99de-dd2da759617d SYSTEM_ENABLEACCESSTOKEN=SecretVariable BUILD_BINARIESDIRECTORY=/home/vsts/work/1/b BUILD_BUILDID=53 ${BUILD_REPOSITORY_URI}/tree/${BUILD_SOURCEBRANCHNAME} ${BUILD_REPOSITORY_URI}/commit/${BUILD_SOURCEVERSION} # athena create table for CloudFront logs CREATE EXTERNAL TABLE IF NOT EXISTS default.cloudfront_logs_stagingallthecloudbits ( `date` DATE, time STRING, location STRING, bytes BIGINT, request_ip STRING, method STRING, host STRING, uri STRING, status INT, referrer STRING, user_agent STRING, query_string STRING, cookie STRING, result_type STRING, request_id STRING, host_header STRING, request_protocol STRING, request_bytes BIGINT, time_taken FLOAT, xforwarded_for STRING, ssl_protocol STRING, ssl_cipher STRING, response_result_type STRING, http_version STRING, fle_status STRING, fle_encrypted_fields INT, c_port INT, time_to_first_byte FLOAT, x_edge_detailed_result_type STRING, sc_content_type STRING, sc_content_len BIGINT, sc_range_start BIGINT, sc_range_end BIGINT ) ROW FORMAT DELIMITED FIELDS TERMINATED BY '\t' LOCATION 's3://dev-agency-website-cloudfrontlogsbucket-jf9ykpajh7n1/cloudfront/logs/staging.allthecloudbits.com/' TBLPROPERTIES ( 'skip.header.line.count'='2' ) # query SELECT * FROM "default"."cloudfront_logs_stagingallthecloudbits" limit 10 # CF dist WAF association myDistribution Type AWS::CloudFront::Distribution Properties: DistributionConfig: WebACLId: !Ref : MyWebACL # aws managed rule groups https://docs.aws.amazon.com/waf/latest/developerguide/aws-managed-rule-groups-list.html AWS::WAFv2::WebACL Type: AWS::WAFv2::WebACL Properties: Name: waf-webacl Scope: CLOUDFRONT Description: CloudFront WAF WebACL DefaultAction: Allow: {} VisibilityConfig: SampledRequestsEnabled: true CloudWatchMetricsEnabled: true MetricName: ExampleWebACLMetric Rules: - Name: RuleWithAWSManagedRules Priority: 0 OverrideAction: Count: {} VisibilityConfig: SampledRequestsEnabled: true CloudWatchMetricsEnabled: true MetricName: RuleWithAWSManagedRulesMetric Statement: ManagedRuleGroupStatement: VendorName: AWS Name: AWSManagedRulesCommonRuleSet ExcludedRules: [] --- ### Flask - URL: https://brianpfeil.com/post/flask/ - Date: 2020-02-15 - Tags: flask code for article  pfeilbr/flask-playground learn flask , the python web application framework server.py - flask server client.py - http client. uses requests to POST the image.jpeg image file load-test.sh - load test via parallel requests Prerequisites pipenv for python environment/packages Running # install all dependencies pipenv install --dev # load python enviroment pipenv shell # run server in debug (livereload - reload on file changes) mode FLASK_DEBUG=1 FLASK_RUN_PORT=5000 FLASK_APP=server.py flask run # run client python client.py # load test chmod +x load-test.sh ./load-test.sh --- ### Bash Script Date and Time - URL: https://brianpfeil.com/post/bash-script-date-and-time/ - Date: 2020-02-10 - Tags: bash code for article  pfeilbr/bash-script-date-and-time-playground examples of working with formatted dates in bash including time operations such as adding time see main.sh Running chmod a+x main.sh fswatch -o main.sh | xargs -n1 -I{} sh main.sh --- ### Multipass - URL: https://brianpfeil.com/post/multipass/ - Date: 2020-02-09 - Tags: linux, virtualization code for article  pfeilbr/multipass-playground learn multipass - command line interface to launch, manage and generally fiddle about with instances of Linux multipass | docs Working with Multipass instances Session # install brew cask install multipass # set to use virtualbox instead of hyperkit, which is the default on macOS (ran into issues with hyperkit) sudo multipass set local.driver=virtualbox # launch default ubuntu instance multipass launch # set the primary instance. if you don't specify the instance a command, this is the instance used multipass set client.primary-name=capital-tapir # list instances multipass ls # open shell (uses ssh under the covers) multipass shell # execute command and exit multipass exec capital-tapir -- ls # mount local host folder into instance # /Users/pfeilbr/tmp (host) => /Users/pfeilbr/tmp (instance) multipass mount ~/tmp capital-tapir # unmount mounts multipass unmount capital-tapir # list info about instance multipass info capital-tapir multipass suspend capital-tapir multipass restart capital-tapir multipass stop capital-tapir multipass delete capital-tapir # purge all deleted instances permanently multipass purge --- ### Python Async Await - URL: https://brianpfeil.com/post/python-async-await/ - Date: 2020-02-05 - Tags: python code for article  pfeilbr/python-async-await-playground learn python async await Prerequisites Pipenv Running git clone pipenv install pipenv run python main.py --- ### TailwindCSS - URL: https://brianpfeil.com/post/tailwindcss/ - Date: 2020-01-16 - Tags: css, framework code for article  pfeilbr/tailwindcss-playground learn tailwindcss Resources https://dev.to/hagnerd/setting-up-tailwind-with-create-react-app-4jd create-react-app README.md below This project was bootstrapped with Create React App . Available Scripts In the project directory, you can run: yarn start Runs the app in the development mode. Open http://localhost:3000 to view it in the browser. The page will reload if you make edits. You will also see any lint errors in the console. yarn test Launches the test runner in the interactive watch mode. See the section about running tests for more information. yarn build Builds the app for production to the build folder. It correctly bundles React in production mode and optimizes the build for the best performance. The build is minified and the filenames include the hashes. Your app is ready to be deployed! See the section about deployment for more information. yarn eject Note: this is a one-way operation. Once you eject, you can’t go back! If you aren’t satisfied with the build tool and configuration choices, you can eject at any time. This command will remove the single build dependency from your project. Instead, it will copy all the configuration files and the transitive dependencies (Webpack, Babel, ESLint, etc) right into your project so you have full control over them. All of the commands except eject will still work, but they will point to the copied scripts so you can tweak them. At this point you’re on your own. You don’t have to ever use eject. The curated feature set is suitable for small and middle deployments, and you shouldn’t feel obligated to use this feature. However we understand that this tool wouldn’t be useful if you couldn’t customize it when you are ready for it. Learn More You can learn more in the Create React App documentation . To learn React, check out the React documentation . Code Splitting This section has moved here: https://facebook.github.io/create-react-app/docs/code-splitting Analyzing the Bundle Size This section has moved here: https://facebook.github.io/create-react-app/docs/analyzing-the-bundle-size Making a Progressive Web App This section has moved here: https://facebook.github.io/create-react-app/docs/making-a-progressive-web-app Advanced Configuration This section has moved here: https://facebook.github.io/create-react-app/docs/advanced-configuration Deployment This section has moved here: https://facebook.github.io/create-react-app/docs/deployment yarn build fails to minify This section has moved here: https://facebook.github.io/create-react-app/docs/troubleshooting#npm-run-build-fails-to-minify --- ### Go Modules - URL: https://brianpfeil.com/post/go-modules/ - Date: 2020-01-15 - Tags: golang code for article  pfeilbr/go-modules-playground learn using golang modules Resources https://blog.golang.org/using-go-modules golang/go/wiki/Modules Session # run tests recursively on change fswatch -o . | xargs -n1 -I{} go test -v ./... # run main on change fswatch -o . | xargs -n1 -I{} go run main.go --- ### Go SDL2 - URL: https://brianpfeil.com/post/go-sdl2/ - Date: 2020-01-13 - Tags: golang code for article  pfeilbr/go-sdl2-playground learn go-sdl2 , the SDL2 wrapped for Go users Resources veandco/go-sdl2-examples --- ### Make - URL: https://brianpfeil.com/post/make/ - Date: 2020-01-13 - Tags: make code for article  pfeilbr/make-playground learn make Resources GNU make manual The GNU Make Book purpose of .PHONY # run `make` whenever Makefile changes fswatch -o Makefile | xargs -n1 -I{} make --- ### Googletest - URL: https://brianpfeil.com/post/googletest/ - Date: 2020-01-02 - Tags: testing, cpp code for article  pfeilbr/googletest-playground learn Googletest , Google Testing and Mocking Framework with CMake based on https://raymii.org/s/tutorials/Cpp_project_setup_with_cmake_and_unit_tests.html # begin: one-time setup to add googletest dependency mkdir lib pushd lib git clone https://github.com/google/googletest/ popd # end: one-time setup #clean ./run.sh clean # build ./run.sh buildandtest # run tests ./run.sh test # build and run tests when file changes under src/ and tst/ ./run.sh watch # build main binary ./run.sh buildmain # run main binary ./run.sh main googletest command line examples # run all tests ./build/tst/app_tst # list tests ./build/tst/app_tst --gtest_list_tests # run specific test ./build/tst/app_tst --gtest_filter=MyLib.adhoc # run all tests in MyLib ./build/tst/app_tst --gtest_filter='MyLib.*' # list command line options ./build/tst/app_tst --help --- ### React Bootstrap - URL: https://brianpfeil.com/post/react-bootstrap/ - Date: 2019-12-29 - Tags: react code for article  pfeilbr/react-bootstrap-playground This project was bootstrapped with Create React App . Available Scripts In the project directory, you can run: yarn start Runs the app in the development mode. Open http://localhost:3000 to view it in the browser. The page will reload if you make edits. You will also see any lint errors in the console. yarn test Launches the test runner in the interactive watch mode. See the section about running tests for more information. yarn build Builds the app for production to the build folder. It correctly bundles React in production mode and optimizes the build for the best performance. The build is minified and the filenames include the hashes. Your app is ready to be deployed! See the section about deployment for more information. yarn eject Note: this is a one-way operation. Once you eject, you can’t go back! If you aren’t satisfied with the build tool and configuration choices, you can eject at any time. This command will remove the single build dependency from your project. Instead, it will copy all the configuration files and the transitive dependencies (Webpack, Babel, ESLint, etc) right into your project so you have full control over them. All of the commands except eject will still work, but they will point to the copied scripts so you can tweak them. At this point you’re on your own. You don’t have to ever use eject. The curated feature set is suitable for small and middle deployments, and you shouldn’t feel obligated to use this feature. However we understand that this tool wouldn’t be useful if you couldn’t customize it when you are ready for it. Learn More You can learn more in the Create React App documentation . To learn React, check out the React documentation . Code Splitting This section has moved here: https://facebook.github.io/create-react-app/docs/code-splitting Analyzing the Bundle Size This section has moved here: https://facebook.github.io/create-react-app/docs/analyzing-the-bundle-size Making a Progressive Web App This section has moved here: https://facebook.github.io/create-react-app/docs/making-a-progressive-web-app Advanced Configuration This section has moved here: https://facebook.github.io/create-react-app/docs/advanced-configuration Deployment This section has moved here: https://facebook.github.io/create-react-app/docs/deployment yarn build fails to minify This section has moved here: https://facebook.github.io/create-react-app/docs/troubleshooting#npm-run-build-fails-to-minify --- ### GLib - URL: https://brianpfeil.com/post/glib/ - Date: 2019-11-11 - Tags: graphics, cpp code for article  pfeilbr/glib-playground learn GNOME | GLib see main.c Prerequisites brew install cmake brew install glib brew install pkg-config Build and Run mkdir -p build && cd build cmake .. && make # run ./glib-playground output Resources Manage C data using the GLib collections GNOME | GLib GNOME/glib repo --- ### Terraform - URL: https://brianpfeil.com/post/terraform/ - Date: 2019-11-05 - Tags: infrastructure-as-code, aws code for article  pfeilbr/terraform-playground learn and experiment with Terraform followed Packer GETTING STARTED docs Notes Execution Plans - describing what it will do and asks for your approval before making any infrastructure changes. This allows you to review changes before Terraform creates, updates, or destroys infrastructure. Resource Graph - Terraform builds a resource graph and creates or modifies non-dependent resources in parallel. Module Demo see mymodule01 cd mymodule01/examples/app01 terraform init terraform plan terraform apply -auto-approve Resources represent infrastructure objects each resource type is implemented by a provider, which is a plugin for Terraform that offers a collection of resource types Meta-Arguments depends_on count - creates that many instances of the resource or module for_each - value is a map or a set of strings, Terraform will create one instance for each member of that map or set provider - use to override tf default provider. e.g. you want to change the region for a given resource lifecycle provisioner and connection Syntax resource TYPE NAME { BODY } Data Sources Terraform Language use hcl or json to declare Blocks - containers for other content and usually represent the configuration of some kind of object Arguments - assign a value to a name Expressions - represent a value, either literally or by referencing and combining other values State by default state is stored locally in *.tfstate file (the default backend) when multiple people or clients are collaborating on infra via tf, state can be stored in a remote backend e.g. state for aws can be stored in S3 and a dynamodb table is used for locking when the state is shared amongst many people and/or systems terraform { backend "s3" { # Replace this with your bucket name! bucket = "terraform-up-and-running-state" key = "global/s3/terraform.tfstate" region = "us-east-2" # Replace this with your DynamoDB table name! dynamodb_table = "terraform-up-and-running-locks" encrypt = true } } Workspaces - allow you to store your Terraform state in multiple, separate, named workspaces CLI # plan terraform plan # apply terraform apply -auto-approve # validate terraform validate # show outputs terraform output Resources Terraform Language Documentation Terraform | AWS Provider Docs Terraform Registry --- ### Serverless Lambda Java - URL: https://brianpfeil.com/post/serverless-lambda-java/ - Date: 2019-10-27 - Tags: serverless, lambda, java code for article  pfeilbr/serverless-lambda-java-playground serverless framework + lambda java Build and Deploy cd ~/projects/serverless-lambda-java-playground serverless create --template aws-java-maven mvn clean install sls deploy --verbose sls invoke --function hello --data '{"msg": "hello"}' --log { "statusCode": 200, "body": "{\"message\":\"Go Serverless v1.x! Your function executed successfully!\",\"input\":{}}", "headers": { "X-Powered-By": "AWS Lambda & serverless" }, "isBase64Encoded": false } Resources How to create a REST API in Java using DynamoDB and Serverless --- ### Wordpress - URL: https://brianpfeil.com/post/wordpress/ - Date: 2019-10-22 - Tags: wordpress code for article  pfeilbr/wordpress-playground learn and experiment with all things WordPress and WordPress VIP (aka Enterprise WordPress) Running using Docker Compose # ensure docker is up and running # start docker-compose up -d # admin ui (login: admin/password01) open http://localhost:8000/wp-admin # site open http://localhost:8000 # root rest api routes curl http://localhost:8000/index.php?rest_route=/ | jq '.' # rest api routes curl http://localhost:8000/\?rest_route\=/wp/v2 | jq '.' # rest api posts example curl http://localhost:8000/?rest_route=/wp/v2/posts | jq '.' # stop docker-compose down Extending WP REST API | Adding Custom Endpoints | Custom Plugin Example based on Adding Custom Endpoints create wp-root/wp-content/plugins/myplugin.php if needed, copy to “live” plugins directory cp wp-root/wp-content/plugins/myplugin.php html/wp-content/plugins/myplugin.php activate via Admin | Plugins access custom endpoint http://localhost:8000/index.php?rest_route=/myplugin/v1/author/1 Running using VVV (Varying Vagrant Vagrants) Setup install System Requirements execute Installation steps Running root directory path ~/vagrant-local vagrant up add VirtualBox port mappings 0.0.0.0:80 -> 80 update /etc/hosts to point to 127.0.0.1 sudo sed -i.bu 's/192.168.50.4/127.0.0.1/g' /etc/hosts visit http://vvv.test Shouldn’t need to do the /etc/hosts and VirtualBox port mapping steps. This is a workaround to deal with an iss where macOS ignores the entries added by the vagrant-hostsupdater plugin in /etc/hosts file. stopping vagrant halt this removes /etc/hosts entries Changes Allow logging in to a WP VIP site (e.g. http://WP-VIP-SITE/wp-admin ) with admin / password. changed wpcom_vip_is_restricted_username @ WP-VIP-SITE/public_html/wp-content/mu-plugins/security.php to function wpcom_vip_is_restricted_username( $username ) { // return 'admin' === $username // || WPCOM_VIP_MACHINE_USER_LOGIN === $username // || WPCOM_VIP_MACHINE_USER_EMAIL === $username; return false; } Remove VIP_MAINTENANCE_MODE not defined error message showing in wp-admin and site UIs added the following to WP-VIP-SITE/public_html/wp-content/client-mu-plugins/plugin-loader.php to get rid of VIP_MAINTENANCE_MODE not defined error message showing in wp-admin and site UIs. define( 'VIP_MAINTENANCE_MODE', false ); Resources Quickstart: Compose and WordPress WordPress VIP Documentation WordPress | REST API Handbook Block Editor Handbook WordPress/gutenberg packages/block-library - Block library for the WordPress editor. WordPress Storybook site developer.wordpress.org codex.wordpress.org Adding Custom Endpoints WPGraphQL - graphql API for WP Automattic/vip-go-mu-plugins - The development repo for mu-plugins used on the VIP Go platform. Automattic/vip-go-mu-plugins-built - The generated repo for mu-plugins used on the VIP Go platform Theme Handbook / Advanced Theme Topics / Child Themes --- ### AWS Control Tower - URL: https://brianpfeil.com/post/aws-control-tower/ - Date: 2019-10-17 - Tags: aws code for article  pfeilbr/aws-control-tower-playground learn AWS Control Tower AWS Control Tower provides the easiest way to set up and govern a secure, compliant, multi-account AWS environment based on best practices established by working with thousands of enterprises. With AWS Control Tower, end users on your distributed teams can provision new AWS accounts quickly. Meanwhile your central cloud administrators will know that all accounts are aligned with centrally established, company-wide compliance policies. Notes Control Tower is the composition of many AWS services AWS SSO - integrated with Microsoft AD on-prem and cloud/azure Organizations Service Control Policies - central control over the maximum available permissions for all accounts in your organization, allowing you to ensure your accounts stay within your organization’s access control guidelines Guardrails - two kinds of guardrails exist: preventive (block) and detective (after the fact notification of non-compliance) implemented as AWS Config - monitor for compliance Service Catalog - self-service provisioning of cloud products AWS landing zone is a solution that helps customers more quickly set up a secure, multi-account AWS environment based on AWS best practices. Resources AWS Control Tower Documentation AWS re:Inforce 2019: Using AWS Control Tower to Govern Multi-Account AWS Environments (GRC313-R) (video) Using AWS Control Tower to govern multi-account AWS environments at scale - GRC313-R - AWS re:Inforce 2019 (slides) AWS Control Tower is now generally available --- ### AWS Serverless Application Repository - URL: https://brianpfeil.com/post/aws-serverless-application-repository/ - Date: 2019-10-14 - Tags: aws, serverless code for article  pfeilbr/aws-serverless-application-repository-playground learn AWS Serverless Application Repository Example using SAM CLI # define deployment bucket BUCKET="sam-deploy-bucket-01" # init sam app sam init --runtime nodejs cd sam-app # " create a Lambda deployment package". # add Metadata section to `template.yml`. see https://docs.aws.amazon.com/en_pv/serverlessrepo/latest/devguide/serverlessrepo-quick-start.html#serverlessrepo-quick-start-hello-world-package-app # package sam package \ --template-file template.yaml \ --output-template-file packaged.yaml \ --s3-bucket $BUCKET # publish to SAR. will be private by default sam publish \ --template packaged.yaml \ --region us-east-1 # create SAM app to consume SAR (embedded SAR) # embed SAR (`Type: AWS::Serverless::Application`) in template. see https://docs.aws.amazon.com/en_pv/serverless-application-model/latest/developerguide/serverless-sam-template.html#serverless-sam-template-application touch embed-serverless-application.yaml # package embedded app sam package \ --template-file embed-serverless-application.yaml \ --output-template-file embed-serverless-application-packaged.yaml \ --s3-bucket $BUCKET # define name for stack STACK_NAME="embed-serverless-application" # deploy embedded app. note usage of `CAPABILITY_AUTO_EXPAND` param sam deploy --template-file ./embed-serverless-application-packaged.yaml --stack-name "$STACK_NAME" --capabilities CAPABILITY_IAM CAPABILITY_AUTO_EXPAND # two stacks are created. the parent (sam-app/embed-serverless-application-packaged.yaml) and the emb embedded SAR (packaged.yaml) # cleanup / remove stack(s) aws cloudformation delete-stack --stack-name "$STACK_NAME" SAR App in Console Nested CFN Stacks Example using aws serverlessrepo CLI can also be done (and is preferred) via sam cli. see above. mkdir sar-cli-example cd sar-cli-example APP_NAME=myapp01 aws serverlessrepo list-applications # create appliction aws serverlessrepo create-application \ --author 'Brian Pfeil' \ --description $APP_NAME \ --name $APP_NAME \ --semantic-version 0.0.1 \ --template-body file://./s3-bucket-template.yaml # fetch ApplicationId APP_ID=$(aws serverlessrepo list-applications | jq --raw-output ".Applications[] | select(.Name == \"$APP_NAME\").ApplicationId") # create request to create cfn template (async operation) TEMPLATE_ID=$(aws serverlessrepo create-cloud-formation-template --application-id "$APP_ID" | jq --raw-output ".TemplateId") # fetch the cfn template TEMPLATE_URL=$(aws serverlessrepo get-cloud-formation-template --application-id "$APP_ID" --template-id "$TEMPLATE_ID" | jq --raw-output ".TemplateUrl") # view cfn template curl "$TEMPLATE_URL" # delete app aws serverlessrepo delete-application --application-id "$APP_ID" --- ### Stackery - URL: https://brianpfeil.com/post/stackery/ - Date: 2019-10-09 - Tags: infrastructure-as-code, aws code for article  pfeilbr/stackery-playground learn stackery . Project based on Stackery Quickstart NodeJS . Comments (as of 2019-10-09) competes with serverless dashboard . many of the same features. secrets service, environment management, team/collaboration support says it supports serverless framework (serverless.yml), but SAM seems to be the first class citizen. SAM since its based on CF, lags behind in feature support over serverless. Serverless fills the gaps with custom plugins. installs a stack (roles, buckets, CodeBuild project, lambdas, SNS topics, etc.) into your account to link your aws account to the stackery SaaS service. the stackery cli feels heavy due to the number of flags and arguments you need to supply. e.g. always specifying env and aws profile. documentation feels a bit all over the place. this may be because the tool doesn’t have strong opinions and is favoring flexibility. Prerequisites docker Session # ensure docker is running # install brew tap stackery/tap brew install stackery-cli stackery login # init stackery init -n stackery-quickstart # visual editor stackery edit # deploy stackery deploy --interactive-setup # cd to function directory cd src/get # invoke stackery local invoke -e pfeilbr-development --aws-profile admin Stackery editor that generates cloudformation (template.yml) or serverless framework (serverless.yml). There is a “link your AWS account with Stackery” one time setup. Creates CF Stack for initial setup. Includes Roles, Bucket, CodeBuild, Lambdas, SNS. Roles created during one time setup Resources Stackery Quickstart NodeJS Stackery Documentation --- ### AWS Cognito Developer Authenticated Identities - URL: https://brianpfeil.com/post/aws-cognito-developer-authenticated-identities/ - Date: 2019-09-26 - Tags: aws, cognito code for article  pfeilbr/aws-cognito-developer-authenticated-identities-playground learn Developer Authenticated Identities (Identity Pools) With developer authenticated identities, you can register and authenticate users via your own existing authentication process, while still using Amazon Cognito to synchronize user data and access AWS resources. Example using AWS CLI Add custom authentication provider (DEVELOPER_PROVIDER_NAME) to your identity pool via “Edit identity pool” UI setup shell variables IDENTITY_POOL_ID="us-east-1:335c1f44-87c9-4bbd-a314-93b47d91fadd" DEVELOPER_PROVIDER_NAME=com.brianpfeil.app01 # this is YOUR applications userid DEVELOPER_PROVIDER_USERID=003 create identity aws cognito-identity get-open-id-token-for-developer-identity --identity-pool-id $IDENTITY_POOL_ID --logins "$DEVELOPER_PROVIDER_NAME=$DEVELOPER_PROVIDER_USERID" example output { "Token": "eyJraWQiOiJ1cy1lYXN0LTExIiwidHlwIjoiSldTIiwiYWxnIjoiUlM1MTIifQ.eyJzdWIiOiJ1cy1lYXN0LTE6YWVjYjU4YTgtMjc3Ni00NDMxLTk3OGMtZDYzOTVlMWI1Mzc5IiwiYXVkIjoidXMtZWFzdC0xOjMzNWMxZjQ0LTg3YzktNGJiZC1hMzE0LTkzYjQ3ZDkxZmFkZCIsImFtciI6WyJhdXRoZW50aWNhdGVkIiwiY29tLmJyaWFucGZlaWwuYXBwMDEiLCJjb20uYnJpYW5wZmVpbC5hcHAwMTp1cy1lYXN0LTE6MzM1YzFmNDQtODdjOS00YmJkLWEzMTQtOTNiNDdkOTFmYWRkOjAwMyJdLCJpc3MiOiJodHRwczovL2NvZ25pdG8taWRlbnRpdHkuYW1hem9uYXdzLmNvbSIsImV4cCI6MTU2OTUyMzg3NSwiaWF0IjoxNTY5NTIyOTc1fQ.F6ST-AQWETHUAAR7JM-IU1ZIFLDEVL9ZBNM49WDOL_RXLLCYEY2KYUICHSGYLERD4WWLHWEJG-AOHFMMS0DUXT-UANA3BENUFFZWWSBAYVD0N2BHCLHZG7PURTRKRDN2XRFGDGQQ2PIMURMWAIPSB0ZCM-EXMSV-QAGOGKE5C2QR0P91BICL_LB1OQRTF9VXANPEMFFSAMZED776WHKR8ZMP7NTXZBMRE453QFW7VGVNKV3KJDTAKSRVZJS6YVW7BXY74_OQUJCFF9KWXJSMTEBNOIMHEFI3LJ25HSDDJ4LMLBGODD_ET4PPSUORIVLGW4UQ-7PJYHCAYTBDV0MXAQ", "IdentityId": "us-east-1:aecb58a8-2776-4431-978c-d6395e1b5379" } get token aws cognito-identity get-open-id-token-for-developer-identity --identity-pool-id $IDENTITY_POOL_ID --identity-id $IDENTITY_ID --logins "$DEVELOPER_PROVIDER_NAME=$DEVELOPER_PROVIDER_USERID" example output { "Token": "eyJraWQiOiJ1cy1lYXN0LTExIiwidHlwIjoiSldTIiwiYWxnIjoiUlM1MTIifQ.eyJzdWIiOiJ1cy1lYXN0LTE6YWVjYjU4YTgtMjc3Ni00NDMxLTk3OGMtZDYzOTVlMWI1Mzc5IiwiYXVkIjoidXMtZWFzdC0xOjMzNWMxZjQ0LTg3YzktNGJiZC1hMzE0LTkzYjQ3ZDkxZmFkZCIsImFtciI6WyJhdXRoZW50aWNhdGVkIiwiY29tLmJyaWFucGZlaWwuYXBwMDEiLCJjb20uYnJpYW5wZmVpbC5hcHAwMTp1cy1lYXN0LTE6MzM1YzFmNDQtODdjOS00YmJkLWEzMTQtOTNiNDdkOTFmYWRkOjAwMyJdLCJpc3MiOiJodHRwczovL2NvZ25pdG8taWRlbnRpdHkuYW1hem9uYXdzLmNvbSIsImV4cCI6MTU2OTUyNDg5OCwiaWF0IjoxNTY5NTIzOTk4fQ.S8JRKAALQZV0FT0GX6WDBE2EZUH2UPHIJLQ1AX9_PQBXLDM4V7UFFVUDXMHGHMZ2T4VMC6R2ILUJATYO05EIKB4HKWPEHSJWHAT8ZUQ9MRVEZ4KJFAY-7ER4LCGKN8MW-ZTZWQRPXUAYGP3RHQFYDV7FGJCJ3GE-MTTCBGXRAY_0H8NNOQE2F1WRO0KPE-Q-8GXF2P89WGFM9FAHZYOBV0FCZYOH8LCAZ7CKQJQ6FO8NYIAQDXDWFJM5-SLMPRYJLBIW88PBLO00ASOP5OGTHFD61JJCUUXFDRB6UTUGM-RUNILJRCTZB5_AB0FXS2YWAG2YZ3_JOFPWDLL-FNQ2UA", "IdentityId": "us-east-1:aecb58a8-2776-4431-978c-d6395e1b5379" } get credentials aws cognito-identity get-credentials-for-identity --identity-id $IDENTITY_ID --logins "cognito-identity.amazonaws.com=$TOKEN" example output { "Credentials": { "SecretKey": "REDACTED", "SessionToken": "REDACTED", "Expiration": 1569527701.0, "AccessKeyId": "REDACTED" }, "IdentityId": "us-east-1:aecb58a8-2776-4431-978c-d6395e1b5379" } NOTE: --logins cognito-identity.amazonaws.com=. See https://aws.amazon.com/blogs/mobile/understanding-amazon-cognito-authentication-part-4-enhanced-flow/ When using an Amazon Cognito token with GetCredentialsForIdentity, you use the key cognito-identity.amazonaws.com in the logins parameter. You can now use AccessKeyId, SecretKey, and SessionToken to access AWS resources. Resources Developer Authenticated Identities (Identity Pools) Understanding Amazon Cognito Authentication Part 2: Developer Authenticated Identities --- ### Gitlab - URL: https://brianpfeil.com/post/gitlab/ - Date: 2019-09-18 - Tags: gitlab code for article  pfeilbr/gitlab-playground learn gitlab. Uses GitLab CE Docker image to run locally Based on steps in docs @ GitLab Docker images mkdir ~/dev/gitlab # not ssh exposed on 2022 to not conflict with mac | System Prefs | Sharing | Remote Login sudo docker run --detach \ --hostname gitlab.example.com \ --publish 443:443 --publish 80:80 --publish 2022:22 \ --name gitlab \ --restart always \ --volume ~/dev/gitlab/config:/etc/gitlab \ --volume ~/dev/gitlab/logs:/var/log/gitlab \ --volume ~/dev/gitlab/data:/var/opt/gitlab \ gitlab/gitlab-ce:latest # ***NOTE*** can take up to 10 min to load # check logs via `sudo docker logs -f gitlab` or Kitematic UI | Container Logs open http://localhost # will need to set root password # login with username: root, password: YOUR_PASSWORD # add ssh key via User Settings | SSH Keys cat ~/.ssh/id_rsa.pub | pbcopy # create project in UI (`project01`) # note port 2022 git clone ssh://git@localhost:2022/root/project01.git # stopping sudo docker stop gitlab Container Logs via Kitematic UI Project view --- ### Serverless Lambda Layers - URL: https://brianpfeil.com/post/serverless-lambda-layers/ - Date: 2019-09-18 - Tags: serverless, lambda code for article  pfeilbr/serverless-lambda-layers-playground learn lambda layers with serverless framework Running see serverless.yml and index.js layers are applied in order, meaning last layer in array is applied last and will overwrite any common files in other layers. # install deps npm i # run via local packages sls npm run sls -- deploy # invoke and view logs npm run sls -- invoke --function hello --log Resources serverless docs | AWS - Layers Part 2 β€” Create Lambda Layers with Serverless Framework and Offline support --- ### AWS Service Catalog - URL: https://brianpfeil.com/post/aws-service-catalog/ - Date: 2019-09-10 - Tags: aws code for article  pfeilbr/aws-service-catalog-playground learn aws service catalog aws-service-catalog-playground Concepts Service Catalog Pipeline CloudFormation Support Provision a Service Catalog Product Instance using CloudFormation Composing Solutions with AWS Service Catalog Provisioned Products Example Use Case | Static Website Resources Concepts products are cloudformation templates portfolio is collection of products access to portfolios is via IAM users, groups, roles IT administrator creates products and portfolios and grants access End user accesses products and deploys them example use cases: approved self-service products from Solution Factory e.g. static web site. S3 + CloudFormation + WAF + ACM (certificate) + Route 53 (hosted zone, domain) e.g. Oracle RDS DB with all security, tags, etc. in place Service Actions - enable end users to perform operational tasks, troubleshoot issues, run approved commands, or request permissions in AWS Service Catalog via SSM docs. can include/reference existing product(s) in your product cloudformation template. This enables modular composition and nesting. Service Catalog Pipeline Service catalog can be used to deliver products to all spoke accounts in an org. Central hub account that provisions AWS Service Catalog Products into spoke accounts on your behalf CloudFormation Support Service Catalog resources can be created using CloudFormation. See AWS Service Catalog resource type reference . Provision a Service Catalog Product Instance using CloudFormation You can provision a Service Catalog Product using the AWS::ServiceCatalog::CloudFormationProvisionedProduct resource type. For example, if you have a service catalog product named MyProduct you can provision an instance of it using the following cfn. AWSTemplateFormatVersion: '2010-09-09' Description: My Service Catalog Provisioned Product Resources: MyProvisionedProduct: Type: AWS::ServiceCatalog::CloudFormationProvisionedProduct Properties: ProductName: MyProduct ProvisioningArtifactName: '1.0' ProvisioningParameters: - Key: param1 Value: "param1value" - Key: param2 Value: "param2Value" Composing Solutions with AWS Service Catalog Provisioned Products AWS Service Catalog now supports obtaining outputs from a Service Catalog provisioned product in an AWS CloudFormation template. Product outputs provide the interface from one product to another. With this new feature, administrators and developers can easily refer to those outputs in order to combine the products needed for their applications, which saves time building applications that use more than one product, such as a three-tier web application. Provisioned product outputs are now available in AWS Service Catalog . Enabled via AWS::ServiceCatalog transform Example cfn from docs // Example 1 AWSTemplateFormatVersion: 2010-09-09 Transform: 'AWS::ServiceCatalog' Resources: ExampleParameter: Type: 'AWS::SSM::Parameter' Properties: Type: String Value: '[[servicecatalog:provisionedproduct:SampleProvisionedProduct:SampleOutputKey]]' // Example 2 AWSTemplateFormatVersion: 2010-09-09 Transform: 'AWS::ServiceCatalog' Resources: ExampleParameter: Type: 'AWS::SSM::Parameter' Properties: Type: String Value: '[[servicecatalog:provisionedproduct:SampleProvisionedProduct:SampleOutputKey]]' // Example 3 AWSTemplateFormatVersion: 2010-09-09 Transform: AWS::ServiceCatalog Resources: ExampleParameter: Type: 'AWS::SSM::Parameter' Properties: Type: String Value: "[[servicecatalog:provisionedproduct:SampleProvisionedProduct:SampleOutputKey]]" // Example 4 AWSTemplateFormatVersion: 2010-09-09 Transform: AWS::ServiceCatalog Resources: ExampleParameter: Type: 'AWS::SSM::Parameter' Properties: Type: String Value: >- [[servicecatalog:provisionedproduct:SampleProvisionedProduct:SampleOutputKey]] // Example 5 AWSTemplateFormatVersion: 2010-09-09 Transform: AWS::ServiceCatalog Resources: ExampleParameter2: Type: 'AWS::SSM::Parameter' Properties: Type: String Value: [[servicecatalog:provisionedproduct:SSMProductProvisionedProduct:SampleOutputKey]] Example Use Case | Static Website The following is a simple example of a “Static Website” product for the service catalog. It’s an S3 bucket with website enabled for it. This product is purposely kept simple to keep the focus on Service Catalog, but a product can be make up of anything that can be expressed via a CloudFormation template. Define Launch Constraint the IAM role the cloudformation stack provisioning runs under Allows you to assign an IAM role that is used to provision the resources at launch, so you can restrict user permissions without impacting users’ ability to provision products from the catalog. Launch constraint for a product must be added at Portfolio level see AWS Service Catalog Launch Constraints Assign Users, Groups, Roles for Portfolio End User Provisioning Constraint Types Template constraints allow you to limit/constrain CloudFormation template parameters. see AWS Service Catalog Template Constraints CloudFormation Outputs End User Provisioned Products List Admin add new product version End user Update Provisioned Product End user view Resource changes End user provisioning update S3 static website hosting routing rules added (the update) “Backing” CloudFormation Stack Details Resources AWS Service Catalog | AWS Management & Governance Blog - all aws blogs posts tagged with “AWS Service Catalog” AWS Service Catalog Documentation service-catalog-tools-workshop.com/ aws-samples/aws-service-catalog-reference-architectures AWS Service Catalog - Getting Started Simplify sharing your AWS Service Catalog portfolios in an AWS Organizations setup | Amazon Web Services AWS re:Invent 2018: Streamlining Application Development with AWS Service Catalog (DEV328) AWS CloudFormation support for AWS Service Catalog products GitHub - AlexRex/cdk-service-catalog: Showcase how to do a service catalog using CDK Standardize compliance in AWS using DevOps and a Cloud Center of Excellence (CCOE) approach | Amazon Web Services - example pipeline to delivery service catalog portfolio across all accounts in org. local evernote search tag:service-catalog --- ### Serverless Plugin CloudFront Lambda Edge - URL: https://brianpfeil.com/post/serverless-plugin-cloudfront-lambda-edge/ - Date: 2019-09-10 - Tags: serverless, cloudfront, lambda code for article  pfeilbr/serverless-plugin-cloudfront-lambda-edge-playground learn silvermine/serverless-plugin-cloudfront-lambda-edge visit https://d3cztrjc4xcpde.cloudfront.net and login with user01/password01 Basic Auth Challenge username: user01 password: password01 users are stored in aws secrets manager and sourced from users.json Usage # deploy all (provision infra, build static site, copy to s3, invalidate cache). npm run deploy # separate component deploys npm run deploy-infrastructure npm run build-static-site # gatsby npm run publish-static-assets-to-bucket npm run cloudfront:invalidate # bucket # not accessible because of Origin Access Identity applied # e.g. http://s3-cf-private-static-site-01-dev.s3-website-us-east-1.amazonaws.com open "http://$(node scripts/get-stack-property.js WebsiteBucketName).s3-website-us-east-1.amazonaws.com/index.html" # cloudfront url. e.g. https://d3cztrjc4xcpde.cloudfront.net open "https://$(node scripts/get-stack-property.js CloudFrontDistributionDomainName)" Removing Auth Update CloudFront Behavior to remove “Viewer Request” “Lambda Function Association”, then invalidate cache on all “*”. Viewing CloudWatch Logs for Lambda@Edge Functions logging takes place in the region of the edge (PoP) location. This will vary based on the client location. Visit CloudFront | Monitoring | Lambda@Edge Functions | YOUR FUNCTION, then click [View Function Metrics] button see Determining the Lambda@Edge Region for more details Lambda@Edge Request { "event": { "Records": [ { "cf": { "config": { "distributionDomainName": "d13ydba49ilc9v.cloudfront.net", "distributionId": "E20V9SS2N0VT6P", "eventType": "viewer-request", "requestId": "NcxNfOh2NaptKOLBlSimVF7AAlRp10OM-F1-CMZsJOGSMEkHUVd25A==" }, "request": { "clientIp": "100.11.96.70", "headers": { "host": [ { "key": "Host", "value": "d13ydba49ilc9v.cloudfront.net" } ], "user-agent": [ { "key": "User-Agent", "value": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.132 Safari/537.36" } ], "authorization": [ { "key": "authorization", "value": "Basic dXNlcjpwYXNz" } ], "upgrade-insecure-requests": [ { "key": "upgrade-insecure-requests", "value": "1" } ], "sec-fetch-mode": [ { "key": "sec-fetch-mode", "value": "navigate" } ], "sec-fetch-user": [ { "key": "sec-fetch-user", "value": "?1" } ], "accept": [ { "key": "accept", "value": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3" } ], "sec-fetch-site": [ { "key": "sec-fetch-site", "value": "none" } ], "accept-encoding": [ { "key": "accept-encoding", "value": "gzip, deflate, br" } ], "accept-language": [ { "key": "accept-language", "value": "en-US,en;q=0.9,nb;q=0.8,fr;q=0.7" } ], "if-none-match": [ { "key": "if-none-match", "value": "\"2cad58ac06c32be3c6384050881bc507\"" } ], "if-modified-since": [ { "key": "if-modified-since", "value": "Tue, 10 Sep 2019 21:32:57 GMT" } ] }, "method": "GET", "querystring": "", "uri": "/" } } } ] } } Resources add origin access identity see the following for cfn markup https://github.com/lroguet/amzn-cloudformation/blob/master/storage-content-delivery/static-website-with-cloudfront.yml Restricting Access to Amazon S3 Content by Using an Origin Access Identity S3 Bucket | Granting Permission to an Amazon CloudFront Origin Identity TODO DNS (route 53) Routing Traffic to an Amazon CloudFront Web Distribution by Using Your Domain Name . Covers [root|sub]domainsG see https://www.brautaset.org/articles/2017/route-53-cloudformation.html . Contains ApexRecordSet (example.com) and WwwRecordSet (www.example.com ) Alias Resource Record Set for a CloudFront Distribution update serverless.yml:iamRoleStatements with dynamic region, account id, and secretsmanager name via serverless-pseudo-parameters plugin Scratch $(node ./scripts/get-stack-property.js ) --- ### AWS Amplify Console - URL: https://brianpfeil.com/post/aws-amplify-console/ - Date: 2019-09-09 - Tags: aws, amplify code for article  pfeilbr/aws-amplify-console-playground learn AWS Amplify Console AWS Amplify Console is a continuous delivery and hosting service for modern web applications. The AWS Amplify Console simplifies the deployment of your application front end and backend. Connect to your code repository and your front end and backend are deployed in a single workflow, on every code commit. Flow You specify a source (github, bitbucket, gitlab, S3, zip file upload). On code change (commit), it checks out the code to a CodeBuild project. Runs your build, test, [backend] deploy, etc. Deploys static web assets (.html, .js, .css, images) to S3 to be served with CloudFront Verifies deployment by visiting root site URL with various deveice form factors (iPhone, iPad, desktop) and taking screenshots. Build settings are defined in amplify.yml . See Configuring Build Settings for YML Specification Syntax. set amplify service role. codebuild env will assume this role and commands will execute in this context. showing assumed role in codebuild env Update SAM deploy bucket policy to allow amplify service (amplify.amazonaws.com) to read/write to bucket. Custom Domain configuration App running on custom domain https://amplify-master.minote.net/ Resources Amplify Console | Getting Started aws-amplify/amplify-console --- ### AWS Polly - URL: https://brianpfeil.com/post/aws-polly/ - Date: 2019-09-06 - Tags: aws, polly code for article  pfeilbr/aws-polly-playground learn aws polly text-to-speech (TTS) Running based on https://docs.aws.amazon.com/sdk-for-javascript/v2/developer-guide/getting-started-browser.html#getting-started-browser-scenario update js code in polly.html with the following AWS.config.region = '<YOUR_REGION>'; AWS.config.credentials = new AWS.CognitoIdentityCredentials({ IdentityPoolId: '<YOUR_IDENTITY_POOL_ID>' }); open polly.html --- ### AWS Delivlib - URL: https://brianpfeil.com/post/aws-delivlib/ - Date: 2019-09-03 - Tags: aws, continuous-delivery code for article  pfeilbr/aws-delivlib-playground learn aws-delivlib , which is a library that leverages AWS Cloud Development Kit (CDK) for defining continuous pipelines for building, testing and publishing code libraries through AWS CodeBuild and AWS CodePipeline. Prerequisites pipeline source github repo must exist. (e.g. pfeilbr/aws-delivlib-playground ) github personal access token must be stored in SSM Parameter named /com/brianpfeil/aws-delivlib-playground/github-personal-access-token Running src/pipeline-hello-world is an example code pipeline where the source is this github repo (pfeilbr/aws-delivlib-playground ). The source nodejs app is src/hello-world with jest tests. Code Pipeline source -> build -> test (linux) -> test (windows) If any of the tests (*.sh) in src/pipeline-hello-world/tests/ change, be sure to do a npm run build && npm run cdk deploy. Internally uses assets.ZipDirectoryAsset from "@aws-cdk/assets" package. mkdir -p src/pipeline-hello-world cd src/pipeline-hello-world cdk init --language typescript # at this time, `aws-delivlib` is not using the most recent version of CDK # need to do the following # see https://github.com/aws/aws-cdk/issues/1733 npm remove aws-cdk @aws-cdk/core aws-delivlib npm i aws-cdk@0.24.1 -D npm i @aws-cdk/cdk@0.24.1 npm i aws-delivlib # for dev npm run watch # build npm run build # generate cfn to stdout npm run cdk synth # deploy stack npm run cdk deploy # make changes # build npm run build # diff npm run cdk diff # delete stack npm run cdk destroy AWS Console | CodePipeline AWS Console | CloudFormation Stack --- ### Drawio - URL: https://brianpfeil.com/post/drawio/ - Date: 2019-08-26 - Tags: drawio code for article  pfeilbr/drawio-playground learn and store draw.io templates see diagrams @ AWS Architecture Center for examples Scratchpad The scratchpad is a temporary working space for keeping commonly used shapes for easy access. See How to use the Scratchpad? exported aws-scratchpad @ aws-scratchpad-panel-export.xml . Note that the scratchpad is stored in the local storage of the browser, which means it’s contents are deleted if cookies are cleared. The scratchpad is automatically saved between each change. You can export the scratchpad as a draw.io library from that dialog, if you want to store it more permanently or share it with others. --- ### Zipkin - URL: https://brianpfeil.com/post/zipkin/ - Date: 2019-08-21 - Tags: distributed-tracing, observability code for article  pfeilbr/zipkin-playground learn zipkin the distributed tracing system Prerequisites docker Running running zipkin server and viewing web ui # run zipkin server (NOTE: this is non-blocking and will return) docker run -d -p 9411:9411 openzipkin/zipkin # open zipkin web ui open http://localhost:9411/zipkin/ running a web app with frontend and backend the sends traces to zipkin server. see frontend.js for how code is instrumented to send traces to zipkin server. # download zipkin javascript example curl -LO https://github.com/openzipkin/zipkin-js-example/archive/master.zip unzip master.zip cd zipkin-js-example-master/web # install dependencies npm install # had to run the following also. this make be fixed npm install node-fetch # in `frontend.js` change `const {wrapAxios} = require("zipkin-instrumentation-axiosjs");` # to `const wrapAxios = require("zipkin-instrumentation-axiosjs");` # create bundle npm run browserify # run frontend and backend DEBUG=true npm start # open app. displays datetime and sends traces to zipkin sever. open http://localhost:8081 # now open zipkin web ui to view traces open http://localhost:9411/zipkin/ # click the [Find Traces] button view JSON trace data by clickint the [JSON] button [ { "traceId": "9ee01dfbbba1a5f2", "parentId": "f107fa7e6a7f29db", "id": "15f42a92f26d5a19", "kind": "SERVER", "name": "get /api", "timestamp": 1566397279334946, "duration": 1509626, "localEndpoint": { "serviceName": "backend", "ipv4": "192.168.1.14" }, "tags": { "http.path": "/api", "http.status_code": "200" }, "shared": true }, { "traceId": "9ee01dfbbba1a5f2", "parentId": "f107fa7e6a7f29db", "id": "15f42a92f26d5a19", "kind": "CLIENT", "name": "get", "timestamp": 1566397279301452, "duration": 1542392, "localEndpoint": { "serviceName": "frontend" }, "tags": { "http.path": "/api", "http.status_code": "200" } }, { "traceId": "9ee01dfbbba1a5f2", "parentId": "9ee01dfbbba1a5f2", "id": "f107fa7e6a7f29db", "name": "pay-me", "timestamp": 1566397279295227, "duration": 1552644, "localEndpoint": { "serviceName": "frontend" } }, { "traceId": "9ee01dfbbba1a5f2", "id": "9ee01dfbbba1a5f2", "kind": "SERVER", "name": "get /", "timestamp": 1566397279276961, "duration": 1572319, "localEndpoint": { "serviceName": "frontend", "ipv4": "192.168.1.14" }, "tags": { "http.path": "/", "http.status_code": "200" } } ] --- ### Serverless Framework Full Lifecycle with Dashboard - URL: https://brianpfeil.com/post/serverless-framework-full-lifecycle-with-dashboard/ - Date: 2019-08-20 - Tags: serverless code for article  pfeilbr/serverless-framework-full-lifecycle-with-dashboard-playground learn and understand how Serverless Framework – Now, Full Lifecycle works. This allows serverless to instrument code, send cloudwatch logs to the serverless SaaS. This allows serverless to provide the Serverless Dashboard features Running serverless from cli to create project The service, app, and org top-level properties added to serverless.yml enable serverless full lifecycle / dashboard service: serverless-with-dashboard-playground-01 app: serverless-with-dashboard-playground-01 org: pfeilbr To disable, add the following to serverless.yml custom: enterprise: collectLambdaLogs: false Serverless framework creates a role during the deploy. This allows for the cloudwatch log group logs to be sent to serverless SaaS app. e.g. arn:aws:iam::529276214230:role/serverless-with-dashboard-EnterpriseLogAccessIamRo-19SDI69RM1KJ4 inline policy { "Version": "2012-10-17", "Statement": [ { "Action": [ "logs:FilterLogEvents" ], "Resource": [ "arn:aws:logs:us-east-1:529276214230:log-group:/aws/lambda/serverless-with-dashboard-playground-01-dev-hello:*" ], "Effect": "Allow" } ] } trust relationship trust serverless aws account 802587217904 { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "AWS": "arn:aws:iam::802587217904:root" }, "Action": "sts:AssumeRole", "Condition": { "StringEquals": { "sts:ExternalId": "ServerlessEnterprise-LGGXBmZw2Z47MmWq6b" } } } ] } Instrumented/wrapped Code Example On deploy, serverless instruments/wraps your code/handlers. You don’t see this locally in your codebase. You only see on the deployment side in AWS. embedded/bundled serverless SDK SERVERLESS_ENTERPRISE wrapped log example serverless hooks logging to stdout and stderr via serverlessSDK. This allows it to log structured JSON logs to cloudwatch logs with the prefix SERVERLESS_ENTERPRISE. This logging is additional, the console.logs are logged independently. To see the log group subscription details aws logs describe-subscription-filters --log-group-name '/aws/lambda/serverless-with-dashboard-playground-01-dev-hello' { "subscriptionFilters": [ { "filterPattern": "?\"REPORT RequestId: \" ?\"SERVERLESS_ENTERPRISE\"", "filterName": "serverless-with-dashboard-playground-01-dev-CloudWatchLogsSubscriptionFilterHelloLogGroup-1SAKWRFMW5JHE", "creationTime": 1566318781436, "logGroupName": "/aws/lambda/serverless-with-dashboard-playground-01-dev-hello", "destinationArn": "arn:aws:logs:us-east-1:802587217904:destination:LGGXBmZw2Z47MmWq6b#VlGYyRJNfvVVgHf8y1#serverless-with-dashboard-playground-01#dev", "distribution": "ByLogStream" } ] } It sends logs to the serverless AWS account (802587217904). The destinationArn: arn:aws:logs:us-east-1:802587217904:destination:LGGXBmZw2Z47MmWq6b#VlGYyRJNfvVVgHf8y1#serverless-with-dashboard-playground-01#dev is a kinesis stream within the serverless AWS account. This is done via Cross-Account Log Data Sharing with Subscriptions Serverless Dashboard | Views “safeguard policies” are evaluated on serverless deploy Send notifications (e.g. email) Resources reddit | Serverless Framework now supports full lifecycle on AWS --- ### AWS SAM Local - URL: https://brianpfeil.com/post/aws-sam-local/ - Date: 2019-08-16 - Tags: aws, sam code for article  pfeilbr/aws-sam-local-playground learn aws-sam-local session cd examples/hello-world sam local invoke "HelloWorld" -e event.json echo '{"name": "Brian" }' | sam local invoke "HelloWorld" # debugging # ensure launch.json localRoot is set to directory where index.js or code exists # e.g. "localRoot": "${workspaceRoot}/examples/hello-world" sam local invoke -e event.json -d 5858 HelloWorld # set breakpoints in code (vscode) # then run debug in vscode # package # NOTE: bucket must exist (`aws s3 mb s3://sam-deploy-bucket-01`) sam package --template-file template.yaml --s3-bucket sam-deploy-bucket-01 --output-template-file packaged.yaml # deploy sam deploy --template-file packaged.yaml --stack-name sam-hello-world-v0 --capabilities CAPABILITY_IAM # invoke # NOTE: you'll need to lookup the "full" function name aws lambda invoke --function-name "sam-hello-world-v1-HelloWorld-L8DLT50DZNIJ" --payload '{"name": "brian"}' output.log; cat output.log # view logs sam logs -n HelloWorld --stack-name sam-hello-world-v1 # --- # invoke lambda via api gateway example cd examples/api-event-source # local development sam local start-api curl http://127.0.0.1:3000/ # package # NOTE: bucket must exist (`aws s3 mb s3://sam-deploy-bucket-01`) sam package --template-file template.yaml --s3-bucket sam-deploy-bucket-01 --output-template-file packaged.yaml # deploy sam deploy --template-file packaged.yaml --stack-name api-event-source-v1 --capabilities CAPABILITY_IAM Resources AWS Serverless Application Model (AWS SAM) Documentation --- ### AWS CloudFormation - URL: https://brianpfeil.com/post/aws-cloudformation/ - Date: 2019-08-14 - Tags: aws, cloudformation code for article  pfeilbr/aws-cloudformation-playground learn aws cloudformation Concepts Stacks Drift detection Resources that support import and drift detection operations - AWS CloudFormation Nested Stacks Stack Sets enabling you to create, update, or delete stacks across multiple accounts and Regions with a single operation You can create a stack set with either self-managed or service-managed permissions self-managed - first create the necessary IAM roles to establish a trusted relationship between the account you’re administering the stack set from and the account you’re deploying stack instances to service-managed (use with Orgs) - deploy stack instances to accounts managed by AWS Organizations in specific Regions. With this model, you don’t need to create the necessary IAM roles; StackSets creates the IAM roles on your behalf. automatic deployment enabled, StackSets automatically deploys to accounts that are added to the target organization or organizational units (OUs) in the future account gate is an optional feature that lets you specify an AWS Lambda function to verify that a target account meets certain requirements before AWS CloudFormation StackSets begins stack operations in that account stackset stacks should be account and region agnostic. same template will be used for all deployments. use SSM parameter store parameters in the target account + region for specific configurations. Running Examples examples in templates/ directory # validate template aws cloudformation validate-template --template-body file://templates/s3-bucket.yaml # deploy aws cloudformation deploy --template-file templates/s3-bucket.yaml --stack-name s3-bucket-stack # list stack output values aws cloudformation describe-stacks --stack-name s3-bucket-stack --query "Stacks[0].Outputs[].OutputValue" common parameters --parameter-overrides Name=Value --tags \ Key1="Value1" \ Key2="Value2" \ Dynamic References Example see templates/dynamic-references-ssm-secrets.yaml and Using Dynamic References to Specify Template Values # validate aws cloudformation validate-template --template-body file://templates/dynamic-references-ssm-secrets.yaml # create stack aws cloudformation deploy --template-file templates/dynamic-references-ssm-secrets.yaml --stack-name dynamic-references-ssm-secrets-stack # uncomment `Outputs` in templates/dynamic-references-ssm-secrets.yaml # update stack aws cloudformation deploy --template-file templates/dynamic-references-ssm-secrets.yaml --stack-name dynamic-references-ssm-secrets-stack # view stack outputs # NOTE: `MySecret01Value` output does not get resolved due to security aws cloudformation describe-stacks --stack-name dynamic-references-ssm-secrets-stack --query "Stacks[0].Outputs[].OutputValue" # clean up aws cloudformation delete-stack --stack-name dynamic-references-ssm-secrets-stack Resources CloudFormation Custom Resources CloudFormation Macros Scratch aws cloudformation deploy --template-file templates/playground.yaml --stack-name playground-stack --capabilities "CAPABILITY_IAM" "CAPABILITY_NAMED_IAM" "CAPABILITY_AUTO_EXPAND" aws cloudformation describe-stacks \ --stack-name "playground-stack" \ --query "Stacks[0].Outputs[?OutputKey=='LambdaName'].OutputValue" --output text playground-stack-MyLambdaFunction-iMVMRs2CA3fm aws lambda invoke \ --cli-binary-format "raw-in-base64-out" \ --function-name "playground-stack-MyLambdaFunction-iMVMRs2CA3fm" \ --payload '{"msg": "hello"}' \ output.log; cat output.log; rm output.log --- ### Intellij Maven App - URL: https://brianpfeil.com/post/intellij-maven-app/ - Date: 2019-08-14 - Tags: java, maven, build-tools code for article  pfeilbr/intellij-maven-app-playground example of IntelliJ Maven App Resources Apache Maven Assembly Plugin / Usage # clean mvn clean # package with all dependencies and make .jar executable # see `pom.xml` for details mvn package # run jar java -jar ./target/app01-1.0-SNAPSHOT-jar-with-dependencies.jar --- ### AWS XRay - URL: https://brianpfeil.com/post/aws-xray/ - Date: 2019-08-13 - Tags: aws, xray code for article  pfeilbr/aws-xray-playground learn AWS X-Ray see index.js Development # run x-ray daemon locally (nodejs code sends trace data to it and it forwards it to aws x-ray service) # see https://docs.aws.amazon.com/xray/latest/devguide/xray-daemon.html cd ~/bin/aws-xray-daemon-macos-3.x ./xray_mac -o -n us-east-1 # run code that generates trace/segment data npm run dev Screenshots Resources aws/aws-xray-sdk-node aws/aws-xray-sdk-node/blob/master/packages/core/README.md --- ### Pipenv - URL: https://brianpfeil.com/post/pipenv/ - Date: 2019-08-07 - Tags: python, packaging code for article  pfeilbr/pipenv-playground learn pipenv Pipenv is a tool that aims to bring the best of all packaging worlds (bundler, composer, npm, cargo, yarn, etc.) to the Python world. Windows is a first-class citizen, in our world. It automatically creates and manages a virtualenv for your projects, as well as adds/removes packages from your Pipfile as you install/uninstall packages. It also generates the ever-important Pipfile.lock, which is used to produce deterministic builds. based on https://docs.python-guide.org/dev/virtualenvs/ Usage # install pip3 install pipenv # create directory cd ~/tmp mkdir pipenv-playground cd pipenv-playground # install dependency pipenv install requests # write some code that uses the dependency touch main.py # run it using the created virtualenv pipenv run python main.py # can also specify `python3` explicitly pipenv run python3 main.py # try with jupyter notebook cd .. mkdir jupyter-notebook-playground cd jupyter-notebook-playground pipenv install jupyter pipenv run jupyter notebook # try with jupyterlab pipenv install jupyterlab pipenv run jupyter lab --- ### Python Packaging - URL: https://brianpfeil.com/post/python-packaging/ - Date: 2019-08-07 - Tags: python code for article  pfeilbr/python-packaging-playground learn python packaging via Packaging Python Projects view example package at https://test.pypi.org/project/example-pkg-pfeilbr/ python3 -m venv .venv source .venv/bin/activate mkdir example-pkg-pfeilbr touch example-pkg-pfeilbr/__init__.py python3 -m pip install --user --upgrade setuptools wheel python3 setup.py sdist bdist_wheel pip install --upgrade pip pip install wheel pip install twine python3 -m twine upload --repository-url https://test.pypi.org/legacy/ dist/* # testing mkdir tmp cd tmp python3 -m venv .venv source .venv/bin/activate pip list python3 -m pip install --index-url https://test.pypi.org/simple/ --no-deps example-pkg-pfeilbr # to install specific version. did this because it was using 0.0.1 from a cache python3 -m pip install --index-url https://test.pypi.org/simple/ --no-deps 'example-pkg-pfeilbr==0.0.2' pip list # test using package echo -e "import example_pkg_pfeilbr\nprint(example_pkg_pfeilbr.name)" | python Example Package This is a simple example package. You can use Github-flavored Markdown to write your content. --- ### Neo4j - URL: https://brianpfeil.com/post/neo4j/ - Date: 2019-08-05 - Tags: neo4j code for article  pfeilbr/neo4j-playground learn neo4j graph database Example Session # update brew update # install brew install neo4j # add neo4j binaries to path export PATH="$PATH:/usr/local/Cellar/neo4j/3.5.8/bin" # start server neo4j start # open web ui open http://localhost:7474/ # it'll prompt for password on first time # user and password is neo4j # you then need to change the password # open neo4j shell (Cypher Shell) cypher-shell -u neo4j -p PASSWORD # create nodes in shell CREATE (p:Person {name:'brian'}); CREATE (p:Person {name:'tricia'}); # create relationship MATCH (brian:Person {name:'brian'}) MATCH (tricia:Person {name:'tricia'}) CREATE (brian)-[:IS_MARRIED_TO]->(tricia) CREATE (tricia)-[:IS_MARRIED_TO]->(brian); # query `IS_MARRIED_TO` relationship MATCH (p1:Person)-[rel:IS_MARRIED_TO]->(p2:Person) RETURN p1 AS spouse1, p2 AS spouse2; # clean up: delete all nodes and relationships MATCH (n) DETACH DELETE n; # exit shell :exit # stop server neo4j stop Web UI Resources neo4j documentation --- ### AWS Transcribe - URL: https://brianpfeil.com/post/aws-transcribe/ - Date: 2019-07-30 - Tags: aws code for article  pfeilbr/aws-transcribe-playground learn aws transcribe speech to text service see src/index.js and example output in data/ Running # install deps npm install # run npm start --- ### Chrome AWS Lambda - URL: https://brianpfeil.com/post/chrome-aws-lambda/ - Date: 2019-07-24 - Tags: chrome, aws, lambda code for article  pfeilbr/chrome-aws-lambda-playground learn the chrome-aws-lambda package for use with aws lambda via serverless framework Screenshot Web Page return screenshot of page from provided url example with https://twitter.com https://SERVICE_ENDPOINT/dev/webpagescreenshot?url=https://twitter.com --- ### YouTube API - URL: https://brianpfeil.com/post/youtube-api/ - Date: 2019-07-22 - Tags: youtube, nodejs code for article  pfeilbr/youtube-api-playground learn youtube api using node.js see src/index.js Running # install deps npm install # setup your .env with your youtube api key cp .env.sample .env # run npm start --- ### AWS CDK - URL: https://brianpfeil.com/post/aws-cdk/ - Date: 2019-07-11 - Tags: aws, cdk code for article  pfeilbr/aws-cdk-playground learn AWS Cloud Development Kit (CDK) see kitchen-sink/README.md Description express resources using general purpose programming languages (ts/js/python/java/C#) constructs - construct levels 1, 2, 3. cfn (L1), CDK (L2), pattern/solution (L3) synth to cfn cloud assemblies - cfn + source code, docker images, assets (s3) aspects - ability to visit each node/resource in stack and apply changes Application -> Stacks -> Constructs Parameters - cfn parameters. can pass in via cdk synth. Runtime context - key-value pairs that can be associated with a stack or construct. Can only be string values (kind of like parameters) [tf|k8s] CDKs jsii - core/foundational tech for multi-language/polyglot support. bind any language to underlying typescript implementation. CDK pipelines for CI/CD Custom Logical Names - shows how to hook into an provide own resource names. Can be used for IAM policies based on resource name prefixes Usage with Permissions Boundaries - class PermissionsBoundary Β· AWS CDK . e.g. PermissionsBoundary.of(this).apply(permissionsBoundariesPolicy); Key Files and Directories bin - entry point to CDK app. imports 1 or more stacks from lib folder lib/*-stack.* - define stacks here which contain constructs cdk.json - cdk configuration test - unit/integration/e2e tests cdk.out - CDK assembly / synth output (cfn, assets, etc.) Common Areas @aws-cdk/core - App, Stack, Stack.account, Stack.region, Stack.tags, Stack.terminationProtection, Construct, Duration, CfnOutput lambda.Function, lambda.Code.fromAsset, lambda.Code.fromInline @aws-cdk/aws-iam - Role, User, Group, PolicyStatement Common Steps # init cdk app cdk init app --language javascript cdk init app --language typescript # list stacks in the app cdk list # [optional] build for ts -> js. not required for js npm run build # synthesize to cfn (`cdk.out`) cdk synth # run tests npm test # compare the specified stack with the deployed stack cdk diff # deploy cdk deploy # force deploy, no prompt cdk deploy --force --require-approval never # delete cdk destroy [STACKS..] CDK Internals Details on the inner workings of CDK. CDK Tree Core of CDK is based on tree structure similar to the DOM. node: ConstructNode - accessed via this.node - root of the tree. node.children node.findChild(id: string) - search for child in tree with id. new s3.Bucket(this, "Assets"). "Assets" is the id. node.tryFindChild(id: string) - same as findChild but won’t throw if id doesn’t exist. Will return undefined. node.defaultChild - reference to primary level 1 construct (Cfn*) common L1 construct methods overrideLogicalId addOverride(propertyPath, value) - e.g. cfnBucket.addOverride("Properties.BucketName", "my-bucket-name-01") need to use for cfn attributes that are outside of the Properties. e.g. DeletionPolicy add bucket name to Metadata - cfnBucket.addOverride("Metadata.BucketName", cfnBucket.ref) addDeletionOverride(propertyPath) - remove a property. e.g. cfnBucket.addDeletionOverride("Properties.BucketName") addPropertyOverride(propertyPath) - cfnBucket.addPropertyOverride("Properties.BucketName", "my-bucket-name-01") all Fn::GetAtt return values can be accessed via a property named Att${Return Value Name}. e.g. cfnBucket.AttArn, cfnBucket.AttDomainName CDK Path every resource defined in CDK tree has a path this.node.path - concatenation of path traversal to the node in the CDK tree. (e.g. MyStack/Assets) this path is added to the Metadata property for each resource in cfn. CDK Identifiers Logical ID - used in cfn template. scoped within stack. calculated using CDK path (sanitized id (no stack id) + hash) Unique ID - uniquely identify resource within CDK application. scoped within CDK application, which may be composed of multiple stacks. sanitized id (including stack) + hash Common CDK Snippets @aws-cdk/core module @aws-cdk/aws-iam module - key module // durations Duration.seconds(300) // 5 minutes Duration.minutes(5) // 5 minutes Duration.hours(1) // 1 hour Duration.days(7) // 7 days Duration.parse('PT5M') // 5 minutes // sizes Size.kibibytes(200) // 200 KiB Size.mebibytes(5) // 5 MiB Size.gibibytes(40) // 40 GiB Size.tebibytes(200) // 200 TiB Size.pebibytes(3) // 3 PiB // create secret const secret = SecretValue.secretsManager('secretId', { jsonField: 'password', // optional: key of a JSON field to retrieve (defaults to all content), versionId: 'id', // optional: id of the version (default AWSCURRENT) versionStage: 'stage', // optional: version stage name (default AWSCURRENT) }); // get default VPC const vpc = ec2.Vpc.fromLookup(stack, 'VPC', { // This imports the default VPC but you can also // specify a 'vpcName' or 'tags'. isDefault: true, }); // custom resource const fn = new lambda.Function(this, 'MyProvider', functionProps); new CustomResource(this, 'MyResource', { serviceToken: fn.functionArn, }); // OR const serviceToken = CustomResourceProvider.getOrCreate(this, 'Custom::MyCustomResourceType', { codeDirectory: `${__dirname}/my-handler`, runtime: CustomResourceProviderRuntime.NODEJS_12_X, description: "Lambda function created by the custom resource provider", }); new CustomResource(this, 'MyResource', { resourceType: 'Custom::MyCustomResourceType', serviceToken: serviceToken }); // bastion host const host = new ec2.BastionHostLinux(this, 'BastionHost', { vpc }); Resources AWS CDK Β· AWS CDK Reference Documentation Infrastructure-as-Code | Constructs | AWS Solutions awslabs/aws-solutions-constructs aws-samples/aws-cdk-examples aws/constructs - Constructs Programming Model panacloud-modern-global-apps/full-stack-serverless-cdk github | search | “filename:cdk.json” Exploring CDK Internals Working with the AWS CDK Explorer - AWS Toolkit for VS Code --- ### AWS Fargate - URL: https://brianpfeil.com/post/aws-fargate/ - Date: 2019-07-10 - Tags: aws, fargate code for article  pfeilbr/aws-fargate-playground learn AWS Fargate # register task definition aws ecs register-task-definition --cli-input-json file://./task-definitions/hello-world.json # run task aws ecs run-task --task-definition "fargate-001" --launch-type "FARGATE" --network-configuration '{"awsvpcConfiguration": {"subnets": ["subnet-154b7928", "subnet-4700526d", "subnet-6cf9a534", "subnet-af5052d9"],"securityGroups": ["sg-047fbd9524f6e2b5e"],"assignPublicIp": "ENABLED"}}' # *** NOTE *** # * "assignPublicIp": "ENABLED" is needed for ecs/fargate to pull image from docker hub # * "securityGroups": must allow outbund traffic # * make sure log group is already created --- ### AWS Glue - URL: https://brianpfeil.com/post/aws-glue/ - Date: 2019-07-09 - Tags: aws, glue code for article  pfeilbr/aws-glue-playground learn and experiment with aws glue Running “Python Shell” Job Local Running/Testing of Script python3 -m venv venv source venv/bin/activate pip install -r requirements.txt python scripts/python3-shell-job-example.py Running/Submitting Python Shell Job cp .env.sample .env # modify .env for your environment # following submits the job (scripts/python3-shell-job-example.py) node src/job-runner.js run-python-shell-script scripts/python3-shell-job-example.py Steps to Run scripts/example-notebook-script-01.py in SageMaker notebook see scripts/example-notebook-script-01.py upload data.csv to S3 create glue crawler for data.csv which results in a table in glue database being created you can verify by previewing the data in athena create aws glue Dev Endpoint no need to specify ssh key create SageMaker notebook SageMaker notebook works just like Zepplin notebook, but less setup steps. open SageMaker notebook and past in code from scripts/example-notebook-script-01.py --- ### AWS Rekognition - URL: https://brianpfeil.com/post/aws-rekognition/ - Date: 2019-07-08 - Tags: aws code for article  pfeilbr/aws-rekognition-playground learn AWS Rekognition see src/examples.js Prerequisites AWS default credentials available to call Rekognition API install imagemagick, graphicsmagick, and ghostscript (for fonts) for labelling image brew install imagemagick brew install graphicsmagick brew install ghostscript Example Labelled Image Using detectLabels API Source Labelled Example Labelled Image Using detectText API Source Labelled Running npm install npm start Video Label Detection Example via CLI # start (it's async) aws rekognition start-label-detection --video "S3Object={Bucket=rekognition-playground,Name=SampleVideo_1280x720_1mb.mp4}" # output { "JobId": "2c9b387607977af21c0839f177bf7034ce1bcd5139810b533dea3deb6361f348" } # in progress aws rekognition get-label-detection --job-id "2c9b387607977af21c0839f177bf7034ce1bcd5139810b533dea3deb6361f348" # output { "Labels": [], "LabelModelVersion": "2.0", "JobStatus": "IN_PROGRESS" } # completed aws rekognition get-label-detection --job-id "2c9b387607977af21c0839f177bf7034ce1bcd5139810b533dea3deb6361f348" # output { "Labels": [ { "Timestamp": 0, "Label": { "Instances": [], "Confidence": 89.929931640625, "Parents": [], "Name": "Animal" } }, { "Timestamp": 0, "Label": { "Instances": [], "Confidence": 69.0970458984375, "Parents": [ { "Name": "Animal" }, { "Name": "Mammal" }, { "Name": "Rodent" } ], "Name": "Bunny" } }, { "Timestamp": 0, "Label": { "Instances": [], "Confidence": 51.81949234008789, "Parents": [ { "Name": "Wildlife" }, { "Name": "Mammal" }, { "Name": "Animal" } ], "Name": "Deer" } }, { "Timestamp": 0, "Label": { "Instances": [], "Confidence": 83.59092712402344, "Parents": [ { "Name": "Plant" } ], "Name": "Grass" } }, { "Timestamp": 0, "Label": { "Instances": [], "Confidence": 50.83674621582031, "Parents": [], "Name": "Green" } }, { "Timestamp": 0, "Label": { "Instances": [], "Confidence": 83.93998718261719, "Parents": [ { "Name": "Animal" } ], "Name": "Mammal" } }, { "Timestamp": 0, "Label": { "Instances": [], "Confidence": 54.92662048339844, "Parents": [ { "Name": "Plant" } ], "Name": "Moss" } }, { "Timestamp": 0, "Label": { "Instances": [], "Confidence": 63.28266906738281, "Parents": [], "Name": "Outdoors" } }, { "Timestamp": 0, "Label": { "Instances": [], "Confidence": 60.88518524169922, "Parents": [ { "Name": "Animal" } ], "Name": "Pet" } }, { "Timestamp": 0, "Label": { "Instances": [], "Confidence": 90.28629302978516, "Parents": [], "Name": "Plant" } }, { "Timestamp": 0, "Label": { "Instances": [], "Confidence": 69.0970458984375, "Parents": [ { "Name": "Animal" }, { "Name": "Mammal" }, { "Name": "Rodent" } ], "Name": "Rabbit" } }, { "Timestamp": 0, "Label": { "Instances": [], "Confidence": 69.0970458984375, "Parents": [ { "Name": "Mammal" }, { "Name": "Animal" } ], "Name": "Rodent" } }, { "Timestamp": 0, "Label": { "Instances": [], "Confidence": 71.99913787841797, "Parents": [ { "Name": "Plant" } ], "Name": "Tree" } }, { "Timestamp": 0, "Label": { "Instances": [], "Confidence": 82.74295806884766, "Parents": [ { "Name": "Plant" } ], "Name": "Vegetation" } }, { "Timestamp": 0, "Label": { "Instances": [], "Confidence": 58.27427291870117, "Parents": [ { "Name": "Animal" } ], "Name": "Wildlife" } }, { "Timestamp": 0, "Label": { "Instances": [], "Confidence": 67.73959350585938, "Parents": [ { "Name": "Animal" } ], "Name": "Zoo" } }, { "Timestamp": 200, "Label": { "Instances": [], "Confidence": 91.4648666381836, "Parents": [], "Name": "Animal" } }, { "Timestamp": 200, "Label": { "Instances": [], "Confidence": 52.55938720703125, "Parents": [ { "Name": "Mammal" }, { "Name": "Animal" }, { "Name": "Rodent" } ], "Name": "Bunny" } }, { "Timestamp": 200, "Label": { "Instances": [], "Confidence": 53.18073654174805, "Parents": [ { "Name": "Wildlife" }, { "Name": "Mammal" }, { "Name": "Animal" } ], "Name": "Deer" } }, { "Timestamp": 200, "Label": { "Instances": [], "Confidence": 81.07315826416016, "Parents": [ { "Name": "Plant" } ], "Name": "Grass" } }, { "Timestamp": 200, "Label": { "Instances": [], "Confidence": 82.8969955444336, "Parents": [ { "Name": "Animal" } ], "Name": "Mammal" } }, { "Timestamp": 200, "Label": { "Instances": [], "Confidence": 55.91206359863281, "Parents": [ { "Name": "Plant" } ], "Name": "Moss" } }, { "Timestamp": 200, "Label": { "Instances": [], "Confidence": 55.965728759765625, "Parents": [], "Name": "Nature" } }, { "Timestamp": 200, "Label": { "Instances": [], "Confidence": 68.73519897460938, "Parents": [], "Name": "Outdoors" } }, { "Timestamp": 200, "Label": { "Instances": [], "Confidence": 54.99771499633789, "Parents": [ { "Name": "Animal" } ], "Name": "Pet" } }, { "Timestamp": 200, "Label": { "Instances": [], "Confidence": 89.0054931640625, "Parents": [], "Name": "Plant" } }, { "Timestamp": 200, "Label": { "Instances": [], "Confidence": 52.55938720703125, "Parents": [ { "Name": "Mammal" }, { "Name": "Animal" }, { "Name": "Rodent" } ], "Name": "Rabbit" } }, { "Timestamp": 200, "Label": { "Instances": [], "Confidence": 61.68227767944336, "Parents": [ { "Name": "Mammal" }, { "Name": "Animal" } ], "Name": "Rodent" } }, { "Timestamp": 200, "Label": { "Instances": [], "Confidence": 74.91455078125, "Parents": [ { "Name": "Plant" } ], "Name": "Tree" } }, { "Timestamp": 200, "Label": { "Instances": [], "Confidence": 82.96375274658203, "Parents": [ { "Name": "Plant" } ], "Name": "Vegetation" } }, { "Timestamp": 200, "Label": { "Instances": [], "Confidence": 60.77254104614258, "Parents": [ { "Name": "Animal" } ], "Name": "Wildlife" } }, { "Timestamp": 200, "Label": { "Instances": [], "Confidence": 74.65026092529297, "Parents": [ { "Name": "Animal" } ], "Name": "Zoo" } }, { "Timestamp": 400, "Label": { "Instances": [], "Confidence": 93.01680755615234, "Parents": [], "Name": "Animal" } }, { "Timestamp": 400, "Label": { "Instances": [], "Confidence": 59.57973098754883, "Parents": [ { "Name": "Wildlife" }, { "Name": "Mammal" }, { "Name": "Animal" } ], "Name": "Deer" } }, { "Timestamp": 400, "Label": { "Instances": [], "Confidence": 78.3232650756836, "Parents": [ { "Name": "Plant" } ], "Name": "Grass" } }, { "Timestamp": 400, "Label": { "Instances": [], "Confidence": 83.4029541015625, "Parents": [ { "Name": "Animal" } ], "Name": "Mammal" } }, { "Timestamp": 400, "Label": { "Instances": [], "Confidence": 52.54331588745117, "Parents": [ { "Name": "Plant" } ], "Name": "Moss" } }, { "Timestamp": 400, "Label": { "Instances": [], "Confidence": 60.17965316772461, "Parents": [], "Name": "Nature" } }, { "Timestamp": 400, "Label": { "Instances": [], "Confidence": 69.8688735961914, "Parents": [], "Name": "Outdoors" } }, { "Timestamp": 400, "Label": { "Instances": [], "Confidence": 87.0956802368164, "Parents": [], "Name": "Plant" } }, { "Timestamp": 400, "Label": { "Instances": [], "Confidence": 56.15915298461914, "Parents": [ { "Name": "Mammal" }, { "Name": "Animal" } ], "Name": "Rodent" } }, { "Timestamp": 400, "Label": { "Instances": [], "Confidence": 74.54655456542969, "Parents": [ { "Name": "Plant" } ], "Name": "Tree" } }, { "Timestamp": 400, "Label": { "Instances": [], "Confidence": 80.760009765625, "Parents": [ { "Name": "Plant" } ], "Name": "Vegetation" } }, { "Timestamp": 400, "Label": { "Instances": [], "Confidence": 65.61725616455078, "Parents": [ { "Name": "Animal" } ], "Name": "Wildlife" } }, { "Timestamp": 400, "Label": { "Instances": [], "Confidence": 80.25843048095703, "Parents": [ { "Name": "Animal" } ], "Name": "Zoo" } }, { "Timestamp": 600, "Label": { "Instances": [], "Confidence": 93.0205078125, "Parents": [], "Name": "Animal" } }, { "Timestamp": 600, "Label": { "Instances": [], "Confidence": 55.57878875732422, "Parents": [ { "Name": "Mammal" }, { "Name": "Animal" }, { "Name": "Wildlife" } ], "Name": "Deer" } }, { "Timestamp": 600, "Label": { "Instances": [], "Confidence": 74.01980590820312, "Parents": [ { "Name": "Plant" } ], "Name": "Grass" } }, { "Timestamp": 600, "Label": { "Instances": [], "Confidence": 82.74808502197266, "Parents": [ { "Name": "Animal" } ], "Name": "Mammal" } }, { "Timestamp": 600, "Label": { "Instances": [], "Confidence": 51.89423751831055, "Parents": [ { "Name": "Plant" } ], "Name": "Moss" } }, { "Timestamp": 600, "Label": { "Instances": [], "Confidence": 55.12575149536133, "Parents": [], "Name": "Nature" } }, { "Timestamp": 600, "Label": { "Instances": [], "Confidence": 66.46729278564453, "Parents": [], "Name": "Outdoors" } }, { "Timestamp": 600, "Label": { "Instances": [], "Confidence": 83.74144744873047, "Parents": [], "Name": "Plant" } }, { "Timestamp": 600, "Label": { "Instances": [], "Confidence": 54.528724670410156, "Parents": [ { "Name": "Mammal" }, { "Name": "Animal" } ], "Name": "Rodent" } }, { "Timestamp": 600, "Label": { "Instances": [], "Confidence": 69.28874206542969, "Parents": [ { "Name": "Plant" } ], "Name": "Tree" } }, { "Timestamp": 600, "Label": { "Instances": [], "Confidence": 74.46104431152344, "Parents": [ { "Name": "Plant" } ], "Name": "Vegetation" } }, { "Timestamp": 600, "Label": { "Instances": [], "Confidence": 61.98466873168945, "Parents": [ { "Name": "Animal" } ], "Name": "Wildlife" } }, { "Timestamp": 600, "Label": { "Instances": [], "Confidence": 85.43339538574219, "Parents": [ { "Name": "Animal" } ], "Name": "Zoo" } }, { "Timestamp": 800, "Label": { "Instances": [], "Confidence": 93.25353240966797, "Parents": [], "Name": "Animal" } }, { "Timestamp": 800, "Label": { "Instances": [], "Confidence": 52.10648727416992, "Parents": [ { "Name": "Mammal" }, { "Name": "Animal" }, { "Name": "Wildlife" } ], "Name": "Deer" } }, { "Timestamp": 800, "Label": { "Instances": [], "Confidence": 75.25298309326172, "Parents": [ { "Name": "Plant" } ], "Name": "Grass" } }, { "Timestamp": 800, "Label": { "Instances": [], "Confidence": 56.252071380615234, "Parents": [ { "Name": "Nature" }, { "Name": "Outdoors" } ], "Name": "Land" } }, { "Timestamp": 800, "Label": { "Instances": [], "Confidence": 83.48870086669922, "Parents": [ { "Name": "Animal" } ], "Name": "Mammal" } }, { "Timestamp": 800, "Label": { "Instances": [], "Confidence": 56.91990280151367, "Parents": [], "Name": "Nature" } }, { "Timestamp": 800, "Label": { "Instances": [], "Confidence": 63.378013610839844, "Parents": [], "Name": "Outdoors" } }, { "Timestamp": 800, "Label": { "Instances": [], "Confidence": 82.87625885009766, "Parents": [], "Name": "Plant" } }, { "Timestamp": 800, "Label": { "Instances": [], "Confidence": 56.5538330078125, "Parents": [ { "Name": "Mammal" }, { "Name": "Animal" } ], "Name": "Rodent" } }, { "Timestamp": 800, "Label": { "Instances": [], "Confidence": 65.29582977294922, "Parents": [ { "Name": "Plant" } ], "Name": "Tree" } }, { "Timestamp": 800, "Label": { "Instances": [], "Confidence": 70.30518341064453, "Parents": [ { "Name": "Plant" } ], "Name": "Vegetation" } }, { "Timestamp": 800, "Label": { "Instances": [], "Confidence": 56.97867965698242, "Parents": [ { "Name": "Animal" } ], "Name": "Wildlife" } }, { "Timestamp": 800, "Label": { "Instances": [], "Confidence": 90.27912139892578, "Parents": [ { "Name": "Animal" } ], "Name": "Zoo" } }, { "Timestamp": 1000, "Label": { "Instances": [], "Confidence": 93.98469543457031, "Parents": [], "Name": "Animal" } }, { "Timestamp": 1000, "Label": { "Instances": [], "Confidence": 76.76435089111328, "Parents": [ { "Name": "Plant" } ], "Name": "Grass" } }, { "Timestamp": 1000, "Label": { "Instances": [], "Confidence": 84.5231704711914, "Parents": [ { "Name": "Animal" } ], "Name": "Mammal" } }, { "Timestamp": 1000, "Label": { "Instances": [], "Confidence": 62.500648498535156, "Parents": [], "Name": "Outdoors" } }, { "Timestamp": 1000, "Label": { "Instances": [], "Confidence": 83.34005737304688, "Parents": [], "Name": "Plant" } }, { "Timestamp": 1000, "Label": { "Instances": [], "Confidence": 56.20842361450195, "Parents": [ { "Name": "Mammal" }, { "Name": "Animal" } ], "Name": "Rodent" } }, { "Timestamp": 1000, "Label": { "Instances": [], "Confidence": 64.69285583496094, "Parents": [ { "Name": "Plant" } ], "Name": "Tree" } }, { "Timestamp": 1000, "Label": { "Instances": [], "Confidence": 71.41803741455078, "Parents": [ { "Name": "Plant" } ], "Name": "Vegetation" } }, { "Timestamp": 1000, "Label": { "Instances": [], "Confidence": 54.289005279541016, "Parents": [ { "Name": "Animal" } ], "Name": "Wildlife" } }, { "Timestamp": 1000, "Label": { "Instances": [], "Confidence": 90.80029296875, "Parents": [ { "Name": "Animal" } ], "Name": "Zoo" } }, { "Timestamp": 1200, "Label": { "Instances": [], "Confidence": 93.2735366821289, "Parents": [], "Name": "Animal" } }, { "Timestamp": 1200, "Label": { "Instances": [], "Confidence": 55.429866790771484, "Parents": [ { "Name": "Mammal" }, { "Name": "Animal" }, { "Name": "Rodent" } ], "Name": "Bunny" } }, { "Timestamp": 1200, "Label": { "Instances": [], "Confidence": 79.42290496826172, "Parents": [ { "Name": "Plant" } ], "Name": "Grass" } }, { "Timestamp": 1200, "Label": { "Instances": [], "Confidence": 84.56671905517578, "Parents": [ { "Name": "Animal" } ], "Name": "Mammal" } }, { "Timestamp": 1200, "Label": { "Instances": [], "Confidence": 54.960243225097656, "Parents": [], "Name": "Outdoors" } }, { "Timestamp": 1200, "Label": { "Instances": [], "Confidence": 83.51371002197266, "Parents": [], "Name": "Plant" } }, { "Timestamp": 1200, "Label": { "Instances": [], "Confidence": 55.429866790771484, "Parents": [ { "Name": "Mammal" }, { "Name": "Animal" }, { "Name": "Rodent" } ], "Name": "Rabbit" } }, { "Timestamp": 1200, "Label": { "Instances": [], "Confidence": 55.429866790771484, "Parents": [ { "Name": "Mammal" }, { "Name": "Animal" } ], "Name": "Rodent" } }, { "Timestamp": 1200, "Label": { "Instances": [], "Confidence": 54.99834060668945, "Parents": [ { "Name": "Plant" } ], "Name": "Tree" } }, { "Timestamp": 1200, "Label": { "Instances": [], "Confidence": 68.27635192871094, "Parents": [ { "Name": "Plant" } ], "Name": "Vegetation" } }, { "Timestamp": 1200, "Label": { "Instances": [], "Confidence": 89.07957458496094, "Parents": [ { "Name": "Animal" } ], "Name": "Zoo" } }, { "Timestamp": 1400, "Label": { "Instances": [], "Confidence": 91.85724639892578, "Parents": [], "Name": "Animal" } }, { "Timestamp": 1400, "Label": { "Instances": [], "Confidence": 54.245033264160156, "Parents": [ { "Name": "Animal" } ], "Name": "Bird" } }, { "Timestamp": 1400, "Label": { "Instances": [], "Confidence": 54.461219787597656, "Parents": [ { "Name": "Mammal" }, { "Name": "Rodent" }, { "Name": "Animal" } ], "Name": "Bunny" } }, { "Timestamp": 1400, "Label": { "Instances": [], "Confidence": 80.37769317626953, "Parents": [ { "Name": "Plant" } ], "Name": "Grass" } }, { "Timestamp": 1400, "Label": { "Instances": [], "Confidence": 51.644439697265625, "Parents": [], "Name": "Green" } }, { "Timestamp": 1400, "Label": { "Instances": [], "Confidence": 81.59090423583984, "Parents": [ { "Name": "Animal" } ], "Name": "Mammal" } }, { "Timestamp": 1400, "Label": { "Instances": [], "Confidence": 50.62571716308594, "Parents": [], "Name": "Outdoors" } }, { "Timestamp": 1400, "Label": { "Instances": [], "Confidence": 51.50884246826172, "Parents": [ { "Name": "Animal" } ], "Name": "Pet" } }, { "Timestamp": 1400, "Label": { "Instances": [], "Confidence": 83.59095764160156, "Parents": [], "Name": "Plant" } }, { "Timestamp": 1400, "Label": { "Instances": [], "Confidence": 54.461219787597656, "Parents": [ { "Name": "Mammal" }, { "Name": "Rodent" }, { "Name": "Animal" } ], "Name": "Rabbit" } }, { "Timestamp": 1400, "Label": { "Instances": [], "Confidence": 54.461219787597656, "Parents": [ { "Name": "Mammal" }, { "Name": "Animal" } ], "Name": "Rodent" } }, { "Timestamp": 1400, "Label": { "Instances": [], "Confidence": 52.90790939331055, "Parents": [ { "Name": "Plant" } ], "Name": "Tree" } }, { "Timestamp": 1400, "Label": { "Instances": [], "Confidence": 69.43914031982422, "Parents": [ { "Name": "Plant" } ], "Name": "Vegetation" } }, { "Timestamp": 1400, "Label": { "Instances": [], "Confidence": 85.9814453125, "Parents": [ { "Name": "Animal" } ], "Name": "Zoo" } }, { "Timestamp": 1600, "Label": { "Instances": [], "Confidence": 92.43132019042969, "Parents": [], "Name": "Animal" } }, { "Timestamp": 1600, "Label": { "Instances": [], "Confidence": 64.58529663085938, "Parents": [ { "Name": "Animal" } ], "Name": "Bird" } }, { "Timestamp": 1600, "Label": { "Instances": [], "Confidence": 77.15897369384766, "Parents": [ { "Name": "Plant" } ], "Name": "Grass" } }, { "Timestamp": 1600, "Label": { "Instances": [], "Confidence": 72.95342254638672, "Parents": [ { "Name": "Animal" } ], "Name": "Mammal" } }, { "Timestamp": 1600, "Label": { "Instances": [], "Confidence": 52.87014389038086, "Parents": [], "Name": "Outdoors" } }, { "Timestamp": 1600, "Label": { "Instances": [], "Confidence": 50.35053634643555, "Parents": [ { "Name": "Animal" } ], "Name": "Pet" } }, { "Timestamp": 1600, "Label": { "Instances": [], "Confidence": 78.85031127929688, "Parents": [], "Name": "Plant" } }, { "Timestamp": 1600, "Label": { "Instances": [], "Confidence": 52.71328353881836, "Parents": [ { "Name": "Plant" } ], "Name": "Tree" } }, { "Timestamp": 1600, "Label": { "Instances": [], "Confidence": 67.42894744873047, "Parents": [ { "Name": "Plant" } ], "Name": "Vegetation" } }, { "Timestamp": 1600, "Label": { "Instances": [], "Confidence": 86.85175323486328, "Parents": [ { "Name": "Animal" } ], "Name": "Zoo" } }, { "Timestamp": 1800, "Label": { "Instances": [], "Confidence": 93.09008026123047, "Parents": [], "Name": "Animal" } }, { "Timestamp": 1800, "Label": { "Instances": [ { "BoundingBox": { "Width": 0.34907206892967224, "Top": 0.08466745913028717, "Left": 0.18098066747188568, "Height": 0.7858670353889465 }, "Confidence": 87.56869506835938 } ], "Confidence": 76.05467224121094, "Parents": [ { "Name": "Animal" } ], "Name": "Bird" } }, { "Timestamp": 1800, "Label": { "Instances": [], "Confidence": 70.30843353271484, "Parents": [ { "Name": "Plant" } ], "Name": "Grass" } }, { "Timestamp": 1800, "Label": { "Instances": [], "Confidence": 62.480037689208984, "Parents": [ { "Name": "Animal" } ], "Name": "Mammal" } }, { "Timestamp": 1800, "Label": { "Instances": [], "Confidence": 72.11174774169922, "Parents": [], "Name": "Plant" } }, { "Timestamp": 1800, "Label": { "Instances": [], "Confidence": 57.063655853271484, "Parents": [ { "Name": "Plant" } ], "Name": "Vegetation" } }, { "Timestamp": 1800, "Label": { "Instances": [], "Confidence": 89.28423309326172, "Parents": [ { "Name": "Animal" } ], "Name": "Zoo" } }, { "Timestamp": 2000, "Label": { "Instances": [], "Confidence": 93.03936004638672, "Parents": [], "Name": "Animal" } }, { "Timestamp": 2000, "Label": { "Instances": [], "Confidence": 54.733070373535156, "Parents": [ { "Name": "Animal" }, { "Name": "Bird" } ], "Name": "Beak" } }, { "Timestamp": 2000, "Label": { "Instances": [ { "BoundingBox": { "Width": 0.358623743057251, "Top": 0.051293861120939255, "Left": 0.1617140769958496, "Height": 0.8379018306732178 }, "Confidence": 92.25386810302734 }, { "BoundingBox": { "Width": 0.7171964049339294, "Top": 0.07636002451181412, "Left": 0.18950828909873962, "Height": 0.8369417190551758 }, "Confidence": 79.23548889160156 }, { "BoundingBox": { "Width": 0.27733245491981506, "Top": 0.002237256383523345, "Left": 0.0026457428466528654, "Height": 0.359114408493042 }, "Confidence": 69.28044128417969 } ], "Confidence": 87.71780395507812, "Parents": [ { "Name": "Animal" } ], "Name": "Bird" } }, { "Timestamp": 2000, "Label": { "Instances": [], "Confidence": 56.62752914428711, "Parents": [ { "Name": "Plant" } ], "Name": "Grass" } }, { "Timestamp": 2000, "Label": { "Instances": [], "Confidence": 52.03357696533203, "Parents": [ { "Name": "Animal" } ], "Name": "Mammal" } }, { "Timestamp": 2000, "Label": { "Instances": [], "Confidence": 62.13314437866211, "Parents": [ { "Name": "Animal" }, { "Name": "Bird" } ], "Name": "Penguin" } }, { "Timestamp": 2000, "Label": { "Instances": [], "Confidence": 59.2055778503418, "Parents": [], "Name": "Plant" } }, { "Timestamp": 2000, "Label": { "Instances": [], "Confidence": 82.38220977783203, "Parents": [ { "Name": "Animal" } ], "Name": "Zoo" } }, { "Timestamp": 2200, "Label": { "Instances": [], "Confidence": 93.34841918945312, "Parents": [], "Name": "Animal" } }, { "Timestamp": 2200, "Label": { "Instances": [], "Confidence": 56.2954216003418, "Parents": [ { "Name": "Animal" }, { "Name": "Bird" } ], "Name": "Beak" } }, { "Timestamp": 2200, "Label": { "Instances": [ { "BoundingBox": { "Width": 0.35335078835487366, "Top": 0.05790714547038078, "Left": 0.15636315941810608, "Height": 0.8401352167129517 }, "Confidence": 93.80805206298828 }, { "BoundingBox": { "Width": 0.7027698755264282, "Top": 0.06268657743930817, "Left": 0.22604236006736755, "Height": 0.8820430040359497 }, "Confidence": 79.38104248046875 }, { "BoundingBox": { "Width": 0.12301671504974365, "Top": 0.08307760953903198, "Left": 0.0, "Height": 0.2910917401313782 }, "Confidence": 67.13511657714844 } ], "Confidence": 92.72528076171875, "Parents": [ { "Name": "Animal" } ], "Name": "Bird" } }, { "Timestamp": 2200, "Label": { "Instances": [], "Confidence": 71.46737670898438, "Parents": [ { "Name": "Animal" }, { "Name": "Bird" } ], "Name": "Penguin" } }, { "Timestamp": 2200, "Label": { "Instances": [], "Confidence": 51.77077865600586, "Parents": [], "Name": "Plant" } }, { "Timestamp": 2200, "Label": { "Instances": [], "Confidence": 82.10325622558594, "Parents": [ { "Name": "Animal" } ], "Name": "Zoo" } }, { "Timestamp": 2400, "Label": { "Instances": [], "Confidence": 94.62578582763672, "Parents": [], "Name": "Animal" } }, { "Timestamp": 2400, "Label": { "Instances": [], "Confidence": 60.21943664550781, "Parents": [ { "Name": "Bird" }, { "Name": "Animal" } ], "Name": "Beak" } }, { "Timestamp": 2400, "Label": { "Instances": [ { "BoundingBox": { "Width": 0.35887837409973145, "Top": 0.06843113154172897, "Left": 0.14393199980258942, "Height": 0.8300164341926575 }, "Confidence": 95.54167938232422 }, { "BoundingBox": { "Width": 0.6892942190170288, "Top": 0.08313678950071335, "Left": 0.2348754107952118, "Height": 0.8645190596580505 }, "Confidence": 73.23546600341797 }, { "BoundingBox": { "Width": 0.2922210097312927, "Top": 0.006060568615794182, "Left": 0.0026990175247192383, "Height": 0.3458782136440277 }, "Confidence": 70.34931945800781 } ], "Confidence": 93.9428482055664, "Parents": [ { "Name": "Animal" } ], "Name": "Bird" } }, { "Timestamp": 2400, "Label": { "Instances": [], "Confidence": 73.18048858642578, "Parents": [ { "Name": "Bird" }, { "Name": "Animal" } ], "Name": "Penguin" } }, { "Timestamp": 2400, "Label": { "Instances": [], "Confidence": 50.8458366394043, "Parents": [], "Name": "Plant" } }, { "Timestamp": 2400, "Label": { "Instances": [], "Confidence": 84.690673828125, "Parents": [ { "Name": "Animal" } ], "Name": "Zoo" } }, { "Timestamp": 2600, "Label": { "Instances": [], "Confidence": 94.66659545898438, "Parents": [], "Name": "Animal" } }, { "Timestamp": 2600, "Label": { "Instances": [], "Confidence": 64.99951934814453, "Parents": [ { "Name": "Bird" }, { "Name": "Animal" } ], "Name": "Beak" } }, { "Timestamp": 2600, "Label": { "Instances": [ { "BoundingBox": { "Width": 0.3687349259853363, "Top": 0.07353333383798599, "Left": 0.1337471306324005, "Height": 0.8266903162002563 }, "Confidence": 94.13199615478516 }, { "BoundingBox": { "Width": 0.6544439196586609, "Top": 0.11645308881998062, "Left": 0.2392454445362091, "Height": 0.8009392619132996 }, "Confidence": 66.88501739501953 }, { "BoundingBox": { "Width": 0.27630096673965454, "Top": 0.0037734773941338062, "Left": 0.0016699910629540682, "Height": 0.3773573935031891 }, "Confidence": 59.929168701171875 } ], "Confidence": 93.93486785888672, "Parents": [ { "Name": "Animal" } ], "Name": "Bird" } }, { "Timestamp": 2600, "Label": { "Instances": [], "Confidence": 71.29080200195312, "Parents": [ { "Name": "Bird" }, { "Name": "Animal" } ], "Name": "Penguin" } }, { "Timestamp": 2600, "Label": { "Instances": [], "Confidence": 54.107154846191406, "Parents": [], "Name": "Plant" } }, { "Timestamp": 2600, "Label": { "Instances": [], "Confidence": 84.884765625, "Parents": [ { "Name": "Animal" } ], "Name": "Zoo" } }, { "Timestamp": 2800, "Label": { "Instances": [], "Confidence": 94.47773742675781, "Parents": [], "Name": "Animal" } }, { "Timestamp": 2800, "Label": { "Instances": [], "Confidence": 64.99027252197266, "Parents": [ { "Name": "Animal" }, { "Name": "Bird" } ], "Name": "Beak" } }, { "Timestamp": 2800, "Label": { "Instances": [ { "BoundingBox": { "Width": 0.38219374418258667, "Top": 0.0916653499007225, "Left": 0.12020987272262573, "Height": 0.7875498533248901 }, "Confidence": 93.746337890625 } ], "Confidence": 93.69722747802734, "Parents": [ { "Name": "Animal" } ], "Name": "Bird" } }, { "Timestamp": 2800, "Label": { "Instances": [], "Confidence": 50.03854751586914, "Parents": [ { "Name": "Plant" } ], "Name": "Grass" } }, { "Timestamp": 2800, "Label": { "Instances": [], "Confidence": 70.23558044433594, "Parents": [ { "Name": "Animal" }, { "Name": "Bird" } ], "Name": "Penguin" } }, { "Timestamp": 2800, "Label": { "Instances": [], "Confidence": 56.43981170654297, "Parents": [], "Name": "Plant" } }, { "Timestamp": 2800, "Label": { "Instances": [], "Confidence": 79.4557876586914, "Parents": [ { "Name": "Animal" } ], "Name": "Zoo" } }, { "Timestamp": 3000, "Label": { "Instances": [], "Confidence": 93.83204650878906, "Parents": [], "Name": "Animal" } }, { "Timestamp": 3000, "Label": { "Instances": [], "Confidence": 56.40727233886719, "Parents": [ { "Name": "Animal" }, { "Name": "Bird" } ], "Name": "Beak" } }, { "Timestamp": 3000, "Label": { "Instances": [ { "BoundingBox": { "Width": 0.3897032141685486, "Top": 0.08954781293869019, "Left": 0.11494585126638412, "Height": 0.7958506941795349 }, "Confidence": 92.31681823730469 }, { "BoundingBox": { "Width": 0.6983879804611206, "Top": 0.09777429699897766, "Left": 0.21733859181404114, "Height": 0.8366428017616272 }, "Confidence": 70.86040496826172 } ], "Confidence": 93.1003189086914, "Parents": [ { "Name": "Animal" } ], "Name": "Bird" } }, { "Timestamp": 3000, "Label": { "Instances": [], "Confidence": 56.9047966003418, "Parents": [ { "Name": "Plant" } ], "Name": "Grass" } }, { "Timestamp": 3000, "Label": { "Instances": [], "Confidence": 64.41144561767578, "Parents": [ { "Name": "Animal" }, { "Name": "Bird" } ], "Name": "Penguin" } }, { "Timestamp": 3000, "Label": { "Instances": [], "Confidence": 62.5224609375, "Parents": [], "Name": "Plant" } }, { "Timestamp": 3000, "Label": { "Instances": [], "Confidence": 50.769561767578125, "Parents": [ { "Name": "Animal" }, { "Name": "Bird" } ], "Name": "Waterfowl" } }, { "Timestamp": 3000, "Label": { "Instances": [], "Confidence": 77.5377426147461, "Parents": [ { "Name": "Animal" } ], "Name": "Zoo" } }, { "Timestamp": 3200, "Label": { "Instances": [], "Confidence": 93.4638900756836, "Parents": [], "Name": "Animal" } }, { "Timestamp": 3200, "Label": { "Instances": [ { "BoundingBox": { "Width": 0.41419991850852966, "Top": 0.0762210413813591, "Left": 0.10949432849884033, "Height": 0.8263728022575378 }, "Confidence": 92.80986785888672 }, { "BoundingBox": { "Width": 0.716448187828064, "Top": 0.07755652815103531, "Left": 0.20436832308769226, "Height": 0.8662547469139099 }, "Confidence": 65.755615234375 } ], "Confidence": 92.78094482421875, "Parents": [ { "Name": "Animal" } ], "Name": "Bird" } }, { "Timestamp": 3200, "Label": { "Instances": [], "Confidence": 62.132137298583984, "Parents": [ { "Name": "Plant" } ], "Name": "Grass" } }, { "Timestamp": 3200, "Label": { "Instances": [], "Confidence": 57.85761642456055, "Parents": [ { "Name": "Animal" }, { "Name": "Bird" } ], "Name": "Penguin" } }, { "Timestamp": 3200, "Label": { "Instances": [], "Confidence": 67.59366607666016, "Parents": [], "Name": "Plant" } }, { "Timestamp": 3200, "Label": { "Instances": [], "Confidence": 53.58717727661133, "Parents": [ { "Name": "Animal" }, { "Name": "Bird" } ], "Name": "Waterfowl" } }, { "Timestamp": 3200, "Label": { "Instances": [], "Confidence": 71.5042495727539, "Parents": [ { "Name": "Animal" } ], "Name": "Zoo" } }, { "Timestamp": 3400, "Label": { "Instances": [], "Confidence": 92.79534912109375, "Parents": [], "Name": "Animal" } }, { "Timestamp": 3400, "Label": { "Instances": [ { "BoundingBox": { "Width": 0.4154813885688782, "Top": 0.07800661772489548, "Left": 0.11423929035663605, "Height": 0.822206437587738 }, "Confidence": 92.5831298828125 }, { "BoundingBox": { "Width": 0.7635012865066528, "Top": 0.10048328340053558, "Left": 0.1485721319913864, "Height": 0.832586407661438 }, "Confidence": 64.7535171508789 }, { "BoundingBox": { "Width": 0.3332541882991791, "Top": 0.0031649908050894737, "Left": 0.0, "Height": 0.39897918701171875 }, "Confidence": 63.40528869628906 } ], "Confidence": 92.79534912109375, "Parents": [ { "Name": "Animal" } ], "Name": "Bird" } }, { "Timestamp": 3400, "Label": { "Instances": [], "Confidence": 61.568965911865234, "Parents": [ { "Name": "Plant" } ], "Name": "Grass" } }, { "Timestamp": 3400, "Label": { "Instances": [], "Confidence": 54.528438568115234, "Parents": [ { "Name": "Bird" }, { "Name": "Animal" } ], "Name": "Penguin" } }, { "Timestamp": 3400, "Label": { "Instances": [], "Confidence": 64.99581146240234, "Parents": [], "Name": "Plant" } }, { "Timestamp": 3400, "Label": { "Instances": [], "Confidence": 51.52910614013672, "Parents": [], "Name": "Water" } }, { "Timestamp": 3400, "Label": { "Instances": [], "Confidence": 55.16550827026367, "Parents": [ { "Name": "Bird" }, { "Name": "Animal" } ], "Name": "Waterfowl" } }, { "Timestamp": 3400, "Label": { "Instances": [], "Confidence": 63.12147903442383, "Parents": [ { "Name": "Animal" } ], "Name": "Zoo" } }, { "Timestamp": 3600, "Label": { "Instances": [], "Confidence": 93.00226593017578, "Parents": [], "Name": "Animal" } }, { "Timestamp": 3600, "Label": { "Instances": [], "Confidence": 57.59740447998047, "Parents": [ { "Name": "Water" } ], "Name": "Aquatic" } }, { "Timestamp": 3600, "Label": { "Instances": [ { "BoundingBox": { "Width": 0.39586177468299866, "Top": 0.09188465774059296, "Left": 0.11620678752660751, "Height": 0.8037567138671875 }, "Confidence": 92.49174499511719 }, { "BoundingBox": { "Width": 0.7152539491653442, "Top": 0.08323486894369125, "Left": 0.2065276801586151, "Height": 0.8613862991333008 }, "Confidence": 63.56003189086914 } ], "Confidence": 93.00226593017578, "Parents": [ { "Name": "Animal" } ], "Name": "Bird" } }, { "Timestamp": 3600, "Label": { "Instances": [], "Confidence": 66.82272338867188, "Parents": [ { "Name": "Plant" } ], "Name": "Grass" } }, { "Timestamp": 3600, "Label": { "Instances": [], "Confidence": 52.20016860961914, "Parents": [], "Name": "Legend of Zelda" } }, { "Timestamp": 3600, "Label": { "Instances": [], "Confidence": 51.27878952026367, "Parents": [ { "Name": "Animal" }, { "Name": "Bird" } ], "Name": "Penguin" } }, { "Timestamp": 3600, "Label": { "Instances": [], "Confidence": 68.2894058227539, "Parents": [], "Name": "Plant" } }, { "Timestamp": 3600, "Label": { "Instances": [], "Confidence": 51.18978500366211, "Parents": [ { "Name": "Animal" } ], "Name": "Sea Life" } }, { "Timestamp": 3600, "Label": { "Instances": [], "Confidence": 59.10639572143555, "Parents": [], "Name": "Water" } }, { "Timestamp": 3600, "Label": { "Instances": [], "Confidence": 55.734310150146484, "Parents": [ { "Name": "Animal" }, { "Name": "Bird" } ], "Name": "Waterfowl" } }, { "Timestamp": 3600, "Label": { "Instances": [], "Confidence": 58.78702926635742, "Parents": [ { "Name": "Animal" } ], "Name": "Zoo" } }, { "Timestamp": 3800, "Label": { "Instances": [], "Confidence": 94.29412078857422, "Parents": [], "Name": "Animal" } }, { "Timestamp": 3800, "Label": { "Instances": [], "Confidence": 61.6994514465332, "Parents": [ { "Name": "Water" } ], "Name": "Aquatic" } }, { "Timestamp": 3800, "Label": { "Instances": [ { "BoundingBox": { "Width": 0.3951866924762726, "Top": 0.11749797314405441, "Left": 0.11840298026800156, "Height": 0.7652148604393005 }, "Confidence": 93.82617950439453 }, { "BoundingBox": { "Width": 0.7300499081611633, "Top": 0.08484145253896713, "Left": 0.19536180794239044, "Height": 0.8607867956161499 }, "Confidence": 70.83800506591797 }, { "BoundingBox": { "Width": 0.33079493045806885, "Top": 0.004325739573687315, "Left": 0.0, "Height": 0.3973473012447357 }, "Confidence": 60.03199005126953 } ], "Confidence": 89.16603088378906, "Parents": [ { "Name": "Animal" } ], "Name": "Bird" } }, { "Timestamp": 3800, "Label": { "Instances": [], "Confidence": 68.32997131347656, "Parents": [ { "Name": "Plant" } ], "Name": "Grass" } }, { "Timestamp": 3800, "Label": { "Instances": [], "Confidence": 55.55190658569336, "Parents": [ { "Name": "Animal" } ], "Name": "Mammal" } }, { "Timestamp": 3800, "Label": { "Instances": [], "Confidence": 68.33173370361328, "Parents": [], "Name": "Plant" } }, { "Timestamp": 3800, "Label": { "Instances": [], "Confidence": 56.37184524536133, "Parents": [ { "Name": "Animal" } ], "Name": "Sea Life" } }, { "Timestamp": 3800, "Label": { "Instances": [], "Confidence": 63.10999298095703, "Parents": [], "Name": "Water" } }, { "Timestamp": 3800, "Label": { "Instances": [], "Confidence": 54.42851257324219, "Parents": [ { "Name": "Animal" }, { "Name": "Bird" } ], "Name": "Waterfowl" } }, { "Timestamp": 3800, "Label": { "Instances": [], "Confidence": 66.54654693603516, "Parents": [ { "Name": "Animal" } ], "Name": "Zoo" } }, { "Timestamp": 4000, "Label": { "Instances": [], "Confidence": 93.92424011230469, "Parents": [], "Name": "Animal" } }, { "Timestamp": 4000, "Label": { "Instances": [], "Confidence": 60.050777435302734, "Parents": [ { "Name": "Water" } ], "Name": "Aquatic" } }, { "Timestamp": 4000, "Label": { "Instances": [ { "BoundingBox": { "Width": 0.38773655891418457, "Top": 0.12097299098968506, "Left": 0.12352852523326874, "Height": 0.7628968954086304 }, "Confidence": 93.34445190429688 }, { "BoundingBox": { "Width": 0.7438606023788452, "Top": 0.11014620214700699, "Left": 0.15275272727012634, "Height": 0.8042365312576294 }, "Confidence": 66.38320922851562 }, { "BoundingBox": { "Width": 0.33265557885169983, "Top": 0.0027915106620639563, "Left": 0.0, "Height": 0.3603067696094513 }, "Confidence": 66.176025390625 } ], "Confidence": 87.69349670410156, "Parents": [ { "Name": "Animal" } ], "Name": "Bird" } }, { "Timestamp": 4000, "Label": { "Instances": [], "Confidence": 65.20291900634766, "Parents": [ { "Name": "Plant" } ], "Name": "Grass" } }, { "Timestamp": 4000, "Label": { "Instances": [], "Confidence": 60.93068313598633, "Parents": [ { "Name": "Animal" } ], "Name": "Mammal" } }, { "Timestamp": 4000, "Label": { "Instances": [], "Confidence": 68.37018585205078, "Parents": [], "Name": "Plant" } }, { "Timestamp": 4000, "Label": { "Instances": [], "Confidence": 55.07685089111328, "Parents": [ { "Name": "Animal" } ], "Name": "Sea Life" } }, { "Timestamp": 4000, "Label": { "Instances": [], "Confidence": 60.50196838378906, "Parents": [], "Name": "Water" } }, { "Timestamp": 4000, "Label": { "Instances": [], "Confidence": 74.234130859375, "Parents": [ { "Name": "Animal" } ], "Name": "Zoo" } }, { "Timestamp": 4200, "Label": { "Instances": [], "Confidence": 94.10073852539062, "Parents": [], "Name": "Animal" } }, { "Timestamp": 4200, "Label": { "Instances": [], "Confidence": 63.6142463684082, "Parents": [ { "Name": "Water" } ], "Name": "Aquatic" } }, { "Timestamp": 4200, "Label": { "Instances": [ { "BoundingBox": { "Width": 0.3899807929992676, "Top": 0.11104846745729446, "Left": 0.13585709035396576, "Height": 0.7780617475509644 }, "Confidence": 69.82188415527344 }, { "BoundingBox": { "Width": 0.7412311434745789, "Top": 0.07028910517692566, "Left": 0.1845070868730545, "Height": 0.8762639164924622 }, "Confidence": 61.936519622802734 } ], "Confidence": 86.78479766845703, "Parents": [ { "Name": "Animal" } ], "Name": "Bird" } }, { "Timestamp": 4200, "Label": { "Instances": [], "Confidence": 61.21832275390625, "Parents": [ { "Name": "Plant" } ], "Name": "Grass" } }, { "Timestamp": 4200, "Label": { "Instances": [], "Confidence": 63.98245620727539, "Parents": [ { "Name": "Animal" } ], "Name": "Mammal" } }, { "Timestamp": 4200, "Label": { "Instances": [], "Confidence": 50.86898422241211, "Parents": [ { "Name": "Bird" }, { "Name": "Animal" } ], "Name": "Penguin" } }, { "Timestamp": 4200, "Label": { "Instances": [], "Confidence": 65.02304077148438, "Parents": [], "Name": "Plant" } }, { "Timestamp": 4200, "Label": { "Instances": [], "Confidence": 56.770111083984375, "Parents": [ { "Name": "Animal" } ], "Name": "Sea Life" } }, { "Timestamp": 4200, "Label": { "Instances": [], "Confidence": 63.6142463684082, "Parents": [], "Name": "Water" } }, { "Timestamp": 4200, "Label": { "Instances": [], "Confidence": 81.00969696044922, "Parents": [ { "Name": "Animal" } ], "Name": "Zoo" } }, { "Timestamp": 4400, "Label": { "Instances": [], "Confidence": 93.77033233642578, "Parents": [], "Name": "Animal" } }, { "Timestamp": 4400, "Label": { "Instances": [], "Confidence": 68.46247863769531, "Parents": [ { "Name": "Water" } ], "Name": "Aquatic" } }, { "Timestamp": 4400, "Label": { "Instances": [], "Confidence": 57.74924087524414, "Parents": [ { "Name": "Animal" }, { "Name": "Bird" } ], "Name": "Beak" } }, { "Timestamp": 4400, "Label": { "Instances": [ { "BoundingBox": { "Width": 0.30696171522140503, "Top": 0.144287109375, "Left": 0.22396674752235413, "Height": 0.7462866306304932 }, "Confidence": 86.27051544189453 } ], "Confidence": 86.7204818725586, "Parents": [ { "Name": "Animal" } ], "Name": "Bird" } }, { "Timestamp": 4400, "Label": { "Instances": [], "Confidence": 60.02400207519531, "Parents": [ { "Name": "Plant" } ], "Name": "Grass" } }, { "Timestamp": 4400, "Label": { "Instances": [], "Confidence": 62.43345260620117, "Parents": [ { "Name": "Animal" } ], "Name": "Mammal" } }, { "Timestamp": 4400, "Label": { "Instances": [], "Confidence": 55.2487907409668, "Parents": [ { "Name": "Animal" }, { "Name": "Bird" } ], "Name": "Penguin" } }, { "Timestamp": 4400, "Label": { "Instances": [], "Confidence": 66.2137680053711, "Parents": [], "Name": "Plant" } }, { "Timestamp": 4400, "Label": { "Instances": [], "Confidence": 58.16676712036133, "Parents": [ { "Name": "Animal" } ], "Name": "Sea Life" } }, { "Timestamp": 4400, "Label": { "Instances": [], "Confidence": 68.46247863769531, "Parents": [], "Name": "Water" } }, { "Timestamp": 4400, "Label": { "Instances": [], "Confidence": 85.66663360595703, "Parents": [ { "Name": "Animal" } ], "Name": "Zoo" } }, { "Timestamp": 4600, "Label": { "Instances": [], "Confidence": 93.82898712158203, "Parents": [], "Name": "Animal" } }, { "Timestamp": 4600, "Label": { "Instances": [], "Confidence": 67.83256530761719, "Parents": [ { "Name": "Water" } ], "Name": "Aquatic" } }, { "Timestamp": 4600, "Label": { "Instances": [], "Confidence": 59.0078239440918, "Parents": [ { "Name": "Bird" }, { "Name": "Animal" } ], "Name": "Beak" } }, { "Timestamp": 4600, "Label": { "Instances": [ { "BoundingBox": { "Width": 0.3058396279811859, "Top": 0.13702332973480225, "Left": 0.2092207968235016, "Height": 0.7735120058059692 }, "Confidence": 89.72554016113281 }, { "BoundingBox": { "Width": 0.17597775161266327, "Top": 0.003950765356421471, "Left": 0.15447595715522766, "Height": 0.19859127700328827 }, "Confidence": 72.09510040283203 }, { "BoundingBox": { "Width": 0.31234344840049744, "Top": 0.007410896942019463, "Left": 0.002845072653144598, "Height": 0.3882608711719513 }, "Confidence": 70.87576293945312 } ], "Confidence": 87.15044403076172, "Parents": [ { "Name": "Animal" } ], "Name": "Bird" } }, { "Timestamp": 4600, "Label": { "Instances": [], "Confidence": 53.97991943359375, "Parents": [ { "Name": "Plant" } ], "Name": "Grass" } }, { "Timestamp": 4600, "Label": { "Instances": [], "Confidence": 61.05377197265625, "Parents": [ { "Name": "Animal" } ], "Name": "Mammal" } }, { "Timestamp": 4600, "Label": { "Instances": [], "Confidence": 57.4560661315918, "Parents": [ { "Name": "Bird" }, { "Name": "Animal" } ], "Name": "Penguin" } }, { "Timestamp": 4600, "Label": { "Instances": [], "Confidence": 60.12495803833008, "Parents": [], "Name": "Plant" } }, { "Timestamp": 4600, "Label": { "Instances": [], "Confidence": 60.812286376953125, "Parents": [ { "Name": "Animal" } ], "Name": "Sea Life" } }, { "Timestamp": 4600, "Label": { "Instances": [], "Confidence": 67.83256530761719, "Parents": [], "Name": "Water" } }, { "Timestamp": 4600, "Label": { "Instances": [], "Confidence": 87.4284439086914, "Parents": [ { "Name": "Animal" } ], "Name": "Zoo" } }, { "Timestamp": 4800, "Label": { "Instances": [], "Confidence": 93.2036361694336, "Parents": [], "Name": "Animal" } }, { "Timestamp": 4800, "Label": { "Instances": [], "Confidence": 64.24736785888672, "Parents": [ { "Name": "Water" } ], "Name": "Aquatic" } }, { "Timestamp": 4800, "Label": { "Instances": [], "Confidence": 61.71356964111328, "Parents": [ { "Name": "Animal" }, { "Name": "Bird" } ], "Name": "Beak" } }, { "Timestamp": 4800, "Label": { "Instances": [ { "BoundingBox": { "Width": 0.30193066596984863, "Top": 0.14368638396263123, "Left": 0.20988507568836212, "Height": 0.7614515423774719 }, "Confidence": 92.7527847290039 }, { "BoundingBox": { "Width": 0.31587615609169006, "Top": 0.007122527342289686, "Left": 0.002042448613792658, "Height": 0.38757696747779846 }, "Confidence": 62.651546478271484 } ], "Confidence": 90.06462860107422, "Parents": [ { "Name": "Animal" } ], "Name": "Bird" } }, { "Timestamp": 4800, "Label": { "Instances": [], "Confidence": 56.28676986694336, "Parents": [ { "Name": "Animal" } ], "Name": "Mammal" } }, { "Timestamp": 4800, "Label": { "Instances": [], "Confidence": 62.933197021484375, "Parents": [ { "Name": "Animal" }, { "Name": "Bird" } ], "Name": "Penguin" } }, { "Timestamp": 4800, "Label": { "Instances": [], "Confidence": 56.79875564575195, "Parents": [], "Name": "Plant" } }, { "Timestamp": 4800, "Label": { "Instances": [], "Confidence": 55.53575897216797, "Parents": [ { "Name": "Animal" } ], "Name": "Sea Life" } }, { "Timestamp": 4800, "Label": { "Instances": [], "Confidence": 64.24736785888672, "Parents": [], "Name": "Water" } }, { "Timestamp": 4800, "Label": { "Instances": [], "Confidence": 86.41097259521484, "Parents": [ { "Name": "Animal" } ], "Name": "Zoo" } }, { "Timestamp": 5000, "Label": { "Instances": [], "Confidence": 93.97762298583984, "Parents": [], "Name": "Animal" } }, { "Timestamp": 5000, "Label": { "Instances": [], "Confidence": 68.58273315429688, "Parents": [ { "Name": "Water" } ], "Name": "Aquatic" } }, { "Timestamp": 5000, "Label": { "Instances": [], "Confidence": 64.67484283447266, "Parents": [ { "Name": "Bird" }, { "Name": "Animal" } ], "Name": "Beak" } }, { "Timestamp": 5000, "Label": { "Instances": [ { "BoundingBox": { "Width": 0.3098117709159851, "Top": 0.17863719165325165, "Left": 0.21030397713184357, "Height": 0.7127665281295776 }, "Confidence": 93.915283203125 } ], "Confidence": 90.92428588867188, "Parents": [ { "Name": "Animal" } ], "Name": "Bird" } }, { "Timestamp": 5000, "Label": { "Instances": [], "Confidence": 52.265533447265625, "Parents": [ { "Name": "Animal" } ], "Name": "Mammal" } }, { "Timestamp": 5000, "Label": { "Instances": [], "Confidence": 64.71392822265625, "Parents": [ { "Name": "Bird" }, { "Name": "Animal" } ], "Name": "Penguin" } }, { "Timestamp": 5000, "Label": { "Instances": [], "Confidence": 54.23300552368164, "Parents": [], "Name": "Plant" } }, { "Timestamp": 5000, "Label": { "Instances": [], "Confidence": 59.28446578979492, "Parents": [ { "Name": "Animal" } ], "Name": "Sea Life" } }, { "Timestamp": 5000, "Label": { "Instances": [], "Confidence": 68.58273315429688, "Parents": [], "Name": "Water" } }, { "Timestamp": 5000, "Label": { "Instances": [], "Confidence": 85.9736099243164, "Parents": [ { "Name": "Animal" } ], "Name": "Zoo" } }, { "Timestamp": 5200, "Label": { "Instances": [], "Confidence": 94.2772216796875, "Parents": [], "Name": "Animal" } }, { "Timestamp": 5200, "Label": { "Instances": [], "Confidence": 64.10331726074219, "Parents": [ { "Name": "Water" } ], "Name": "Aquatic" } }, { "Timestamp": 5200, "Label": { "Instances": [], "Confidence": 67.0080337524414, "Parents": [ { "Name": "Animal" }, { "Name": "Bird" } ], "Name": "Beak" } }, { "Timestamp": 5200, "Label": { "Instances": [ { "BoundingBox": { "Width": 0.38834601640701294, "Top": 0.17760764062404633, "Left": 0.22096411883831024, "Height": 0.7267040610313416 }, "Confidence": 83.2437973022461 } ], "Confidence": 91.10344696044922, "Parents": [ { "Name": "Animal" } ], "Name": "Bird" } }, { "Timestamp": 5200, "Label": { "Instances": [], "Confidence": 66.91466522216797, "Parents": [ { "Name": "Animal" }, { "Name": "Bird" } ], "Name": "Penguin" } }, { "Timestamp": 5200, "Label": { "Instances": [], "Confidence": 52.75025939941406, "Parents": [], "Name": "Plant" } }, { "Timestamp": 5200, "Label": { "Instances": [], "Confidence": 57.49009323120117, "Parents": [ { "Name": "Animal" } ], "Name": "Sea Life" } }, { "Timestamp": 5200, "Label": { "Instances": [], "Confidence": 64.10331726074219, "Parents": [], "Name": "Water" } }, { "Timestamp": 5200, "Label": { "Instances": [], "Confidence": 84.29118347167969, "Parents": [ { "Name": "Animal" } ], "Name": "Zoo" } } ], "LabelModelVersion": "2.0", "JobStatus": "SUCCEEDED", "VideoMetadata": { "Format": "QuickTime / MOV", "FrameRate": 25.0, "Codec": "h264", "DurationMillis": 5280, "FrameHeight": 720, "FrameWidth": 1280 } } --- ### AWS Codebuild Local - URL: https://brianpfeil.com/post/aws-codebuild-local/ - Date: 2019-07-03 - Tags: aws, codebuild code for article  pfeilbr/aws-codebuild-local-playground Resources Announcing Local Build Support for AWS CodeBuild Running # build CodeBuild image locally git clone https://github.com/aws/aws-codebuild-docker-images.git cd aws-codebuild-docker-images/ubuntu/standard/2.0 docker build -t aws/codebuild/standard:2.0 . # pull CodeBuild local agent docker pull amazon/aws-codebuild-local:latest --disable-content-trust=false # cd to directory containing `buildspec.yml` # NOTE: script was previously downloaded to `~/bin/codebuild_build.sh` cd basic-01 codebuild_build.sh -i 'aws/codebuild/standard:2.0' -a 'artifact-output' -c --- ### Golang Debug In Docker with Delve - URL: https://brianpfeil.com/post/golang-debug-in-docker-with-delve/ - Date: 2019-06-24 - Tags: golang, docker code for article  pfeilbr/golang-debug-in-docker-with-delve-playground based on Debugging Go using Delve, Docker and VS Code post make run # or make build first # set breakpoint in vscode on line 18, then Debug | Start Debugging Resources dlv debug cli docs --- ### Cognito Federated to Salesforce and S3 Presigned URL - URL: https://brianpfeil.com/post/cognito-federated-to-salesforce-and-s3-presigned-url/ - Date: 2019-06-13 - Tags: cognito, salesforce, s3 code for article  pfeilbr/cognito-federated-to-salesforce-and-s3-presigned-url-playground Login with Cognito Federated to Salesforce Example see index.html see ~/Dropbox/notes/static-s3-cognito-authentication.md on local machine also for greater detail on setup # local dev # start server. serves static files from current directory '.'. listens on ssl 443 sudo node server.js open https://localhost # on s3 open https://pfeil-static-site-01.s3.amazonaws.com/index.html # on cloudfront open https://d3tz189emgpc7c.cloudfront.net/index.html # copy updated file to s3 aws s3 cp index.html s3://pfeil-static-site-01/index.html # invalidate cloudfront dist after copy # if any login issues # clear site data for https://pfeil-dev-ed.my.salesforce.com in chrome dev tools Pre-signed URL Example see index.js node index.js outputs pre-signed url. (uses ~/.aws/credentials and pre-signed URL generated under that users context/permission) PUT / upload file to S3 via curl -X PUT -T hello.txt -L '<pre-signed url>' --- ### Azure Kubernetes Service AKS - URL: https://brianpfeil.com/post/azure-kubernetes-service-aks/ - Date: 2019-05-18 - Tags: azure, kubernetes code for article  pfeilbr/azure-kubernetes-service-aks-playground learn azure kubernetes service Running # create resource group az group create --name "pfeilbr-aks-01" --location eastus # create cluster az aks create \ --resource-group "pfeilbr-aks-01" \ --name "pfeilbr-aks-01" \ --node-count 1 \ --location eastus \ --enable-addons monitoring \ --generate-ssh-keys # set connectivity to cluster for kubectl az aks get-credentials --resource-group "pfeilbr-aks-01" --name "pfeilbr-aks-01" # list cluster nodes kubectl get nodes # deploy kubectl apply -f azure-vote.yaml # monitor deployment progress kubectl get service azure-vote-front --watch # open "EXTERNAL-IP" in browser to view webapp # delete cluster az group delete --name "pfeilbr-aks-01" --yes --no-wait --- ### Azure Nodejs ARM Resources - URL: https://brianpfeil.com/post/azure-nodejs-arm-resources/ - Date: 2019-05-15 - Tags: azure, nodejs code for article  pfeilbr/azure-nodejs-arm-resources-playground manipulate azure resources (deployments, etc.) programmatically via nodejs --- ### Azure CLI - URL: https://brianpfeil.com/post/azure-cli/ - Date: 2019-05-14 - Tags: azure, cli code for article  pfeilbr/azure-cli-playground azure cli usage examples # version info az -v # help az -h # find examples az find "az functionapp list" # interactive login az login # log in with a service principal using client secret. az login --service-principal -u http://azure-cli-2019-05-09-16-09-40 -p '[CLIENT_SECRET_HERE]' --tenant brianpfeilgmail.onmicrosoft.com az logout az account show az account show | jq '.tenantId' TENANT_ID=$(az account show --query tenantId --output tsv) az account list az account get-access-token az account list-locations # list account fields in table format az account list --output table --query '[].{Name:name, SubscriptionId:id, TenantId:tenantId}' # details of signed in user az ad signed-in-user show # ad az ad -h # ad service principals az ad sp -h # create a service principal and configure its access to Azure az ad sp create-for-rbac # list service principals az ad sp list # list service principals with subset of properties az ad sp list --query "[].{id:appId, tenant:appOwnerTenantId, name:displayName}" # list role assignments for service principal az role assignment list --assignee 'http://azure-cli-2019-05-09-16-09-40' # list role assignments for user (user sign-in name) az role assignment list --assignee 'dev01@brianpfeilgmail.onmicrosoft.com' # list role assignments for user (object id) az role assignment list --assignee '38d35c72-5a26-464c-bbb3-c4487a1d4779' # list role assignments for service principal az role assignment list --assignee "http://service-principal-01" # reset service principal password. if you forgot or lost it, this is the only way to get it. az ad sp credential reset --name "http://service-principal-01" # output # { # "appId": "00133c8e-a08e-490e-ae7c-872ea2debf1e", # "name": "http://service-principal-01", # "password": "09tGSBcRsl_Gml7DI7VkRniFu_r_xxxxxx", # "tenant": "b0579be4-503f-48ca-9bd2-ca22100857dd" # } # login with service principal az login --service-principal --username "http://service-principal-01" --password "09tGSBcRsl_Gml7DI7VkRniFu_r_xxxxxx" --tenant "b0579be4-503f-48ca-9bd2-ca22100857dd" # get access token (bearer) # NOTE: must be logged in with `service-principal-01` above for this example # or may need to add client id to authorize application. see following screenshot # <https://www.evernote.com/l/AAFJLMG88QhDgqehvtS8P-qkVuCcmFFUhCMB/image.png> ACCESS_TOKEN=$(az account get-access-token --resource 'https://brianpfeilmyfn01.azurewebsites.net') # call endpoint protected by azure ad. e.g. functions function endpoint curl --header "Authorization: Bearer ${ACCESS_TOKEN}" https://brianpfeilmyfn01.azurewebsites.net/api/HttpExample # show service principal details az ad sp show --id "http://service-principal-01" # get access token (Bearer) that can be used as `Authorization` header # current user az account get-access-token # for specific resource az account get-access-token --resource 'https://vault.azure.net' # list app registrations az ad app list --query '[].{displayName: displayName}' --output table # create resource group az group create --name "group01" --location eastus # list resource group names az group list | jq '.[].name' # delete resource group az group delete --name "group01" # deploy arm template examples az group deployment create --name "my-deployment-01" --resource-group "my-resource-group-01" --template-file template.json --parameters @parameters.json az group deployment create --name cosmosdbaccountdeployment01 --resource-group group01 --template-file azure-cosmos-db-account.json --parameters name="account01" location="eastus" locationName="East US" defaultExperience="DocumentDB" # interactive/repl mode. immediately exits / doesn't work as of 2019-05-14 az interactive # list all resources az resource list az role assignment list az role definition list az role definition list --custom-role-only true --output json | jq '.[] | {"roleName":.roleName, "roleType":.roleType}' az role definition list | jq '.[].description' # list storage account az storage account list | jq '.[].name' --- ### Azure JS Cognitive Services - URL: https://brianpfeil.com/post/azure-js-cognitive-services/ - Date: 2019-05-13 - Tags: azure, javascript code for article  pfeilbr/azure-js-cognitive-services-playground learn azure cognitive services Examples runHandwrittenOCRForURL() - uses computer vision batchReadFile API to perform handwritten OCR on a image of a form (image sourced from public URL). Resources cognitive-services/computer-vision docs Running Create new cognitive services computer vision resource run az login, then npm run provision OR via portal Copy .env.sample to .env and populate with values via the following steps. CLIENT_ID, DOMAIN SUBSCRIPTION_ID COGNITIVE_SERVICES_ENDPOINT COGNITIVE_SERVICES_KEY run npm start --- ### Azure Nodejs ARM Storage - URL: https://brianpfeil.com/post/azure-nodejs-arm-storage/ - Date: 2019-05-10 - Tags: azure, nodejs code for article  pfeilbr/azure-nodejs-arm-storage-playground manipulate azure storage accounts programmatically via nodejs Resources Azure Storage modules for Node.js --- ### Azure SDK for Node Authentication - URL: https://brianpfeil.com/post/azure-sdk-for-node-authentication/ - Date: 2019-05-10 - Tags: azure, sdk code for article  pfeilbr/azure-sdk-for-node-authentication-playground learn authentication methods for azure sdk for nodejs Service Principal Authentication Based on azure-sdk-for-node/Documentation/Authentication.md clientId, secret, and domain are defined in .env. Copy .env.sample to .env and populate with values via the following steps. getting clientId and domain run az ad sp list getting / creating secret navigate to Portal | Azure Active Directory | App registrations create new client secret if needed OR use existing --- ### Azure Serverless Framework - URL: https://brianpfeil.com/post/azure-serverless-framework/ - Date: 2019-05-10 - Tags: azure, serverless code for article  pfeilbr/azure-serverless-framework-playground learn serverless framework for azure source @ src/services/hello-world see Azure - Credentials for getting setup --- ### Python 2.X WebSocket Client - URL: https://brianpfeil.com/post/python-2.x-websocket-client/ - Date: 2019-05-10 - Tags: python code for article  pfeilbr/python-2.x-websocket-client-playground learn python websocket-client module prerequisites python 2.x node.js: for local websocket server. code is from https://glitch.com/~socketio-basic . uses ws: a Node.js WebSocket library virtualenv: pip install virtualenv or sudo pip install virtualenv running # server (node.js) - run in own terminal (blocking) cd src/server npm i npm start # client - run in separate terminal cd src/client virtualenv venv source venv/bin/activate pip install -r requirements.txt python client.py example run --- ### Azure Cosmos JS SDK - URL: https://brianpfeil.com/post/azure-cosmos-js-sdk/ - Date: 2019-05-08 - Tags: azure, javascript, sdk code for article  pfeilbr/azure-cosmos-js-sdk-playground learn azure cosmos db javascript sdk . Running provision resources # create group az group create --name group01 --location "East US" # create cosmos db account az group deployment create \ --name cosmosdbaccountdeployment01 \ --resource-group group01 \ --template-file arm-templates/azure-cosmos-db-account.json \ --parameters name="account01" location="eastus" locationName="East US" defaultExperience="DocumentDB" copy .env.sample to .env and populate with your information from Keys run npm start --- ### GatsbyJS - URL: https://brianpfeil.com/post/gatsbyjs/ - Date: 2019-03-09 - Tags: gatsbyjs code for article  pfeilbr/gatsbyjs-playground learn Gatsby Plugin custom plugin at plugins/gatsby-source-random original README.md from starter below Gatsby's default starter Kick off your project with this default boilerplate. This starter ships with the main Gatsby configuration files you might need to get up and running blazing fast with the blazing fast app generator for React. Have another more specific idea? You may want to check out our vibrant collection of official and community-created starters . πŸš€ Quick start Create a Gatsby site. Use the Gatsby CLI to create a new site, specifying the default starter. # create a new Gatsby site using the default starter gatsby new my-default-starter https://github.com/gatsbyjs/gatsby-starter-default Start developing. Navigate into your new site’s directory and start it up. cd my-default-starter/ gatsby develop Open the source code and start editing! Your site is now running at http://localhost:8000! Note: You’ll also see a second link: http://localhost:8000/___graphql. This is a tool you can use to experiment with querying your data. Learn more about using this tool in the Gatsby tutorial . Open the my-default-starter directory in your code editor of choice and edit src/pages/index.js. Save your changes and the browser will update in real time! 🧐 What’s inside? A quick look at the top-level files and directories you’ll see in a Gatsby project. . β”œβ”€β”€ node_modules β”œβ”€β”€ src β”œβ”€β”€ .gitignore β”œβ”€β”€ .prettierrc β”œβ”€β”€ gatsby-browser.js β”œβ”€β”€ gatsby-config.js β”œβ”€β”€ gatsby-node.js β”œβ”€β”€ gatsby-ssr.js β”œβ”€β”€ LICENSE β”œβ”€β”€ package-lock.json β”œβ”€β”€ package.json └── README.md /node_modules: This directory contains all of the modules of code that your project depends on (npm packages) are automatically installed. /src: This directory will contain all of the code related to what you will see on the front-end of your site (what you see in the browser) such as your site header or a page template. src is a convention for β€œsource code”. .gitignore: This file tells git which files it should not track / not maintain a version history for. .prettierrc: This is a configuration file for Prettier . Prettier is a tool to help keep the formatting of your code consistent. gatsby-browser.js: This file is where Gatsby expects to find any usage of the Gatsby browser APIs (if any). These allow customization/extension of default Gatsby settings affecting the browser. gatsby-config.js: This is the main configuration file for a Gatsby site. This is where you can specify information about your site (metadata) like the site title and description, which Gatsby plugins you’d like to include, etc. (Check out the config docs for more detail). gatsby-node.js: This file is where Gatsby expects to find any usage of the Gatsby Node APIs (if any). These allow customization/extension of default Gatsby settings affecting pieces of the site build process. gatsby-ssr.js: This file is where Gatsby expects to find any usage of the Gatsby server-side rendering APIs (if any). These allow customization of default Gatsby settings affecting server-side rendering. LICENSE: Gatsby is licensed under the MIT license. package-lock.json (See package.json below, first). This is an automatically generated file based on the exact versions of your npm dependencies that were installed for your project. (You won’t change this file directly). package.json: A manifest file for Node.js projects, which includes things like metadata (the project’s name, author, etc). This manifest is how npm knows which packages to install for your project. README.md: A text file containing useful reference information about your project. πŸŽ“ Learning Gatsby Looking for more guidance? Full documentation for Gatsby lives on the website . Here are some places to start: For most developers, we recommend starting with our in-depth tutorial for creating a site with Gatsby . It starts with zero assumptions about your level of ability and walks through every step of the process. To dive straight into code samples, head to our documentation . In particular, check out the Guides, API Reference, and Advanced Tutorials sections in the sidebar. πŸ’« Deploy --- ### Apollo Client Local State - URL: https://brianpfeil.com/post/apollo-client-local-state/ - Date: 2019-02-27 - Tags: apollo code for article  pfeilbr/apollo-client-local-state-playground Apollo client Local state management see src/App.js --- ### Jest Testing Framework - URL: https://brianpfeil.com/post/jest-testing-framework/ - Date: 2019-02-23 - Tags: jest code for article  pfeilbr/jest-testing-framework-playground --- ### Glitch Local Development - URL: https://brianpfeil.com/post/glitch-local-development/ - Date: 2019-02-20 - Tags: glitch code for article  pfeilbr/glitch-local-development-playground Develop Glitch apps locally. Based on noise-machines/git-glitched . The default Glitch development experience is to use the browser based editor. This allows you to use your favorite local editor. Resources Possible to code locally and push to glitch with git? Code locally, push to glitch via git? Changing a remote’s URL . Glitch changed the git repo path since this project was first created and it had to be updated Initial Project Setup Get the writable git repo location from the Glitch UI via Tools | Git Import, Import, Export Get the URL + User Name. The User Name acts as the username+password. It is a GUID. e.g. a63dde92-f3db-411b-a4cf-e2834ed707e4 Clone glitch repo locally Example Clone (this doesn’t work. username changed) git clone https://a63dde92-f3db-411b-a4cf-e2834ed707e4@api.glitch.com/git/glitch-local-development-playground Development (live reload from local changes) npm run dev You must have the browser based editor view open for live reload to work. Pushing Changes to Github There are two remotes, one for glitch and one for github. The branch named local contains the most recent up to date changes. Run the following to push to github. git push -u github local git push -u github master When viewing on github in browser be sure to change branch to local to see the latest. --- ### React Hooks - URL: https://brianpfeil.com/post/react-hooks/ - Date: 2019-02-11 - Tags: react code for article  pfeilbr/react-hooks-playground learn React Hooks introduced in React 16.8 see src/App.tsx --- ### Pivotal Cloud Foundry - URL: https://brianpfeil.com/post/pivotal-cloud-foundry/ - Date: 2019-01-31 - Tags: paas code for article  pfeilbr/pivotal-cloud-foundry-playground learn Pivotal Cloud Foundry Resources PCF on your Local Workstation with PCF Dev tutorial Cloud Foundry Documentation Why Yes, Cloud Foundry IS Like Your Own Heroku! Example Apps see README.md in subdirectories of apps/ Notes stacks is a prebuilt root file system (rootfs) that supports a specific operating system (https://docs.cloudfoundry.org/devguide/deploy-apps/stacks.html ) e.g. cflinuxfs3: The Linux cflinuxfs3 stack is derived from Ubuntu Bionic 18.04. uses buildpacks to support different languages/run-times (based on heroku buildpack scheme). buildpacks are layered on top of stacks supports Deploy an App with Docker Restaging your app stops your app and restages it, by compiling a new droplet and starting it. Task : A task is an app or script whose code is included as part of a deployed app, but runs independently in its own container. cf cli session cf help # install PCF Dev <https://pivotal.io/pcf-dev> ./pcfdev-v0.30.0+PCF1.11.0-osx # start environment (VM, etc). takes ~5 mins. cf dev start # To begin using PCF Dev, please run: # cf login -a https://api.local.pcfdev.io --skip-ssl-validation # Apps Manager URL: https://apps.local.pcfdev.io # Admin user => Email: admin / Password: admin # Regular user => Email: user / Password: pass cf login -a https://api.local.pcfdev.io --skip-ssl-validation # web UI open https://apps.local.pcfdev.io # login: admin/admin cd ~/dev # fetch sample git clone https://github.com/cloudfoundry-samples/spring-music cd spring-music # build ./gradlew assemble # deploy app cf push --hostname spring-music # show logs cf logs spring-music --recent # stream logs cf logs spring-music # view app resources cf app spring-music # show app ENV cf env spring-music # run a task (simple `ls`) cf run-task spring-music "ls" --name my-task # view task output cf logs spring-music --recent # list tasks for an app (running, failed, succeeded) cf tasks spring-music # cancel a task cf terminate-task spring-music TASK-ID # stop app cf stop spring-music # show routes cf routes # stop environment (VMs, etc.) cf dev stop Screenshots login --- ### fswatch - URL: https://brianpfeil.com/post/fswatch/ - Date: 2019-01-29 - Tags: tools code for article  pfeilbr/fswatch-playground learn fswatch file change monitor github @ emcrisostomo/fswatch Example Script see watch.sh Example Usage # get file path information for changed files fswatch -0 . # print file path information for each changed file on separate line fswatch -0 . | xargs -0 -n 1 -I {} echo {} --- ### Storybook UI Development Environment - URL: https://brianpfeil.com/post/storybook-ui-development-environment/ - Date: 2019-01-25 - Tags: react, ui, ui-components code for article  pfeilbr/storybook-ui-development-environment-playground learn Storybook UI Development Environment Running add stories to src/components/*.stories.js run storybook via npm run storybook --- ### Jupyter Kernel Gateway - URL: https://brianpfeil.com/post/jupyter-kernel-gateway/ - Date: 2019-01-19 - Tags: jupyter code for article  pfeilbr/jupyter-kernel-gateway-playground learn Jupyter Kernel Gateway Development # build image docker build -t pfeilbr/jupyter-kernel-gateway . # run docker run -it --rm -p 8888:8888 pfeilbr/jupyter-kernel-gateway # view swagger json open http://0.0.0.0:8888/api/swagger.json # expose to public internet ngrok http 8888 # view in swagger ui open https://petstore.swagger.io # enter https://NGROK_HOST/api/swagger.json in explore # e.g. # https://0278ef82.ngrok.io/api/swagger.json # needed to add CORS params in `docker-stacks-image/Dockerfile` `CMD` statement # running `browser-client-example.js` in chrome DevTools open http://0.0.0.0:8888/api/swagger.json # open DevTools and run code in Sources | Snippets # push docker image docker push pfeilbr/jupyter-kernel-gateway Screenshots --- ### AWS API Gateway WebSockets Serverless - URL: https://brianpfeil.com/post/aws-api-gateway-websockets-serverless/ - Date: 2019-01-08 - Tags: aws, serverless code for article  pfeilbr/aws-api-gateway-websockets-serverless-playground update 2019-10-03: websocket events are now “natively supported in serverless framework” no longer need to use serverless-websockets-plugin. see https://serverless.com/framework/docs/providers/aws/events/websocket/ below uses serverless-websockets-plugin learn AWS API Gateway Websockets with serverless framework Using API Gateway WebSockets with the Serverless Framework serverless-websockets-plugin/example Control Access to a WebSocket API in API Gateway API Gateway Limits for Configuring and Running a WebSocket API --- ### Rust - URL: https://brianpfeil.com/post/rust/ - Date: 2018-12-21 - Tags: rust code for article  pfeilbr/rust-playground learn Rust programming language Prerequisites nodemon - enables compile and run on file change. Development # run on file(s) change (ensure `nodemon` is installed) # turn off dead_code warning nodemon --exec "RUSTFLAGS=\"$RUSTFLAGS -A dead_code\" cargo run" src/main.rs # watch tests (run `cargo test` when any file changes) cargo watch -x check -x test # show output (e.g. `println!`) cargo watch -x 'test -- --nocapture' # to get a stack backtrace on panic RUST_BACKTRACE=1 cargo run Notes compiles to LLVM IR guarantees memory safety no GC. compiler generates code at compile time to clean up of memory for variable that go out of scope, etc. no overhead at runtime for this. this is what makes it performant. zero-cost abstractions excellent compiler messages if you managed to appease the compiler, there’s a good chance your code will work–barring any logic flaws out-of-bounds access at runtime causes the program to immediately stop ownership rules apply across multiple threads either one mutable reference (read/write) OR many immutable (read-only) references no Null, only Option traits are like interfaces in other languages includes macros for metaprogramming dtolnay/cargo-expand: Subcommand to show result of macro expansion concurrency support via threads, channels, mutex can call C code via extern Lifetimes are, in some ways, Rust’s most distinctive feature rust lifetimes are not in other languages. essentially tells the compiler when it can generate code to free memory/variables. the responsibility the runtime GC in other languages is shifted to you to take care of and think of (zero cost abstraction? not so much … there is the human cost). see Understanding lifetimes in Rust - LogRocket Blog for details. cargo workspaces Option and Result see Option and Result | Learning Rust Option // An output can have either Some value or no value/ None. enum Option<T> { // T is a generic and it can contain any type of value. Some(T), None, } Result If a function can produce an error, we have to use a Result used for all I/O, conversions, etc. anything that can fail. // A result can represent either success/ Ok or failure/ Err. enum Result<T, E> { // T and E are generics. T can contain any type of value, E can be any error. Ok(T), Err(E), } Smart Pointers (Box, Rc, Arc, Cell) Box is for single ownership (single thread, stored on heap). Rc is for multiple ownership (single thread, stored on heap). Arc is for multiple ownership, but threadsafe (multi-thread, stored on heap). Cell is for “interior mutability” for Copy types; that is, when you need to mutate something behind a &T. Resources The Rust Programming Language book Rust by Example Table of Contents - Rust Cookbook rust-analyzer - Bringing a great IDE experience to the Rust programming language. GitHub - watchexec/cargo-watch: Watches over your Cargo project's source. Rust: A Language for the Next 40 Years - Carol Nichols A collection of notable Rust blog posts Understanding lifetimes in Rust - LogRocket Blog The Little Book of Rust Macros Yoututbe | Jon Gjengset - rust videos dtolnay/cargo-expand: Subcommand to show result of macro expansion Wrapper Types in Rust: Choosing Your Guarantees --- ### AWS CodePipeline - URL: https://brianpfeil.com/post/aws-codepipeline/ - Date: 2018-12-20 - Tags: aws, codepipeline code for article  pfeilbr/aws-codepipeline-playground learn AWS CodePipeline Using CodeBuild for a “custom” CodePipeline deploy phase CodeBuild gives you a amz linux server and you can run whatever you want. This is an example using it for the “deployment” phase. During the “build” phase in buildspec.yml, the post_build command runs mv deploy-buildspec.yml buildspec.yml. The newly renamed buildspec.yml is what gets used in the “deploy” phase. Exporting Pipeline Definition/Metadata via CodePipeline.getPipeline method node scripts/get-pipeline.js > /pipeline-definition-export.json --- ### Twit Twitter API Client - URL: https://brianpfeil.com/post/twit-twitter-api-client/ - Date: 2018-11-19 - Tags: twitter, api, nodejs code for article  pfeilbr/twit-twitter-api-client-playground learn twit , the Twitter API Client for node. Supports both the REST and Streaming API. see index.js for examples Running cp .env.sample .env # update .env with your twitter developer consumer and access values (e.g. app defined @ https://developer.twitter.com) # Uncomment examples in `index.js` to run # dev / live reload npm run dev Resources https://github.com/ttezel/twit https://developer.twitter.com/en/docs --- ### AWS Amplify JS App - URL: https://brianpfeil.com/post/aws-amplify-js-app/ - Date: 2018-11-13 - Tags: aws, amplify, javascript code for article  pfeilbr/aws-amplify-js-app-playground learn AWS Amplify Framework getting started walkthrough @ https://aws-amplify.github.io/docs/js/start?ref=amplify-js-btn&platform=purejs Session mkdir aws-amplify-js-app-playground cd aws-amplify-js-app-playground/\\n mkdir src touch package.json index.html webpack.config.js src/app.js touch README.md npm i npm start npm install --save aws-amplify amplify init amplify status amplify add analytics amplify push amplify add hosting amplify publish amplify status --- ### AWS ECR - URL: https://brianpfeil.com/post/aws-ecr/ - Date: 2018-11-13 - Tags: aws, ecr code for article  pfeilbr/aws-ecr-playground learn Amazon Elastic Container Registry Docker Basics for Amazon ECR tutorial Session cd ~/projects mkdir aws-ecr-playground code aws-ecr-playground touch Dockerfile touch README.md docker build -t hello-world . touch .gitignore docker build -t hello-world . docker-machine ip docker run -p 80:80 hello-world docker-machine ip aws ecr create-repository --repository-name hello-world docker tag hello-world 529276214230.dkr.ecr.us-east-1.amazonaws.com/hello-world $(aws ecr get-login --no-include-email) docker push 529276214230.dkr.ecr.us-east-1.amazonaws.com/hello-world --- ### Dynogels - URL: https://brianpfeil.com/post/dynogels/ - Date: 2018-10-22 - Tags: aws, javascript, dynamodb code for article  pfeilbr/dynogels-playground learn dynogels - DynamoDB data mapper for node.js this code uses dynogels-promisified to leverage async/await. Promisifies by appending Async to each method name. --- ### Node Jenkins API - URL: https://brianpfeil.com/post/node-jenkins-api/ - Date: 2018-10-18 - Tags: jenkins code for article  pfeilbr/node-jenkins-api-playground project to learn node-jenkins Install npm install jenkins copy .env.sample to .env and update with “your” values Run Examples node index.js shows getting details of job and building a job --- ### Autotools - URL: https://brianpfeil.com/post/autotools/ - Date: 2018-10-10 - Tags: autotools code for article  pfeilbr/autotools-playground learn Autotools Autotools is composed of several tools: aclocal, autoconf, automake and others, belonging to two packages: Automake and Autoconf Prerequisites brew install automake Build and Run brew install automake mkdir hello_world cd hello_world touch main.c touch configure.ac touch Makefile.am aclocal autoconf automake --add-missing ./configure make ./helloworld # make distrubution .tar.gz make dist # test that the distribution tarball make distcheck # Use Makefile to install the program make install Resources The magic behind configure, make, make install Using Autotools --- ### Vcpkg - URL: https://brianpfeil.com/post/vcpkg/ - Date: 2018-09-27 - Tags: cpp, package-manager code for article  pfeilbr/vcpkg-playground learn vcpkg C++ package manager see README.md file(s) in sub diredctories Resources https://github.com/Microsoft/vcpkg https://docs.microsoft.com/en-us/cpp/vcpkg --- ### AWS SDK CPP - URL: https://brianpfeil.com/post/aws-sdk-cpp/ - Date: 2018-09-25 - Tags: aws, sdk, cpp code for article  pfeilbr/aws-sdk-cpp-playground learn aws-sdk-app C++ SDK see README.md file(s) in subdirectories of src install aws-sdk-cpp cd ~/dev git clone https://github.com/aws/aws-sdk-cpp.git cd aws-sdk-cpp mkdir build cd build cmake .. make make install build and run example cd src/s3-example mkdir -p build && cd build cmake .. && make # run ./s3-example output Key Install File Locations /usr/local/lib/cmake/AWSSDK - cmake modules /usr/local/include - headers installed /usr/local/lib - libraries installed (e.g. libaws-cpp-sdk-core.dylib) --- ### Google Test CPP Test Framework - URL: https://brianpfeil.com/post/google-test-cpp-test-framework/ - Date: 2018-09-25 - Tags: google, cpp code for article  pfeilbr/google-test-cpp-test-framework-playground learn googletest / gtest C++ test framework see README.md in each subdirectory under src gtest install # download source cd ~/dev curl -LO https://github.com/google/googletest/archive/release-1.8.1.zip unzip release-1.8.1.zip cd googletest-release-1.8.1 mkdir build cd build cmake .. make make install --- ### Modern CPP - URL: https://brianpfeil.com/post/modern-cpp/ - Date: 2018-09-24 - Tags: cpp code for article  pfeilbr/modern-cpp-playground based on nesteruk/ModernCpp playground.cpp - main file scripts/ - build, run, etc. scripts NOTE (to self): developed and ran on 2016 MacBook Pro. Depends on local file locations. development build and run on file contents change (live reload) # start watcher ./scripts/build-and-run-on-change playground.cpp # edit and save `playground.cpp` for live reload --- ### V8 JavaScript Engine - URL: https://brianpfeil.com/post/v8-javascript-engine/ - Date: 2018-09-21 - Tags: v8 code for article  pfeilbr/v8-javascript-engine-playground project to learn and develop with V8 Root V8 source directory is @ ~/dev/v8 on local machine Build Source and Sample(s) Follow steps at https://github.com/v8/v8/wiki/Building-from-Source up until, but not inlcuding tools/dev/v8gen.py ... generate “debug” build files tools/dev/v8gen.py x64.debug change contents of v8/out.gn/x64.debug/args.gn to is_debug = true target_cpu = "x64" v8_enable_backtrace = true v8_enable_slow_dchecks = true v8_optimized_debug = false v8_monolithic = true v8_use_external_startup_data = false is_component_build = false needed to enable debugging compile with ninja -C out.gn/x64.debug compile debug version of samples/hello-world.cc with g++ -g -I. -Iinclude samples/hello-world.cc -o out.gn/x64.debug/hello_world -lv8 -lv8_libbase -lv8_libplatform -licuuc -licui18n -Lout.gn/x64.debug/ -pthread -std=c++0x -g enables debug version move into output directory pushd out.gn/x64.debug set library path and run LD_LIBRARY_PATH=./ ./hello_world return to v8 root popd Developing configure: modify script/env for your setup build: ./scripts/build src/playground.cc run: ./scripts/run src/playground.cc build and run on change (dev): ./scripts/build-and-run-on-change src/playground.cc vscode debugging launch.json configuration { "version": "0.2.0", "configurations": [ { "name": "(lldb) Launch", "type": "cppdbg", "request": "launch", "program": "${workspaceFolder}/out.gn/x64.debug/hello_world", "args": [], "stopAtEntry": false, "cwd": "${workspaceFolder}/out.gn/x64.debug", "environment": [{"name": "LD_LIBRARY_PATH", "value": "./"}], "externalConsole": true, "MIMode": "lldb" } ] } ${workspaceFolder} is v8 root directory --- ### CMake - URL: https://brianpfeil.com/post/cmake/ - Date: 2018-09-20 - Tags: cmake code for article  pfeilbr/cmake-playground project to learn CMake see README.md file for each sub directory --- ### Lerna - URL: https://brianpfeil.com/post/lerna/ - Date: 2018-09-02 - Tags: nodejs, npm, tools code for article  pfeilbr/lerna-playground learn lerna A tool for managing JavaScript projects with multiple packages Prerequisites install lerna npm install --global lerna Example Workflow # init lerna repo mkdir lerna-playground cd lerna-playground git init lerna init # create mathlib cd packages mkdir mathlib cd mathlib npm init -f touch index.js # write mathlib code # create client that depends on mathlib cd .. mkdir client cd client npm init -f # add mathlib lerna add mathlib # create client code touch index.js # run client node index.js --- ### VueJS webpack - URL: https://brianpfeil.com/post/vuejs-webpack/ - Date: 2018-09-01 - Tags: vuejs code for article  pfeilbr/vuejs-webpack-playground A Vue.js project Build Setup # install dependencies npm install # serve with hot reload at localhost:8080 npm run dev # build for production with minification npm run build # build for production and view the bundle analyzer report npm run build --report # run unit tests npm run unit # run e2e tests npm run e2e # run all tests npm test For a detailed explanation on how things work, check out the guide and docs for vue-loader . --- ### WebAssembly - URL: https://brianpfeil.com/post/webassembly/ - Date: 2018-08-22 - Tags: webassembly code for article  pfeilbr/webassembly-playground see README.md in sub directories --- ### Lua C API - URL: https://brianpfeil.com/post/lua-c-api/ - Date: 2018-07-24 - Tags: lua, c code for article  pfeilbr/Lua-c-api-playground learn and experiment with Lua C API see Lua-c-api-playground/main.cpp Xcode Project Config --- ### AWS Appsync - URL: https://brianpfeil.com/post/aws-appsync/ - Date: 2018-06-09 - Tags: aws code for article  pfeilbr/aws-appsync-playground learn and experiment with AWS AppSync , the managed GraphQL service. Components GraphQL Schema Resolvers resolver request/response mapping templates DataSources DynamoDB, ElasticSearch, Lambda (as of 2018-06-09) AppSync Resource Types API AWS::AppSync::GraphQLApi createGraphqlApi Schema AWS::AppSync::GraphQLSchema createType startSchemaCreation DataSource AWS::AppSync::DataSource createDataSource Resolver AWS::AppSync::Resolver createResolver FunctionConfiguration AWS::AppSync::FunctionConfiguration createFunction ApiKey AWS::AppSync::ApiKey createApiKey ApiCache AWS::AppSync::ApiCache createApiCache Resources nabeelthedev/chat-app - example using CloudFormation for all AppSync and other resource provisioning. sbstjn/appsync-resolvers-example - Example project for AppSync, GraphQL, and AWS Lambda resolvers using Go. sbstjn/go-appsync-graphql-cloudformation - AWS AppSync GraphQL API Proxy with Lambda, CloudFormation, and SAM AWS JavaScript SDK | AWS.AppSync Build your own multi-user photo album app with React, GraphQL, and AWS Amplify gabehollombe-aws/react-graphql-amplify-blog-post serverless-appsync-plugin example service urql - GraphQL client --- ### JSON Schema - URL: https://brianpfeil.com/post/json-schema/ - Date: 2018-06-06 - Tags: json code for article  pfeilbr/json-schema-playground project to learn JSON schema see main.spec.ts for examples run examples/tests npm run test or npm run test:watch uses Ajv schema validator --- ### Jenkins Pipeline GitHub - URL: https://brianpfeil.com/post/jenkins-pipeline-github/ - Date: 2018-05-30 - Tags: jenkins code for article  pfeilbr/jenkins-pipeline-github-playground learn and experiment with Jenkins automation server. # run docker run -p 8080:8080 -p 50000:50000 -v jenkins_home:/var/jenkins_home --privileged jenkins/jenkins:lts # open web ui open http://localhost:8080/job/projects/job/jenkins-pipeline-github-playground/configure github project example https://github.com/pfeilbr/jenkins-pipeline-github-playground screenshots --- ### Alexa Skills - URL: https://brianpfeil.com/post/alexa-skills/ - Date: 2018-05-17 - Tags: aws, alexa code for article  pfeilbr/alexa-skills-playground sage skill data driven skill. modify config.js then deploy NOTE: skills/sage/lambda/custom/config.js is generated. do not edit directly! # build cd skills/sage/lambda/custom npm install # deploy cd skills/sage npm run deploy npm build generates models/en-US.json and lambda/custom/config.js scripts/run performs file generation Development Workflow # configure ask cli ask init # create new skill ask new # clone an existing skill ask clone --skill-id amzn1.ask.skill.1aee9e3d-ec1c-4e07-8239-19fea42c3036 # deploy ask deploy Payload Examples LaunchRequest Payload Example (simplified) { "type": "LaunchRequest", "requestId": "amzn1.echo-api.request.310bdf96-793a-4e14-b586-e8f680ffe52f", "timestamp": "2018-05-15T17:32:42Z", "locale": "en-US", "shouldLinkResultBeReturned": false } LaunchRequest Payload Example (full) { "requestEnvelope": { "version": "1.0", "session": { "new": true, "sessionId": "amzn1.echo-api.session.314b8ce7-06d6-4cd5-8822-7644d2f5a799", "application": { "applicationId": "amzn1.ask.skill.1aee9e3d-ec1c-4e07-8239-19fea42c3036" }, "user": { "userId": "amzn1.ask.account.AFPKUMM66XYI7MZMSUYLSGUUJDILTGYDXEMEL2WFFLEIG2TIQIE3NSIDODHS7VW2UKCA4EYR2QQS6QDZA5FAKMIQI5ASRRAEEXMNFMF6KSYQYM6YSVZ4UGU52L3SMJ3T73LOQPSFYBTIUQRPBLOFFQPEGIW35AP2FMQP5IP55QR4M6LZ2RGEEHNORLF6GWJOD3VSZIQGNLIC4MI" } }, "context": { "AudioPlayer": { "playerActivity": "IDLE" }, "Display": { "token": "" }, "System": { "application": { "applicationId": "amzn1.ask.skill.1aee9e3d-ec1c-4e07-8239-19fea42c3036" }, "user": { "userId": "amzn1.ask.account.AFPKUMM66XYI7MZMSUYLSGUUJDILTGYDXEMEL2WFFLEIG2TIQIE3NSIDODHS7VW2UKCA4EYR2QQS6QDZA5FAKMIQI5ASRRAEEXMNFMF6KSYQYM6YSVZ4UGU52L3SMJ3T73LOQPSFYBTIUQRPBLOFFQPEGIW35AP2FMQP5IP55QR4M6LZ2RGEEHNORLF6GWJOD3VSZIQGNLIC4MI" }, "device": { "deviceId": "amzn1.ask.device.AEOU6OLEMMGRPZDLADWRH5FELI7QBDGR4XXXM2EGU53EJQWESEUGNQF7WZKY3SUJFR2SFLQ7DB2KUJ4QAXTYH2DJGWR3K36SPOIGNFVPTFHPHN34LBW234VMAU5IM5PJUFYCAI3QHPTUBAXZT6VIBPFBMXXX3BLQFNRO3IJ6WFVXBPDFAKRP6", "supportedInterfaces": { "AudioPlayer": {}, "Display": { "templateVersion": "1.0", "markupVersion": "1.0" } } }, "apiEndpoint": "https://api.amazonalexa.com", "apiAccessToken": "<REMOVED>" } }, "request": { "type": "LaunchRequest", "requestId": "amzn1.echo-api.request.310bdf96-793a-4e14-b586-e8f680ffe52f", "timestamp": "2018-05-15T17:32:42Z", "locale": "en-US", "shouldLinkResultBeReturned": false } }, "context": { "callbackWaitsForEmptyEventLoop": true, "logGroupName": "/aws/lambda/aws-serverless-repository-alexaskillskitnodejsfact-6CKGX2O4QCIR", "logStreamName": "2018/05/15/[$LATEST]239394be80f1496387bc6961cede7161", "functionName": "aws-serverless-repository-alexaskillskitnodejsfact-6CKGX2O4QCIR", "memoryLimitInMB": "128", "functionVersion": "$LATEST", "invokeid": "f8d0e1c0-5865-11e8-9638-0df1b13913aa", "awsRequestId": "f8d0e1c0-5865-11e8-9638-0df1b13913aa", "invokedFunctionArn": "arn:aws:lambda:us-east-1:529276214230:function:aws-serverless-repository-alexaskillskitnodejsfact-6CKGX2O4QCIR" }, "attributesManager": {}, "responseBuilder": {}, "serviceClientFactory": { "apiConfiguration": { "apiClient": {}, "apiEndpoint": "https://api.amazonalexa.com", "authorizationValue": "<REMOVED>" } } } IntentRequest Payload Example (simplified) { "type": "IntentRequest", "requestId": "amzn1.echo-api.request.7bfbb240-78c3-4622-b255-53a8dcc34376", "timestamp": "2018-05-15T17:34:12Z", "locale": "en-US", "intent": { "name": "GetNewFactIntent", "confirmationStatus": "NONE" } } IntentRequest Payload Example (full) { "requestEnvelope": { "version": "1.0", "session": { "new": true, "sessionId": "amzn1.echo-api.session.204f5c22-869d-406e-bd59-9d866477cfcc", "application": { "applicationId": "amzn1.ask.skill.1aee9e3d-ec1c-4e07-8239-19fea42c3036" }, "user": { "userId": "amzn1.ask.account.AFPKUMM66XYI7MZMSUYLSGUUJDILTGYDXEMEL2WFFLEIG2TIQIE3NSIDODHS7VW2UKCA4EYR2QQS6QDZA5FAKMIQI5ASRRAEEXMNFMF6KSYQYM6YSVZ4UGU52L3SMJ3T73LOQPSFYBTIUQRPBLOFFQPEGIW35AP2FMQP5IP55QR4M6LZ2RGEEHNORLF6GWJOD3VSZIQGNLIC4MI" } }, "context": { "AudioPlayer": { "playerActivity": "IDLE" }, "Display": { "token": "" }, "System": { "application": { "applicationId": "amzn1.ask.skill.1aee9e3d-ec1c-4e07-8239-19fea42c3036" }, "user": { "userId": "amzn1.ask.account.AFPKUMM66XYI7MZMSUYLSGUUJDILTGYDXEMEL2WFFLEIG2TIQIE3NSIDODHS7VW2UKCA4EYR2QQS6QDZA5FAKMIQI5ASRRAEEXMNFMF6KSYQYM6YSVZ4UGU52L3SMJ3T73LOQPSFYBTIUQRPBLOFFQPEGIW35AP2FMQP5IP55QR4M6LZ2RGEEHNORLF6GWJOD3VSZIQGNLIC4MI" }, "device": { "deviceId": "amzn1.ask.device.AEOU6OLEMMGRPZDLADWRH5FELI7QBDGR4XXXM2EGU53EJQWESEUGNQF7WZKY3SUJFR2SFLQ7DB2KUJ4QAXTYH2DJGWR3K36SPOIGNFVPTFHPHN34LBW234VMAU5IM5PJUFYCAI3QHPTUBAXZT6VIBPFBMXXX3BLQFNRO3IJ6WFVXBPDFAKRP6", "supportedInterfaces": { "AudioPlayer": {}, "Display": { "templateVersion": "1.0", "markupVersion": "1.0" } } }, "apiEndpoint": "https://api.amazonalexa.com", "apiAccessToken": "<REMOVED>" } }, "request": { "type": "IntentRequest", "requestId": "amzn1.echo-api.request.7bfbb240-78c3-4622-b255-53a8dcc34376", "timestamp": "2018-05-15T17:34:12Z", "locale": "en-US", "intent": { "name": "GetNewFactIntent", "confirmationStatus": "NONE" } } }, "context": { "callbackWaitsForEmptyEventLoop": true, "logGroupName": "/aws/lambda/aws-serverless-repository-alexaskillskitnodejsfact-6CKGX2O4QCIR", "logStreamName": "2018/05/15/[$LATEST]239394be80f1496387bc6961cede7161", "functionName": "aws-serverless-repository-alexaskillskitnodejsfact-6CKGX2O4QCIR", "memoryLimitInMB": "128", "functionVersion": "$LATEST", "invokeid": "2e99cd8c-5866-11e8-ac6c-091c252114f4", "awsRequestId": "2e99cd8c-5866-11e8-ac6c-091c252114f4", "invokedFunctionArn": "arn:aws:lambda:us-east-1:529276214230:function:aws-serverless-repository-alexaskillskitnodejsfact-6CKGX2O4QCIR" }, "attributesManager": {}, "responseBuilder": {}, "serviceClientFactory": { "apiConfiguration": { "apiClient": {}, "apiEndpoint": "https://api.amazonalexa.com", "authorizationValue": "<REMOVED>" } } } --- ### GraphQL - URL: https://brianpfeil.com/post/graphql/ - Date: 2018-04-30 - Tags: graphql code for article  pfeilbr/graphql-playground project to learn and experiment with graphql using Apollo server location - `src/server' entrypoint - src/server/index.ts dev (reload on change) - npm run dev run - npm start client location - `src/client' entrypoint - src/client/App.tsx run - npm start ensure server is running on port 3000 Examples Run via GraphiQL @ http://localhost:3000/graphiql query { books { id, title, author } } mutation { createBook(title: "How To", author: "John Doe") { id, title author } } subscription onBookAdded { bookAdded { id title author } } Screenshots subscriptiion push over websocket Scratch --- ### SFML MacOS - URL: https://brianpfeil.com/post/sfml-macos/ - Date: 2018-02-27 - Tags: macos, graphics, cpp code for article  pfeilbr/sfml-macos-playground learn SFML , the Simple and Fast Multimedia Library --- ### AWS IoT - URL: https://brianpfeil.com/post/aws-iot/ - Date: 2018-02-10 - Tags: aws, iot code for article  pfeilbr/aws-iot-playground learn and experiment with AWS IoT uses aws-iot-device-sdk-js for programmatic access. see index.js copy .env.template to .env and populate with your values Object Hierarchy thing -> certificate -> policy --- ### AWS IAM - URL: https://brianpfeil.com/post/aws-iam/ - Date: 2018-02-08 - Tags: aws, iam code for article  pfeilbr/aws-iam-playground serverless-project-policy-templates/ - serverless project policy templates for the various roles (admin, support, lambda, apigateway). they aim to isolate permissions by matching resources against a project prefix --- ### AWS EMR - URL: https://brianpfeil.com/post/aws-emr/ - Date: 2018-02-03 - Tags: aws, emr code for article  pfeilbr/aws-emr-playground examples of using aws-emr. hive, pig, spark, etc. Dynamodb Export/Import to/from S3 Examples -- link hive table to dynamodb table CREATE EXTERNAL TABLE hiveLogTable (id string, data string) STORED BY 'org.apache.hadoop.hive.dynamodb.DynamoDBStorageHandler' TBLPROPERTIES ("dynamodb.table.name" = "log", "dynamodb.column.mapping" = "id:id,data:data"); -- query dynamodb table using hive SQL SELECT * FROM hiveLogTable; -- export dynamodb table to s3 INSERT OVERWRITE DIRECTORY 's3://com.brianpfeil.scratch/emr/dynamodb/export/table/log/' SELECT * FROM hiveLogTable; -- create export table that will be comma delimited CREATE EXTERNAL TABLE s3_export(id string, data string) ROW FORMAT DELIMITED FIELDS TERMINATED BY ',' LOCATION 's3://com.brianpfeil.scratch/emr/dynamodb/export/table/log/'; -- export dynamodb table to s3 as comman delimited file (csv) INSERT OVERWRITE TABLE s3_export SELECT * FROM hiveLogTable; -- link hive table to s3 path CREATE EXTERNAL TABLE s3_import(id string, data string) ROW FORMAT DELIMITED FIELDS TERMINATED BY ',' LOCATION 's3://com.brianpfeil.scratch/emr/dynamodb/export/table/log/'; -- import csv data from s3 into dynamodb table INSERT OVERWRITE TABLE hiveLogTable SELECT * FROM s3_import; --- ### Golang Dep - URL: https://brianpfeil.com/post/golang-dep/ - Date: 2018-02-03 - Tags: golang code for article  pfeilbr/golang-dep-playground project to learn to use dep the dependency management tool for go # create new project dep init # add new import(s) in source code # pull down imports into ./vendor/ dep ensure # can always run `dep ensure` to sync everything up --- ### Imagenet - URL: https://brianpfeil.com/post/imagenet/ - Date: 2018-01-31 - Tags: machine-learning, python code for article  pfeilbr/imagenet-playground project to explore ImageNet image data for ML applications Running run jupyter notebook via kjupyter command see imagenet-classify.ipynb --- ### OpenCV Python - URL: https://brianpfeil.com/post/opencv-python/ - Date: 2018-01-31 - Tags: opencv, python code for article  pfeilbr/opencv-python-playground code is split between jupyter notebook(s) and python (.py) files to create virtualenv (only required for first time setup) python3 -m venv .env Running activate virtualenv source .env/bin/activate display webcam video python opencv-webcam.py example run screenshot --- ### AWS ECS - URL: https://brianpfeil.com/post/aws-ecs/ - Date: 2017-11-11 - Tags: aws, ecs code for article  pfeilbr/aws-ecs-playground learn AWS ECS Examples ecs-fargate-load-balanced-http-service-example example ECS Fargate private (vpc only access) load balanced (internal ALB) http service (nginx) ecs-fargate-task-example example ECS Fargate nodejs task that is manually run ecs-fargate-task-with-efs-example example ECS Fargate nodejs task with EFS file share that is manually run ecs-on-ec2-with-bottlerocket-example example ECS on EC2 using Bottlerocket AMI for ECS container instance ecs-cli-example example using ecs-cli to serve simple docker compose nodejs web app Notes Fargate Task CPU and memory min cpu: 256 (.25 vCPU), max cpu: 4096 (4 vCPU) min mem: 512 MiB, max mem: 30 GB AWS::ECS::TaskDefinition.ExecutionRoleArn - role that grants the Amazon ECS container agent permission to make AWS API calls on your behalf. For example, permission to pull ECR images and create log streams. See Amazon ECS task execution IAM role . The arn:aws:iam::${AWS::AccountId}:role/ecsTaskExecutionRole role is available by default. AWS::ECS::TaskDefinition.TaskRoleArn - role that grants containers in the task permission to call AWS APIs on your behalf. e.g. access S3, secrets manager, etc. See IAM roles for tasks AWS::ECS::Service.Role - no need to specify in typical use case. Amazon ECS uses the service-linked role named AWSServiceRoleForECS to enable Amazon ECS to call AWS APIs on your behalf. see Service-linked role for Amazon ECS AWS::ECS::Service.NetworkConfiguration - required for task definitions that use the awsvpc network mode to receive their own elastic network interface, and it is not supported for other network modes AWS::ECS::Service.NetworkConfiguration.AwsvpcConfiguration.AssignPublicIp - Whether the task’s elastic network interface receives a public IP address. The default value is DISABLED. (DISABLED | ENABLED) --- ### Serverless Python - URL: https://brianpfeil.com/post/serverless-python/ - Date: 2017-11-10 - Tags: serverless, python code for article  pfeilbr/serverless-python-playground experiment with the serverless framework using python based on https://serverless.com/blog/serverless-python-packaging/ session # install `serverless framework` if not already installed npm install serverless -g # create serverless python project serverless create --template aws-python3 --name numpy-test --path numpy-test cd numpy-test # create isolated virtualenv virtualenv venv --python=python3 touch handler.py code . # add code for `handler.py` # activate virtualenv source venv/bin/activate # run to test python handler.py pip install numpy # save dependencies pip freeze > requirements.txt cat requirements.txt # install dependencies (don't need to execute if starting from scratch) pip install -r requirements.txt python handler.py npm init --force # add serverless `serverless-python-requirements` plugin npm install --save serverless-python-requirements # deploy sls deploy # invoke lambda and output log sls invoke -f numpy --log # cleanup sls remove --- ### Boto - URL: https://brianpfeil.com/post/boto/ - Date: 2017-10-30 - Tags: boto code for article  pfeilbr/boto-playground learn and experiment with Boto 3 / AWS SDK for Python Initial setup git clone <this repo> # ensure python 3.x python3 -m venv venv source venv/bin/activate # install dependencies pip install boto3 pip install -U python-dotenv # save dependencies pip freeze > requirements.txt Running # setup virtualenv source venv/bin/activate pip install -r requirements.txt # run script python s3-example.py # exit virtualenv / deactivate deactivate --- ### Minecraft Spigot Plugin - URL: https://brianpfeil.com/post/minecraft-spigot-plugin/ - Date: 2017-10-29 - Tags: minecraft code for article  pfeilbr/minecraft-spigot-plugin-playground minecraft-spigot-plugin-playground Learn plugin development for the spigot minecraft server. Project Setup and Configuration see Creating a blank Spigot plugin in IntelliJIDEA Add spigot.jar Artifacts Remote Debug Configuration Developing (hot swap/live coding) launch minecraft server in debug mode cd ./minecraft-spigot-server ./start.command # java -agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=5005 -jar spigot-1.12.2.jar Launch Debug confifguration in IntelliJ Set breakpoints, etc. in code Launch minecraft and play Breakpoints will be hit. Use Evaluate Code Fragment to explore while debugger is paused You can edit code, save, and build (cmd-F9). It’ll live apply/hot swap without stopping server. Resources https://www.spigotmc.org/wiki/intellij-debug-your-plugin/ --- ### Pusher - URL: https://brianpfeil.com/post/pusher/ - Date: 2017-10-21 - Tags: javascript, real-time code for article  pfeilbr/pusher-playground learn pusher Run Client https://jsbin.com/modutol/edit?html,js,output Server triggered event node index.js # this will cause alert with message in client web browser --- ### Deep and Machine Learning - URL: https://brianpfeil.com/post/deep-and-machine-learning/ - Date: 2017-09-24 - Tags: machine-learning, python code for article  pfeilbr/deep-and-machine-learning-playground docker image + jupyter notebook UI for learning and experimenting with deep and machine learning. docker image based on https://github.com/floydhub/dl-docker - All-in-one Docker image for Deep Learning docker hub image @ https://hub.docker.com/r/pfeilbr/deep-and-machine-learning/ Running docker run -it -p 8888:8888 -p 6006:6006 -v /Users/brianpfeil/projects/deep-and-machine-learning-playground:/root/sharedfolder pfeilbr/deep-and-machine-learning:v2 bash # this lands you at a bash prompt in the container # start jupyter jupyter notebook # open browser to http://localhost:8888/ To save container changes for example to save software installs (e.g. apache mxnet was added to image) # list images to find commit id docker ps -a # commit changes # docker commit 5b4a6fb7117b pfeilbr/deep-and-machine-learning:TAG # e.g. docker commit 5b4a6fb7117b pfeilbr/deep-and-machine-learning:v2 # push new image to docker hub # docker push pfeilbr/deep-and-machine-learning:TAG # e.g. docker push pfeilbr/deep-and-machine-learning:v2 --- ### Jupyter - URL: https://brianpfeil.com/post/jupyter/ - Date: 2017-09-09 - Tags: jupyter code for article  pfeilbr/jupyter-playground learn and experiment with jupyter notebooks jupyterlab (early preview as of this - 2017-09-09) # install conda install -c conda-forge jupyterlab # run jupyter lab # notebook server now running @ http://localhost:8888 screenshots jupyter notebook (classic UI) # run jupyter notebook screenshots jupyter qtconsole The Qt console is a very lightweight application that largely feels like a terminal, but provides a number of enhancements only possible in a GUI, such as inline figures, proper multi-line editing with syntax highlighting, graphical calltips, and much more. The Qt console can use any Jupyter kernel. # install conda install qtconsole conda install pyqt # run jupyter qtconsole screenshots IJavascript Kernel IJavascript is a Javascript kernel (node.js) for the Jupyter notebook # install npm install -g ijavascript # registers the IJavascript kernel with Jupyter ijsinstall screenshots --- ### AWS KMS - URL: https://brianpfeil.com/post/aws-kms/ - Date: 2017-08-31 - Tags: aws, kms code for article  pfeilbr/aws-kms-playground learn how to use kms to encrypt and decrypt data see index.js KMS Key Policies Key policies are the primary way to control access to AWS KMS keys. Every KMS key must have exactly one key policy. The statements in the key policy document determine who has permission to use the KMS key and how they can use it. Every KMS key must have exactly one key policy Each key policy is effective only in the Region that hosts the KMS key. IAM policies by themselves are not sufficient to allow access to a KMS key, though you can use them in combination with a key policy. In a key policy, you use “*” for the resource, which effectively means “this KMS key.” A key policy applies only to the KMS key it is attached to. via Key policies in AWS KMS The following is needed to grant access to a key via an IAM policy. For example, the following would need to exist on the key policy. Then you would need to grant access to the key via say a policy attached to a role. Allows the AWS account (root user) 111122223333 full access to the KMS key, and thus enables IAM policies in the account to allow access to the KMS key. { "Sid": "Enable IAM policies", "Effect": "Allow", "Principal": {"AWS": "arn:aws:iam::111122223333:root"}, "Action": "kms:*", "Resource": "*" }, Resources Key policies in AWS KMS --- ### Scribbletune - URL: https://brianpfeil.com/post/scribbletune/ - Date: 2017-08-26 - Tags: nodejs, music code for article  pfeilbr/scribbletune-playground learn Scribbletune . Generate musical patterns with JavaScript and export as MIDI files using Node.js install timidity command line midi player via brew install timidity timidity man docs --- ### Puppeteer - URL: https://brianpfeil.com/post/puppeteer/ - Date: 2017-08-22 - Tags: testing, browser code for article  pfeilbr/puppeteer-playground learn puppeteer Puppeteer is a Node library which provides a high-level API to control headless Chrome over the DevTools Protocol. It can also be configured to use full (non-headless) Chrome. see examples --- ### S3 Website - URL: https://brianpfeil.com/post/s3-website/ - Date: 2017-08-17 - Tags: s3, aws, static-site code for article  pfeilbr/s3-website-playground learn s3-website npm module usage session # install deps npm install # create static website ./node_modules/s3-website/s3-website.js create # deploy static website com.brianpfeil.s3-website-playground.site01 ./node_modules/s3-website/s3-website.js deploy # view open http://com.brianpfeil.s3-website-playground.site01.s3-website-us-east-1.amazonaws.com # NOTE: using `./node_modules/s3-website/s3-website.js` to avoid global install and # doesn't work with `npx` because of https://www.evernote.com/l/AAH8gou8OCZKF6rfnqA2Tom3BU7c0xuFTM0B/image.png --- ### React Router 4 - URL: https://brianpfeil.com/post/react-router-4/ - Date: 2017-07-29 - Tags: react code for article  pfeilbr/react-router-4-playground project to learn and experiment with react-router 4 built with create-react-app see src/index.js install deps and run $ yarn $ yarn start upgrade all deps $ yarn upgrade react react-dom react-redux react-router react-router-dom react-router-redux redux react-scripts --- ### AWS Kinesis - URL: https://brianpfeil.com/post/aws-kinesis/ - Date: 2017-07-18 - Tags: aws, kinesis code for article  pfeilbr/aws-kinesis-playground learn aws kinesis example session create kinesis stream create lambda (logs stream event) kinesis stream triggers lambda put a record to kinesis stream via cli get shard iterator via cli get records via cli ) view event in lambda cloudwatch logs --- ### Serverless TypeScript webpack - URL: https://brianpfeil.com/post/serverless-typescript-webpack/ - Date: 2017-07-11 - Tags: serverless, typescript code for article  pfeilbr/serverless-typescript-webpack-playground project to learn/explore serverless + typescript + webpack Session example testing and deployment session # cd to service dir cd src/services/logger # local test ../../../node_modules/.bin/sls webpack invoke -f scan --data '{"key1":"value1", "key2":"value2"}' # deploy ../../../node_modules/.bin/sls deploy # test logger endpoint - create log item curl -d '{"key1":"value1", "key2":"value2"}' -H "Content-Type: application/json" -X POST https://4qodphulj8.execute-api.us-east-1.amazonaws.com/dev/logger # lambda-limits service # invoke concurrent lambda executions test # ***warning** understand/check resource usage curl -X POST https://m1v5komsge.execute-api.us-east-1.amazonaws.com/dev/lambda-limits/concurrent # kinesis-playground # PUT record curl -d '{"key1":"value1", "key2":"value2"}' -H "Content-Type: application/json" -X POST https://ijkemqb2r8.execute-api.us-east-1.amazonaws.com/dev/stream/put # tail remote logs ../../../node_modules/.bin/sls logs -f hello -t Notes updated tsconfig.json with "target": "es2015" needed to add the following for webpack + awesome-typescript-loader to run without errors yarn add @types/node --dev yarn add @types/async --dev --- ### Ethereum - URL: https://brianpfeil.com/post/ethereum/ - Date: 2017-01-17 - Tags: blockchain, cryptocurrency code for article  pfeilbr/ethereum-playground Ethereum Playground learn and experiment with Ethereum decentralized blockchain based platform TODO walk all the testnet transaction and get counts of the addresses to see which addresses produce the most transactions walk all testnet transactions and plot transaction count over time for the top X% of addresses that produce the most transactions Files and Directories of Interest Production ~/Library/Ethereum ~/Library/Ethereum/geth/chaindata TEST-NET ~/Library/Ethereum/testnet ~/Library/Ethereum/testnet/geth/chaindata Ethereum Wallet Ethereum Wallet Screenshot Geth CLI command line interface for running a full ethereum node # attach to existing geth instance (e.g. Ethereum Wallet GUI app's geth instance). opens up a javascript repl geth attach # js console geth console # open testnet console geth --testnet console Ethereum Block Explorer production - https://etherscan.io/ testnet - https://testnet.etherscan.io/ Ethereum Javascript API Web3 JavaScript Ðapp API web3.js npm module --- ### Lua - URL: https://brianpfeil.com/post/lua/ - Date: 2017-01-10 - Tags: lua code for article  pfeilbr/lua-playground learn and experiment with the Lua programming language Resources Learn Lua in 15 Minutes Lua - Object Oriented - Tutorialspoint --- ### Node Google Cloud - URL: https://brianpfeil.com/post/node-google-cloud/ - Date: 2016-12-09 - Tags: google code for article  pfeilbr/node-google-cloud-playground Speech API --- ### Apex Lambda - URL: https://brianpfeil.com/post/apex-lambda/ - Date: 2016-12-06 - Tags: apex, lambda code for article  pfeilbr/apex-lambda-playground learn Apex serverless architecture Example Session apex init apex deploy apex logs --follow apex invoke hello echo -n '{"name": "brian"}' | apex invoke hello apex list echo -n '{"q": "heroku"}' | apex invoke search # file upload example # see screenshot of how to access file contents from lambda @ # http://static-content-01.s3-website-us-east-1.amazonaws.com/2__apex_logs_-f__apex__1DF7126D.png curl -i -F name=test -F filedata=@README.md https://UPDATEME.execute-api.us-east-1.amazonaws.com/prod/playground_api-gateway --- ### Heroku Container - URL: https://brianpfeil.com/post/heroku-container/ - Date: 2016-11-04 - Tags: heroku code for article  pfeilbr/heroku-container-playground Learn heroku container support. Based on Heroku Container Registry and Runtime article # install container support heroku plugins:install heroku-container-registry # login to registry heroku container:login # build docker build -t my-nodejs-app . # run docker run -it --rm -p 8000:8000 --name my-running-app my-nodejs-app # visit http://localhost:8000 # create heroku app heroku create # push. NOTE: app name of "stormy-badlands-73151" will be different heroku container:push web --app stormy-badlands-73151 # check that it is up and running heroku ps --app stormy-badlands-73151 # visit in browser heroku open --app stormy-badlands-73151 # update flow # make change to `server.js` # build and push heroku container:push web --app stormy-badlands-73151 --- ### NextJS - URL: https://brianpfeil.com/post/nextjs/ - Date: 2016-11-01 - Tags: javascript, react code for article  pfeilbr/nextjs-playground Learn Next.js , a small framework for server-rendered universal JavaScript webapps, built on top of React, Webpack and Babel Developing $ npm run dev --- ### KeystoneJS - URL: https://brianpfeil.com/post/keystonejs/ - Date: 2016-10-20 - Tags: javascript, ui, framework code for article  pfeilbr/keystonejs-playground Learn and experiment with KeystoneJS Install and Run $ npm install # create dir for mongodb files $ mkdir -p data/db # start mongodb $ mongod --dbpath data/db & # start webapp $ node keystone.js # visit http://localhost:3000/ Admin Login Email: brian.pfeil@gmail.com Password: admin --- ### Kubernetes - URL: https://brianpfeil.com/post/kubernetes/ - Date: 2016-10-18 - Tags: kubernetes code for article  pfeilbr/kubernetes-playground learn kubernetes. based on tutorial at http://kubernetes.io/docs/tutorials/kubernetes-basics Install and start minikube $ cd ~/tmp $ curl -Lo minikube https://storage.googleapis.com/minikube/releases/v0.11.0/minikube-darwin-amd64 && chmod +x minikube && sudo mv minikube /usr/local/bin/ # verify install $ minikube version # start $ minikube start minikube VMs, etc. are stored in ~/.minikube $ tree ~/.minikube . β”œβ”€β”€ addons β”œβ”€β”€ apiserver.crt β”œβ”€β”€ apiserver.key β”œβ”€β”€ ca.crt β”œβ”€β”€ ca.key β”œβ”€β”€ ca.pem β”œβ”€β”€ cache β”‚Β β”œβ”€β”€ iso │ │ └── minikube-0.7.iso │ └── localkube β”œβ”€β”€ cert.pem β”œβ”€β”€ certs β”‚Β β”œβ”€β”€ ca-key.pem β”‚Β β”œβ”€β”€ ca.pem β”‚Β β”œβ”€β”€ cert.pem │ └── key.pem β”œβ”€β”€ config β”œβ”€β”€ key.pem └── machines β”œβ”€β”€ minikube β”‚Β β”œβ”€β”€ boot2docker.iso β”‚Β β”œβ”€β”€ config.json β”‚Β β”œβ”€β”€ disk.vmdk β”‚Β β”œβ”€β”€ id_rsa β”‚Β β”œβ”€β”€ id_rsa.pub │ └── minikube β”‚Β β”œβ”€β”€ Logs │ │ └── VBox.log β”‚Β β”œβ”€β”€ minikube.vbox │ └── minikube.vbox-prev β”œβ”€β”€ server-key.pem └── server.pem REST API start proxy to allow access $ kubectl proxy Access API $ curl http://127.0.0.1:8001/api { "kind": "APIVersions", "versions": [ "v1" ], "serverAddressByClientCIDRs": [ { "clientCIDR": "0.0.0.0/0", "serverAddress": "10.0.2.15:8443" } ] } visit API endpoint at http://127.0.0.1:8001/api visit kubernetes-dashboard at http://127.0.0.1:8001/api/v1/proxy/namespaces/kube-system/services/kubernetes-dashboard/#/workload?namespace=default Accessing Services The cluster ip address as exposed to host machine can be obtained via $ kubectl cluster-info --- ### Phaser - URL: https://brianpfeil.com/post/phaser/ - Date: 2016-08-02 - Tags: javascript, game, framework code for article  pfeilbr/phaser-playground project to learn the Phaser HTML5 game framework stackem game Simple drop and stack game Play @ https://pfeilbr.github.io/phaser-playground/ Developing # start live reload server live-server --- ### Emscripten - URL: https://brianpfeil.com/post/emscripten/ - Date: 2016-06-02 - Tags: web-assembly, tools code for article  pfeilbr/emscripten-playground learn and experiment with emscripten Running Follow the Download and install instructions Run the following # base install location # ~/dev/emsdk_portable # source in environment to update PATH and make tools available # could add to .bash_profile to make available in all shells $ source ~/dev/emsdk_env.sh # needed to add python2 symlink (see https://github.com/kripken/emscripten/issues/3872) $ cd /usr/local/bin $ ln -s /usr/bin/python2.7 python2 # compile c file to javascript -> generates output/hello_world.js $ emcc hello_world.c -o output/hello_world.js $ node output/hello_world.js # compile c file and generate html to view it -> generates output/hello.html and output/hello.js # NOTE: overwrites $ emcc hello_world.c -o output/hello.html --- ### GitBook - URL: https://brianpfeil.com/post/gitbook/ - Date: 2016-05-20 - Tags: react, documentation code for article  pfeilbr/gitbook-playground GitBook Toolchain Documentation Setup $ npm install gitbook-cli -g $ mkdir gitbook-playground $ cd gitbook-playground/ $ gitbook init # MAKE EDITS - see http://toolchain.gitbook.com/ # server via built-in webserver $ gitbook serve # generate pdf $ gitbook pdf ./ ./mybook.pdf Introduction Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum. Section 1 Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum. Subsection 1 Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum. Section 2 Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum. Subsection 1 Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum. --- ### TypeScript Node - URL: https://brianpfeil.com/post/typescript-node/ - Date: 2016-05-18 - Tags: typescript code for article  pfeilbr/typescript-node-playground example app showing how to setup and develop a typescript based node app using the typings TypeScript Definition Manager Initial Setup $ npm init --force $ typings install --ambient node $ npm install request --save $ typings install --ambient request // tsc was complaining about `form-data` $ typings install --ambient form-data // for `Promise`, `Object.assign`, etc. $ typings install --ambient es6-shim // compile from .ts to .js $ tsc Developing // live re-compile and re-run on changes via nodemon $ npm run dev // NOTE: app will be ran 2xs on every change when editing in Atom since atom-typescript will re-compile also in addition to `tsc -w` in the `dev` npm script --- ### Angular CLI App - URL: https://brianpfeil.com/post/angular-cli-app/ - Date: 2016-05-09 - Tags: angular, cli code for article  pfeilbr/angular-cli-app-playground learn and experiment with angular-cli --- ### React Redux - URL: https://brianpfeil.com/post/react-redux/ - Date: 2016-05-04 - Tags: react code for article  pfeilbr/react-redux-playground learn and experiment with react redux react-redux react-router-redux Based on jackielii/simplest-redux-example Running npm install npm start open http://localhost:8080/webpack-dev-server/ in the browser Files index.html - host page index.js - entry point main.css - styles webpack.config.js - webpack config --- ### Typings - URL: https://brianpfeil.com/post/typings/ - Date: 2016-05-02 - Tags: typescript code for article  pfeilbr/typings-playground typings-playground learn and experiment with TypeScript typings # install typescript $ npm install typescript --global # install typings $ npm install typings --global # install node typings $ typings install node --ambient --save # compile index.ts to index.js $ tsc # run $ node index.js see contents of tsconfig.json, which tells tsc how to compile Resources How to add type definitions to a TypeScript project typings DefinitelyTyped/DefinitelyTyped http://definitelytyped.org/ --- ### Nexe - URL: https://brianpfeil.com/post/nexe/ - Date: 2016-04-05 - Tags: nodejs, distribution, packaging, tools code for article  pfeilbr/nexe-playground Learn and experiment with nexe nexe lets you create a single executable out of your node.js apps Install $ npm install nexe -g Running There are two methods to run nexe, command line and by specifying the paramaters in package.json Command line $ nexe -r 4.2.4 -i ./index.js -o ./main.nex Downloads node source 4.2.4 for the platform it’s running on in ./tmp directory, compiles it, uses browserify on ./index.js then bundles it all an outputs ./main.nex package.json nexe property of package.json specifies all the parameters $ nexe # output is ./main.nex, which is specified in the nexe.output property --- ### Node OData - URL: https://brianpfeil.com/post/node-odata/ - Date: 2016-03-18 - Tags: nodejs, odata code for article  pfeilbr/node-odata-playground node-odata-playground This is an example of a custom OData 4.0 provider that can be consumed by salesforce Lightning Connect Setup $ npm install Running $ node custom-odata-provider-example.js # run ngrok to expose local server publicly $ ngrok 1337 OData Endpoints you can visit http://localhost:1337 http://localhost:1337/$metadata http://localhost:1337/$metadata#PIMSProduct http://localhost:1337/PIMSProduct?$count=false http://localhost:1337/todos?$count=false Using brian.pfeil.1@gmail.com salesforce org. Screenshots Query Example External Data Source External Object Filter Criteria / Where Clause --- ### Angular2 - URL: https://brianpfeil.com/post/angular2/ - Date: 2016-01-13 - Tags: angular, framework code for article  pfeilbr/angular2-playground Angular2 Playground Project to learn and experiment with Angular2 Running $ npm install $ npm start make changes and page will live reload --- ### Atom Golang Setup - URL: https://brianpfeil.com/post/atom-golang-support/ - Date: 2015-12-02 - Tags: golang, atom I’ve recently switched to Atom from Sublime Text for web development. The transition was relatively painless since many of the keyboard shortcuts and capabilities are the same in. I really enjoy usig Atom and want to have that same experience with programming in Go . This documents my Go setup in Atom. Install and Configuration Steps Install go-plus Set the go-plus GOPATH setting Ideally the Atom application process should have GOPATH in its ENV and you shouldn’t have to set this. Install atom build Set build Panel Visibility setting to Keep Visible Add .atom-build.json to root of your project directory { "cmd": "$GOROOT/bin/go run {FILE_ACTIVE}", "shell": true, "env": { "GOROOT": "/usr/local/go", "GOPATH": "/Users/pfeilbr/go" } } Keyboard Shortcuts cmd-alt-b - run current file cmd-alt-g - goto definition ctrl-space - code completion --- ### Beefy - URL: https://brianpfeil.com/post/beefy/ - Date: 2015-10-16 - Tags: nodejs, tools code for article  pfeilbr/beefy-playground Beefy Playground Learn and experiment with Beefy , which makes working with Browserify a little nicer. Running Clone this repo Install deps $ npm install Run with live reload and es6 support (babelify ) $ beefy index.js --live --open -- -t babelify --- ### Flexbox - URL: https://brianpfeil.com/post/flexbox/ - Date: 2015-10-15 - Tags: css code for article  pfeilbr/flexbox-playground Flexbox Playground Project to learn and experiment with the CSS3 Flexbox layout module. index.html is the entry point Resources A Visual Guide to CSS3 Flexbox Properties A Complete Guide to Flexbox --- ### Node Coveralls - URL: https://brianpfeil.com/post/node-coveralls/ - Date: 2015-09-29 - Tags: testing, nodejs code for article  pfeilbr/node-coveralls-playground node-coveralls-playground To learn and experiment with coveralls test coverage service. Setup Steps Install deps $ npm install mocha coveralls mocha-lcov-reporter --save-dev Install istanbul $ npm install istanbul --save-dev -g If running from command line, ensure .coveralls.yml is in current directory. It should contain a line with repo_token: REPO_TOKEN Running To run on the tests in the test/ directory. $ istanbul cover ./node_modules/mocha/bin/_mocha --report lcovonly -- -R spec && cat ./coverage/lcov.info | ./node_modules/coveralls/bin/coveralls.js && rm -rf ./coverage You can view the raw coverage output in the ./coverage directory. --- ### Heroku Deploy - URL: https://brianpfeil.com/post/heroku-deploy/ - Date: 2015-09-23 - Tags: heroku code for article  pfeilbr/heroku-deploy-playground To learn and experiment with various heroku deployment options such as github and dropbox --- ### Heroku Docker - URL: https://brianpfeil.com/post/heroku-docker/ - Date: 2015-09-23 - Tags: heroku, docker code for article  pfeilbr/heroku-docker-playground NOTE: This process and the use of the heroku docker command has been deprecated Heroku Docker Playground To learn heroku-docker to Build, run and deploy Heroku apps with Docker Deploy Note Renamed app to “heroku-docker-playground” via heroku web app. Need to push via the following. heroku docker:release –app heroku-docker-playground Resources Build and Deploy with Docker --- ### Mocha - URL: https://brianpfeil.com/post/mocha/ - Date: 2015-09-22 - Tags: mocha code for article  pfeilbr/mocha-playground Mocha Playground Area to learn and explore the Mocha JavaScript test framework Install Mocha $ npm install -g mocha Install THIS and dependencies Clone repo then $ npm install Run Tests $ mocha --- ### OSX Audio - URL: https://brianpfeil.com/post/osx-audio/ - Date: 2015-09-15 - Tags: osx code for article  pfeilbr/osx-audio-playground Playground to learn the various audio frameworks and APIs Server server/ contains a node server (websocket) that the playground code interacts with --- ### Metaforce - URL: https://brianpfeil.com/post/metaforce/ - Date: 2015-09-09 - Tags: salesforce, ruby, gem code for article  pfeilbr/metaforce-playground Playground to learn and experiment with the metaforce gem for interacting with the salesforce metadata api. --- ### Docker Hub Automated Build - URL: https://brianpfeil.com/post/docker-hub-automated-build/ - Date: 2015-08-20 - Tags: docker code for article  pfeilbr/docker-hub-automated-build-playground Playground to try out Docker Hub Automated Builds from GitHub see https://docs.docker.com/docker-hub/github/ Github Services --- ### WebdriverIO - URL: https://brianpfeil.com/post/webdriverio/ - Date: 2015-08-20 - Tags: nodejs, testing, automation, browser code for article  pfeilbr/webdriverio-playground Playground to learn and experiment with the node webdriverio package Running Selenium Standalone Server (Preferred / lighter weight) Start selenium Standalone server # run the following in it's own shell and leave running $ java -jar selenium-server/selenium-server-standalone-2.53.0.jar Run script $ node index.js Running Selenium Hub and Node(s) Using Docker start hub (clients connect to this) $ docker run -d -P --name selenium-hub -e GRID_TIMEOUT=10000 selenium/hub start chrome node (hub delegates work to it) $ docker run -d --link selenium-hub:hub selenium/node-chrome Get host and port for “selenium-hub” container and use them in your client. Example client config var options = { host: "192.168.99.100", port: 32768, desiredCapabilities: { browserName: 'chrome' } }; see index.js --- ### Spark - URL: https://brianpfeil.com/post/spark/ - Date: 2015-06-05 - Tags: spark code for article  pfeilbr/spark-playground Project to learn and experiment with the Spark cluster computing system. Setup spark installed at ~/dev/spark-2.2.0-bin-hadoop2.7 and is part of PATH in .bash_profile export SPARK_HOME=~/dev/spark-2.2.0-bin-hadoop2.7 export PATH=$SPARK_HOME/bin:$PATH Running an example $ spark-submit dataframes-and-sql-examples.py Docker $ docker run -i -t -P --name spark -v /Users/pfeilbr/Dropbox/mac01/Users/brianpfeil/projects/spark-playground:/src sequenceiq/spark:latest bash Resources Spark Documentation --- ### V8 - URL: https://brianpfeil.com/post/v8/ - Date: 2015-05-19 - Tags: v8 code for article  pfeilbr/v8-playground Project to learn and experiment with the V8 javascript engine. Building V8 with Xcode Install depot-tools . And add to PATH environment variable. Clone repo $ git clone https://chromium.googlesource.com/v8/v8.git Build $ cd v8 $ build/gyp_v8 -Dtarget_arch=x64 Background This project was created with a new Application | Command Line Tool with the language set to C++ in Xcode. The static libraries from the v8 build were copied in The v8 include files were copied from v8/include to v8-includes Resources Google V8 Docs --- ### HTML Video Embed - URL: https://brianpfeil.com/post/html-video-embed/ - Date: 2015-05-12 - Tags: html code for article  pfeilbr/html-video-embed-playground Project to learn and experiment with various HTML video players Usage Start a local webserver python -m SimpleHTTPServer 8000 Visit the .html files to view video embeds. e.g. http://localhost:8000/test-screencast-jwplayer.html Resources JW Player Video.js --- ### webpack - URL: https://brianpfeil.com/post/webpack/ - Date: 2015-04-03 - Tags: javascript, bundler, packaging code for article  pfeilbr/webpack-playground A project to learn and experiment with the webpack module bundler. Development Start local webpack server $ webpack-dev-server --progress --colors --debug This serves all changes out of memory. Remember to run webpack by itself to persist changes Open http://localhost:8080/webpack-dev-server/bundle Make code changes. Changes will live reload. Persist changes by running $ webpack Directory and File Details public/ - client side files public/lib/index.js - webpacks entry point react-app.js - react application example demonstrates the usage of jsx loader css loader es6 support angular-app.js - angular application example demonstrates the usage of ng-cache loader for views and partials --- ### ES6 - URL: https://brianpfeil.com/post/es6/ - Date: 2015-03-30 - Tags: javascript code for article  pfeilbr/es6-playground Area to learn and experiment with ECMAScript 6 using the Babel JavaScript compiler. Running $ npm install Edit app.js $ node index.js --- ### React Native - Initial Thoughts - URL: https://brianpfeil.com/post/react-native-initial-thoughts/ - Date: 2015-03-27 - Tags: javascript, ios, react, react native React Native was released as open source yesterday. There’s been quite a bit of build up and excitement since the React.js Conf 2015 video . The video showed a real app being built with a web development style workflow (your favorite editor with livereload). I’ve built some apps using React.js and really appreciate the simplicity and development workflow. Having the same for native app development definitely is exciting. I’ve gottent to spend a few hours with it and it looks like it’s living up to the hype. I started in with the tutorial , which is a movies app backed by Rotten Tomatoes data. The main UI is a run of the mill List View with each cell containing a poster image, and the movie title and year. It shows fetching data over the network, some of the concerns with rendering list views, and the react core concept of only re-rendering thing that’ve changes with ListView.DataSource.rowHasChanged. They sprinkle in ES6 syntax within the examples, and some of it I had to look up. This is a good way to give people a gentle introduction to it. I can see how it might be a bit distracting if your trying to learn something new. Having to learn language level features on top of a new framework could be offputting for some. Yep. It’s All in that single file. It does a good job with starting off with a single minimal view and layering on the different concepts to get you to some more than hello world, but not a real app. One of the things that stood out to me and made it much easier to follow is that all the code is in a single file. UI, style, logic, and data. This is huge from a learning and user uptake perspective. For larger apps, it will get cumbersome quickly and using separate .js files and require them in will make it managaeble. I’ve seen way too much time spent in other javascript frameworks first introducing the file and directory layout . There are as many different ways to lay things out as there are javascript frameworks these days. I agree it’s important for maintainability and collaboration, but when starting out, I feel it should be kept simple, and only restructure when needed. Choice is good, but I hope facebook explicitly documents their recommended layout and people just use it. Native Modules: Call for Community Contributions If react native doesn’t provide the UI component or capability you need, they allow extension via Native Modules . Bridging is never pretty as it always feels like you need to switch contexts and be aware this is a bridge component. They do their best to minimize this and in the spirit of react it’s lightweight and only requires the minimal amount of hooks to do it’s job. You only need to implement the RCTBridgeModule protocol to expose native code. Here’s the example // CalendarManager.h #import "RCTBridgeModule.h" @interface CalendarManager : NSObject <RCTBridgeModule> @end // CalendarManager.m @implementation CalendarManager - (void)addEventWithName:(NSString *)name location:(NSString *)location { RCT_EXPORT(); RCTLogInfo(@"Pretending to create an event %@ at %@", name, location); } @end var CalendarManager = require('NativeModules').CalendarManager; CalendarManager.addEventWithName('Birthday Party', '4 Privet Drive, Surrey'); As you can see, it’s minimally intrusive on both the objc and js sides. Native Modules is the community aspect of this effort and if done right will give it staying power. The productivity and developer workflow benefits of npm install awesomesauce is the one of the core underlying reasons for javascript/node popularity. A decision needs to be made on whether npm or a native packager like Cocoapods will be the vehicle for packaging and distributing native modules. To keep it x-platform and support android, npm would be the obvious choice. It’s important for the react native team to make a decision on this and define this from the start. People will fill the gaps that react native doesn’t include. The easier you make it for them to publish for community, the better off the ecosystem will be. Fetch Networking and more specifically the ability to fetch HTTP resources is a core capability of a majority of applications today. While walking through the tutorial I saw the use of fetch , but had never seen it before. I thought it was something that facebook defined and injected that bridged into iOS’s networking. It does this, but in addition it’s based on the fetch networking API , which is going through the standard process. Hopefully a higher profile project like react native will give it some legs. Onward Native development on the native target platform will always be more popular. It doesn’t strive to or need to try to supplant native development. Capturing the web development community alone is a huge audience. More importantly, the react community is full of super star developers that will acively champion things they like. If it reached the success of a Xamarin and as large a community, it would be quite an achievement. It’s early and if you don’t have a lot free time or care to be on the bleeding edge, I’d recommend waiting a bit before using it to develop a production app. As with anything new and in interest of the long-term goals, I’m sure there will be bugs to work through, breaking changes, API changes, additional development and debugging tooling, etc. Even if I don’t dive right in, at a minimum I’ll keep up with it’s progress via rss feeds and twitter. Kudos to the react native team for a great launch and wish them the best. --- ### Concurrent Downloader in Go - URL: https://brianpfeil.com/post/concurrent-downloader-in-go/ - Date: 2015-03-26 - Tags: golang, concurrency Go has been getting a lot of traction among the coding elite due to it’s simplicity, speed, and most touted, it’s ability to enable concurrent solutions. Go is a very small language. If you have experience with a few languages, you can learn in a few days and fit and keep it all in your head. With most other languages, I’m accustomed to looking up documentation or examples on stackoverflow while I code. With Go, I find I don’t do it as much. It’s a language influenced by C and the rest of curly brace family of languages. It compiles down into standalone binaries with no dependencies. I can’t emphasize how great this is for building cross-platform tools. You write your code once, and can cross-compile to any of core three OSs (osx , linux , windows ) from any one of them. No need to run a windows VM to build for windows. The most visible feature and probably most discussed is it’s concurrency model. It provides the primitive channel construct, with along with the syntax makes tackling concurrency problems more intuitive than the typical thread construct provided by a majority of languages. In order to get a thorough understanding of concurrency in go, I thought of a real world problem that lends itself to being solved via concurrency. The ATP podcast with Marco Arment was food for thought for an example real world. Downloading web content and indexing it in an efficient manner is a problem many web based services do. The process of indexing web content is computationally expensive, and only doing it when neccessary is desired. If the web content hasn’t changed, then we shouldn’t waste compute resources indexing the content. One way to detect this is by doing an MD5 hash of the web contents (HTML page) and only index it if it’s changed. The first step is fetching the contents of the web resource via a URL. The built-in net/http package turns this into a one-liner resp, err := http.Get(url) We can then get the body body, err := ioutil.ReadAll(resp.Body) and create an MD5 hash of it hash := md5.Sum(body) Now onto the interesting part of parallelizing. We create a simple structure to hold the information needed to do the work. In this case it’s simply a URL type URLJob struct { url string } We also need to create a structure to hold the results type URLResult struct { URL string Body string MD5 string } With these in place we con now focus on the work with our worker function. func worker(id int, jobs <-chan URLJob, results chan<- URLResult) { for j := range jobs { fmt.Println("worker", id, "processing job", j.url) results <- fetch(j.url) } } Let’s focus on the arguments. jobs <-chan URLJob is an array of channels where you can send URLJob instances to it. results chan<- URLResult is an array of channels where we can send the results to. We then invoke via a goroutine go worker(w, jobs, results) This is where the magic happens. go func creates a goroutine that is ran concurrently by the Go runtime. In our example, waiting on the network to respond with the contents of the URL is the limiting factor. In this case, all these http GET requests will be fired off essentially at once, and as the responses return our results channel array will be updated. You could request 1000s of URLs at a time and assuming the response contents are not huge, this wouldn’t move the needle noticably on CPU and memory for a reasonably sized machine. Parallelism is baked into all hardware these days and is the way the computing hardware industry holds true to Moores law. Modern languages like Go need to surface these advances, but without the usual cognitive load of using threads. The holy grail of writing your code as if everything is syncronous and having it parallelized automatically for you to make use of all the cores, etc. seems a ways off or most likely will never happen. In the meantime Go is making a good run at it and moving us forward in this space. Example code for the concurrent downloader example is on github at pfeilbr/concurrent-downloader . --- ### Running Elasticsearch on AWS Elastic Beanstalk - URL: https://brianpfeil.com/post/running-elasticsearch-on-elastic-beanstalk/ - Date: 2015-03-25 - Tags: docker, elastic beanstalk, aws, elastic search TL;DR code on github @ pfeilbr/Elastic-Beanstalk-Dockerrun.aws.json-Example This article will show you the steps to run elasticsearch on AWS Elastic Beanstalk . This example uses the elasticsearch docker image as an example. Once running you can visit http://<beanstalk domain>/?pretty an you will see the root elastic search JSON response. Initial Deployment Modify Dockerrun.aws.json for your needs. Dockerrun.aws.json reference Commit changes to repo $ git commit -a -m "changes" assumes Dockerrun.aws.json has already been added to repo. if not, git add . Create eb app $ eb init # populate details NOTE: select Docker. Populate all details Create environment for app $ eb create dev-env IMPORTANT Must immediately update the ec2 instance with tags so it doesn’t get terminated. Enable termination protection on the ec2 instance. Deploying Updates Modify Dockerrun.aws.json for your needs. Commit changes $ git commit -a -m "my updates" Deploy to eb $ eb deploy NOTE: Takes between 3-5 min to deploy changes Notes Port mapping details specific to aws beanstalk --- ### Dockerized Node.js App on Elastic Beanstalk Example - URL: https://brianpfeil.com/post/dockerized-nodejs-app-on-elastic-beanstalk-example/dockerized-nodejs-app-on-elastic-beanstalk-example/ - Date: 2015-03-25 - Tags: nodejs, docker, elastic beanstalk, aws Example of developing and deploying a dockerized Node.js app to Elastic Beanstalk source on Github at pfeilbr/Elastic-Beanstalk-Docker-Node.js-Example Local Development Workflow Edit code. e.g. index.js Build image $ docker build --tag="pfeilbr/eb-docker-node-example" . Run $ docker run -p 80:80 -it -rm -name eb-docker-node-example pfeilbr/eb-docker-node-example Get docker host ip (optional. only if using boot2docker ) $ boot2docker ip Open browser to http://<boot2docker ip> Initial Deployment Init git repo $ git init . Add files to repo $ git add . Commit changes $ git commit -m "init" Create eb app $ eb init # populate details Populate all details Create environment for app $ eb create dev-env IMPORTANT Must immediately update the ec2 instance with tags so it doesn’t get terminated. Enable termination protection on the ec2 instance. Output Deploying Updates Modify code and test via [Local Development Workflow] Commit changes $ git commit -a -m "my updates" Deploy to eb $ eb deploy NOTE: Takes between 3-5 min to deploy changes Output Establish Interactive Bash Shell in Running Docker Container ssh into ec2 docker host server $ eb ssh Get container name $ sudo docker ps # save off name of container Connect/attach with an interactive bash session $ sudo docker exec -i -t <container name> bash Example Session with Output --- ### Cross-platform Desktop Apps with NW.js - URL: https://brianpfeil.com/post/cross-platform-desktop-apps-with-nw-js/ - Date: 2015-02-15 - Tags: javascript, nodejs, desktop, nw.js Post is based on the pfeilbr/nwjs-playground project. NW.js Playground Project to learn and experiment with NW.js . NW.js enables the creation of cross platform desktop applications. Your UI is built with standard html, js, and css technologies. Access to the underlying desktop OS capabilites are provided by giving you access to node.js . You can use the Node.js core modules plus any of the 3rd party NPM modules; Installing NW.js Download and unpack nwjs app for your platform from http://nwjs.io On OSX you’ll end up with /Applications/nwjs.app Developing Your App create index.html and package.json files see nw.js quick-start for details This app is a minimal example with an added app.js file for javascript. It’s sets the document.body to 'Hello There'. Running Create alias in ~/.bash_profileto ease the use from terminal # alias to nw alias nw="/Applications/nwjs.app/Contents/MacOS/nwjs" Run $ cd ~/projects/nwjs-playground $ nw . Packaging as app on OSX Create .nw file $ cd ~/projects/nwjs-playground $ zip -r ../${PWD##*/}.nw * .nw file will be created along side the project directory make copy of nwjs.app $ cp -r /Applications/nwjs.app /Applications/myapp.app Copy .nw file into myapp.app bundle $ cp ../nwjs-playground.nw /Applications/myapp.app/Contents/Resources/app.nw --- ### Karma - URL: https://brianpfeil.com/post/karma/ - Date: 2015-02-15 - Tags: javascript, testing code for article  pfeilbr/karma-playground Project to learn and experiment with the Karma javascript test runner. Karma runs your javascript tests in multiple browsers. Quickstart Tutorial Change to project directory $ cd ~/projects/karma-playground New NPM project $ npm init accept all defaults Install karma # Install Karma: $ npm install karma --save-dev # Install plugins that your project needs: $ npm install karma-jasmine karma-chrome-launcher --save-dev Install global karma cli so we can type karma without the full path from anywhere. $ npm install -g karma-cli Create sample test. Using Jasmine for our example. $ mkdir test $ touch test/sample1.js // test/sample1.js describe("A suite", function() { it("contains spec with an expectation", function() { expect(true).toBe(true); }); }); Generate the karma configuration file $ karma init my.conf.js Start karma # Start Karma using your configuration: $ karma start my.conf.js You’ll see Chrome or the browsers you chose start --- ### NWjs - URL: https://brianpfeil.com/post/nwjs/ - Date: 2015-02-15 - Tags: nodejs, cross-platform, framework code for article  pfeilbr/nwjs-playground NW.js playground Project to learn and experiment with NW.js . NW.js enables the creation of cross platform desktop applications. Your UI is built with standard html, js, and css technologies. Access to the underlying desktop OS capabilites are provided by giving you access to node.js . You can use the Node.js core modules plus any of the 3rd party NPM modules; Installing NW.js Download and unpack nwjs app for your platform from http://nwjs.io On OSX you’ll end up with /Applications/nwjs.app Developing Your App create index.html and package.json files see nw.js quick-start for details This app is a minimal example with an added app.js file for javascript. It’s sets the document.body to 'Hello There'. Running Create alias in ~/.bash_profileto ease the use from terminal # alias to nw alias nw="/Applications/nwjs.app/Contents/MacOS/nwjs" Run cd ~/projects/nwjs-playground nw . Packaging as app on OSX Create .nw file cd ~/projects/nwjs-playground zip -r ../${PWD##*/}.nw * .nw file will be created along side the project directory make copy of nwjs.app cp -r /Applications/nwjs.app /Applications/myapp.app Copy .nw file into myapp.app bundle cp ../nwjs-playground.nw /Applications/myapp.app/Contents/Resources/app.nw --- ### Sauce Labs - URL: https://brianpfeil.com/post/sauce-labs/ - Date: 2015-02-04 - Tags: testing, automation code for article  pfeilbr/sauce-labs-playground Sauce Labs Playground Playground to learn, explore, and play with Sauce Labs Sauce Labs Node.js Tutorial tutorial/ directory is from following the tutorial @ Sauce Labs Node.js Tutorial tutorial/test/sauce/tutorial-specs.js contains the tests and is where you can modify the code. Quickstart Bypasses almost all tutorial steps because they’ve been completed cd tutorial/ grunt Example Test Run Output Screenshot --- ### Go Postgres - URL: https://brianpfeil.com/post/go-postgres/ - Date: 2015-01-11 - Tags: golang, postgres code for article  pfeilbr/go-postgres-playground go-postgres-playground playground for go lib/pq package Dependencies go get github.com/lib/pq Running go run main.go --- ### Nodemailer - URL: https://brianpfeil.com/post/nodemailer/ - Date: 2014-12-06 - Tags: nodejs, gmail, email code for article  pfeilbr/nodemailer-playground Playground for exploring the use of Nodemailer to send email Usage Update index.js with email password before running. npm install npm start OR node index.js --- ### Kendo UI - URL: https://brianpfeil.com/post/kendo-ui/ - Date: 2014-08-02 - Tags: javascript, ui, framework code for article  pfeilbr/kendo-ui-playground Kendo UI Playground Place to play around and learn Kendo UI --- ### iOS Swift - URL: https://brianpfeil.com/post/ios-swift/ - Date: 2014-07-13 - Tags: ios code for article  pfeilbr/ios-swift-playground ios-swift-playground Scratchpad to learn and explore swift --- ### Heroku Connect - URL: https://brianpfeil.com/post/heroku-connect/ - Date: 2014-07-05 - Tags: heroku Heroku Connect was announced at Dreamforce ‘13 shortly after salesforce aquired cloudconnect . Cloudconnect is the original product created by Adam Gross , formerly of Dropbox that became Heroku Connect. There’s a rich developer ecosystem around web apps built with popular web frameworks such as Rails , Django , Express , Symfony , etc. All of these frameworks assume a relational database as a backend, which because of it’s age, usage, and rich set of existing toolsets, makes it easy to get started with. Salesforce’s force.com platform wanted gain this ease of use and speed at which apps could be developed. Salesforce has it’s own data storage solution that is built for scale and multitenatcy and doesn’t cleanly map to the standard relational model. Connect solves this problem by bi-directionaly synchronizing your salesforce data to a heroku postgres database. Once in postgres, all the popular web frameworks with a huge developer community can be used to develop salesforce apps. Heroku Connect Setup Connect is provided as a heroku add-on , which enables easy intregration with heroku apps. A prerequisite is to have a postgres database provisioned for connect to synchronize your salesforce data to. Once your database is in place, you install the add-on via heroku addons:add herokuconnect -a appname Here’s what it looks like The rest of the setup and configuration takes place in the Heroku Connect web interface. You access it by logging into heroku and navigate to your app, then click the Heroku Connect link in the Add-ons section. This will lauch the web ui. On first lauch you will be greeted with a setup screen. This will automatically use to the DATABASE_URL config variable, which was put in place when the postgres database was provisioned. The final setup looks like Next you need to authorize Connect to access your salesforce org You allow permission You are then ready to configure the data you want to use with Connect. Resources Introducing Heroku Connect: Connecting Clouds and Customers Heroku Connect Dev Center article --- ### Kue - URL: https://brianpfeil.com/post/kue/ - Date: 2014-06-13 - Tags: javascript, queue code for article  pfeilbr/kue-playground Playground for learning / playing with Kue . Kue is a priority job queue for node. Running redis-server # start redis server node index.js -m process # run job processor node index.js -m create # run job creator in another terminal View the Kue web ui at http://localhost:3000 --- ### JSForce - URL: https://brianpfeil.com/post/jsforce/ - Date: 2014-06-12 - Tags: salesforce, javascript code for article  pfeilbr/jsforce-playground Area to learn and play with JSForce . index.js contains the code. migrator.js migrates salesforce metadata from a source org to a target org To Run $ ./node_modules/.bin/babel-node migrator.js sharing-rules-operations.js created country code firewall criteria based sharing rules To Run $ ./node_modules/.bin/babel-node sharing-rules-operations.js account-fetcher.js Developing $ DEBUG=account* nodemon -e js account-fetcher.js --- ### Mac Swift - URL: https://brianpfeil.com/post/mac-swift/ - Date: 2014-06-05 - Tags: mac code for article  pfeilbr/mac-swift-playground Swift Playground Swift playground for exploring the swift programming language Playground Specific Functions There are a set of functions that enable the code you write to interact with the plaground UI. For example, you can display a UIView live. The XCPlayground module provides these functions XCPSetExecutionShouldContinueIndefinitely(continueIndefinitely:Bool) Allows async code to run. For example, dispatch_* calls and networking. XCPCaptureValue(identifier: String, value:AnyObject) Displays the value in the view pane with the identifier as the title XCPShowView(identifier: String, view: NSView) Display the provided view in the view pane with the identifier as the title --- ### Go - URL: https://brianpfeil.com/post/go/ - Date: 2013-12-31 - Tags: golang code for article  pfeilbr/go-playground A set of example Go programs for learning Each subdirectory contains an example Running go run filename e.g. go run hello.go Resources Go Documentation --- ### Minotes - URL: https://brianpfeil.com/post/minotes/ - Date: 2013-06-21 - Tags: ios, app There are many choices for notes apps out in the App Store. So many that it’d be a couple days effort just to look into and evaluate them all. Everything from the built-in simple Notes app to the feature packed Evernote . A notes app is one of those utility apps that gets a lot of mileage like mail and weather. Something with high usage you want the most efficient and streamlined app. It’s also something that becomes personal and people get attached to. I wanted something extremely simple, clean up, and fast. I started with the simplest possible solution of a single text view you could type in. From that, the ability to create multiple notes was added, but I didn’t want the freedom to create as many notes as you want. Having constraints on the number of notes forces the user to create more meaningful notes and manage / clean up notes. The goal with minotes is not to have an archive of all historical notes with search, etc. The type of notes appropriate for this app are short-lived ones and in general have a lifespan of around a week. I landed on a UI metaphor similar to mobile safari with a max of 9 notes. Preferences around typography and text size is something that is really needed in order to be useful for more than a specific set of users. For example, older people may need a little bit larger font size, whereas younger users may want to cram more text on the screen and are fine with a smaller font size. In order to keep it simple, there are three typefaces to choose from and in sizes, small, medium, large. It’s an opinionated app built with my specific needs in mind. It’s doesn’t try to appeal to all users universally, but if you like simple purpose built apps, check it out. Minotes Product Page --- ### Hosting Static Sites on Heroku - URL: https://brianpfeil.com/post/hosting-static-sites-on-heroku/ - Date: 2012-09-16 - Tags: web, heroku I was recently creating a web site for a family member, and was looking around for the best place to host it. It’s a simple personal site that only contains static content. There are many options out there from the freebie wordpress options to rackspace and the like. None of the “web publishing platform” type sites were a fit because I wanted full control over the layout, and look and feel of the site. I’ve been working with Heroku for my day job for web applications, and it’s been simple to get things up and running at no cost. Heroku allows you to create a web application for free, and only if you need to scale to support the traffic do you have to pay. One of the limitations of using the free plan with heroku is that if your app hasn’t recieved any traffic within an hour, it puts it into an idle state. They call this dyno idling and when a request comes in and your dyno is in an idle state, it has to spin it back up, and that takes time. In my experience, I’ll go to hit the site and it takes approx. 5 seconds to respond. It’s free, so I can understand when they do this to save on compute resources, and if you have pay / have multiple dynos, they never with idle your app. To work around this you just need to have at least 1 request come in an hour. There are many ways to do this, and I took the simple approach of using a monitoring service (pingdom ) to check/ping the site for availability every 5 minutes. You could also implement by using a background worker process. There’s a great Creating Static Sites in Ruby with Rack walkthrough on the heroku dev center that will have you up and running in few minutes. They provide a fully functional sample application that you can modify to make you own. --- ### RubyMotion - URL: https://brianpfeil.com/post/rubymotion/ - Date: 2012-05-11 - Tags: ruby RubyMotion is a toolchain that lets you develop iOS using Ruby . I’m a big fan of Ruby and have been using it on and off since 2001. It’s been a huge timesaver in many cases allowing me to quickly script solutions. When I heard it was coming to iOS I had to check it out. The toolchain and the way you use the various components matches the rails development process. Open your favorite text editor and a terminal, and you’re ready to go. For example to create a new project. motion create project Interfacing with the toolchain is all rake driven. Edit your files, type rake device, and your app is compiled and launches in the iOS Simulator. It’s a very nice and familiar development process. One of the big benefits with using Ruby is concise code. Compared to Objective-C , you’ll end up with a lot less code for an app. For example with Ruby, you could put your entire app in a single .rb file. With Objective-C you have the .h and .m files for every class, and you spend quite a bit of time switching between them all. Ruby helps ease the app development process, but most of the learning and skill with iOS development is understanding the various iOS frameworks. This doesn’t go away with RubyMotion. You still have to take the time to learn these. Part of learning is reviewing the code samples that Apple provides, and they’re all in Objective-C. Knowing the native language of a platform is always the best thing, and if someone wanted to get started with iOS development, I’d recommend learning Objective-C. RubyMotion costs $200, and this might be a barrier for it taking off. The Ruby community is used to things being open source and free. It’d be nice if they took the approach Apple does with Xcode. With a free Apple Developer account, you can create apps, but your limited to running them in the simulator. Once you pay the $99 for to be a part of the developer program, you can put apps on devices and the App Store. RubyMotion could do the same, free for apps running on the simulator, and purchase to put apps in the App Store. The creator of it is former Apple employee, Laurent Sansonetti , who also led the development of the MacRuby project, which tries to achieve the same goal by allowing devs to create OS X apps with Ruby. Apple supported the MacRuby project heavily in the past, but it seems to have slowed with the rise of iOS. I have to think that Sansonetti proposed Ruby on iOS while he was at Apple, but it didn’t get support internally. Apple doesn’t like platforms being built on top of it’s own platforms because they loose control and if the apps created using the platform don’t match the user experience Apple wants, they can’t do anything about it (remember flash). I’m skeptical that it’ll take off, but am definitely rooting for it. --- ### Emailing Todos to Dropbox - URL: https://brianpfeil.com/post/emailing-todos-to-dropbox/ - Date: 2011-11-20 - Tags: productivity, dropbox, appengine Between work and home,I’ve always got more things to do than I can keep straight my head. Like most people, I keep a to-do list so I don’t have to worry about forgetting it. My to-do list is a simple bulleted text (Markdown ) file that I keep in Dropbox so I can access it from anywhere. # TODOS * get gas * get leaf bags * send thank you cards # Completed * get propane * get burgers and hot dogs This setup works fairly well, but they’re times when I think of something I need to do, but I don’t get it into this list because of the time it takes to add it. The usual scenario is that I’m not sitting in front of a computer, and I need to add it using my iPhone. There are plenty of to-do and dropbox text file editor apps our there to choose from (way too many), but none of them make it fast enough to enter, and fit my to-do-list-text-file-in-dropbox setup. The best case scenario would be an app, that opens awaiting my next to-do, and I’d just type it in, and it’d add it to my to-do list text file in dropbox in the right format. I thought about making an app that did just that, but I knew there had to be an easier way. That’s when email popped into my head. It’s dead simple, and available everywhere. I’d never have an excuse not to get something on the list. It’s also fast and easy to send email. Solution My solution is to send an email to a special “todo” email address where the subject would be the todo item, the email would be processed, and the item added to my to-do list text file in dropbox. Google App Engine lets you run web apps on Google infrastructure for free, and they offer a way to receive and process email. Dropbox provides a Dropbox Python SDK to get at your files, which is works out nicely since Python is the primary language used for App Engine development. You can take a look at the Google App Engine Receiving Email Documentation , but the gist of it is that an email sent to NAME@APP.appspotmail.com will result in a HTTP POST request sent to /_ah/mail/NAME@APP.appspotmail.com with the POST body containing the contents of the email. Google provides a nice InboundMailHandler class to make handling inbound email a breeze. The first thing we need to do is to map a handler for inbound emails. I’m using the Django Python web framework in my app engine app, which provides some connivence mechanisms. To map a URL to a handler we add the following to the urls.py file. urlpatterns = patterns('', (r'^_ah/mail/todo@myapp.appspotmail.com`, 'emailengine.views.todo_email_handler'), We mapped the url to todo_email_handler method def todo_email_handler(request): if request.POST: message = mail.InboundEmailMessage(request.raw_post_data) logging.info("Received a message from: " + message.sender) if is_whitelisted_email_address(message.sender): mgr = DropboxManager() mgr.add_todo(message.subject) logging.info("Received a message from: " + message.sender + ", Todo:" + message.subject) return HttpResponse('ok') In order to prevent anyone from adding to-do items to my list, I added the is_whitelisted_email_address method to restrict processing of emails to only those that come from me. def is_whitelisted_email_address(email): result = False for whitelist_email in EMAIL_ADDRESS_WHITELIST: if email.lower().find(whitelist_email) != -1: result = True return result The work to add the to-do item to the TODO.txt file in dropbox is mgr = DropboxManager() mgr.add_todo(message.subject) I created the DropboxManager class as a simple wrapper around the Dropbox API. This keeps the code tidy, and all the dropbox specific code like the app key, secret, and token in one place. Conclusion I’m sure one of the many existing to-do list apps fits the needs of most people. This solution required a little work upfront, but has already proven to be a timesaver. --- ### Best Fit Web Font Sizing - URL: https://brianpfeil.com/post/best-fit-web-font-sizing/ - Date: 2011-09-27 - Tags: html, javascript, fonts I’ve been working on a web app that targets both smartphones and tablets. The large variation in screen sizes has sent me down the path of using a dynamic proportional layout that adapts to fit the available space. For example, I’ve allocated 20% of the available vertical space to the header section that displays a title. The jQuery UI.Layout Plug-in has worked great for laying out the content areas, but I ran into a wall when it came to sizing my text proportionately. The basic problem is that I have a box, and I want to display some text in it at the largest font size without it being wrapped or clipped. The only way to determine the bounding rectangle a string of text with certain font characteristics, is to create it, add it to the DOM, and then measure it. The following function does just that. function sizeWithText(text, cssStyles) { // create temp element to hold our text var e = document.createElement('span'); e.appendChild(document.createTextNode(text)); // apply any styles that have been passed in // to our element - these can affect the text size for (var prop in cssStyles) { e.style[prop] = cssStyles[prop]; } // hide our temp element e.style['visibility'] = 'hidden'; // add to DOM in order to have it render document.body.appendChild(e); // get the bounding rectangle dimensions var s = {w: e.offsetWidth, h: e.offsetHeight}; // remove from DOM document.body.removeChild(e); return s; } The cssStyles parameter holds the other css style attributes that you’d like to apply to the text. For example, you might have a font-weight: bold attribute that increases the size of the text, and we want to make sure we account for it. Now we can use this function to check whether text with a font size and a set of styles will fit in our box. We set the font size to 1 and continuously increase it by 1 check whether it’ll fit at every iteration. As soon as it doesn’t, we stop. function bestFitTextSize(text, css, width, height) { var pixel = 1; do { css['font-size'] = (pixel++) + 'px'; s = sizeWithText(text, css); } while ( (s.w < width) && (s.h < height) ) return pixel - 2; } This is a brute force and inefficient way to do the calculation, and there are improvements that could be made. We could start at a reasonable font size like 6px, increment by standard font sizes, etc., but this is fine for my usage where I only do it once on app load. Here’s a code sample that shows how the previous functions are used. // box we want to fill with text var c = document.getElementById('content'); // out text var text = 'Lorem ipsum dolor sit amet'; // styles var cssStyles = { 'font-family': 'Impact', 'font-style': 'normal', 'font-weight': 'bolder', 'letter-spacing': '1px', 'text-shadow': '3px 3px 3px white' }; // size the text to fit function applyBestFitText() { // get the pixel size for the font var px = bestFitTextSize(text, cssStyles, c.offsetWidth, c.offsetHeight); cssStyles['font-size'] = px + 'px'; // set the text c.innerHTML = text; // apply our styles for (var prop in cssStyles) { c.style[prop] = cssStyles[prop]; } } // adjust if the size changes window.addEventListener('resize', applyBestFitText, false); // call for first time adjustment applyBestFitText(); The complete example is available as a gist --- ### Metro UI: Initial Thoughts - URL: https://brianpfeil.com/post/metro-ui-initial-thoughts/ - Date: 2011-09-21 - Tags: windows, win8, metro I’ve been digesting all the new information around Windows 8 and Metro over the past week. I’ve watched the BUILD Conference Channel 9 Videos , read the articles and docs on the Windows Dev Center and the Building Windows 8 blog . The best resource for your time on learning what Metro is all about from a design perspective is 8 traits of great Metro style apps . The Metro Style interface is a totally new and original approach to UI. Microsoft put a lot of thought into the design of it, and they demonstrated this by providing background during the BUILD sessions on how they studied how users held and used slate devices, what areas of the screen where reachable by their thumbs, etc. The concept of dedicating all the available screen space to content with no chrome makes perfect sense. It doesn’t makes sense for all applications, especially those rich in functionality. For example, Microsoft showed Photoshop and it’s many tool palettes and menus, and it was clear that you wouldn’t make a Metro Style Photoshop app. Metro makes sense for “consumery” type apps like casual games, productivity apps, and rich media apps; the same things you see on an iPad or Android tablet. This is one of the reasons that they need to support “classic” windows app, and they can’t make a clean break to the Metro Style UI. One potential problem with the immersive content, and no chrome approach is the lack of discoverability to take actions on the content. Metro relies heavily on gestures that take place at the edges of the screen. For example to display the charm bar, you place your finger off to the right of the screen where there isn’t any content and touch isn’t recognized then swipe in towards the center of the screen. Another example is the application bar at the bottom where you have to swipe up to summon it. Is a user gonna know to do this? Will devices come with an in-your-face tutorial on these things when you first boot it up? There have been many studies that have proven if it’s out of site then users won’t discover it. For touch, there are only two gestures you should assume your users know, and that’s tap and swipe to scroll; everything else is power user territory. There’s another issue around using a mouse in Metro. Microsoft made the statement that Metro is a natural fit for a mouse in addition to touch. I’ve been using the Developer Preview running in VMware, and using a mouse with Metro is very tedious. The level of indirection you have between the mouse and what happens on the screen makes the things that work well with touch feel like a large amount of effort. I actually resorted to using the windows 8 keyboard shortcuts . to activate the standard toolbars and menus because it was so painful. If the apps interaction is limited to clicks and/or keyboard input with no application bar, then the mouse could work, but that’s only gonna be a handful of apps. The tiles interface for the home screen was very visually appealing in all the BUILD demos with nice typography, rich pictures, and pastel colors. This is great and demos well when Microsoft controls all the apps, but what happens when you have a whole community of app developers who have their own opinions on design and style building Metro apps. The app that doesn’t use a pastel color for their tile background, shows a poorly shot overexposed photo that the user took on their phone, uses an ugly icon, etc. There’s nothing from keeping people from doing this. Microsoft is aware of this and they emphasized many times in the BUILD sessions to use the templates they provide in Visual Studio 11. This is a problem on any platform, and it happens on iOS and Android, but the reason it’s a bigger issue for Metro is that it’s critical to have in place for the immersive experience. All you have is content. You don’t have standard chrome title bars and toolbars with a standard look and feel to make your app fit in. Content is everything, and if it’s poor, then it’ll stick out like a sore thumb. It’s still early. Microsoft always previews things and gets them out early to users, so I’m sure there’ll be some tweaks. This is a big change for them, and the most recent thing as big as this has been there move into the game console space with XBox. Time will tell. --- ### Salesforce and jQuery Mobile - URL: https://brianpfeil.com/post/salesforce-and-jquery-mobile/ - Date: 2011-09-11 - Tags: salesforce, jquery, mobile I wanted to do a project to learn salesforce , but not a trivial hello world style example. I had recently played around with jQuery Mobile , but didn’t do anything significant with it. I decided to put together a basic mobile CRM app. Here’s what the finished app looks like. Salesforce provides a full-featured environment for free to anyone. It takes no time to sign-up for this Developer Edition , and you get access to everything. If you’re just getting started with salesforce , I can’t recommend enough the online tutorials, workbooks, and documentation at developer.force.com . Accessing the App Data Our app will contain Contact and Account data. We’ll use Apex Web Service methods to provide the data. We can get all our Contacts with the following: webService static List<Contact> getContacts() { ApexPages.StandardSetController setCon = new ApexPages.StandardSetController( Database.getQueryLocator( [select id, name, lastname, firstname, title, department, phone, email from Contact])); return (List<Contact>)setCon.getRecords(); } Next we need to call the web service method from the client code. We’ll use the force.com ajax apex javascript libraries to access our data. Add these script references to your page <head> section. <script src="/soap/ajax/15.0/connection.js"></script> <script src="/soap/ajax/15.0/apex.js"></script> To call the web service method from JavaScript we use this simple one-liner. var contacts = sforce.apex.execute("MyWebService", "getContacts", {}); Building the UI Our mobile app UI will be 100% jQuery Mobile , and we can use force.com Sites to provide pure web content. Sites by default provides configurable headers, footers, navigation, etc. that you can apply themes to. We don’t want any of this in our app since we’ll be controlling everything in the UI. To turn off the default stylesheets, header, and navigation elements your apex:page element should set the corresponding attributes, and should look similar to the following: <apex:page showHeader="false" sidebar="false" standardStylesheets="false" contentType="text/html"> Now we’ll add in the jQuery Mobile references. <link rel="stylesheet" href="http://code.jquery.com/mobile/1.0b2/jquery.mobile-1.0b2.min.css" /> <script src="http://code.jquery.com/jquery-1.6.2.min.js"></script> <script src="http://code.jquery.com/mobile/1.0b2/jquery.mobile-1.0b2.min.js"></script> The beauty of jQuery Mobile is that all you have to do is markup your html with the correct attributes to get a mobile UI. You don’t need to write any JavaScript. Usually you serve up the jQuery Mobilized html from the server. We want our app to run and function when the device is offline, so we’re going to generate the html at run-time on the client using JavaScript to remove the dependency on the server. To minimize the amount of html we embed in our JavaScript, we’re going to use jQuery Templates . jQuery Templates allows us to write our html for the UI as we usually would, and to leave placeholders for the pieces we want to replace. Here’s an example of a template used to generate a list page. <script id="object-list-page-template" type="text/x-jquery-tmpl"> <div data-role="page" id="${type}-list-page" data-url='${type}-list-page'> <div data-role="header"> <h1>${pageTitle}</h1> <a id='${type}-add-link' data-obj-type='${type}' class='ui-btn-right object-add-link' data-icon="add">Add</a> </div> <div data-role="content"> <ul data-role="listview" id="${type}-list" data-filter="true"> </ul> </div> <div data-role="footer" data-id='footer-nav-bar' data-position="fixed"> <div data-role="navbar"> <ul> <li><a href="#contact-list-page" data-transition="fade" class="${(type == 'contact') ? 'ui-btn-active' : ''} ui-state-persist">Contacts</a></li> <li><a href="#account-list-page" data-transition="fade" class="${(type == 'account') ? 'ui-btn-active' : ''} ui-state-persist">Accounts</a></li> </ul> </div> </div> </div> </script> The things wrapped in ${} are replaced with the values. The way we use the template is by calling the $.tmpl method. var $page = $('#object-list-page-template').tmpl(data); jQuery Templates are very powerful, and really help with the maintainability of your code. Check out the documentation to learn more. Conclusion This provides an overview of the core pieces of the solution and the technologies used. If you like to dive in deeper, the complete source is available on github , and the working app is available here (must be viewed in a WebKit based browser). --- ### Automatic Network Location - URL: https://brianpfeil.com/post/automatic-network-location/ - Date: 2011-08-05 - Tags: mac, network I bounce back and forth between work and home with a MacBook Pro. At work, I’m behind a firewall, and have to go through a web proxy to get to public internet sites. I have 2 network locations setup, home and work. When I get to work I manually set my location to work, and when I get home, I manually set it to home again. I got tired of remembering to do this so I looked around, and found scselect . scselect is a command line utility that comes with OS X that lets you set your location by name scselect "home" Now all I needed was a way to determine whether I was at work or home and then run scselect with the corresponding location. My work assigned ip address has a set prefix, so I can use that to say I’m at work and anything else would default to home. # exit code is 0 if on work network and 1 otherwise ifconfig | grep -i "inet 59.33" > /dev/null In order to prevent setting the location to the same location we need to find out what the current location is. Running scselect with no arguments displays a list of locations and the active one is marked with an asterisk (*). Here’s my list. > scselect Defined sets include: (* == current set) 575844ED-8466-479C-9567-3F0B7D767EE9 (home) 3DF4B8B9-2E92-4F61-B684-74E0D0D38DEE (Automatic) * 6064213B-532D-43C9-8941-DC72B6487955 (work) The output is a bit messy, and we need to parse out the active location. #!/usr/bin/env ruby # example output: * 6064213B-532D-43C9-8941-DC72B6487955 (work) output = `scselect 2>&amp;1 | grep ' \\* '` # parse out the location. location = output.scan(/\(.*\)/).first.gsub(/[\(\)]/, "") # location = work We now compare the active location with the location our ip address tells us. If different, we call scselect with the new location. Complete script is available here --- ## Projects ### Minotes - URL: https://brianpfeil.com/projects/minotes/ Minotes - minimalist notes - is designed from the ground up to provide fast and clean note taking. Rather than packing features into a crowded interface, Minotes provides a minimal streamlined interface for your notes. clean β€” The interface of Minotes is intentionally simple. It provides just what you need, without any distractions fast β€” Minotes is optimized from the ground up to be fast and fluid, whether you are on the fanciest new iPhone or a more vintage model. beautiful β€” From its thoughtfully laid out navigation to its select set of settings, Minotes is designed to make you want to open it. Minotes Product Page --- ### Animal Fun - URL: https://brianpfeil.com/projects/animal-fun/ Animal Fun is an iPhone and iPad app I wrote for my son when he was 3 to help him learn about animals while also entertaining him. It’s a free app available on the App Store and the source code is available on github --- ### Vehicle Fun for Kids - URL: https://brianpfeil.com/projects/vehicle-fun-for-kids/ Fun and easy interactive vehicle app designed by parents for young children who enjoy the look and sound of vehicles. A terrific collection of real vehicle pictures and sounds that play with one easy tap. You’ll also hear the name of the vehicle along with the sound. It will keep those little ones busy for quite a while! They’ll get their fill of cars, trucks, and boats! Vehicle Fun for Kids Product Page --- ### Alpha Sound - URL: https://brianpfeil.com/projects/alpha-sound/ Alpha Sound is a simple ABC’s learning program for children. Children learn their letters by seeing and hearing the letter names. It’s a free app available on the App Store Alpha Sound Product Page --- ### Music Sound Touch - URL: https://brianpfeil.com/projects/music-sound-touch/ Fun and easy interactive music app designed by parents for young children who enjoy the look and sound of musical instruments. A terrific collection of real musical instrument pictures and sounds that play with one easy tap. You’ll also hear the name of the instrument along with the sound. It will keep those little ones busy for quite a while! Music Sound Touch Product Page --- ### Mobile Account & Contact Manager - URL: https://brianpfeil.com/projects/mobile-account-contact-manager/ Mobile Contact and Account management web application built using the salesforce platform for the backend and jQuery Mobile for the frontend. Provides the same functionality when disconnected by using HTML5 Application Cache and local storage . See Salesforce and jQuery Mobile post for details source code --- ### iOS Enterprise App Store - URL: https://brianpfeil.com/projects/ios-enterprise-app-store/ Enterprises that are members of the iOS Developer Enterpise Program can distribute their employee apps with their own App Store using Apple’s over-the-air distribution . With this solution, you just drop your app .ipa file and a file with information about the app into a directory, and it’s immediately available for users to install. The backend is a Sinatra web app. On the front-end there’s a built-in jQuery Mobile web interface, and also a separate native iPhone app. App Store Web App Native iPhone App Store Client source code ---