Posts

  • Jan 14, 2019

    AWS Notes - Database

    Aurora

    • 5 times faster then MySQL
    • Start with 5 GB to 10 GB, 32vCpu, 244 GB memory
    • 2 copies of data in one AZ, 3 AZs
    • 2 types of replica, Aurora Replica, MySQL replica
    • Up to 15 read replica
    • Aurora PostgreSQL does not currently support cross-region replicas.

    RDS

    • Automated backup is enabled by default
    • Snapshots are created manually
    • Restore creates a new instance
    • Multi AZs is for disaster recovery, read replica is to improve performance.
    • Read replica is not available for SQL Server and Oracle database.
    • Max size is 16TB for SQL Server.
    • When RDS is doing a backup the IO may be briefly suspended.
    • Changes to RDS backup window takes effect immediately.
    • Up to 5 read replica
    • Encryption of existing RDB instance is not supported, you have to create a snapshot of the instance, encrypt the snapshot and restore from the encrypted copy.
    • Backup can be kept up to 35 days.
    • Multi AZ use synchronous synchronisation for standby instance.
    • Read replica use asynchronous synchronisation.
    • Read replica can be in a different region.

    ElasticCache

    • Memcache
    • Redis

    DynamoDB

    • Value cannot exceeds 400kb.
    • Spread 3 different data centers. It is automatically replicated across multiple AZs.
    • By default it is eventual consistent read. It can support strongly consistent read with a higher cost.
    • DynamoDB synchronously replicates data across three facilities in an AWS Region, giving you high availability and data durability.
    • DynamoDB is designed to scale without limits. However, if you want to exceed throughput rates of 10,000 write capacity units or 10,000 read capacity units for an individual table, you must first contact Amazon . If you want to provision more than 20,000 write capacity units or 20,000 read capacity units from a single subscriber account, you must first contact us to request a limit increase.

    Redshift

    • A single node can have 160 GB data.
    • Up to 128 compute node.
    • Only support in one AZ.
    • Columnar data storage.
  • Jan 9, 2019

    AWS Notes - Route53

    • A CNAME record assigns an Alias name to a Canonical Name. This is a normal DNS capability as defined in the RFCs. By design CNAMEs are not intended to point at IP addresses.
    • Route53 has a security feature that prevents internal DNS from being read by external sources. The work around is to create a EC2 hosted DNS instance that does zone transfers from the internal DNS, and allows itself to be queried by external servers.
    • Route 53 supports alias resource record sets, which enables routing of queries to a CloudFront distribution, Elastic Beanstalk, ELB, an S3 bucket configured as a static website, or another Route 53 resource record set
    • Alias records are not standard for DNS RFC and are an Route 53 extension to DNS functionality
    • Alias records help map the apex zone (root domain without the www) records to the load balancer DNS name as the DNS specification requires “zone apex” to point to an ‘A’ record (ip address) and not to an CNAME
    • Route 53 automatically recognizes changes in the resource record sets that the alias resource record set refers to for e.g. for a site pointing to an load balancer, if the ip of the load balancer changes, Route 53 will reflect those changes automatically in the DNS answers without any changes to the hosted zone that contains resource record sets
    • If an alias resource record set points to a CloudFront distribution, a load balancer, or an S3 bucket, the time to live (TTL) can’t be set; Route 53 uses the CloudFront, load balancer, or Amazon S3 TTLs.
  • Jan 6, 2019

    AWS Notes - EC2

    • EC2 supports On Demand, Reserved, Spot, Dedicated Hosts access.
    • SSD: General Purpose SSD, Provisioned IOPS SSD.
    • Magnetic: Throughput Optimized HDD, Cold HDD, Magnetic
    • One EC2 instance can have multiple security group.
    • All inbound traffic is blocked by default and all outbound traffic is allowed by default.
    • Inbound rules are automatically applied to outbound rules.
    • EFS can be mounted to multiple EC2 instances
    • EBS is replicated in multiple physical devices in the same AZ.
    • There is a soft limit of 20 instances per region.
    • You cannot create an unencrypted volume from an encrypted snapshot or encrypt an existing volume
    • ELB can help to support statefull service
    • Once a VPC is set to Dedicated hosting, it is not possible to change the VPC or the instances to Default hosting. You must re-create the VPC. Further information
    • There is no limit to the number of EC2 instances you can have in the Auto Scaling group. However, there might an EC2 limitation in your account that can be increased by logging a support ticket.
    • ELB can span multiple AZs within a region. It cannot span multiple regions.
    • The customer is responsible for the security of anything running on the hypervisor, and therefore the operating system and the security of data are the customer’s responsibility.
    • With proper scripting and scaling policies, the On-demand instances behind the Spot instances will deliver the most cost-effective solution because the on-demand will only spin up if the spot instances are not available.
    • EBS, S3 and EFS all allow the user to configure encryption at rest using either the AWS Key Management Service (KMS) or, in some cases, using customer provided keys. The exception on the list is Elasticache for Memcached which does not offer a native encryption service, although Elasticache for Redis does.
    • The Public IP address is not managed on the instance: It is, instead, an alias applied as a network address translation of the Private IP address.
    • AWS does not copy launch permissions, user-defined tags, or Amazon S3 bucket permissions from the source AMI to the new AMI.
    • You can not encrypt an exiting EBS volume.
    • AWS reserves both the first four and the last IP address in each subnet’s CIDR block.
  • Jan 2, 2019

    AWS Notes - Tips

    • Kinesis for social media, news feeds, logs.
    • Redshift for Business intelligence.

    • Elastic Map Reduce for Big data processing

    • Cloudtrail is per AWS account per region

    • Can consolidate logs using S3 bucket cross multiple accounts

    • AWS Organisation Service Control Policies will override individual account policies.

    • To connect your data center with AWS, you will need a customer gateway on your side and a virtual private gateway on AWS.
    • An internet gateway is to connect a VPC to the internet and NAT gateway connect the servers running in private subnet to the internet.
    • A VPC endpoint enables you to privately connect your VPC to supported AWS services.
    • You cannot do vpc peering cross region. But can do it cross accounts.
    • You cannot creat VPC peering when the two VPCs have matching or overlapping CIDR blocks.
    • VPC peering does not support transitive peering relationships.
    • SSL certificates will only be useful to encrypt data in transit, not data at rest.
    • ELB can span multiple AZs within a region. It cannot span multiple regions.
    • The customer is responsible for the security of anything running on the hypervisor, and therefore the operating system and the security of data are the customer’s responsibility.
  • Dec 31, 2018

    AWS Notes - Services

    SWF

    • Workers
    • Deciders
    • In SWF a task is only assigned once and never duplicated.

    SNS

    • Push mechanism
    • Messages are stored across multiple AZs
    • Topics
    • Subscribers can be HTTP, HTTPS, Email, Email-JSON, SQS, Application, Lambda

    Elastic Transcoder

    API Gateway

    • Caching TTL

    Kinesis

    • Kinesis stream
    • Kinesis firehose
    • Kinesis analytics
  • Dec 30, 2018

    AWS Notes - SQS

    • Messages can contain up to 256kb of text in any format.
    • Standard Queue more than one copy of a message might be delivered out of order.
    • FIFO queue ensure ::ordering:: and a message is delivered ::once:: only.
    • SQS is ::pull based::.
    • Messages can be kept from 1 min to 14 days, default is 4 days.
    • SQS guarantees messages will be processed at least once.
    • SQS visibility timeout: default is 30 sec, up to 12 hours.
    • Short polling vs Long polling
  • Dec 29, 2018

    AWS Notes - VPC

    • IP ranges:

      10.0.0.0 ~ 10.255.255.255 (10/8 prefix)
      172.16. 0.0 ~ 172.31.255.255 (172.16/12 prefix)
      192.168.0.0 ~ 192.168.255.255 (192.168/16 prefix)

    • Default 5 VPCs in a region
  • Dec 28, 2018

    AWS Notes - S3

    • S3 one zone infrequent access is for recreatable data. It has the same durability but reduced availability (99.95%)
    • Signed URL and signed cookies can be used to securely access S3.
    • S3 use read after write consistency. New objects can be read after write but update or delete will be eventual consistent.
  • Dec 7, 2018

    Avoid Typing Twice When Defining Action Names

    Found a nice package which can help defining actions a lot easier by avoiding type the key/value twice when defining action names.

    The package is KeyMirror. Simply import it into the source code and then you can build a dictionary with key/value pairs are of the same values.

    var keyMirror = require('keyMirror');
var COLORS = keyMirror({blue: null, red: null});
var myColor = COLORS.blue;
var isColorValid = !!COLORS[myColor];
  • Dec 5, 2018

    Martin Fowler Refactoring 2nd

    This is on my reading list.

    Martin Fowler - Refactoring 2nd

  • Dec 4, 2018

    Danger of not awaiting async

    Recently found a bug in a production system where exceptions happened when the code tried to commit data into database.

    The exception was

    InvalidOperationException
The transaction has already been committed or rolled back.

    The cause of the exception was because a very deep invocation of an asynchronous handler was not awaited. If not awaited, the method will keep running outside of the scope of the sql transaction and when the method tries to commit into database, the exception will be thrown.

  • Dec 3, 2018

    Use docker compose with environment variables

    Environment variables can be defined inside docker compose files. Most of the time, some environment variables are used repeatedly, for example api gateway urls, DNS servers and identity server urls.

    We can declare reusable variables inside a docker compose file and pull the values out of it into a ::.env:: file under the same folder.

  • Oct 28, 2018

    eshopOnContainers Switch to Local Database on Dev Environment

    It is too heavy to run sql server in a docker container on a development machine. The goal of this article is to introduce a way to use sql server express installed locally instead.

    I have cloned a copy from the main repo and updated the configuration to use local database instead of running a separate sql server inside docker.

    It can be found from here

  • Aug 6, 2018

    Integrate AWS Services With On-Premise Servers

    Summary

    In this blog, I will introduce a solution which enables integration of AWS services with your on-premise servers. The idea is to utlise the felxibilities of the cloud while maintaining security of on-premise servers.

  • Jul 29, 2018

    Quartznet Database (Mysql) Example

    • You can download the source code on Github.
    • A set of data tables need to be generated in the database. I have included the sql scripts in the source.
  • Jun 25, 2017

    Continuously Deploy Spring Boot Applications With Docker

    Spring Boot is a lightweight and fully fledged framework written in Java. It can be used to build RESTful APIs as well as console applications. It provides functionalities such as Dependency Injection, Database Connection Pool Management out of the box so that developers can focus on the business logics instead of struggling with nuts and bolts.

  • Apr 30, 2017

    Writing SOLID Code to Enable Continuous Delivery & Integration

    Why do we need Continuous Delivery & Continuous Integration?

    In today’s highly competitive IT industry no one can afford to spend years developing a piece of software. There are always smart people with same or similar ideas and users can always find alternatives if they are unhappy with what you can provide. In this sense, it is always important to get to the market faster.

  • Apr 13, 2017

    D3 Projection on Google Maps

    Summary

    This blog explains how to use d3 to draw shapes on google maps and enable users to interact with them.

    You can find the source code on Plunker

subscribe via RSS