From 4a497599faea975e97f518aa9ab8ad0d2ccbaf83 Mon Sep 17 00:00:00 2001 From: dtomlinson Date: Thu, 30 Apr 2020 03:18:56 +0100 Subject: [PATCH] updating notes --- runningnotes.md | 248 ++++++++++++++++++++++++++++++++++++++++++------ tempnotes.md | 86 ++++++++++++++--- todo.md | 42 ++++---- 3 files changed, 315 insertions(+), 61 deletions(-) diff --git a/runningnotes.md b/runningnotes.md index 15d3906..2a47c34 100644 --- a/runningnotes.md +++ b/runningnotes.md @@ -1,3 +1,45 @@ + +* [Running strapi in different modes](#Runningstrapiindifferentmodes) +* [Strapi documentation](#Strapidocumentation) +* [API Examples using HTTPIE](#APIExamplesusingHTTPIE) + * [Authenticate with the API](#AuthenticatewiththeAPI) + * [Get a Single Content Type](#GetaSingleContentType) + * [Use query parameters to filter for Multiple Content Type](#UsequeryparameterstofilterforMultipleContentType) +* [S3 Upload Addon](#S3UploadAddon) + * [AWS Resources](#AWSResources) + * [Configuration](#Configuration) +* [Fix Version Numbers](#FixVersionNumbers) +* [Strapi in git](#Strapiingit) +* [Cloudformation](#Cloudformation) + * [Creating templates](#Creatingtemplates) + * [Adding resources](#Addingresources) + * [Using parameters](#Usingparameters) + * [Using outputs](#Usingoutputs) + * [Using functions](#Usingfunctions) + * [Examples](#Examples) + * [Short form](#Shortform) + * [Outputs](#Outputs) + * [Referencing other resources internally.](#Referencingotherresourcesinternally.) + * [Pesudeo references](#Pesudeoreferences) + * [Referencing other resources from external templates](#Referencingotherresourcesfromexternaltemplates) + * [Deploy a stack/template](#Deployastacktemplate) + * [Passing in parameters](#Passinginparameters) + * [Tags](#Tags) + * [Updating stack](#Updatingstack) + * [Failure](#Failure) + * [Stacks](#Stacks) + * [Snippets](#Snippets) + * [Deploy a template/stack](#Deployatemplatestack) + * [Destroy a stack](#Destroyastack) +* [Tags](#Tags-1) + * [Cloudformation default tags](#Cloudformationdefaulttags) + + + + # Running notes deocument that the db has to be done from cli arg, but the configs can be done via files. @@ -15,7 +57,7 @@ Tie this in with a cloudformation template + hooking it up Try setting the database name using cloudformation template -## Running strapi in different modes +## Running strapi in different modes You should use development for developing strapi and then deploy it to production. @@ -42,7 +84,7 @@ There is no current plans to allow for this, as well as no plans to move these m Due to the reasons I explained above I am going to mark this as closed but please do feel free to discuss. ``` -## Strapi documentation +## Strapi documentation @@ -54,17 +96,17 @@ You should change the production URL server url in the documentation settings. Edit the file `./extensions/documentation/documentation/1.0.0/full_documentation.json` and change `YOUR_PRODUCTION_SERVER` to the ELB URL of your environment. -## API Examples using HTTPIE +## API Examples using HTTPIE -### Authenticate with the API +### Authenticate with the API `http http://strapi-prod.eu-west-1.elasticbeanstalk.com/auth/local identifier=apiuser password=password` -### Get a Single Content Type +### Get a Single Content Type `http http://strapi-prod.eu-west-1.elasticbeanstalk.com/tests Authorization:"Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpZCI6MiwiaWF0IjoxNTg3ODY3NzQ4LCJleHAiOjE1OTA0NTk3NDh9.McAi1b-F3IT2Mw90652AprEMtknJrW66Aw5FGMBOTj0"` -### Use query parameters to filter for Multiple Content Type +### Use query parameters to filter for Multiple Content Type You can use query parameters to filter requests made to the API. @@ -74,17 +116,17 @@ The syntax is `?field_operator=value`, e.g `?title_contains=test`, after the end `http "http://strapi-prod.eu-west-1.elasticbeanstalk.com/tests?title_contains=test" Authorization:"Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpZCI6MSwiaXNBZG1pbiI6dHJ1ZSwiaWF0IjoxNTg3ODY3NzMwLCJleHAiOjE1OTA0NTk3MzB9.XXdoZUk_GuOION2KlpeWZ7qwXAoEq9vTlIeD2XTnJxY"` -## S3 Upload Addon +## S3 Upload Addon You should add the `strapi-provider-upload-aws-s3` extension using NPM. Make sure you add the same version of Strapi you are using. `npm i strapi-provider-upload-aws-s3@3.0.0-beta.20` -### AWS Resources +### AWS Resources You should have an S3 bucket with public access, and an AWS account that has a policy to access the bucket. -### Configuration +### Configuration You should create a settings file at `./extensions/upload/config/settings.json`. @@ -136,7 +178,7 @@ if (process.env.NODE_ENV === "production") { } ``` -## Fix Version Numbers +## Fix Version Numbers When using Strapi you should make sure the version numbers for **all** dependencies in `./package.json` are fixed for Strapi modules. You cannot mix and match and upgrade arbitrarily. @@ -163,7 +205,7 @@ An example is: } ``` -## Strapi in git +## Strapi in git To have a strapi project in github you should remove the: @@ -179,11 +221,11 @@ When cloning from the repo you should then do a: You can then run Strapi with `npm run develop` or `NODE_ENV=production npm run start`. -## Cloudformation +## Cloudformation (example of deploying an S3 bucket with static site `index.html`.) -### Creating templates +### Creating templates To create a cloudformation template you should create a `template.yaml`. This yaml file should have at the top: @@ -194,13 +236,13 @@ Description: A simple CloudFormation template Then you should add a `Resources` key and populate this with all the infrastructure you need to provision. -### Adding resources +### Adding resources Documentation for all AWS resources is: . A good approach is to use the GUI to create an object, and then lookup the cloudformation template as you go along. -### Using parameters +### Using parameters @@ -220,11 +262,11 @@ Parameters: Description: Enter t2.micro, m1.small, or m1.large. Default is t2.micro. ``` -### Using outputs +### Using outputs -### Using functions +### Using functions A list of all Cloudformation functions is: . @@ -234,7 +276,11 @@ A list of all Cloudformation functions is: Examples + +##### Select, GetAZs and Ref + +Example of `Fn::Select`, `Fn::GetAZs` and `!Ref`: ```yaml PublicSubnet1: @@ -246,7 +292,90 @@ PublicSubnet1: - Fn::GetAZs: !Ref "AWS::Region" ``` -### Outputs +##### GetAtt + +`Fn::GetAtt` differs from `Ref` in that `!GetAtt` gets an attribute of a resource, whereas `Ref` will reference the actual resource itself. An attribute is a return value of a resource. For example, a VPC resource has a `DefaultSecurityGroup` as an attribute that you can access. + +To see attributes that you can reference with `!GetAtt`, you should check the Cloudformation documentation for the resource in question and look at the "Return Values" header: . + +An example would be using `Fn::GetAtt` to export a return value for some object in a template: + +```yaml +Outputs: + PublicVPCIDDefaultSecurityGroup: + Description: The VPC ID. + Value: !GetAtt PublicVPC.DefaultSecurityGroup + Export: + Name: !Sub "${AWS::StackName}-PublicVPCIDDefaultSecurityGroup" +``` + +Long syntax: `Fn::GetAtt: [ logicalNameOfResource, attributeName ]` + +##### Sub + +A really good resource for Cloudformation functions is: . + +Using `Fn::Sub` allows you to substitue a variable into the string you are trying to create. You might want to substitue an input parameter in for example. + +```yaml +AppDnsRecord: + Type: AWS::Route53::RecordSet + Properties: + HostedZoneId: !ImportValue HostedZone-zone-id + Name: + Fn::Sub: + - "myapp.${HostedZoneName}" + - HostedZoneName: !ImportValue HostedZone-zone-name +``` + +Here we have referenced `${HostedZoneName}` - this is a temporary parameter in the sub command. At this point it does not exist, which is why we create a map which defines this variable as the second argument to Sub. In this example it is using `Fn::ImportValue` to import a resource from another Cloudformation stack. + +As this second argument is a map (denoted by the `:`), we can have multiple key,value pairs. + +```yaml +Name: + Fn::Sub: + - "myapp.${SubDomain}.${HostedZoneName}" + - HostedZoneName: !ImportValue HostedZone-zone-name + SubDomain: !ImportValue HostedZone-subzone-name +``` + +Note here that the second definition of the key, value pair does not have a leading `-`. We don't want to pass another argument to the `Sub` command, rather, we want to define additional key,value pair to be substituted in. + +If our import value name also depended on an input parameter (say our imported value name depeneded on a stack name) we would have to use nested sub functions. In the above example we are simply importing a static import value, the string is hardcoded, if we wanted this to be dynamic, and be populated from an input parameter, then we can use: + +```yaml +Parameters: + Route53StackName: + Type: String + +Resources: + AppDnsRecord: + Type: AWS::Route53::RecordSet + Properties: + HostedZoneName: + Fn::ImportValue: !Sub "${Route53StackName}-zone-name" + Name: + Fn::Sub: + - "myapp.${ZoneName}" + - ZoneName: + Fn::ImportValue: !Sub "${Route53StackName}-zone-name" +``` + +Pay attention to the double indentation after `ZoneName`!. + +We have to use the long form of `Fn::ImportValue` here and not the shorthand - this is a Cloudformation restriction. + +#### Short form + +If you are writing templates in yaml there is a long and shortform available. + +An example for the `Sub` function: + +- Longform `Fn::Sub: String` +- Shortform `!Sub String` + +### Outputs You can use the `Outputs:` header in your Cloudformation templates to specify outputs to be used in other Cloudformation templates. @@ -267,7 +396,7 @@ Outputs: You can refer to these in ELB `./config` files for example - allowing you to dynamically link to other AWS resources in your ELB environment. -### Referencing other resources +### Referencing other resources internally. You can reference other resources in the template. This is useful say if you want to define a VPC and a subnet and reference the VPC from the subnet. @@ -277,17 +406,74 @@ To do this you should use the `!Ref` function: VpcId: !Ref PublicVPC ``` -#### Pesudeo references +Note that this is a special syntax, it doesn't have `Fn::` in the long form and the short form, `!Ref` is actually longer than the long form in this case. + + + +#### Pesudeo references You can also reference certain AWS references: . Examples include `AWS::AccountId` and `AWS::StackName` among others. -### Deploy a stack/template +### Referencing other resources from external templates + +Say we have a Cloudformation template where we have created a VPC: + +```yaml +Outputs: + PublicSubnet0ID: + Description: The ID of the subnet. + Value: !Ref PublicSubnet0 + Export: + Name: !Sub "${AWS::StackName}-PublicSubnet0" +``` + +We want to be able to use this, dynamically, in another template. + +To do this we can use the `Fn::Sub` and `Fn::ImportValue` functions. + +```yaml +Parameters: + StackName: + Description: The stack name of another CloudFormation template. This is used + to prepend the name of other resources in other templates. + Type: String +Resources: + RDSSubnetGroup: + Type: AWS::RDS::DBSubnetGroup + Properties: + DBSubnetGroupDescription: A subnet group for the RDS instance. + SubnetIds: + - Fn::ImportValue: !Sub "${StackName}-PublicSubnet0" + - Fn::ImportValue: !Sub "${StackName}-PublicSubnet1" +``` + +### Deploy a stack/template To deploy, you should run the command: `aws cloudformation deploy --template-file template.yaml --stack-name static-website` -### Tags +### Passing in parameters + +You can define parameters in its own section in a Cloudformation template: + +```yaml +Parameters: + StackName: + Description: The stack name of another CloudFormation template. This is used + to prepend the name of other resources in other templates. + Type: String +``` + +You can set a default value which will be used if no value is passed in. + +To pass values in using the CLI you should use the `--parameter-overrides` argument and pass them in as key=value pairs seperated by a space: + +```bash +--parameter-overrides StackName=temp-vpc +``` + +### Tags When setting tags you can set them on individual resources in the Cloudformation template: @@ -307,13 +493,13 @@ Tags: Alternatively if you have many tags to be shared across all resources you can set them when you use the CLI to deploy: `--tags git=web-dev owner=home project=strapi-elb test=true deployment=cloudformation` -### Updating stack +### Updating stack To update a stack you can use `deploy`. Note that the default behaviour is to create the new resources side by side, then once successful remove the old ones. You may run into errors when updating certain resources (updating a VPC subnet will fail as it has to create the new subnet alongside the existing one). You should remove the old stack by doing `delete-stack` first. `aws cloudformation delete-stack --stack-name temp-vpc --profile admin` -### Failure +### Failure If something goes wrong, you can use `describe-stack-events` and pass the `stack-name` to find the events leading up to the failure: `aws cloudformation describe-stack-events --stack-name strapi-s3`. @@ -321,7 +507,7 @@ If this is the first time you are creating a stack you will not be able to re-de You can delete a stack by running: `aws --profile admin cloudformation delete-stack --stack-name strapi-s3`. -### Stacks +### Stacks @@ -329,9 +515,9 @@ A cloudformation stack is a collection of AWS resources that you can manage as a Using stacks means AWS will treat all resources as a single unit. They must all be created or destroyed successfully to be created or deleted. If a resource cannot be created, Cloudformation will roll the stack back to the previous configuration and delete any interim resources that were created. -### Snippets +### Snippets -#### Deploy a template/stack +#### Deploy a template/stack `aws --profile admin cloudformation deploy --template-file ./01-stack-storage.yaml --stack-name strapi-s3` @@ -339,11 +525,11 @@ You can pass paramter values in with `--paramter-overrides KEY=VALUE`: `--parameter-overrides TestParameter="some test string"` -#### Destroy a stack +#### Destroy a stack `aws --profile admin cloudformation delete-stack --stack-name strapi-s3` -## Tags +## Tags Suggested tags for all AWS resources are: @@ -356,7 +542,7 @@ Suggested tags for all AWS resources are: | environment | environment resource belongs to | `dev`, `prod` | | deployment | AWS tool used for deployment | `cloudformation`, `elb` | -### Cloudformation default tags +### Cloudformation default tags For Cloudformation resources the following tags get applied automatically: diff --git a/tempnotes.md b/tempnotes.md index 31714ed..d256e2f 100644 --- a/tempnotes.md +++ b/tempnotes.md @@ -1,6 +1,29 @@ + + +- [Decoupling](#Decoupling) +- [Creating Database + VPC + Subnets in Cloudformation](#CreatingDatabaseVPCSubnetsinCloudformation) +- [Single instance (no load balancer)](#Singleinstancenoloadbalancer) + _ [EC2::VPC](#EC2::VPC) + _ [Enable DNS](#EnableDNS) + _ [EC2::Subnet](#EC2::Subnet) + _ [EC2::InternetGateway](#EC2::InternetGateway) + _ [EC2::VPCGatewayAttachment](#EC2::VPCGatewayAttachment) + _ [AWS::EC2::RouteTable](#AWS::EC2::RouteTable) + _ [AWS::EC2::Route](#AWS::EC2::Route) + _ [AWS::EC2::SubnetRouteTableAssociation](#AWS::EC2::SubnetRouteTableAssociation) +- [Running notes](#Runningnotes) +- [Reference an input parameter, or a resource ID from inside current template](#ReferenceaninputparameteroraresourceIDfrominsidecurrenttemplate) \* [Using `Fn::Sub`](#UsingFn::Sub) +- [Dynamically referencing resources from another stack.](#Dynamicallyreferencingresourcesfromanotherstack.) + + + + # Temp Notes -## Decoupling +## Decoupling When creating an ELB instance with `--single` and `--database` the following is created as part of the ELB deployment: @@ -10,7 +33,7 @@ When creating an ELB instance with `--single` and `--database` the following is Is the security group created without a databse? (probably yes...) -## Creating Database + VPC + Subnets in Cloudformation +## Creating Database + VPC + Subnets in Cloudformation Template from AWS showing cross-stack referencing and creating and referencing a VPC: . @@ -24,7 +47,7 @@ You should use this VPC for you RDS instance. Creating a VPC for ELB (with RDS) -## Single instance (no load balancer) +## Single instance (no load balancer) Example cloudformation template that ELB uses: . @@ -56,13 +79,13 @@ You should then create a public route table and associate it with the VPC you ha You should then create a public route. You can then attach the internet gateway attachment to this route and specify a list of IPs that will go out to the internet. To allow all trafic to the internet set a `DestinationCidrBlock` of `0.0.0.0/0`. -### EC2::VPC +### EC2::VPC -#### Enable DNS +#### Enable DNS Enable `EnableDnsHostnames` + `EnableDnsSupport` - this allows resources in the VPC to use DNS in AWS. -### EC2::Subnet +### EC2::Subnet Go to the EC2 dashboard to find all availability zones. Create a subnet for each zone. @@ -71,33 +94,70 @@ Go to the EC2 dashboard to find all availability zones. Create a subnet for each - `CidrBlock` - `MapPublicIpOnLaunch` -### EC2::InternetGateway +### EC2::InternetGateway -### EC2::VPCGatewayAttachment +### EC2::VPCGatewayAttachment - `VpcId` - `InternetGatewayId` -### AWS::EC2::RouteTable +### AWS::EC2::RouteTable - `VpcId` -### AWS::EC2::Route +### AWS::EC2::Route - `RouteTableId` - `DestinationCidrBlock` - `GatewayId` -### AWS::EC2::SubnetRouteTableAssociation +### AWS::EC2::SubnetRouteTableAssociation - `SubnetId` - `RouteTableId` -## Running notes +## Running notes -If we specify the VPC + Subnets from Cloudformation in a config file, will it create the security groups automatically for the EC2 instances? +If we specify the VPC + Subnets from Cloudformation in a config file, will it create the security groups automatically for the EC2 instances? - Yes Database can use existing subnets. Database needs a security group creating EC2 security groups automatically created and associated with the VPC. Use aws:ec2:vpc (https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/command-options-general.html#command-options-general-ec2vpc) + +### Database + +Needs: + +- `AWS::RDS::DBSubnetGroup` +- `AWS::EC2::SecurityGroupIngress` +- `AWS::RDS::DBInstance` + +Default ports: + +| Database Engine | Default Port | +| -------------------- | ------------ | +| Aurora/MySQL/MariaDB | 3306 | +| PostgreSQL | 5432 | +| Oracle | 1521 | +| SQL Server | 1433 | +| DynamoDB | 8000 | + + + +## Work Commands + +### deploy + +`aws --profile admin cloudformation deploy --template-file ./02-stack-vpc.yaml --stack-name temp-vpc --tags git=web-dev owner=home project=strapi-elb test=true deployment=cloudformation` + +`aws --profile admin cloudformation deploy --template-file ./03-stack-rdsinstance.yaml --stack-name temp --parameter-overrides StackName=temp-vpc` + +### delete + +`aws --profile admin cloudformation delete-stack --stack-name temp-vpc` + +`aws --profile admin cloudformation delete-stack --stack-name temp` + +List of all RDS Engines available under "Engine" header: . + diff --git a/todo.md b/todo.md index 6a583a0..d0822c0 100644 --- a/todo.md +++ b/todo.md @@ -57,31 +57,39 @@ Recreate env with database, check the DB subnets - are they the same as the EC2 Can we use cloudformation functions (imports) in .config files under option_settings? (reference a VPC that already exists?) (https://www.reddit.com/r/aws/comments/a2uoae/is_there_a_way_to_reference_an_elastic_beanstalk/) Yes? Yes but only certain functions: . You cannot use FN::ImportValue to reference a resource in another Cloudformation stack. -Append the stackname to the outputs for the VPC and Subnets - -{ "Fn::ImportValue": "awseb-e-abcdefghijklmno-AutoScalingGroup" } - Use join for subnets Name: !Join [ ":", [ !Ref "AWS::StackName", AccountVPC ] ] Updating 07 config to accept a parameter +For DB -Append the stackname to the outputs for the VPC and Subnets +Follow -!Sub "${AWS::StackName}-VPCID" +Create a subnet group (add each subnet to this group) +Create a security Group for DB +Create a SecurityGroupIngress +Use existing public VPC and subnet -Name: !Sub "${AWS::StackName}-ELBStrapiPublicVPC" -Name: !Sub "${AWS::StackName}-ELBStrapiSubnet0" -Name: !Sub "${AWS::StackName}-ELBStrapiSubnet1" -Name: !Sub "${AWS::StackName}-ELBStrapiSubnet2" +What is the RDS hostname inside the container? -temp-vpc-ELBStrapiPublicVPC -temp-vpc-ELBStrapiSubnet0 -temp-vpc-ELBStrapiSubnet1 -temp-vpc-ELBStrapiSubnet2 +Follow a naming convention for likewise cloudformation templates -{ "Fn::ImportValue": "awseb-e-abcdefghijklmno-AutoScalingGroup" } +E.g. -Use join for subnets Name: !Join [ ":", [ !Ref "AWS::StackName", AccountVPC ] ] +```yaml +Outputs: + PublicVPCID: + Description: The VPC ID. + Value: !Ref PublicVPC + Export: + Name: !Sub "${AWS::StackName}-ELBStrapiPublicVPC" +``` -Updating 07 config to accept a parameter +Defines a VPC. We can then pass in the stackname to another CF template and it can reference this VPC. The VPC names are static between projects (they don't have to be but here they are). + + +Check if the security group of the CF RDS matches that of ELB RDS. They should specify the same (one postgres inbound on 5432 and all traffic all/all inbound) + +Do we need a seperate security group for the database? - The answer should be that there is a seperate security group for RDS, and one for EC2. EC2 one should be created by ELB automatically. + +Does the db and the ec2 instances share the same VPC?