AWS BILLS TO IMPACTS

[da-ekchajzer]

NOM

AWS BILLS TO IMPACTS

STORY

As a FinOps, I want to retrieve the impacts of my AWS instances from my billing documents.

GUIDELINES

• Parse AWS billing documents
• Create API objects
• Retrieve an aggregate impacts

HELP

RESOURCES

• AWS route documentation : http://hackaton.boavizta.org/Reference/cloud_route/
• Example of CSV :

Instance Type	c5d.4xlarge($)	c5.9xlarge($)	c5.4xlarge($)	i3.xlarge($)	c5.12xlarge($)	c5.2xlarge($)	c5.xlarge($)	m5.xlarge($)	r5.4xlarge($)	c5n.4xlarge($)	m5d.4xlarge($)	m5n.4xlarge($)	c5a.4xlarge($)	r5d.4xlarge($)	m5.2xlarge($)	m5.large($)	m5.4xlarge($)	m5dn.4xlarge($)	t3.medium($)	r3.4xlarge($)	r5.2xlarge($)	t3.xlarge($)	m3.xlarge($)	c4.large($)	c5a.xlarge($)	c5n.2xlarge($)	t3.small($)	m5n.2xlarge($)	i3.large($)	m3.medium($)	c5.large($)	t3a.medium($)	c5d.2xlarge($)	t3.large($)	m4.xlarge($)	r5.xlarge($)	m5d.2xlarge($)	m5dn.2xlarge($)	r5d.xlarge($)	r5dn.xlarge($)	r5ad.xlarge($)	t2.small($)	t2.micro($)	t3.nano($)	r5a.xlarge($)	t3a.small($)	c4.2xlarge($)	i3en.xlarge($)	r4.xlarge($)	r6g.2xlarge($)	Total cost ($)	c5d.4xlarge(Hrs)	c5.9xlarge(Hrs)	c5.4xlarge(Hrs)	i3.xlarge(Hrs)	c5.12xlarge(Hrs)	c5.2xlarge(Hrs)	c5.xlarge(Hrs)	m5.xlarge(Hrs)	r5.4xlarge(Hrs)	c5n.4xlarge(Hrs)	m5d.4xlarge(Hrs)	m5n.4xlarge(Hrs)	c5a.4xlarge(Hrs)	r5d.4xlarge(Hrs)	m5.2xlarge(Hrs)	m5.large(Hrs)	m5.4xlarge(Hrs)	m5dn.4xlarge(Hrs)	t3.medium(Hrs)	r3.4xlarge(Hrs)	r5.2xlarge(Hrs)	t3.xlarge(Hrs)	m3.xlarge(Hrs)	c4.large(Hrs)	c5a.xlarge(Hrs)	c5n.2xlarge(Hrs)	t3.small(Hrs)	m5n.2xlarge(Hrs)	i3.large(Hrs)	m3.medium(Hrs)	c5.large(Hrs)	t3a.medium(Hrs)	c5d.2xlarge(Hrs)	t3.large(Hrs)	m4.xlarge(Hrs)	r5.xlarge(Hrs)	m5d.2xlarge(Hrs)	m5dn.2xlarge(Hrs)	r5d.xlarge(Hrs)	r5dn.xlarge(Hrs)	r5ad.xlarge(Hrs)	t2.small(Hrs)	t2.micro(Hrs)	t3.nano(Hrs)	r5a.xlarge(Hrs)	t3a.small(Hrs)	c4.2xlarge(Hrs)	i3en.xlarge(Hrs)	r4.xlarge(Hrs)	r6g.2xlarge(Hrs)	Total usage (Hrs)
Instance Type Total	36350.2603570022	30614.5202426906	24316.8331034817	9326.6919432198	8104.1107502527	4855.8905596833	4136.5406810251	2770.7789043742	1725.7048499975	1587.3778770002	1572.5797419984	1305.8370539994	1250.8661040004	1233.6222369998	1128.8064806767	1125.0435701273	936.1562239995	874.9240509973	759.6965529723	580.4618090012	297.1334089999	291.6871495004	286.44	177.460359804	144.110458184	137.966512	106.172781096	84.588503	83.5433819862	71.424	61.739	51.9171376346	50.138778	39.726104538	32.4023970003	28.9672449999	28.5207710001	27.8938040001	24.303523	17.100862	12.017345	11.2837706464	10.7312	9.0794185684	6.642005	5.597889	0.368046	0	-7.105427357601E-15	-2.1316282072803E-14	136655.658944458	39350.279724	16768.742489	30460.513066	64835	3723.049446	11814.604438	24126.475004	15049.279999	5384.991932	5246.132779	4143.136387	3480.223602	3379.313599	3102.154721	2979.103056	12648	3048.723324	2523.009998	28763.399448	2009.227844	2005.909442	2789.594722	744	1408.415554	762.377776	936.199166	7654.480278	507.324721	1488	744	744	2232	376.348333	743	372.172772	1849.565278	151.71833	170.757775	305.066387	233.618332	164.171389	744.156388	1488	2098.208613	90.737777	726.944445	2.859722	10416	2721.085836	4464	331970.073892
2022-01-01	36350.2603570022	30614.5202426906	24316.8331034817	9326.6919432198	8104.1107502527	4855.8905596833	4136.5406810251	2770.7789043742	1725.7048499975	1587.3778770002	1572.5797419984	1305.8370539994	1250.8661040004	1233.6222369998	1128.8064806767	1125.0435701273	936.1562239995	874.9240509973	759.6965529723	580.4618090012	297.1334089999	291.6871495004	286.44	177.460359804	144.110458184	137.966512	106.172781096	84.588503	83.5433819862	71.424	61.739	51.9171376346	50.138778	39.726104538	32.4023970003	28.9672449999	28.5207710001	27.8938040001	24.303523	17.100862	12.017345	11.2837706464	10.7312	9.0794185684	6.642005	5.597889	0.368046	0	-7.105427357601E-15	-2.1316282072803E-14	136655.658944458	39350.279724	16768.742489	30460.513066	64835	3723.049446	11814.604438	24126.475004	15049.279999	5384.991932	5246.132779	4143.136387	3480.223602	3379.313599	3102.154721	2979.103056	12648	3048.723324	2523.009998	28763.399448	2009.227844	2005.909442	2789.594722	744	1408.415554	762.377776	936.199166	7654.480278	507.324721	1488	744	744	2232	376.348333	743	372.172772	1849.565278	151.71833	170.757775	305.066387	233.618332	164.171389	744.156388	1488	2098.208613	90.737777	726.944445	2.859722	10416	2721.085836	4464	331970.073892

• File : Exemple Cost Explorer - Usage Type Group EC2 Running Hours - Group By Instance Type.csv

[demeringo]

I am not familiar with the project but https://github.com/cloud-carbon-footprint/cloud-carbon-footprint seems to have the ability to access and analyze the billing and usage data from AWS, GCP or Azure.

According to the doc, it seems to imply using Athena or other vendor specific query tool to extract data from the usage report stored in a S3 bucket or equivalent by the cloud provider. So maybe this is a bit too complex or expensive for our use case.

[da-ekchajzer]

I have investigated the cloud carbon foundation implementation, which is heavy to implement, and I don't think that it would be our added value to develop such middleware (correct me if I am wrong).

The idea is to compute cold data (a manual export of EC2 instances usage). Such approach was used by @bpetit for azure instances. I am running some instances in a personal AWS environment to obtain the template of export (the one we want to process).

I think we can get an export : https://docs.aws.amazon.com/cost-management/latest/userguide/ce-saving.html

But I need to wait 24h for service to be available.

If you have a way to get a template or even anonymized data, feel free to propose !

[demeringo]

I do not have much data at hand.

I attach a sample cost report (ec2 monthly cost and hours of usage by instance type).

In this case we have only a single instance type (t2micro) but it can give an idea of the structure.
monthly-ec2-instance-hours-costs.csv

[da-ekchajzer]

Thank you very much I had a contributor giving me the same type of export which I guess is sufficient ! (See in the first comment).

See you tomorrow!

[bpetit]

Should we move this issue to https://github.com/Boavizta/cloud-scanner/ ?