cv/street_group_tech_test
1
0
Fork 0
mirror of https://github.com/dtomlinson91/street_group_tech_test synced 2025-12-22 11:55:45 +00:00
Code Issues Projects Releases Wiki Activity

adding latest beam pipeline code for dataflow

Browse Source
This commit is contained in:
Daniel Tomlinson
2021-09-26 15:57:00 +01:00
parent e6ec110d54
commit 9f53c66975
2 changed files with 53 additions and 13 deletions
Download Patch File Download Diff File Expand all files Collapse all files

17
notes/dataflow.md Normal file
View File

@@ -0,0 +1,17 @@
# DataFlow
<https://cloud.google.com/dataflow/docs/quickstarts/quickstart-python>
Export env variable:
`export GOOGLE_APPLICATION_CREDENTIALS="/home/dtomlinson/git-repos/work/street_group/street_group_tech_test/street-group-0c490d23a9d0.json"`
Run the pipeline:
python -m analyse_properties.main \
--region europe-west2 \
--input gs://street-group-technical-test-dmot/input/pp-monthly-update-new-version.csv \
--output gs://street-group-technical-test-dmot/input/pp-monthly-update-new-version \
--runner DataflowRunner \
--project street-group \
--temp_location gs://street-group-technical-test-dmot/tmp