cv/street_group_tech_test
1
0
Fork 0
mirror of https://github.com/dtomlinson91/street_group_tech_test synced 2025-12-22 11:55:45 +00:00
Code Issues Projects Releases Wiki Activity

moving dataflow notes

Browse Source
This commit is contained in:
Daniel Tomlinson
2021-09-26 17:15:17 +01:00
parent bb71d55f8c
commit fded858932
1 changed files with 8 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files

24
notes/documentation/dataflow.md Normal file
View File

@@ -0,0 +1,24 @@
# DataFlow
<https://cloud.google.com/dataflow/docs/quickstarts/quickstart-python>
Export env variable:
`export GOOGLE_APPLICATION_CREDENTIALS="/home/dtomlinson/git-repos/work/street_group/street_group_tech_test/street-group-0c490d23a9d0.json"`
Run the pipeline:
python -m analyse_properties.main \
--region europe-west2 \
--input gs://street-group-technical-test-dmot/input/pp-monthly-update-new-version.csv \
--output gs://street-group-technical-test-dmot/output/pp-monthly-update-new-version \
--runner DataflowRunner \
--project street-group \
--temp_location gs://street-group-technical-test-dmot/tmp
## Errors
Unsubscriptable error on window:
<https://stackoverflow.com/questions/42276520/what-does-object-of-type-unwindowedvalues-has-no-len-mean>