Theme
Description
With a more flexible structure, data pipelines can be implemented in a way that is hard to understand (see HU1.4 - Less structure can lead to hard to understand implementation).
Because developers are not forced to follow a specific structure, to create easily readable code they must be disciplined and experienced enough to maintain good coding habits.
Additionally, due to the ease of use of Python it often gets used to implement data pipeline prototypes in a script-like fashion. Without developer discipline the danger of just promoting these prototypes to production code in the end exists, meaning hard to understand and unstructured data pipeline code is shared.
Representative Quotes
1: The concept of blocks: you can manually create this in python, but hardly anybody will do this (…) 3: The structure: in Jayvee you are forced to give one whole pipeline description, in Python you can get away without doing this.
- S26
It depends on the experience of the user; if well experienced in Python, it would be easier for him to understand the Python code faster. For a beginner in coding I think the fact that Jayvee is self explaining makes it easier to understand data pipelines.
- S28
Must use good programming habits to write good readable pipelines.
- S29
The code was not clearly written. Using a proper functional approach would have been better so I could understand what a particular function is doing.
- S46