Skip to content

MINOR: Add logging to Datalake processing table names #20968

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 2 commits into from
Apr 28, 2025
Merged

MINOR: Add logging to Datalake processing table names #20968

merged 2 commits into from
Apr 28, 2025
+1 −1

Conversation

ayush-shah
Copy link
Member

This is to add proper logging since debug logging doesn't provide any name and if the processing starts, if the file fails due to any columns from within, it doesn't show which file it is a part of.

Type of change:

  • Bug fix
  • Improvement
  • New feature
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Documentation

Checklist:

  • I have read the CONTRIBUTING document.
  • My PR title is Fixes <issue-number>: <short explanation>
  • I have commented on my code, particularly in hard-to-understand areas.
  • For JSON Schema changes: I updated the migration scripts or explained why it is not needed.

@github-actions github-actions bot added Ingestion safe to test Add this label to run secure Github workflows on PRs labels Apr 24, 2025
@ayush-shah ayush-shah marked this pull request as ready for review April 27, 2025 05:16
@ayush-shah ayush-shah requested a review from a team as a code owner April 27, 2025 05:16
@github-actions GitHub Actions
Copy link
Contributor

🛡️ TRIVY SCAN RESULT 🛡️

Target: openmetadata-ingestion-base-slim:trivy (debian 12.10)

Vulnerabilities (23)

Package Vulnerability ID Severity Installed Version Fixed Version
libperl5.36 CVE-2024-56406 🚨 HIGH 5.36.0-7+deb12u1 5.36.0-7+deb12u2
linux-libc-dev CVE-2024-26982 🚨 HIGH 6.1.129-1 6.1.133-1
linux-libc-dev CVE-2024-35866 🚨 HIGH 6.1.129-1 6.1.133-1
linux-libc-dev CVE-2024-50246 🚨 HIGH 6.1.129-1 6.1.133-1
linux-libc-dev CVE-2024-53166 🚨 HIGH 6.1.129-1 6.1.133-1
linux-libc-dev CVE-2024-58002 🚨 HIGH 6.1.129-1 6.1.133-1
linux-libc-dev CVE-2025-21702 🚨 HIGH 6.1.129-1 6.1.133-1
linux-libc-dev CVE-2025-21855 🚨 HIGH 6.1.129-1 6.1.133-1
linux-libc-dev CVE-2025-21858 🚨 HIGH 6.1.129-1 6.1.133-1
linux-libc-dev CVE-2025-21905 🚨 HIGH 6.1.129-1 6.1.133-1
linux-libc-dev CVE-2025-21919 🚨 HIGH 6.1.129-1 6.1.133-1
linux-libc-dev CVE-2025-21920 🚨 HIGH 6.1.129-1 6.1.133-1
linux-libc-dev CVE-2025-21926 🚨 HIGH 6.1.129-1 6.1.133-1
linux-libc-dev CVE-2025-21928 🚨 HIGH 6.1.129-1 6.1.133-1
linux-libc-dev CVE-2025-21934 🚨 HIGH 6.1.129-1 6.1.133-1
linux-libc-dev CVE-2025-21945 🚨 HIGH 6.1.129-1 6.1.133-1
linux-libc-dev CVE-2025-21968 🚨 HIGH 6.1.129-1 6.1.133-1
linux-libc-dev CVE-2025-21979 🚨 HIGH 6.1.129-1 6.1.133-1
linux-libc-dev CVE-2025-21991 🚨 HIGH 6.1.129-1 6.1.133-1
linux-libc-dev CVE-2025-21993 🚨 HIGH 6.1.129-1 6.1.133-1
perl CVE-2024-56406 🚨 HIGH 5.36.0-7+deb12u1 5.36.0-7+deb12u2
perl-base CVE-2024-56406 🚨 HIGH 5.36.0-7+deb12u1 5.36.0-7+deb12u2
perl-modules-5.36 CVE-2024-56406 🚨 HIGH 5.36.0-7+deb12u1 5.36.0-7+deb12u2

🛡️ TRIVY SCAN RESULT 🛡️

Target: Java

Vulnerabilities (46)

Package Vulnerability ID Severity Installed Version Fixed Version
com.fasterxml.jackson.core:jackson-databind CVE-2020-36518 🚨 HIGH 2.11.4 2.13.2.1, 2.12.6.1
com.fasterxml.jackson.core:jackson-databind CVE-2021-46877 🚨 HIGH 2.11.4 2.12.6, 2.13.1
com.fasterxml.jackson.core:jackson-databind CVE-2022-42003 🚨 HIGH 2.11.4 2.12.7.1, 2.13.4.2
com.fasterxml.jackson.core:jackson-databind CVE-2022-42004 🚨 HIGH 2.11.4 2.12.7.1, 2.13.4
com.fasterxml.jackson.core:jackson-databind CVE-2020-36518 🚨 HIGH 2.13.0 2.13.2.1, 2.12.6.1
com.fasterxml.jackson.core:jackson-databind CVE-2021-46877 🚨 HIGH 2.13.0 2.12.6, 2.13.1
com.fasterxml.jackson.core:jackson-databind CVE-2022-42003 🚨 HIGH 2.13.0 2.12.7.1, 2.13.4.2
com.fasterxml.jackson.core:jackson-databind CVE-2022-42004 🚨 HIGH 2.13.0 2.12.7.1, 2.13.4
com.google.code.gson:gson CVE-2022-25647 🚨 HIGH 2.2.4 2.8.9
com.google.protobuf:protobuf-java CVE-2021-22569 🚨 HIGH 2.5.0 3.16.1, 3.18.2, 3.19.2
com.google.protobuf:protobuf-java CVE-2021-22570 🚨 HIGH 2.5.0 3.15.0
com.google.protobuf:protobuf-java CVE-2022-3509 🚨 HIGH 2.5.0 3.16.3, 3.19.6, 3.20.3, 3.21.7
com.google.protobuf:protobuf-java CVE-2022-3510 🚨 HIGH 2.5.0 3.16.3, 3.19.6, 3.20.3, 3.21.7
com.google.protobuf:protobuf-java CVE-2024-7254 🚨 HIGH 2.5.0 3.25.5, 4.27.5, 4.28.2
com.google.protobuf:protobuf-java CVE-2021-22569 🚨 HIGH 3.3.0 3.16.1, 3.18.2, 3.19.2
com.google.protobuf:protobuf-java CVE-2021-22570 🚨 HIGH 3.3.0 3.15.0
com.google.protobuf:protobuf-java CVE-2022-3509 🚨 HIGH 3.3.0 3.16.3, 3.19.6, 3.20.3, 3.21.7
com.google.protobuf:protobuf-java CVE-2022-3510 🚨 HIGH 3.3.0 3.16.3, 3.19.6, 3.20.3, 3.21.7
com.google.protobuf:protobuf-java CVE-2024-7254 🚨 HIGH 3.3.0 3.25.5, 4.27.5, 4.28.2
com.google.protobuf:protobuf-java CVE-2021-22569 🚨 HIGH 3.7.1 3.16.1, 3.18.2, 3.19.2
com.google.protobuf:protobuf-java CVE-2021-22570 🚨 HIGH 3.7.1 3.15.0
com.google.protobuf:protobuf-java CVE-2022-3509 🚨 HIGH 3.7.1 3.16.3, 3.19.6, 3.20.3, 3.21.7
com.google.protobuf:protobuf-java CVE-2022-3510 🚨 HIGH 3.7.1 3.16.3, 3.19.6, 3.20.3, 3.21.7
com.google.protobuf:protobuf-java CVE-2024-7254 🚨 HIGH 3.7.1 3.25.5, 4.27.5, 4.28.2
com.nimbusds:nimbus-jose-jwt CVE-2023-52428 🚨 HIGH 9.8.1 9.37.2
commons-io:commons-io CVE-2024-47554 🚨 HIGH 2.11.0 2.14.0
commons-io:commons-io CVE-2024-47554 🚨 HIGH 2.8.0 2.14.0
dnsjava:dnsjava CVE-2024-25638 🚨 HIGH 2.1.7 3.6.0
io.airlift:aircompressor CVE-2024-36114 🚨 HIGH 0.21 0.27
net.minidev:json-smart CVE-2021-31684 🚨 HIGH 1.3.2 1.3.3, 2.4.4
net.minidev:json-smart CVE-2023-1370 🚨 HIGH 1.3.2 2.4.9
org.apache.avro:avro CVE-2024-47561 🔥 CRITICAL 1.11.0 1.11.4
org.apache.avro:avro CVE-2023-39410 🚨 HIGH 1.11.0 1.11.3
org.apache.avro:avro CVE-2024-47561 🔥 CRITICAL 1.7.7 1.11.4
org.apache.avro:avro CVE-2023-39410 🚨 HIGH 1.7.7 1.11.3
org.apache.derby:derby CVE-2022-46337 🔥 CRITICAL 10.14.2.0 10.14.3, 10.15.2.1, 10.16.1.2, 10.17.1.0
org.apache.hadoop:hadoop-common CVE-2022-25168 🔥 CRITICAL 3.3.2 2.10.2, 3.2.4, 3.3.3
org.apache.ivy:ivy CVE-2022-46751 🚨 HIGH 2.5.1 2.5.2
org.apache.mesos:mesos CVE-2018-1330 🚨 HIGH 1.4.3 1.6.0
org.apache.spark:spark-hive-thriftserver_2.12 CVE-2024-23945 🚨 HIGH 3.3.4 3.4.2
org.apache.thrift:libthrift CVE-2019-0205 🚨 HIGH 0.12.0 0.13.0
org.apache.thrift:libthrift CVE-2020-13949 🚨 HIGH 0.12.0 0.14.0
org.apache.zookeeper:zookeeper CVE-2023-44981 🔥 CRITICAL 3.6.2 3.7.2, 3.8.3, 3.9.1
org.xerial.snappy:snappy-java CVE-2023-34455 🚨 HIGH 1.1.8.4 1.1.10.1
org.xerial.snappy:snappy-java CVE-2023-43642 🚨 HIGH 1.1.8.4 1.1.10.4
org.yaml:snakeyaml CVE-2022-1471 🚨 HIGH 1.31 2.0

🛡️ TRIVY SCAN RESULT 🛡️

Target: Node.js

No Vulnerabilities Found

🛡️ TRIVY SCAN RESULT 🛡️

Target: Python

Vulnerabilities (1)

Package Vulnerability ID Severity Installed Version Fixed Version
Werkzeug CVE-2024-34069 🚨 HIGH 2.2.3 3.0.3

🛡️ TRIVY SCAN RESULT 🛡️

Target: /etc/ssl/private/ssl-cert-snakeoil.key

No Vulnerabilities Found

🛡️ TRIVY SCAN RESULT 🛡️

Target: /ingestion/pipelines/extended_sample_data.yaml

No Vulnerabilities Found

🛡️ TRIVY SCAN RESULT 🛡️

Target: /ingestion/pipelines/lineage.yaml

No Vulnerabilities Found

🛡️ TRIVY SCAN RESULT 🛡️

Target: /ingestion/pipelines/sample_data.json

No Vulnerabilities Found

🛡️ TRIVY SCAN RESULT 🛡️

Target: /ingestion/pipelines/sample_data.yaml

No Vulnerabilities Found

🛡️ TRIVY SCAN RESULT 🛡️

Target: /ingestion/pipelines/sample_usage.json

No Vulnerabilities Found

🛡️ TRIVY SCAN RESULT 🛡️

Target: /ingestion/pipelines/sample_usage.yaml

No Vulnerabilities Found

@github-actions GitHub Actions
Copy link
Contributor

🛡️ TRIVY SCAN RESULT 🛡️

Target: openmetadata-ingestion:trivy (debian 12.9)

Vulnerabilities (6)

Package Vulnerability ID Severity Installed Version Fixed Version
libperl5.36 CVE-2024-56406 🚨 HIGH 5.36.0-7+deb12u1 5.36.0-7+deb12u2
libxslt1.1 CVE-2024-55549 🚨 HIGH 1.1.35-1 1.1.35-1+deb12u1
libxslt1.1 CVE-2025-24855 🚨 HIGH 1.1.35-1 1.1.35-1+deb12u1
perl CVE-2024-56406 🚨 HIGH 5.36.0-7+deb12u1 5.36.0-7+deb12u2
perl-base CVE-2024-56406 🚨 HIGH 5.36.0-7+deb12u1 5.36.0-7+deb12u2
perl-modules-5.36 CVE-2024-56406 🚨 HIGH 5.36.0-7+deb12u1 5.36.0-7+deb12u2

🛡️ TRIVY SCAN RESULT 🛡️

Target: Java

Vulnerabilities (46)

Package Vulnerability ID Severity Installed Version Fixed Version
com.fasterxml.jackson.core:jackson-databind CVE-2020-36518 🚨 HIGH 2.11.4 2.13.2.1, 2.12.6.1
com.fasterxml.jackson.core:jackson-databind CVE-2021-46877 🚨 HIGH 2.11.4 2.12.6, 2.13.1
com.fasterxml.jackson.core:jackson-databind CVE-2022-42003 🚨 HIGH 2.11.4 2.12.7.1, 2.13.4.2
com.fasterxml.jackson.core:jackson-databind CVE-2022-42004 🚨 HIGH 2.11.4 2.12.7.1, 2.13.4
com.fasterxml.jackson.core:jackson-databind CVE-2020-36518 🚨 HIGH 2.13.0 2.13.2.1, 2.12.6.1
com.fasterxml.jackson.core:jackson-databind CVE-2021-46877 🚨 HIGH 2.13.0 2.12.6, 2.13.1
com.fasterxml.jackson.core:jackson-databind CVE-2022-42003 🚨 HIGH 2.13.0 2.12.7.1, 2.13.4.2
com.fasterxml.jackson.core:jackson-databind CVE-2022-42004 🚨 HIGH 2.13.0 2.12.7.1, 2.13.4
com.google.code.gson:gson CVE-2022-25647 🚨 HIGH 2.2.4 2.8.9
com.google.protobuf:protobuf-java CVE-2021-22569 🚨 HIGH 2.5.0 3.16.1, 3.18.2, 3.19.2
com.google.protobuf:protobuf-java CVE-2021-22570 🚨 HIGH 2.5.0 3.15.0
com.google.protobuf:protobuf-java CVE-2022-3509 🚨 HIGH 2.5.0 3.16.3, 3.19.6, 3.20.3, 3.21.7
com.google.protobuf:protobuf-java CVE-2022-3510 🚨 HIGH 2.5.0 3.16.3, 3.19.6, 3.20.3, 3.21.7
com.google.protobuf:protobuf-java CVE-2024-7254 🚨 HIGH 2.5.0 3.25.5, 4.27.5, 4.28.2
com.google.protobuf:protobuf-java CVE-2021-22569 🚨 HIGH 3.3.0 3.16.1, 3.18.2, 3.19.2
com.google.protobuf:protobuf-java CVE-2021-22570 🚨 HIGH 3.3.0 3.15.0
com.google.protobuf:protobuf-java CVE-2022-3509 🚨 HIGH 3.3.0 3.16.3, 3.19.6, 3.20.3, 3.21.7
com.google.protobuf:protobuf-java CVE-2022-3510 🚨 HIGH 3.3.0 3.16.3, 3.19.6, 3.20.3, 3.21.7
com.google.protobuf:protobuf-java CVE-2024-7254 🚨 HIGH 3.3.0 3.25.5, 4.27.5, 4.28.2
com.google.protobuf:protobuf-java CVE-2021-22569 🚨 HIGH 3.7.1 3.16.1, 3.18.2, 3.19.2
com.google.protobuf:protobuf-java CVE-2021-22570 🚨 HIGH 3.7.1 3.15.0
com.google.protobuf:protobuf-java CVE-2022-3509 🚨 HIGH 3.7.1 3.16.3, 3.19.6, 3.20.3, 3.21.7
com.google.protobuf:protobuf-java CVE-2022-3510 🚨 HIGH 3.7.1 3.16.3, 3.19.6, 3.20.3, 3.21.7
com.google.protobuf:protobuf-java CVE-2024-7254 🚨 HIGH 3.7.1 3.25.5, 4.27.5, 4.28.2
com.nimbusds:nimbus-jose-jwt CVE-2023-52428 🚨 HIGH 9.8.1 9.37.2
commons-io:commons-io CVE-2024-47554 🚨 HIGH 2.11.0 2.14.0
commons-io:commons-io CVE-2024-47554 🚨 HIGH 2.8.0 2.14.0
dnsjava:dnsjava CVE-2024-25638 🚨 HIGH 2.1.7 3.6.0
io.airlift:aircompressor CVE-2024-36114 🚨 HIGH 0.21 0.27
net.minidev:json-smart CVE-2021-31684 🚨 HIGH 1.3.2 1.3.3, 2.4.4
net.minidev:json-smart CVE-2023-1370 🚨 HIGH 1.3.2 2.4.9
org.apache.avro:avro CVE-2024-47561 🔥 CRITICAL 1.11.0 1.11.4
org.apache.avro:avro CVE-2023-39410 🚨 HIGH 1.11.0 1.11.3
org.apache.avro:avro CVE-2024-47561 🔥 CRITICAL 1.7.7 1.11.4
org.apache.avro:avro CVE-2023-39410 🚨 HIGH 1.7.7 1.11.3
org.apache.derby:derby CVE-2022-46337 🔥 CRITICAL 10.14.2.0 10.14.3, 10.15.2.1, 10.16.1.2, 10.17.1.0
org.apache.hadoop:hadoop-common CVE-2022-25168 🔥 CRITICAL 3.3.2 2.10.2, 3.2.4, 3.3.3
org.apache.ivy:ivy CVE-2022-46751 🚨 HIGH 2.5.1 2.5.2
org.apache.mesos:mesos CVE-2018-1330 🚨 HIGH 1.4.3 1.6.0
org.apache.spark:spark-hive-thriftserver_2.12 CVE-2024-23945 🚨 HIGH 3.3.4 3.4.2
org.apache.thrift:libthrift CVE-2019-0205 🚨 HIGH 0.12.0 0.13.0
org.apache.thrift:libthrift CVE-2020-13949 🚨 HIGH 0.12.0 0.14.0
org.apache.zookeeper:zookeeper CVE-2023-44981 🔥 CRITICAL 3.6.2 3.7.2, 3.8.3, 3.9.1
org.xerial.snappy:snappy-java CVE-2023-34455 🚨 HIGH 1.1.8.4 1.1.10.1
org.xerial.snappy:snappy-java CVE-2023-43642 🚨 HIGH 1.1.8.4 1.1.10.4
org.yaml:snakeyaml CVE-2022-1471 🚨 HIGH 1.31 2.0

🛡️ TRIVY SCAN RESULT 🛡️

Target: Node.js

No Vulnerabilities Found

🛡️ TRIVY SCAN RESULT 🛡️

Target: Python

Vulnerabilities (4)

Package Vulnerability ID Severity Installed Version Fixed Version
Werkzeug CVE-2024-34069 🚨 HIGH 2.2.3 3.0.3
apache-airflow-providers-common-sql CVE-2025-30473 🚨 HIGH 1.21.0 1.24.1
h11 CVE-2025-43859 🔥 CRITICAL 0.14.0 0.16.0
setuptools CVE-2024-6345 🚨 HIGH 65.5.1 70.0.0

🛡️ TRIVY SCAN RESULT 🛡️

Target: /etc/ssl/private/ssl-cert-snakeoil.key

No Vulnerabilities Found

🛡️ TRIVY SCAN RESULT 🛡️

Target: /home/airflow/.local/lib/python3.10/site-packages/openmetadata_managed_apis-1.8.0.0.dev0.dist-info/METADATA

No Vulnerabilities Found

🛡️ TRIVY SCAN RESULT 🛡️

Target: /home/airflow/openmetadata-airflow-apis/openmetadata_managed_apis.egg-info/PKG-INFO

No Vulnerabilities Found

@ayush-shah ayush-shah merged commit 7dd987a into main Apr 28, 2025
21 of 24 checks passed
@ayush-shah ayush-shah deleted the minor-datalake branch April 28, 2025 06:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ulixius9 ulixius9 ulixius9 approved these changes

Assignees
No one assigned
Labels
Ingestion safe to test Add this label to run secure Github workflows on PRs
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@ayush-shah @ulixius9