How to ingest a simple JSON file using Openmetadata API

[baharhmohammadi]

Hi Openmetadata team, thanks for the great work!
I have openmetadata running locally and I'd like to know how I can create custom entities using openmetadata API. The example json file I'm trying to ingest is the following:

[ { "namespace": "monorail.schemas.accounts_user_action.v2_1", "type": "record", "name": "accounts_user_action", "doc": "Fired anytime a user do a meaningful action (SMS login, email login, etc)", "fields": [ { "name": "user_uuid", "type": "string", "doc": "The Account UUID of the user", "privacy_handlers": "", "privacy": null }, { "name": "action", "type": "string", "doc": "The performed action (sms_sign_in, verify_phone, etc)", "privacy_handlers": "", "privacy": null }, { "name": "success", "type": "boolean", "doc": "The success of the action", "privacy_handlers": "", "privacy": null }, { "name": "client_timestamp_ms", "type": "long", "doc": "Unix timestamp in milliseconds (UTC) when event was fired.", "privacy_handlers": "", "privacy": null }, { "name": "email", "type": "string", "doc": "The user email address", "privacy_handlers": "", "privacy": { "type": "EMAIL", "tokenize": false } }, { "name": "client_uuid", "type": "string", "doc": "The Client UUID acting on behalf of the user.", "privacy_handlers": "", "privacy": null }, { "name": "phone", "type": "string", "doc": "The user phone number", "privacy_handlers": "", "privacy": { "type": "PHONE", "tokenize": false } }, { "name": "error_details", "type": "string", "doc": "The error details (code, message, etc) if any", "privacy_handlers": "", "privacy": null }, { "name": "event_details", "type": "string", "doc": "A JSON holding non PII information related to the action (current scope, etc)", "privacy_handlers": "", "privacy": null }, { "name": "benchmark", "type": "boolean", "doc": "If the action is done for benchmarking purpose.", "privacy_handlers": "", "privacy": null } ] } ]

I tried running metadata ingest command and passing this json file's path but it's not working, I'd like to know how I can achieve ingesting via Openmetadata API. Please let me know, thanks!

[sureshms]

@baharhmohammadi, want to understand what this entity is. Is it a document or schema of a Kafka topic? Current list of entities supported in OM is available here - https://github.com/open-metadata/OpenMetadata/tree/main/catalog-rest-service/src/main/resources/json/schema/entity

OpenMetadata does not allow arbitrary JSON document/records to be stored as entities. It must be one of the entities supported by the system or a new entity type needs to be created for that.

We can work together on modeling this as a metadata entity with more details from your answers.

[harshach]

@baharhmohammadi can you tell us what issues did you face. Also we are available on slack it will be easier to interact there, can you join slack.open-metadata.org. Please ping us with the issues you faced for using Kafka connector.

[harshach]

Few things here

1. Currently our connector support's confluent's schema registry
2. If you have your own registry or other home grown solution this requires little amount of work to get connected
3. Yes you can create a topic by directly ingesting as well

{ "cleanupPolicies": [ "delete" ], "description": "string", "maximumMessageSize": 0, "minimumInSyncReplicas": 0, "name": "topic_name" "partitions": 0, "replicationFactor": 0, "retentionSize": 0, "retentionTime": 0, "schemaText": "{jsonSchema}", "schemaType": "Json", "service": { "id": "uuid", "type": "messagingService" } }

You can use the payload like above and fill in the details and either use POST (to create) , PUT (create or update) a topic along with schema "/v1/topics"

1. you see "service" key at the end . This a reference to your Kafka cluster service. You can go to the OpenMetadata UI and create it

2. Once you create the service, take the service id by making a get call on api/v1/services/messagingServices you'll get a response like this

take the id value from there and replace the uuid part in the above json payload. 3. lastly you'll see schemaText and add your JsonSchema data in there as a json string . Rest of the fields are for topic configs

[harshach]

@baharhmohammadi please reach out to us on https://slack.open-metadata.org . We are available 24x7. Would like to understand what issues you are facing.

[baharhmohammadi]

great, thanks so much, I posted the errors I'm receiving on Slack in Support channel

[rae89]

Is this still a valid approach? I tried the suggestion provided in this post and got the following error: { "code": 400, "message": "Unrecognized field \"schemaText\" (class org.openmetadata.schema.api.data.CreateTopic), not marked as ignorable (17 known properties: \"messageSchema\", \"partitions\", \"name\", \"displayName\", \"topicConfig\", \"owner\", \"cleanupPolicies\", \"service\", \"tags\", \"replicationFactor\", \"description\", \"retentionSize\", \"maximumMessageSize\", \"retentionTime\", \"sampleData\", \"extension\", \"minimumInSyncReplicas\"])\n at [Source: (org.glassfish.jersey.message.internal.ReaderInterceptorExecutor$UnCloseableInputStream); line: 11, column: 18] (through reference chain: org.openmetadata.schema.api.data.CreateTopic[\"schemaText\"])" }