Remove too long atomic fields

[oguzhanunlu]

No description provided.

[istreeter]

For some loaders event_id is a a required field in the table. For example, see here how we create the snowflake events table.

Same is true of v_collector and v_etl, although I cannot tell you why it is this way!!

I'm fine with setting fields to null, and I understand how this helps with the wider design of incomplete events. But be aware this makes it technically possible that enrich can emit events which cannot be loaded.

cc @colmsnowplow

[oguzhanunlu]

thanks for the heads up @istreeter ! That makes me think if we should

• keep a list of atomic field names to be excluded from this check or
• take prefix of the field as much as its' limit allow or
• let it be and accept the possibility

[benjben]

In my opinion we should not set event_id, v_collector and v_etl to null, we should leave them untouched.

Contrary to atomic.events table, incomplete.events doesn't have lengths limits for the atomic fields.
Ideally we would like each field set in incomplete.events table to mean that it is valid and would be the same in atomic.events table.
But the very priority of incomplete events is to know what the problems are so that they can get fixed, and if an event can't get loaded because a required field is missing, we will never know about the problems.
If one of these three fields is too long, failure entity will tell us so that's fine.

BTW when we have removed all the lengths limits for all destinations, we will remove the checks for atomic fields lengths from Enrich.